CaySentinel - REST API demo for ESP 8266


#1

Here is a REST API demo for ESP 8266 with Arduino C.

The genesis of this program is to solve the problem of a 8266 rebooting from a power failure and just having its original program, but no knowledge of what happened before the outage. The simple solution is that the 8266 would write its state to the EPROM every 15 seconds, but that would exhaust the 100,000 write life of the EPROM in about a week or two. So the better solution is that the 8266 would query cayenne to see what was happening the last time that the 8266 reported its status.

As well, I have been fixated on the time gap between sending data, and the timestamp that data has when I read it back from the database.

Normally the simple controller like 8266 just sends its readings through MQTT up the chain of command, but with REST the 8266 can fetch information from Cayenne just like the Windows/Android/etc systems that run the user interface or the supervisory control.

So this 8266 program reboots by asking Cayenne when it last reported its status (the time), and how many times its has rebooted (a counter), and the status or a 2-State and a Slider. The program simulates a machine that produces two sine waves. The machine may be on or off (the 2-state) and the sine wave might be 1 to 6 cycles per hour (the slider). The second sine wave runs one cycle per hour.

The main REST query requires an OAuth2 access tokenand is called as follows:

int rest_query_cayenne(device, sensor)

returning 1 if successful, along with global variables v, ts, device_type and unit matching the json that is returned by cayenne,

{“v”:“1526700604”,“ts”:“2018-05-19T03:30:05.283Z”,“unit”:“seconds”,“device_type”:“analog”}

The query works for regular data points, as well as sliders, but it does not seem to work for “buttons”, so you cannot find out if the current state of a button is 0 or 1, but you can echo that button event back to cayenne as a 2-state, so you can see on the dashboard if the 8266 got the command and turned the machine or the relay to the state that you wanted it.

I also did a REST write

int rest_update_cayenne(channel, v, type, unit)

returning 1 is successful and following the cayenne virtualwrite call.

MQTT is a simpler form of write, but the REST write has the merit that cayenne echoes the write back to the 8266 so you will get a message in CAYENNE_IN_DEFAULT() and you can confirm that cayenne got the information.

I could not find any reliable REST implementation for 8266, but there are good SSL HTTP and a JSON functions.

To use MQTT you just need to the username, password, device_id, and channel number that are created by cayenne when you add a sensor, which are numbers of the form ea767930-5b0f-11e8-9907-5defa4aa8de2 (and 1,2,3 for channel number) and fixed.

But for REST you need an OAuth2 access_token - a 500ish digit number - and a OAuth2 refresh token - a 50ish digit number - which are generated using your cayenne identity of the for mrpeanut@gmail.com and peanutpassword. Since you don’t want your email and password hardcoded into the 8266, this program uses the refresh_token that you can generate with your email/password, and that is hardcoded into the program, and then the program generates the access_token every time it reboots, and also once per day, and keeps refresh_token in the the EPROM, as the OAuth2 security expires the access_token, but will give you a new access_token and refresh_token without needing your email and password.

So this program starts with 4 queries

  • last time reported
  • how many boots
  • machine on/off
  • cycle frequency

and then the once per minute sends the current time to cayenne using MQTT and REST, and queries cayenne for the time from the previous minute. Since it knows when the last time was sent, it can calculate the difference between the time when the send was done, and the timestamp in the cayenne database. I call this the Lag time, and with my internet the MQTT send is timestamped about 1 second after I sent it, and the REST send is timestamped about 4 seconds after the send. And since the REST send is immediately echoed back to the 8266, I calculated the milliseconds it took to make 2 journeys across the internet, which is usually about 3 seconds. All this values are reported back to cayenne (using MQTT).

As well, if the “machine is on”, it sends the 2 sine waves every 15 seconds.

I shared the dashboard, including the controls to turn on/off the machine and changes the frequency - so you can control the “sine wave machine” from anywhere in the world! Find it at this link

https://cayenne.mydevices.com/shared/5b31c1877754a2057f6aa182/project/23b659ad-e01d-4f02-888a-4701dcd11d47

which should look like this:

In this snapshot, the sine frequency was changed half way through the hour, and the lag mqtt is around 1 second, lag rest around 4 seconds, and rest roundtrip around 3000 milliseconds.

These times do fluxuate with internet problems and such, you can see by backing through the hours and looking at daily results.

Here is an example with there is a flaw in the data transmission (lower graph), and you can see it in the simulated data (upper graph). Also at the right side of that graph the “machine” is shut off, so there are no datapoints in the sine wave for a few minutes.

You can find the program at

The functions that do the REST API queries and sends need the other functions that do the OAuth2 token fetching and renewing. If you want to run the program as it is, you will need to do get your refresh_token (from your cayenne username/password), and also get your mqtt username/password and get your device and sensor numbers and edit all that into the program … so not trivial, but do-able.

The 8266 console gives you a running update

cayenne_rest_mqtt_v2.3  Aug 11, 2018 -jz
Free Heap 35896
>>>>> Starting Cayenne ...

Waiting for Internet time....
Sat Aug 11 16:55:14 2018

Get took 0
Read from EEPROM: 
76df3fa74f1peanut5b284d95c7085854e4a56
1534006277
<<<<< Update OAuth2 token

at=eyJhbGc....49f3
Refreshed access_token and refresh_token at 1534006517
Writing to EPROM ...
a691115peanut1bc1a5b4d552fa4cd3e7d449f3
1534006517
Put took 35 ms, bytes = 844
Fetching parameters from cayenne about last known state ...


Query cayenne for last running time 
   Device: ea767930-5b0f-11e8-9907-5defa4aa8de2
   Sensor: 6c6d96b0-6a1c-11e8-8e48-275f329dc9d5
Results from cayenne 
      v = 1534006464.00
     ts = 2018-08-11T16:54:24.739Z
 device = analog
   unit = seconds

8266 was down for 57.00 seconds or 0.02 hours

Total reboots = 50.00
Machine is ON, = 1.00
cycles is = 6.00

     >>>>> MQTT Time Write 1534006533
     >>>>> REST Time Write 1534006535
<<<<< rest time Send has arrived 2950 MilliSeconds

<<<<< MQTT Time Lag -0.42
     >>>>> MQTT Time Write 1534006594
<<<<< REST Time Lag -4.01
     >>>>> REST Time Write 1534006596
<<<<< rest time Send has arrived 2980 MilliSeconds

<<<<< MQTT Time Lag -0.41
     >>>>> MQTT Time Write 1534006655
<<<<< REST Time Lag -4.22
     >>>>> REST Time Write 1534006658
<<<<< rest time Send has arrived 3174 MilliSeconds

ok, i spent more time on this post than the program.
Not really, but further info will be as requested.


Sharing value of device 1 into device 2
#2

Nice post @jameszahary


#3

https://cayenne.mydevices.com/shared/5b7a457a63015d03ba5e91c6/project/ef5692cb-85d0-4f65-a4ac-e749b5a14c62

I rewrote the CaySentinel program in nodejs as the first step in a Raspberry PI system to drive a display that doesn’t need a PC or Android phone … and I discovered some interesting things.

The previous version was on an 8266 and the results suggested that it took about 1 second for MQTT to get my data timestamped in Cayenne, and about 4 seconds for REST to do the same. And about 3 seconds for a REST update from the 8266 to travel to cayenne and be sent back and received by the 8266. This data is displayed on the third graph, and the blue line on the top graph.

Those numbers were not great, but reasonable I thought.

But when running with nodejs on a big PI (second graph) it appears that nodejs/pi will send MQTT data and get it timestamped in cayenne in about 50 ms (versus 1000 ms for 8266), and REST will send data and get it timestamped in cayenne in about 500 ms (versus 4000 ms for 8266). As well, the roundtrip time for a REST update is about 300 ms on nodejs/pi versus 3000 ms on 8266.

Further, on the left side of the second graph, the nodejs was running on a Windows i5 PC, and clearly the MQTT improves from 225 ms to the 50 ms for the PI. The REST system takes about the same time, but has higher variance on the PI ???

So, my conclusion is that the 8266 is really slowing things down. 80 MHz on a dedicated CPU is nice quick action on sensors, but handing control to wifi code and the cayenne.loop(), seems to add a second or two to the process of moving the data, where the PI and its modern kernel can move things out the door much quicker.


#4

you can try with 160MHz max frequency and see what is the difference but it will be still less then pi.
i still prefer esp8266 over pi :wink:


#5

Yes I’m a fan of 8266 too, but I think it is the Arduino C mini-kernel that is slowing it down rather than the raw speed. Have you tried Mongoose or other mini-os for 8266?


#6

Had gone through Mongoose last year but dint dig into it much.