Timestamped data


#1

My device makes multiple measurements available with each sample. It also generates a timestamp for the group of measurements. When I send the data to Cayenne I have to slit up the group into separate MQTT messages. How do I send data to Cayenne with the timestamp info so that the group appears in Cayenne charts at the sampling time, not the time Cayenne received the MQTT message? Is there any way to send the entire group of readings with one timestamp? Maybe a JSON message with everything?


#2

@eptak tagging for this.


#3

Yes:

v1/USERNAME/things/THING_ID/data/json
payload is of this format

[
     {"channel":100,"type":"rssi","unit":"dbm","value":10,"name":"RSSI"},
     {"channel":101,"type":"snr","unit":"db","value":10,"name":"SNR"},
     {"channel":1,"type":"temp","unit":"c","value":10,"name":"Temperature"},
     {"channel":2,"type":"rel_hum","unit":"p","value":10,"name":"Humidity"},
     {"channel":4,"type":"lum","unit":"lux","value":10,"name":"Luminosity"},
     {"channel":5,"type":"motion","unit":"d","value":motion,"name":"Motion"}, 
     {"channel":6,"type":"co2","unit":"ppm","value":10,"name":"CO2"},
     {"channel":107,"type":"voltage","unit":"v","value":8,"name":"VDD"}
  ]

#4

I have pondered this question, and notwithstanding the database clog problems before and slightly after April 12, the data moves quite quickly through the internet and into the database within 1 second, and is timestamped when it arrives. And since this is usually 15 second data, the “within a second” is probably adequate.

This is an 8266 series, that goes through wifi, then fiber internet Canadian phone company to the server in LaLa Land (I’m guessing), with the Cayenne server time on the left, and my 8266 time on the right, and they are usually within a second.

If there is bad internet on either end, or other problems, they could diverge.

On your response adam with v1/USERNAME/things/THING_ID/data/json, is that a mqtt command that can send a multi-item update to the server? I assume if I am using the mqtt arduino library with the virtualwrite command, I am sending each datapoint separately. Is there a virtualwrite for multiple points? Or do I have to build my own json, and send it through the http post/get/etc system and bypass the cayenne mqtt library?

As an aside, I would be interested in the secret document all all the REST and MQTT interactions that your servers can accommodate, and are legal and kosher. I have been bypassing the summary database to get some non-summarized data, now and then, but have to reverse engineer it from the commands my browser is sending. :grinning:


#5

You will have to send the payload manually. I still have a project here that is not using the Cayenne library (instead uses paho), you can get an idea of how to send the data there.

topic = "v1/" + username + "/things/" + clientid + "/data/json"
payload = '[{"channel":100,"type":"rssi","unit":"dbm","value":10,"name":"RSSI"},{"channel":101,"type":"snr","unit":"db","value":10,"name":"SNR"},{"channel":1,"type":"temp","unit":"c","value":10,"name":"Temperature"},{"channel":2,"type":"rel_hum","unit":"p","value":10,"name":"Humidity"},{"channel":4,"type":"lum","unit":"lux","value":10,"name":"Luminosity"},{"channel":5,"type":"motion","unit":"d","value":motion,"name":"Motion"}, {"channel":6,"type":"co2","unit":"ppm","value":10,"name":"CO2"},{"channel":107,"type":"voltage","unit":"v","value":8,"name":"VDD"}]'
mqttc.publish(topic, payload=payload , retain=True)

*I didn’t test the format of the payload varaible in Python. It’s been a long time since I did any Python programming so take that as a hint rather than copy this and it will work


#6

I got that format from @ecoqba on Slack. Maybe he can help if you have any other questions.


#7

Thanks for the great JSON syntax. Now all the widgets appear at once, very nice! All the data gets the same timestamp.

Is there any way to set the timestamp used by Cayenne? I understand the Internet speeds can be quick, but my full network is not. For example, timestamped data may get queued up while a modem comes online and connects. I send the data to a time indexed database, and to Cayenne, and sometimes other sites. I need the time values to correspond to those in the data, not when it showed up at Cayenne, so it can be synchronized with the database and other sites.

Maybe a valid “type” could be a timestamp? Or could one of the JSON fields on each channel be “timestamp”?


#8

For everyone trying to use any of the LPWAN technologies and only connecting to the network at periodic times, this can be real problem. Some way to capture the time that the sample was taken would be great.


#9

You can send your own time with extra virtualwrite’s.

Just note the time when you read your sensor, and the delay in internet travel and the variance in delay are avoided. Conceivably they could add a “time” field, just like a “unit” field, which would be better.

Here is some code (arduino code for 8266)

// ~~~~~ declarations ~~~~~
#include <time.h>

int timezone = -6; // -6 for daylight savings, -7 regular mountain
struct tm * timeinfo;
time_t boottime;
time_t now;

// ~~~~~ setup ~~~~~

configTime(timezone * 3600, 0, “pool.ntp.org”, “time.nist.gov”);
Serial.println("\nWaiting for time");
while (!time(nullptr)) {
Serial.print(".");
delay(1000);
}
Serial.println("");
now = time(nullptr);
Serial.println(ctime(&now));
boottime = now;

// ~~~~~ CAYENNE_OUT_DEFAULT ~~~~~
now = time(nullptr);
// Cayenne.virtualWrite(1, now, “UnixTime”, “Seconds”); // not UTC
long seconds = now - boottime;
hours = seconds / 3600.0;
Cayenne.virtualWrite(0, hours, “Uptime”, “Hours”);
time_t now = time(nullptr);
Serial.print(ctime(&now));
timeinfo = localtime (&now);
float timefloat = timeinfo->tm_hour*100 + timeinfo->tm_min + timeinfo->tm_sec/100.0;
Cayenne.virtualWrite(2, timefloat, “Time”, “HHMM.SS”);


#10

That will just create a new Cayenne datapoint with time information, and it will be timestamped by Cayenne when it receives the message. I am trying to associate a timestamp with the data being sent, so when it is shown in a chart, the time information on the X axis is the time the data was collected, not the time when Cayenne receives the message.


#11

I agree – not a perfect solution, but … not bad. :grinning: