Using Node-RED as a Local Fallback Server


#1

Required items:

Raspberry Pi with the newest image of Raspbian Jessie (Node-RED is pre-installed)
A client device (Arduino, Pi, or any other microcontroller that has an MQTT library)
Arduino IDE

Overview:

Cayenne will be a display for our data and easy to use interface.
The Raspberry Pi will be a local server for our client device to send data to. Through Node-RED we can create functions that are currently missing from Cayenne such as time delay buttons, GPIO interrupts, and add sensors that are currently not supported for Raspberry Pi.

Setup:
Cayenne:
Log in to your dashboard and click Add new…>Device/Widget
Click the Bring Your Own Thing button - keep this window open for later
That’s it for now!

Raspberry Pi server:
Install a local MQTT server using apt-get:

sudo apt-get install mosquitto

Set up Node-RED to start on boot using the command below:

sudo systemctl enable nodered.service

Now you can either start Node-RED by rebooting or using “sudo service nodered start”

The Node-RED web interface can be accessed by going to your Pi’s IP address and appending :1880 to the end (ex. http://192.168.0.10:1880)

For now we will create a flow to receive an MQTT command from Cayenne and another to send a value to Cayenne using MQTT. The very basic flow below can be imported in to Node-RED by clicking Settings (three lines in the top right)>Import>Clipboard then click where you want to place it on the flow tab.

[{"id":"bb4f8d87.5978a8","type":"mqtt in","z":"841dbd34.248ee8","name":"Cayenne Input","topic":"See Post","qos":"2","broker":"94fad99f.8e2728","x":120,"y":540,"wires":[["d8185bc2.e76a78","3369d01a.b20798"]]},{"id":"d8185bc2.e76a78","type":"debug","z":"841dbd34.248ee8","name":"","active":false,"console":"false","complete":"false","x":350,"y":500,"wires":[]},{"id":"56762e2e.625f48","type":"inject","z":"841dbd34.248ee8","name":"Send Temp Value of 20c","topic":"","payload":"temp,c=20","payloadType":"str","repeat":"","crontab":"","once":false,"x":150,"y":600,"wires":[["5d89c061.6d73"]]},{"id":"5d89c061.6d73","type":"mqtt out","z":"841dbd34.248ee8","name":"Cayenne Output Channel 2","topic":"See Post","qos":"0","retain":"true","broker":"94fad99f.8e2728","x":400,"y":600,"wires":[]},{"id":"6648df8b.44f7f8","type":"mqtt out","z":"841dbd34.248ee8","name":"Cayenne Output Channel 1","topic":"See Post","qos":"0","retain":"true","broker":"94fad99f.8e2728","x":780,"y":520,"wires":[]},{"id":"3369d01a.b20798","type":"function","z":"841dbd34.248ee8","name":"Send Back Received Value","func":"msg.payload = msg.payload.slice(-1);\n\nreturn msg;","outputs":1,"noerr":0,"x":400,"y":540,"wires":[["6648df8b.44f7f8","1656dfff.8cb4e8","30b1895d.a1218e"]]},{"id":"1656dfff.8cb4e8","type":"debug","z":"841dbd34.248ee8","name":"","active":true,"console":"false","complete":"false","x":730,"y":480,"wires":[]},{"id":"c42d2749.18218","type":"mqtt in","z":"841dbd34.248ee8","name":"Arduno Input","topic":"arduino_1_data","qos":"2","broker":"58cb32ce.d41f54","x":110,"y":660,"wires":[["919819a4.c4f0e8"]]},{"id":"919819a4.c4f0e8","type":"function","z":"841dbd34.248ee8","name":"Send Data Received From Client To Cayenne","func":"var MQTTusername = global.get(\"MQTTusername\");\nvar MQTTclientID = global.get(\"MQTTclientID\");\nvar msgarray = msg.payload.split(',');\nvar channel;\n\nfor (i = 0; i < msgarray.length; i++){\n    if (msgarray[i].match(/channel/gi)){\n        channel = msgarray[i].split('=')[1];\n        msgarray.splice(i, 1);\n    }\n}\n\nmsg.payload = msgarray.join(\",\");\nmsg.topic = \"v1/\" + MQTTusername + \"/things/\" + MQTTclientID + \"/data/\" + channel;\n\nreturn msg;","outputs":1,"noerr":0,"x":460,"y":660,"wires":[["e17e25f8.3d3508"]]},{"id":"e17e25f8.3d3508","type":"mqtt out","z":"841dbd34.248ee8","name":"Cayenne Output Arduino Input","topic":"","qos":"0","retain":"true","broker":"94fad99f.8e2728","x":790,"y":660,"wires":[]},{"id":"30b1895d.a1218e","type":"mqtt out","z":"841dbd34.248ee8","name":"Client Output","topic":"CayenneInput","qos":"0","retain":"true","broker":"58cb32ce.d41f54","x":730,"y":580,"wires":[]},{"id":"736cbb2b.50afec","type":"inject","z":"841dbd34.248ee8","name":"Setup","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":true,"x":90,"y":420,"wires":[["f1692282.cd432"]]},{"id":"f1692282.cd432","type":"function","z":"841dbd34.248ee8","name":"Set Global Variables","func":"global.set(\"MQTTusername\",\"See Post\");\nglobal.set(\"MQTTclientID\",\"See Post\");\n\nreturn msg;","outputs":1,"noerr":0,"x":380,"y":420,"wires":[["131e7e5a.1370ba"]]},{"id":"131e7e5a.1370ba","type":"debug","z":"841dbd34.248ee8","name":"","active":false,"console":"false","complete":"false","x":730,"y":420,"wires":[]},{"id":"94fad99f.8e2728","type":"mqtt-broker","z":"","broker":"mqtt.mydevices.com","port":"1883","clientid":"See Post","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"willTopic":"","willQos":"0","willPayload":"","birthTopic":"","birthQos":"0","birthPayload":""},{"id":"58cb32ce.d41f54","type":"mqtt-broker","z":"","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"willTopic":"","willQos":"0","willPayload":"","birthTopic":"","birthQos":"0","birthPayload":""}]

After placing the flow edit the Set Global Variables node with your MQTT Username/Password listed on your Cayenne dashboard for the device we added earlier.

Edit the Cayenne Input node by double clicking it. Change the topic to fit your settings, the channel should be 1. Below is the breakdown of what the topic means:

v1/username/things/clientID/cmd/channel
v1 - This is the API version which for now will always be v1
username - Change this to the MQTT username listed on your Cayenne dashboard for the device we added earlier
things - This can stay things - not exactly sure what this is
clientID - Change this to the Client ID listed on your Cayenne dashboard for the device we added earlier
cmd - This is the type of message.  cmd is a command message from the Cayenne server
channel -  This is where the data ends up on the dashboard.  Each widget must have a unique channel

In the end your Topic string should look like this:

v1/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/things/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/cmd/X

After the topic is set click on the pencil next to the server. On this page paste in your Client ID in the Client ID field then click the Security tab and paste in your username and password from the Cayenne dashboard.

Next edit the Cayenne Output Channel 1 node by double clicking it. Change the topic to fit your settings. You can use basically the same topic from the input node, but this time we want to change the message type to digital to tell the Cayenne server we received the command so it updates the button on the dashboard. The channel should be 1. Your Topic string should look like this:

v1/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/things/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/digital/X

Finally, edit the Cayenne Output Channel 2 node by double clicking it. Make the same changes as the Cayenne Output Channel 1 node but notice the second to last parameter is data instead of digital this time. Sending the type data tells the server we are sending it a value to update a widget. The channel should be set to 2.

v1/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/things/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/data/X

After the 2 MQTT nodes are configured you can click Deploy to save the changes. At this point the MQTT settings page should go away and take you to a blank dashboard. You can test that communication to the server is working by clicking the blue button to the left of the Send Temp Value of 20c inject node. You should see a temporary widget appear on the blank dashboard. From here if you want to make it permanent just click the + in the upper right corner.

To test that values are being received from the server, click Add new…>Device/Widget>Custom Widgets>Button. Enter the following settings:

Name - Enter a name.  Button is a good choice
Device - Select your MQTT device
Channel - Enter a 1
Icon - Whatever your preference.  Light is usually the one I select

Click Add Widget then click on the button that shows up on the dashboard. Go back to Node-RED and click on the Debug tab in the top right. You should see two debug messages:

XXXXXXXXXXXXXXX,1 which means your message was received. 1 means the button was turned on and 0 means it was turned off. The first part of the message is a random sequence.
1 - 1 means the button was turned on and 0 means it was turned off. This is the value sent back to the server to verify the command was successful and update the button on the dashboard.

Client Device:
Arduino:
For now I’ll just go over configuring an ESP8266 device to send data to our local server (Raspberry Pi coming soon?)
Copy this sketch to your Arduino (I’m using an ESP8266 - you may have to include more libraries for your device). This sketch simply sends some hard coded values to your dashboard to verify everything is working. Note - you will have to change SSID to your wireless network name, PASSWORD to your wireless password, and SERVER_IP to the IP address of your Pi.

#include <PubSubClient.h>
#include <ESP8266WiFi.h>

const char* ssid = "SSID";
const char* password = "PASSWORD";
const char* mqtt_server = "SERVER_IP";

WiFiClient espClient;
PubSubClient client(espClient);
long lastMsg = 0;
char msg[50];
int value = 0;

void setup() {
  int timeoutcounter = 0;
  
  Serial.begin(9600);
  WiFi.begin(ssid, password);

  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  while (WiFi.status() != WL_CONNECTED) {
    timeoutcounter++;
    delay(500);
    Serial.print(".");
    if (timeoutcounter >= 30){
      Serial.println("Failed to connect to wireless - sleeping for 5 minutes");
      delay(100);
      ESP.deepSleep(300000000, WAKE_RF_DEFAULT);
      delay(100);
    }
  }
  
  Serial.println("");
  Serial.println("WiFi connected");

  client.setServer(mqtt_server, 1883);
  client.setCallback(callback);
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client")) {
      Serial.println("connected");
      client.subscribe("CayenneInput");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Sleep for 5 minutes before retrying
      Serial.println("Failed to connect to MQTT server - sleeping for 5 minutes");
      delay(100);
      ESP.deepSleep(300000000, WAKE_RF_DEFAULT);
      delay(100);
    }
  }
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i=0;i<length;i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  float t = 20.2;
  float h = 30.3;
  float hif = 40.4;
  
  String mqtt_message = "temp,f=" + String(t) + ",channel=10";
  mqtt_message.toCharArray(msg, 50);

  Serial.print("Publish message: ");
  Serial.println(msg);
  client.publish("arduino_1_data", msg);

  mqtt_message = "rel_hum,rel_hum=" + String(h) + ",channel=11";
  mqtt_message.toCharArray(msg, 50);

  Serial.print("Publish message: ");
  Serial.println(msg);
  client.publish("arduino_1_data", msg);

  mqtt_message = "temp,f=" + String(hif) + ",channel=12";
  mqtt_message.toCharArray(msg, 50);

  Serial.print("Publish message: ");
  Serial.println(msg);
  client.publish("arduino_1_data", msg);
  delay(5000);
}

After you upload the code to your ESP you should see 3 new widgets on your MQTT device dashboard. Here you can click the + on the widget to add it permanently, or just use this as a validation that everything is working. From here you can modify the sketch to send any data you want to the dashboard. There are a couple important parts to sending data to the dashboard. If you are using my Node-RED flow, the string you send to the Node-RED server must have 3 comma separated values: message type, value, and channel (ex. “temp,f=20.2,channel=10”) For the value you must include the unit value from the list of data types in the Cayenne docs. Here is a list of the valid data types and units: http://www.cayenne-mydevices.com/docs/#bring-your-own-thing-api-supported-data-types

To receive data from Cayenne click the button we created earlier. This will send a 0 or a 1 to your ESP through the CayenneInput MQTT Topic. To act on the received data from Cayenne you will have to modify the callback function. In here you can do anything, put the device to sleep, toggle a GPIO pin, flash an LED, etc. The possibilities are endless. The one downfall you will notice is that the received data is only process once per loop. There are some ways around this, but I won’t get in to that now. I will most likely update this post to include these changes.

I’ll add to this post over time. Right now I have planned to add a section on using the Raspberry Pi with this setup and some other cool stuff like adding in your own features such as the time delay button. If there is anything you would like to see here just post a request.


DHT11, DHT22 Sensors
Local version of Cayenne project
New virtual sensor/device as generic MQTT
MQtt send commands to change state
Custom HTML front-end
Cayenne to node-red mqtt server issues
Node-red to Cayenne
Mxl 90614 sensor
ESP8266 "RAW" MQTT messages using the PubSub Library question
MQTT Dash Phone Tool Setup Help
Rasp p3 + arduino
Mysensors Gateway
Local MQTT broker bridging to Cayenne
Sigfox to Cayenne
Email trigger for events
Question about PIR and Light
Esp8266
SONOFF button to change relay state
Offline behaviour
Line Graph - no data available
Triggering events from the outside
#2

@adam Fantastic informative post, thank you Adam! I want to use Cayenne as part of my dissertation but was worried that the criticism for the system would be that should the internet link fail the system wouldn’t function. This largely solves that issue, just a shame that Node Red seems to need quite a bit of learning compared to the easiness of setting things up in Cayenne!


Node-RED / MQTT API
#3

Yes, Node-RED has quite the learning curve. I’ve been using it for about 2 years now and still learning about new ways to do things. If you need help with anything Node-RED let me know.


#4

Thanks @mike306dt for spotting the missing Set Global Variables part of the tutorial which cases the data to not make it back to the Cayenne server. I updated the project to include it.


#5

Hi Adam,

Thank you very much for investing efforts into community education.

Couple of days I am struggling to resolve problems related with launching of nodered service. Mainly, problems are related to screwed nodejs environment on Raspbian Jessie v8. Googled got solution and tried tens of them. Nothing helps. Nodered seems impossible option on my R-Pi.

Is there a chance to connect Mosquitto broker with Cayenne without nodered?


#6

As far as the node-red issues there are a few way to fix that. The simplest is to just get a fresh image of Raspbian on an SD card since the new version has Node-RED installed by default. You can also use the following commands to uninstall and reinstall NodeJS and Node-RED:

node-red-stop
sudo apt-get remove nodered
sudo apt-get remove nodejs nodejs-legacy
sudo apt-get remove npm   # if you installed npm
curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
sudo apt-get install -y build-essential python-rpi.gpio nodejs
sudo npm cache clean
sudo npm install -g --unsafe-perm  node-red

You can also send MQTT messages without Node-RED but the process to do that depends on what you plan to send them with.