MQTT Manually Publishing / Subscribing

Cayenne API's Bring Your Own Thing (MQTT)
1

Hi Everyone,

First of al I am not a MQTT specialist or programmer.
I try to find easy software to show the data that I collect.

I am using my raspberry (192.168.1.20) as a MQTT broker for my Tinkerforge ( [Home | Tinkerforge] (Home | Tinkerforge)) temp, humidity and air pressure sensors without connections to the raspberry GPIO

Using MQTTFX

I can subscribe to the 3 raspberry mqtt sensor data:

887×725 78.4 KB

and see the data. So I know it is working.

When configuring mqttfx with the cayenne mqtt.mydevices.com broker address and the client ID, User Name and password from the bring your own thing page, I see the device on my cayenne dashboard.

But I cannot subscribe to my raspberry sensors as the cayenne is the broker, not the raspberry(I think?)

Is there a way to use the cayenne as a MQTT CLIENT and have a kind of cayenne widget “mqtt subscriber client” where I could put the ip of my local raspberry broker, the subscriber info (in my case: tinkerforge/bricklet/barometer/vGZ/air_pressure for example ) and some field formatting to display the data of this sensor.

From what I read in the MQTT Manually Publishing / Subscribing docs, I do not see/find this possibility.
As I am not a programmer, I prefer the drag and drop, configure some fields and see the result :slight_smile:
It could be used for every mqtt sensor connected to any broker.

Is this possible and if yes, how do I do?

Another idea is to have the cayenne IOT on my local network and have the broker ip as my pi address and not the mqtt.mydevices.com address???Or am I completely wrong?

Thanks for helping.

Patrick

2

Hi @dewit281,

Thanks for the detailed post! Sorry it’s been some days since you’ve had a response.

@croczey Can you chime in here? :slight_smile:

-Benny

3

Hmm…I’m kind of confused as to what you are trying to accomplish. How MQTT works is you have a server that listens on port 1883 and clients that subscribe to and publish to the server. Subscribing will receive data from the server and publishing pushes data to the server. So…from what I read I’m getting that you want the current server/client roles reversed so the Pi is the “server” and Cayenne subscribes to it? Typically it’s not a good idea for a few reasons. 1. You have to add port forwarding to you router and 2. You now have a port open directly on the internet which can lead to security issues. If possible change your broker to client mode and connect to the Cayenne server normally.

4

Hi Adam,

Yes, it is exactly what I try; have the raspberry or other board as broker/server and the cayenne IOT as mqtt client.
Also ideally I would like cayenne IOT on my local network and not on the internet site.
Have already a VPN to my local Synology NAS and could make my own secure connection to the local cayenne IOT.

But first, is it possible to reverse the roles and how do I do this ? I do not find info on the manually publishing/subscribing part of the documentation.
How do I configure the cayenne IOT as a generic mqtt client subscriber?with the MQTTFX client??
Once subscribed, import the data to a sensor to be shown on the cayenne dashboard.
Parsing the “brute” mqtt data into the cayenne format via a small config form without any programming codes.

Advantage: any board, sensor,… could be integrated in the Cayenne IOT without any user programming.
Make the mqtt for this board-sensor work with the doc of this board and sensor (not a Cayenne problem), in the cayenne subscribe to this mqtt data, configure with “form like screen” where the parsed data (unit, value, timestamp etc,) goes to the right field to show the data in a good format on the dashboard and you have a super tool for people without any programming experience.

As long as the local cayenne IOT doesn’t work, can I at least start with the “non secure” port forwarding?
Any help, tips, workaround how to do this would be greatly appreciated.

Thanks

Patrick

5

Using the Cayenne servers as a client is not supported right now, and I don’t really think it should be. In the Cayenne world they are the server so anything sending data to Cayenne needs to be set up as a client. Generally anything collecting data and sending it to with MQTT will be set up as a client not a broker. The only scenario I can see your setup being required would be if you have created a local MQTT server on your Pi that sends out MQTT messages to clients that are subscribed, but in that case you wouldn’t need Cayenne anyway. I guess I’m still not following what you are using to send the data out. You mention MQTTFX, but what are you using to read your sensors and send the data with MQTT?

6

Hi Adam,

Thanks for the info that cayenne can not be used as client.

My sensor is a small board (trinkerforge brick with temp, humidity and air pressure sensor) with his own IP (192.168.1.41) connected to a raspberry PI (192.168.1.20).
On this Pi a small python script is running as a mqtt broker with software from tinkerforge to get the sensor data on a regular interval via my internal network.
I do not us the raspberry GPIO’s . When using whatever client (mqttfx or other) I can subscribe to the pi and get de sensor(s) data.

Saw in the “bring your own thing” doc that there is a MQTT manually publishing/subscribing paragraph so I thought it was possible to subscribe in the Cayenne software to get the raspberry mqtt data. In this paragraph they mention the mqttfx software to do this, so I installed it and tried to configure it without success. From there my forum question to see if someone could help me.

But with what you told in the last comment, I think it is a impossible task.

I loved the dashboard-sensor display used in the cayenne software.
It is much better than the “brut data dump” of the usual mqtt clients.

Hopeful that it ill be possible in the future.

Thanks for your explanations.

Patrick

7

Got it, that makes sense now. Do you have the python script? It would be very easy to connect to the Cayenne servers and send the data as a client. You can see my post here if you want to try for yourself DHT11/DHT22 with Raspberry Pi

8

Hi Adam,

I got the info from the tinkerforge site Doc | Tinkerforge and some help from the tinkerforge forum.

I installed following steps on my raspberry with latest Jessie lite to make it a mqtt broker for my sensors:

Last line is to start the Brick-MQTT-Proxy as process with 5 seconds interval to send the data

Then using mqttfx as a client on my PC, I could subscribe to see the data.
I configure the mqttfx as shown in my first post screenshots and I can see the data for the air-pressure sensor as an example.

Can this help you?

Your help is greatly appreciated.
Thanks

Patrick

9

hi @dewit281. does it have to be on your local pc? cant you just send the sensor data to your Rpi then just publish the data to Cayenne using the API - BYOT from the Rpi? it doesnt have to be connected to the pin really.
Sorry cant helped since im doing BYOT as well. i thought @adam is right, it’s much more easy.

10

Hi Adam,

Took your file and tried to make some modification to fit my settings

Code:

import paho.mqtt.client as mqtt
import time
import sys

time.sleep(30) #Sleep to allow wireless to connect before starting MQTT

mqttc = mqtt.Client(client_id=“xxxx”)
mqttc.username_pw_set(“yyyy”,password=“zzzzz”)
mqttc.connect(“mqtt.mydevices.com”, port=1883, keepalive=60)

topic_temp = “v1/username/things/clientid/data/1”
topic_humidity = “v1/username/things/clientid/data/2”
topic_air-press = “v1/username/things/clientid/data/3”

while True:
try:
temp = mosquitto_sub -v -t tinkerforge/bricklet/temperature/t6Q/temperature
humidity= mosquitto_sub -v -t tinkerforge/bricklet/humidity/uk9/humidity
air-press= mosquitto_sub -v -t tinkerforge/bricklet/barometer/vGZ/air_pressure

    if temp is not None:
        temp = "temp,c=" + str(temp)
        mqttc.publish(topic_temp, payload= ?, retain=True)

    if humidity is not None:
        humidity = "humidity,%=" + str(humidity)
        mqttc.publish(topic_humidity, payload= ? , retain=True)

    if air-press is not None:
        temp = "air-press,mBar=" + str(air-press)
        mqttc.publish(topic_tair-press, payload= ? , retain=True)

    time.sleep(5)
except (EOFError, SystemExit, KeyboardInterrupt):
    mqttc.disconnect()
    sys.exit()

“xxxx”, "yyyy"and “zzzz” I replaced by my by bring you own thong data.
But when I run the script, I got an error in line 13:

temp = mosquitto_sub -v -t tinkerforge/bricklet/temperature/t6Q/temperature I suppose I have to get the published (not subscribed) data from the sensor???

I have also trouble with the payload=

Can you put me on the right track?

Thanks

Patrick

11

I think we can go about this a little differently. Are you reading the sensor directly from the Pi that you are running this script? What is the sensor model number? We’ll get your data to the dashboard somehow haha.

12

Hi Adam,

I am running the script on the same PI that is my mqtt broker.

I got a little bit further.
I concentrated to one of my sensors, the temperature sensor.
If I type mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature at the command prompt, I get this:

pi@pi20:~/tinkerforge $ mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature
{“_timestamp”:1486541777.09539,“temperature”:2406}

so I can get the timestamp and temp value of the temp sensor and every 5 seconds I get a new value.

I modified the script (cayenne-mqtt.py) by removing the lines for humidity and air-pressure and

while True:
try:
temp = “mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature”

print temp #to see the value of temp
if temp is not None:
temp = “temp,c=” + str(temp)
mqttc.publish(topic_temp, payload= temp , retain=True)

13

sorry I did not finish my post; pressed the wrong button!!!

I receive thi every 5 seconds:
pi@pi20:~/tinkerforge $ python cayenne-mqtt.py
mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature
mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature

So I did not get the timestamp and temp value in the temp variable.
What do I do wrong?

is this a wrong variable type issue or do I make a concept error?

I also put the same “temp” in the mqttc.publish payload.(last line)
is this ok?

Thanks for your help

Patrick

14

“mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature” is a bash command but when you put that in your python code it’s just reading it as a string. Try this:

import paho.mqtt.client as mqtt
import time
import sys
import subprocess

time.sleep(30) #Sleep to allow wireless to connect before starting MQTT

mqttc = mqtt.Client(client_id="xxxx")
mqttc.username_pw_set("yyyy",password="zzzzz")
mqttc.connect("mqtt.mydevices.com", port=1883, keepalive=60)

topic_temp = "v1/username/things/clientid/data/1" #change username/clientid here
topic_humidity = "v1/username/things/clientid/data/2" #change username/clientid here
topic_air-press = "v1/username/things/clientid/data/3" #change username/clientid here

while True:
try:
temp = subprocess.Popen(mosquitto_sub -v -t tinkerforge/bricklet/temperature/t6Q/temperature)
humidity= subprocess.Popen(mosquitto_sub -v -t tinkerforge/bricklet/humidity/uk9/humidity)
air-press= subprocess.Popen(mosquitto_sub -v -t tinkerforge/bricklet/barometer/vGZ/air_pressure)

          
    if temp is not None:
        temp = "temp,c=" + str(temp)
        mqttc.publish(topic_temp, payload= ?, retain=True)

    if humidity is not None:
        humidity = "humidity,%=" + str(humidity)
        mqttc.publish(topic_humidity, payload= ? , retain=True)

    if air-press is not None:
        temp = "air-press,mBar=" + str(air-press)
        mqttc.publish(topic_tair-press, payload= ? , retain=True)

    time.sleep(5)
except (EOFError, SystemExit, KeyboardInterrupt):
    mqttc.disconnect()
    sys.exit()
15

Hi adam,

Added line import subprocess.
Modified the clientid in the topic_temp to the clientid value (xxxx) I received from cayenne without any quotes
Modified temp to
temp = subprocess.Popen(mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature)

Changed the payload value to temp:
if temp is not None:
#temp = “temp,c=” + str(temp)
mqttc.publish(topic_temp, payload= temp , retain=True)

saved the file and ran it:

pi@pi20:~/tinkerforge $ python cayenne-mqtt.py
Traceback (most recent call last):
File “cayenne-mqtt.py”, line 16, in
temp = subprocess.Popen(mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature)
NameError: name ‘mosquitto_sub’ is not defined

And receive this error message above.

But I see the “bring your own thing dashboard” jumping from offline to something else in a fraction of seconds until the script stops with this error.
So I think, the connection to cayenne dashboard is working, but no sensor data yet.

Again, many thanks for helping me.

Patrick

16

oh, sorry. Put quotes around the command

temp = subprocess.Popen(“mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature”)

17

Hi Adam,
Added the quotes and ran cayenne-mqtt.py

pi@pi20:~/tinkerforge $ python cayenne-mqtt.py
Traceback (most recent call last):
File “cayenne-mqtt.py”, line 16, in
temp = subprocess.Popen(“mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature”)
File “/usr/lib/python2.7/subprocess.py”, line 710, in init
errread, errwrite)
File “/usr/lib/python2.7/subprocess.py”, line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

thought it was a python 2 vs python 3 problem so ran:

pi@pi20:~/tinkerforge $ python3 cayenne-mqtt.py
Traceback (most recent call last):
File “cayenne-mqtt.py”, line 1, in
import paho.mqtt.client as mqtt
ImportError: No module named ‘paho’
pi@pi20:~/tinkerforge $

As a absolute enthusiast beginner with python and after some Google search on both errors, I am stuck.
I had an error about mixing spaces and tabs that I could resolve.
but those errors are to much for me.

What do I do wrong?

Thanks

Patrick

18

Sorry about that again…not on a good start here haha. Popen requires more than 1 argument, I was thinking of subprocess.call when I put that in there. Try:

temp = subprocess.Popen(“mosquitto_sub”, “-t /tinkerforge/bricklet/temperature/t6Q/temperature”)

or

temp = subprocess.call(“mosquitto_sub -t /tinkerforge/bricklet/temperature/t6Q/temperature”, shell=True)

19

Hi Adam,
With the first one solution I receive a error: buffersize must be an integer
The second one seems to be working.
here is a printscreen from the moment I logon to PI with putty:

login as: pi
pi@192.168.1.20’s password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Feb 12 09:21:51 2017 from 192.168.1.2
pi@pi20:~ $ cd /home/pi/tinkerforge
pi@pi20:~/tinkerforge $ python brick-mqtt-proxy.py --brickd-host 192.168.1.41 --brickd-port 4223 --broker-host localhost --broker-port 1883 --update-interval 5&
[1] 2151
pi@pi20:~/tinkerforge $ python cayenne-mqtt.py
{“_timestamp”:1486889999.273564,“temperature”:2456}
{“_timestamp”:1486890034.483148,“humidity”:272}
{“_timestamp”:1486890034.895567,“air_pressure”:1017743}
{“_timestamp”:1486890039.948072,“air_pressure”:1017742}
{“_timestamp”:1486889999.273564,“temperature”:2456}
{“_timestamp”:1486890034.483148,“humidity”:272}
{“_timestamp”:1486890039.948072,“air_pressure”:1017742}
{“_timestamp”:1486890045.001824,“air_pressure”:1017750}
{“_timestamp”:1486890045.001824,“air_pressure”:1017750}
{“_timestamp”:1486889999.273564,“temperature”:2456}
{“_timestamp”:1486890034.483148,“humidity”:272}
{“_timestamp”:1486890045.001824,“air_pressure”:1017750}
{“_timestamp”:1486890049.521626,“humidity”:271}
{“_timestamp”:1486890049.521626,“humidity”:271}
{“_timestamp”:1486890049.521626,“humidity”:271}
{“_timestamp”:1486890050.097411,“air_pressure”:1017752}
{“_timestamp”:1486890050.097411,“air_pressure”:1017752}
{“_timestamp”:1486890050.097411,“air_pressure”:1017752}
{“_timestamp”:1486889999.273564,“temperature”:2456}
{“_timestamp”:1486890049.521626,“humidity”:271}
{“_timestamp”:1486890050.097411,“air_pressure”:1017752}
{“_timestamp”:1486890055.1549,“air_pressure”:1017749}
{“_timestamp”:1486890055.1549,“air_pressure”:1017749}
{“_timestamp”:1486890055.1549,“air_pressure”:1017749}
{“_timestamp”:1486890055.1549,“air_pressure”:1017749}
^Cpi@pi20:~/tinkerforge $

first of allI have two questions:

  1. I have to manually start the broker at the command prompt every time I restart the pi with :

    cd /home/pi/tinkerforge
    python brick-mqtt-proxy.py --brickd-host 192.168.1.41 --brickd-port 4223 --broker-host localhost --broker-port 1883 --update-interval 5&

can I be done automatically? I tried to put the commands in a .sh file and put it the crontab.(found this on Google). But only the python brick-mqtt-proxy.py start without the --brickd-host 192.168.1.41 --brickd-port 4223 --broker-host localhost --broker-port 1883 --update-interval 5& arguments. So I do not have the 5 seconds refreshment but one value.

2.is this normal that with the 5 seconds interval (from question 1), I receive every 5 seconds the values of my 3 sensors in random order and sometimes more than once; as example see the last 4 lines of my print screen; 4 identical timestamps with 4 identical air_pressure values.
Is this normal mqtt behavior?

Then I went to the cayenne dash board and the dashboard went from

858×528 56.1 KB

to this screen as soon I a ran the python cayenne-mqtt.py command at the command prompt.

856×530 48.8 KB

Yellow band at the top disappeared, so I think communication is working but still no sensor data.

how do I fix this?

Thanks

Patrick

20

Ok, that looks much better. Lets see your full updated script again and go from there. As far as running it on boot, you can use cron to do it. Put this line in your crontab:

“@reboot python /home/pi/tinkerforge/brick-mqtt-proxy.py --brickd-host 192.168.1.41 --brickd-port 4223 --broker-host localhost --broker-port 1883 --update-interval 5&”

I usually include an " &" at the end but it’s not necessary.