Flow to turn SerialGateway into MQTT-SerialGateway

[cranky]

After installing and working with Mosquitto, I realized that this was going to be easier than implement a SQlite db. However, I already have a SerialGateway and don't have an i2c/spi ethernet device for my microcontrollers. Not a big deal; I have Node-Red.

No, why would I want Mosquitto? It's the MQTT broker, and a really good lean one. The configuration sucks, but time for learning, eh? Ideally, instead of having "In-Serial" and "Out-Serial" littered all around in your code, you can have clear MQTT topic subscriptions and topic outputs in your code.

It also allows you to use other Controllers as well. Let's get started.

1. Install Mosquitto for your platform.
2. (optional) Configure Mosquitto for security. Login/pass and generate SSL certs. If this is public facing, you will need LetsEncrypt certs for encrypted Websockets
3. Node-Red flow:

[{"id":"72a5a797.020098","type":"mqtt-broker","z":"b2cff5f2.9cc8b","broker":"localhost","port":"8883","clientid":"","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":"15","cleansession":true,"willTopic":"","willQos":"0","willRetain":null,"willPayload":"","birthTopic":"","birthQos":"0","birthRetain":null,"birthPayload":""},{"id":"fe37ebc7.c69d68","type":"serial-port","z":"b2cff5f2.9cc8b","serialport":"/dev/ttyUSB0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false},{"id":"1d5d1eec.5b3009","type":"debug","z":"b2cff5f2.9cc8b","name":"Serial to MQTT debug","active":false,"console":"false","complete":"true","x":635,"y":48,"wires":[]},{"id":"6c0c081e.aac08","type":"serial in","z":"b2cff5f2.9cc8b","name":"","serial":"fe37ebc7.c69d68","x":99,"y":98,"wires":[["4f1327d4.d482c"]]},{"id":"88f499d1.fae8c","type":"mqtt in","z":"b2cff5f2.9cc8b","name":"","topic":"MySensors/#","broker":"72a5a797.020098","x":115.5,"y":158,"wires":[["b006de82.c402b"]]},{"id":"b006de82.c402b","type":"function","z":"b2cff5f2.9cc8b","name":"convert topic to Serial protocol","func":"msg.topic = msg.topic.replace(\"MySensors\\/\", \"\"); // Gets rid of MySensors preamble\nfor (i=0;i<4;i++){\n    msg.topic = msg.topic.replace(\"\\/\", \";\");\n}\nmsg.payload = msg.topic + \";\" + msg.payload + \"\\n\";\nreturn msg;","outputs":1,"noerr":0,"x":334,"y":158,"wires":[["3f76ac9e.e553cc","543d809e.d80758"]]},{"id":"3f76ac9e.e553cc","type":"debug","z":"b2cff5f2.9cc8b","name":"MQTT to Serial debug","active":false,"console":"false","complete":"true","x":600,"y":207,"wires":[]},{"id":"543d809e.d80758","type":"serial out","z":"b2cff5f2.9cc8b","name":"","serial":"fe37ebc7.c69d68","x":562,"y":157,"wires":[]},{"id":"f8db4507.4574c8","type":"mqtt out","z":"b2cff5f2.9cc8b","name":"","topic":"To MQTT","qos":"","retain":"","broker":"72a5a797.020098","x":614,"y":98,"wires":[]},{"id":"4f1327d4.d482c","type":"mysdecenc","z":"b2cff5f2.9cc8b","name":"","x":254,"y":98,"wires":[["4fe1aca3.083e44"]]},{"id":"4fe1aca3.083e44","type":"function","z":"b2cff5f2.9cc8b","name":"Formats MQTT topic","func":"msg.topic = \"MySensors\" + \"/\" + msg.nodeId + \"/\" + msg.childSensorId + \"/\" + msg.messageType + \"/\" + msg.ack + \"/\" + msg.subType;\nreturn msg;","outputs":1,"noerr":0,"x":439,"y":98,"wires":[["f8db4507.4574c8","1d5d1eec.5b3009"]]}]

Citation: The MQTT submission format (along with challenge/response network flow) is NOT documented(on library or serial API page) . This makes any work past basic Library and Serial coding laborious.

Would be willing to make proper documentation. Given from comments elsewhere, "source is documentation", to which I respectfully disagree.

Arduino-development/libraries/MySensors/core/MyGatewayTransportMQTTClient.cpp Line #26 has the hint

// Topic structure: MY_MQTT_PUBLISH_TOPIC_PREFIX/NODE-ID/SENSOR-ID/CMD-TYPE/ACK-FLAG/SUB-TYPE

This is what it looks like:

Now when you proceed to inject packets into the MySensors network, you can send data to the appropriate topic or send data directly to the serial port.

The structure on MQTT that the data is stored is the following:

MySensors/nodeId/childSensorId/messageType/ack/subType
And the msg.payload is your data

Note that MySensors/ is user configurable within this MQTT-SerialGateway. I chose MySensors. You can change this by modifying "Formats MQTT topic" and "Convert Topic to Serial Protocol" and change "MySensors" to the name you want. Don't use spaces. MQTT doesn't always like that.

This does allow that you can have 2 or more gateways and have different topics for different networks to increase the number of your devices past 256.

I also have not included the messageType or subType substitutions found on the lists in Serial API. That would have included an extra step if that was stored in MQTT to translate between the messageType(number) and messageType(variable name). Although I may write a quick lookup depending on how bad I need that.

[barduino]

Hi @cranky

I'm very interested in this.

I was about to start coding a nodeJS app to create a SerialMQTTGateway

I'm going to try yours...

Actually first snag...

I can't get the node-red-contrib-mysensor installed...

So the mysencdec is missing

Any tips?

Here is my log:

RVB:~/.node-red rvb$ pwd
/Users/rvb/.node-red
RVB:~/.node-red rvb$ npm install node-red-contrib-mysensors
node-red-contrib-mysensors@0.7.5 ../node_modules/node-red-contrib-mysensors
RVB:~/.node-red rvb$ node-red -v


Welcome to Node-RED
===================

13 Jan 10:12:44 - [info] Node-RED version: v0.12.5
13 Jan 10:12:44 - [info] Node.js  version: v0.12.2
13 Jan 10:12:44 - [info] Loading palette nodes
13 Jan 10:12:45 - [warn] ------------------------------------------
13 Jan 10:12:45 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
13 Jan 10:12:45 - [warn] ------------------------------------------
13 Jan 10:12:45 - [info] Settings file  : /Users/rvb/.node-red/settings.js
13 Jan 10:12:45 - [info] User directory : /Users/rvb/.node-red
13 Jan 10:12:45 - [info] Flows file : /Users/rvb/.node-red/flows_RVB.local.json
13 Jan 10:12:45 - [info] Server now running at http://127.0.0.1:1880/
13 Jan 10:12:45 - [info] Starting flows
13 Jan 10:12:45 - [info] [inject:462ebb50.b9d144] repeat = 300000
13 Jan 10:12:45 - [info] Started flows

What am I doing wrong?

Cheers

[cranky]

if you do

npm install node-red-contrib-mysensors

It should work. If it doesn't, can you post the full text of the error please?

Oh, also, I chopped out the requirements for node-red-contrib-mysensors. Now, you should only need Serial, Function, MQTT, and Debug. Try this. Beware, I didn't test it, as I don't have easy remote access to my Broker.

[{"id":"938db5f1.fc32d","type":"mqtt-broker","z":"61b3db25.ac0dcc","broker":"localhost","port":"8883","clientid":"","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":"15","cleansession":true,"willTopic":"","willQos":"0","willRetain":null,"willPayload":"","birthTopic":"","birthQos":"0","birthRetain":null,"birthPayload":""},{"id":"e8212410.a623d","type":"serial-port","z":"61b3db25.ac0dcc","serialport":"/dev/ttyUSB0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false},{"id":"142d3129.87f8a7","type":"debug","z":"61b3db25.ac0dcc","name":"Serial to MQTT debug","active":false,"console":"false","complete":"true","x":791,"y":214,"wires":[]},{"id":"3637dd4b.cb9c4a","type":"serial in","z":"61b3db25.ac0dcc","name":"","serial":"e8212410.a623d","x":255,"y":264,"wires":[["e3ff437d.acd2a"]]},{"id":"40b2ff18.06e41","type":"mqtt in","z":"61b3db25.ac0dcc","name":"","topic":"MySensors/#","broker":"938db5f1.fc32d","x":271.5,"y":324,"wires":[["f638c51f.95fb78"]]},{"id":"f638c51f.95fb78","type":"function","z":"61b3db25.ac0dcc","name":"convert topic to Serial protocol","func":"msg.topic = msg.topic.replace(\"MySensors\\/\", \"\"); // Gets rid of MySensors preamble\nfor (i=0;i<4;i++){\n    msg.topic = msg.topic.replace(\"\\/\", \";\");\n}\nmsg.payload = msg.topic + \";\" + msg.payload + \"\\n\";\nreturn msg;","outputs":1,"noerr":0,"x":490,"y":324,"wires":[["4b33e37a.1bab14","51bc5462.90ca1c"]]},{"id":"4b33e37a.1bab14","type":"debug","z":"61b3db25.ac0dcc","name":"MQTT to Serial debug","active":false,"console":"false","complete":"true","x":756,"y":373,"wires":[]},{"id":"51bc5462.90ca1c","type":"serial out","z":"61b3db25.ac0dcc","name":"","serial":"e8212410.a623d","x":718,"y":323,"wires":[]},{"id":"e6240f60.af5c78","type":"mqtt out","z":"61b3db25.ac0dcc","name":"","topic":"To MQTT","qos":"","retain":"","broker":"938db5f1.fc32d","x":770,"y":264,"wires":[]},{"id":"e3ff437d.acd2a","type":"function","z":"61b3db25.ac0dcc","name":"Formats MQTT topic","func":"var sensor = msg.payload.split(\";\");\nmsg.topic = \"MySensors\" + \"/\" + sensor[0] + \"/\" + sensor[1] + \"/\" + sensor[2] + \"/\" + sensor[3] + \"/\" + sensor[4];\nmsg.payload = sensor[5];\nreturn msg;","outputs":1,"noerr":0,"x":498,"y":265,"wires":[["e6240f60.af5c78","142d3129.87f8a7"]]}]

[cranky]

Oh, it's that error

Delete the sheet, and re-copy. It will show up then. That, btw, is an existing bug in NR.

[barduino]

@cranky

as you can see on my log the npm install does not return any errors.

I had to manually create a nodes directory and install the files manually.

Anyway I guess I don't need it anymore since you removed the dependency

Thanks

Cheers

[cranky]

@barduino . How peculiar. I think it might be the way RasPi chose to include Node-red. It appears that you might need to do my above npm command with sudo... You shouldn't need to hand-create node_modules or node-red-contrib-mysensors. npm should have done that all for you, along with any potential dependencies. Adafruit's help goes into Pi-weirdness: https://learn.adafruit.com/raspberry-pi-hosting-node-red/installing-further-plugins

But I hope the mysensors removal works well. Like I said, haven't tested it yet

Good luck!
Cranky

[barduino]

@cranky,

This is actually a MAC-weirdness... no worries.

Well it connected to the amazon MQTT server no problems there.

But i don't have my gateway with me, will test later.

Thanks!

[cranky]

Gladly There's more coming out. I just provided the flow to handle "DHCP" sensors (or the ones that do REQ ID 255;255;3;0;3;\n)

One step at a time....

[gregl]

Very cool!
Im currently using the dev branch MQTTClient Gateway sketch - but will keep my eye on this!

[fets]

@cranky congrats for your work.
Do you you have in your luggage a flow to "turn SerialGateway into EthernetGateway" ?

[cranky]

Unfortunately, I do not. Until when I just looked at the source "MyGatewayTransportEthernet.cpp", I mistakenly assumed that it too published to an MQTT broker.

It doesn't look too terribly hard to do; with UDP being easier. But my main caveat here is that I don't have an SPI ethernet port for an arduino. That means I wouldn't be able to test my code.

If there was a packet documentation for TCP and UDP, I could write code for the reference. Again, past library and serial documentation, there is none I can refer to.

Of if some hardware fairy would be willing to send me a compatible SPI ethernet port, I could do it

---

documentation

MY_PORT = (default) 5003
(TCP/UDP) choice
(server mode / client mode) choice. IP dest. reqd for being client
buffer receiving: 100 B
buffer sending: 120 B

[cranky]

Ok, as you saw, I got the MQTT<-->TCP connector done , and that's live

Now, I have my GH up and populated. It's pretty sparse with documentation right now, but that will change.
https://github.com/jwcrawley/node-red-mysensors-flows/

What's cool is I can use a the "Watch" NR node and watch for updates in my working directory and auto-push

All you need is your SSH keys set up.

Then you watch for /.git/COMMIT_EDITMSG and when a change hits that, do a

git add (repo directory)

Then read the /.git/COMMIT_EDITMSG into msg.payload and then send it to an EXEC node as

git commit -m {{payload}}

And then

git push origin master

..... And there you have it: automated local to Github.

[Mathieu Chambon]

I think you made one mistake in the flow, you need to remove "To MQTT" in the Topic item and put "To MQTT" in the Name item else all the message go in "To MQTT" topic.

I have start to make the same flow and I found this post, thanks for the job.