nodered "injected" between domoticz and mysensors

[tbowmo]

Been playing with a small setup the last couple of days, where I have injected node-red between my serial GW, and domoticz.

I've made my own node, that deocdes / encodes mysensors serial protocol, only debugging things so far.

I have created a set of dummy serial ports (ToDomoticz) with socat, domoticz is attached to the other end of the socat "pipe". And receives the messages that comes in through node-red.

My goal (right now) is a proof of concept, where I can inject special sensor nodes (non mysensor stuff) as mysensors nodes.

Another goal I have thought about, was to have both domoticz and pidome running side by side, with the same sensor network (Why? because I can! and want to tinker with the system). Only one of them will be handing out node-id's though.

I am looking at publishing the mysensors node through npm, so others can install it easily in their setup.

[scalz]

@tbowmo: node red is so powerful. very interesting work

[fets]

@tbowmo congrats that's almost what I had in mind but haven't nodered knowledge and time to dig in.
If I understand clearly, your serail gateway is not directly connected you domoticz but to node-red ?
I didn't know socat, seems powerfull, how did you configure it with serial gateway ?
What do mysensors node do actually ?

[tbowmo]

@fets

mysensors GW is connected to node-red, which then is connected to domoticz. So all mysensors messages flow through node-red.

the mysensors node, splits the mysensors serial api messages (described here), to an object with the following properties:

msg.nodeId = tokens[0];
msg.childSensorId = tokens[1];
msg.messageType = tokens[2];
msg.ack = tokens[3];
msg.subType = tokens[4];
msg.payload = tokens[5];

You can then use node-red to act uppon the different parts, route it to whatever you want it to.

The mysensors node, can also do the reverse, that is if a msg object is sent in, which contains the above properties, it will assemble that into a mysensors compatible serial api message.

That means, that you can build messages (in node-red) from other devices, and inject that into what ever application you have, that is listening to the output of node-red. Or send it back to a mysensor node.

socat is a tool on unix, that can make a pipe between files, networks, etc.. It's a multipurpose relay (swiss army knife :))

[fets]

thanks for your explanation @tbowmo.
Your domoticz server and node-red aren't on the same machine ?

[bpair]

I have been playing with a similar setup but connecting to openHAB via MQTT via Mosquitto. I am also looking at using the REST API to connect to openHAB. I really like the simplicity of node-red and I want to be able to send the data to several places (to a file, to influxdb/grafana, maybe to thingspeak). I have been frustrated with the controller options as I do not think they can solve every issue.

One thing to know about node-red is that it is single threaded. I don't think it is an issue with a small setup and with small node-red sketches but if you have a large number of messages you could start to see a delay in processing. It probably won't affect your project but good to know. I believe i saw somewhere that the ethernet gateway can only handle 4 threads anyway and I guess if you are using the serial gateway it is single-threaded as well.

There are now a number of posts on this forum about node-red, as well as MQTT. I think the forum needs some new categories. Controllers/Hardware/General is not enough. Maybe Gateways and Brokers and Bridges should be added.

I certainly would like a place to share node-red work. All the code that exists for the ethernet MQTT gateways could be converted to javascript and node-red . It provides a solution for code that has trouble fitting in the space available on the arduino.

One of the issues I have with this setup is translating node IDs and therefore also which piece assigns them. If you plan to send data to a database like influxdb from node-red it is better to send a descriptive name rather than node-id=20, child-id=0, so do you maintain a map of these translations in node-red and then duplicate the map in your controller? I don't know the answer.

[tbowmo]

@bpair

This is my first stab at node-red, so haven't decided the whole setup yet.. Just playing with it, and see what features it has, and what I can make out of it..

I'm thinking about making a second nodetype as well, that can wrap a standard node-red msg.payload, into a mysensors serial api "datagram", where you can choose the node_id, message type etc. in dropdowns, when configuring that node.

[gbfromhb]

@bpair

"One thing to know about node-red is that it is single threaded. I don't think it is an issue with a small setup and with small node-red sketches but if you have a large number of messages you could start to see a delay in processing."

This single thread will handle a trimenduse load, because of the way it is done. It receives a request, and then asynchronously processes this request, it then looks to see if any asynchronous promises have completed (callback). The only concern I have with this single thread is if you were doing CPU intensive calculations

[bpair]

@gbfromhb Good to know. Thanks!

[diggs]

This is also something I have started playing with. Having node-red in the system seems like a powerful way to move and connect information around the place between the mysensor system and all sorts of interfaces.

It would be good to have an area where we could share the nodes and codes etc.

What code do you use for parsing the serial data into its different parts?

At the moment I have the mysensors serial gateway talking to domoticz directly (as standard) with the node-red serial node picking up serial data in parallel to then send off other places (influxdb, mqtt etc). At the moment I can grab the serial data, but no idea how to parse it out for useful sending on.

[tbowmo]

ok, my stab at a couple of nodes for this, should be install-able with the following command

npm install -g node-red-contrib-mysensors

This will add two extra nodes, that can be used with mysensors serial GW API.

[tbowmo]

a small example that I made, using my node-red-contrib-mysensors nodes (needs to be installed for this to work)

This routes data from a mysensors GW, to both domoticz and pidome. It filters out node-id request, so they are not sent to pidome (domoticz is the one that hands out id's to mysensors nodes).

I have a pulse counting arduino on my utility meter, this outputs strings as <watts>;<wh>\n, which I split into two different messages with a function, and dumps into mysensor serial protocol

To connect with domoticz / pidome, the following pipes are made with socat

socat PTY,link=/dev/ttyS80,mode=666,group=dialout,raw PTY,link=/dev/ttyUSB20,mode=666,group=dialout,raw &
socat PTY,link=/dev/ttyS81,mode=666,group=dialout,raw PTY,link=/dev/ttyUSB21,mode=666,group=dialout,raw &
socat PTY,link=/dev/ttyS82,mode=666,group=dialout,raw PTY,link=/dev/ttyUSB22,mode=666,group=dialout,raw &

It makes 3 virtual serial ports that I can use to attach pidome etc. to domoticz. I put it in /etc/rc.local

domoticz is using /dev/ttyUSB20 as a mysensors serial GW.

[{"id":"acfe3226.5301d","type":"serial-port","serialport":"/dev/ttyACM0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":true},{"id":"66f437ac.990bc8","type":"serial-port","serialport":"/dev/ttyS81","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":true},{"id":"b5105521.4aefa8","type":"serial-port","serialport":"/dev/ttyS80","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":true},{"id":"821a7286.7de59","type":"serial-port","serialport":"/dev/ttyUSB0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":true},{"id":"5b8bba4c.a47444","type":"serial in","name":"From Mysensors","serial":"821a7286.7de59","x":257,"y":196,"z":"67deb464.98214c","wires":[["1656727d.e9a98e","4755dfb8.b8aa2"]]},{"id":"47b5936e.b84a6c","type":"serial out","name":"To Mysensors","serial":"821a7286.7de59","x":1166,"y":305,"z":"67deb464.98214c","wires":[]},{"id":"602db35f.9fd24c","type":"serial in","name":"From Domoticz","serial":"b5105521.4aefa8","x":248,"y":348,"z":"67deb464.98214c","wires":[["47b5936e.b84a6c","f58d12f9.0a72f"]]},{"id":"4755dfb8.b8aa2","type":"serial out","name":"To Domoticz","serial":"b5105521.4aefa8","x":1164,"y":213,"z":"67deb464.98214c","wires":[]},{"id":"1656727d.e9a98e","type":"mysdecenc","name":"","x":560,"y":157,"z":"67deb464.98214c","wires":[["5dbdeccd.a24214"]]},{"id":"3094c378.cf6b3c","type":"debug","name":"","active":true,"console":"false","complete":"true","x":1033,"y":468,"z":"67deb464.98214c","wires":[]},{"id":"5dbdeccd.a24214","type":"function","name":"Filter ID requests","func":"if (msg.messageType != 3 &\n    msg.subType != 3) {\n    return msg;\n}","outputs":1,"noerr":0,"x":738,"y":157,"z":"67deb464.98214c","wires":[["2ce8f789.d31708"]]},{"id":"293de58.fd6c21a","type":"serial in","name":"From PiDome","serial":"66f437ac.990bc8","x":246,"y":266,"z":"67deb464.98214c","wires":[["47b5936e.b84a6c","4755dfb8.b8aa2"]]},{"id":"f58d12f9.0a72f","type":"serial out","name":"To PiDome","serial":"66f437ac.990bc8","x":1162,"y":386,"z":"67deb464.98214c","wires":[]},{"id":"2ce8f789.d31708","type":"mysdecenc","name":"","x":915,"y":156,"z":"67deb464.98214c","wires":[["f58d12f9.0a72f"]]},{"id":"ff3ea1e4.00c16","type":"serial in","name":"Meter input","serial":"acfe3226.5301d","x":229,"y":475,"z":"67deb464.98214c","wires":[["3c7fe6d9.c3801a"]]},{"id":"3c7fe6d9.c3801a","type":"function","name":"Split meter data","func":"var pl = msg.payload.split(';');\n\nmsg1 = {payload : parseFloat(pl[0])*1000};\nmsg2 = {payload : parseFloat(pl[1])/1000};\nreturn [msg1, msg2];","outputs":"2","noerr":0,"x":427,"y":475,"z":"67deb464.98214c","wires":[["e1a7231a.1e58e"],["b06687e3.4f9978"]]},{"id":"e1a7231a.1e58e","type":"mysencap","name":"V_WATT node 99","nodeid":"99","childid":"1","subtype":"17","x":642,"y":445,"z":"67deb464.98214c","wires":[["ede87fe8.12178"]]},{"id":"b06687e3.4f9978","type":"mysencap","name":"V_KWM node 99","nodeid":"99","childid":"2","subtype":"18","x":640,"y":488,"z":"67deb464.98214c","wires":[["ede87fe8.12178"]]},{"id":"ede87fe8.12178","type":"mysdecenc","name":"","x":868,"y":468,"z":"67deb464.98214c","wires":[["3094c378.cf6b3c","4755dfb8.b8aa2","f58d12f9.0a72f"]]}]```

[tbowmo]

one word of caution, using the above scenario..

You need to have something picking up the "serial port data" in the other end of the socat pipes, otherwise nodered clog up, and start to use 100% cpu. I discovered that the "hard way".

As long as the socat pipe is emptied, node-red is running happily

[tbowmo]

just updated the plugin to node-red, so it now can send presentation messages as well (and internal messages, if you want), when using the "mysencap" node.

The presentation message is sent 1 second after a flow is deployed, or when node-red is started..

[John]

This looks really cool! What is PiDome's role in this picture?

[tbowmo]

@John

It would just be a proof of concept. However I haven't succeeded in bringing in Pidome, as it relies on USB device descriptors for the serial port, and so I can't choose the socat pipe from nodered.

One could use it for testing new sensor types, mock them up in nodered and inject messages to the controller (pidome, domoticz etc)

[John]

Yep, that's correct, to make sure the correct interface is loaded (serial/hid/etc..). If you would get it to trigger udev with a serial description it should catch it.

Nonetheless in about a week or two, three i think the fixed udev triggered mechanism will be supplemented with an user selectable list of available ports as i'm quite interested in this mockup.

[tbowmo]

just made a small update, so it now can create a more full presentation round, to another controller, with both sketch name, and version strings..

Use the following to update your plugin..

npm update node-red-contrib-mysensors

[bpair]

@tbowmo Thanks for the update. I hope to try using socat as a way to test different controllers easily. I am frustrated with the controller options and I feel like as my project progresses being able to easily change controllers might be very desirable. This architecture makes that possible.

I have a function in my node-red flow that enriches the messages from MySensors. I realize this begins to do what a controller might do, but I need a simple way to translate "Node:20, Child:1" to "Tank 1 - Water Temperature" so that I can publish that to a database or dweet.io or easily display in a graph. I might know what a graph of Node:20:Child1=25.6 means but nobody else will.

I am new to json but am attempting to use that. Ideally node-red would allow for easily manage this mapping information via a web page and store it. I am setting this globally:

[{"id":"d5d724ec.cba8c","type":"subflow","name":"On Startup","in":[],"out":[{"x":454,"y":160,"wires":[{"id":"dcd80855.bc4688","port":0}]}]},{"id":"c7328a87.d34f28","type":"inject","name":"On Startup","topic":"server-startup","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"x":148,"y":160,"z":"d5d724ec.cba8c","wires":[["dcd80855.bc4688"]]},{"id":"dcd80855.bc4688","type":"function","name":"Save startup date","func":"msg.startupDate = msg.payload;\nmsg.payload = \"\";\nreturn msg;","outputs":1,"noerr":0,"x":318,"y":160,"z":"d5d724ec.cba8c","wires":[[]]},{"id":"302ddf01.cbf0b8","type":"subflow:d5d724ec.cba8c","name":"","x":127,"y":214,"z":"5791498e.8d6878","wires":[["841c3e7c.b41288"]]},{"id":"841c3e7c.b41288","type":"function","name":"SetGlobalVars","func":"node.log(\"Add sensorNodes to global context\");\n\nvar sNodes = [   \n        {nodeId:30, nodeLabel:\"Greenhouse\", \n            sensors:[\n                {childSensorId:1, childSensorLabel:\"Air\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:2, childSensorLabel:\"Air\", sensorTypeLabel:\"Humidity\"},\n                {childSensorId:130, childSensorLabel:\"AirBattery\", sensorTypeLabel:\"Voltage\"}\n            ] \n        }, \n        {nodeId:50, nodeLabel:\"Aquaponics\", \n            sensors:[\n                {childSensorId:0, childSensorLabel:\"Water\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:20, childSensorLabel:\"Air\", sensorTypeLabel:\"Humidity\"},\n                {childSensorId:21, childSensorLabel:\"Air\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:30, childSensorLabel:\"Light\", sensorTypeLabel:\"Lux\"}\n            ]\n        }\n    ];\n\ncontext.global.sensorNodes = sNodes;\n\n    \nmsg.payload = context.global.sensorNodes;\n\nreturn msg;","outputs":1,"noerr":0,"x":357,"y":213,"z":"5791498e.8d6878","wires":[[]]},{"id":"8ad9f121.3ea3b","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":127,"y":135,"z":"5791498e.8d6878","wires":[["841c3e7c.b41288"]]}]

and then using it:

[{"id":"d14e00ff.cc9f7","type":"function","name":"MySensorIdInfo","func":"var sensorNode = getByNodeId(context.global.sensorNodes, \"\"+msg.nodeId);\nif (sensorNode != null) {\n    console.log(\"NodeLabel = \" + sensorNode.nodeLabel);\n    msg.nodeLabel = sensorNode.nodeLabel;\n        \n    var childNode = getByChildId(sensorNode.sensors, \"\"+msg.childSensorId);\n    if (childNode != null) {\n        console.log(\"ChildNodeLabel = \" + childNode.childSensorLabel);\n        msg.childSensorLabel = childNode.childSensorLabel;\n        msg.sensorTypeLabel = childNode.sensorTypeLabel;\n    } else {\n        msg.childSensorLabel = msg.childSensorId;\n        msg.sensorTypeLabel = \"undefined\";\n    }\n} else {\n    node.error(\"No sensor match found in sensorNodes\");\n    msg.nodeLabel = msg.nodeId;\n    msg.childSensorLabel = msg.childSensorId;\n    msg.sensorTypeLabel = \"undefined\";   \n}\n\nif (msg.sensorTypeLabel == \"undefined\") {\n    return null;\n}\n\n//debug\nnode.log(\"nodeId = \" + msg.nodeId + \" :--: \" + \n\"nodeLabel = \" + msg.nodeLabel + \" :--: \" + \n\"childSensorId = \" + msg.childSensorId + \" :--: \" + \n\"childSensorLabel = \" + msg.childSensorLabel + \" :--: \" + \n\"sensorTypeLabel = \" + msg.sensorTypeLabel);\n\nreturn msg;\n\n//------- Helper Functions ----------//\n\nfunction getByNodeId(arr, value) {\n  for (var i=0, iLen=arr.length; i<iLen; i++) {\n    if (arr[i].nodeId == value) return arr[i];\n  }\n}\n\nfunction getByChildId(arr, value) {\n  for (var i=0, iLen=arr.length; i<iLen; i++) {\n    if (arr[i].childSensorId == value) return arr[i];\n  }\n}","outputs":1,"noerr":0,"x":614,"y":390,"z":"c3095d41.89f1e8","wires":[["6e57151f.370ffc","ae35a271.90c3b"]]}]

The one other piece I added was a response to the MetricOrImperial question. Maybe some of this is useful. Thanks for your work.

[Fabien]

@bpair said:

@tbowmo Thanks for the update. I hope to try using socat as a way to test different controllers easily. I am frustrated with the controller options and I feel like as my project progresses being able to easily change controllers might be very desirable. This architecture makes that possible.

I have a function in my node-red flow that enriches the messages from MySensors. I realize this begins to do what a controller might do, but I need a simple way to translate "Node:20, Child:1" to "Tank 1 - Water Temperature" so that I can publish that to a database or dweet.io or easily display in a graph. I might know what a graph of Node:20:Child1=25.6 means but nobody else will.

I am new to json but am attempting to use that. Ideally node-red would allow for easily manage this mapping information via a web page and store it. I am setting this globally:

[{"id":"d5d724ec.cba8c","type":"subflow","name":"On Startup","in":[],"out":[{"x":454,"y":160,"wires":[{"id":"dcd80855.bc4688","port":0}]}]},{"id":"c7328a87.d34f28","type":"inject","name":"On Startup","topic":"server-startup","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"x":148,"y":160,"z":"d5d724ec.cba8c","wires":[["dcd80855.bc4688"]]},{"id":"dcd80855.bc4688","type":"function","name":"Save startup date","func":"msg.startupDate = msg.payload;\nmsg.payload = \"\";\nreturn msg;","outputs":1,"noerr":0,"x":318,"y":160,"z":"d5d724ec.cba8c","wires":[[]]},{"id":"302ddf01.cbf0b8","type":"subflow:d5d724ec.cba8c","name":"","x":127,"y":214,"z":"5791498e.8d6878","wires":[["841c3e7c.b41288"]]},{"id":"841c3e7c.b41288","type":"function","name":"SetGlobalVars","func":"node.log(\"Add sensorNodes to global context\");\n\nvar sNodes = [   \n        {nodeId:30, nodeLabel:\"Greenhouse\", \n            sensors:[\n                {childSensorId:1, childSensorLabel:\"Air\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:2, childSensorLabel:\"Air\", sensorTypeLabel:\"Humidity\"},\n                {childSensorId:130, childSensorLabel:\"AirBattery\", sensorTypeLabel:\"Voltage\"}\n            ] \n        }, \n        {nodeId:50, nodeLabel:\"Aquaponics\", \n            sensors:[\n                {childSensorId:0, childSensorLabel:\"Water\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:20, childSensorLabel:\"Air\", sensorTypeLabel:\"Humidity\"},\n                {childSensorId:21, childSensorLabel:\"Air\", sensorTypeLabel:\"Temperature\"},\n                {childSensorId:30, childSensorLabel:\"Light\", sensorTypeLabel:\"Lux\"}\n            ]\n        }\n    ];\n\ncontext.global.sensorNodes = sNodes;\n\n    \nmsg.payload = context.global.sensorNodes;\n\nreturn msg;","outputs":1,"noerr":0,"x":357,"y":213,"z":"5791498e.8d6878","wires":[[]]},{"id":"8ad9f121.3ea3b","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":127,"y":135,"z":"5791498e.8d6878","wires":[["841c3e7c.b41288"]]}]

and then using it:

[{"id":"d14e00ff.cc9f7","type":"function","name":"MySensorIdInfo","func":"var sensorNode = getByNodeId(context.global.sensorNodes, \"\"+msg.nodeId);\nif (sensorNode != null) {\n    console.log(\"NodeLabel = \" + sensorNode.nodeLabel);\n    msg.nodeLabel = sensorNode.nodeLabel;\n        \n    var childNode = getByChildId(sensorNode.sensors, \"\"+msg.childSensorId);\n    if (childNode != null) {\n        console.log(\"ChildNodeLabel = \" + childNode.childSensorLabel);\n        msg.childSensorLabel = childNode.childSensorLabel;\n        msg.sensorTypeLabel = childNode.sensorTypeLabel;\n    } else {\n        msg.childSensorLabel = msg.childSensorId;\n        msg.sensorTypeLabel = \"undefined\";\n    }\n} else {\n    node.error(\"No sensor match found in sensorNodes\");\n    msg.nodeLabel = msg.nodeId;\n    msg.childSensorLabel = msg.childSensorId;\n    msg.sensorTypeLabel = \"undefined\";   \n}\n\nif (msg.sensorTypeLabel == \"undefined\") {\n    return null;\n}\n\n//debug\nnode.log(\"nodeId = \" + msg.nodeId + \" :--: \" + \n\"nodeLabel = \" + msg.nodeLabel + \" :--: \" + \n\"childSensorId = \" + msg.childSensorId + \" :--: \" + \n\"childSensorLabel = \" + msg.childSensorLabel + \" :--: \" + \n\"sensorTypeLabel = \" + msg.sensorTypeLabel);\n\nreturn msg;\n\n//------- Helper Functions ----------//\n\nfunction getByNodeId(arr, value) {\n  for (var i=0, iLen=arr.length; i<iLen; i++) {\n    if (arr[i].nodeId == value) return arr[i];\n  }\n}\n\nfunction getByChildId(arr, value) {\n  for (var i=0, iLen=arr.length; i<iLen; i++) {\n    if (arr[i].childSensorId == value) return arr[i];\n  }\n}","outputs":1,"noerr":0,"x":614,"y":390,"z":"c3095d41.89f1e8","wires":[["6e57151f.370ffc","ae35a271.90c3b"]]}]

The one other piece I added was a response to the MetricOrImperial question. Maybe some of this is useful. Thanks for your work.

Thank you, it's very usefull !

[vickey]

@tbowmo May you please share your node-red set up by exporting your setup as flow? If feasible for you