MQTT Broker gateway


  • Code Contributor

    How to Set-up MQTTGateway with OpenHAB

    What is MQTTGateway:
    MQTTGateway will act as a broker, a server. And any MQTT clients should be able to connect to it (Please let me know if you experience problems).
    MQTTGateway is a very simple implementation. It will not care if the client subscribes to anything or not.
    Every node package it receives it will publish to the client. Sketch name and version is also implemented.
    Most clients handle this well and just ignores messages it hasn't been subscribed yet.
    There are some settings in MQTT that can be good to take a look at.
    You will need to configure TCP_IP and TCP_PORT.
    MQTTGateway will also give nodes an ID between MQTT_FIRST_SENSORID and MQTT_LAST_SENSORID. The last position is stored in EEprom.

    As standard MQTT_SEND_SUBSCRIPTION is enabled. This basically means that the broker will send a empty payload to the node upon a subscribe package. This can be useful. Just keep in mind that you can receive empty payload packages.

    The MQTTGateway will also present its SketchName and SketchVersion.

    Lets get started!
    First follow https://github.com/openhab/openhab/wiki/Quick-Setup-an-openHAB-Server You can setup and try with DEMO configuration first.

    Then we need to unpack org.openhab.binding.mqtt into addons folder * NOTE (During development of MQTTGateway I found a bug inside openhab mqtt client which is solved in 1.6.0 nightly build. Use this nightly addon if you experience problems)

    • openhab.cfg
      All you need to do here is to add
     mqtt:mysensor.url=tcp://192.168.0.234:1883
     mqtt:mysensor.clientId=MQTT
    

    ClientID does not matter at all (just keep it short). And IP and Port is self explanatory.

    Now you can start OpenHAB, it will connect but nothing more will happen.
    This is because we hasn't subscribed to anything and OpenHAB will just ignore every incoming package.

    So here comes a sample configuration of OpenHAB.

    • items/test.items
     Group all
     Group node1  (all)
     Group node2  (all)
     Group sketch (all)
    
     Number node1_temp  "Temp [%.1f °C]"     (node1,all)  {mqtt="<[mysensor:MyMQTT/20/10/V_TEMP:state:default]"}
     Number node1_humid "Humid [%.1f %%Rh]" (node1,all)  {mqtt="<[mysensor:MyMQTT/20/11/V_HUM:state:default]"}
    
     Switch node2_sw1  "sw2"  (node2,all)  {mqtt=">[mysensor:MyMQTT/21/1/V_LIGHT:command:ON:1],>[mysensor:MyMQTT/21/1/V_LIGHT:command:OFF:0]"}
     Switch node2_sw2  "sw2 send + recieve example"  (node2,all)      {mqtt=">[mysensor:MyMQTT/21/2/V_LIGHT:command:ON:1],>[mysensor:MyMQTT/21/2/V_LIGHT:command:OFF:0],<[mysensor:MyMQTT/21/2/V_LIGHT:command:MAP(1on0off.map)]"}
     
     String sketch20 "Sketch name 20 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/20/255/V_SKETCH_NAME:state:default]"}
     String sketch21 "Sketch name 21 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/21/255/V_SKETCH_NAME:state:default]"}
     String sketch22 "Sketch name 22 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/22/255/V_SKETCH_NAME:state:default]"}
     String sketch23 "Sketch name 23 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/23/255/V_SKETCH_NAME:state:default]"}
     String sketch24 "Sketch name 24 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/24/255/V_SKETCH_NAME:state:default]"}
     String sketch25 "Sketch name 25 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/25/255/V_SKETCH_NAME:state:default]"}
     String sketch26 "Sketch name 26 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/26/255/V_SKETCH_NAME:state:default]"}
     String sketch27 "Sketch name 27 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/27/255/V_SKETCH_NAME:state:default]"}
     String sketch28 "Sketch name 28 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/28/255/V_SKETCH_NAME:state:default]"}
     String sketch29 "Sketch name 29 [%s]"  (sketch,all) {mqtt="<[mysensor:MyMQTT/29/255/V_SKETCH_NAME:state:default]"}
    

    We also need to create a sitemap :

    • sitemap/test.sitemap
     sitemap demo label="Menu"
     Frame label="test" {
       Group item=node1 label="Temp hum node"
       Group item=node2 label="Switch node"
       Group item=sketch label="Sketch names"
       Group item=all label="Everything"
     }
    

    and a simple transformation file :

    • transform/1on0off.map
    1=ON
    0=OFF
    

    If everything worked and I didn't forget anything just fire up :

    http://[Openhab]:8080/openhab.app?sitemap=test

    After you restart one of your node's the data and sketch name should be visible.
    Openhab will forget everything after every restart. Use persistence to fix this (https://github.com/openhab/openhab/wiki/rrd4j-Persistence)


  • Mod

    @Damme Hey man, thanks for the extensive documentation!
    Would you mind if I split this post into its own topic? I think your MQTT broker deserves its own topic; let's reserve this thread for OpenHAB+MQTT related stuff.

    Update Split done!


  • Code Contributor

    @Yveaux sure



  • So have I got this right.....?

    This will mean that I can build an arduino with an ethernet port and a radio, and this will then plug in to an ethernet port and be what my other ardiuno sensors will connect to, and it will send everything to my Vera?


  • Code Contributor

    @Homer For Vera you should build http://mysensors.org/build/ethernet_gateway

    MQTTGateway is a MQTT broker (server), used for mqttclient like openhab for example.



  • @Damme said:

    @Homer For Vera you should build http://mysensors.org/build/ethernet_gateway

    MQTTGateway is a MQTT broker (server), used for mqttclient like openhab for example.

    Ok. I was hoping that this device would enable me to have some of my sensors connected to ethernet so I can power them as well. Rather than sending information over wireless, they would do so via ethernet.

    There isn't a way to do as I have described above, right?


  • Code Contributor

    @Homer I'm sorry but I don't think I really understand your question.
    read http://mysensors.org/build/network
    Each node (with sensors) talk wireless to A gateway, which then talks to a controller (in your case Vera) this can be via RS232 OR ethernet.

    If you have something that talks MQTT (like openhab) as a controller, then your choice could be sensors <-> MQTTgateway <-> openhab instead.


  • Code Contributor

    I can confirm it's working with UIPEthernet.h and the ENC28J60 module as well. You need to turn off debugging in MyConfig.h to make the code fit.


  • Code Contributor

    @ToSa Thanks! I havn't had time to get and test that module.


  • Code Contributor

    @ToSa said:

    I can confirm it's working with UIPEthernet.h and the ENC28J60 module as well. You need to turn off debugging in MyConfig.h to make the code fit.

    Maybe that was a little too fast - it starts up and starting OpenHAB finds the gateway and connects. When I reset a node it appears that the gateway crashes (OpenHAB reports disconnect) and only a reset of the gateway helps to get it to work again. Problem is it's hard to debug without debugging enabled due to code size :(


  • Code Contributor

    @ToSa if possible you could make a tcpdump over the mqqt port and I'll look at it. Also a serial dump from the node would be great


  • Code Contributor

    @Damme said:

    @ToSa if possible you could make a tcpdump over the mqqt port and I'll look at it. Also a serial dump from the node would be great

    After changing the include from Ethernet.h to UIPEthernet.h the sketch fits into the m328 only with DEBUG (and TCPDUMP) turned off:

    DEBUG / TCPDUMP turned off:
    Sketch uses 30,626 bytes (99%) of program storage space. Maximum is 30,720 bytes.
    Global variables use 1,330 bytes (64%) of dynamic memory, leaving 718 bytes for local variables. Maximum is 2,048 bytes.

    DEBUG / TCPDUMP turned on:
    Sketch uses 32,354 bytes (105%) of program storage space. Maximum is 30,720 bytes.
    Global variables use 1,393 bytes (68%) of dynamic memory, leaving 655 bytes for local variables. Maximum is 2,048 bytes.
    processing.app.debug.RunnerException: Sketch too big;

    The only thing I have right now to look at is the OpenHAB output (appears that it's not happening when I reset a node but rather random):

    pi@openhab ~/runtime $ ./start.sh
    Launching the openHAB runtime...
    osgi> 19:54:19.384 INFO o.o.c.internal.CoreActivator[:61] - openHAB runtime has been started (v1.6.0).
    19:54:39.316 INFO o.o.i.t.mqtt.MqttService[:106] - MQTT Service initialization completed.
    19:54:39.322 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'
    19:54:42.838 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'logging.persist'
    19:54:46.082 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'exec.persist'
    19:54:46.261 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'rrd4j.persist'
    19:54:46.494 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'db4o.persist'
    19:54:47.009 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.items'
    19:55:01.187 INFO o.o.i.s.i.DiscoveryServiceImpl[:72] - mDNS service has been started
    19:55:10.219 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.sitemap'
    19:55:19.042 INFO o.o.io.rest.RESTApplication[:143] - Started REST API at /rest
    19:55:24.936 INFO o.o.u.w.i.s.WebAppServlet[:79] - Started Classic UI at /openhab.app
    19:55:35.271 ERROR o.o.i.t.m.i.MqttBrokerConnection[:478] - Error starting consumer
    org.eclipse.paho.client.mqttv3.MqttException: Connection lost
    at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
    Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    19:55:35.298 ERROR o.o.i.t.m.i.MqttBrokerConnection[:532] - MQTT connection to broker was lost
    org.eclipse.paho.client.mqttv3.MqttException: Connection lost
    at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
    Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    19:55:35.305 ERROR o.o.i.t.m.i.MqttBrokerConnection[:536] - MQTT connection to 'mysensor' was lost: Connection lost : ReasonCode 32109 : Cause : Connection reset
    19:55:35.310 INFO o.o.i.t.m.i.MqttBrokerConnection[:543] - Starting connection helper to periodically try restore connection to broker 'mysensor'
    19:55:45.317 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'


  • Code Contributor


  • Code Contributor

    @Damme said:

    @ToSa wireshark dump https://www.wireshark.org/ or tcpdump http://www.tcpdump.org/

    Thought you were referring to the "#define TCPDUMP" in MyMQTT.h :)

    I did "sudo tcpdump host 10.0.1.99 -i eth0 -w dump.pcap" on the RPi that is running the OpenHAB runtime (10.0.1.6 is the IP of the RPi with OpenHAB / 10.0.1.99 is the IP of the MQTT gateway) and then did the following:

    • power-on MQTT gateway
    • start OpenHAB and wait until startup finished
    • power-on node with DallasTemperature

    The pcap file is attached (not knowing a lot about the protocol it looks like a series of keep-alive messages only and suddenly a connection reset): dump2.pcap

    I do not see any messages arriving at OpenHAB... looking at the debug output from the node it appears that the "find parent" message is correctly handled by the gateway but the "node ID request" is not.

    ** OpenHAB output: **

    pi@openhab ~/runtime $ ./start.sh
    Launching the openHAB runtime...
    osgi> 21:55:31.305 INFO o.o.c.internal.CoreActivator[:61] - openHAB runtime has been started (v1.6.0).
    21:55:50.955 INFO o.o.i.t.mqtt.MqttService[:106] - MQTT Service initialization completed.
    21:55:50.974 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'
    21:55:57.178 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'logging.persist'
    21:55:59.989 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'exec.persist'
    21:56:00.137 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'rrd4j.persist'
    21:56:00.303 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'db4o.persist'
    21:56:01.162 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.items'
    21:56:13.732 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.sitemap'
    21:56:16.231 INFO o.o.i.s.i.DiscoveryServiceImpl[:72] - mDNS service has been started
    21:56:31.603 INFO o.o.io.rest.RESTApplication[:143] - Started REST API at /rest
    21:56:37.387 INFO o.o.u.w.i.s.WebAppServlet[:79] - Started Classic UI at /openhab.app
    21:59:50.613 ERROR o.o.i.t.m.i.MqttBrokerConnection[:532] - MQTT connection to broker was lost
    org.eclipse.paho.client.mqttv3.MqttException: Connection lost
    at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
    Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    21:59:50.621 ERROR o.o.i.t.m.i.MqttBrokerConnection[:536] - MQTT connection to 'mysensor' was lost: Connection lost : ReasonCode 32109 : Cause : Connection reset
    21:59:50.625 INFO o.o.i.t.m.i.MqttBrokerConnection[:543] - Starting connection helper to periodically try restore connection to broker 'mysensor'
    22:00:00.633 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'

    ** The serial debug output of the node shows (more than that but always repeating as I did several resets of the node): **

    sensor started, id 255
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    send: 255-255-255-255 s=255,c=3,t=7,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
    req node id
    send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:


  • Code Contributor

    @ToSa Thats odd. Everything looks good...(almost, there are some excess of trailing 0x00 but shouldn't matter, but I'll look at that anyways..)
    Can you compile with myconfig debug enabled, but define tcpdump disabled? or is it too big that way too?


  • Plugin Developer

    @Damme

    I'm currently busy implementing an mqtt client plugin, and i'm going to use mysensors for it's first predefined implementation so i have a couple of questions

    I took a look at the api page and saw this in the mqtt documentation:

    [MQTT_BROKER_PREFIX]/[NodeID]/[SensorID]/V_[SensorType]
    

    Does this means as example:

    MyMQTT/[0-255]/[0-255]/[STRING]/[VALUE]
    

    OR this?

    MyMQTT/[0-255]/[0-255]/[0-endvarindex]/[VALUE]
    
    • Are variable types defined by their index used or by name?
    • Also it states it keeps track of node id's how are these reported by the broker, should the controller still hand out free id's?

    [EDIT]
    I currently do not have the hardware present to at least set it up to test and look at what exactly is happening.
    [/EDIT]


  • Code Contributor

    @John MyMQTT/[0-255]/[0-255]/[STRING]/[VALUE]

    MQTT.cpp :

    sprintf(&buffer[buffsize],"/%i/%i/V_%s", msg.sender, msg.sensor, getType(convBuf, &vType[msg.type]));
    

    note V_%s -> MyMQTT/[0-255]/[0-255]/V_[STRING]/[VALUE]
    also note there is no trailing / before payload, as http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/mqtt-v3r1.html
    http://mqtt.org/wiki/doku.php/topic_format
    I choose not to have trailing slash, but if needed its easy to make it configurable.

    and get type translates from this list

      char V_0[] PROGMEM = "TEMP";		//V_TEMP
      char V_1[] PROGMEM = "HUM";		//V_HUM
      char V_2[] PROGMEM = "LIGHT";		//V_LIGHT
      char V_3[] PROGMEM = "DIMMER";		//V_DIMMER
      char V_4[] PROGMEM = "PRESSURE";	//V_PRESSURE
      char V_5[] PROGMEM = "FORECAST";	//V_FORECAST
      char V_6[] PROGMEM = "RAIN";		//V_RAIN
      char V_7[] PROGMEM = "RAINRATE";	//V_RAINRATE
      char V_8[] PROGMEM = "WIND";		//V_WIND
      char V_9[] PROGMEM = "GUST";		//V_GUST
      char V_10[] PROGMEM = "DIRECTON";	//V_DIRECTON
      char V_11[] PROGMEM = "UV";		//V_UV
      char V_12[] PROGMEM = "WEIGHT";		//V_WEIGHT
      char V_13[] PROGMEM = "DISTANCE";	//V_DISTANCE
      char V_14[] PROGMEM = "IMPEDANCE";	//V_IMPEDANCE
      char V_15[] PROGMEM = "ARMED";		//V_ARMED
      char V_16[] PROGMEM = "TRIPPED";	//V_TRIPPED
      char V_17[] PROGMEM = "WATT";		//V_WATT
      char V_18[] PROGMEM = "KWH";		//V_KWH
      char V_19[] PROGMEM = "SCENE_ON";	//V_SCENE_ON
      char V_20[] PROGMEM = "SCENE_OFF";	//V_SCENE_OFF
      char V_21[] PROGMEM = "HEATER";		//V_HEATER
      char V_22[] PROGMEM = "HEATER_SW";	//V_HEATER_SW
      char V_23[] PROGMEM = "LIGHT_LEVEL";	//V_LIGHT_LEVEL
      char V_24[] PROGMEM = "VAR1";		//V_VAR1
      char V_25[] PROGMEM = "VAR2";		//V_VAR2
      char V_26[] PROGMEM = "VAR3";		//V_VAR3
      char V_27[] PROGMEM = "VAR4";		//V_VAR4
      char V_28[] PROGMEM = "VAR5";		//V_VAR5
      char V_29[] PROGMEM = "UP";		//V_UP
      char V_30[] PROGMEM = "DOWN";		//V_DOWN
      char V_31[] PROGMEM = "STOP";		//V_STOP
      char V_32[] PROGMEM = "IR_SEND";	//V_IR_SEND
      char V_33[] PROGMEM = "IR_RECEIVE";	//V_IR_RECEIVE
      char V_34[] PROGMEM = "FLOW";		//V_FLOW
      char V_35[] PROGMEM = "VOLUME";		//V_VOLUME
      char V_36[] PROGMEM = "LOCK_STATUS";	//V_LOCK_STATUS
      char V_37[] PROGMEM = "DUST_LEVEL";	//V_DUST_LEVEL
      char V_38[] PROGMEM = "VOLTAGE";	//V_VOLTAGE
      char V_39[] PROGMEM = "CURRENT";	//V_CURRENT
      char V_40[] PROGMEM = "";		//
      char V_41[] PROGMEM = "";		//
      char V_42[] PROGMEM = "";		//
      char V_43[] PROGMEM = "";		//
      char V_44[] PROGMEM = "";		//
      char V_45[] PROGMEM = "";		//
      char V_46[] PROGMEM = "";		//
      char V_47[] PROGMEM = "";		//
      char V_48[] PROGMEM = "";		//
      char V_49[] PROGMEM = "";		//
      char V_50[] PROGMEM = "";		//
      char V_51[] PROGMEM = "";		//
      char V_52[] PROGMEM = "";		//
      char V_53[] PROGMEM = "";		//
      char V_54[] PROGMEM = "";		//
      char V_55[] PROGMEM = "";		//
      char V_56[] PROGMEM = "";		//
      char V_57[] PROGMEM = "";		//
      char V_58[] PROGMEM = "";		//
      char V_59[] PROGMEM = "";		//
      char V_60[] PROGMEM = "Started!\n";	//Custom for MQTTGateway
      char V_61[] PROGMEM = "SKETCH_NAME";	//Custom for MQTTGateway
      char V_62[] PROGMEM = "SKETCH_VERSION"; //Custom for MQTTGateway
      char V_63[] PROGMEM = "UNKNOWN"; 	//Custom for MQTTGateway
      char V_64[] PROGMEM = "FW_CREQ"; 	//Custom for MQTTGateway
      char V_65[] PROGMEM = "FW_CRES"; 	//Custom for MQTTGateway
      char V_66[] PROGMEM = "FW_REQ"; 	//Custom for MQTTGateway
      char V_67[] PROGMEM = "FW_RES"; 	//Custom for MQTTGateway
    

    The MQTTgateway hands out ID's; configured in MQTT.h:

      #define MQTT_FIRST_SENSORID	20  		// If you want manually configured nodes below this value. 255 = Disable
      #define MQTT_LAST_SENSORID	254 		// 254 is max! 255 reserved.
      #define MQTT_BROKER_PREFIX	"MyMQTT"	// First prefix in MQTT tree, keep short!
      #define MQTT_SEND_SUBSCRIPTION 1		// Send empty payload (request) to node upon MQTT client subscribe request.
      // NOTE above : Beware to check if there is any length on payload in your incommingMessage code:
      // Example: if (msg.type==V_LIGHT && strlen(msg.getString())>0) otherwise the code might do strange things.

  • Plugin Developer

    @Damme
    Clear!

    And how about the internal messages? Are these somewhat compatible with the Serial API? Like this one now has a "I_GATEWAY_READY" internal var, this one complies then with the "Started!\n" message?

    Also about the presentation (if the node does it), are these also passed through?

    I know it's a bit question banging... But i just want to do it right.


  • Code Contributor

    @John No worries! :)
    Presentation is On Todo.. Or Rather To think About list is presentation . I dont do anything with that as it is (Just ignores it). I havn't had any good idea for that yet.

    Internal is nothing more or less, it connects and set a variable that it is connected. :) But sure, there could be transport for internal messages also.
    I do have an experimental version there I can send internal messages to reboot node, reboot gateway and set clear read eeprom values. (over mqtt-protocol) but this s not finnished yet.. Ideas are welcome!


  • Code Contributor

    @John About Presentation ; My idea was that next version of the variable system I will either store some bytes in the MQTT gateway about what S_types to V_types, or hopefully the next protocol contains both those so the GW doesnt have to remember. And then add an extra directory to the MQTT path ( MyMQTT/[0-255]/[0-255]/S_[STRING]/V_[STRING]/[VALUE]


Log in to reply
 

Looks like your connection to MySensors Forum was lost, please wait while we try to reconnect.