Guide: Setting up and testing MQTT Client Gateway

[hek]

(this post currently applies to the development branch)

You can run the MQTT gateway on an Arduino+W5100 ethernet module or the ESP8266. Connect radio and ethernet module exactly like for the normal gateway.

Download and install the MySensors development branch

The use one of the following sketches depending on the hardware you use:
https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/examples/GatewayW5100MQTTClient/GatewayW5100MQTTClient.ino
or
https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/examples/GatewayESP8266MQTTClient/GatewayESP8266MQTTClient.ino

The topic resembles the serial protocol. You must define you preferred subscribe and publish prefix in the sketch. The topic is build like this:
MY_MQTT_PUBLISH_TOPIC_PREFIX/FROM-NODE-ID/SENSOR-ID/CMD-TYPE/ACK-FLAG/SUB-TYPE

Read more about the serial protocol here. An example topic for data received from your gateway could look like this:

mygateway1-out/2/1/1/0/49

If you want to send data to your sensors use MY_MQTT_SUBSCRIBE_TOPIC_PREFIX defined in your sketch. Should be looking like this (using default sketch):

mygateway1-in/2/1/1/0/49

You can test your gateway using mosquitto as broker.

Make sure to set MY_IP_GATEWAY_ADDRESS in gateway pointing to you computers ip-number (on the same lan).

Start the broker (leave this running):
> mosquitto

Now start your gateway and you should see something like this in the gateway log. Note that I have a GPS sensor running in my radio network which send data to the gateway all the time:

0;0;3;0;9;gateway started, id=0, parent=0, distance=0
0;0;3;0;9;Attempting MQTT connection...
0;0;3;0;9;MQTT connected
0;0;3;0;9;read: 2-2-0 s=1,c=1,t=49,pt=0,l=22,sg=0:55.722519;13.018120
0;0;3;0;9;Sending message on topic: mygateway1-out/2/1/1/0/49
0;0;3;0;9;read: 2-2-0 s=1,c=1,t=49,pt=0,l=22,sg=0:55.722519;13.018121
0;0;3;0;9;Sending message on topic: mygateway1-out/2/1/1/0/49

Subscribe to messages (in another shell on your computer)
> mosquitto_sub -v -t 'mygateway1-out/#'

You should see the messages like you saw in the gateway-log showing up like this:

mygateway1-out/2/1/1/0/49 55.722519;13.018121;13
mygateway1-out/2/1/1/0/49 55.722519;13.018114;12
mygateway1-out/2/1/1/0/49 55.722527;13.018120;11
mygateway1-out/2/1/1/0/49 55.722534;13.018122;10

Sending (publishing) messages to your sensor network:
> mosquitto_pub -t 'mygateway1-in/2/1/1/0/49' -m '0,29'

[Mike Cayouette]

@hek said:

MY_MQTT_SUBSCRIBE_TOPIC_PREFIX

You must have read my mind, I was coming here to ask how do you subscribe? LOL

Excellent job as usual.

Mike

[Mike Cayouette]

Not sure I'm doing this properly. I started with only this

#define MY_MQTT_TOPIC_PREFIX "mygateway"

My sketch was able to subscribe and publish to the mygateway topic.

I then went to this:

#define MY_MQTT_TOPIC_PREFIX "mygateway"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

but my gateway was not able to receive anything from "mygateway1-in" but it was still receiving from "mygateway".

I then went to this:

#define MY_MQTT_TOPIC_PREFIX "mygateway"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"

but still not able to send or get anything from "mygateway1-in" or "mygateway1-out"

What am I doing wrong?

Also, when I send the following from mosquitto:
mygateway/106/3/1/0/2

I get a st=fail error.

"0;0;3;0;9;Message arrived on topic: mygateway/106/3/1/0/2
0;0;3;0;9;send: 0-0-0-106 s=3,c=1,t=2,pt=0,l=3,sg=0,st=fail:100"

Any help would would be appreciated.

Thank you,

Mike

[Mike Cayouette]

@hek said:
MY_MQTT_SUBSCRIBE_TOPIC_PREFIX

I am unable to find this anywhere in the mysensors library files. I find MY_MQTT_TOPIC_PREFIX in MyGatewayTransportMQTTClient.cpp, but that is it.

Should MY_MQTT_SUBSCRIBE_TOPIC_PREFIX not be there also?

Mike

[hek]

@Mike-Cayouette
Had to let the pull request containing MY_MQTT_SUBSCRIBE_TOPIC_PREFIX /MY_MQTT_PUBLISH_TOPIC_PREFIX build before merging it. Now it's in place.

"st=fail" means that your gw doesn't receive any ack on the sent radio message.

[Mike Cayouette]

@hek said: "st=fail" means that your gw doesn't receive any ack on the sent radio message

Thank you for the quick response. Do I have to recompile all my sketches using the dev branch? Right now all my sensor sketches are compiled with version 1.5.

Everything works fine with my serial gateway. When I do my testing with the ESP MQTT gateway I turn off my serial gateway to avoid conflicts and when I restart one of my sensors it sends the sketch version and it presents its self and I see this in the gateways serial interface and I also see it in mosquitto. My sketches seem to have no problem sending to the gateway, but the gateway does not appear to be able to send anything to the sketch.

MIke

[hek]

@Mike-Cayouette said:

Do I have to recompile all my sketches using the dev branch? Right now all my sensor sketches are compiled with version 1.5.

No, shouldn't be needed.

[Mike Cayouette]

@hek

I remembered that ack is disabled on my sketches, as I did not see a point in using it with the serial gateway connected to node-red. I will recompile my sketch with ack enabled.

Thank you for your help.

Mike

[hek]

Not the same type of ack. This is the one sent automatically by the NRF radio. You can't disable it in sketch.

[Mike Cayouette]

I have tried a few things but without success, I keep getting the following output on the gateway when I try to send a message to a sensor

0;0;3;0;9;send: 0-0-0-106 s=3,c=1,t=2,pt=0,l=1,sg=0,st=fail:0

Oddly the sensors have no problem sending a message to the gateway. For example when I reset my dimmer the serial output on the gateway is the following

;0;3;0;9;read: 106-106-0 s=255,c=0,t=17,pt=0,l=3,sg=0:1.5
0;0;3;0;9;Sending message on topic: mygateway1-out/106/255/0/0/17
0;0;3;0;9;read: 106-106-0 s=255,c=3,t=6,pt=1,l=1,sg=0:0
0;0;3;0;9;Sending message on topic: mygateway1-out/106/255/3/0/6

I have used an external power supply rather than connect the radio to the 3.3v pin on the nodemcu breakout board but that did not change anything, I also added the following but without luck

#define MY_RF24_PA_LEVEL RF24_PA_LOW

The only thing I have not done was compile the sensor with the library in the dev branch, but I was under the impression I did not need to.

Any help would be appreciated.

Thank you,

Mike

[ahmedadelhosni]

@Mike-Cayouette Can you check what I have found and tell me whether it temporary solved your problem or not ? http://forum.mysensors.org/topic/2193/gatewayesp8266mqttclient-in-development-branch/11

[Mike Cayouette]

@ahmedadelhosni That worked, I can now send messages from my gateway.

Thank you for your help

Mike

[ahmedadelhosni]

@Mike-Cayouette Glad that it worked. Please note that these would be temporary workarounds for MQTT support only. I guess that won't work if the sensor node is far away from the gateway and needs a repeater in the middle, because in that workaround you don't check your look up table.

[Mike Cayouette]

@ahmedadelhosni I have not had to add any repeaters to my network yet so I should be good until a more permanent solution is worked into the library. Thank you again for you solution.

@hek Do you expect a more permanent fix for this solution to be pushed to the dev branch?
http://forum.mysensors.org/topic/2193/gatewayesp8266mqttclient-in-development-branch/11

Thank you,

Mike

[hek]

Ok, verified sending a message to a node here now when repeater mode is enabled. Works fine and message is received.

Here is the change I made. Soon merged into development.
https://github.com/mysensors/Arduino/pull/251

[Lars65]

Great work. I use an ESP8266 12E, and it works just fine. :)

[John]

All though this topic is related to the development branch, has the topic structure been fixed (except for prefix)?

[ahmedadelhosni]

@hek What about the first issue which I reported about ? Still returns TRUE and returns.

if (strcmp_P(str, MY_MQTT_SUBSCRIBE_TOPIC_PREFIX) != 0) {
	// Message not for us or malformed!
       	return;
}

Is it reproducible at your side ?

[hek]

@ahmedadelhosni

No, I couldn't reproduce any problems with topic naming.

I use the default subscribe/publish prefix in the sketch and can send messages without any problem.

[Samuel235]

@hek said:

You can run the MQTT gateway on an Arduino+W5100 ethernet module or the ESP8266. Connect radio and ethernet module exactly like for the normal gateway.

Have we managed to get a MQTT Gateway working on a Arduino with an ENC28J60 module rather than a w5100?