Switching from v1.5 MQTT gateway to ESP8266 MQTT Client gateway

[mbj]

I decided to test the change from the old (v 1.4 and 1.5) MQTT Gateway (broker) to the new gateway which is a MQTT client. I use Openhab (on a RPi) as controller and must say that the old MQTT gateway has been working fine all the time so the change was triggered mostly because the current gateway is stated to be obsolete for the next MySensors release.

This is just a brief summary the experience so far:

As I had a couple of Nodemcu ESP8266 and no W5100 laying around, the natural choice was to test the ESP8266MQTTClientGateway from the development branch. This is a client why also a broker must be installed, in my case Mosquitto on the Openhab RPi system.

The ESP8266 installation was painless. It worked more or less "out of the box" after a couple of small compilation errors due to choosing DHCP. These were easily eliminated and I also switched to using a fixed IP anyhow.

Mosquitto became a hassle. Everything seemed to be working, I could subscribe/publish messages which were visible over the network but Mosquitto refused to accept connections from the ESP8266 but worked fine for other connections. I tried to change MQTT version as well as other things but without luck. For the install I used the Mosquitto Wheezy repository to get the latest version but something was clearly wrong.

Did bite the tongue and made a completely fresh Raspbian installation based on the Jessie version (was on the list anyhow), a fresh Openhab installation and a Mosquitto from the Raspbian Jessie repository. Believe it or not, Mosquitto started to work as it should.

Then Openhab became the nightmare (I am not exaggerating). First I made a mistake with not observing that the MySensors version of the MQTT topic had changed quite a bit from v1.5. My own fault.

Anyhow, this means that Openhab item definitions have to be adjusted to the new topic structure (not a big job, "find and replace" in a good editor does it swiftly).

Once the items were OK I hoped for success but that was in vain. The MQTT messages from/to the gateway via Mosquitto were completely ignored by Openhab even though the connection to Mosquitto was accepted and established with no error loggings at all.

Of course I tried the old binding to the version 1.5 gateway again and this worked like a charm. I made some test items in Openhab and tested bindings to Mosquitto on the RPi and Mosquitto on Windows and found that Openhab showed outbound commands for these test items on the bus and also sent them to the broker but refused to read anything coming in.

The bindings used were of the transport configuration type which had worked well for the v1.5 gateway. After many hours of work and as a last resort I decided to test adding an event bus binding configuration to the Mosquitto broker definitions in Openhab. And even if I had made an error in the topic definitions the MQTT messages STARTED ROLLING IN!!!!!

So just by adding a stateSubscribeTopic and a commandSubscribeTopic in an event bus configuration Openhab seemed to have loaded up what was missing. This was never needed for the binding to the v1.5 MQTT gateway and there is no logic in why it made a difference for the Mosquitto binding. Especially so because the topic definitions I made for the new binding were wrong.

Now the ESP8266 test installation has worked without glitches for a day and the messages keep rolling in to Openhab. State changes as well as commands sent back to the sensor boxes via the ESP8266 gateway works fine as well.

And I am keeping my fingers crossed.

I assume I am not the only one using Openhab and the v1.4 - v1.5 MQTT Gateway and hope this might give some info to the "topic" of changing the gateway.

[Yveaux]

@mbj Thanks for posting your results and solutions!

[Matt Pitts]

@mbj Great to see you have this working, I have also been trying to get the W5100 MQTT Client up and running with OpenHAB. I think I'm also there.

Would you mind posting your openhab.cfg binding configs?? as I'm struggling with this too.

Thanks
Matt

[mbj]

@Matt-Pitts It works really well so far. Here are the relevant
parts of my openhab.cfg:

Openhab MQTT to Mosquitto

mqtt:mosq1.url=tcp://localhost:1883
mqtt:mosq1.clientId=MyMQTT-2

Openhab MQTT Eventbus broker

mqtt-eventbus:broker=mosq1
mqtt-eventbus:stateSubscribeTopic=MyMQTT-2/stateUpdates/${item}/state
mqtt-eventbus:commandSubscribeTopic=MyMQTT-2/commandUpdates/${item}/command

The event bus broker part should not be needed but was the one that made it work.

Good luck.

[Matt Pitts]

Thanks, that's really helpful, sorry for my ignorance but where do you setup the client ID? Am i right in assuming the client ID is the one from the mqtt client sketch??

[mbj]

@Matt-Pitts Just here and nowhere else. It is an arbitrary definition because an auto id will be generated if not stated. Since it seemed to be needed for the broker part I added it even though. The Openhab pages for the MQTT binding will tell you the details.

Update: Should have added that the clientID here is a private affair between Openhab and Mosquitto.

The item bindings will typically look like:

incoming to Openhab: {mqtt="<[mosq1:mygateway1-out/22/1/1/0/17:state:default]"}
outgoing from Openhab {mqtt=">[mosq1:mygateway1-in/22/1/1/0/24:state:*:default]"}

I guess this will answer your question, all designations are confusing. Took me time to sort it out.

[Matt Pitts]

@mbj Thanks again, I had figured out the item binding which are working fine, although I don't see any messages in the openhab log when they get a status update?? although they are obviously updating

So I setup the openhab.cfg with the following, I think that's correct, but I still don't see any MQTT messages when running tail -f /var/log/openhab/openhab.log

I was a little confused by your setting because your topic was the same as your clientid, I assume that was just coincidence?

I have used the topic ID's for the eventbus broker from the mysenors sketch in the bindings in the openhab.cfg which are below.

################################# MQTT Transport ######################################
#
# Define your MQTT broker connections here for use in the MQTT Binding or MQTT
# Persistence bundles. Replace <broker> with a id you choose.
#

# URL to the MQTT broker, e.g. tcp://localhost:1883 or ssl://localhost:8883
mqtt:matt_broker.url=tcp://localhost:1883

# Optional. Client id (max 23 chars) to use when connecting to the broker.
# If not provided a default one is generated.
mqtt:matt_broker.clientId=openhabmqtt

# Optional. User id to authenticate with the broker.
mqtt:matt_broker.user=mattpitts74

# Optional. Password to authenticate with the broker.
mqtt:matt_broker.pwd=landy90

# Optional. Set the quality of service level for sending messages to this broker. 
# Possible values are 0 (Deliver at most once),1 (Deliver at least once) or 2 
# (Deliver exactly once). Defaults to 0.
#mqtt:<broker>.qos=<qos>

# Optional. True or false. Defines if the broker should retain the messages sent to
# it. Defaults to false.
#mqtt:<broker>.retain=<retain>

# Optional. True or false. Defines if messages are published asynchronously or
# synchronously. Defaults to true.
#mqtt:<broker>.async=<async>

# Optional. Defines the last will and testament that is sent when this client goes offline
# Format: topic:message:qos:retained <br/>
#mqtt:<broker>.lwt=<last will definition>



#######################################################################################
#####                        Binding configurations                               #####
#######################################################################################
mqtt-eventbus:broker=matt_broker
mqtt-eventbus:stateSubscribeTopic=mygateway1-out/stateUpdates/${item}/state
mqtt-eventbus:commandSubscribeTopic=mygateway1-in/commandUpdates/${item}/command

Should I be seeing the mqtt messages in the log as they come in?? as currently they don't seem to be showing up???

[mbj]

@Matt-Pitts I have been out for the evening so pls excuse for not replying. I will take a look at it tomorrow but in principle every message which match an item in Openhab will be found in the event.log, at least for the installation I have (i assume you run Openhab as a daemon, if being run from a shell command you will see any matching messages in the opehab.log also)
So if not shown in the event.log it has not been received OK. A few things to check are:

• The message structure has changed from the 1.5 version to the development branch sketches. What I listed above is according the new format. In principle 2 new fields are added and V_XXX is changed to the corresponding numerical code. If the message topic and type is different from what is stated in the item definition nothing will be picked up by Openhab.

• Check the messages from the gateway using for example the mosquitto _sub client. This will tell you if the messaged are sent and what format they have.

[Matt Pitts]

@mbj now worries, thanks for helping me out, I'm running openHAB as a serivce my my rpi2 along with mosquitto.

I have noticed these occasional messages in the event.log:

2016-02-13 20:36:25.823 [DEBUG] [o.o.i.m.i.MyOpenHABServiceImpl] - store(lounge_Temp), state = 23.0

I also checked by subscribing with mosquitto _sub client and get the following.

mosquitto_sub -d -t mygateway1-out/3/0/1/0/1
Client mosqsub/14104-raspberry sending CONNECT
Client mosqsub/14104-raspberry received CONNACK
Client mosqsub/14104-raspberry sending SUBSCRIBE (Mid: 1, Topic: mygateway1-out/3/0/1/0/1, QoS: 0)
Client mosqsub/14104-raspberry received SUBACK
Subscribed (mid: 1): 0
Client mosqsub/14104-raspberry sending PINGREQ
Client mosqsub/14104-raspberry received PINGRESP
Client mosqsub/14104-raspberry sending PINGREQ
Client mosqsub/14104-raspberry received PINGRESP
Client mosqsub/14104-raspberry sending PINGREQ
Client mosqsub/14104-raspberry received PINGRESP
Client mosqsub/14104-raspberry received PUBLISH (d0, q0, r0, m0, 'mygateway1-out/3/0/1/0/1', ... (4 bytes))
37.0
Client mosqsub/14104-raspberry sending PINGREQ
Client mosqsub/14104-raspberry received PINGRESP

I don't really know what this should look like, but from my very basic understanding it looks like messages are at least being published. Otherwise OpenHab wouldn't be able to do this

[mbj]

@Matt-Pitts When running Openhab as a daemon some typical listings in my event.log are:

2016-02-14 08:05:24 - node2_pulsecounter_in state updated to 30568
2016-02-14 08:05:24 - Kwh_node2 state updated to 30.5680
2016-02-14 08:05:24 - Kwh_diff state updated to 0.8050
2016-02-14 08:05:25 - Temp_node2_freezer state updated to -19.4
2016-02-14 08:05:53 - Temp_dining state updated to 21.9

Your listing:
2016-02-13 20:36:25.823 [DEBUG] [o.o.i.m.i.MyOpenHABServiceImpl] - store(lounge_Temp), state = 23.0
shows that Openhab has received this value and updated the state. My guess is that you run Openhab with debug output enabled. At least I get this logging format but then in openhab.log when I run from a debug shell.

For moquitto you will see all messages if you subscribe to topic # and if you set the -v flag also the output is a bit more verbose, example:
mosquitto_sub -h 192.168.1.107 -t # -v (I need the -h switch because I do not run it from localhost)

A typical output gives topic and payload and looks like:
mygateway1-out/24/1/1/0/0 -10.6
mygateway1-out/24/2/1/0/1 96

Your topic line will show just this the topic mygateway1-out/3/0/1/0/1 and the -d flag shows debug values.

Since you get a log message and also the graph these values of course must be coming in so if this is the only incoming value you have all is OK. If you have more sensors sending which do not show up in the log then most likely there are errors in the mqtt binding for these items.

Just a comment, in your previous post about the openhab.cfg one line for the broker states:

mqtt-eventbus:commandSubscribeTopic=mygateway1-in/commandUpdates/${item}/command

I think this should be changed to
mqtt-eventbus:commandSubscribeTopic=mygateway1-out/commandUpdates/${item}/command

[DirkB19]

@mbj said:

mqtt-eventbus:broker=mosq1
mqtt-eventbus:stateSubscribeTopic=MyMQTT-2/stateUpdates/${item}/state
mqtt-eventbus:commandSubscribeTopic=MyMQTT-2/commandUpdates/${item}/command

Hi mbj,
I'm following you on this thread now Because I'm also on OpenHAB.

Just some questions ,
1 Could you confirm if the following topic structure change is correct in my item bindings ;
BEFORE =
Number BadK_TempDH "T. BADKAMER [%.1f °C]" { mqtt="<[RPIBroker:MyMQTT/25/11/V_TEMP:state:default]"
AFTER =
Number BadK_TempDH "T. BADKAMER [%.1f °C]" { mqtt="<[RPIBroker:MyMQTT/25/11/1/0/0::state:default]"

1. Think it matters if the in- and outgoing TOPIC is the same ?
   So instead of
   incoming to Openhab: {mqtt="<[RPIBroker:mygateway1-out/22/1/1/0/17:state:default]"}
   outgoing from Openhab {mqtt=">[RPIBroker:mygateway1-in/22/1/1/0/24:state::default]"}
   I would have ;
   incoming to Openhab: {mqtt="<[RPIBroker:MyMQTT/22/1/1/0/17:state:default]"}
   outgoing from Openhab {mqtt=">[RPIBroker:MyMQTT/22/1/1/0/24:state::default]"}

Thing is, I wanna make sure before I start changing 40-50 item definitions.
Thx again,
DirkB

[mbj]

@DirkB19 Hi Dirk,
Looking at this link http://forum.mysensors.org/topic/2352/guide-setting-up-and-testing-mqtt-client-gateway you will find the new topic to be MY_MQTT_PUBLISH_TOPIC_PREFIX/FROM-NODE-ID/SENSOR-ID/CMD-TYPE/ACK-FLAG/SUB-TYPE . Thus the difference is the added /CMD-TYPE/ACK-FLAG/ plus that the V_XXXX is changed to the corresponding number. So to me your item 1 change looks perfectly OK.

Your item 2 is a bit unclear to me and these are of course dependent on your definitions in the sketch. I did use the preset data so my incoming to Openhab typically looks like:

{mqtt="<[mosq1:mygateway1-out/22/1/1/0/25:state:default]"}

And the outgoing (in this case a command) :

{mqtt=">[mosq1:mygateway1-in/27/1/1/0/24:command:*:default]"}

The in- and outgoing need a different definition (mygateway-out versus mygateway-in in these examples) as far as I can understand from the publish/subscribe logic in MQTT.

Hope it makes sense.

[DirkB19]

@mbj

thanks, meanwhile I've changed all my item definitions, and I can confirm the following ;

1. new topic definition is OK, it works
2. I confirm it works with the same MY_MQTT_PUBLISH_TOPIC_PREFIX for both in and out, simply "MyMQTT" in my case.

Next hurdles are ;
3. convert all my nodes to DEVELOPMANT BRANCH LIBRARIES, using info from : http://forum.mysensors.org/topic/2742/problem-compiling-sketch-on-library-1-5-devel/2
I've got one node converted now and it seems to work, except that the DHT22 sensor now seems to report in Fahrenheit (before it was in Celcius) ? Strange is that before I uploaded the node with dev libs I had accidentally reset it and from then (even with 1.5 LIBS) it started reporting temp in Fahrenheit to my MQTT client gateway on DEV LIBS ?

[mbj]

@DirkB19
I have not looked into what happens when using same prefix so this is just an unconfirmed thought:

The MQTT broker relays any message received to anybody listening. So when using same prefix for publish and subscribe there ought to be a risk that your gateway reads and forwards to the sending node same message which it just did send.

Many nodes do not read incoming messages and then this does not matter but for nodes reading incoming messages then there might be some issues if , for example, the outgoing message being interpreted as an incoming setpoint.

Would be interesting to hear if anybody knowing the details could comment on this.

I have at least one node running with DHT22 and the code from the development branch and without any problems. Just one thought - this code is using a different DHT library compared to the v1.5 version if I remember right. Nothing you have overlooked there?

[martinhjelmare]

For the mqtt model of communication to be able to support two way communication, publish and subscribe topics can't be the same at each device.

Comm goes like this:

point1 -> publish -> broker -> subscribe -> point2
point1 <- subscribe <- broker <- publish <- point2

So each device, point1 and point2, in the example above, must have at least two different topics, one for publishing and one for subscribing to messages, to enable both sending and receiving messages.

For pure sensors devices that only send/publish messages, the publish topic is the only relevant one. But since for mysensors, this is set up at the gateway, for all nodes, no matter if they are pure sensors or if they are actuators, you have to have different publish and subscribe prefixes to the topics.

From the dev branch 5100 mqtt gateway sketch:

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

I'm not sure exactly how the mqtt is set up in openhab, but if openhab should be able to both send and receive messages via mqtt it will need to handle both publish and subscribe topics and those should not be the same.

[mbj]

@martinhjelmare
Thank's for the explanation which follows what is my own understanding of the MQTT setup and procedures.

When defining items in Openhab obviously there is no problem handling both publish and subscribe topics.

[DirkB19]

And still, I have it working both ways with the same MY_MQTT_PUBLISH_TOPIC_PREFIX for in and out going messages.
I have sensors and actuators both working (still on 1.5 LIBS but I guess that has nothing to do with it ?).
But just to be sure, I will change it to 2 different TOPIC_PREFIXes as suggested in above posts.
Thx.

[mbj]

@DirkB19 Think you will get another advantage by changing as well - must be a lot easier to monitor the MQTT messages when topic clearly shows the difference.