MQTT Client Gateway / Node Controller / OTA

[CrankyCoder]

I am moving forward with some of my builds. I love the mqtt client gateway. This works great with my existing mqtt setup. There are 2 things I am unsure of.

1. is there a node controller type of system for the mqtt stuff? like myscontroller?
2. if I am using the mqtt client gateway. Is it possible to do OTA updates? If so, is there an example/documentation?

Thanks guys!

[maghac]

@Jason-Brunk
Did you get anywhere with this? I am also looking for OTA updates using the MQTT client gateway.

[CrankyCoder]

not really.

I was looking at this. https://www.mysensors.org/controller/mycontroller

Maybe using it for JUST the ota stuff and inventory.

[maghac]

@Jason-Brunk Ok, I'll have a look. Lika you say, maybe possible to use it only for OTA updates. I'm using HomeAssistant and I'm not ready to switch (especially to something which is version 0.0.3.Alpha2 )

[CrankyCoder]

Im with you. I don't want to switch off openhab just to be able to do updates

[Samuel235]

I'm also here with this predicament. I use OpenHAB and W5100 client gateway and would like to be able to push OTA updates. I'm also not really liking the idea of having to disconnect OpenHAB and then connect MyController to just issue updates. Would we need to add each device into MyController manually to see them? I've been playing with it but i can't even get it to connect properly yet.

Also, thinking about it, would we need to disconnect OpenHAB as its only on MQTT and technically doesn't interface with the gateway/mysensors other than taking readings off of MQTT topics, its not like a serial connection to the gateway or anything.....

[maghac]

Yes, I was hoping that since the only interface to the gateway is actually through MQTT, the controller wouldn't "own" it and MyController could also connect to it.

I didn't actually get as far as MyController since I never managed to burn the bootloader required for OTA updates and haven't had time to play with it more

[Samuel235]

@maghac - i'm going to play with the bootloader later today, if i can get that on some nanos (just to test) then i will load up MyController and connect to the MQTT. If i can get this working i can't see any issues with this plan, as like we have both said, the controller isn't owning the gateway. So, I personally don't see any issues AS LONG AS we do not set any rules or anything to trigger on anything within the MyController environment. The moment we do that it will then conflict with OpenHAB (or any other controller for that matter).

However, I may take a look around to see if we have any decent way of implementing an interface for tablets/scene controllers with MyController, if so this could even be an easier controller. Not sure if it has much capabilities like OpenHAB does with other protocols though. I run a Windows server at home so I'm okay with throwing that onto there if needs be. But i would like to stick with OpenHAB as I have learnt how it works (some what anyway).

Do you know of any info or guides on MyController?

I keep getting MyController and MYSController mixed up too, its a shame that MYSController doesn't support MQTT, that would have been perfect for our needs.

[gohan]

So basically if we want to make ota updates we need to temporarily replace the mqtt gateway with another ethernet gateway, wait for the sensors to reconnect, run updates and restore back the mqtt, correct?

[Samuel235]

@gohan - If MyController is able to push OTA updates over MQTT (which i'm about to test, give me an hour or two) then no. My theory is that because MQTT doesn't connect the controller directly to the gateway then we shouldn't need to shut anything down, only start MyController up and push the updates.

[vikasjee]

@Samuel235 What are your results?

[Samuel235]

@vikasjee - I've upgraded my gateway to MySensorsV2.1 and having some pretty bad results with just powering up. I'm waiting for people with V2.1 MQTT gateways to reply to threads. Do you run with MQTT gateways on V2.1 by any chance?

[maghac]

I am running a v2.1 MQTT Gateway on a NodeMCU (so it connects to the network with wifi) and it works fine.

It's a bit picky with the power supply but I think that's more a hardware than a software issue.

[Samuel235]

@maghac - I'm not sure how that setup works, but are you able to see any sort of serial monitor to debug or anything?

[maghac]

Yes, when I program it with Arduino IDE I use the serial monitor like normal with other boards. Never had any problems with it.

If you go back to an older version, does it work then?

[Samuel235]

@maghac I downgraded my boardDefs last night to 1.6.11 like advised on this forum. I'm about to install arduino IDE 1.6.11 and replace my latest install of Arduino and attempt to use this. What arduino IDE environment do you use, out of curiosity?

[Samuel235]

Exact same behaviour is happening. This is what is being printed to the serial monitor:

IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;!TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=NACK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;!TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=NACK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;!TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=NACK:0

I'm not even certain that anything is wrong here. It just keeps reconnecting to the MQTT server, but it is actually working but i'm concerned about why its doing such inside of the serial monitor. I read somewhere that it does a 'touching base' case to keep it alive or something, but i also recall someone saying that was every 15 seconds or so, this is every 8 seconds (about). Do you happen to know what its sending to a MQTT topic 0/255/0/0/18 for, is this standard behaviour or just mine?

[maghac]

Not sure if mine did that, I can check later.

I'm using the latest Arduino IDE (1.8.1 or something like that)

[Samuel235]

@maghac - The same as the one I've been using apart from the last test then. Thanks for confirming that.

If you wouldn't mind checking a serial output on your later, that would be really appreciated.

[maghac]

Type 18 is the heartbeat message type, I think this is the last element in your MQTT topic so it looks fine I think. Can you monitor the MQTT broker and see what messages are passing through it? (e.g if you can see something being sent by the sensors)

[Samuel235]

@maghac - I know the sensors are sending messages, the gateway is responding as it should to them, i was just concerned about the gateway keep connecting itself to the MQTT server, i was a little worried in case there is a feature in Mosquitto broker that declines the connection if it detects any sort of brute force or anything like that.

[Samuel235]

@maghac

On my MQTT server i'm sure enough receiving a message from the gateway every time it says its reconnected in the serial monitor, sending its version number to the server. By the looks of the serial monitor, its actually reconnecting and not just sending a new message to the MQTT topic. I'm confused to why this is doing this, I had just assumed that the heartbeat message is just a re-sending on a message on that topic. Would like to know if this is the normal behaviour of the gateway to keep constantly reconnecting.

[maghac]

Mine doesn't reconnect. Here's the serial monitor on startup:

0;255;3;0;9;MCO:BGN:INIT GW,CP=RNNGE--,VER=2.1.1
0;255;3;0;9;TSF:LRT:OK
0;255;3;0;9;TSM:INIT
0;255;3;0;9;TSF:WUR:MS=0
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 14
cnt 

connected with hacker, channel 9
dhcp client start...
0;255;3;0;9;TSM:INIT:TSP OK
0;255;3;0;9;TSM:INIT:GW MODE
0;255;3;0;9;TSM:READY:ID=0,PAR=0,DIS=0
0;255;3;0;9;MCO:REG:NOT NEEDED
f r0, scandone
....ip:192.168.1.215,mask:255.255.255.0,gw:192.168.1.1
.IP: 192.168.1.215
0;255;3;0;9;MCO:BGN:STP
OTA init
Ready
IP address: 192.168.1.215
0;255;3;0;9;MCO:BGN:INIT OK,TSP=1
IP: 192.168.1.215
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: mysensors-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: mysensors-in/200/255/3/0/6
0;255;3;0;9;!TSF:RTE:200 UNKNOWN
0;255;3;0;9;!TSF:MSG:SEND,0-0-200-200,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/254/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-254-254,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/222/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-222-222,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/222/255/3/0/1
0;255;3;0;9;!TSF:MSG:SEND,0-0-222-222,s=255,c=3,t=1,pt=0,l=10,sg=0,ft=0,st=NACK:1485810579
0;255;3;0;9;Message arrived on topic: mysensors-in/123/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-123-123,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/255/255/3/0/4
0;255;3;0;9;TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=4,pt=0,l=3,sg=0,ft=0,st=OK:234
0;255;3;0;9;Message arrived on topic: mysensors-in/227/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-227-227,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/227/1/1/0/24
0;255;3;0;9;!TSF:MSG:SEND,0-0-227-227,s=1,c=1,t=24,pt=0,l=3,sg=0,ft=0,st=NACK:102
0;255;3;0;9;Message arrived on topic: mysensors-in/233/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-233-233,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;Message arrived on topic: mysensors-in/234/255/3/0/6
0;255;3;0;9;!TSF:MSG:SEND,0-0-234-234,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=17,pt=5,l=4,sg=0:7160
0;255;3;0;9;Sending message on topic: mysensors-out/227/1/1/0/17
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=24,pt=5,l=4,sg=0:530
0;255;3;0;9;Sending message on topic: mysensors-out/227/1/1/0/24
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=18,pt=7,l=5,sg=0:0.5300
0;255;3;0;9;Sending message on topic: mysensors-out/227/1/1/0/18
pm open,type:2 0
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=17,pt=5,l=4,sg=0:7175
0;255;3;0;9;Sending message on topic: mysensors-out/227/1/1/0/17
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=24,pt=5,l=4,sg=0:570
0;255;3;0;9;Sending message on topic: mysensors-out/227/1/1/0/24
0;255;3;0;9;TSF:MSG:READ,227-227-0,s=1,c=1,t=18,pt=7,l=5,sg=0:0.5700

After this, it's just continously getting data from sensor 227 (my power meter). My mosquitto broker is getting this:

mysensors-out/0/255/0/0/18 2.1.1
mysensors-out/227/1/1/0/17 7160
mysensors-out/227/1/1/0/24 530
mysensors-out/227/1/1/0/18 0.5300
mysensors-out/227/1/1/0/17 7175
mysensors-out/227/1/1/0/24 570
mysensors-out/227/1/1/0/18 0.5700
mysensors-out/227/1/1/0/17 7195

etc etc - it's just the one message on startup.

[Samuel235]

@maghac - Right okay. Atleast i now have someone else's to compare too. So, something is pushing it off the network OR it can't hold its connection.

My start up sequence:

0;255;3;0;9;MCO:BGN:INIT GW,CP=RNNGA--,VER=2.1.1
0;255;3;0;9;TSM:INIT
0;255;3;0;9;TSF:WUR:MS=0
0;255;3;0;9;TSM:INIT:TSP OK
0;255;3;0;9;TSM:INIT:GW MODE
0;255;3;0;9;TSM:READY:ID=0,PAR=0,DIS=0
0;255;3;0;9;MCO:REG:NOT NEEDED
IP: 192.168.0.22
0;255;3;0;9;MCO:BGN:STP
0;255;3;0;9;MCO:BGN:INIT OK,TSP=1
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;!TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=NACK:0

The start is different to mine, that is probably because you're running that on a ESP8266, right?

I have connected to the MQTT server with MQTTFX, which is a MQTT monitoring tool. Not too familiar with it right now but i can't see the server dropping any connections or anything, it almost seems like the gateway is stuck in a loop.

[gohan]

Did you try to swap the mega for an arduino uno? Or swap the w5100 shield? Just to see if there is any difference, so we can exclude any hw issues

[Samuel235]

@gohan - The only thing that be causing this is software as the 1.6 branch works perfectly on this setup with the same sketch but back when it was 1.6 development.

So, if i'm thinking correctly, even if there is a issue between software and hardware, it should be fixable through software, hope this makes sense to anyone other than myself. I don't have an UNO, the only other thing i have is a nano. I'm currently using a W5100 shield, not module, but i do have some modules here, but they were a pain to get working before and hence why i made the switch to the shield.

EDIT: I can confirm that the shield is working properly as i have ran a http test sketch and i'm able to communicate from my web browser to the arduino. So it must be a software issue.

[Samuel235]

Update: I have tried MySensors versions V2.0, V2.1.1 and even the development branch. From what I can work out, V1.6 when it was a development branch got released as V2.0. But that isn't working either, not sure what state it was in before I changed to V2.1.1 as i never monitored the gateway before i reinstalled the new version.

Either way, none are working, they're all giving me the same result. Therefor i'm attempting to find an issue else where on the network other than MySensors. I'm going to try a new MQTT broker, on a different machine. If this gives the same result then its time to troubleshoot my network. Then if that doesn't show anything i think its safe to revert back to looking through my Gateway software.

[Samuel235]

I can confirm that the issues is not MySensors, I've just created a virtual machine running Ubuntu and installed Mosquitto there. Connected the gateway and perfect functionality. Everything is working as intended with a nice silent serial output until a message is sent to the gateway.

[maghac]

I suppose this is on your own LAN, so there are no firewalls or anything that could block the MQTT traffic? Can you see anything in the mosquitto log?

[Samuel235]

@maghac - It is on the LAN, no firewalls that are stopping it unless they're on the ubuntu box themselves but i had a quick look at that at the start of my issues and the two ports were open for MQTT. So unless something is up with the allowance settings, then its not that. I'm about to find the location of the MQTT logs to check that now. I'm connected to the MQTT server with MQTT.fx and the log is filling with commands, can't see anything obvious right now. Will check the logs on the machine to see any issues recorded.

Would you expect to see anything in any of the logs specifically or just check them all? Do you happen to know where they're located?

[Samuel235]

UPDATE: The issue has been resolved.

I have no clue why it has only just became and issue because my OpenHAB hasn't been upgraded or modified since day one. But the issue was that the OpenHAB install was attempting to join the MQTT broker (on the same system) with the same ID as the MySensors Gateway. Once i changed my config of OpenHAB MQTT connection, its all working perfect now.

Please, if you see anyone having this issue, point them to this message. I may add it to the controller section or ask a MOD/ADMIN to add it to the OpenHAB controller page if they feel its something that maybe worth pointing out.

Thank you all for your continued help with this. Now, time to get back to the issue at heart. I will be testing the use of MyController today without having to disconnect OpenHAB at all.

[mfalkvidd]

@Samuel235 great that you found the problem, and thanks for reporting back.

[Samuel235]

@mfalkvidd - In the process of monitoring the network now to check everything is still normal behaviour. I may be getting a random message published, keeping an eye on this. I would say if I do have an issue, it has nothing to do with software though

I'm looking into MyController as we speak just for the OTA updates alone then returning to normal controller.

[Samuel235]

Update: I've got a MQTT/persistence error on MyController, so i'm waiting on a software developer for them to advise me on a solution.

[jkandasa]

@Samuel235 What is the error are you facing? Did you report anywhere?

[Samuel235]

@jkandasa - You have replied to my post, just about to post a reply for you.

For others joining about this error, please see linked thread as this thread is just for the use of MyController for OTA updates only.

[Samuel235]

Still applying configuration on my OpenHAB and MyController setup. At the moment, it looks like its a very possible pairing to work for our needs. But I'm not completely installed together yet. In theory, it is working. Just applying that now

• Got them both installed.
• Got OpenHAB2 configured.
• Need to change my MyController port.
• Need to sort out the bootloader issue i'm having with MyBootloader not going to sketch properly in boot-up.

[Samuel235]

Everything working except for the node is not automatically rebooting on new firmware command. Will update on manual reboot.

[maghac]

@Samuel235 Great news! How does your final setup look like - how did you configure MyController? And what is the steps to follow when publishing an OTA update to the nodes?

Maybe the reboot issue can be solved by manually sending a message to the node to reboot it? Assuming there is such a functionality in the protocol of course.

[Samuel235]

@maghac - I have been speaking to a couple of people, one of which is Jkandasa himself, to my understanding, using MYSBootloader does not enable us to issue a firmware update and then to automatically reboot itself. However, as you suggested, there is a reboot option in MyController. As far as I understand (I may be slightly wrong here, but i don't think i am) every time a MYSBootloader enabled node boots up, it asks MyController if there is a firmware update for it (is there a linked firmware to me that i do not have?) and if so, it goes ahead and downloads it.

I'm happy to confirm i have this all working now.

A vague outline of what we have to do:

• Install MyController.
• Edit the conf files (On linux its: <MYCONTROLLER INSTALL LOCATION>/conf/mycontroller.properties) to change the port that it uses from 8443 to anything you wish, i have set mine to 9443 and save the changes to that conf file.
  issue the start command inside of the /bin folder and then use your browser to go to https://<IPOFMYCONTROLLERMACHINE>:<PORTNUMBER> then you should be greeted with the login page for your mycontroller install.
• log in with admin and admin (change these).
• Navigate to the Settings>MQTT Broker page and disable the MQTT broker functionality.
• Navigate to Resources>Gateway and then click Add gateway. This is where you enter the MQTT settings to enable MyController to find your gateway and nodes.
• Give the gateway a name, then select the MySensors type and MQTT. Add your details here for the MQTT server.
• Once added, you can select your gateway and in the drop down menu (top right above the list of gateways) you can select discovery, this should automatically discover your nodes.

To send a new firmware to the node, follow the guide on MySensors to create the hex file. then follow the section related to firmware on the user guide of MyController. Once you have saved a new firmware, you then go into the nodes list and associate that node with the new firmware by editing the settings on that node and there should be a firmware box that your able to drop down and select an uploaded firmware. Then you can select the node and issue a reboot from the drop down at the top right of the list of nodes. The version control of the firmware files that you upload is really awesome! I must thank @jkandasa for that!

If you have any issues at all, please ask more questions here, i'm willing to help. I may even write a little more in-depth guide for others to use. That way we can always link people to this thread for the information to do such.

I have a few little bits of config left to do in terms of security of the server and then i'm all set.

[maghac]

Wow, I was actually able to get this to work now. Like you I also noticed that the nodes do not reboot automatically, but since you can execute a reboot from the MyController UI, it's not such a big problem. I also learnt that you need to use smartSleep() rather than sleep() in your sketch, otherwise the node won't process the incoming reboot request.

It also takes a while for the firmware to upload, but that's OK. Still saves a lot of time compared to manually uploading new sketches.

Next step is to try this on some "real" battery-operated nodes. Currently I'm only testing it on one test node which doesn't do anything except blink the LED.

[Samuel235]

@maghac - yeah, i read about that smartsleep() function. I'm really glad that you managed to get this to work dude! I don't care how long the firmware takes to upload, as long as i know its doing it, and i monitor my MQTT server broker for this. Its not about saving time for me, its about the fact i couldn't get to some nodes that are in the ceiling, so this just saves literally having to rip floors up or lights down. Its awesome!

I'm yet to install it on any nodes in practical uses yet. But my latest board is running it and soon as tests have finished, i will be sure to use it in full time production.

[rozpruwacz]

@Samuel235 great stuff ! Worked like a charm one question, when using with dualoptiboot, how to check if firmware upload is done ? now I have to stop the gateway and run it in debug mode, then i can see all the messages. In MYSController there is log with all the messages, is there any such log in the MyController ?

[p359]

@Samuel235 thanks for listing your steps. I cannot get my laptop to link to mqtt on pi gateway. Do you have to have mycontroller on the pi? I have all working on laptop, can log in and disable inbuilt mqtt broker, just unable to connect to mosquitto on pi. I use tcp://(ip of pi):1883 Thankyou for any help!

[gohan]

Are you sure mosquitto is running? What conf are you using for it?

[p359]

@gohan Hi. I'll say tentative yes, all mysensors coms from nodes go through mosquitto to domoticz via pi gateway. Ready to be told otherwise if config needed? Cheers

[p359]

Progress report. Looks like config issue. On mosquitto config, if I comment out

"Binding 127.0.0.1"

Then can connect via laptop to mosquitto via mycontroller. Issue then is mysensor gateway fails and can't see nodes or data haha.

Will have to put my controller onto pi unless some other way to bind?

[gohan]

Comment that line in the conf file and try again, as that allows connections only from the local host and not from the network

[p359]

@gohan sorry, my post not so clear. I comment this out as you suggest then mysensor nodes stop reporting in to gateway.

[gohan]

can you login to the mqtt server with any mqtt client for computer (like mqtt spy or similar)?