💬 Building a MQTT Gateway

hek

This thread contains comments for the article "Building a MQTT Gateway" posted on MySensors.org.

Nicklas Starkel

Quick question.
I have built GatewayW5100MQTTClient according to mysensors.
It works and relays my nodes.
However, my old gateway with MqTT delivered the messages with my selected prefix and then node/child number + variable/sensor.
Ex MyMQTT/1/1/S_TEMP
Now the new gateway with MqTT delivers node and child and more like this:
MyMQTT/1/1/x/x/x where X is some number.
It does not display S_TEMP.
And what are the extra "numbers"?

And it also displays when the node starts up some MqTT messages that are empty ex:
MyMQTT/1/255/3/0/1 with no information.
I do not know what publishes these..

For relevance, the sketches I upload are the ones in the new mysensors library, ex Atmospheric Pressure Sensor.

FotoFieber

The new MQTT gateway exposes the serial api to MQTT. It has less functionality.

The old client gateway was a controller and translated the protocol to more readable messages and had a node-id handling.

I had the same problem and switched to node-red with a serial gateway.

hek

https://forum.mysensors.org/topic/2352/guide-setting-up-and-testing-mqtt-client-gateway

Nicklas Starkel

@FotoFieber

How do you mean it is a problem? Everything works, just needed to know what the extra was
I too use NODE RED, but just listening on MQTT and then putting the info into mysql databases.

@hek
Thank you! I have used search, but it is difficult when you do not know what you are searching for And all information about 2.0 is overwhelming at the time being for me.

hek

Thanks, good feedback @Nicklas-Starkel .

I just updated the main MQTT page, moving some of the instructions from the forum.

Ernst79

Seems that the W5100 sketch of the MQTT gateway is missing #include <SPI.h>.
I had to add it before #include <Ethernet.h>.

siod

As Ernst79 already mentioned: please add #include <SPI.h> as a switch to your code, thank you!

hek

https://github.com/mysensors/MySensors/issues/601

Martin Tellblom

Is it possible to have the MQTT client also act as a network gateway?

hek

Nope, sorry. You have to have to introduce a broker or controller acting as MQTT server.

Martin Tellblom

The old network gateway with mqtt seems to be removed and the client version is the one that id preferred. Is it possible to have both mqtt server and network gateway in that version?

Sorry for my ignorance but I would like to know how to connect the gateway to the sensor.
As I understand it, the radio communication is responsible for it, so can I put arduino + NRF24L01 + sensor at the other end?
Thank you!

Martin Tellblom

@DIM said:

my ignorance but I would like to k

The gateway is a arduino with NRF24L01+ (or RFM69) and serial/network/RS485 component. The sensor is another arduino with NRF24L01+ (or RFM69) and the sensor that are attached. The gateway is then acting as a translator from the NRF24L01+ (or RFM69) radiosignal to the medium that can talk to your controller

I thought that the gateway could be done only with the ESP8266 module.
Thanks!

CurlyWurly

If you're using an ESP8266 and you find it "SOFT WDT resetting" whenever a node tries to send a message, try moving the wire from GPIO4 to GPIO5 - There are a few esp12E boards which have been incorrectly screen printed - worked for me with this board!
https://twitter.com/bdcatalin/status/663434209265078272

Nicklas Starkel

Regarding ACK flag in MQTT.
If a sensor sends data to the gateway, and it requests an ACK.
It is the gateway that responds with an ACK, not the controller.

How do one utilize the ACK flag in MQTT?
Or rather, how do one get the ACK to display in a MQTT message?

void receive(const MyMessage &message) {
int value = debouncer.read();
//We only expect one type of message from controller. But we better check anyway.
 if (message.isAck()) {
    Serial.println("This is an ack from gateway");
 }
}

Should I write a codesnippet like the above but one that sends a message back to the gateway, that the ACK was received?
And also, if an ACK was not received that the sensor should try to send the message again.

Graham

I have set up an MQTT gateway running on a Wemos D1, and a motion sensor running on an arduino nano.

It seems to work successfully, but I had to define MY_NODE_ID in the sensor sketch because the negotiation for this is done by the controller. Presumably, I'm going to have to program my controller to respond to the mqtt message being sent to negiotiate the id, but I cannot find any documentation on the correct reply. Does someone have a link?

The mqtt message being sent to negotiate the node id was

mygateway1-out/255/255/3/0/3 (null)

proddy

I believe the node id will always be 0 for gateways so the examples are a little misleading. For example the output shows connections with a node id of 2 (e.g. mygateway1-out/2/1/1/0/49 55.722519;13.018121;13)

mpp

Can this be used with RabbitMQ as broker?

stephenmhall

Not sure if you want to leave the GPS on there unless that isn't your house with the double hot tub in tha back garden

Mountek Angel

Hi, is it possible to configure the MQTT gateway to let me in to port 5003 via telnet and read the communication message (not MQTT payloads) there also?

proddy

Not sure if this helps but for debugging I use MQTT spy (https://github.com/eclipse/paho.mqtt-spy/wiki) for tracking the comms

gohan

There is also mqtt fx that can be used to see what's going on on the mqtt server

Mountek Angel

I know what is going on the MQTT server and I am subscribed to the topics the Mysensors gateway created and publish data to it.

But, I want to know what is going on the GATEWAY as well. I'd like to see messages like these when I telnet to the gateway on port 5003 (that's the default as I remember)

0;255;3;0;9;Starting gateway (RNNGA-, 2.0.0)
0;255;3;0;9;TSM:INIT
0;255;3;0;9;TSM:RADIO:OK
0;255;3;0;9;TSM:GW MODE
0;255;3;0;9;TSM:READY
IP: 192.168.0.109
0;255;3;0;9;No registration required
0;255;3;0;9;Init complete, id=0, parent=0, distance=0, registration=1
0;255;3;0;9;TSP:MSG:READ 255-255-255 s=255,c=3,t=7,pt=0,l=0,sg=0:
0;255;3;0;9;TSP:MSG:BC
0;255;3;0;9;TSP:MSG:FPAR REQ (sender=255)
0;255;3;0;9;TSP:CHKUPL:OK (FLDCTRL)
0;255;3;0;9;TSP:MSG:GWL OK
0;255;3;0;9;TSP:MSG:SEND 0-0-255-255 s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=bc:0
0;255;3;0;9;TSP:MSG:READ 255-255-255 s=255,c=3,t=7,pt=0,l=0,sg=0:

gohan

There is the tool MYSController that connects to the gateway and shows all the messages

Samuel235

My gateway seems to be constantly repeating the following on the serial monitor:

IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0

Does this look like its in a reset loop or would you say there is some other issue going on?

If needed, my config is a Arduino Mega 2560 witha w5100 shield on top. I have specified the IP address of itself, the subnet, gateway, mqtt client IP (Controller), the mqtt topics, port, changed my LED pins.

I have a node with a button next to me, when i press it the message is picked up by the gateway. Below is what happens in the serial monitor when i press the node button.

IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0
0;255;3;0;9;TSF:MSG:READ,3-3-0,s=3,c=1,t=2,pt=2,l=2,sg=0:1
0;255;3;0;9;Sending message on topic: Gateway1-out/3/3/1/0/2

I'm a little confused at what point to troubleshoot here because I seem to think that for some reason it keeps refreshing its connection to the MQTT server. Not too sure why it is pushing a message to the topic of "Gateway1-out/0/255/0/0/18" either.

mfalkvidd

@Samuel235 see if the solution suggested by tekka in https://forum.mysensors.org/post/47225 can help (downgrade board definitions to 1.6.11)

mfalkvidd

@Samuel235
In case anyone needs to know where to downgrade: Tools -> Boards ->Boards Manager -> Select version for Arduini AVR Boards.

Samuel235

@mfalkvidd - It is still seeming to repeatedly reconnect to the MQTT server. I've issued a ping command to the arduino from the mqtt server and its constantly pinging without an issue, even while the serial monitor shows that its reconnecting. I'de be interested to know if anyone else is running the MQTT client gateway with MySensors V2.1. I used to run it when it was first introduced into the development environment in 1.6 and its been fine for over a year, but i upgraded to V2.1 and now i get this. So, i don't actually know if it is supposed to keep reconnecting to the MQTT server. I have read somewhere here that it sends a wake/refresh packet/message to the mqtt server just to keep it alive, but they said that was every 15 seconds by default. This is happening every 5/6 seconds.

IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0
IP: 192.168.0.22
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: Gateway1-out/0/255/0/0/18
0;255;3;0;9;Message arrived on topic: Gateway1-in/4/1/1/0/2
0;255;3;0;9;TSF:MSG:SEND,0-0-4-4,s=1,c=1,t=2,pt=0,l=1,sg=0,ft=0,st=OK:0

gohan

i think you need to post your code just to make more complete info.

Samuel235

I would like to know if anyone else's MQTT Gateway does this?

If this is the normal procedure I can carry on with config work for the nodes. Its pretty weird, the serial monitor is showing this but the gateway seems to be functioning. Nodes are talking to it. I'm not sure on that output topic though, why would it be sending something to Gateway1-out/0/255/0/0/18?

Samuel235

@gohan - I have not touched the MySensors.h or MyConfig.h but the sketch is below:

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik Ekblad
 *
 * DESCRIPTION
 * The W5100 MQTT gateway sends radio network (or locally attached sensors) data to your MQTT broker.
 * The node also listens to MY_MQTT_TOPIC_PREFIX and sends out those messages to the radio network
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error
 *
 * See http://www.mysensors.org/build/esp8266_gateway for wiring instructions.
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 *
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 *
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */


// Enable debug prints to serial monitor
#define MY_DEBUG

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_CLIENT

// Set this node's subscribe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "Gateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "Gateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// W5100 Ethernet module SPI enable (optional if using a shield/module that manages SPI_EN signal)
//#define MY_W5100_SPI_EN 4

// Enable Soft SPI for NRF radio (note different radio wiring is required)
// The W5100 ethernet module seems to have a hard time co-operate with
// radio on the same spi bus.
#if !defined(MY_W5100_SPI_EN) && !defined(ARDUINO_ARCH_SAMD)
#define MY_SOFTSPI
#define MY_SOFT_SPI_SCK_PIN 14
#define MY_SOFT_SPI_MISO_PIN 16
#define MY_SOFT_SPI_MOSI_PIN 15
#endif

// When W5100 is connected we have to move CE/CSN pins for NRF radio
#ifndef MY_RF24_CE_PIN
#define MY_RF24_CE_PIN 5
#endif
#ifndef MY_RF24_CS_PIN
#define MY_RF24_CS_PIN 6
#endif

// Enable these if your MQTT broker requires usenrame/password
//#define MY_MQTT_USER "username"
//#define MY_MQTT_PASSWORD "password"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 192,168,0,22

// If using static ip you need to define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 192,168,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0

// MQTT broker ip address or url. Define one or the other.
//#define MY_CONTROLLER_URL_ADDRESS "m20.cloudmqtt.com"
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 0, 21

// The MQTT broker port to to open
#define MY_PORT 1883

/*
// Enable inclusion mode
#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
// Uncomment to override default HW configurations
//
//
//
*/

#define MY_DEFAULT_ERR_LED_PIN 11  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  12  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  13  // the PCB, on board LED

#include <Ethernet.h>
#include <MySensors.h>

void setup()
{
}

void presentation()
{
	// Present locally attached sensors here
}


void loop()
{
	// Send locally attached sensors data here
}

EDIT: My issue was resting on the MQTT Server, had OpenHAB and MySensors gateway using same ClientID.

mlava

@Gabor-Bognar I also need to be able to do this.
Is there a solution to have a port 1883 for MQTT and port 5003 for telnet?

Gefkuz

Hi,

There is a missing parameter in your sketch :
// The MAC address can be anything you want but should be unique on your network.
// Newer boards have a MAC address printed on the underside of the PCB, which you can (optionally) use.
// Note that most of the Ardunio examples use "DEAD BEEF FEED" for the MAC address.
#define MY_MAC_ADDRESS 0x90, 0xA2, 0xDA, 0x10, 0xEB, 0x1D

In normal way, this line is note really important.
But if you want to use two separate MQTT on the same network, you should add this line and modify the mac adress on the two MQTT...

Bye

scottdube

Is it possible to have a single device acting as a ethernet gateway and an mqtt gateway?

gohan

No, but if you really really need the 2 connections you could use node-red to read data from the ethernet gateway and mirror them to mqtt

scottdube

I am wondering why you cannot built the ethernet gateway and combine the mqtt client sketch into it? I am trying to shift towards mqtt (experiment) while still having sensors available in Vera. I have the MQTT plugin on vera but have not yet figured out how to use the child device to control devices via MQTT. It seems like a waste to control the MQTT device only through Vera and I would never be able to shift away from vera that way. I'm new to this so I am still trying to understand the entire picture.

Thanks!

mfalkvidd

@scottdube my guess is that nobody with sufficient knowledge has had enough motivation and time to see if it can be done - especially since other solutions (nodered, myscontroller) already exist.

scottdube

@mfalkvidd I am missing something, the reason I want it to do both is that I want the flexibility to use Node Red and my Vera controller. I was thinking that as I built new sensors they would run through node red but be available to vera as well. I'm sure I just don't yet understand how people are designing their hybrid systems yet. My solution was to have the mysensors gateway also put the sensors values out to the mqtt broker and vice versa but there must be another way I am not seeing yet...

Thanks!

gohan

maybe it is time to switch away from Vera to something else that still uses zwave?

scottdube

@gohan Maybe that is the answer, I'll research that a bit... That was what I was hoping to do over time with my strategy...

mfalkvidd

@scottdube you can let Vera connect to Nodered.

See these threads for some ideas on what other people have done
https://forum.mysensors.org/topic/3891/2-controllers-and-1-gateway-expected-behavior/5
https://forum.mysensors.org/topic/4792/node-red/2
https://forum.mysensors.org/topic/9106/new-node-red-contrib-mysensors-out/2

joaoabs

I'd like to have a very simple (intra-home) web page in my raspberry PI that would be updated with all of my nodes information and that could also turn on/off nodes with actuators (basically I don't want to have a fancy controller).
If I understood correctly, with this MQTT Ethernet Gateway I could implement this by running Mosquito that would subscribe information from each node, through this MQTT gateway.
Does this mean I can avoid having a controller?
Cheers,

gohan

Technically speaking yes but I don't see the point in reinventing the wheel

รอเรือ

I rebuilt my serial gateway to a MQTT gateway. FWIW I had to connect 5V to my WizNET (W5100) Ethernet module (3.3V did not work) I couldn't make DHCP work so I used a fixed IP. Finally I had to replace the capacitor over the radio power feed from 4.7 uF capacitor to a 100 uF capacitor (That made a big difference!). Works great now.

Guillermo Schimmel

It should have:

void presentation() {
sendSketchInfo("Gateway node", "1.0");
}

In order to show the right topology in some tools, i.e. mycontroller.org

Thanks

sionut

Hi, is there any case for this gateway ? I wouldn't want all those wires to lay around in plain view..

mfalkvidd

@sionut "this gateway" is a bit ambiguous. The mqtt gateway can be based on different Arduinos and use different expansion boards, which will make a big difference for casing. Which variant are you planning to build?

sionut

That's a good question. I didn't realize that the arduino board is not specified (neither in this tutorial nor in the ethernet one).
However, I see that the ethernet gateway project (https://www.mysensors.org/build/ethernet_gateway) mentions that any arduino board would be fine. Since I'm only a beginner, I don't know what board would suffice for this kind of project. Could you recommend me the cheapest Arduino board that can do the job ?

mfalkvidd

@sionut if you have wifi, I would suggest esp8266 (wemos d1 mini if you want a specific board recommendation). It has built-in wifi so you don’t need an expansion board. Guide: https://www.mysensors.org/build/esp8266_gateway It can be run in mqtt mode if you prefer mqtt over plain tcp.

If you prefer ethernet, the Arduino Uno seems to be the one that matches the W5100 shield.

sionut

Great. I have wifi, so I will use the esp8266. I think I will continue with this board here as it's something I can find in a local store nearby.

I just need one more clarification though: in the ESP8266 gateway page I see the following paragraph:

Just connect the radio, install gateway software and you're good to go.

Now, considering that the board already has WIFI, why do I need the radio ?

mfalkvidd

@sionut the "normal" use of a MySensors network is to have multiple nodes using a cheap radio (nrf24) and a low-power microcontroller (Arduino Pro Mini is much more power efficient than the esp8266).

The gateway connects the nodes (using nrf24) to your controller. Without a radio, the gateway can't talk to the nodes. See this page in the gatting started guide for a more complete explanation.

However, if you don't need battery-powered nodes you could get a bunch of esp8266 and configure them all as gateways without adding radios to them.

AlaskaDIY

Is it feasible to have MySensors talking to an MQTT Broker like Mosquito (running on a rpi) via radio so using say a USB gateway AND also have WiFi modules talking to Mosquito and then having Mosquito talking to HomeAssistant? I see where this is doable via MySensors and radio links OR it can be done using WiFi modules but what if you want both?

electrik

Yes that is possible. Mysensors connects through the gateway to MQTT and the WiFi modules to mosquitto directly. You can use the mysensors protocol also over the WiFi modules I believe.
I'm not sure if it is possible with an usb gateway though, but for sure with a WiFi based gateway like ESP 32.
Once you are on mqtt you can connect to and from all devices/controllers that support it.

gohan

@alaskadiy I'd make a raspberry gateway with radio directly on the raspberry, much cleaner solution

AlaskaDIY

@gohan With a radio directly on the rPi would mosquito talk to both the radio and WiFi? I assume it would, I would think mosquito would just be a service that the radio or WiFi sends/receives data to but have not seen anyone talk about it.

Thanks

AlaskaDIY

@electrik Sounds like what I want to do can be done, thanks.

mfalkvidd

@alaskadiy no. The gateway (running as a service) would talk to the radio and to mosquitto. Mosquitto is unable to talk directly to the radio.

Guillermo Schimmel

Hi guys. I'm using an Arduino Uno+5100 based MQTT gateway. It works fine for it's gateway function but I'm not beeing able to send local sensor data. Is that possible?

mfalkvidd

@guillermo-schimmel it should be possible to call present() and send() just like on a regular node. Could you share your sketch and the debug output?

Guillermo Schimmel

@mfalkvidd thanks. I just did that on a dedicated post. It's driving me crazy.

https://forum.mysensors.org/topic/10109/ethernet-mqtt-gateway-slow-to-connect

mfalkvidd

@guillermo-schimmel ok, lets discuss there then so we don't split the effort and information in separate threads.

evb

Hi, receiving an upvote on a relatively old topic of mine, did remember me that this information could help others searching to build a MQTT gateway with a RFM69 radio, a W5100 Ethernet shield and a simple Arduino UNO R3.

https://forum.mysensors.org/topic/10358/solved-mqtt-gateway-based-on-arduino-uno-uno-compatible-w5100-ethernet-shield-rfm69hw-radio

People starting with MySensors, normally read as first step, the documentation available.
Every open source project is structurally a bit different and sometimes it is difficult to know how to contribute to the project.
For example, the open source project HomeAssistant (a possible controller for MySensors) uses a standard git repository for its documentation where anyone can make a pull request with their contribution. The moderators then review each contribution and either accept it or not. You have the official documentation and the 'user guides' documentation.
@mfalkvidd, maybe a workflow to consider? It can help to keep the documentation up to date faster and assess contributions better and faster.
Most open source projects run on volunteers who spend their free time on these projects, so I'm happy to return some knowledge.

In the meantime, I've learned here and realized that under every documentation page, there is a forum topic.
Unfortunately, documentation is usually not really up to date and the topic can be very long. Very hard to read for beginners.

That's why I created some forum topics with my first steps in the Mysensors world to overcome some outdated documentation. But of course these are hard to find among all the other topics and if you don't know anything about their existence, it's even harder.
And like everything, my topic is also already outdated, because, if I am correct, today a W5200 Ethernet shield is existing...

(for your information, my mqtt gateway is still alive and kicking )

tssk

Does anyone know if it would be possible to add static information that would be available over HTTP (default port 80)?

My idea is to put some kind of description/useful information in the MQTT GW itself. So in case Controller fails. There is some information resource what is the purpose, functionality and how to use it in some limited/emergency way eg. from phone or any computer.

EDIT: Thinking of it - it might also provide some kind of configuration interface...

frits

@tssk Nice idea! This should work, at the cost of some other feature which doesn't fit into ATMega's flash anymore.
Here's my gateway:

#define BUILD_MQTT_GATEWAY // either build mqtt gateway or ethernet gateway
// #define USE_SIGNING
#define MY_RADIO_RF24
// #define MY_RADIO_RFM69
#define WEBFRONTEND // provide a webfrontend at port 80 that shows gateway config
#define MY_DEBUG

#define SN "MySensors W5100 Gateway"
#define SV "0.2"

#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mysgw1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mysgw1-in"
#define MY_MQTT_CLIENT_ID "mysgw1"
#define MY_MQTT_CONTROLLER_IP_ADDRESS 192, 168, 99, 20
#define MY_MQTT_PORT 1883
#define MY_MYSGW_PORT 5003

#define MY_IP_ADDRESS 192,168,99,13
#define MY_MAC_ADDRESS 0xDE, 0xAD, 0xBA, 0xBE, 0x99, 0x13
#define MY_SIGNING_ATSHA204_PIN 17
#define MY_RFM69_NETWORKID (200)
#define MY_RFM69_TX_POWER_DBM 20
#define MY_SERIALDEVICE Serial

#if !defined(MY_W5100_SPI_EN) && !defined(ARDUINO_ARCH_SAMD)
  #define MY_SOFTSPI
  #define MY_SOFT_SPI_SCK_PIN (14)
  #define MY_SOFT_SPI_MISO_PIN (16)
  #define MY_SOFT_SPI_MOSI_PIN (15)
#endif

#ifdef MY_RADIO_RF24
  #define MY_RF24_CE_PIN (5)
  #define MY_RF24_CS_PIN (6)
#elif defined MY_RADIO_RFM69
  #define MY_IS_RFM69HW
  #define MY_RFM69_NEW_DRIVER
  #define MY_RF69_SPI_CS (14)
#else
  #error "only rfm24 or rf69 supported"
#endif

#ifdef BUILD_MQTT_GATEWAY
  #define MY_GATEWAY_MQTT_CLIENT
  #define MY_PORT MY_MQTT_PORT
  #define MY_CONTROLLER_IP_ADDRESS MY_MQTT_CONTROLLER_IP_ADDRESS
#else
  #define MY_GATEWAY_W5100
  #define MY_PORT MY_MYSGW_PORT
#endif

#ifdef USE_SIGNING
  #define MY_SIGNING_ATSHA204_PIN 17
  #define MY_SIGNING_ATSHA204 //!< Hardware signing using ATSHA204A
  #define MY_SIGNING_REQUEST_SIGNATURES
#endif

void printIpAddress ( Stream &s, uint8_t a, uint8_t b, uint8_t c, uint8_t d )
{
  s.print ( a );
  s.print ( "." );
  s.print ( b );
  s.print ( "." );
  s.print ( c );
  s.print ( "." );
  s.print ( d );
}

void printMacAddress ( Stream &s, uint8_t a, uint8_t b, uint8_t c, uint8_t d, uint8_t e, uint8_t f )
{
  s.print ( a, HEX );
  s.print ( ":" );
  s.print ( b, HEX );
  s.print ( ":" );
  s.print ( c, HEX );
  s.print ( ":" );
  s.print ( d, HEX );
  s.print ( ":" );
  s.print ( e, HEX );
  s.print ( ":" );
  s.print ( f, HEX );
}

void printConfig ( Stream &s, char * lineBreak )
{
  s.print ( F ( "***********************************************" ) );
  s.print ( lineBreak );
  s.print ( SN );
  s.print ( lineBreak );
  s.print ( "Version " );
  s.print ( SV );
  s.print ( lineBreak );
  s.print ( F("own mac address: ") );
  printMacAddress ( s, MY_MAC_ADDRESS );
  s.print ( lineBreak );
  s.print ( F("own network address: ") );
  printIpAddress ( s, MY_IP_ADDRESS );
  s.print ( lineBreak );
#ifdef BUILD_MQTT_GATEWAY
    s.print ( F("Gateway type: mqtt" ) );
    s.print ( lineBreak );
    s.print ( F ( "mqtt controller address: " ) );
    printIpAddress ( s, MY_CONTROLLER_IP_ADDRESS );
    s.print ( lineBreak );
    s.print ( F ( "mqtt controller port: " ) );
    s.print ( MY_MQTT_PORT );
    s.print ( lineBreak );
    s.print ( F ( "mqtt client id: " ) );
    s.print ( MY_MQTT_CLIENT_ID );
    s.print ( lineBreak );
    s.print ( F ( "mqtt publish prefix: " ) );
    s.print ( MY_MQTT_PUBLISH_TOPIC_PREFIX );
    s.print ( lineBreak );
    s.print ( F ( "mqtt subscribe prefix: " ) );
    s.print ( MY_MQTT_SUBSCRIBE_TOPIC_PREFIX );
    s.print ( lineBreak );
#else
    s.print ( F ( "Gateway type: MysGW" ) );
    s.print ( lineBreak );
    s.print ( F ( "network port: " ) );
    s.print ( MY_MYSGW_PORT );
    s.print ( lineBreak );
#endif
  s.print ( F ( "Radio type: " ) );
#ifdef MY_RADIO_RF24
    s.print ( F ( "NRF24" ) );
    s.print ( lineBreak );
#elif defined MY_RADIO_RFM69
    s.print ( F (  "RFM69" ) );
    s.print ( lineBreak );
    s.print ( F ( "network id: " ) );
    s.print ( MY_RFM69_NETWORKID );
    s.print ( lineBreak );
#else
    s.print ( F ( "bad radio defined" ) );
    s.print ( lineBreak );
#endif
  s.print ( F ( "signing: " ) );
#ifdef USE_SIGNING
    s.print ( "yes" );
#else
    s.print ( "no" );
#endif
  s.print ( lineBreak );
}

#include <Ethernet.h>
#include <MySensors.h>
#ifdef WEBFRONTEND
  byte mac[] = { MY_MAC_ADDRESS };
  IPAddress ip ( 192,168,99,13 );
  EthernetServer server ( 80 );
#endif

void setup() 
{
  uint32_t enter = hwMillis();
  while (hwMillis() - enter < (uint32_t)500);
  while(!MY_SERIALDEVICE); // For USB enabled devices, wait for USB enumeration before continuing
  printConfig( MY_SERIALDEVICE, "\n" );
#ifdef WEBFRONTEND
  Ethernet.begin(mac, ip);
#endif
}

void presentation() { }

void loop() 
{
#ifdef WEBFRONTEND
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println("Connection: close");  // the connection will be closed after completion of the response
          client.println();
          client.println("<!DOCTYPE HTML>");
          client.println("<html>");
          printConfig ( client, "<br />" );
          client.println("</html>");
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
  #endif
}

It will deliver this website to the browser:

***********************************************
MySensors W5100 Gateway
Version 0.2
own mac address: DE:AD:BA:BE:99:13
own network address: 192.168.99.13
Gateway type: mqtt
mqtt controller address: 192.168.99.20
mqtt controller port: 1883
mqtt client id: mysgw1
mqtt publish prefix: mysgw1-out
mqtt subscribe prefix: mysgw1-in
Radio type: NRF24
signing: no

tssk

@frits Cool! So you already had this?

I was exactly thinking of using most simple HTTP server example from Arduino examples (although you have it nicely dynamic using the defines etc.). But I was not sure if it would not interfere with mysensors library in some way or if maybe mysensors already had such functionality build-in (because if I remember correctly Ethernet gateway return some data to browser of you call the 5003 port).

frits

@tssk said in Building a MQTT Gateway:

@frits Cool! So you already had this?

no, not as a webserver. I only print some node information to Serial.

Webserver seems to work and not to interfere with mysensors core. My (MQTT-)Gateway doesn't act as a server, so there's only one listener. But multiple listeners (webserver on port 80 and Ethernet Gateway on port 5003 for example) should also be possible. I've not tested this though.

linkinpio

Hello,
I'm new here so sorry if this is not the right place for this question.
But I am trying to build Arduino MQTT GW to act as a relay/light controller and report/interact with HomeAssistant.
I think I have worked out most of my problems and got it running as I wanted to, but I was testing failover senarios as the controller should be able to work independently of any other device as long it has power.
My config is kind of simple Arduino Mega with W5500 Ethernet module configured as MQTT GW
And the problem is that when network or HA/MQTT broker is unavailable when Arduino is restarted/booting it tries to reconnect to MQTT broker constantly and interupts the main loop everytime for around 3s which makes board basically unresponsive.

00:45:07.094 -> 549536 !GWT:RMQ:FAIL
00:45:07.094 -> Loop count = 179
00:45:07.094 -> 549538 GWT:TPC:IP=192.168.11.111
00:45:08.107 -> 550542 GWT:RMQ:CONNECTING...
00:45:10.130 -> 552547 !GWT:RMQ:FAIL
00:45:10.130 -> Loop count = 180
00:45:10.130 -> 552550 GWT:TPC:IP=192.168.11.111

Any ideas how I could disable reconnect function if that happens for example after 3 attempts? So the controller can work normally even if there is no connection to MQTT broker?

OldSurferDude

@linkinpio

I don't have a solution, but I, too, may run into that problem. So I'm curious how it can be resolved.

Perhaps post in the Hardware or Development sections. Fortunately, some really knowledgeable people (eg. @mfalkvidd and @NeverDie ) monitor pretty much all of the forum subjects.

OSD

electrik

@linkinpio in the pubsubclient library, you can change the timeout on connecting to the mqtt broker. Default it is 15s, reducing it to e.g. 2s will increase the availability of the main loop.

mfalkvidd

@OldSurferDude yes I have read it already. But unfortunately my impression is that my knowledge of how the code works is woefully insufficient.

linkinpio

@electrik thank you for your suggestion, but are you talking about MQTT_SOCKET_TIMEOUT ?
If yes I did set this to 1s

#define MQTT_SOCKET_TIMEOUT 1

and did not change a thing

21:32:05.081 -> Loop count = 153
21:32:05.081 -> 471258 GWT:TPC:IP=192.168.11.111
21:32:06.098 -> 472262 GWT:RMQ:CONNECTING...
21:32:08.071 -> 474266 !GWT:RMQ:FAIL
21:32:08.071 -> Loop count = 154

If not please do let me know what you had in mind so I can check it out.

@OldSurferDude thanks for confirming that I have not missed anything when doing a resarch on this matter, if there will be no new suggestions I think I might have to post it to Development as dont see anyone suggesting any good workaround or a solution.

electrik

No, you're right, it doesn't solve your issue. It does increase the availability of the main loop though.

linkinpio

@electrik
Have you tested this in your enviroment/sketch?
Did it increase availability for your sketch?
If yes what was the single loop time before and after changing this?
I did test is on my example and it does not increase the availability for the main loop for my sketch, also you can see the printout from the serial monitor, the loop is run in the same 3s intervals like before, so maybe I am doing something wrong?

electrik

Yes, that's what I meant. But the value seems to be 2 seconds in your log. Where have you placed the define?
You can also use

setSocketTimeout(1)

On the pubsubclient object.

linkinpio

@electrik I've put the

#define MQTT_SOCKET_TIMEOUT 1

in the beginning where all other #defined constants are and it does not seeams to take this config.
I tried to use

setSocketTimeout(1)

but I can't seems to get the sketch to work with <PubSubClient.h>
I've added

EthernetClient ethClient;
PubSubClient client(ethClient);
client.setSocketTimeout(1);

but getting following error.

GatewayW5x00MQTTClient.ino:172:1: error: 'client' does not name a type
 client.setSocketTimeout(1);
 ^~~~~~

exit status 1

Compilation error: 'client' does not name a type

any ideas?

electrik

@linkinpio said in Building a MQTT Gateway:

client.setSocketTimeout(1);

That is the right syntax, but you have to move the line in setup()

linkinpio

@electrik
tried it but also no luck

C:\Users\p.\Documents\Arduino\GatewayW5x00MQTTClient\GatewayW5x00MQTTClient.ino: In function 'void setup()':
C:\Users\p.\Documents\Arduino\GatewayW5x00MQTTClient\GatewayW5x00MQTTClient.ino:213:10: error: 'class PubSubClient' has no member named 'setSocketTimeout'
   client.setSocketTimeout(1);
          ^~~~~~~~~~~~~~~~

exit status 1

Compilation error: 'class PubSubClient' has no member named 'setSocketTimeout'```

electrik

This looks like you're not using the latest (development) version of the mysensors library. That has the latest pubsubclient library included.
Looking at the intervals between the errors again, I'm doubtful if it will make a big difference though. The mysensors library will need to be modified to fulfill your request I guess

linkinpio

I think I'm ussing the latest one

but as you said I don't think it will fix my issue though.
But looking at the examples I found that they have a connect function which then can be used to reconnect whenever needed not constantly like it happens now,

But not sure how I could integrate that functionality in this sketch though, as I'm not that experienced with programming
To you think it will be possible?

/*
 Reconnecting MQTT example - non-blocking

 This sketch demonstrates how to keep the client connected
 using a non-blocking reconnect function. If the client loses
 its connection, it attempts to reconnect every 5 seconds
 without blocking the main loop.

*/

#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>

// Update these with values suitable for your hardware/network.
byte mac[]    = {  0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
IPAddress ip(172, 16, 0, 100);
IPAddress server(172, 16, 0, 2);

void callback(char* topic, byte* payload, unsigned int length) {
  // handle message arrived
}

EthernetClient ethClient;
PubSubClient client(ethClient);

long lastReconnectAttempt = 0;

boolean reconnect() {
  if (client.connect("arduinoClient")) {
    // Once connected, publish an announcement...
    client.publish("outTopic","hello world");
    // ... and resubscribe
    client.subscribe("inTopic");
  }
  return client.connected();
}

void setup()
{
  client.setServer(server, 1883);
  client.setCallback(callback);

  Ethernet.begin(mac, ip);
  delay(1500);
  lastReconnectAttempt = 0;
}


void loop()
{
  if (!client.connected()) {
    long now = millis();
    if (now - lastReconnectAttempt > 5000) {
      lastReconnectAttempt = now;
      // Attempt to reconnect
      if (reconnect()) {
        lastReconnectAttempt = 0;
      }
    }
  } else {
    // Client connected

    client.loop();
  }

}

💬 Building a MQTT Gateway

Suggested Topics

Missing Notification Settings
Bug Reports • • edoardoo

💬 Water Meter Pulse Sensor
Announcements • • hek

💬 Components
Announcements • • hek

Merry Christmas and Happy New 2023
Announcements • • mfalkvidd

💬 Real Time Clock Module, LCD Display and Controller Time
Announcements • • hek

💬 Building a Orange Pi Gateway
Announcements • • hek

58
Online

11.4k
Users

11.1k
Topics

112.6k
Posts

💬 Building a MQTT Gateway

Suggested Topics

Missing Notification Settings Bug Reports • • edoardoo

💬 Water Meter Pulse Sensor Announcements • • hek

💬 Components Announcements • • hek

Merry Christmas and Happy New 2023 Announcements • • mfalkvidd

💬 Real Time Clock Module, LCD Display and Controller Time Announcements • • hek

💬 Building a Orange Pi Gateway Announcements • • hek

58Online

11.4kUsers

11.1kTopics

112.6kPosts

Missing Notification Settings
Bug Reports • • edoardoo

💬 Water Meter Pulse Sensor
Announcements • • hek

💬 Components
Announcements • • hek

Merry Christmas and Happy New 2023
Announcements • • mfalkvidd

💬 Real Time Clock Module, LCD Display and Controller Time
Announcements • • hek

💬 Building a Orange Pi Gateway
Announcements • • hek

58
Online

11.4k
Users

11.1k
Topics

112.6k
Posts