💬 Building a MQTT Gateway

[mfalkvidd]

@sionut if you have wifi, I would suggest esp8266 (wemos d1 mini if you want a specific board recommendation). It has built-in wifi so you don’t need an expansion board. Guide: https://www.mysensors.org/build/esp8266_gateway It can be run in mqtt mode if you prefer mqtt over plain tcp.

If you prefer ethernet, the Arduino Uno seems to be the one that matches the W5100 shield.

[sionut]

Great. I have wifi, so I will use the esp8266. I think I will continue with this board here as it's something I can find in a local store nearby.

I just need one more clarification though: in the ESP8266 gateway page I see the following paragraph:

Just connect the radio, install gateway software and you're good to go.

Now, considering that the board already has WIFI, why do I need the radio ?

[mfalkvidd]

@sionut the "normal" use of a MySensors network is to have multiple nodes using a cheap radio (nrf24) and a low-power microcontroller (Arduino Pro Mini is much more power efficient than the esp8266).

The gateway connects the nodes (using nrf24) to your controller. Without a radio, the gateway can't talk to the nodes. See this page in the gatting started guide for a more complete explanation.

However, if you don't need battery-powered nodes you could get a bunch of esp8266 and configure them all as gateways without adding radios to them.

[AlaskaDIY]

Is it feasible to have MySensors talking to an MQTT Broker like Mosquito (running on a rpi) via radio so using say a USB gateway AND also have WiFi modules talking to Mosquito and then having Mosquito talking to HomeAssistant? I see where this is doable via MySensors and radio links OR it can be done using WiFi modules but what if you want both?

[electrik]

Yes that is possible. Mysensors connects through the gateway to MQTT and the WiFi modules to mosquitto directly. You can use the mysensors protocol also over the WiFi modules I believe.
I'm not sure if it is possible with an usb gateway though, but for sure with a WiFi based gateway like ESP 32.
Once you are on mqtt you can connect to and from all devices/controllers that support it.

[gohan]

@alaskadiy I'd make a raspberry gateway with radio directly on the raspberry, much cleaner solution

[AlaskaDIY]

@gohan With a radio directly on the rPi would mosquito talk to both the radio and WiFi? I assume it would, I would think mosquito would just be a service that the radio or WiFi sends/receives data to but have not seen anyone talk about it.

Thanks

[AlaskaDIY]

@electrik Sounds like what I want to do can be done, thanks.

[mfalkvidd]

@alaskadiy no. The gateway (running as a service) would talk to the radio and to mosquitto. Mosquitto is unable to talk directly to the radio.

[Guillermo Schimmel]

Hi guys. I'm using an Arduino Uno+5100 based MQTT gateway. It works fine for it's gateway function but I'm not beeing able to send local sensor data. Is that possible?

[mfalkvidd]

@guillermo-schimmel it should be possible to call present() and send() just like on a regular node. Could you share your sketch and the debug output?

[Guillermo Schimmel]

@mfalkvidd thanks. I just did that on a dedicated post. It's driving me crazy.

https://forum.mysensors.org/topic/10109/ethernet-mqtt-gateway-slow-to-connect

[mfalkvidd]

@guillermo-schimmel ok, lets discuss there then so we don't split the effort and information in separate threads.

[evb]

Hi, receiving an upvote on a relatively old topic of mine, did remember me that this information could help others searching to build a MQTT gateway with a RFM69 radio, a W5100 Ethernet shield and a simple Arduino UNO R3.

https://forum.mysensors.org/topic/10358/solved-mqtt-gateway-based-on-arduino-uno-uno-compatible-w5100-ethernet-shield-rfm69hw-radio

People starting with MySensors, normally read as first step, the documentation available.
Every open source project is structurally a bit different and sometimes it is difficult to know how to contribute to the project.
For example, the open source project HomeAssistant (a possible controller for MySensors) uses a standard git repository for its documentation where anyone can make a pull request with their contribution. The moderators then review each contribution and either accept it or not. You have the official documentation and the 'user guides' documentation.
@mfalkvidd, maybe a workflow to consider? It can help to keep the documentation up to date faster and assess contributions better and faster.
Most open source projects run on volunteers who spend their free time on these projects, so I'm happy to return some knowledge.

In the meantime, I've learned here and realized that under every documentation page, there is a forum topic.
Unfortunately, documentation is usually not really up to date and the topic can be very long. Very hard to read for beginners. :-(

That's why I created some forum topics with my first steps in the Mysensors world to overcome some outdated documentation. But of course these are hard to find among all the other topics and if you don't know anything about their existence, it's even harder.
And like everything, my topic is also already outdated, because, if I am correct, today a W5200 Ethernet shield is existing...

(for your information, my mqtt gateway is still alive and kicking ;-) )

[tssk]

Does anyone know if it would be possible to add static information that would be available over HTTP (default port 80)?

My idea is to put some kind of description/useful information in the MQTT GW itself. So in case Controller fails. There is some information resource what is the purpose, functionality and how to use it in some limited/emergency way eg. from phone or any computer.

EDIT: Thinking of it - it might also provide some kind of configuration interface...

[frits]

@tssk Nice idea! This should work, at the cost of some other feature which doesn't fit into ATMega's flash anymore.
Here's my gateway:

#define BUILD_MQTT_GATEWAY // either build mqtt gateway or ethernet gateway
// #define USE_SIGNING
#define MY_RADIO_RF24
// #define MY_RADIO_RFM69
#define WEBFRONTEND // provide a webfrontend at port 80 that shows gateway config
#define MY_DEBUG

#define SN "MySensors W5100 Gateway"
#define SV "0.2"

#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mysgw1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mysgw1-in"
#define MY_MQTT_CLIENT_ID "mysgw1"
#define MY_MQTT_CONTROLLER_IP_ADDRESS 192, 168, 99, 20
#define MY_MQTT_PORT 1883
#define MY_MYSGW_PORT 5003

#define MY_IP_ADDRESS 192,168,99,13
#define MY_MAC_ADDRESS 0xDE, 0xAD, 0xBA, 0xBE, 0x99, 0x13
#define MY_SIGNING_ATSHA204_PIN 17
#define MY_RFM69_NETWORKID (200)
#define MY_RFM69_TX_POWER_DBM 20
#define MY_SERIALDEVICE Serial

#if !defined(MY_W5100_SPI_EN) && !defined(ARDUINO_ARCH_SAMD)
  #define MY_SOFTSPI
  #define MY_SOFT_SPI_SCK_PIN (14)
  #define MY_SOFT_SPI_MISO_PIN (16)
  #define MY_SOFT_SPI_MOSI_PIN (15)
#endif

#ifdef MY_RADIO_RF24
  #define MY_RF24_CE_PIN (5)
  #define MY_RF24_CS_PIN (6)
#elif defined MY_RADIO_RFM69
  #define MY_IS_RFM69HW
  #define MY_RFM69_NEW_DRIVER
  #define MY_RF69_SPI_CS (14)
#else
  #error "only rfm24 or rf69 supported"
#endif

#ifdef BUILD_MQTT_GATEWAY
  #define MY_GATEWAY_MQTT_CLIENT
  #define MY_PORT MY_MQTT_PORT
  #define MY_CONTROLLER_IP_ADDRESS MY_MQTT_CONTROLLER_IP_ADDRESS
#else
  #define MY_GATEWAY_W5100
  #define MY_PORT MY_MYSGW_PORT
#endif

#ifdef USE_SIGNING
  #define MY_SIGNING_ATSHA204_PIN 17
  #define MY_SIGNING_ATSHA204 //!< Hardware signing using ATSHA204A
  #define MY_SIGNING_REQUEST_SIGNATURES
#endif

void printIpAddress ( Stream &s, uint8_t a, uint8_t b, uint8_t c, uint8_t d )
{
  s.print ( a );
  s.print ( "." );
  s.print ( b );
  s.print ( "." );
  s.print ( c );
  s.print ( "." );
  s.print ( d );
}

void printMacAddress ( Stream &s, uint8_t a, uint8_t b, uint8_t c, uint8_t d, uint8_t e, uint8_t f )
{
  s.print ( a, HEX );
  s.print ( ":" );
  s.print ( b, HEX );
  s.print ( ":" );
  s.print ( c, HEX );
  s.print ( ":" );
  s.print ( d, HEX );
  s.print ( ":" );
  s.print ( e, HEX );
  s.print ( ":" );
  s.print ( f, HEX );
}

void printConfig ( Stream &s, char * lineBreak )
{
  s.print ( F ( "***********************************************" ) );
  s.print ( lineBreak );
  s.print ( SN );
  s.print ( lineBreak );
  s.print ( "Version " );
  s.print ( SV );
  s.print ( lineBreak );
  s.print ( F("own mac address: ") );
  printMacAddress ( s, MY_MAC_ADDRESS );
  s.print ( lineBreak );
  s.print ( F("own network address: ") );
  printIpAddress ( s, MY_IP_ADDRESS );
  s.print ( lineBreak );
#ifdef BUILD_MQTT_GATEWAY
    s.print ( F("Gateway type: mqtt" ) );
    s.print ( lineBreak );
    s.print ( F ( "mqtt controller address: " ) );
    printIpAddress ( s, MY_CONTROLLER_IP_ADDRESS );
    s.print ( lineBreak );
    s.print ( F ( "mqtt controller port: " ) );
    s.print ( MY_MQTT_PORT );
    s.print ( lineBreak );
    s.print ( F ( "mqtt client id: " ) );
    s.print ( MY_MQTT_CLIENT_ID );
    s.print ( lineBreak );
    s.print ( F ( "mqtt publish prefix: " ) );
    s.print ( MY_MQTT_PUBLISH_TOPIC_PREFIX );
    s.print ( lineBreak );
    s.print ( F ( "mqtt subscribe prefix: " ) );
    s.print ( MY_MQTT_SUBSCRIBE_TOPIC_PREFIX );
    s.print ( lineBreak );
#else
    s.print ( F ( "Gateway type: MysGW" ) );
    s.print ( lineBreak );
    s.print ( F ( "network port: " ) );
    s.print ( MY_MYSGW_PORT );
    s.print ( lineBreak );
#endif
  s.print ( F ( "Radio type: " ) );
#ifdef MY_RADIO_RF24
    s.print ( F ( "NRF24" ) );
    s.print ( lineBreak );
#elif defined MY_RADIO_RFM69
    s.print ( F (  "RFM69" ) );
    s.print ( lineBreak );
    s.print ( F ( "network id: " ) );
    s.print ( MY_RFM69_NETWORKID );
    s.print ( lineBreak );
#else
    s.print ( F ( "bad radio defined" ) );
    s.print ( lineBreak );
#endif
  s.print ( F ( "signing: " ) );
#ifdef USE_SIGNING
    s.print ( "yes" );
#else
    s.print ( "no" );
#endif
  s.print ( lineBreak );
}

#include <Ethernet.h>
#include <MySensors.h>
#ifdef WEBFRONTEND
  byte mac[] = { MY_MAC_ADDRESS };
  IPAddress ip ( 192,168,99,13 );
  EthernetServer server ( 80 );
#endif

void setup() 
{
  uint32_t enter = hwMillis();
  while (hwMillis() - enter < (uint32_t)500);
  while(!MY_SERIALDEVICE); // For USB enabled devices, wait for USB enumeration before continuing
  printConfig( MY_SERIALDEVICE, "\n" );
#ifdef WEBFRONTEND
  Ethernet.begin(mac, ip);
#endif
}

void presentation() { }

void loop() 
{
#ifdef WEBFRONTEND
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println("Connection: close");  // the connection will be closed after completion of the response
          client.println();
          client.println("<!DOCTYPE HTML>");
          client.println("<html>");
          printConfig ( client, "<br />" );
          client.println("</html>");
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
  #endif
}

It will deliver this website to the browser:

***********************************************
MySensors W5100 Gateway
Version 0.2
own mac address: DE:AD:BA:BE:99:13
own network address: 192.168.99.13
Gateway type: mqtt
mqtt controller address: 192.168.99.20
mqtt controller port: 1883
mqtt client id: mysgw1
mqtt publish prefix: mysgw1-out
mqtt subscribe prefix: mysgw1-in
Radio type: NRF24
signing: no

[tssk]

@frits Cool! So you already had this?

I was exactly thinking of using most simple HTTP server example from Arduino examples (although you have it nicely dynamic using the defines etc.). But I was not sure if it would not interfere with mysensors library in some way or if maybe mysensors already had such functionality build-in (because if I remember correctly Ethernet gateway return some data to browser of you call the 5003 port).

[frits]

@tssk said in 💬 Building a MQTT Gateway:

@frits Cool! So you already had this?

no, not as a webserver. I only print some node information to Serial.

Webserver seems to work and not to interfere with mysensors core. My (MQTT-)Gateway doesn't act as a server, so there's only one listener. But multiple listeners (webserver on port 80 and Ethernet Gateway on port 5003 for example) should also be possible. I've not tested this though.

[linkinpio]

Hello,
I'm new here so sorry if this is not the right place for this question.
But I am trying to build Arduino MQTT GW to act as a relay/light controller and report/interact with HomeAssistant.
I think I have worked out most of my problems and got it running as I wanted to, but I was testing failover senarios as the controller should be able to work independently of any other device as long it has power.
My config is kind of simple Arduino Mega with W5500 Ethernet module configured as MQTT GW
And the problem is that when network or HA/MQTT broker is unavailable when Arduino is restarted/booting it tries to reconnect to MQTT broker constantly and interupts the main loop everytime for around 3s which makes board basically unresponsive.

00:45:07.094 -> 549536 !GWT:RMQ:FAIL
00:45:07.094 -> Loop count = 179
00:45:07.094 -> 549538 GWT:TPC:IP=192.168.11.111
00:45:08.107 -> 550542 GWT:RMQ:CONNECTING...
00:45:10.130 -> 552547 !GWT:RMQ:FAIL
00:45:10.130 -> Loop count = 180
00:45:10.130 -> 552550 GWT:TPC:IP=192.168.11.111

Any ideas how I could disable reconnect function if that happens for example after 3 attempts? So the controller can work normally even if there is no connection to MQTT broker?

[OldSurferDude]

@linkinpio

I don't have a solution, but I, too, may run into that problem. So I'm curious how it can be resolved.

Perhaps post in the Hardware or Development sections. Fortunately, some really knowledgeable people (eg. @mfalkvidd and @NeverDie ) monitor pretty much all of the forum subjects.

OSD