Broken gateway with local sensors on MQTT ESP8266 GW

[mrumpf]

I try to add the same sensors I have on my nodes also to the gateway.
Unfortunately the gateway does not seem to be able to receive messages from the nodes anymore when local sensors are attached.
On the nodes I get some MSG:SEND:FAIL output which indicates that the gateway is not responding anymore.
I learned from the forum that I should not use sleep(10000) in the loop() function to avoid that the gateway goes to sleep. The debug output however was telling me that sleep was not possible because the gateway is in repeater mode. Instead wait(10000) should be used.
But even with wait() I get the same behavior. The gateway is simply not responding to messages from the nodes anymore.
When I comment the code for the local sensors everything is fine again and I see debug output of messages from the nodes on the gateway.
I also tried to remove the sleep/wait altogether but that also did not work. The node was again increasing the fail counter in this case, e.g. "TSM:FAIL:CNT=7".

I'm new to MySensors and I could not find a hint neither in the forum, nor in the documentation.

[tekka]

@mrumpf Can you share more details about your setup & sensors used, also posting your sketch and the full debug log are a good starting point for troubleshooting.

[mrumpf]

The strange thing is that after 930865s a re-init was done and afterwards the messages from node 1 were received successfully.

...
930865 TSM:FAIL:RE-INIT
930869 TSM:INIT
930877 TSM:INIT:TSP OK
930881 TSM:INIT:GW MODE
930886 TSM:READY:ID=0,PAR=0,DIS=0
930892 TSM:READY:NWD REQ
930932 TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=20,pt=0,l=0,sg=0,ft=0,st=OK:
959517 TSF:MSG:READ,1-1-255,s=255,c=3,t=7,pt=0,l=0,sg=0:
959528 TSF:MSG:BC
959531 TSF:MSG:FPAR REQ,ID=1
959536 TSF:PNG:SEND,TO=0
959540 TSF:CKU:OK
959544 TSF:MSG:GWL OK
959609 TSF:MSG:SEND,0-0-1-1,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=OK:0
961577 TSF:MSG:READ,1-1-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1
961587 TSF:MSG:PINGED,ID=1,HP=1
961600 TSF:MSG:SEND,0-0-1-1,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=OK:1
962080 TSF:MSG:READ,1-1-0,s=0,c=1,t=23,pt=3,l=2,sg=0:0
962090 GWT:TPS:TOPIC=mygateway1-out/1/0/1/0/23,MSG SENT
962102 TSF:MSG:READ,1-1-0,s=2,c=1,t=1,pt=7,l=5,sg=0:31.76
962112 GWT:TPS:TOPIC=mygateway1-out/1/2/1/0/1,MSG SENT
962124 TSF:MSG:READ,1-1-0,s=1,c=1,t=0,pt=7,l=5,sg=0:23.56
962134 GWT:TPS:TOPIC=mygateway1-out/1/1/1/0/0,MSG SENT
962145 TSF:MSG:READ,1-1-0,s=3,c=1,t=4,pt=7,l=5,sg=0:963.05
962156 GWT:TPS:TOPIC=mygateway1-out/1/3/1/0/4,MSG SENT
...

and on the node side

...
113469 TSF:MSG:SEND,1-1-255-255,s=255,c=3,t=7,pt=0,l=0,sg=0,ft=0,st=OK:
115478 !TSM:FPAR:NO REPLY
115480 TSM:FPAR
115517 TSF:MSG:SEND,1-1-255-255,s=255,c=3,t=7,pt=0,l=0,sg=0,ft=0,st=OK:
117526 !TSM:FPAR:FAIL
117528 TSM:FAIL:CNT=7
117530 TSM:FAIL:DIS
117532 TSF:TDI:TSL
177537 TSM:FAIL:RE-INIT
177539 TSM:INIT
177545 TSM:INIT:TSP OK
177549 TSM:INIT:STATID=1
177551 TSF:SID:OK,ID=1
177553 TSM:FPAR
177590 TSF:MSG:SEND,1-1-255-255,s=255,c=3,t=7,pt=0,l=0,sg=0,ft=0,st=OK:
178278 TSF:MSG:READ,0-0-1,s=255,c=3,t=8,pt=1,l=1,sg=0:0
178284 TSF:MSG:FPAR OK,ID=0,D=1
179599 TSM:FPAR:OK
...

Might this be related to an unstable power supply? I do not have the capacitor yet. Both devices (gateway via ESP8266 USB & node via PL-2303HX) are attached to the USB ports of my 2015 Macbook Pro.

[mrumpf]

My sketch for the Gateway looks like this:

#define MY_DEBUG
#define MY_BAUD_RATE 57600

#define MY_RADIO_RF24
#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"
#define MY_MQTT_CLIENT_ID "mygateway1"
#define MY_WIFI_SSID "*****"
#define MY_WIFI_PASSWORD "*********************"
#define MY_HOSTNAME "mygateway1"
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 58
// #define MY_CONTROLLER_URL_ADDRESS "test.mosquitto.org"
#define MY_PORT 1883


#define LOCAL_SENSORS
#define LOCAL_SENSORS_BH1750
#define LOCAL_SENSORS_BME280

#include <ESP8266WiFi.h>
#include <SPI.h>
#include <MySensors.h>

#ifdef LOCAL_SENSORS
#include <i2cdetect.h>
#include <Wire.h>

#ifdef LOCAL_SENSORS_BH1750
#include <BH1750FVI.h>
// light
#define CHILD_ID_LIGHT 0
BH1750FVI lightSensor(13, BH1750FVI::k_DevAddress_L, BH1750FVI::k_DevModeContHighRes);
MyMessage lightMsg(CHILD_ID_LIGHT, V_LIGHT_LEVEL);
#endif

#ifdef LOCAL_SENSORS_BME280
// temperature & humidity & pressure
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#define CHILD_ID_TEMPERATURE 1
#define CHILD_ID_HUMIDITY 2
#define CHILD_ID_PRESSURE 3
#define CHILD_ID_DEWPOINT 4
Adafruit_BME280 bme;
MyMessage temperatureMsg(CHILD_ID_TEMPERATURE, V_TEMP);
MyMessage humidityMsg(CHILD_ID_HUMIDITY, V_HUM);
MyMessage pressureMsg(CHILD_ID_PRESSURE, V_PRESSURE);
#endif

#endif // LOCAL_SENSORS

void setup()
{
#ifdef LOCAL_SENSORS

#ifdef LOCAL_SENSORS_BH1750
  Wire.begin(D3, D4);
  //Wire.setClock(100000);

  Serial.println("Init: BH1750 (Light)");
  lightSensor.begin();
#endif

#ifdef LOCAL_SENSORS_BME280
  Serial.println("Init: BME280 (Temperature, Humidity, Pressure)");
  if (!bme.begin(0x77)) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    while (1);
  }
#endif

#endif
}

void presentation()
{
#ifdef LOCAL_SENSORS
  // Send the sketch version information to the gateway and Controller
  sendSketchInfo("Light/Temperature/Humidity/Pressure Gateway", "1.0");

  // Register all sensors to gateway (they will be created as child devices)
#ifdef LOCAL_SENSORS_BH1750
  present(CHILD_ID_LIGHT, S_LIGHT_LEVEL);
#endif

#ifdef LOCAL_SENSORS_BME280
  present(CHILD_ID_TEMPERATURE, S_TEMP);
  present(CHILD_ID_HUMIDITY, S_HUM);
  present(CHILD_ID_PRESSURE, S_BARO);
#endif
  
#endif
}

void loop()
{
#ifdef LOCAL_SENSORS
  i2cdetect();

#ifdef LOCAL_SENSORS_BH1750
  // light
  uint16_t lux = lightSensor.GetLightIntensity();
  Serial.print("Light: ");
  Serial.println(lux);
  send(lightMsg.set(lux));
#endif

#ifdef LOCAL_SENSORS_BME280
  // humidity
  float humidity = bme.readHumidity();
  Serial.print("Humidity: ");
  Serial.println(humidity);
  send(humidityMsg.set(humidity, 2));

  // temperature
  float temperatureC = bme.readTemperature();
  Serial.print("Temperature: ");
  Serial.println(temperatureC);
  send(temperatureMsg.set(temperatureC, 2));

  // pressure 
  float pressure = bme.readPressure() / 100.0F;
  Serial.print("Pressure: ");
  Serial.println(pressure);
  send(pressureMsg.set(pressure, 2));
#endif

  wait(10000);
#endif
}

[mrumpf]

The node sketch:

#define MY_RADIO_RF24
#define MY_NODE_ID 1
#define MY_DEBUG
#define MY_BAUD_RATE 57600

#include <SPI.h>
#include <MySensors.h>
#include <i2cdetect.h>
#include <Wire.h>

// light
#include <BH1750FVI.h>
#define CHILD_ID_LIGHT 0
BH1750FVI lightSensor(13, BH1750FVI::k_DevAddress_L, BH1750FVI::k_DevModeContHighRes);
MyMessage lightMsg(CHILD_ID_LIGHT, V_LIGHT_LEVEL);

// temperature & humidity & pressure
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#define CHILD_ID_TEMPERATURE 1
#define CHILD_ID_HUMIDITY 2
#define CHILD_ID_PRESSURE 3
#define CHILD_ID_DEWPOINT 4
Adafruit_BME280 bme;
MyMessage temperatureMsg(CHILD_ID_TEMPERATURE, V_TEMP);
MyMessage humidityMsg(CHILD_ID_HUMIDITY, V_HUM);
MyMessage pressureMsg(CHILD_ID_PRESSURE, V_PRESSURE);

void setup()
{
  Serial.begin(57600);

  i2cdetect();

  Serial.println("Init: BH1750 (Light)");
  lightSensor.begin();

  Serial.println("Init: BME280 (Temperature, Humidity, Pressure)");
  if (!bme.begin(0x76)) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    while (1);
  }
}

void presentation()
{
  sendSketchInfo("Light/Temperature/Humidity/Pressure Sensor", "1.0");

  present(CHILD_ID_LIGHT, S_LIGHT_LEVEL);
  present(CHILD_ID_TEMPERATURE, S_TEMP);
  present(CHILD_ID_HUMIDITY, S_HUM);
  present(CHILD_ID_PRESSURE, S_BARO);
}

void loop()
{  
  uint16_t lux = lightSensor.GetLightIntensity();
  Serial.print("Light: ");
  Serial.println(lux);
  send(lightMsg.set(lux));

  float humidity = bme.readHumidity();
  Serial.print("Humidity: ");
  Serial.println(humidity);
  send(humidityMsg.set(humidity, 2));

  float temperatureC = bme.readTemperature();
  Serial.print("Temperature: ");
  Serial.println(temperatureC);
  send(temperatureMsg.set(temperatureC, 2));

  float pressure = bme.readPressure() / 100.0F;
  Serial.print("Pressure: ");
  Serial.println(pressure);
  send(pressureMsg.set(pressure, 2));
  
  sleep(10000);
}

[mrumpf]

Setup (IDE: VSCode@MacOSX):

• Node: Arduino Pro Mini (Clone)
  • BME280 - Temperature & Humidity & Pressure
  • BH1750 - Light
  • nRF24L01 - Radio
  • PL-2303HX (replaced 5V by DTR) -- Connection
• Gateway: NodeMCU ESP8266 (Amica)
  • BME280 - Temperature & Humidity & Pressure
  • BH1750 - Light
  • nRF24L01 - Radio
  • ESP8266 USB -- Connection

[tekka]

@mrumpf Could be a power issue or one of the included libraries interfering with the SPI. The full debug log (from the beginning) may give additional hints - please upload them here (as .zip)

[mrumpf]

Here are the logs.
Again, after 930000 seconds the gateway did a re-init and after that the 2 devices could communicate.

Uploading a file gives me an error that only image types are supported!?

OK, please try to extract the image. It has a zip appended :)

$ unzip -l 1551391305174-archive.zip.png
Archive:  1551391305174-archive.zip.png
warning [1551391305174-archive.zip.png]:  59815 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  Length      Date    Time    Name
---------  ---------- -----   ----
    31016  02-28-2019 22:18   gateway.serial.log
        0  02-28-2019 22:19   __MACOSX/
      210  02-28-2019 22:18   __MACOSX/._gateway.serial.log
    18271  02-28-2019 22:18   node.serial.log
      210  02-28-2019 22:18   __MACOSX/._node.serial.log
---------                     -------
    49707                     5 files

[tekka]

@hek Is there a pic-only limitation with the file uploader?

@mrumpf said in Broken gateway with local sensors on MQTT ESP8266 GW:

Here are the logs.
Again, after 930000 seconds the gateway did a re-init and after that the 2 devices could communicate.
[0_1551388989628_Archive.zip](Uploading 100%)
Uploading a file gives me an error that only image types are supported!?

[mrumpf]

My current assumption is that there is a incompatibility with the BH1750 sensor library. When I set the symbols "LOCAL_SENSORS" and "LOCAL_SENSORS_BME280" it works. I will continue tomorrow evening...

[tekka]

@mrumpf From the GW log:

21822 MCO:BGN:INIT OK,TSP=0

I see the RF24 sanity check failing and this is likely due to the BH1750 initialization code meddling with the RF24 SPI. Most probably you can fix the issue by moving the sensor setup code to preHwInit(), i.e.

#define MY_DEBUG
#define MY_BAUD_RATE 57600

#define MY_RADIO_RF24
#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"
#define MY_MQTT_CLIENT_ID "mygateway1"
#define MY_WIFI_SSID "*****"
#define MY_WIFI_PASSWORD "*********************"
#define MY_HOSTNAME "mygateway1"
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 58
// #define MY_CONTROLLER_URL_ADDRESS "test.mosquitto.org"
#define MY_PORT 1883


#define LOCAL_SENSORS
#define LOCAL_SENSORS_BH1750
#define LOCAL_SENSORS_BME280

#include <ESP8266WiFi.h>
#include <SPI.h>
#include <MySensors.h>

#ifdef LOCAL_SENSORS
#include <i2cdetect.h>
#include <Wire.h>

#ifdef LOCAL_SENSORS_BH1750
#include <BH1750FVI.h>
// light
#define CHILD_ID_LIGHT 0
BH1750FVI lightSensor(13, BH1750FVI::k_DevAddress_L, BH1750FVI::k_DevModeContHighRes);
MyMessage lightMsg(CHILD_ID_LIGHT, V_LIGHT_LEVEL);
#endif

#ifdef LOCAL_SENSORS_BME280
// temperature & humidity & pressure
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#define CHILD_ID_TEMPERATURE 1
#define CHILD_ID_HUMIDITY 2
#define CHILD_ID_PRESSURE 3
#define CHILD_ID_DEWPOINT 4
Adafruit_BME280 bme;
MyMessage temperatureMsg(CHILD_ID_TEMPERATURE, V_TEMP);
MyMessage humidityMsg(CHILD_ID_HUMIDITY, V_HUM);
MyMessage pressureMsg(CHILD_ID_PRESSURE, V_PRESSURE);
#endif

#endif // LOCAL_SENSORS

void setup()
{

}

void preHwInit()
{
#ifdef LOCAL_SENSORS

#ifdef LOCAL_SENSORS_BH1750
  Wire.begin(D3, D4);
  //Wire.setClock(100000);

  Serial.println("Init: BH1750 (Light)");
  lightSensor.begin();
#endif

#ifdef LOCAL_SENSORS_BME280
  Serial.println("Init: BME280 (Temperature, Humidity, Pressure)");
  if (!bme.begin(0x77)) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    while (1);
  }
#endif

#endif
}

void presentation()
{
#ifdef LOCAL_SENSORS
  // Send the sketch version information to the gateway and Controller
  sendSketchInfo("Light/Temperature/Humidity/Pressure Gateway", "1.0");

  // Register all sensors to gateway (they will be created as child devices)
#ifdef LOCAL_SENSORS_BH1750
  present(CHILD_ID_LIGHT, S_LIGHT_LEVEL);
#endif

#ifdef LOCAL_SENSORS_BME280
  present(CHILD_ID_TEMPERATURE, S_TEMP);
  present(CHILD_ID_HUMIDITY, S_HUM);
  present(CHILD_ID_PRESSURE, S_BARO);
#endif
  
#endif
}

void loop()
{
#ifdef LOCAL_SENSORS
  i2cdetect();

#ifdef LOCAL_SENSORS_BH1750
  // light
  uint16_t lux = lightSensor.GetLightIntensity();
  Serial.print("Light: ");
  Serial.println(lux);
  send(lightMsg.set(lux));
#endif

#ifdef LOCAL_SENSORS_BME280
  // humidity
  float humidity = bme.readHumidity();
  Serial.print("Humidity: ");
  Serial.println(humidity);
  send(humidityMsg.set(humidity, 2));

  // temperature
  float temperatureC = bme.readTemperature();
  Serial.print("Temperature: ");
  Serial.println(temperatureC);
  send(temperatureMsg.set(temperatureC, 2));

  // pressure 
  float pressure = bme.readPressure() / 100.0F;
  Serial.print("Pressure: ");
  Serial.println(pressure);
  send(pressureMsg.set(pressure, 2));
#endif

  wait(10000);
#endif
}

This also explains why the GW "recovers" after 930000ms ~= 15mins, which happens to be the GW transport sanity check interval followed by a successful transport re-init:

920856 !TSF:SAN:FAIL
920859 TSM:FAIL:CNT=1
920863 TSM:FAIL:DIS
920867 TSF:TDI:TSL
...
930871 TSM:FAIL:RE-INIT
930876 TSM:INIT
930884 TSM:INIT:TSP OK 

[mrumpf]

Thanks for your analysis! I used the BH1750FVI library which is listed in the Arduino 1.8.8 IDE under Manage libraries. The MySensors wiki however references the claws library.
The name of the include file is different "BH1750FVI.h" (PeterEmbedded) to "BH1750.h" (claws).
After switching to the claws library everyting works as expected!

Maybe this could be added to the Wiki to make sure people use the right library!
Thanks for your time!

[tekka]

@mrumpf Thanks for your feedback - just for reference, did you try the BH1750FVI library and the sensor init code in preHwInit()?

[mrumpf]

@tekka No, but I can do that...

It works as well. Thanks again!

Here is the code:

// Before radio initialization
void preHwInit() 
{
  Serial.begin(57600);
  Serial.println("preHwInit()");

  Wire.begin(D3, D4);
  i2cdetect();

  Serial.println("Init: BME280 (Temperature, Humidity, Pressure)");
  if (!bme.begin(0x77)) {
    Serial.println("Could not find a valid BME280 sensor, check wiring!");
    while (1);
  }

  Serial.println("Init: BH1750 (Light)");
  lightSensor.begin();
}

setup() is empty.

[mfalkvidd]

@mrumpf said in Broken gateway with local sensors on MQTT ESP8266 GW:

Thanks for your analysis! I used the BH1750FVI library which is listed in the Arduino 1.8.8 IDE under Manage libraries. The MySensors wiki however references the claws library.
The name of the include file is different "BH1750FVI.h" (PeterEmbedded) to "BH1750.h" (claws).
After switching to the claws library everyting works as expected!

Maybe this could be added to the Wiki to make sure people use the right library!
Thanks for your time!

Great work!

The wiki says ”This example uses the external BH1750 library found here. Please install it and restart the Arduino IDE before trying to compile.” and links to the library. Do you have any siggestions on how we could make it clearer?

[mrumpf]

How about adding some warning message (highlighted) that the default Arduino IDE library requires the initialization in the preHwInit() function and not in the setup() function.
My mistake was missing the sentence "This example uses the external BH1750 library found here. Please install it and restart the Arduino IDE before trying to compile."
I was just using the library that comes with the IDE.