[Solved] ESP8266 Wi-Fi Gateway + Domoticz: how to correctly forward sensor values

[Neo Mod]

After a bit of initial troubles (mainly due to me being a total novice at this) I was finally able to obtain a working “WiFi Gateway” sketch that connects to my network and sends the data from the only one sensor attached to the gateway itself. (a capacitive soil moisture sensor)

This is the sketch I’m using on a NODE-MCU ESP8266 board, compiled using the lates beta library of MySensors as reccomended by @mfalkvidd in another post (thank you again!):

// Enable debug prints to serial monitor
#define MY_DEBUG

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (non attached at the moment)
//#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_ESP8266

#define MY_ESP8266_SSID "****"
#define MY_ESP8266_PASSWORD "****"

// Enable UDP communication
//#define MY_USE_UDP  // If using UDP you need to set MY_CONTROLLER_IP_ADDRESS below

// Set the hostname for the WiFi Client. This is the hostname
// it will pass to the DHCP server if not static.
#define MY_ESP8266_HOSTNAME "test-sensor-gateway"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
//#define MY_IP_ADDRESS 192,168,178,87

// If using static ip you can define Gateway and Subnet address as well
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0

// The port to keep open on node server mode
#define MY_PORT 5003

// How many clients should be able to connect to this gateway (default 1)
#define MY_GATEWAY_MAX_CLIENTS 2

// Controller ip address. Enables client mode (default is "server" mode).
// Also enable this if MY_USE_UDP is used and you want sensor data sent somewhere.
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE

// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
// Led pins used if blinking feature is enabled above
//#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED

#if defined(MY_USE_UDP)
#include <WiFiUdp.h>
#endif

#include <ESP8266WiFi.h>
#include <MySensors.h>

#define CHILD_ID 0

int AirValue = 694;
int WaterValue = 344;
int intervals = (AirValue - WaterValue)/3; 
int soilMoistureValue = 0;

MyMessage msg(CHILD_ID, V_LEVEL);
unsigned long SLEEP_TIME = 30000;
int sensorPin = A0;
  
void setup()
{
  // Setup locally attached sensors
}

void presentation()
{
  // Present locally attached sensors here
  sendSketchInfo("Soil Moisture Sensor Capacitive", "1.0");
  present(CHILD_ID, S_MOISTURE);

}

void loop()
{
  // Send locally attached sensors data here
  int soilMoistureValue;
  soilMoistureValue = analogRead(sensorPin);
 

  if(soilMoistureValue > WaterValue && soilMoistureValue < (WaterValue + intervals))
    {
      send(msg.set(100));
    
    }
    else if(soilMoistureValue > (WaterValue + intervals) && soilMoistureValue < (AirValue - intervals))
      {
        send(msg.set(50));
     
      }
    else if(soilMoistureValue < AirValue && soilMoistureValue > (AirValue - intervals))
    {
       send(msg.set(0));
        
    
     }
    
sleep(SLEEP_TIME);
// note-to-self: search for possible wait() alternative or deep-sleep
}

As far as I can see from the serial monitor the data is recovered from the sensor and also transmitted (below a brief extract of the monitor itself) but I’m having a hard time understanding how to have Domoticz read this value.

0;255;3;0;9;2068414 MCO:SLP:WUP=-2
0;255;3;0;9;2068531 MCO:SLP:MS=30000,SMS=0,I1=255,M1=255,I2=255,M2=255
0;255;3;0;9;2068604 MCO:SLP:WUP=-2
0;255;3;0;9;2068721 MCO:SLP:MS=30000,SMS=0,I1=255,M1=255,I2=255,M2=255
0;255;3;0;9;2068794 MCO:SLP:WUP=-2
0;255;3;0;9;2068911 MCO:SLP:MS=30000,SMS=0,I1=255,M1=255,I2=255,M2=255
0;255;3;0;9;2068984 MCO:SLP:WUP=-2

The Gateway/Node and the Sensor appear inside the Domoticz interface, but the value stays always zero.
I tried different options, but apparently I’m missing some pieces of the puzzle.

Sorry again if the question is "silly": I've read all the tutorials I could find on the topic but unfortunately the majority of them uses a different sensor (the conductive one) and so have a lot of instructions and a "calibrating" section that's far from my actual knowledge level.

On the bright side I'm able to read data continuously from the sensor using a diffferent sketch (very simple).

P.S. I know that the sleep(SLEEP_TIME); function is not supported by ESP8266 and that I have to find a proper solution to lower power consumption.

[Neo Mod]

After some more digging I managed to get a report passed from the Gateway to Domoticz using this:

void loop()
{
  // Send locally attached sensors data here
  int soilMoistureValue;
  soilMoistureValue = analogRead(sensorPin);
  delay(5000);
  
  if (client.connect(domoticz_server,port)) {
  
    client.print("GET /json.htm?type=command&param=udevice&idx=");
    client.print(idx);
    client.print("&nvalue=0&svalue=");
    client.print(soilMoistureValue);
    client.println(" HTTP/1.1");
    client.print("Host: ");
    client.print(domoticz_server);
    client.print(":");
    client.println(port);
    client.println("User-Agent: Arduino-ethernet");
    client.println("Connection: close");
    client.println();

    client.stop();
    }

Although I'm not sure this the right way, actually I feel like I'm missing the point of using MySensors WiFi Gateway if I had to push a json to Domoticz in order to read the sensor.

[Neo Mod]

Update 3:

I have adapted some code from another sketch used for humidity, and included some power-optimizations from another sketch.

Luckly enough the whole thing is working, but still Domoticz refuses to accept the sensor. I believe it's an error on their side.

[Neo Mod]

Update 4:

Problem solved!

The sensor now reports correctly to Domoticz and ESP8266 also correctly enters and exits DeepSleep to save power.

For those interested this is my (dirty) sketch:

// Enable debug prints to serial monitor
#define MY_DEBUG

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
//#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_ESP8266

#define MY_ESP8266_SSID "iNeo"
#define MY_ESP8266_PASSWORD "****"

// Enable UDP communication
//#define MY_USE_UDP  // If using UDP you need to set MY_CONTROLLER_IP_ADDRESS below

// Set the hostname for the WiFi Client. This is the hostname
// it will pass to the DHCP server if not static.
#define MY_ESP8266_HOSTNAME "test-sensor-gateway"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
//#define MY_IP_ADDRESS 192,168,178,87

// If using static ip you can define Gateway and Subnet address as well
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0

// The port to keep open on node server mode
#define MY_PORT 5003

// How many clients should be able to connect to this gateway (default 1)
#define MY_GATEWAY_MAX_CLIENTS 2

// Controller ip address. Enables client mode (default is "server" mode).
// Also enable this if MY_USE_UDP is used and you want sensor data sent somewhere.
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE

// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
// Led pins used if blinking feature is enabled above
//#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED

#if defined(MY_USE_UDP)
#include <WiFiUdp.h>
#endif

#include <ESP8266WiFi.h>
#include <MySensors.h>

#define CHILD_ID 1

//declare reference values if you want to calibrate your sensor. I decided to output RAW data in order to have a better understanding of my soil.

//int AirValue = 694;
//int WaterValue = 344;
//int intervals = (AirValue - WaterValue)/3; 

int soilMoistureValue = 0;

//presenting the sensor this way will help domoticz associate to a soil sensor, altough not capacitative. support lacks. so the graphic will be inverse. keep in mind.

MyMessage msgMoisture(CHILD_ID, V_LEVEL);

// the only one analog pin available for sensor connected directly to the gateway.

int sensorPin = A0;

 
void setup()
{
  // Setup locally attached sensors
  
}

void presentation()
{
 
  sendSketchInfo("Soil Moisture Sensor Capacitive", "1.0");
  present(CHILD_ID, S_MOISTURE);

}

void loop()
{
 
// wake-up the sensor, pick a reading
  pinMode(sensorPin, INPUT_PULLUP);
  analogRead(sensorPin);

// just as precaution, wait a little bit to stabilize the reading. not sure is needed with a capacitive one.
  sleep(1250);

  int soilMoistureValue = analogRead(sensorPin);

// turn off the sensor, again not sure if needed but helpfull anyway to prevent power-leak
  pinMode(sensorPin, OUTPUT);
  digitalWrite(sensorPin, LOW);

//send data to domoticz or anything else capable of listening. tested only on domoticz.
// little verbose for monitoring purpose. serial.print can be commented.  

  Serial.print("Soil Value: ");
  Serial.println(soilMoistureValue);
  send(msgMoisture.set((soilMoistureValue)));

//security delay: wait to be sure that Gateway sends data to Domoticz over WiFi and Domoticz records it.
  wait(300000);

//finally go to deep sleep for 4.5h (since soil moisture level won't change drastically in a few hours this will help preserve battery)
  ESP.deepSleep(16200000);
}

[Yveaux]

@Neo-Mod well, great to hear you solved it and thanks for the extensive write up!

[Neo Mod]

Update 5:

Apparently I was a bit too optimistic: after about 12h of running this sketch I discovered that:

1. The DeepSleep mode wasn't correctly managed: in fact the ESP wen't into DeepSleep but never emerged.

This was due to my misunderstanding of a crucial step: in order for the Internal Clock to wake the ESP after entering DeepSleep mode a connection between the RST pin and the D0 pin is needed.

After doing so (thankfully the NodeMCU board has all pins exposed and accessible) the ESP.deepSleep command was actually working, sending the board into power-saving mode.

But then I've discovered the second issue:

1. Domoticz configuration, with the WiFi Gateway configured inside the "Hardware" tab, doesn't bond well with deepsleep.

I've spent countless hours reading and researching on this matter, and probably my conclusion isn't quite perfect but here is what I understand: domoticz has the somehow bad habit of regularly "pinging" the gateway.

When the gateway is always-on the WiFi is active and so the Modem hence it will respond to a ping request and Domoticz is happy.

But in DeepSleep the only thing active is the Internal Clock and so there is no response dfor Domoticz.(for instance: if you want to change the sketch and the ESP is in DeepSleep not even the RST button will work. you'll have to sever the RST-D0 connection and power-cycle the board).
Hence Domoticz starts and endless cycle of "ping --> gateway did not answer --> throw error --> start 30 sec timer --> retry" that ultimately led to the sensors being off-line.

Sadly even tough the ESP was coming out of deep-sleep every 2h and sending a measurement, if Domoticz was casually in the middle of a "30s waiting due to ping error" it won't accept the message from the gateway, keeping the sensor off-line basically even if it wasn't.

So I had to fall back on the json API to update Domoticz, creating a dummy sensor and having it updated every time the ESP exits DeepSleep. (this is important for me since I'm planning to have this sensor battery operated).

I don't know how this will impact my future plan to add radio sensors, battery operated, which report back to this gateway. If someone has any idea about that I would gladly hear them out, please!

In the meantime this is the new (and hopefully final) sketch for this sensor: there still some things that I need to do, like implementing a "previous measurement control value" and some safety check.

The previous sketch is still valid but only for always active gateway (or I believe for no more than a couple minutes of delay between each reading)

// Enable debug prints to serial monitor
#define MY_DEBUG

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
//#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_ESP8266

#define MY_ESP8266_SSID "*****"
#define MY_ESP8266_PASSWORD "****"

// Enable UDP communication
//#define MY_USE_UDP  // If using UDP you need to set MY_CONTROLLER_IP_ADDRESS below

// Set the hostname for the WiFi Client. This is the hostname
// it will pass to the DHCP server if not static.
#define MY_ESP8266_HOSTNAME "GatewayMoisture"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
//#define MY_IP_ADDRESS 192,168,1,10

// If using static ip you can define Gateway and Subnet address as well
//#define MY_IP_GATEWAY_ADDRESS 192,168,1,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0

// The port to keep open on node server mode
#define MY_PORT 5003

// How many clients should be able to connect to this gateway (default 1)
#define MY_GATEWAY_MAX_CLIENTS 2

// Controller ip address. Enables client mode (default is "server" mode).
// Also enable this if MY_USE_UDP is used and you want sensor data sent somewhere.
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE

// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
// Led pins used if blinking feature is enabled above
//#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED

#if defined(MY_USE_UDP)
#include <WiFiUdp.h>
#endif

#include <ESP8266WiFi.h>
#include <MySensors.h>

// define the analog pin where the sensor is connected. node-mcu board has only one, A0
#define sensorPin A0
//define the child_id for this sensor. it's not strictly useful here, but it helps domoticz and further developments
#define CHILD_ID 1

// define domotics server IP address and PORT
const char* domoticz_server = "192.168.0.3"; //Domoticz IP
int port = 2121; //Domoticz Port
// define virtual sensor ID given by domoticz upon dummy sensor creation.

// this section is commented since I connected my sensor directly to the gateway,
// and I'm using the gateway only for one sensors at the moment, hence I'm putting the value
// manually inside the request string.
// int idx = 1; //IDX for virtual sensor can be found in Setup -> Devices

float soilMoistureValue;

MyMessage msgMoisture(CHILD_ID, V_LEVEL);

void presentation()
{
  // send sketch info and present the sensor as moisture. not sure if this is needed anymore since the API method. 
  sendSketchInfo("Soil Moisture Capacitive", "1.0");
  present(CHILD_ID, S_MOISTURE);

}

// creates a client that can connect to to a specified internet IP address and port using the client.connect()

WiFiClient client;

void setup()
{
  // best practice, as suggested by many tutorials, to initiate the sensor after deepsleep
  int sensorPin = A0;
}

void loop()
{
  // Send locally attached sensors data here, quite useless in this case since we are using json, but still I'm keeping for further development
  soilMoistureValue = analogRead(sensorPin);
  send(msgMoisture.set(soilMoistureValue, 1));

// a little verbosity, only active on serial monitor if my_debug is true.
#ifdef MY_DEBUG
  Serial.print("Soil Value: ");
  Serial.println(soilMoistureValue);
  Serial.println("Now preparing to send data to domoticz via json...");
#endif
  
  //update virtual sensor data with json, needs a saftey check to determine if WiFi is alive. altough this is not really necessary as it's a logic fallback: thanks to mysesors wifi gateway library, if the wifi connection is not ready
  // the gateway will not get any further and won't try to update sensor data.
  
  if (client.connect(domoticz_server, port)) {
  
    client.print("GET /json.htm?type=command&param=udevice&idx=1&nvalue=");
    client.print(soilMoistureValue);
    client.println(" HTTP/1.1");
    client.println();
    client.stop();
    }

  delay(2200); //just to be sure that the value has been forwarded to domoticz server, in case the network is very busy.
  // let's put the esp into deepsleep. change the first number to the seconds you want the ESP to be in deeps sleep. so for example if you want to deep sleep for 1h then 1h is 3600 s, then you have (3600 * 1000000, WAKE_RF_DEFAULT)
  // be aware: THE DEFAULT UNIT FOR DEEPSLEEP COUNTING IS MICROSECODS AND NOT MILLISECONDS! this gave me a 2hrs headache before getting it right.
  
  ESP.deepSleep(1800 * 1000000, WAKE_RF_DEFAULT);
  delay(100); // wait for deep sleep to happen! this is really important and usefull! 
}

@Yveaux It's my pleasure to contribute to the forum even if a bit of my experience could help others and somehow give back a little of the time others have spent on my topics full of questions then I'm happy.

[Yveaux]

@Neo-Mod said in [Solved] ESP8266 Wi-Fi Gateway + Domoticz: how to correctly forward sensor values:

When the gateway is always-on the WiFi is active and so the Modem hence it will respond to a ping request and Domoticz is happy.

In mySensors architecture (likely in all wireless networks) a gateway is supposed to be always on, without sleeping. How would a sleeping gateway be able to process incoming requests, either from the network or the controller?
Save yourself a lot of trouble and keep your gateway always on!

[Neo Mod]

@Yveaux thank you for your help! yes, that makes a lot of sense actually. My ignorance on the matter kicked-in once again... sorry

What I'm missing so is how to create a sensor node which can be operated on battery and take measurements every "x" seconds/minutes/hours: shall I leave the Gateway always on and then configure differently another Node-MCU to act as a Sensor Node ?

Edit: In fact all of this was meant, for me, to be a "test run" bedore developing the whole system. I got a little off-road with the deepSleep and Battery thing, but I wanted to have a real-word-test of my sensor left for a day in the soil while sending reliable data to controller. As I'm reading even more documents and forum post, it's getting clear to me that the best thing would be "ask to the comunity" their experience with HW nodes, thus I can decide which one is best for my case.