Domoticz data timeout

[gbuico]

Hi. Let me first describe the scenario.
Raspberry Pi3 as controller using Domoticz and a NodeMCU v3 Lolin as gateway with sensors on board.
No radio components on NodeMCU... just the built in WiFi, so NodeMCU and Raspberry are connected through "Ethernet".
Since this configuration is part of a security system I'm currently testing what happens in each and every case of failure.
Originally I had no "data timeout" defined for NodeMCU hardware in Domoticz.
In this scenario, simulating a power failure on NodeMCU side, the restart of the gateway was not going after the point in which it asks Domoticz to activate the link... because Domoticz was thinking that the previous session was still active (very bad).
In this situation, a restart of Domoticz was solving the issue... but of course not something you can imagine in an operational project.

Second scenario, same as above, but data timeout set to 1 minute in Domoticz.
With this setting, in the same condition of gateway power failure, Domoticz restarts the connection and everything would be fine... BUT since the Sketch on NodeMCU is set to send data only if one of its sensors changes status.. this will happen seldomly and Domoticz does a restart of the gateway every minute
Still... from the console of the node, I see that a PING is sent with a frequency lower than a minute... but Domoticz seems to ignore it.

Am I missing something? How are managing the "keepalive" between the gateway and the controller???

Here is a copy of what I see on the console:
0;255;3;0;9;Client 0 connected
0;255;3;0;9;Client 0: 0;0;3;0;2;
0;255;3;0;9;Client 0: 0;0;3;0;2;Get Version
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0: 0;0;3;0;18;PING
0;255;3;0;9;Client 0 disconnected
0;255;3;0;9;Client 0 connected
0;255;3;0;9;Client 0: 0;0;3;0;2;
0;255;3;0;9;Client 0: 0;0;3;0;2;Get Version
0;255;3;0;9;Client 0: 0;0;3;0;18;PING

and the corresponding log in Domoticz:

2018-01-02 19:15:00.850 Error: MySensors1 hardware (7) nothing received for more than 1 Minute!....
2018-01-02 19:15:01.851 Error: Restarting: MySensors1
2018-01-02 19:15:02.822 MySensors: TCP/IP Worker stopped...
2018-01-02 19:15:03.824 MySensors: trying to connect to: 192.168.1.7:5003
2018-01-02 19:15:04.824 MySensors: connected to: 192.168.1.7:5003
2018-01-02 19:15:04.824 MySensors: Gateway Ready...
2018-01-02 19:15:05.825 MySensors: Gateway Version: 2.1.1
2018-01-02 19:15:05.826 MySensors: Gateway Version: 2.1.1
2018-01-02 19:16:00.064 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-02 19:16:00.081 dzVents: Debug: Event trigger type: time
2018-01-02 19:16:30.838 Error: MySensors1 hardware (7) nothing received for more than 1 Minute!....
2018-01-02 19:16:31.839 Error: Restarting: MySensors1
2018-01-02 19:16:32.834 MySensors: TCP/IP Worker stopped...
2018-01-02 19:16:33.836 MySensors: trying to connect to: 192.168.1.7:5003
2018-01-02 19:16:34.836 MySensors: connected to: 192.168.1.7:5003
2018-01-02 19:16:34.836 MySensors: Gateway Ready...
2018-01-02 19:16:35.837 MySensors: Gateway Version: 2.1.1
2018-01-02 19:16:35.837 MySensors: Gateway Version: 2.1.1

[mfalkvidd]

@gbuico this is unfortunately how Domoticz is designed to work. See https://forum.mysensors.org/post/77400 for an earlier discussion.

A workaround is to set the timeout larger than the shortest update interval of your nodes. But if you do that, remember that Domoticz considers an update with the same value as before (for example a pir that says it has still not detected movement) as no update. You need something that changes.

Maybe add a "dummy" pir as a local sensor on the gateway, and toggle the state every 10 seconds?

[gbuico]

@mfalkvidd , to tell you the truth I was fearing that the answer would have been this one.
I should bettere say "these ones" because in the post you mentioned I also found confirmation to a doubt I started to have since some time.... should I continue with Domoticz or not
As I said in my introduction I have close to 50 years of IT experience and I loved that project because you get started very quickly... but... if you are someone who likes "architectures"... it's a death.
Plus, you may see yourself, I posted several times in the forum and I almost never received even a comment.
I also saw in the other topic that you were close to the idea to jump to another controller... did you do it??? I gave a quick look to OpenHAB and found it far away from my mentality.. I've not tried anything else and would love to hear other experiences...

Back to the specific issue, I was thinking to force a status update before the data timeout expires.. but I see it's not working if the status is unchanged.
Even if it won't change anything, let me comment that to an "architecture fanatic"as I am, this behavior is just unacceptable. You may decide not to process a device event if there is no change, but you cannot say that you have not received anything from the corresponding hardware because this is just a lie.
Whatever... even if I'm getting tired to survive with patches for the deficiencies of Domoticz, the quick and dirty solution right now seems to implement a fake sensor which changes its status with a proper frequency.
I'm just having troubles to use the tmr.now() in my sketch because the compiler tells me "tmr" is not defined... will see how to solve.
In the meanwhile, advices from you and whomever has experience with other controllers with a better design and a better integration with MySensors it's highly appreciated.

P.S. Another reason why I kept going with Domoticz is that I'm using an RFLink 433 radio interface (nano shop) and I couldn't find another controller supporting this piece of hardware

[sundberg84]

@gbuico - it would be best to have this integrated into Domoticz. In the same time if Domoticz went down you had the same issue? Wouldnt it be the best to create a external node with an UPS that pings Domoticz and the gateway and alerts with a test/sms if it went down?

[gbuico]

@sundberg84 , in the architecture I have in mind for real operations the Raspberry with the main instance of Domoticz will be under UPS, so in my tests I'm assuming that it will be always on.
Probably I may use a small databank between the plug and NodeMCU, so this should protect me against a normal power loss.
Still... the protection with the databank is certainly weaker than a real UPS and this is the reason why I'm doing.
Anyway... wherever you place a monitoring system you have only two solutions:

1. data timeout disabled - with this setting, when the nodeMCU restarts then remains waiting for long time... may be hours.. because Domoticz doesn't answer to the initialization protocol.
2. data timeout enabled - in this case Domoticz would surely restart the communication after the defined timeout... but would also disconnect and reconnect with the same frequency if there is no change in the sensor status.

The second solution, at least, does not interrupts the overall functionality... but let me say it... this is a severe design error in Domoticz if this is the way it is expected to work.
In the meanwhile... I went through all the presentation and the overall documentation of "Home Assistant" and, unless someone will dis encourage me from doing it, tonight I'll give it a try

[mfalkvidd]

@gbuico for Domoticz, I have the following cron job to restart in case Domoticz dies:

/3  *  *   *   *     /usr/sbin/service domoticz status || /usr/sbin/service domoticz restart

[gbuico]

Never heard of cron jobs before... you never stop learning
Still, from what I understand this restart is triggered when Domoticz dies... not when a node has lost contact.
Btw.. The Home Assistant installation is in progress on another SD card.
If you care, the reason why it attracted me is because it seems to have full support for Piface Digital 2, RFLink and MySensors. Second, the documentation seems really well done. A small minus, 32G SD card is recommended... which is big... still I'm currently installing on a 8G card I had available in the house...
Will let you know

[kimot]

Sorry but I think using pure NodeMCU with Domoticz ( without other MySensors nodes connected ) like MySensors gateway is "masochistic".
I suggest something like ESPeasy for this situation. If it supports your sensors types, you get more communication "freedom" then.

[gbuico]

@mfalkvidd , after a full day of work, trying to implement my configuration on Home Assistant, OpenHAB and Jeedom, I'm back here with a low tail and ready to go back to Domoticz.
Without going in details, each of the above failed implementing either the RFLink or Piface Hardware. None of them seemed to provide a "logical" user interface like Domoticz.
So... here I am... with one day totally lost and trying to recover.
What I want to do now is to define a fake sensor in NodeMCU that will send a change within the data timeout.
May I kindly ask you if you have a working sketch using the tmr function as I can't manage to compile a proper one (tmr not defined)
Thanks in advance

[mfalkvidd]

@gbuico I have never needed something like that, so I don't have any code.

But I took some code from the motion sensor example and came up with this:

#define CHILD_ID 1   // Id of the sensor child

// Initialize motion message
MyMessage msg(CHILD_ID, V_TRIPPED);

void presentation()
{
	// Register all sensors to gw (they will be created as child devices)
	present(CHILD_ID, S_MOTION);
}

bool tripped = false;
void loop()
{
 tripped = !tripped;

	send(msg.set(tripped?"1":"0"));  // Send tripped value to gw
    wait(20000);
}

Sorry for the formatting, currently using mobile.

[kimot]

tmr.now() is NodeMCU LUA function, not MySensors or Arduino.

NodeMCU

[gbuico]

@kimot , sure, I know that tmr.now() is a NodeMCU function and I'm trying to use it in the "void loop()" sketch of my NodeMCU to send a "changed status" at regular interval.
Unfortunately I'm still getting compile errors as if the function has to be declared or requires an "include".
Back to your post above where you mention ESPEasy... I don't know this alternative, but from what I'm reading is just a sketch that you produce with a specific tool instead of using Arduino IDE software, so I cannot see how this can be better or easier compared to Mysensors since the "dumb side" seems to be in Domoticz.
Still since you know both, I'm sure that you have a good reason... so, may I ask you to spend a couple of words more to explain why you think this can make things simpler? Thanks a lot

[kimot]

@gbuico
tmr.now() is LUA function
NodeMCU LUA is firmware, which you load to your NodeMCU ( or other ESP2866 ) and then write your programs in LUA scripts, which stores in ESP2866.

MySensors is writen In Ardfuino language ( C++ ), so Arduino IDE compile your program and load it to ESP2866. And there is not tmr.now(). Use millis() instead off.

ESPeasy is firemware to load to ESP2866.
It runs on ESP2866 and you can create sensors and write simple "program" when connected to ESP webpage.
There are any timeouts, you can send message one per day or month, no problem.
You do not need compile ESPeasy itself. Use compiled file and download it to your ESP2866.

First you create your sensors in Domoticz as "Dummy" devices and remembers their "idx".
Then connect with browser to your sensor with ESPeasy running and through its webpage you can define different sensors ( sensor type, GPIO where connected, its domoticz idx , sending interval, etc )
For example, for DS18b20 you select pin for 1-wire bus, then you can see all sensors on this bus and select which you need. Temperature is automatically send to Domoticz.
Different sensor - no problem, select for example pins for i2c bus, discover this bus and then select sensor.
Display - select i2c address and it is done.
But what I really love on ESPeasy is its communication model.
If you cannot send some info through ESPeasy autmatically, you can write json command and send to Domotic whateverelse.
Need send data from Domoticz to your ESPeasy - no problem.
You can send commands from Domoticz with parameters and receive it in ESPeasy.
You can transfer everything this way.
No compilation, only write in your browser and hit "Apply" button.

I like MySensors very much, but for wi-fi connected single sensors exists easier ways.

But my dream is feature for MySensor ethernet gateway, when I can send to gateway something like this:
http://gatewayip/control?cmd=23;5;24.5
which means = to_node, sensor_id, value.

Or opposite way, send from sensor to gateway command with domoticz sensor idx, type of domoticz value (svalue or nvalue ) and value itself and gateway will send to domoticz appropriate json command.

[gbuico]

@kimot , I have to confess that I'm still a bit lost when I have to understand which language I'm using in each environment, but this is because I keep jumping from something to something else. Your explanation was very clear and allowed me to solve the "timing problem" if a few minutes.
At the same time... you gave me enough reason to test ESPeasy... and to say it all I have already installed another NodeMCU on the breadboard and starting right now to play
Thanks again

[kimot]

@gbuico
Ok.
Only do not forget, that ESPeasy webpage is mainly for setting, so it is not refreshed automatically like Domoticz. You must manually refresh it when check, if sensor value changed, for example.
video

[gbuico]

My tests with ESPeasy stopped before beginning because I realized that I need a USB to TTL converter that I don't have a t moment... Delayed...

This gave me time to implement the change in NodeMCU sketch which addresses and solves the original problem of this topic.
I'm attaching here the second part of the modified sketch omitting the WiFi part.
In short, I defined a fake "switch" sensor in NodeMCU and I'm sending its modified status to Domoticz every 45 seconds (data timeout in Domoticz is 1 minute).
Important to underline that the new "fake sensor" can remain in the list of "not used" devices in Domoticz.
Here is the code:

// Start of the sensor specific code

#include <SPI.h>
#include <MySensors.h>
#include <pins_arduino.h>
#include <Bounce2.h>
#define CHILD_ID0 0
#define CHILD_ID1 1
#define CHILD_ID2 2
Bounce debouncer1 = Bounce();
Bounce debouncer2 = Bounce(); 
int oldValue1=-1;
int oldValue2=-1;
int tempo=millis();
int ultimo=0;
MyMessage msg1(CHILD_ID0, V_TRIPPED);
MyMessage msg2(CHILD_ID1, V_TRIPPED);
MyMessage msg3(CHILD_ID2, V_TRIPPED);
void setup()
{
  // Setup magnetic contact on D0 with pull-up
  pinMode(D0,INPUT_PULLUP);
  // After setting up reed switch, setup debouncer
  debouncer1.attach(D0);
  debouncer1.interval(5);
  // Setup magnetic contact on D1 with pull-up
  pinMode(D1,INPUT_PULLUP);
  // After setting up reed switch, setup debouncer
  debouncer2.attach(D1);
  debouncer2.interval(5);
}

void presentation()
{
	// Present locally attached sensors here
	present(CHILD_ID0, S_DOOR);
  present(CHILD_ID1, S_DOOR);
  present(CHILD_ID2, S_DOOR);
}


//  Check if digital input has changed and send in new value
void loop() 
{
  debouncer1.update();
  debouncer2.update();
  // Get the update value
  int value1 = debouncer1.read();
  int value2 = debouncer2.read();
  if (millis() > tempo + 45000)
  {
     Serial.print (millis());
     Serial.print (ultimo);
     tempo = millis();
     send(msg3.set(ultimo));
     ultimo = ultimo + 1;
     if (ultimo > 1)
     {
        ultimo = 0; 
     }
  }
  
  if (value1 != oldValue1)
	{
     // Send in the new value
     send(msg1.set(value1==HIGH ? 1 : 0));
     oldValue1 = value1;
	}
  if (value2 != oldValue2)
  {
     // Send in the new value
     send(msg2.set(value2==HIGH ? 1 : 0));
     oldValue2 = value2;
  }
}```

[gbuico]

Domoticz log changed from this:

2018-01-05 14:19:04.819 MySensors: Gateway Ready...
2018-01-05 14:19:05.820 MySensors: Gateway Version: 2.1.1
2018-01-05 14:19:05.820 MySensors: Gateway Version: 2.1.1
2018-01-05 14:20:00.270 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:20:00.293 dzVents: Debug: Event trigger type: time
2018-01-05 14:20:30.855 Error: MySensors1 hardware (7) nothing received for more than 1 Minute!....
2018-01-05 14:20:31.856 Error: Restarting: MySensors1
2018-01-05 14:20:32.829 MySensors: TCP/IP Worker stopped...
2018-01-05 14:20:33.831 MySensors: trying to connect to: 192.168.1.7:5003
2018-01-05 14:20:34.831 MySensors: connected to: 192.168.1.7:5003
2018-01-05 14:20:34.831 MySensors: Gateway Ready...
2018-01-05 14:20:35.832 MySensors: Gateway Version: 2.1.1
2018-01-05 14:20:35.832 MySensors: Gateway Version: 2.1.1
2018-01-05 14:21:00.379 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:21:00.402 dzVents: Debug: Event trigger type: time
2018-01-05 14:22:00.486 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:22:00.510 dzVents: Debug: Event trigger type: time
2018-01-05 14:22:00.846 Error: MySensors1 hardware (7) nothing received for more than 1 Minute!....
2018-01-05 14:22:01.847 Error: Restarting: MySensors1
2018-01-05 14:22:02.841 MySensors: TCP/IP Worker stopped...
2018-01-05 14:22:03.842 MySensors: trying to connect to: 192.168.1.7:5003
2018-01-05 14:22:04.843 MySensors: connected to: 192.168.1.7:5003

To this:

2018-01-05 14:55:00.413 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:55:00.436 dzVents: Debug: Event trigger type: time
2018-01-05 14:55:35.108 (MySensors1) Light/Switch (Security Sensor)
2018-01-05 14:56:00.520 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:56:00.543 dzVents: Debug: Event trigger type: time
2018-01-05 14:56:20.113 (MySensors1) Light/Switch (Security Sensor)
2018-01-05 14:57:00.127 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:57:00.151 dzVents: Debug: Event trigger type: time
2018-01-05 14:57:05.120 (MySensors1) Light/Switch (Security Sensor)
2018-01-05 14:57:50.124 (MySensors1) Light/Switch (Security Sensor)
2018-01-05 14:58:00.235 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:58:00.259 dzVents: Debug: Event trigger type: time
2018-01-05 14:58:35.129 (MySensors1) Light/Switch (Security Sensor)
2018-01-05 14:59:00.344 dzVents: Debug: Dumping domoticz data to /home/pi/domoticz/scripts/dzVents//domoticzData.lua
2018-01-05 14:59:00.367 dzVents: Debug: Event trigger type: time
2018-01-05 14:59:20.134 (MySensors1) Light/Switch (Security Sensor)

[kimot]

@gbuico
"and to say it all I have already installed another NodeMCU on the breadboard and starting right now to play"

"My tests with ESPeasy stopped before beginning because I realized that I need a USB to TTL converter that I don't have a t moment... Delayed..."

NodeMCU is with USB to TTL converter onboard.
Or how you programme it now?

[pjr]

I think with domoticz its might be enough to call sendHeartbeat() from your gateway every 30 seconds.
It should update the m_LastHeartbeatReceive of the HW and thats what mainworker uses when it defines if the HW should be "rebooted".

Here is a example(NOT TESTED):

#define HEARTBEAT_SEND_FREQUENCY 30000
unsigned long lastHeartBeatSend;
....
void setup()
{
  lastHeartBeatSend  = 0;
}
...
void loop()
{
  unsigned long now = millis();
  if(now - lastHeartBeatSend > HEARTBEAT_SEND_FREQUENCY)
  {
    sendHeartbeat();
    lastHeartBeatSend = now;
  }
}

See:
https://github.com/domoticz/domoticz/blob/8bb0420c3ba4a10bbacd6f2e2f9b40b06ff0c7ac/main/mainworker.cpp#L12864
https://github.com/domoticz/domoticz/blob/8bb0420c3ba4a10bbacd6f2e2f9b40b06ff0c7ac/hardware/MySensorsBase.cpp#L487
https://www.mysensors.org/download/sensor_api_20

[gbuico]

@kimot , right now I'm using the Arduino IDE application which allows me to upload to NodeMCU just using the USB cable.
Reading the ESPeasy documentation I understand that this type of connection is not an option and you must have a USB to TTL interface.
I'd be very happy to hear that I don't need it... otherwise I'm going to get one on monday...

[gbuico]

@pjr , what you are suggesting is exactly what I wanted at the beginning..... but... my understanding from the first answer I got and for this https://www.domoticz.com/forum/viewtopic.php?f=42&t=9775&sid=1fea4af7b12f84f656ca4aa825eb6115
is that the sendHeartbeat() function is basically accepted but disregarded in Domoticz, meaning, would not create problems but not fixing the issues.
Of course this is my understanding and would be glad to have additional opinions/tests

[pjr]

@gbuico That thread is pretty old.

The current(and most likely working) functionality is implemented here:
https://github.com/domoticz/domoticz/commit/2d4e690c82b166ff8597a7deb44be17f3ad2560f#diff-a9f78798ba1bded6b56b3909b2a42848

[gbuico]

@pjr, thanks for your update. I went to check the solution proposed in the link you attached... and let me confess... got the concept, but it's complexity is currently not something at my reach.
Meaning.. as my solution is very small and working, for the time been I'll stick to it and spend my time for further function implementation.
My only worry... your example was just a few lines... understandable... how did they end up with 1700 lines of code or this?
Whatever.. I'l go back to it when I have some more time as I prefer an official implementation to a patch like mine...

[kimot]

@gbuico
USB to TTL serial is on board on NodeMCU and Wemos.

Download
ESPEasy mega dev.13

Upload to your NodeMCU using "FlashESP8266.exe" from .zip
Upload to NodeMCU

Use "....._normal_4096.bin" file

Power off - on !!! your NodeMCU ( reconnect USB cable )

Make basic setup:
With smart phone or notebook connect to your NodeMCU, which works like wifi acces point now ( name like "ESP_Easy_0" )
If password required - "configesp"
Type in browser address 192.168.4.1
wifi setup
Set your home wifi access info in config page.
You will see actual IP your NodeMCU in your wifi network.
Connect to this IP and make basic setup.
( you can set Fixed IP, IP of your Domoticz, etc )
ESPEasy setup