Packet 'freezes' WiFi Gateway ESP8266

[novicit]

I am trying to set up a ESP8266 gateway using NodeMCU module exactly as in the 'build' instructions. Am using the development branch from Feb 1. It works fine until it gets a "read and forward" packet as shown below in the serial monitor output:

0;255;3;0;9;read: 23-23-0 s=1,c=1,t=0,pt=7,l=5,sg=0:16.5
0;255;3;0;9;read: 23-23-0 s=0,c=1,t=1,pt=7,l=5,sg=0:30.0
0;255;3;0;9;read: 20-20-0 s=1,c=1,t=0,pt=7,l=5,sg=0:-2.6
0;255;3;0;9;read: 20-20-0 s=0,c=1,t=1,pt=7,l=5,sg=0:96.9
0;255;3;0;9;read: 20-20-0 s=255,c=3,t=0,pt=1,l=1,sg=0:188
0;255;3;0;9;read: 22-22-0 s=255,c=3,t=0,pt=1,l=1,sg=0:89
0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
[Gateway freezes here. When it restarts, it prints the next line]
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0
0;255;3;0;9;Client 0 connected
0;255;3;0;9;Client 1 disconnected
0;255;3;0;9;Client 1 connected
0;255;3;0;9;Client 2 disconnected
0;255;3;0;9;Client 2 connected
0;255;3;0;9;Client 2: 0;0;3;0;2;Get Version
0;255;3;0;9;read: 26-26-0 s=255,c=3,t=0,pt=1,l=1,sg=0:102
0;255;3;0;9;read: 20-20-0 s=1,c=1,t=0,pt=7,l=5,sg=0:-2.9
0;255;3;0;9;read: 20-20-0 s=255,c=3,t=0,pt=1,l=1,sg=0:188
0;255;3;0;9;read: 22-22-0 s=255,c=3,t=0,pt=1,l=1,sg=0:90
0;255;3;0;9;read: 26-26-0 s=0,c=1,t=0,pt=7,l=5,sg=0:1.2

Of 7 nodes I have, 3 have sent in these 'read and forward' packets, which 'freeze' the gateway and disconnect both MYScontroller and Domoticz from the server. My question ---- What does the 'read and forward' packet mean? (I don't believe the nodes are repeaters as I used gw.begin(NULL, 22, false).)

The packets appear randomly, from minutes to hours between them. And 90% of the time the gateway freezes requiring a manual restart. I logged for about 2 hours using Serial Monitor, MYScontroller logging, and Domoticz logs. The freeze occurred 7 times in the period.

Also, after logging the data above and removing the WiFi gateway, I put back the previous IBoard 328p ethernet gateway and everything worked fine.

What does the 'read and forward' packet mean? What originates it? Why not a problem with the original gateway. (Also, I believe I had the same problem when I tried WiFi gateway version 1.5, but I did not do all the logging before switching to the development branch.)

Not sure it matters, but below is a list of all the freezes I monitored:

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
No freeze
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 21-21-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze
0;255;3;0;9;send: 0-0-21-21 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 23-23-255 s=255,c=3,t=7,pt=0,l=0,sg=0
No freeze, long 4+ minute delay before clearing by itself
0;255;3;0;9;send: 0-0-23-23 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
No freeze
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze, Red led for tx,rx,error locked on
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=fail:0

0;255;3;0;9;read and forward: 23-23-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze
0;255;3;0;9;send: 0-0-23-23 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze
0;255;3;0;9;send: 0-0-22-22 s=255,c=3,t=8,pt=1,l=1,sg=0,st=ok:0

0;255;3;0;9;read and forward: 22-22-255 s=255,c=3,t=7,pt=0,l=0,sg=0
Freeze & I give up

Any helpful comments appreciated.

[Yveaux]

@novicit said:

gw.begin(NULL, 22, false)

Just a quick idea: Are you sure all nodes & gateway have unique ID's?

[novicit]

@Yveaux, Yes I save each sketch separately and was able to check them. Also in the serial monitor snippet above before the 'freeze', you can see the different node ID's. Also, when using the previous 328 gateway all report properly. I think I would get problems if there were duplicate node ID's also on that gateway.

To help, what do I mean when I say 'freeze'. A couple of times the gateway cleared by itself, but after many minutes. And during this period no other nodes were able to report. Other times the red LED for tx, rx, error lights and never goes off. In these cases I had to power down/reset the gateway to recover.

In a couple of other cases, I observed the red led blinking periodically as if it was getting node data, but it had broken the connection with Domoticz (and MYScontroller). In these cases all I had to do was tell Domoticz to settings-hardware-update the mysensors controller. This seemed to cause the gateway to restart, then I could connect again.

But, based on the logs, all problems trigger only after the packets I have shown.

Thanks,

[Yveaux]

@novicit Apparently when it freezes it is doing something/waiting for something, because if it enters an endless loop the infamous wdt reset will occur.
Would have to look in the code what could be the case.

[Yveaux]

@novicit The 'read and forward' is printed from here: https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/core/MyTransport.cpp#L97
Apparently it tries to route a message. I suspect something goes wrong with the routing and the gateway keeps sending messages until one gets lost or so.
Probably @hek can grasp what might go wrong here.

[novicit]

@Yveaux , thank. I was thinking, ... of the 7 nodes I have active, only three have the MYSBootloader on them. These are the same nodes which sent the 'read & forward' packets. This morning before work, I powered up one of the 3 nodes and it immediately tried to download firmware, and that caused the gateway to freeze. (I was still half asleep at the time though.) Do you know if the WiFi Gateway works well with MYSBootloader? Just a thought. Tonight I might change the boot loaders to see what happens.

[Yveaux]

@novicit said:

MYSBootloader

Never used it, but in principle it should work.
Had a quick look at the code. The MYSBootloader reads its parent node from the eeprom (https://github.com/mysensors/MySensorsBootloaderRF24/blob/master/MYSBootloader.c#L197). Maybe the eeprom contents do not match your current network setup?

[kr0815]

any solution on this ?

I saw the same with a node acting as repeater, it immediately crashes the ESP ..

[novicit]

@kr0815 Sorry for not getting back to you sooner. No, I have not come up with a solution for this. Basically I spent much of my free time for a week investigating and working to solve it - within my level of skills - but was unable to isolate the problem. I hope to get back to it when more time allows.

I have seen another forum thread (perhaps yours) that had many of the same problem characteristics as mine. In that thread @hek was involved. Between @hed and @yveaux, that is a lot of talent. The problem is having the problem be repeatable at another installation so it can be identified. In the meantime I installed my standard wired ethernet gateway and have had no problems.

[kr0815]

Hello Novici,
thanks a lot for your reply

I wrote somethong about it here:

http://forum.mysensors.org/topic/3141/big-wish-a-working-system-including-up-to-date-reference/19

so there must be something wrong in the code, if it´s working with the older regular Code, but not with the developer code

What i wonder is , that Hek doesn´t get the same problems in his test setup