ESP8266 gateway and sensor node not talking - noobie

[Fabien]

I have same bug : http://forum.mysensors.org/topic/2538/esp8266-and-rfm69w-loop-reboot

[hek]

Are you getting the same error in dev branch?

[Fabien]

With the last git developpement it works. except when a node start presentation, gateway reboot. And sometimes it reboots randomless.
node is a sensebender with MySensors 1.6 and gateway with 1.6 too.
Important : You have to define MY_RF69_IRQ_NUM = MY_RF26_IRQ_PIN on ESP8266

#define MY_RADIO_RFM69
#define MY_RFM69_FREQUENCY   RF69_868MHZ
#define MY_RF69_IRQ_PIN 15
#define MY_RF69_SPI_CS 2
#define MY_RF69_IRQ_NUM 15

Result is

connected with PandoraBox_1BB657, channel 1
dhcp client start...
.ip:192.168.2.165,mask:255.255.255.0,gw:192.168.2.1
.IP: 192.168.2.165
0;0;3;0;9;Init complete, id=0, parent=0, distance=0
chg_B:-120

Soft WDT reset

ctx: cont 
sp: 3ffef9a0 end: 3ffefc60 offset: 01b0

>>>stack>>>
3ffefb50:  00000001 3ffefd00 3ffeff68 00000001  
3ffefb60:  00000024 3ffefd00 3ffeff68 402022f2  
3ffefb70:  3ffe8496 00000000 3ffefd00 40202378  
3ffefb80:  3ffe8496 00000000 3ffefd00 40202563  
3ffefb90:  3ffefd1e 00000001 3ffefd00 40202a43  
3ffefba0:  3fffdc20 00000000 3ffeec28 3ffefcd8  
3ffefbb0:  3fffdc20 3ffefd1d 3ffefd00 40202e0c  
3ffefbc0:  40209c8d 00000000 3ffeec28 40203ed1  
3ffefbd0:  00000000 00000002 3ffefe58 402039d0  
3ffefbe0:  00000000 3ffefd00 3ffeff68 40202338  
3ffefbf0:  40101500 402059d6 3ffeff50 3ffefc20  
3ffefc00:  3ffefc10 00000004 3ffefcf9 40202358  
3ffefc10:  3fffdc20 00000000 00000000 3ffeec30  
3ffefc20:  3fffdc20 00000000 3ffeec28 3ffeec30  
3ffefc30:  3fffdc20 00000000 3ffeec28 40203aee  
3ffefc40:  00000000 00000000 3ffeec28 402041a0  
3ffefc50:  00000000 00000000 3ffeec40 40100114  
<<<stack<<<
�!�L�<1���0;0;3;0;9;Starting gateway (RRNGE-, 1.6.0-beta)
0;0;3;0;9;Radio init successful.
scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
pm open phy_2,type:2 0 0
cnt 
chg_B:-60

connected with PandoraBox_1BB657, channel 1
dhcp client start...
.ip:192.168.2.165,mask:255.255.255.0,gw:192.168.2.1
.IP: 192.168.2.165
0;0;3;0;9;Init complete, id=0, parent=0, distance=0
chg_B:-120
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.6
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:51
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.4

And something strange, I only recieve Humidity one time. (sensebender defaut sketch with FORCE_TRANSMIT_INTERVAL 0 )

[Fabien]

@hek you are right it works fine with

#define TIME_OUT 500

in https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/drivers/RFM69/RFM69.cpp#L47

0�~?�4�!�{�OAa�0;0;3;0;9;Starting gateway (RRNGE-, 1.6.0-beta)
0;0;3;0;9;Radio init successful.
scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
pm open phy_2,type:2 0 0
cnt 
chg_B:-60

connected with PandoraBox_1BB657, channel 1
dhcp client start...
.ip:192.168.2.165,mask:255.255.255.0,gw:192.168.2.1
.IP: 192.168.2.165
0;0;3;0;9;Init complete, id=0, parent=0, distance=0
chg_B:-120
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.8
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:54
0;0;3;0;9;read: 1-1-0 s=255,c=0,t=17,pt=0,l=10,sg=0:1.6.0-beta
0;0;3;0;9;read: 1-1-0 s=255,c=4,t=0,pt=6,l=10,sg=0:FFFFFFFFFFFFFFFF0300
0;0;3;0;9;read: 1-1-0 s=255,c=3,t=11,pt=0,l=17,sg=0:Sensebender Micro
0;0;3;0;9;read: 1-1-0 s=255,c=3,t=12,pt=0,l=3,sg=0:1.4
0;0;3;0;9;read: 1-1-0 s=2,c=0,t=7,pt=0,l=0,sg=0:
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:54
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.7
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:50
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.7
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:51
0;0;3;0;9;Client 0 connected
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:24.5
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=1,pt=2,l=2,sg=0:50

I don't try with other timeout values.

[hek]

Ok, great. We need to increase timeout then.

[Fabien]

@hek Yes and perhaps a feature to set MY_RF69_IRQ_NUM = MY_RF26_IRQ_PIN when using ESP8266. Do You want to test other timeout values ? Is it critical ?

[hek]

@Fabien said:

MY_RF26_IRQ_PIN

I cannot find this defined anywhere...

[Fabien]

sorry MY_RF69_IRQ_PIN ...

[Fabien]

Work with 51ms timeout with my config. I set 60ms to be "secure"

[Yveaux]

@hek said:

Ok, great. We need to increase timeout then.

Better add a yield() call in those blocking wait loops.
When quiclkly looking over the code of the RF69 driver, the RFM69::initialize call could in theory block for 1.5 sec...
That's a watchdog reset waiting to happen!

[hek]

Yep,

https://github.com/mysensors/Arduino/pull/287

[Yveaux]

@hek :+1:

[Mersing]

This is working great !

Tx for this !

[chrille]

I am seeing something similar with RFM69 (specifically RFM69CW) and ESP8266. I have build an ESP8266 gateway based on the GatewayESP8266 sketch and this is what I changed

#define MY_RADIO_RFM69
#define MY_GATEWAY_ESP8266
#define MY_RFM69_FREQUENCY   RF69_868MHZ
#define MY_RF69_IRQ_PIN 4
#define MY_RF69_SPI_CS 15
#define MY_RF69_IRQ_NUM 4

I build a sensor node with the DallasTemperatureSensor sketch on an Arduino Mini Pro and defined

#define MY_RADIO_RFM69
#define MY_RFM69_FREQUENCY   RF69_868MHZ

When I start up the gateway I see

0;255;3;0;9;Starting gateway (RRNGE-, 2.0.0-beta)
0;255;3;0;9;Radio init successful.
....scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 8
cnt

connected with AP, channel 1
ip:62.x.x.x,mask:255.255.255.128,gw:62.x.x.x
.IP: 62.x.x.x
0;255;3;0;9;Init complete, id=0, parent=0, distance=0

So far so good - I assume all the radio wiring is OK, since the radio init is successful

On the sensor node I see

Starting sensor (RRNNA-, 2.0.0-beta)
Radio init successful.
find parent 
send: 255-255-255-255 s=255,c=3,t=7,pt=0,l=0,sg=0,st=bc:
find parent
send: 255-255-255-255 s=255,c=3,t=7,pt=0,l=0,sg=0,st=bc:
find parent
send: 255-255-255-255 s=255,c=3,t=7,pt=0,l=0,sg=0,st=bc:
Init complete, id=255, parent=255, distance=255

As soon as the sensor node is started the ESP8266 crashes with a WDT reset

Soft WDT reset

ctx: cont
sp: 3ffef370 end: 3ffef630 offset: 01b0

>>>stack>>>
3ffef520:  00000001 3ffef6d4 3ffef964 00000000
3ffef530:  00000024 3ffef6d4 3ffef964 40202436
3ffef540:  3ffe8432 00000000 3ffef6d4 402024bc
3ffef550:  3ffe8432 00000000 3ffef6d4 402026a7
3ffef560:  3ffef6f2 00000001 3ffef6d4 40202b87
3ffef570:  3fffdae0 00000000 3ffee600 3ffef6ac
3ffef580:  3fffdae0 3ffef6f1 3ffef6d4 40202cf4
3ffef590:  3fff1210 00000000 3ffee600 402042b4
3ffef5a0:  00000000 00000002 3ffef82c 40203d20
3ffef5b0:  00000000 3ffef6d4 3ffef964 4020247c
3ffef5c0:  401057ff 00000400 00000103 3ffef5f0
3ffef5d0:  3ffef5e0 00000004 3ffef6cd 4020249c
3ffef5e0:  3ffe8920 00000000 00000000 3ffee608
3ffef5f0:  3fffdae0 00000000 3ffee600 3ffee608
3ffef600:  3fffdae0 00000000 3ffee600 40203e6c
3ffef610:  feefeffe feefeffe 3ffee600 402045b4
3ffef620:  feefeffe feefeffe 3ffee610 40100114
<<<stack<<<

The libraries are 2.0 beta as from a few days ago. I consider the power to the ESP8266 to be OK, since I use the same for en NRF24 based gateway and also for several other ESP modules.

From the thread I understand that known issues should be fixed in the current beta. Any ideas on how to move on?

[Yveaux]

@chrille Try adding the following line to your setup() function:

ESP.wdtDisable();

It should disable the (software) watchdog on ESP. See if it continues, or if it will now just hang.

[chrille]

@Yveaux Just tried that. The change removed the watchdog message and the stack trace, but the module still reboots

[chrille]

...I also tried to change TIME_OUT in rfm69.cpp to 500 as suggested by Fabien. Doesn't make any difference

[Yveaux]

@chrille said:

the module still reboots

Probably the hardware watchdog kicks in then...

[chrille]

@Yveaux I did some further tests and it may be related to the inclusion mode feature. At least disabling this makes things more stable - but I still haven't got the radio side working reliable. When this is working I will go back to the gateway and see if I can find out what triggers the crash

[chrille]

@Yveaux I think I have narrowed down the issue now to this code in RFM69.cpp

void RFM69::sendACK(const void* buffer, byte bufferSize) {
debug(PSTR("RFM69: void sendACK\n"));
  byte sender = SENDERID;
  debug(PSTR("RFM69: sendACK - entering while()\n"));
  while (!canSend()) receiveDone();
  sendFrame(sender, buffer, bufferSize, false, true);
}

(The debug lines are added by me)

Most of the time the WDT kicks in, after logging "entering while()" - occasionally it will reach sendFrame() and log a "read: " line but then the WDT kicks in when it gets to the while() loop again.
sendAck is called from sendFrame, that disables interrupts, by calling select()

With a bunch of additional debug turned on, this is what I get on the console

0;255;3;0;9;Init complete, id=0, parent=0, distance=0
chg_B:-60
chg_B:-120
0;255;3;0;9;RFM69: interruptHandler
0;255;3;0;9;RFM69: IRQ handler - select chip
0;255;3;0;9;RFM69: IRQ handler - release chip
0;255;3;0;9;RFM69: IRQ handler - read RSSI
0;255;3;0;9;RFM69: receiveDone() - payload
0;255;3;0;9;RFM69: void sendACK
chg_B:-60
0;255;3;0;9;RFM69: sendACK - entering while()
0;255;3;0;9;RFM69: sendFrame()
0;255;3;0;9;RFM69: interruptHandler
0;255;3;0;9;RFM69: IRQ handler - read RSSI
0;255;3;0;9;read: 255-255-0 s=255,c=3,t=3,pt=0,l=0,sg=0:
0;255;3;0;9;RFM69: interruptHandler
0;255;3;0;9;RFM69: IRQ handler - select chip
0;255;3;0;9;RFM69: IRQ handler - release chip
0;255;3;0;9;RFM69: IRQ handler - read RSSI
0;255;3;0;9;RFM69: receiveDone() - payload
0;255;3;0;9;RFM69: void sendACK
0;255;3;0;9;RFM69: sendACK - entering while()

Soft WDT reset

ctx: cont
sp: 3ffefa70 end: 3ffefd30 offset: 01b0

>>>stack>>>
3ffefc20:  00000001 3ffefdd4 3ffeffdc 000000ff
3ffefc30:  00000024 3ffefdd4 3ffeffdc 402022da
3ffefc40:  3ffe8496 00000000 3ffefdd4 40202360
3ffefc50:  00000000 00000000 3ffefdd4 4020254b
3ffefc60:  3ffefdf2 00000000 3ffefdd4 402037f9
3ffefc70:  3fffdc20 00000000 3ffefdd4 3ffefdac
3ffefc80:  3fffdc20 3ffefdf1 3ffefdd4 4020385c
3ffefc90:  00000003 00000000 3ffeecf8 40204035
3ffefca0:  00000003 00000000 00000000 00000000
3ffefcb0:  3ffefd58 3ffefdd4 3ffeffdc 3ffeed00
3ffefcc0:  3ffefe00 00000000 3ffefe98 40203aec
3ffefcd0:  00000003 00000000 000000ff 00000000
3ffefce0:  000000ff 00000002 00000000 00000000
3ffefcf0:  3fffdc20 00000000 3ffeecf8 3ffeed00
3ffefd00:  3fffdc20 00000000 3ffeecf8 40203c06
3ffefd10:  00000000 00000000 3ffeecf8 4020432c
3ffefd20:  00000000 00000000 3ffeed10 40100114
<<<stack<<<

Any ideas on how to fix this, would be greatly appreciated!