[SOLVED] latest git-snapshot causes freezes

cimba007

Only thing I can say so far is that this method-call causes the freez:

Hardware setup:

node = 1mhz arduino pro mini (internal osciallator) with jboard and nrf24
gateway = esp8266

the node is connected to an i2c adxl345 acceleration sensor

 send(_message,true);

prepare rssi<\r><\n>
12075 TSF:MSG:SEND,14-14-0-0,s=1,c=1,t=28,pt=0,l=11,sg=0,ft=0,st=OK:rssi: 8|0|2<\n>
Update<\r><\n>
12173 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0<\n>
 | hwretry: 0<\r><\n>
12255 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0<\n>
12320 TSF:MSG:ACK<\n>
prepare to sleep<\r><\n>
12369 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255<\n>
12419 MCO:SLP:TPD<\n>
12435 MCO:SLP:WUP=1<\n>
12

Gateway

0;255;3;0;9;TSF:MSG:SEND,0-0-14-14,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0<\n>
0;255;3;0;9;Sending message on topic: mygateway-out/14/0/1/0/16<\n>
0;255;3;0;9;TSF:MSG:READ,14-14-0,s=0,c=1,t=16,pt=2,l=2,sg=0:0<\n>
0;255;3;0;9;TSF:MSG:ACK REQ<\n>
0;255;3;0;9;TSF:MSG:SEND,0-0-14-14,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0<\n>
0;255;3;0;9;Sending message on topic: mygateway-out/14/0/1/0/16<\n>
0;255;3;0;9;TSF:MSG:READ,14-14-0,s=0,c=1,t=16,pt=2,l=2,sg=0:0<\n>
0;255;3;0;9;TSF:MSG:ACK REQ<\n>
0;255;3;0;9;TSF:MSG:SEND,0-0-14-14,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0<\n>
0;255;3;0;9;Sending message on topic: mygateway-out/14/0/1/0/16<\n>
0;255;3;0;9;Sending message on topic: mygateway-out/0/0/1/0/15<\n>
0;255;3;0;9;Sending message on topic: mygateway-out/0/0/1/0/15<\n>

whole project:

node-debug-log:0_1475363295558_adxl345_vibrationtest_fabo_rf.rar

You can see that the output stops while flushing an millis output ...

I am using my own milliseconds-wrapper but this should make no difference. I can say that the freezes occurs within 1-5minutes with the setup from the attached file.

I speed up the process of the loop to better reproduce the error. I noticed that the hangup occurs somewhere during the end of the loop. In addition to that he freeze does not occur if I send no messages to the gateway/controller at all but do only internal processing.

Tomorrow I might clean up my sketch and try a minimal setup and reproduce the hangup.

One possibility I am looking into is a interference of:

TWBR = 0;

At the same time I got a random gateway crash:

0;255;3;0;9;Sending message on topic: mygateway-out/14/0/1/0/16<\n>
Fatal exception 28(LoadProhibitedCause):<\n>
epc1=0x402026ac, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000003, depc=0x00000000<\n>
<\r><\n>
Exception (28):<\r><\n>
epc1=0x402026ac epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000003 depc=0x00000000<\r><\n>
<\r><\n>
ctx: cont <\r><\n>
sp: 3ffef980 end: 3ffefbe0 offset: 01a0<\r><\n>
<\r><\n>
>>>stack>>><\r><\n>
3ffefb20:  00000000 3ffeff7c 3ffefdc4 4020589f  <\r><\n>
3ffefb30:  001e8480 00085f6b 3ffefdc4 40202f98  <\r><\n>
3ffefb40:  00000001 00000011 00000000 3ffeebac  <\r><\n>
3ffefb50:  3fffdad0 00085f6b 3ffefc8c 40203282  <\r><\n>
3ffefb60:  3fffdad0 00085f6b 3ffefc8c 40203bb8  <\r><\n>
3ffefb70:  3fffdad0 00085f6b 0000ea60 402031bd  <\r><\n>
3ffefb80:  3fffdad0 00085f6b 0000ea60 4020463d  <\r><\n>
3ffefb90:  3fffdad0 00000000 0000ea60 4020465e  <\r><\n>
3ffefba0:  00000001 00000000 0000ea60 40204683  <\r><\n>
3ffefbb0:  00000000 00000000 3ffeeb80 40204b49  <\r><\n>
3ffefbc0:  3fffdad0 00000000 3ffeeb80 40204d83  <\r><\n>
3ffefbd0:  feefeffe feefeffe 3ffeebc0 40100114  <\r><\n>
<<<stack<<<<\r><\n>
<\r><\n>
 ets Jan  8 2013,rst cause:2, boot mode:(1,6)<\r><\n>
<\r><\n>
<\r><\n>
 ets Jan  8 2013,rst cause:4, boot mode:(1,6)<\r><\n>
<\r><\n>
wdt reset<\r><\n>

tekka

@cimba007 You are living on the bleeding edge :) Please provide additional information such as HW setup and debug logs of both, GW and node. Thanks

tekka

@cimba007 Please post new messages instead of editing previous post - this makes it easier in terms of understanding and following the chronology of the thread.

cimba007

Okay .. I am currently investigating:

TWBR = 0;

Might be somehow related as removing it causes no hangups so far...maybe I am seeing things with the newest snapshots :D Strangely the code was working fine until I added the mysensor-part ...

wopps .. again

264077 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0<\n>
264142 TSF:MSG:ACK<\n>
prepare to sleep<\r><\n>
264208 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255<\n>
264273 MCO:SLP:TPD<\n>
264290 MCO:SLP:WUP=1<\n>
.30

So i2c-speed is probably unrelated .. as even after commeting the TWBR = 0; out the hangup repeated.

Next I will try to remove the acceleration-sensor readings ...

cimba007

And again ...

145440 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255<\n>
145489 MCO:SLP:TPD<\n>
145522 MCO:SLP:WUP=1<\n>
.152002<\r><\n>
Update<\r><\n>
145522 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:1<\n>
 | hwretry: 1<\r><\n>
145522 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:1<\n>
145522 TSF:MSG:ACK<\n>

But this time at a differenct part ...

  fabo3axis.readIntStatus();
  sleep(digitalPinToInterrupt(3),RISING,16);

Removing this as well ..

cimba007

and .. again ..

.79366<\r><\n>
Update<\r><\n>
76038 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:1<\n>
 | hwretry: 0<\r><\n>
76038 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:1<\n>
76038 TSF:MSG:ACK<\n>

My loop currently looks like this:

void loop() { 

  motionSensor.set(1);  motionSensor.waitsend(50);
  /*
  uint8_t s = fabo3axis.readIntStatus();
  if(ADXL345_INT_ACTIVITY & s){
    Serial.print("activity ");
    Serial.println(millis());
    motionSensor.set(1);  motionSensor.waitsend(50);
  }
  else
  {
    Serial.println(s,HEX);
    motionSensor.set(0);  motionSensor.waitsend(50);
  }
  */
  Serial.println("prepare to sleep");
  Serial.flush();
  
  // Pending Int-Stati might prevent interrupt!
  //fabo3axis.readIntStatus();
  sleep(digitalPinToInterrupt(3),RISING,16);
  
  // helper.h
  //extern unsigned long millis_offset;
  millis_offset += 16;
  Serial.print(".");
  Serial.println(mymillis());
}

All referencec to the acceleration sensor removed .. the interrupt should not trigger as the interrupt-register is not "cleard" by issuing the readIntStatus ..

cimba007

I forgot to mention that the node is battery powered @ 2x AA-Alkaline-Battery @ ~2,5volt .. no LDO involved .. and an NRF24+LNA+PA version of the NRF24 ..

I got like 4-5 boards with this exact same setup and they run totally fine with the latest stable release ..

.. investigating the root cause might be difficulty as my code is a little bit customized with a little wrapper .. will try out a fresh sketch tomorrow ..

cimba007

Okay ... I switched back between snapshot from yesterday and stable 2.0.0 and all I can say is that the stable is perfectly fine and the snapshot freezes.

prepare to sleep
17416 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255
17481 MCO:SLP:TPD
17498 MCO:SLP:WUP=1
.18

Nothing on the code has been changed. Only difference is that I had to modifie latest stable to include a few delays for my slow 1mhz node ..

PS: I currently don't have the time to debug this issue further .. last time it took me some days to figure out the delay-issue of slow nodes ...

tekka

@cimba007 Could you upload a minimal sketch (ideally without external dependencies) to reproduce this?

cimba007

// http://www.earth.org.uk/OpenTRV/Arduino/bootloader/ATmega328P-1MHz/README.txt
#define MY_NODE_ID 14

#define MY_BAUD_RATE 9600 // DONT GO HIGHER AS 57600 !!! 115200 might cause DEADLOCKS!!!

// Enable debug prints to serial monitor
#define MY_DEBUG
#define MY_DEBUG_LIGHT

#define MY_RF24_PA_LEVEL RF24_PA_HIGH

// Enable and select radio type attached
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#include <SPI.h>
#include <MySensors.h>
//#include "helper.h"

//MySensorChild motionSensor = MySensorChild(0, S_MOTION, V_TRIPPED, "");

void before()
{
  //wdt_enable(WDTO_8S);
}

void presentation()
{
  // Send the sketch version information to the gateway and Controller
  char datetime[30]; // 22 + 1 should ben enough
  snprintf(datetime, 30, "%s | %s", __DATE__, __TIME__);
  #ifdef MY_DEBUG_LIGHT
  Serial.print(F("SketchInfo: "));
  Serial.println(datetime);
  #endif

  sendSketchInfo("adxl345", datetime);
}

#include <Wire.h>
#include <FaBo3Axis_ADXL345.h>

FaBo3Axis fabo3axis;

//void ADXL_ISR()
//{
//
//}

void setup()
{
  Serial.begin(9600); // シリアルの開始デバック用
  
  Serial.println("Checking I2C device...");
  
  //if(fabo3axis.searchDevice()){
  //  Serial.println("I am ADXL345");
  //}
  Serial.println("Init...");
  
  fabo3axis.configuration();
  fabo3axis.powerOn();
  //fabo3axis.enableTap();
  fabo3axis.enableActivity();
  //attachInterrupt(digitalPinToInterrupt(3), ADXL_ISR, RISING);
  fabo3axis.readIntStatus();
  
  // http://www.avrfreaks.net/forum/wire-library-increasing-i2c-speed
  //TWBR = ((F_CPU / 400000) - 16) / 2;
  TWBR = 0;
  Serial.print("i2c @ "); Serial.print(TWBR); Serial.println(); 
}


void loop() { 
  //wdt_reset();
  MyMessage msg(0, V_TRIPPED);
  //motionSensor.set(1);  motionSensor.waitsend(50);
  //send(msg,true);
  
  //uint8_t s = fabo3axis.readIntStatus();
  uint8_t s = 1;
  if(false){
    Serial.print("activity ");
    //Serial.println(millis());
    msg.set(1);
    send(msg,true);
    //motionSensor.set(1);  motionSensor.waitsend(50,60000);
  }
  else
  {
    Serial.print(s,HEX);
    msg.set(0);
    send(msg,true);
    //motionSensor.set(0);  motionSensor.waitsend(50,60000);
  }
  
  //Serial.println("prepare to sleep");
  Serial.flush();
  
  // Pending Int-Stati might prevent interrupt!
  fabo3axis.readIntStatus();
  sleep(digitalPinToInterrupt(3),RISING,16);

  //millis_offset += 16;
  Serial.print(" | ");
  Serial.println(hwMillis());
}

bool my_ack_received;
void receive(const MyMessage &message)
{
  if (message.isAck())
    my_ack_received = true;
}

I suspect the sleep/interrupt combination.

Could not remove the accelerometer cause it is the source of the interrupts

cimba007

no sleep = fine
no i2c-communiction/no interrupts = fine

both = lockup

cimba007

I noticed that I can make the node crash if I trigger some interrupts by force (moving the accelerometer) .. nice ..

I guess the sleep/interrupt code has some kind of bug since the latest rework in the developer branch.

ikkeT

I hope you catch it, as I believe I'm hit by the same bug. It the happens similar way on Sensebender Micro & NRF24L01+. I have a thread on troubleshooting forum with topic of interrupt mystery. I'm arduino newbie, so I'm not much help. I'm just about to try it on Arduino Pro mini just for comparison.

Somehow the enbling of interrupts lock it within a minute or so. If no interrupt enabled, no locking.

BR,
Ikke

cimba007

@ikkeT Are you using the latest stable or snapshot version?

tekka

@cimba007 Thanks, I'll try to reproduce it. What Arduino IDE & AVR Board defs are you using?
@ikkeT Same here, can you upload your (minimal) sketch to reproduce and also indicate MySensors, Arduino IDE and AVR board def versions?

scalz

@cimba007
just an idea..i don't know if it can help at all..but it looks there is a while loop in FaBo3Axis::readI2c
do you think if adding some debug here you get something??
i was just thinking..a freeze that's weird..no wdt?

cimba007

@tekka Arduiono 1.6.12

##############################################################
# Add the new board to boards.txt (normally located at "C:\Program Files\Arduino\hardware\arduino\avr"
# The *.bootloader.* etries only matters if you want to program bootloader (and fuses) from Arduino IDE. 
# See http://www.engbedded.com/fusecalc (select Atmega328p) for interpretation of fuse values and how 
# extended fuses are written in different applications (07h in Arduino IDE = FFh in Atmel studio).
##############################################################

apm96.name=APM Optiboot internal 1MHz noBOD 9600baud

apm96.upload.tool=avrdude
apm96.upload.protocol=arduino
apm96.upload.maximum_size=32256
apm96.upload.speed=9600
apm96.bootloader.tool=avrdude
apm96.bootloader.low_fuses=0x62
apm96.bootloader.high_fuses=0xde
apm96.bootloader.extended_fuses=0x06
apm96.bootloader.path=optiboot_v50
apm96.bootloader.file=atmega328_1a.hex
apm96.bootloader.unlock_bits=0x3F
apm96.bootloader.lock_bits=0x2F
apm96.build.board=AVR_APM96
apm96.build.mcu=atmega328p
apm96.build.f_cpu=1000000L
apm96.build.core=arduino
apm96.build.variant=standard

##############################################################

I changed the fuses to include BDOUT @ 1,8Volt
VCC of my setup is 2-3Volt

cimba007

@scalz Thanks for the hint with the while-loop. I changed the i2c-read the following way:

void FaBo3Axis::readI2c(uint8_t register_addr, uint8_t num, uint8_t buffer[])
{
  Wire.beginTransmission(_i2caddr);
  Wire.write(register_addr);
  Wire.endTransmission();

  Wire.requestFrom(_i2caddr, num);

  int i = 0;
  uint8_t limit = num;
  while(Wire.available() && limit--)
  {
	Serial.print("r");
    buffer[i] = Wire.read();
    i++;
  }
}

Now it should under no circumstanced read more then the number given. Sadly this had no impact:

 | 34291
34291 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0
34357 TSF:MSG:ACK
134390 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0
r34471 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255
34537 MCO:SLP:TPD
34553 MCO:SLP:WUP=1
 | 34553
34553 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0
34553 TSF:MSG:ACK
134553 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0

Just before the sleep ..

I then added a Serial.flush() to the "r"-output ..

void FaBo3Axis::readI2c(uint8_t register_addr, uint8_t num, uint8_t buffer[])
{
	Serial.print("readI2c");
	Serial.flush();
  Wire.beginTransmission(_i2caddr);
  Wire.write(register_addr);
  Wire.endTransmission();

  Wire.requestFrom(_i2caddr, num);

  int i = 0;
  uint8_t limit = num;
  while(Wire.available() && limit--)
  {
	Serial.print("r");
	Serial.flush();
    buffer[i] = Wire.read();
    i++;
  }
}

54706 MCO:SLP:WUP=1
 | 54706
54706 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0
54706 TSF:MSG:ACK
154706 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0

Now the freeze is clearly before entering the sleep function ..

but then .. on another round ..

 | 56623
56623 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0
56623 TSF:MSG:ACK
156623 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0
readI2c

readI2cgot # from slave: 1
r78987 MCO:SLP:MS=16,SMS=0,I1=1,M1=3,I2=255,M2=255
79052 MCO:SLP:TPD
79069 MCO:SLP:WUP=1
 | 79069
79069 TSF:MSG:READ,0-0-14,s=0,c=1,t=16,pt=2,l=2,sg=0:0
79069 TSF:MSG:ACK
179069 TSF:MSG:SEND,14-14-0-0,s=0,c=1,t=16,pt=2,l=2,sg=0,ft=0,st=OK:0
readI2c

So .. your tip was very good .. it seems to be related to the

  Wire.beginTransmission(_i2caddr);
  Wire.write(register_addr);
  Wire.endTransmission();

-call .. only one questions remains .. why was it working perfectly on the stable library?

Just to be sure I will run exactly the same code in a non-stop loop with the stable library ..

tekka

@cimba007 What AVR board defs are you on?

cimba007

@tekka said:

AVR board def

What do you mean with AVR board def?

https://forum.mysensors.org/topic/4999/latest-git-snapshot-causes-freezes/17

I am using an arduino pro-mini china clone (328p) with 8mhz internal oscillator and DIV_8 fuse.

The bootloader is an "APM Optiboot 1Mhz" .. cant find the link right now.

[SOLVED] latest git-snapshot causes freezes

12

11.7k

11.2k

113.0k