NRF5 Hardware module crash after while

[Didou]

Re: nRF5 action!

I have build up a few NRF modules, including main supply power, Relais, Heating control by pilot wire, all to fit within a 2 DIN unit module within power panel.

They all suffer the same issue, after some time could be few hours or even more than a day they do crash on trying sending something

I have spent nearly a month reading left and right about this in mysensor but did not find ant thread on such issue. I have also tryed either the lastes stable or the dev release without any change. This is propably something evident I dont figure out and thick you help !

I did attach part of the code and part of the debug output when failing

/*
 * This example sketch shows how you can manage the nRF5 pin mapping as part of your code.
 * You can use the method for any nRF51822 or nRF52832 board or module.
 * 
 * Most components, like UART, SPI, Wire Bus, of the nRF5 series chips don't
 * have a fixed pin mapping. There are some pins with restrictions like analog
 * inputs, NFC or pins near the radio module. Please refer the latest
 * documentation about pin restrictions at http://infocenter.nordicsemi.com 
 * 
 * To use the custom pin mapping you have to do following steps:
 * 
 * 1. Install "arduino-nrf5" like described at
 *    https://github.com/sandeepmistry/arduino-nRF5/
 * 2. Install the "My Sensors nRF5 Boards" with the board manager like
 *    explained at https://github.com/mysensors/ArduinoBoards
 * 3. Copy the files "MyBoardNRF5.cpp" and "MyBoardNRF5.h" from
 *    "MyBoardNRF5" example into your sketch.
 * 4. Modify pin mappings in "MyBoardNRF5.cpp" and "MyBoardNRF5.h" to fit
 *    your requirements.
 * 5. Select "MyBoardNRF5 nrf52832" or "MyBoardNRF5 nrf52822" as your board.
 *    Choose the correct parameters and programmer in the Tools menu.
 */

/* Module DIN 6xPilotes
 *  LED verte Power
 *  LED Rouge état Pilote (6x)
 *    P0.20 -> LED D4 Pilote 1
 *    P0.25 -> LED D8 Pilote 2
 *    P0.19 -> LED D3 Pilote 3
 *    P0.23 -> LED D6 Pilote 4
 *    P0.18 -> LED D2 Pilote 5
 *    P0.24 -> LED D7 Pilote 6
 *  LED Bleu Communication Radio
 *    P0.22 -> LED D5 Comunication
 */

#define MY_DEBUG
#define MY_SPECIAL_DEBUG
#define MY_DEBUG_VERBOSE
#define MY_DEBUG_VERBOSE_NRF5_ESB
// #define MY_TRANSPORT_SANITY_CHECK


#define MY_RADIO_NRF5_ESB
#define MY_NODE_ID 121
//#define MY_PASSIVE_NODE

// Inverses the behavior of leds
#define MY_WITH_LEDS_BLINKING_INVERSE

// Flash leds on rx/tx/err
// Uncomment to override default HW configurations
//#define MY_DEFAULT_ERR_LED_PIN 22  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  22  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  22  // Transmit led pin

#define INTER_MESSAGES_WAIT 50

#include <MySensors.h>
#include <Wire.h>
#include "Adafruit_MCP23017.h"

#define LED_ON 0
#define LED_OFF 1

#define LED_PILOTE_1 20 
#define LED_PILOTE_2 25 
#define LED_PILOTE_3 19
#define LED_PILOTE_4 23
#define LED_PILOTE_5 18
#define LED_PILOTE_6 24

#define NB_LED_PILOTE 6

const int LEDS_PILOTE[6] = {
  LED_PILOTE_1,
  LED_PILOTE_2,
  LED_PILOTE_3,
  LED_PILOTE_4,
  LED_PILOTE_5,
  LED_PILOTE_6
};

typedef struct {
   char mode_code;
   int code_A;
   int code_B;
} modeFilPilote;

const modeFilPilote modesFilPilote[4] = {
  {'O', 0, 1}, // Mode Off
  {'G', 1, 0}, // Mode hors gel
  {'E', 0, 0}, // Mode éco
  {'C', 1, 1}  // Mode confort
};

typedef struct {
   int childID;
   char *name;
   char lastOrder;
   int pin_A;
   int pin_B;
   bool newState;
} FilPilote;

FilPilote ListFilPilote[6] = {
  {0, "Pilote1", 'O', 0, 8, true},
  {1, "Pilote2", 'O', 1, 9, true},
  {2, "Pilote3", 'O', 2, 10, true},
  {3, "Pilote4", 'O', 3, 11, true},
  {4, "Pilote5", 'O', 4, 12, true},
  {5, "Pilote6", 'O', 5, 13, true}
};

#define NB_PILOTE 6

int LedTimer = 0;
int BatteryTimer = 0;

MyMessage msg_S_HEATER(14,V_STATUS); 

Adafruit_MCP23017 mcp;

//-------------------------------------
void nRF_Init() {
  NRF_POWER->DCDCEN = 0; // Not reducing the radio current   
  NRF_PWM0  ->ENABLE = 0;
  NRF_PWM1  ->ENABLE = 0;
  NRF_PWM2  ->ENABLE = 0;
  NRF_TWIM1 ->ENABLE = 0;
  NRF_TWIS1 ->ENABLE = 0;
  //NRF_RADIO->TXPOWER = 8;
}

void disableNfc() {
  NRF_NFCT->TASKS_DISABLE = 1;
  NRF_NVMC->CONFIG = 1;
  NRF_UICR->NFCPINS = 0;
  NRF_NVMC->CONFIG = 0;
}

void turnOffAdc() {
  if (NRF_SAADC->ENABLE) {
    NRF_SAADC->TASKS_STOP = 1;
    while (NRF_SAADC->EVENTS_STOPPED) {}
    NRF_SAADC->ENABLE = 0;
    while (NRF_SAADC->ENABLE) {}
  }
}

//void sleepI2C() {
//  NRF_TWI1->ENABLE=TWI_ENABLE_ENABLE_Disabled << TWI_ENABLE_ENABLE_Pos;  // I2C Begin is then required on resume (sensor.begin();)
//  *(volatile uint32_t *)0x40004FFC = 0;
//  *(volatile uint32_t *)0x40004FFC;
//  *(volatile uint32_t *)0x40004FFC = 1;
//}
//---------------------------------------

void preHwInit() 
{ 
  // Configure Leds ports 
  pinMode(22, OUTPUT);  
  
  for (int i = 0; i < NB_LED_PILOTE; i++) {
    pinMode(LEDS_PILOTE[i], OUTPUT); 
    digitalWrite(LEDS_PILOTE[i], LED_OFF); 
  }  

  // Test Leds
  for(int i=0; i < NB_LED_PILOTE +1; i++)
  {
    if (i-1>=0)digitalWrite(LEDS_PILOTE[i-1], LED_OFF);
    if (i < NB_LED_PILOTE)digitalWrite(LEDS_PILOTE[i], LED_ON);
    wait(250);
  }  
}

void before()
{    
  nRF_Init();
  disableNfc();
  turnOffAdc();       
  wait(200);
}

void setup() 
{  
  mcp.begin();      // use default address 0  

  // Load Pilot saved states
  for (int i = 0; i < NB_PILOTE; i++) {
    ListFilPilote[i].lastOrder = loadState(ListFilPilote[i].childID);    
  }     

  // Set MCP ports
  for (int i = 0; i <= 15; i++) {
    mcp.pinMode(i, OUTPUT);
    mcp.digitalWrite(0, HIGH);
  }
}

void presentation()
{
  sendSketchInfo("DIN 6xPILOTES Gite 1", "0.8");
  wait(INTER_MESSAGES_WAIT);  

  // Present Pilote
  for (int i = 0; i < NB_PILOTE; i++) 
  {
    present(ListFilPilote[i].childID, S_HEATER, ListFilPilote[i].name);  
    wait(INTER_MESSAGES_WAIT); 
  }  
}

void loop() 
{
  /*
   * Off: Half upper                 Led: Off
   * Confort: Nothing                Led: On 
   * Eco: Half upper & lower (Full)  Led: Blinking 50/50
   * Frizing Protect: Half lower     Led: Blinking 10/90
   */
   
  for(int i=0; i < NB_LED_PILOTE; i++)
  {
    switch (ListFilPilote[i].lastOrder)
    {      
      case 'G': // horsgel
       if (LedTimer <9) digitalWrite(LEDS_PILOTE[i], LED_OFF); 
       else digitalWrite(LEDS_PILOTE[i], LED_ON);        
       break;
      case 'E': // Eco
       if (LedTimer < 5) digitalWrite(LEDS_PILOTE[i], LED_ON); 
       else digitalWrite(LEDS_PILOTE[i], LED_OFF); 
       break;
      case 'O': // off
       digitalWrite(LEDS_PILOTE[i], LED_OFF); 
       break;
      case 'C': // Confort
       digitalWrite(LEDS_PILOTE[i], LED_ON); 
       break;
    }       
  }
  LedTimer++;
  if (LedTimer > 9) LedTimer = 0;
  BatteryTimer++;
  if (BatteryTimer > 599)
  {
    BatteryTimer = 0;
    #ifdef MY_DEBUG
      Serial.println("Send Battery Level");      
    #endif
    sendBatteryLevel(100);
    //  sendHeartbeat();
  }
  
  wait(100);
}

void receive(const MyMessage &message)
{   
 if (message.type == V_HVAC_FLOW_STATE)
  {
    for(int p=0; p < NB_PILOTE; p++) 
    {
      if (message.sensor == ListFilPilote[p].childID)
      {
        #ifdef MY_DEBUG
          Serial.print("--Incoming change for: ");
          Serial.print(ListFilPilote[p].name);
          Serial.print("  --Command: ");
          Serial.print(message.getString()[0]);
          Serial.println();
        #endif

        for(int e=0; e < 4; e++)
        {
          if(message.getString()[0] == modesFilPilote[e].mode_code)
          {
            // Set new state
            mcp.digitalWrite(ListFilPilote[p].pin_A, modesFilPilote[e].code_A); 
            mcp.digitalWrite(ListFilPilote[p].pin_B, modesFilPilote[e].code_B);        
            
            // Save state
            ListFilPilote[p].lastOrder = modesFilPilote[e].mode_code;
            saveState(ListFilPilote[p].childID, modesFilPilote[e].mode_code); 

            // Send status update
            send(msg_S_HEATER.setSensor(ListFilPilote[p].childID).set((const char *)(modesFilPilote[e].mode_code)));
            return;
          }
        }
        // Unknown Order
        #ifdef MY_DEBUG
          Serial.print("--Unknown Order: ");
          Serial.println(message.getString()[0]);
        #endif
      }
      // Wrong Child_ID
      #ifdef MY_DEBUG
          Serial.print("--Wrong ChildID: ");
          Serial.println(message.sensor);
        #endif
    }
  }
}

and the logs

05:08:52.392 -> 72652589 NRF5:SND:TO=0,LEN=8,PID=0,NOACK=0
05:08:52.392 -> 72652596 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:08:52.392 -> 72652601 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:09:52.387 -> Send Battery Level
05:09:52.387 -> 72712609 NRF5:SND:TO=0,LEN=8,PID=1,NOACK=0
05:09:52.387 -> 72712616 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:09:52.387 -> 72712621 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:10:52.433 -> Send Battery Level
05:10:52.433 -> 72772629 NRF5:SND:TO=0,LEN=8,PID=2,NOACK=0
05:10:52.433 -> 72772636 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:10:52.433 -> 72772641 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:11:52.429 -> Send Battery Level
05:11:52.429 -> 72832649 NRF5:SND:TO=0,LEN=8,PID=3,NOACK=0
05:11:52.429 -> 72832656 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:11:52.429 -> 72832661 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:12:52.459 -> Send Battery Level
05:12:52.459 -> 72892669 NRF5:SND:TO=0,LEN=8,PID=0,NOACK=0
05:12:52.459 -> 72892676 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:12:52.459 -> 72892681 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:13:52.481 -> Send Battery Level
05:13:52.481 -> 72952689 NRF5:SND:TO=0,LEN=8,PID=1,NOACK=0
05:13:52.481 -> 72952696 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:13:52.481 -> 72952701 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:14:52.511 -> Send Battery Level
05:14:52.511 -> 73012709 NRF5:SND:TO=0,LEN=8,PID=2,NOACK=0
05:14:52.511 -> 73012716 NRF5:SND:END=1,ACK=1,RTRY=2,RSSI=-45,WAKE=8
05:14:52.511 -> 73012721 TSF:MSG:SEND,121-121-0-0,s=255,c=3,t=0,pt=1,l=1,sg=0,ft=0,st=OK:100
05:15:52.484 -> Send Battery Level
05:15:52.484 -> 73072729 NRF5:SND:TO=0,LEN=8,PID=3,NOACK=0

[evb]

@Didou, if I understand it well, you have a few NRF modules build.
Each module consist of one NRF5 board, relais, pilot connections for heating control and powered by a onboard 220V AC convertor module. All this packed together in a 2 DIN unit module inside your house power panel = electrical cabinet of your house?

If my understanding is correct, the first thing I think of, is electrical influence of the high AC voltages inside your electrical cabinet. Don't know where you live, but are we speaking of 220/250V or 380V?
Low voltage applications powered by 5V or 3.3V do not like noisy environments like a electrical cabinet.

How long are the pilot wires to your heaters? Long wires act as antennas...
Did you try to operate them outside your electrical cabinet on a test basis?

[ncollins]

@Didou There is currently a bug in the NRF5 transport layer that causes them to lock up. It seems more likely to happen when the connection between the gateway/repeater is weak or too far.

You can read more about it here: https://forum.mysensors.org/post/108301.

My understanding there is a fix pending review and @waspie has been testing it with some NRF51822 modules.

[waspie]

@ncollins
Seems to be working well on NRF51, not so much on NRF52. I backed off the changes for NRF52 modules and they're working ok with latest alpha. I think I need a better sketch for them...

Anyway, NRF51822 is doing much much better.

@Didou follow the links posted by @ncollins

The first big dip on 11/29 was a deadlock. I tried some change after that that still didn't work and lead to another deadlock on 12/7. Using the fix from github you can see how much more stable battery reporting has been since 12/9 (small spike when it was plugged in for programming) when i flashed it using the fixed code. This one might finally be fixed.

[Didou]

@waspie @ncollins Thanks for feedback, when trying the fix are you taking the full github dev or only the proposed fix for this issue ? To manage the deadloack are you using the watchdog ?

[Didou]

@ncollins Will try that just have to understand how to just add the fix to the latest stable release

[Didou]

@evb You do undertstand it very well, it was also my first understanding and to avoid any issue relating to EMC I did also tried outside the cabinet, not using my own power supply adding extra capacitors and lately power up the CPU module with the NFC on batteries .... In all cases the crash issue is still there !

I was pretty surprises as there is quiet a lot of different modules been built on openhardwareio and I have seen no report about this issue, so i was conviced the issue was mine !

[evb]

@Didou let's then keep our fingers crossed that it's just a software problem

[waspie]

@Didou @evb
I download the dev version AND replaced the two files with the ones from the proposed fix.

I think that's the right path - it seems to be showing positive results.

[ncollins]

@Didou for my testing, I pulled the entire branch referenced in the merge request. I believe that is MySensors-development + nrf_esb_redesign changes. But you're probably safe to just replace the two files touched in the commits.

[Didou]

@ncollins OK thanks, I did try that pulling diretly from github, so branch + files (Alpha-1)

I did upgrade 3 modules and keep many others with previous to have a comparison basis

Let's cross fingers

It is about 12 hours for now, none of the updated one did crash and one for the unpatched did crash. I need to wait at lest 24 to 36 hours to confirm

[ncollins]

@Didou Which NRF5 module are you using exactly?

I also believe this code from your sketch is unnecessary, given MySensors handles most of it:

void nRF_Init() {
  NRF_POWER->DCDCEN = 0; // Not reducing the radio current   
  NRF_PWM0  ->ENABLE = 0;
  NRF_PWM1  ->ENABLE = 0;
  NRF_PWM2  ->ENABLE = 0;
  NRF_TWIM1 ->ENABLE = 0;
  NRF_TWIS1 ->ENABLE = 0;
  //NRF_RADIO->TXPOWER = 8;
}

void disableNfc() {
  NRF_NFCT->TASKS_DISABLE = 1;
  NRF_NVMC->CONFIG = 1;
  NRF_UICR->NFCPINS = 0;
  NRF_NVMC->CONFIG = 0;
}

void turnOffAdc() {
  if (NRF_SAADC->ENABLE) {
    NRF_SAADC->TASKS_STOP = 1;
    while (NRF_SAADC->EVENTS_STOPPED) {}
    NRF_SAADC->ENABLE = 0;
    while (NRF_SAADC->ENABLE) {}
  }
}

[Didou]

@ncollins I'm using NRF52832 Ebyte modules

[Didou]

@Didou After a few days of test of your NRF fix, I can say there is a huge improuvement either for USB, AC supply or battery powered nodes, they just do not failed ! this is great. thanks for your efforts improving the mysensor code

It's a while since the mysensor master has been updated, any ideas when it's goanna happend, it will also be good for others to benefits

[waspie]

This post is deleted!

[waspie]

@ncollins

Are you currently working with any NRF5x? Do you have a working motion/contact sketch for the NRF52?