What is the correct way to implement a WDT, for reset on a Sleeping node?

[Nigel31]

Hi,

I have a node, which is both remote (middle of the garden, measuring oil level, and external temperature) ,and battery powered, it sleeps for 10 Mins, then wakes, measures and reports. All is mostly well, however occasionally, it fails to report for an extended period, and I have to manually reset it, when all is then well for weeks.
I have tried to implement using the onboard WDT, however this results in the board never really initialising, and sending data.
I can build an external WDT, however before revisiting this, I thought I should ask for other wisdom, as there may be an established method for doing this, that I can't seem to find.

Many thanks

[electrik]

@njwyborn are you sure the board is not responding? Perhaps the message is never delivered due to connection issues? What type of transceiver are you using?

[Nigel31]

RFM69 Series, Signal strength is good, circa -77db.
When this occurs, the battery is significantly drained, if I don't reset it, This leads me to believe that it is hanging mid process, and leaving the sensors powered.
The sensors are powered via logic level mosfets, which are controlled via digital pins on the processor.
As I say, it works fine for weeks on end, then hangs on an indeterminate frequency, this has worked in this fashion for 8 months, the code transmits the data several times for each loop, which is once every 10 mins, I (when I have specifically checked on occasion) get all the transmits of a loop. The ultrasonic sensor, can pull in the order of 60mA, when operating, hence the power mosfet(s).
I have concluded thus, that the "system" is hanging after the mosfet is enabled, but before it is disabled. There may be myriad reasons as to why, but the system is otherwise OK, when reset, hence the wish to employ a hardware WDT, as the onboard WDT is used by the Sleep function, I don't seem able to employ that, at least I haven't been able to successfully.

Thus I am asking mysensors people (so to speak) as how I may implement a WDT, other than building a external circuit.

regards

[zboblamont]

@njwyborn No offence but would it not be better to figure out or obviate the cause other than use the nuclear option of a reboot which may not actually fix the problem ? Your range should be well within parameters so the problem probably lies with sensors locking up processor somehow, the question to answer is why.
eg...
I have several pro-mini/rfm69 nodes around, only two of which have "locked up" in over a year of operation. The two problem nodes turned out to be locked due to accumulated condensation on the downfacing ultrasonic sensors in underground tanks during intensely cold weather. When accessible a wipe across the face recovered it, sometimes a bang on the conduit would clear it, today they were wiped with a glove after a week long outage, now reporting as normal.
What I did to get around the issue when buried under a metre of snow at -15 was for the sketch to make X attempts, if it got a reading which was within parameters send it, otherwise at X attempts, go to sleep.
This could go on for days, but at least it saved battery power, and IF the drip release in the interim, reporting would recommence.
If the sensor starts up differently on reboot to sleep recovery, something is mis-set, otherwise there's the equivalent of the drip scenario IMHO...

[mfalkvidd]

@njwyborn to my knowledge, there is no easy way to use the wdt while sleeping.

What was the battery voltage when the node died? There have been a few cases where the voltage got below the accepted range (2.34V for a 8MHz atmega328 for example) which resulted in undefined behavior (mcu hang).

[Nigel31]

Hi,

3.8v on a 8Mhz , Rocket Scream unit, with onboard regulator, and a input range 3.4 – 6 V
This system is a Lithium, charged by solar, even through the winter, it has maintained circa 3.8v, until the week of no data receiving (my assumption of locking up with sensor powered).

I am uncertain as to why or how a condensated sensor surface would cause the processor to lock, incorrect readings for sure, I suppose it is conceivable that the library I am using might get stuck in a loop, and thus never return to the main loop of the program, thus never sleep, or turn off the power to the sensor. The library is located here -
[https://bitbucket.org/teckel12/arduino-new-ping/downloads/](link url) The installed version is 1.9.0, I see there is an update, Ill re-build and upload any way, and see if that helps over time.

However to return to the origional question, it seems that using the onboard WDT is not really possible, I did try enabling it at the top of the main loop, and disabling before sleep, but that didn't allow it to run either.

Unless there are any other suggestions, Ill, have to employ a TPL5110 timer, and power cycle the board every interval. Not what I really want to do, anybody have a better solution?

regards

[zboblamont]

@njwyborn
I see from your post you referenced "new-ping", so by pure coincidence you are referring to ultrasonic also.
In my own case they are 5v JSN type single transducer "waterproof" heads which have been reliable except in -10 -20 weather conditions. They are relay controlled boards with 5v supply, fired every hour from an onboard RTC - I ultimately discovered the problem was a build up of condensation ultimately forming a drip on the face of the transducer, which interferes with both emit and receive.
After a wipe yesterday, today both are reading consistently after an absence of around a week.
You said a reboot resolves it so perhaps yours is a different issue, or perhaps you are inadvertently shaking the head somehow when rebooting?
I can certainly recommend limiting the number of attempts to get a valid result, battery loss in the last week of NIL reporting has been minimal...

[DavidZH]

I have an idea that it still might be a low voltage situation. You say your ultrasonic sensor consumes about 60mA. When your battery voltage is at 3.8V those currents combined with the spikes caused by your radio can pull your battery down to below 3.3V.
I have an outdoor sensor with a Moteino as board and that uses a MCP1703 as voltage regulator. Same family as the MCP1700-33 of your board. When my supply voltage drops in the direction of the regulated voltage, the MCP faults into some sort of short circuit mode which draws an insane amount of power but does not output any usable voltage. And that will drain your battery quick, fast, in a hurry... (️ AvE )

My sensor is being topped up by a solar panel so theoretically, it should never run out. But theory and real world are NOT the same. Your board is also capable of charging a LiIon battery with both the Vin and 5V connections. The BQ24074 will charge your batt.

Hope this helps you towards your solution.

[zboblamont]

@davidzh Indeed possibly voltage related, but unlikely due to be the radio which should not be operational during the reading. An interesting/curious observation on that voltage regulator behaviour though.
Possibly the fault lies not with the processor but the ultrasonic module if on the edge of it's operating voltage, both board versions trialled here were occasionally flaky unless provided a solid 5v, despite allegedly 3.3-5.5v capable.

[Nigel31]

Many thanks, Do you think then that given I wish to remain with a lithium battery, which otherwise has performed suitably, that inclusion of a step-up regulator, to provide a 5v rail for the ultrasonic sensor?
Do you imagine the battery drain with the additional regulator to be a problem?
Would you supply the whole system with the stepped up 5v, or just the ultrasonic?
The radio and temperature sensor are supplied via the 3.3v regulator on the mcu board, the ultrasonic, via a logic level Nch mosfet, fed off the main battery rail.
The ultrasonic device is supposed to have a supply range of 3.3 - 5.5v

regards

[Nigel31]

As a simple fix, i've stepped up the whole supply , with the addition of a step up booster to 5v, I'll see the results!

[zboblamont]

@njwyborn Ditto on the stated voltage for the two ultrasonics I have, they didn't work reliably at 3.3v if at all, but the common pro-mini has an onboard booster down to <1v input from 2AA alkaline.

I can only observe on the two different solutions I trialed, both of which used a a small DPDT latching signal relay fired by mosfet triggers. (ca 50mA for 6ms On/Off)
The original uses a Master 3.3, the relay firing a Slave 5v pro-mini on a 6v battery, in turn powering the Ultrasonic and doing all the processing then passing results on I2C via a level converter to the Master to radio in.
The second uses only the 3.3, the relay powers up a step-up to power from RAW for the ultrasonic and level converter, turned off after either a correct result or 10 attempts.
The first is reasonably frugal but uses 6AA in total, is clunky, problematic, bulky.... Probably it will be changed to the 2nd version when the snows clear.
The second version is only 3 months in, but thus far seems reasonably frugal on the 2AA cells, probably >9 months battery life.
In both cases the problem of power demand was identified as the ultrasonic head developing a drip blinding the head, hence the max X attempts to minimise battery drain when it doesn't get a reading.
The second version is for sure the simplest and most reliable.
Not sure how efficient the 5v booster is but the greatest power loss is probably the remaining LED on the 5v booster (US LED was removed).
The pins on the pro-mini used for the relay are the onboard LED pins which I'm reluctant to remove as they confirm the hourly routine and whether the result was successful or not.
You might try a variable voltage to the Ultrasonic to see where it falls over, but be careful if the pins on your board are not 5v tolerant (hence level converter)
Hope this helps.

[Nigel31]

Hi All,

So in an effort to resolve this, I have rebuilt this device entirely, using all new hardware, and to an extent, topology.
The origional was built on verroboard, and utilised a couple of logic level mosfets to power sensors when needed, sleeping for 10 mins between waking, sampling, transmitting.
I rebuilt using one of the easyPCB boards I have, which are employed in other nodes, very reliably.
I then uprated the battery size, powering the MCU board with raw battery voltage, and using a low dropout 3.3 regulator to power this board. A new utlrasonic board was employed, and powered via a stepup converter to 5v. Level switching IC was used to convert logic levels. The ultrasonic sensor (jsn-sr04t-2.0) was left powered constantly, having a specified consumption when not active of circa 5mA (hence the bigger battery)
During testing it became apparent that when actually starting the sampling for distance measurement, the drain was sufficient to occasionally cause the MCU to reset, adding 220 microF capacitance resolved this, for over 24 hours on the bench.
I deployed this, with a stripped down and potted sensor, to minimise surface area, and have the sensor surface lowermost physically, to atleast help with the possibility of condensate on the surface.This has been running for over a week, but locked up last night. As part of the code changes (power switching no longer required) I added a text reporting entry present(CHILD_ID_TEXT1, S_INFO,"UpdateTxt",false); and used this at several points in the sketch, as a poor man's debug tool. The last entry was AFTER the ultrasonic sampling (fully completed) and before the transmission of the data to the gateway. The battery voltage was 4.11v as reported (fairly accurate).

My thoughts are that perhaps I can stop the node sleeping, and see if the solar charging keeps up with the drain, and employ the inbuilt WDT, alternatively I will have to build a long interval WDT, to enable me to sleep the node, or possibly sleep for short intervals, and loop to reset a external WDT, sampling and sending on the same sort of 10 min interval as current.

Any experience or thoughts?

Regards and thanks Nigel

[zboblamont]

@njwyborn No offence intended Nigel but perhaps you might stand back a little from the WDT and solar concerns, and concentrate on the efficiency of the Node.
I've just graphed the 2AA alkalines for the US Water Tank Node, as attached :
This Node sleeps between hourly wakeups from the RTC, and aside the initial drain in January prompting me to limit the program sample loops, there has been no dramatic voltage drain, it probably will exceed 4 months. There are no smoothing caps at all in circuit aside what are on the boards, and it has never skipped a beat aside from condensation build up on the sensor (<-5c I noted) causing an abort from sending in a crazy reading or after 10 attempts without success.
Presumably you have a need for 10 minute samples, but am completely baffled on running the JSN constantly. It takes <100ms to warm up from the latching relay kicking on with zero consumption, and aside from the drip events has proven reliable...
Will change the conduit from gunbarrel to plastic over the summer as I suspect the condensation source is the tube (heatsink to external -5 to -16), but putting off the solar charger notion until I figure out whether it causes more problems than it's worth, and watching your progress accordingly..

[Nigel31]

Hi @zboblamont
No offence taken at all. I chose to keep the sensor powered simply as a debug process, to eliminate "extra" hardware and code, in an effort to see if somehow that was contributing to my woes! I will re-instigate powering the JSN sensor with the mosfet, as clearly there is no indication that powering up the sensor was causing a brownout, and thus lockup. the other battery nodes have run for months with no issues on 2xAA batteries, with much life remaining in the batteries, and no lockups. The 10 min interval is because the same node also reports external temperature, which I use to accelerate or retard the central heating timing, and set point.
Returning the efficiency of the node to the original however won't help me with the locking up issue, for which I don't seem (as yet) to be able to tie down, and given that a reset ALWAYS "fixes" things, this is why I am pursuing a WDT as a "solution"
Given that the only real differences between this problematic (rebuilt with alt hardware) node and others is limited to

1. Powersupply, lipo battery and solar vs 2xAA Alkaline
2. A ultrasonic sensor
   I can eliminate (1) by powering the node with 2xAA, and see what the results are, and I can change the sensor type to a Sharp GP2Y0A60SZLF Analog Distance Sensor (which was my original design before deployment), however the sensor output isn't linear, and whilst I came up with a quadratic equation to approximate the distance, it wasn't ideal.

I am intrigued with your comments regarding limiting the number of pings / time limiting the loop. In my sketch I utilise the new ping library with a median request of 10 pings, I am assuming that the library employs timed loops to create a "window" for pings, based on the maximum distance you tell the library you are interested in, and thus (hopefully) it never gets stuck in a loop waiting for a ping which will never come, or it has already missed. Do you use a library, or do you use your own code to generate and measure pings, thus calculate time / distance?

Many thanks
Regards Nigel

[zboblamont]

@njwyborn Thought I had already posted the sketch, but here it is anyhow. It had not been tidied up from memory, so some irrelevant stuff may persist from the previous I2C iteration, and could be more efficiently revised. It is all from forum published stuff, nothing fancier than a simple timing routine...

#include <T2WhisperNode.h>

#include <Wire.h>
#include <RTClibExtended.h>

// Enable debug prints to serial monitor
//#define MY_DEBUG //Comment out once all working saving memory

// Enable and select radio type attached
#define MY_RADIO_RFM69
#define MY_RFM69_FREQUENCY RFM69_433MHZ  // Define for frequency setting. Needed if you're radio module isn't 868Mhz (868Mhz is default in lib)
#define MY_RFM69_NETWORKID 101  // Default is 100 in lib. Uncomment it and set your preferred network id if needed
#define MY_NODE_ID 7  //Manually set the node ID here. Comment out to auto assign
//A define should be added to provide minimum power output to provide decent RSSI to the Gateway if ATC not available
#include <MyConfig.h>
#include <SPI.h>
#include <MySensors.h>
#define SN "Node7water3vWNfinalRTC"
#define SV "4"
#define FIRST_CHILD_ID 1
#define SECOND_CHILD_ID 2
#define THIRD_CHILD_ID 3
#define REED_PIN 3   // Arduino Digital I/O pin for button/reed switch
#define BatteryOn 14
#define BatteryIn A6
#define RelayOn 6
#define RelayOff 9
#define Trigpin 17
#define Echopin 16

int halfday=12;
int tankdepth, tankpercent,tankvolume,distance,test,test2,counter;
long duration;
volatile int ultrasonic;
RTC_DS3231 RTC;      //we are using the DS3231 RTC

// Change to V_LIGHT if you use S_LIGHT in presentation below
MyMessage msg1(FIRST_CHILD_ID, V_LEVEL);
MyMessage msg2(SECOND_CHILD_ID, V_LEVEL);
MyMessage msg3(THIRD_CHILD_ID, V_VOLTAGE);
unsigned long SLEEP_TIME = 0;//1800000; Sleep time between reports (in milliseconds)

void setup ()
  {
   pinMode(REED_PIN, INPUT_PULLUP);//Output of DS3231 INT pin is connected to D3 INT1 for RTC wake-up
   pinMode(BatteryOn, OUTPUT);//WN battery sample activation
   pinMode(BatteryIn, INPUT);//WN ADC for battery read
   pinMode(Trigpin, OUTPUT);
   pinMode(Echopin,INPUT);
   pinMode(RelayOn, OUTPUT);//US on
   pinMode(RelayOff, OUTPUT);//US off
   analogReference(INTERNAL);//All voltage reads are to 1.1 internal datum
   VOFF();
   sleep(100);//Allow things to settle was 1000
   Wire.begin ();
   RTC.begin();
   RTC.adjust(DateTime(__DATE__, __TIME__));   //set RTC date and time to COMPILE time

//clear any pending alarms
   RTC.armAlarm(1, false);
   RTC.clearAlarm(1);
   RTC.alarmInterrupt(1, false);
   RTC.armAlarm(2, false);
   RTC.clearAlarm(2);
   RTC.alarmInterrupt(2, false);

  //Set SQW pin to OFF
   RTC.writeSqwPinMode(DS3231_OFF);

  //Set alarm1 every hour at XX:50
   RTC.setAlarm(ALM1_MATCH_MINUTES, 0, 50, 0, 0);//set wake-up time here
   RTC.alarmInterrupt(1, true);//Alarm on
  }
  
 void presentation(){
  // Send the sketch version information to the gateway and Controller
  sendSketchInfo(SN, SV);

  // Register binary input sensor to sensor_node (they will be created as child devices)
  // You can use S_DOOR, S_MOTION or S_LIGHT here depending on your usage.
  // If S_LIGHT is used, remember to update variable type you send in. See "msg" above.
  present(FIRST_CHILD_ID, S_DUST);
  present(SECOND_CHILD_ID, S_DUST);
  present(THIRD_CHILD_ID, S_MULTIMETER);
}

void loop(){
// Power up US
    VON();
   sleep(500);//Delay to let the 5v kit stabilise
   digitalWrite(Trigpin,HIGH);
   sleep (100); 
   digitalWrite(Trigpin,LOW);
   sleep(100);/// Allow decay of pulse
   int voltread;
   counter=0;
   tankdepth=0;
   while ((tankdepth<350||tankdepth>1000)&&counter<=10){//10 attempts to get proper range
   READULTRASONIC();
   counter++;
   } 
//Normal range should be 363 to 989, completely empty 1520, //First overnight runs got depth full as 363mm so formula revise
//   float tankpercent=100-((989-ultrasonic)/6);
  float tankpercent=100-((989-tankdepth)/6.26);//626 effective range
   int tankvolume=(989-tankdepth)*2.3727;//Set to low level cut off
   if(tankdepth>350&&tankdepth<1000){//Don't report if value is outwith known range
   VOFF();
   sleep(100);
   send(msg1.set(tankvolume));//This is actual volume
   sleep(100);
   send(msg2.set(tankpercent,1));//This is remainder to start of 39cm Freeboard
   sleep(100);  
   }
//   }
   VOFF();//If there are 2 yellow flashes it reported and switched off 1 if it failed to read
// Call for battery reading, send in every 12th RTC call?
   if(halfday==12){
   digitalWrite(BatteryOn, HIGH);
   voltread = analogRead(BatteryIn);
   float voltage = (7.272 * voltread) / 1024;
   sleep(5);
   digitalWrite(BatteryOn, LOW);
   send(msg3.set(voltage,2));
   sleep(50);
   voltread=0;
   sleep(50);
   halfday=0;//Reset for 12 hourly cycle
}
   halfday++;//Increment hourly cycle

// Rearm alarm
   Wire.begin();
   RTC.begin();
   RTC.alarmInterrupt(1, true);
   sleep(50);
   Wire.end();
   sleep(50);
//Go deep sleep until RTC interrups
   sleep(digitalPinToInterrupt(REED_PIN),FALLING, SLEEP_TIME);
sleep(100);///Time to recover
//First clear the alarm
   Wire.begin();
   RTC.begin();
   RTC.armAlarm(1, false);
   RTC.clearAlarm(1);
   RTC.alarmInterrupt(1, false);
  }  // end of loop

void VON(){
  digitalWrite(RelayOn, HIGH);
  sleep (10);
  digitalWrite(RelayOn,LOW);
}

void VOFF(){
  digitalWrite(RelayOff, HIGH);
  sleep (10);
  digitalWrite(RelayOff,LOW);
}

  void READULTRASONIC(){//JSN-SR04-2.0
//Main loop checks valid range on a defined number of readings
  tankdepth=0;
  duration=0;
  distance=0;
  test=1;
  test2=3;
// Normal range should lie between 389 and 989 absolutely empty is 1520
while (test!=test2){// Get two consecutive readings
  digitalWrite(Trigpin, LOW);
  delayMicroseconds(100);
  digitalWrite(Trigpin, HIGH);
  delayMicroseconds(150);
  digitalWrite(Trigpin, LOW);
  duration = pulseIn(Echopin, HIGH);
  distance = duration/5.82;//This is in mm
 if (test!=distance){
    test=distance;
    distance=0;
   }
   else{
    test2=distance;
  }
  delay(100);///// 500 originally but why???????
  }
  tankdepth=distance;
 }

The sketch uses the onboard RTC to wake the Node every hour because it is sufficient to track the Water Tank level, and verify the tank is topped up from the borehole overnight. The 5v booster and 3.3v line to the level shifter is via a dpdt latching relay in this iteration to minimise power leakage, two AAs provide supply. As this Arduino is short on pins, the two onboard LED pins switch the relay, a handy visual check as the 1/2 yellow flashes indicate success or failure if the sensor was blinded by a drip.

Note your explanation on the 10 minute sample, but question whether it makes sense? External temp hourly should be plenty accurate for setback control, but curious how you're using it anyway, and will PM...

Hope this helps...

[Nigel31]

@zboblamont . Have modified hardware, re-instated mosfet power of the Ultrasonic sensor AND am currently using 3xAA Alkaline batteries, The UC is now powered again from RAW battery voltage, as seemingly (not definately) stepping up to 5v didn't help much, and added noise (undoubtedly) and power overhead, however small.
It's working at the moment. I know thats several changes in one go, however given the mtbf, I will be at this for many months. If this is no more stable than any other versions, I will re-instate the LIPO battery / Solar charging to the circuit.

I see you are indeed "doing your own" pings as it were, however for the moment I will persist with the newping library, as the last "failure" definitely occurred post "pinging". Also you use sleep's rather than wait's. Ill have to look into this method myself.

I have investigated several single chip WDT's however they all have short reset time requirements (1.2s max, most a few hundred ms) so it looks like I would definitely need to make my own circuit, seems strange, but I can't find anything with a "interval" in the many mins. Unless I sleep in a loop, reset the WDT, sleep again, and count say 1200 loops of 500ms sleep, then continue on with my main loop.

Regards Nigel

[zboblamont]

@njwyborn As an afterthought, forgot to ask if your mosfets are switching power to the US or Ground ? Reason being I vaguely recall reading these SR04s should be solid earthed and not switched.

I'm happy enough with the current setup and results, and no ill effects from the 5v booster contrary to expectations of noise, though it was parked in the case with the ground plane facing inward (old habits) and components facing out.
Good luck anyhow, you'll get there...

[Nigel31]

Hi. Power side switching.