[Solved] Strange behavior of RFM69 OTA burning firmware after some research

[yury]

Found strange behavior of RFM69 OTA burning firmware after some research
OTA + RFM 69 (possible) glitches.
Used MYScontroller – works super flawless
Tried both MySensors Library 2.0 and MySensors Library v2.1.0
Various boards made by Koresh 8 Mghz Atmega with RFM69 radios

The issue: On larger sketches (like more than 20kbytes), OTA always gives CRC error. Smaller ones (4k or less) could be burned 1 out of 3 tries.

In the picture below the blues screen is comparing two files (right column is HEX and two lines actually is single 16 bit OTA payload)
Bin.txt file is parsed Arduino hex output to the way it is identical to JDEC flash content. TXT2 file is JDEC flash content.
Log indicates NACK error for the block B=0065 and LEN !=29 (assume it is 0 length payload). After that, somehow instead of re-requesting that corrupted or 0 sized payload, it duplicates previously received (marked in yellow) into new address. This can happen sometimes 2-3 or 10 times during OTA process

I spend many hours playing with MyOTAFirmwareUpdate , transport and RFM driver trying “fixing” logic there but failed. Solution found Koresh by increasing from 40ms to 100ms retryWaitTime in RFM69.h

virtual bool sendWithRetry(uint8_t toAddress, const void* buffer, uint8_t bufferSize,  uint8_t retries=2, uint8_t retryWaitTime=100); //!< sendWithRetry (40ms roundtrip req for 61byte packets) 

So may be something strange happens when rfm ack is received outside the circle in sendWithRetry function.

My and Koresh concern it is not “real” solution. Since this kind of re-requesting of 0 sized payloads should be dealt very well in protocol layer. Otherwise if there is lots of traffic going on
In the network, there will be lots of this kind of 0 payload situations.
The fix actually gives good results (10 out of 10 without CRC errors). It works now but… I think would be great to look into deeply.

@tekka

[tekka]

@yury interesting finding, can you also post the MYSController log?

Edit: Ok, this issue has a dual cause: RFM69 driver re-transmission timing and a missing payload check for incoming messages (needed as a consequence of the timing issue) - I will elaborate this today or later this week.

[yury]

I will reproduce tonight. with MYSController log.

[tekka]

@yury Please try with this, I've quickly added a check:

https://github.com/tekka007/MySensors/tree/OTAUpdate
PR: https://github.com/mysensors/MySensors/pull/718

Edit: PR merged, try with the latest development: https://github.com/mysensors/MySensors/tree/development

[yury]

@tekka

with your new PR updated all works good now. I could not replicate @ 40 ms in retryWaitTime the mentioned above. so I put 10 ms like a stress test. It has survived! with lots of WRONG FWB but with CRC OK!

the logs

serial
https://drive.google.com/open?id=0B11zpaGZaUo5c3NMT1VuSGoxU3c

last OTA is correspoding to serial log
https://drive.google.com/open?id=0B11zpaGZaUo5MVh2cUg2cWMxWER0aHEwWHhoY1VidEpuOURn

[Koresh]

Just 2 cents... We couldn't replicate 40 ms in retryWaitTime the mentioned above in another location. And tonight Yury couldnt replicate this case at home too. But yesterday it was typical situation. So I can assume it was caused by high environment radio noise (city center with many potential RF sources). I think it is an ideal situation to test boards and improve library stability

[tekka]

@yury @Koresh Thanks for your feedback and thorough analysis, that helped to identify the issue.

[napo7]

Hi,

I might be a bit dumb , but I can't find RFM69 OTA bootloader ?

Has anyone done it ?

[scalz]

@napo7
this is sensebender micro bootloader, same as dualoptiboot from lowpowerlab
https://www.mysensors.org/about/ota

[napo7]

Ah ah ! I was sure that I was dumb !!

I'll have to give a try to port MYSBootLoader to rfm69 so ....

[tekka]

@napo7 There is something cooking in the lab...will post more in a bit

[napo7]

Well, I'll wait a bit, so !

BTW :how much is a bit for you ?

[tekka]

@napo7 Hmm...1/8th of a byte I cannot give you an ETA right now, but feel free to write your own bootloader implementation of the RFM69 driver.

[napo7]

Well done ! I must admit I wasn't ready to read such a word play
Hum... In fact, I did a small try, cloned the RF24 bootloader, tried to search which pieces of the code I have to modify, but didn't go further....
If you was about to release it in the future days (I'm willing to help/debug/test), I'd have waited, but my projects are going to be on a rush....

I'll try to go a bit further in making my own version

[tekka]

@napo7 perfect!

[yury]

The bootloader and OTA update script + RFM69 work super nice with MYScontroller. I use it few times a week. Big Big Big ups to Tekka. Especially I like the sensebender micro bootloader boots instantly ))). Enjoy everybody!

[napo7]

Hi @tekka,

Any news about RFM69 for MYSBootloader ?

I've had another look in the code, too much complicated for my skills !

[tekka]

@napo7 No, nothing that is close to a release - I've done some PoC with rfm69 bootloading but this needs more time.

[napo7]

Hi @tekka
Would you share your poc so I could try to enhance it ?

Regards