Over the air updates

ToSa

Agree it's not an easy exercise to shrink the RF code into the bootloader section.

I did an OTA bootloader within the limited space available in the 328p using a different RF module (which shouldn't be a major issue as the specs were pretty similar) and a different protocol (which might be an issue). I had to rewrite the init/rx/tx components of the RF module driver to be rather static and honestly no longer stick to the OSI model but go right through from app layer to hardware layer to make it fit and I had to reduced the use-cases in a way that the node in bootloader mode essentially only sends very specific package types and reacts to very specific package types being received.

I did not add any specific hardware (getting back to the initial comment about more complex PCB). The bootloader in inside the 328p and software reboot was handled by setting the watchdog to a short period and start an endless loop (instead of adding some additional hardware components to handle that).

I'll share more details when I had a chance to look into the protocol a bit further probably over the weekend. If I can't find the time, I'll share the code as it is...

ToSa

I couldn't get the connection working by just downloading and installing the Arduino and Raspberry components from GIT. It appears that even the defaults for the connection settings are different (e.g. 1M vs 2M transfer rate). I did not dig deeper but I would highly recommend to merge the projects or at least reference on a common set of header files for e.g. protocol specific details (e.g. the enums). That way the risk to run out of sync is way smaller. That said I don't have a running environment yet to test but started reviewing the protocol...

the max bootloader size is 2048 words -> 4096 bytes (still not a lot)
there are two ways to initiate the bootloader: either on power on of the sensor node or by submitting a specific package to the node that causes a reboot
during bootloading the node does not route any packets to other nodes - it is a pure leaf node for that short period of time even if it's a router under normal condition
the bootloader first of all submits its identity (unique address / type of node / current firmware version) to the master and waits for some time for a response. If the identity is not known (first start / no address known), a new address is requested similar to DHCP. If no response, it does a few retries and then finally starts the existing program. Fir this step to work the details about the node can't be stored "somewhere" in the EEPROM but need to be ad a well defined address so that both the bootloader and the program itself can access the same data.
the master replies with details about the latest program version for the sensor node type
if the response from the master lists the same version as the one installed, the node boots into the existing program
if the master has a new program version then the node starts fetching the new program in small chunks and writes to the 328p program mem - once done it reboots running through the full process again assuming that this time the program is the latest available

In my previous setup I used the master node as software distributor - but now realized that some folks might want to keep the two separate, hence I'll add the option to use a different node for software distribution. This should allow e.g. users with a Vera to keep the Vera as master but use e.g. a Linux PC with an RF module connected (e.g. USB<->ATmega<->nRF24L) for software updates...

Announcing / requesting a new address etc is already covered by the protocol (auto assignment of radioID). A couple of packet types need to be added for the submission of the program etc. ll of this looks doable at this point.

I'll post about progress in this thread.

hek

@ToSa said:

I couldn't get the connection working by just downloading and installing the Arduino and Raspberry components from GIT. It appears that even the defaults for the connection settings are different (e.g. 1M vs 2M transfer rate). I did not dig deeper but I would highly recommend to merge the projects or at least reference on a common set of header files for e.g. protocol specific details (e.g. the enums). That way the risk to run out of sync is way smaller. That said I don't have a running environment yet to test but started reviewing the protocol...

The RPI code is in a weird state right now.
http://forum.mysensors.org/topic/8/code-for-beta-testing#19

Your OTA knowledge is very valuables. It might be too complicated to make Vera the firmware distributor (because of the plugin-system-limitations). It might be better to leave this functionality to the RPI gateway only where we have full control.

Keep us updated ;)

axillent

I see tow possible solutions:

firmware upload is done by a separate device connected direclty to PC and with no any relations with RPI or vera
In this case the upload can be very similar to how we upload firmware using USB
RPI can be a helper. I see similar thing with ConnectPort X2 from Digi. It is Xbee to Ethernet gateway and alos it is a Xbee/Zigbee router node.
I can select Zigbee device from the list using WEB interface of ConnectPort X2 and can use file upload.
This one looks more robust and may be a right design but for DIY we do it will require more steps to download updated firmware

ToSa

Now that the Arduino and the RPi talk to eachother (http://forum.mysensors.org/topic/8/code-for-beta-testing#27) I'm looking into ota updates.

The bootloader needs to know what firmware to request / what firmware the sensor is running - finally it needs to know what hardware is build into the sensor to load the correct firmware update. One option would be to have a separate bootloader for each sensor hardware - but that would be a nightmare to maintain over time.
Looking at the code the sketch info provides this kind of detail - but the sketch info is part of the firmware which ends up in a catch 22. We need to store the sketch info in EEPROM at an address known to the bootloader as well as to the firmware later on. Instead of storing the sketch name I would recommend to store some kind of a sensor type ID to reduce EEPROM consumption - sensors with the same hardware would get the same sensor type ID. The ota bootloader could then check with the firmware provider if there is a new version firmware for this kind of hardware...

I would propose the following: instead of storing separate bytes for address / relay / ... in EEPROM, create a struct and store that struct in EEPROM - load the struct from EEPROM to RAM on startup and check if the crc is valid... The most sophisticated approach would probably be to add a kind of unique MAC address to each sensor and to separate hardware specific and software specific configuration - something like:

struct HardwareConfig {
uint16_t hwType;
uint8_t hwVersion;
uint8_t macAddress[6];
uint16_t crc;
};

struct SensorConfig {
uint8_t address;
uint8_t relayAddress;
uint16_t fwVersion;
uint16_t crc;
};

Rough bootloader code could look like this;

--if hardware config is not valid (crc)
----// this is the probably the very first startup of the sensor after flashing of the bootloader
----send "who am i" request to firmware provider
----wait for "who am i" response from firmware provider
----store response in hardware config -> EEPROM
--if sensor config is not valid (CRC)
----// this is equivalent to the current implementation of "no radioID in EEPROM"
----request radio ID (same way as implemented already)
----store response in sensor config -> EEPROM
--send "what is the latest firmware for hardware type x" request to firmware provider
--wait for "what is the latest firmware for hardware type x" response from firmware provider
--if newer firmware available
----invalidate firmware version/crc in EEPROM
----for each block (small chunks due to max packet size)
------send firmware request to firmware provider
------wait for firmware response from firmware provider
------write to progmem
----write new firmware version/crc to EEPROM
--boot into sensor firmware

Any comments/suggestions?
If none then I'll share some code in the next few days to get the ota going - will start with the most sophisticated and then reduce / adjust if needed e.g. to get the code small enough to fit into the bootloader section :-)

hek

Looks good!

It would be great to have store a link to a specific git-url sketch and/or sha(ish) somehow. This way if user switches controller or if loses all data the new can pick up where the last died.
If we later setup a headless build-server in the cloud we could automate the whole compilation/upload process. That would be super clean!
Is it possible to add a hook in the bootloader to trigger a reset (which do the actual upgrade) from the normal sensor program? This way we can "push" a new firmware to a sensor from controller side.

Also note that relaying nodes keeps routing information in EEPROM (256 bytes).

Cheers
Henrik

ToSa

I need to transfer binary data for the ota updates, to keep the packet rate at a minimum - but the code is currently tailored for text. Especially using \0 for message termination and printing message directly in e.g. debug statements won't work. As this is pretty deep in the Sensor class reused all over the place I'm a bit hesitant to change (e.g. Sensor::readMessage()). This wouldn't be an issue for the bootloader as I won't use the Sensor class anyways (too big) but for the RPi the entire chain would be used: RadioGateway <- Gateway <- Relay <- Sensor ...

hek

Yes you are right, I'm doing some work on allowing binary payload right now. If fact I'm considering letting all data be transferred "binary". But there is some cross platform (endian) issues that needs to be solved.
OT: Hmm.. Is both arduino and rpi using ieee floats? Can I send the 4 float-bytes over the air? Need to do some googling...

axillent

Let me add my cents. I do not think that ota updates should use the same message structure as Sensor class is using. We only need one common thing - a clear identification of ota message.
Sensor class should ignore any ota message. Actually it will ignore it without any class modification because we are ignoring any message with bad CRC.

hek

@axillent

If we want to route firmware messages through relaying nodes it must have the same structure.
We need to introduce a new message type to identify firmware messages.

axillent

@hek that is true.
but do we plan to relay ota updates?
sure we can do, but is it a resonable complication?
even zvawe standard is not relaying inclusion/exclusion messages

ToSa

From my pov that's one of the biggest benefits of ota updates : you can do updates "in place" without the need to move the sensor towards the gateway or the other way around.
If we use the same message structure, the additional complexity is limited: gateway and relay nodes know how to deal with it and the only additional step for the sensor is to find the correct relay address. Error handling (switching relay during ota update etc.) would be limited or not existent keeping the bootloader as small as possible - if something unexpected happens like a disappearing relay during the update, the entire update would fail, the sensor reboots and tries again.

hek

@ToSa

Yes, agree!
Need to discuss something with you. Are you available on your registered forum email?

axillent

Probably we can reuse this http://ncrmnt.org/wp/2014/02/27/rf24boot-a-universal-over-the-air-bootloader-for-all-those-ucs/

ToSa

Quick update: I have the low level hardware access code ready (ability to communicate with the nRF24 without the library as the library is too big for the bootloader) and most of the other arduino side boodloader code as well. The raspberry pi side of the story is behind as binary data submission and a database layer are a prereq. I started based on the initial mongodb setup in the 1.4 dev branch but not sure if that's the strategy longer term.
I had some initial success testing the bootloader with some dirty hacks on the raspberry side (removing all debugging that would fail on binary data / removing the handling of trailing 0 etc.) when my hardware started to fail. I replaced the arduino / the nRF24 on both ends and even the raspberry Pi - without success... loaded old code that I knew was working on both ends and it still doesn't work... Both Arduino and RPi seem to work fine but once the first packet arrives from the Arduino to the RPi it reports retrieval and then is stuck unless I reboot the RPi... I'll retry once I'm back from China in two weeks - don't expect to hear anything from my end in the meantime as I won't able to take any hardware with me.

@axillent : the universal bootloader is great but would not be able to utilize the infrastructure (routing / packet format) to communicate and hence would not allow to update sensors that are out of reach for direct communication to the central node providing the updates (gateway or separate).

ToSa

The OTA bootloader was merged into the development branch some time ago. It consists of two components at this point: the OTA bootloader itself and a quick&dirty NodeJSController that connects through a standard SerialGateway or EthernetGateway and is used as repository and sketch distributor for the sensors.
I've created another pull request just now to include a couple of additional tweaks/fixes and an installation guide to get you started (NodeJsController/Readme.html).

Damme

@ToSa I might have missed it but is there any documentation of the protocol used to transmit OTA?
(I looked in the source and might have missed it .. o:) ) How big is the bootloader installed?

Zeph

The initial description sounds like a "pull" architecture, where the sensor node's bootloader figures out whether it needs to update itself and then invokes the bootloading of the appropriate binary.

I have some tendency towards a more "push" oriented approach, where the central code can (1) ask the node about it's current code and version if it has any doubt and (2) command the node to go into bootloading mode.

The advantage is that we don't have to anticipate the future upgrade path in the sensor node's code, and different nodes even with the same hardware could be "told" to program themselves with different code.

In my own case, I might want to change the code in some nodes to go to a higher bandwidth "christmas lights control" mode, then later change it back to a low bandwidth "sensor reports" mode. In other words, there is no implied "upgrade sequence for node type 23", just the ability to arbitrarily reload code in any node from the central controlling software (i'm deliberately being vague about what that central software is: a smart gateway, or a HA controller via a gateway, or a separate laptop or whatever).

I would initiate doing that by changing a config in one place. The config could be as simple as a text file with lines containing a node identifier and a reference to which hex file (or binary equivalent) we currently want in that node. The central software could compute a checksum or hash of the desired binary code, and the node could report the checksum of the current PROGMEM, from which the central software could decide to commend the node to go into OTA bootloading mode.

The code running in the sensor node needs very little to support this. At minimum - nothing, you just do a power cycle and the update happens while the bootloader has control. (A variant of this uses a reed switch to trigger rebooting, so you don't even have to open the case of a battery powered node). Or for nodes that are physically inaccessible, there could be "send me the hash of your current PROGMEM" and "reboot into the bootloader" commands added to the MySensors operational set.

(The "push" OTA bootloading process could differ some in the details eg: it could be initiated by the node checking with central for any update, rather than central sending a command to the node, and othewise work as above. The key thing in the push approach is that the sensor node just reloads whatever program central wants it to load, which is controlled by flexible config at central rather than expecting the sensor node to decide what it will next be programmed with)

(edit: the other difference is that "the proper code to load" is controlled per node, not per node type. So central could load the same code into every type 19 node, or it could differ per physical node.)

Would the OTA bootloader you are writing accommodate "push" bootloading like this, as well as "pull"?

Zeph

Out of curiosity, what's the real world speed like, when doing the bootloading over the MySensor network, with and without a relay node in the middle? Presumably you want reliable delivery so as to not load corrupt code.

Damme

@Zeph
Hmm, My approach would be at the server side decide 'Node 23 needs an update''
Send RESET node 23 (Hmm, I dont know if soft reset executes the bootloader?)
Node 23 ask server 'Do you have an update for me?'
Server : YES! and throws it away

Over the air updates

23

11.7k

11.2k

113.1k