Over the air updates

ToSa

@Zeph: yep - that means in your case you don't really care about the node type. In other scenarios where you have 60 nodes installed and 20 of them are relay nodes, another 20 are switch detectors and another 20 are temperature sensors (all of them having the same hardware setup) the node type is pretty useful. For your specific need you probably should not care about the node type at all - maybe set node type == nodeID and that's it. The additional 2byte payload should not matter too much.

The ideal setup from my perspective would look like this (dreaming): based on the information shared back (combination of sensors and pin connections) the controller would reassemble the source code and build a new sketch for the given configuration, compile it and send it (I'm not kidding - I worked on a very similar approach a few years back).

Reality is: this is meant to be a bootloader for MySensors. The way MySensors currently works is that the combination you mentioned (18B20 temp sensor on pin 7 and power blind relays on pins 5 and 6 and an IR detector on pin 12) requires a specific sketch to be loaded that has these pin assignments etc. hard-coded.

This is the piece of code you would want to adjust - at this point it pulls all available firmware records for the given type and sorts descending by version - which delivers the highest available version back as the first record:

db.collection('firmware', function(err, c) {
	c.findOne({
		$query: {
			'type': fwtype
		},
		$orderby: {
			'version': -1
		}
	}, function(err, result) {

Instead the "expected firmware" type and version could be an attribute for the given node in the "node" collection which is manually maintained:

db.collection('node', function(err, c) {
	c.findOne({
		'id': destination
	}, function(err, noderesult) {
		db.collection('firmware', function(err, c) {
			c.findOne({
				'type': noderesult.expected_firmware_type,
				'version': noderesult.expected_firmware_version
			}, function(err, result) {

Damme

@ToSa Still I wonder if there is any OTA bootloader / protocol readme :) (So I dont have to dissect the nodejs code to write my own implementation)

ToSa

@Damme look at NodeJsController/Readme.html - actually for now better look at this version which has a couple of updates (will send another pull request tomorrow for the documentation as well as some minor changes).

If you are looking for tech documentation (protocol etc.) that's not yet included but the communication is fairly easy (complexity is mainly to make it robust - not kill a node if something goes wrong etc.):

the bootloader is using the same procedure to find its parent / request a nodeID etc. as a normal MySensors sketch would do
then a config request / config response is exchanged between node and controller
assuming an update is needed a series of code block requests / responses is executed until the full firmware is submitted

Data is submitted as binary - you can see the message payload details in MyOtaBootloader.h:

typedef struct
{
uint16_t type;
uint16_t version;
} FirmwareConfigRequest;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t blocks;
uint16_t crc;
} FirmwareConfigResponse;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t block;
} FirmwareRequest;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t block;
uint8_t data[FIRMWARE_BLOCK_SIZE];
} FirmwareResponse;

Zeph

@ToSa said:

yep - that means in your case you don't really care about the node type. In other scenarios where you have 60 nodes installed and 20 of them are relay nodes, another 20 are switch detectors and another 20 are temperature sensors (all of them having the same hardware setup) the node type is pretty useful

Suppose you do have 20 identical temperature nodes. It's trivially simple tor the server to tell each one of them to update to the same code in the "push by node" model. Not only that, but the server gets to decide when to allocate the bandwidth for each node.

Unfortunately, in the "pull by node type" model, you have no way to update some nodes of the given type and not other nodes of that type.

The "push by node" model easily handles any case the "pull by node type" model does, but the opposite is not true.

To even approach the "push by node" dynamics with "pull by node type" design, you have to have two concepts of "node type" which must not be conflated.

node type for purposes of the user interface
node type for purposes of updating the code in the ATMega328p

When you say "20 temperature nodes" the concept of "node type" would be meaningful in the first sense if you mean "20 nodes containing only a temperature sensor for the HA Controller to display".

But for updating the PROGMEM, the concept of "node type" needs to be "20 nodes containing only a temperature sensor of type DHT-11 on pin 6".

A node with a DHT-22 or 18b20 on pin 6, or a node with a DHT-11 on pin 5, would be the same "node type" for purposes of the user interface (which doesn't care), but different "node types" for purposes of updating PROGMEM.

Once you start considering node type = node id in some cases, it becomes simpler to just ignore the already messy and problematic "node type for purpose of update" concept (as seen by the node) and just do updates per node, period. If you want it, you get the functionally of "update all nodes that use identical code" essentially for free at the server end with the push by node model anyway, PLUS the ability to update individual nodes of any to to run any code you want, and when you the server want to schedule it. I don't see the downside of push-by-node here.

At worst, the server could have a table of node-id to "node type" for lookup and then follow your same dynamics. That's not how I'd do it (this model allows even simpler and more flexible options), but it would be a tiny "shim" to allow the more flexible "push by node" model to emulate the "pull by node type" dynamics if a given implementer so desired.

(Just by the way, this discussion is for me fun and mutually respectful brainstorming, I hope it lands that way).

ToSa

@Zeph not sure what you are asking for as I mentioned above that you can use the bootloader as it is today to just update specific nodes (by nodeID, update one and not update another even if they have the same node type). The implementation is not a "pull by node type" but it's a "pull by node ID, node type, node version" - which information the controller uses to decide if an update should be executed is is up to you!!!

Terminology: the "node type" I'm referring to means the specific setup of the hardware - only if that's the same then the node type would be the same (combination of sensors / pin connections / same sketch to be used). The back-end cares about a node type because it needs to know which sketch to use/send.
The user interface ideally never cares about a node type but really cares about the specific sensor type(s). This "translation" needs to happen in the background no matter if you use an OTA bootloader or not.

Examples:

Let's assume you have two nodes in the living room - the user interface should just show "living room temperature" no matter if the temperature sensor is connected to node 1 together with the light switch or connected to node 2 together with the blinds. This "translation" needs to happen anyways - ideally in the controller.
Let's assume you have two temperature sensors connected to one node - one measures the room temperature at 1.5m height and one is a floor temperature (not unusual for floor heating). Just knowing that there are two temperature sensors but not knowing which one is which will not be sufficient for the heating controller to make the correct adjustments. Again that translation from "node 23 with one XYZ temp sensor on pin 5 and one XYZ temp sensor on pin 7" to "node 23 temp sensor at pin 5 is the floor temperature" needs to happen anyways.

Zeph

OK, maybe we are converging in some ways. I'll try to list the similarities as well as differences.

So we agree that a "node type" doesn't mean a generic "termperature node" but a very specific "my sketch for controlling an 18b20 on pin 7".

Let's suppose the sketch was called "DevDuino_18b20_7.ino" (for the moment let's leave out auto-scripting).
This compiled into DevDuino_18b20_7.hex and the binary equivalent.

We would know that nodes 7, 12, and 15 should have the latest version of this sketch. (node 4 might also measure temperature but with different hardware or pin configuration, so it would not use this sketch).

If we just want to update the sketch, we would send copies of the new binary to nodes 7, 12, and 15.

So far I think we are on nearly the same page, at the generic level described. Where the "push by node" and "pull by node type" models differ is in where the knowledge that nodes 7, 12, and 15 run the same code resides. In push-by-node, the Server knows that it should send the same code to them; in the "pull by node type" model, those three nodes themselves know they want updates for a given numeric "node type".

Also, each node knows it has a given version of the firmware of its node type, and decides when to upgrade by comparing that with what the server offers.

The differences are highlighted better when we make more than a version update.

Suppose we decide to make use of pin 2 of node 7 to control an LED. At this point we need to load different firmware into node 7, and we write a sketch called "DevDuino_18b20_7_LED_2.ino". (forgive the naming, it's an example). So now we want to change the overall system configuration so that the server will load DevDuino_18b20_7_LED_2.hex into node 7. (Nodes 12 and 15 still have the other sketch without LED control and maybe always will, the new sketch is not a new version of the old sketch)

I am suggesting that all you have to do is create the new sketch (or rather it's hex or binary compiled form) and configure the server to send that to node 7 instead of the previous sketch. It doesn't matter one bit to the node whether it's switching to a completely new sketch versus a new version of the current sketch. That's in the server logic only, not OTA.

I think you are saying that your node.js server can accomodate changing what sketch (not just version) runs in each node, because it can load arbitrarily different (or identical) binary files to each node by node id, with no limitations based on what "type" that node used to be, right? Or not?

In my thinking, the node has no need to know that it's running "node type 453, version 16".

The server is free to conceptually organize firmwares by "node type" and "version" if it wishes, but those concepts do not need to be pushed down to the node level.

Pull by node type:

 Node: I'm type 453, what is the latest version of 453?
 Server: Queries database for max version of node type 453, and say "Latest 453 version is 17"
 Node: Checks that it has version 16, asks server to send 453 ver 17 for OTA programming
 Server: sends version requested by node
 (programming done)
 Node: ask server for lastest version for node type 453
 Server: ... 17
 Node: I'm version 17, no change needed

Push by node:

Node: I'm node 7 and my PROGMEM signature if 0x54FE
Server: My config says node 7 should have the latest DevDuino_18b120_7_LED_7.hex" with Sig 0x3EE5
Server: Please load the following binary into your PROGMEM (sends appropriate version)
(programming done)
Node: I'm node 7 and my PROGMEM sig is 0x3EE5
Server: Mark that one as updated

Notice that there's no problem of forgetting to update the version number - if the signature (eg: CRC or hash) in PROGMEM isn't what the server wants there, then it starts an OTA programming session, period. Even if the sig was wrong because the programming had a glitch rather than because it is out of date, the server knows it's wrong and sends again.

In the push-by-node model, the system is not limited to "updating all nodes with node type 453 versions <= 16 to version 17" -- it can load an entirely different sketch (node type 763 version 0 if the server thinks in those terms) into the node if it wishes. And the node doesn't care, it doesn't need to know what "node type" it is or compare versions (that's in the logic of the server), all the node needs to know is that the server wants it to load some code into PROGMEM, period.

That's how I imagine things working. As I have understood you, you are pushing the concept of "node type" and "version number" and the comparison of version numbers down to the node itself, rather than letting the server handle that (if it chooses). I don't see the advantage of that; the server seems a more logical place for that information - both simpler and more flexible.

Your node.js implementation might organize source code for nodes by numeric "node type" (where your node type 7 isn't the same as my node type 7) and by version within node type. That would still be supported by "push by node" model.

But another server might choose to organize firmwares by string filename plus signature (eg; CRC). It's configured with a simple table of node_id,filename. It computes the signature of the binary file to be compared with that the node reports it has in PROGMEM. There's no numeric "node type" or "version number" needed. (If you really want to also keep obsolete versions of the binary firmware on the server, there are easy workarounds fore that too). The config is dead simple: (NodeID, Filename)* Or optionally (nodelist, Filename)* if you want to reduce the number of times you spell out the filename. Use the latest (or only) copy of the given filename.

The cool thing about the "push by node" approach is that the same node bootloader can easily accomodate both server approaches (node type # + version # OR nodeID->filename) - since the only concepts the node uses are

"I can tell you the signature of what I have in PROGMEM now", and
"If you tell me to I'll load something of your choice".

For OTA bootloading the node doesn't know need to know or care about "node type" numbers or versions, nor about file names or date stamps.

So I'm not trying to eliminate your concept of the server of assigning numeric ID's to each combination of sensors and pins, and using ordered versions numbers within each "version type". I just don't see why those concepts need to also be pushed down into the node and OTA bootloader protocol. With the "push by node" model, there''s more flexibility to organize changing firmwares in the server as you prefer OR in other ways, with no meaningful cost, because the node end of the OTA programming system has been distilled to just the essence that it really needs to understand, leaving higher levels of management as a server-internal affair.

Damme

@ToSa I've been looking through the ota bootloader and noticed there are alot of uint16_t wich can be replaced with uint8_t.. saves 128bytes of code. Still needs ~900bytes less until 1024 words bootloader though but is makes more space for other stuff :)

JeJ

When i'm trying to run the NodeJsController.js script i always end up with "Error: Cannot open /dev/ttyAMA0"

I'm running a RPi with a serial gateway.

Any ideas?

mikeones

On my RPi, the serial gateway is detected as dev/ttyUSB0.

Damme

@ToSa I've been working on getting OTA to work with MQTTgateway with some success.

But I do have problem with some packages missing and I think the communication should be something like this;
bootloader checks id and version and server said there is an update. (no change from today)
but then:

[bootloader] 0000 has CHK FF(just filler in first package) REQ 0000 type 01 version 01
[server] load 0000 from hex, send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK FF, REQ 0010 type 01 version 01
[server] (checksum mismatch) send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK C7, REQ 0010 type 01 version 01
[server] load 0010 from hex send addr 0010 0C94A3050C94D0050C9480100C945003 00
And so on.. :)

what do you think about this? the total package is 32bytes, mysensors header is 7bytes. and this layout would need 19 bytes from server to bootloader..
I Have also seen some intel hex that is not in order 0010 0020 0030 etc but it could jump address. I do not think arduino ide does this but you never know..

EDIT:
I havn't read this one yet but I guess there is alot of good stuff in it :)
http://www.nordicsemi.com/eng/nordic/download_resource/10878/2/94069421

ToSa

@JeJ @mikeones : how is your serial gateway connected? using a USB-Rs232 cable or via the GPIO pins on the RPi? Did you check the Readme.md in the NodeJsController directory?

Zeph

@Damme said:

what do you think about this? the total package is 32bytes, mysensors header is 7bytes. and this layout would need 19 bytes from server to bootloader..

16 bit offset, 16 data bytes, one byte checksum, right?

I Have also seen some intel hex that is not in order 0010 0020 0030 etc but it could jump address. I do not think arduino ide does this but you never know..

I see that your descriptions say "0010 from hex" etc, but I thought you would be fetching from a binary blob to satisfy requests from the bootloader. As in:

Server reads the Intel hex and uses it to fill in an array of bytes. (one time, or each time a given file is requested)
Server sends requested 16 byte chunks of that array to bootloader

In that case, it doesn't matter what order the original hex lines are in, or even if they are 16 or 32 bytes wide (or less than 16 bytes at the end).

Damme

@Zeph true (array) and Yes, so the node can request same address twice (might be a timeout) and verify checksum on every 16byte data.

ToSa

@Damme said:

@ToSa I've been working on getting OTA to work with MQTTgateway with some success.

great!

[bootloader] 0000 has CHK FF(just filler in first package) REQ 0000 type 01 version 01
[server] load 0000 from hex, send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK FF, REQ 0010 type 01 version 01
[server] (checksum mismatch) send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK C7, REQ 0010 type 01 version 01
[server] load 0010 from hex send addr 0010 0C94A3050C94D0050C9480100C945003 00
And so on.. :)

what do you think about this? the total package is 32bytes, mysensors header is 7bytes. and this layout would need 19 bytes from server to bootloader..

If I understand correctly you would send the CRC of the previous bloak back to the server together with the request for the next package - the server would then send the next block if the CRC is correct or resend the previous block if the CRC is not ok...
I'm wondering how the bootloader would ever run into that situation. The package itself is checksum'ed already and wouldn't be treated as correctly received package if the checksum is incorrect. Only if the previous block was received correctly the next block is requested. If that doesn't happen within a given amount of time the same block is requested again.
Can you explain a bit further what issues you are running into?

@Damme said:

I Have also seen some intel hex that is not in order 0010 0020 0030 etc but it could jump address. I do not think arduino ide does this but you never know..

Yes, I've seen that as well - actually Codebender does that sometimes and the "HEX file loader" function on the NodeJsController would need to be adjusted to address that (I'm not reading from the .hex file when a package arrives but from a byte array in Mongo)
Probably the best approach for this one would be to add the next address the bootloader should request to the previous package (e.g. "here is the data for 0x0000 - next request should be for 0x0010) and allow for variable block length because that's another one where Codebender sometimes uses <16 byte rows in the hex (not just at the end). The code on the bootloader side would get a little more complex as the flash is still written in pages and these might not line-up anymore with the 16byte blocks.

@Damme said:

I havn't read this one yet but I guess there is alot of good stuff in it :)
http://www.nordicsemi.com/eng/nordic/download_resource/10878/2/94069421

That chip is a totally different animal - same RF but 8051 MCU. I'll have a look at the AppNote tomorrow.

ToSa

@Damme said:

@ToSa I've been looking through the ota bootloader and noticed there are alot of uint16_t wich can be replaced with uint8_t.. saves 128bytes of code. Still needs ~900bytes less until 1024 words bootloader though but is makes more space for other stuff :)

I'll have a look. I've taken the code from an earlier project and adjusted to MySensors - didn't review the variable types that much. I'm using CRC16 as well where CRC8 might be sufficient...

EDIT: got it down a little from 0x0E18 to 0x0DD0 (72 bytes) changing a few loop counters from uint16 to uint8. I don't want to change type to 8bit looking at the large amount of sensors people are asking for / working on. FOr version I'm planning to keep some of these running for a long time with as little maintenance as possible. With some software improvements over time and minor version changes during development 16bit for version seems to be the better fit as well.

mikeones

@ToSa I use a Mini-B USB cable between my PRi and my gateway.

ToSa

@mikeones said:

dev/ttyUSB0

Then /dev/ttyUSB0 is correct. /dev/ttyAMA0 would only be valid for the on-board serial port on the GPIO pinheads.
Are you running into any issues once you set the port in NodeJsGateway accordingly?

JeJ

@ToSa I have my gateway connected via the GPIO and i have followed the steps in the Readme.md.
I will try to use a USB-Rs232 cable and see what happens.

ToSa

@JeJ one potential reason is that the port is already in use. I mentioned somewhere that the startup script doesn't yet stop the NodeJsController correctly. Maybe you already have a NodeJsController process running? Try "sudo killall node" and then try starting it again. To check if the port itself is working you can try to open a simple terminal (minicom etc.) and reset the gateway.

Zeph

@ToSa said:

With some software improvements over time and minor version changes during development 16bit for version seems to be the better fit as well.

Hmm. That seems like overkill, if I'm understanding correctly. (So maybe I am not understanding).

What I heard was:

Each sensactuator node has a "node type" and a "version" within that node type. Each combination of sensors and pin assignments has a unique "node type" (within a given wireless network). A node can only be OTA updated to a newer (higher) version of the same "node type" of the current firmware, and all nodes of that "node type" will be updated.

And extra byte for "version" isn't a big deal tho.

Will there be one or two bytes for "node type"?

Over the air updates

16

11.7k

11.2k

113.1k