MySensors protocol format
-
Lately we've had some discussion on this forum about the MySensors protocol format.
Everything is stuffed into a single fixed header now (in 1.3 and 1.4b) and not all fields are used with all message types.
This leads to unused data being transfered over the air and complicates protocol extensions. Furthermore discussion is ongoing of migrating MySensors to the RadioHead library (http://www.airspayce.com/mikem/arduino/RadioHead/index.html) which will replace the routing and allow reliable datagram transmission.@hek 'challenged' me to suggest a possible breakdown of the protocol, so here it is
I think that in this discussion protocol diagram charts can help a lot. I searched the internet for some online graphical editor, and this http://interactive.blockdiag.com/packetdiag/ seemed to be the best solution. It doesn't allow collaborative editing, but at least allows others to edit the diagrams easily and share on this forum. I'll paste a link with every diagram which can be used to edit it. Afterwards you'll have to paste the generated diagram by hand.
Ok, let's start with the current 1.4b MySensors protocol header (numbered as 2 in the header) as a reference:
The divisions at the top represent bits (numbered 0..7) and each row represents a single byte. Payload has a variable size and is therefore shown as one big block of data.
I would break this header down as follows:
The network header:
It contains all the routing information of a packet. The ack-flags are used for 'reliable' transmission and it contains the length of the payload (the actual MySensors data) following.
This header will completely be replaced by the RadioHead library, when implemented.The MySensors header:
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwMuoJCBrrmVQkFRfkk-ULtCWWpRcWZ-HlDcQtfQ1ApoZG5uYl4KVy0AXHIchA
Very basic header: protocol version (for future compatibility -- we could even leave this out) and command. Each command defines a specific structure following this header.Presentation command header:
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwMuoJCBrrmVQnFqXnF-kYKnC1DAQtfQFC5SUlmQChQzNNM1MrZSKEstKs7Mz1MoLinKzEtXiAZaCDffOJarFgAQaydpUse (during sensor startup) to tell the gateway/controller the types of sensor(s) available at a node. Sensor ID is also called child ID, sensor type is one of S_DOOR, S_MOTION, S_SMOKE etc.
Version string is the same as currently sent by the sensor. Not sure why we need a version string...** The Set & Req command header**:
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwOgCBAZ6JpbKZQl5pSmKpRUFqQCRSx0DU1hQtFAU-GGGMcqcNVyAQCe1h7SThe set & request commands are identical, only the direction of the data differs, so they share the same header structure.
Stores the value type (e.g. V_TEMP,V_HUM, V_LIGHT, etc) and the value itself (variable amount of bytes).As an example, sending a sensor value to the gateway now takes the MySensors header (4 bytes) followed by the Set command (1 byte + data) giving a total of 5 bytes header. With 1.4b this takes 7 bytes.
Some open items:
- C_STREAM command: currently a separate command, but I think this could just as well by a value type sent using GET/SET. If not we need to define it.
- C_INTERNAL command: mainly used for routing and battery level reporting. I think battery level can be sent just as a regular a sensor value. Should routing be done by RadioHead the routing related internal messages are no longer required. For the ones remaining we need to define them.
- I did not pay much attention to the size of certain fields. Some can be bit-stuffed to reduce the data size.
Please be gentle with me
and let's have a fruitful discussion!
-
@Yveaux looks good, protocol version and command could be in the same byte Imo. And internal could have more uses in future. I only use reboot
(which can be coded in sensor..) Battery I send as custom sensor
-
@Damme said:
looks good, protocol version and command could be in the same byte Imo.
There are a couple of ways to merge them into one byte (or any fixed length). One is to allocate a fixed N bits for protocol and 8-N bits for command. (Actually that's the current proposal with 8 + 8 in two bytes, but it could be 2+6 in one byte, etc)
Another way more flexibly partitiions the space: use commands (0 to N1) for protocol 0, (N1+1 to N2) for protocol 2, etc. (Actually there might be some universal commands at the start, available in all protocols, like REBOOT).
Compare this to 2+6 bit fixed partitioning. If in protocol 0 you only defined 9 commands before you went to protocol 1, you wouldn't be tossing away 64 of the 256 possible values, only 9 of them. A range compare to determine whether you (a node) understand the current protocol is quick and easy. For example, rather than testing for (byte & 0xC0 == 0x40) you can test for (byte >= 9 && byte <=15), if you only understand commands 9..15.
This can even support incremental growth within a protocol. Suppose that while the second protocol was still evolving, you add a new 8th command to the current 7 commands with range (9..15), perhaps for a crypto security function for door locks. You do not need to "bump up" the protocol version and force all nodes to use a new range of (16-23), you can just add command 16 so the full current protocol ranges is now (9..16). .Those nodes which don't need this function still accept commands (9..15), which is all they need. You only need to "discard" command codes and start a new range when you go to a new and completely incompatible protocol. (Perhaps you can query for the range supported by a node).
If you ever start getting close to using up the byte, you can use two byte codes, reserving for example codes 0xF0-0xFF to hold the top 4 bits of a 12 bit command range (the following byte holding the low 8 bits). In all likelyhood this won't actually be needed, before you get that far many years down the road, we'll probably replace this with something radically different anyway, but nevertheless you don't have to worry about being trapped in 8 bits. In fact, since old nodes do a >= && <= test and reject any out of range command, you don't even have to decide in advance how many code values at the top of (0..255) to reserve for the extended command codes (ie: you could use 0xFC..0xFF instead). Only nodes written after you make that decision need to care.
-
@Yveaux said:
Very basic header: protocol version (for future compatibility -- we could even leave this out) and command. Each command defines a specific structure following this header.
Presentation command header:
Maybe you could call the "XXXX command header" the "XXXX command payload".
If I understand your proposal as written, there are 3 "headers" (network, mysensors and then command specific) and no payload.
-
@Zeph sure, it's more than just a header as I included the data too.
-
One other thought. How would this interact with the OTA programming facility?
-
@Zeph regarding protocol version & command partitioning; another possibility is to only store a command and no version. If later on a modified version of command x Is needed we simply introduce a new command x2 and carry on. All other commands stay unchanged.
-
@Zeph said:
One other thought. How would this interact with the OTA programming facility?
Protocol support has to be changed to the new format. You already have the same issue with 1.3 and 1.4b. This is one of the reasons I want to have this discussion asap.
-
@Yveaux said:
Protocol support has to be changed to the new format. You already have the same issue with 1.3 and 1.4b. This is one of the reasons I want to have this discussion asap.
I am agreeing, just wondering about the specifics. Like: what would the packets for OTA programming look like in this new format.
I like how you are separating the two layers with two headers, with the idea that part of it would be replaced when using RadioHead, and part of it would be the same (plus the payload). I'm just trying to see the implications.
One of the issues is that, as I understand it, the OTA bootloader would include a trimmed down version of the RF24 library and a limited version of the MySensors protocol within it's limited space. If we are using RadioHead as the transport medium instead of RF24 + MySensors network layer, how does that work?
-
@Zeph I would need to re-write (or at least adjust) that part of the bootloader. Would need to see what radiohead does to the header before I can judge if it's still possible to fit it into the limited bootloader space but at this point I'm pretty positive... current bootloader size is 3608 of 4096 bytes and maybe there is still potential to reduce it further. I was planning to use the remaining size for encryption though which might or might not be feasible with 400 bytes only...
The bootloader would never use the radiohead library due to its size but would mimic the header format. If you are looking for multiple radio modules, the effort to reduce the "driver" to fit into the bootloader section would be unique for each radio module type
-
@ToSa I personally wouldn't have too much troubles if the bootloader runs its own protocol, e.g. the current MySensors implementation. If it communicates using a different channel compared to your regular MySensors channel it can easily co-exist. You will require a separate gateway (and maybe router(s)) to support the bootloader, however.
But this solution all depends on whether this is acceptible to your application...
-
@Yveaux The main purpose of the OTA bootloader is to avoid the need to grab the node and connect it to a PC for an update but instead do the update "in-place". Using the existing "infrastructure" to relay messages is a must as we don't want to implement separate code in each relaying node to handle the bootloader messages separately.
-
@ToSa I understand, but I clearly stated is was my personal opinion
-
@Yveaux
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJx9kctOwzAQRff9ilm2labyI69aYlHRLFiURxuxRaZx24jUKbELVMC_MwkNikBi5znXYx-P3wcA66p8LXK_gwtIqLRVbh52ptjuPBERdqjbE7ABIYaxglI7D0OGIgxHxBLkoQJnbG7qHuYRCqkgN84XVvuisr1QBAoKp9dPzZq6a_N8LiKUXMGBKuOhNHZLt1NfJJs2KVBGCl5M7c7nSd7yGOVU0ZP2e23zhvP2moBhQMJDZw661r6qRzAGuLxZLGbXcwUMlukdfACHVZoBjJuOBEPyWRnrqvpq3lMOI5Lokux0ML0sCjAm63tdHs2vKBYYk9qtPpWVJrXJZEJTFBIeT964dn4Mk38kBayyZTpbfPslCU6FgvTN1xqcP242BKcSOZM_U7PHfSvAgrj9CBYg5_yvQ9g5fH4BuWqGBgI wanted to write some text but need to run!
-
@Damme OK, then RUN!
I'm interested to read your accompanying text....
Please also export the diagram as SVG and include it in the discussion....
-
@Yveaux
not much to say, there is room for 5 extra commands before needing to change protocol version. but I haven't put much thought in future backward compatibility.. the stream art I think OTA developer has to take a look at. I haven't had time to learn that yet. but the more space for payload the better. I have tested stream to send a small image once. but now then I think about it I don't care if the package is set or req.
-
@Yveaux said:
It contains all the routing information of a packet. The ack-flags are used for 'reliable' transmission and it contains the length of the payload (the actual MySensors data) following.
This header will completely be replaced by the RadioHead library, when implemented.Th ack flags is actually not replaced by the radiohead library (it only takes care of acking the first hop). We still need to implement this part in the MySensors library.
Also note you've increased payload-length field. This is only interesting if we use radios with supporting payloads larger than 32 bytes. So the question is if we just should allow the least common divisor here or not.
Here is a proposal reflecting the discussions in the other thread. The datatype gets one extra bit (its crowded now).
-
@hek My 5 cents (more like 2 Euros...):
- Is there a reason why you draw this as one monotonic packet? Don't you like the idea of nested structures?
- When using nested structures and defining commands, a command number has to be stored to indicate the data following the header. This command can differentiate between set-command and req-command, hence the set/request bit is not required.
- The payload length should be part of the network header. In fact, the payload length shouldn't have to be stored at all as the nRF24L01+ already stores it in its header. For radio's which don't pass it in their native header (or don't have a native header at all) it can be added to the network header.
- I would like to get rid of the whole isack/reqack idea... I think it should be handled by the network layer, when enabled, either between two nodes or end-to-end. It should be a configuration of the network connection and should operate at a lower level then MySensors. If RadioHead doesn't offer end-to-end (which I think it does; have a look at RHReliableDatagram) then a separate layer between MySensors and RadioHead should handle this. A solid communications library should provide send-and-forget methods; a user (sensor node) should not have to keep track whether its message got delivered or not. Btw. RadioHead uses message ID's to recognize when an ack to a message arrives.
- The encrypted-flag might need more states than just yes/no -- but that's for the future. Good to mention it here, though.
- Encryption, like end-to-end acknowledgement or (de)fragmentation, should also be handled on a lower level. Maybe move it to the network header?
-
@Yveaux said:
Don't you like the idea of nested structures?
Yes, I do. Hmm.. Ok... it might be good to keep the command-filed for future things we might not have anticipated. Agree on keeping it.
Actually thought some time about skipping the length field also. The length information could be handled differently in the library. But what should we do with the 5 bits that frees up?
I totally missed @Damme post a couple of hours ago where he combined command and version in one byte. Might be a good thing. This would allow 16 different commands and versions.
And yes, the best thing would be if the transportation layer would handle end-to-end acking.
-
So... how about something like this packed?
typedef struct { uint8_t last; // 8 bit - Id of last node this message passed uint8_t sender; // 8 bit - Id of sender node (origin) uint8_t destination; // 8 bit - Id of destination node uint8_t ack; // 1 bit - Request an ack - Indicator that receiver should send an ack back. // 1 bit - Is ack messsage - Indicator that this is the actual ack message. // 6 bit - Reserved for future use. Perhaps encryption? char header[MAX_MESSAGE_LENGTH-4]; } transportation; typedef struct { uint8_t command; // 8 bit - Command type char commandPayload[MAX_MESSAGE_LENGTH - sizeof(transportation) - 1]; } header; typedef struct { uint8_t sensorId; // 8 bit - child sensor id uint8_t sensorType; // 8 bit - sensor type uint8_t valueType_dataType; // 5 bit - value type // 3 bit - data type union { uint8_t bValue; unsigned long ulValue; long lValue; unsigned int uiValue; int iValue; struct { float fValue; uint8_t fPrecision; // Number of decimals when serializing }; char data[MAX_MESSAGE_LENGTH - sizeof(header) - 4]; }; } commandSetReq;But this would only allow 32 different valueTypes per sensorType. But it should be enough. The worst case "Node" would require 22 unique values including the 10 V_CONFIG/V_VAR.
-
@hek said:
version_command
Almost there
- I think we could drop the protocol version number and use a full byte for command type.
When, in the future, a command gets extended we can just reserve a new command-number for it. All existing applications will only recognize the old command-number and ignore the new one. As @zeph already pointed out this allows for more efficient usage of the 8 bits and a new version will not immediately disqualify an onld node completely as it will stil listen & respond to old commands. - It might be better to not include the remaining part of the message in a structure (e.g. header[MAX_MESSAGE_LENGTH-4], commandPayload[MAX_MESSAGE_LENGTH - sizeof(transportation) - 1] as this will make it very inefficient to store single instances of it (e.g. declaring a local instance of transportation or header will always clame a full message size)
- As a second though, it might be best to declare everything as union's, so the structures will overlap eachother (will become one badass strusture then....)
- I think I'll dive a bit deeper into the RadioHead library to understand what is already possible with it and to be able to exchange some ideas with @kolaf
Edit @hek you were right about the RadioHead library:
RHRouter (and therefore RHMesh) use reliable hop-to-hop delivery of messages using hop-to-hop acknowledgements, but not end-to-end acknowledgements.
- I think we could drop the protocol version number and use a full byte for command type.
-
We should also consider interoperability with the the MQTT delivery concepts of "at most once", "at least once" and "exactly once" so the interface is smooth.
There are 4 dataflows where this interoperability could might be at least considered, I think.
controller to MQTT MQTT to node node to MQTT MQTT to controller
-
I think it's important that if we use structs and unions here, we understand that this is just a convenience for discussion, not the specification. For discussion we can assume that the structs will have 1 byte alignment for themselves and for all members, and will use little endianness.
What really matters to the spec is that the bytes in the order they go OTA be exactly the same for all senders and all receivers.
Those bytes might be interpreted in place as a struct (via static cast of a pointer into a byte buffer for example) IN SOME CASES, depending on CPU and on compiler and its options. In other cases it might be necessary to pack and unpack dynamically to avoid alignment and endianness issues.
Just a caveat to watch out for.
-
@Yveaux said:
- I think we could drop the protocol version number and use a full byte for command type.
When, in the future, a command gets extended we can just reserve a new command-number for it. All existing applications will only recognize the old command-number and ignore the new one. As @zeph already pointed out this allows for more efficient usage of the 8 bits and a new version will not immediately disqualify an onld node completely as it will stil listen & respond to old commands.
The value of having defined groups of command associated with a given protocol version is that you can get a pretty good idea of what a node will understand (or send). The first approach was to have some fixed number of bits for a protocol id, and some number for command-within-protocol (like 8+8 or 5+3 or whatever). I'm glad you see the advantages of adding command codes sequentially instead, especially if we are using one byte.
I was suggesting that a given protocol version define a contiguous range of commands (plus some universal ones at the beginning of the codes). One caveat was that the "current protocol version" would be allowed to add new commands at the end of the sequence - the "last command allowed in this version" doesn't get frozen until a version is superceded by another one. Any given node need only understand the universal commands plus the range corresponding to the protocol version it was written to. So each version has the common universal commands plus a non-overlapping range of command specific to this version.
The disadvantage is that you may have to redefine new command codes for commonly needed commands like "set value" every time you move to the next version. So in the first version, command 4 might be set (part of range 3..9). Then in the next version with commands (10..17) we have to define a new "set value" within that range, maybe code 11. Nodes implementing the second version only understand commands 0..2 (universal) and 10..17, they have no need to understand version one commands 3..9. A gateway would not expect a version 2 node to understand command 6. The hub could ask the node for the start and end of the range of commands it can accept.
===
One variant, which you are suggesting I think, is to ignore versoin based commadn code sub-ranges and just keep all commands allowable to all nodes. This has the advantage that "Set value" command 4 (in the above example) can continue to be used "forever", and not need to be reallocated code 11 to fit within the new version's range.
But it may over time cause fragmentation or bloat - either we carry around the code to handle all commands (eg: 1.3 and 1.4 both), or the set of commands allowed once you switch to a new protocol version is non-contiguous (it contains some but not all commands from the older protocol version). I actually considered this, and the possibility of having the node respond to "what commands can you handle" with a bitmap instead of a start and end value, but decided that was getting too complicated. That was just my judgement call tho.
(If you can correctly anticpate this kind of command in the "universal commands" that are part of all versions, the issue goes away. In the first appraoch, maybe "set value" is command 2 and globally available, not needing to be allocated a code in each versions's code range) But sometimes your concept changes more than you anticipated and "set value" might need to become a new command type in newer protocols).
Deciding among these options depends somewhat on one's philosophy of where configuration information should be stored - in the nodes or in the hub. There are many answers to this question at different granularities. If you accept the concept of storing much config on the hub, then another approach emerges:
- each node can use a basic common protocol to report on first connecting what version of the full protocol it implements
- that per-node protocol version is stored in the hub
- the hub has bitmaps or tables of what command codes work for each protocol version it understands
- if the hub understands more than one version, it can communicate with nodes using any of those versions
In this case, you don't make every packet include an explicit (fixed bits allocated) or implicit (code range) protocol version. But the hub could still know which commands work and don't work for any given node.
- I think we could drop the protocol version number and use a full byte for command type.
