MySensors protocol format

[Yveaux]

Lately we've had some discussion on this forum about the MySensors protocol format.
Everything is stuffed into a single fixed header now (in 1.3 and 1.4b) and not all fields are used with all message types.
This leads to unused data being transfered over the air and complicates protocol extensions. Furthermore discussion is ongoing of migrating MySensors to the RadioHead library (http://www.airspayce.com/mikem/arduino/RadioHead/index.html) which will replace the routing and allow reliable datagram transmission.

@hek 'challenged' me to suggest a possible breakdown of the protocol, so here it is ;-)

I think that in this discussion protocol diagram charts can help a lot. I searched the internet for some online graphical editor, and this http://interactive.blockdiag.com/packetdiag/ seemed to be the best solution. It doesn't allow collaborative editing, but at least allows others to edit the diagrams easily and share on this forum. I'll paste a link with every diagram which can be used to edit it. Afterwards you'll have to paste the generated diagram by hand.

Ok, let's start with the current 1.4b MySensors protocol header (numbered as 2 in the header) as a reference:

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJxFj90KwjAMhe_3FHmBwNZ2PxZ8EhEpa3DF2c41KEN8dzN0epfznZBz8iwA-jQ-gucB9tCJjMnTaaBwHliIqje07ZiyEFRia2F0mWXusKotZIqeZpFVg0pb8JQ5RMchRYHKoGosTHPiJIFwpzl_nRZ1JY5bxuQ8jBTPPAjXCrWx0F898DLRSiRlppvrL6uQayF_Zzmxk0THbts1JRppuKkO60_FnNaKdYON_mcepNHvY30sXm8Ork7E

The divisions at the top represent bits (numbered 0..7) and each row represents a single byte. Payload has a variable size and is therefore shown as one big block of data.

I would break this header down as follows:

The network header:

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJw1ilsKwyAQAP89xV5AiObRIPQsRbJLXCraxoVQSu_eDSR_M8N8DcBS884oCe4wq5aK9EjEaxItfrzS9Qyd0dTZW4AcmyjP1o0BGhWkTdVN1vcBkJpwicK1aPRDAG5xeR6s90bvUybbuwCv-Mk1ImQqqyTz-wPilStB

It contains all the routing information of a packet. The ack-flags are used for 'reliable' transmission and it contains the length of the payload (the actual MySensors data) following.
This header will completely be replaced by the RadioHead library, when implemented.

The MySensors header:

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwMuoJCBrrmVQkFRfkk-ULtCWWpRcWZ-HlDcQtfQ1ApoZG5uYl4KVy0AXHIchA
Very basic header: protocol version (for future compatibility -- we could even leave this out) and command. Each command defines a specific structure following this header.

Presentation command header:

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwMuoJCBrrmVQnFqXnF-kYKnC1DAQtfQFC5SUlmQChQzNNM1MrZSKEstKs7Mz1MoLinKzEtXiAZaCDffOJarFgAQaydp

Use (during sensor startup) to tell the gateway/controller the types of sensor(s) available at a node. Sensor ID is also called child ID, sensor type is one of S_DOOR, S_MOTION, S_SMOKE etc.
Version string is the same as currently sent by the sensor. Not sure why we need a version string...

** The Set & Req command header**:

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJyr5lJQSM7PKc9MKclQsFWwAHLz8lNS4zNSM9MzSoAiRqYwIZgaEwOgCBAZ6JpbKZQl5pSmKpRUFqQCRSx0DU1hQtFAU-GGGMcqcNVyAQCe1h7S

The set & request commands are identical, only the direction of the data differs, so they share the same header structure.
Stores the value type (e.g. V_TEMP,V_HUM, V_LIGHT, etc) and the value itself (variable amount of bytes).

As an example, sending a sensor value to the gateway now takes the MySensors header (4 bytes) followed by the Set command (1 byte + data) giving a total of 5 bytes header. With 1.4b this takes 7 bytes.

Some open items:

• C_STREAM command: currently a separate command, but I think this could just as well by a value type sent using GET/SET. If not we need to define it.
• C_INTERNAL command: mainly used for routing and battery level reporting. I think battery level can be sent just as a regular a sensor value. Should routing be done by RadioHead the routing related internal messages are no longer required. For the ones remaining we need to define them.
• I did not pay much attention to the size of certain fields. Some can be bit-stuffed to reduce the data size.

Please be gentle with me ;-) and let's have a fruitful discussion!

[Damme]

@Yveaux looks good, protocol version and command could be in the same byte Imo. And internal could have more uses in future. I only use reboot :) (which can be coded in sensor..) Battery I send as custom sensor

[Zeph]

@Damme said:

looks good, protocol version and command could be in the same byte Imo.

There are a couple of ways to merge them into one byte (or any fixed length). One is to allocate a fixed N bits for protocol and 8-N bits for command. (Actually that's the current proposal with 8 + 8 in two bytes, but it could be 2+6 in one byte, etc)

Another way more flexibly partitiions the space: use commands (0 to N1) for protocol 0, (N1+1 to N2) for protocol 2, etc. (Actually there might be some universal commands at the start, available in all protocols, like REBOOT).

Compare this to 2+6 bit fixed partitioning. If in protocol 0 you only defined 9 commands before you went to protocol 1, you wouldn't be tossing away 64 of the 256 possible values, only 9 of them. A range compare to determine whether you (a node) understand the current protocol is quick and easy. For example, rather than testing for (byte & 0xC0 == 0x40) you can test for (byte >= 9 && byte <=15), if you only understand commands 9..15.

This can even support incremental growth within a protocol. Suppose that while the second protocol was still evolving, you add a new 8th command to the current 7 commands with range (9..15), perhaps for a crypto security function for door locks. You do not need to "bump up" the protocol version and force all nodes to use a new range of (16-23), you can just add command 16 so the full current protocol ranges is now (9..16). .Those nodes which don't need this function still accept commands (9..15), which is all they need. You only need to "discard" command codes and start a new range when you go to a new and completely incompatible protocol. (Perhaps you can query for the range supported by a node).

If you ever start getting close to using up the byte, you can use two byte codes, reserving for example codes 0xF0-0xFF to hold the top 4 bits of a 12 bit command range (the following byte holding the low 8 bits). In all likelyhood this won't actually be needed, before you get that far many years down the road, we'll probably replace this with something radically different anyway, but nevertheless you don't have to worry about being trapped in 8 bits. In fact, since old nodes do a >= && <= test and reject any out of range command, you don't even have to decide in advance how many code values at the top of (0..255) to reserve for the extended command codes (ie: you could use 0xFC..0xFF instead). Only nodes written after you make that decision need to care.

[Zeph]

@Yveaux said:

Very basic header: protocol version (for future compatibility -- we could even leave this out) and command. Each command defines a specific structure following this header.

Presentation command header:

Maybe you could call the "XXXX command header" the "XXXX command payload".

If I understand your proposal as written, there are 3 "headers" (network, mysensors and then command specific) and no payload.

[Yveaux]

@Zeph sure, it's more than just a header as I included the data too.

[Zeph]

One other thought. How would this interact with the OTA programming facility?

[Yveaux]

@Zeph regarding protocol version & command partitioning; another possibility is to only store a command and no version. If later on a modified version of command x Is needed we simply introduce a new command x2 and carry on. All other commands stay unchanged.

[Yveaux]

@Zeph said:

One other thought. How would this interact with the OTA programming facility?

Protocol support has to be changed to the new format. You already have the same issue with 1.3 and 1.4b. This is one of the reasons I want to have this discussion asap.

[Zeph]

@Yveaux said:

Protocol support has to be changed to the new format. You already have the same issue with 1.3 and 1.4b. This is one of the reasons I want to have this discussion asap.

I am agreeing, just wondering about the specifics. Like: what would the packets for OTA programming look like in this new format.

I like how you are separating the two layers with two headers, with the idea that part of it would be replaced when using RadioHead, and part of it would be the same (plus the payload). I'm just trying to see the implications.

One of the issues is that, as I understand it, the OTA bootloader would include a trimmed down version of the RF24 library and a limited version of the MySensors protocol within it's limited space. If we are using RadioHead as the transport medium instead of RF24 + MySensors network layer, how does that work?

[ToSa]

@Zeph I would need to re-write (or at least adjust) that part of the bootloader. Would need to see what radiohead does to the header before I can judge if it's still possible to fit it into the limited bootloader space but at this point I'm pretty positive... current bootloader size is 3608 of 4096 bytes and maybe there is still potential to reduce it further. I was planning to use the remaining size for encryption though which might or might not be feasible with 400 bytes only...

The bootloader would never use the radiohead library due to its size but would mimic the header format. If you are looking for multiple radio modules, the effort to reduce the "driver" to fit into the bootloader section would be unique for each radio module type :(

[Yveaux]

@ToSa I personally wouldn't have too much troubles if the bootloader runs its own protocol, e.g. the current MySensors implementation. If it communicates using a different channel compared to your regular MySensors channel it can easily co-exist. You will require a separate gateway (and maybe router(s)) to support the bootloader, however.
But this solution all depends on whether this is acceptible to your application...

[ToSa]

@Yveaux The main purpose of the OTA bootloader is to avoid the need to grab the node and connect it to a PC for an update but instead do the update "in-place". Using the existing "infrastructure" to relay messages is a must as we don't want to implement separate code in each relaying node to handle the bootloader messages separately.

[Yveaux]

@ToSa I understand, but I clearly stated is was my personal opinion :)

[Damme]

@Yveaux
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJx9kctOwzAQRff9ilm2labyI69aYlHRLFiURxuxRaZx24jUKbELVMC_MwkNikBi5znXYx-P3wcA66p8LXK_gwtIqLRVbh52ptjuPBERdqjbE7ABIYaxglI7D0OGIgxHxBLkoQJnbG7qHuYRCqkgN84XVvuisr1QBAoKp9dPzZq6a_N8LiKUXMGBKuOhNHZLt1NfJJs2KVBGCl5M7c7nSd7yGOVU0ZP2e23zhvP2moBhQMJDZw661r6qRzAGuLxZLGbXcwUMlukdfACHVZoBjJuOBEPyWRnrqvpq3lMOI5Lokux0ML0sCjAm63tdHs2vKBYYk9qtPpWVJrXJZEJTFBIeT964dn4Mk38kBayyZTpbfPslCU6FgvTN1xqcP242BKcSOZM_U7PHfSvAgrj9CBYg5_yvQ9g5fH4BuWqGBg

I wanted to write some text but need to run!

[Yveaux]

@Damme OK, then RUN! :racehorse:
I'm interested to read your accompanying text....
Please also export the diagram as SVG and include it in the discussion....

[Damme]

@Yveaux
not much to say, there is room for 5 extra commands before needing to change protocol version. but I haven't put much thought in future backward compatibility.. the stream art I think OTA developer has to take a look at. I haven't had time to learn that yet. but the more space for payload the better. I have tested stream to send a small image once. but now then I think about it I don't care if the package is set or req.

[hek]

@Yveaux said:

It contains all the routing information of a packet. The ack-flags are used for 'reliable' transmission and it contains the length of the payload (the actual MySensors data) following.
This header will completely be replaced by the RadioHead library, when implemented.

Th ack flags is actually not replaced by the radiohead library (it only takes care of acking the first hop). We still need to implement this part in the MySensors library.

Also note you've increased payload-length field. This is only interesting if we use radios with supporting payloads larger than 32 bytes. So the question is if we just should allow the least common divisor here or not.

Here is a proposal reflecting the discussions in the other thread. The datatype gets one extra bit (its crowded now).

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJxtkE9LxDAQxe_7Keaoh2D_pGk3ICJYEASFRfAgsoRm2AZDUpPsShG_uxNoF0Fze7_3mHmTrw3A4O2n0WmEa-hIOq9xP6I5jIlI1axozWyLDaGCtRIuHvvnl6fdA9z3t3f97hKsigmWR6GOlY2Ev7GITmNYU6VgVf3PMI0xGaeS8S7HKs4qIWEKPnmqDCcMkazstKwuyVGz9UqDRXdII_G6krQqXQX8ONKsTGjPokAN75lwCSaugtqiG8I8JdQ3GQhWbyVolRSkeUJCvGC8zXNd9AGG0VgNRueCvGNNc3aWeCOYoKUnZY-4MsFZ-6vvK11z_u_6bfP9A4PMcVE

[Yveaux]

@hek My 5 cents (more like 2 Euros...):

• Is there a reason why you draw this as one monotonic packet? Don't you like the idea of nested structures?
• When using nested structures and defining commands, a command number has to be stored to indicate the data following the header. This command can differentiate between set-command and req-command, hence the set/request bit is not required.
• The payload length should be part of the network header. In fact, the payload length shouldn't have to be stored at all as the nRF24L01+ already stores it in its header. For radio's which don't pass it in their native header (or don't have a native header at all) it can be added to the network header.
• I would like to get rid of the whole isack/reqack idea... I think it should be handled by the network layer, when enabled, either between two nodes or end-to-end. It should be a configuration of the network connection and should operate at a lower level then MySensors. If RadioHead doesn't offer end-to-end (which I think it does; have a look at RHReliableDatagram) then a separate layer between MySensors and RadioHead should handle this. A solid communications library should provide send-and-forget methods; a user (sensor node) should not have to keep track whether its message got delivered or not. Btw. RadioHead uses message ID's to recognize when an ack to a message arrives.
• The encrypted-flag might need more states than just yes/no -- but that's for the future. Good to mention it here, though.
• Encryption, like end-to-end acknowledgement or (de)fragmentation, should also be handled on a lower level. Maybe move it to the network header?

[hek]

@Yveaux said:

Don't you like the idea of nested structures?

Yes, I do. Hmm.. Ok... it might be good to keep the command-filed for future things we might not have anticipated. Agree on keeping it.

Actually thought some time about skipping the length field also. The length information could be handled differently in the library. But what should we do with the 5 bits that frees up? :)

I totally missed @Damme post a couple of hours ago where he combined command and version in one byte. Might be a good thing. This would allow 16 different commands and versions.

And yes, the best thing would be if the transportation layer would handle end-to-end acking.

[hek]

So... how about something like this packed?

typedef struct
{
	uint8_t last;            	 // 8 bit - Id of last node this message passed
	uint8_t sender;          	 // 8 bit - Id of sender node (origin)
	uint8_t destination;     	 // 8 bit - Id of destination node
	uint8_t ack; 	        	 // 1 bit - Request an ack - Indicator that receiver should send an ack back.
								  // 1 bit - Is ack messsage - Indicator that this is the actual ack message.
								  // 6 bit - Reserved for future use. Perhaps encryption?
	char header[MAX_MESSAGE_LENGTH-4];

} transportation;

typedef struct
{
	uint8_t command;     // 8 bit - Command type
	char commandPayload[MAX_MESSAGE_LENGTH - sizeof(transportation) - 1];
} header;

typedef struct
{
	uint8_t sensorId;            // 8 bit - child sensor id
	uint8_t sensorType; 		 // 8 bit - sensor type
	uint8_t valueType_dataType;  // 5 bit - value type
	                             // 3 bit - data type

	union {
		uint8_t bValue;
		unsigned long ulValue;
		long lValue;
		unsigned int uiValue;
		int iValue;
		struct { 
			float fValue;
			uint8_t fPrecision;   // Number of decimals when serializing
		};
		char data[MAX_MESSAGE_LENGTH - sizeof(header) - 4];
	};

} commandSetReq;

But this would only allow 32 different valueTypes per sensorType. But it should be enough. The worst case "Node" would require 22 unique values including the 10 V_CONFIG/V_VAR.