MySensors protocol format

[Yveaux]

@ToSa I understand, but I clearly stated is was my personal opinion :)

[Damme]

@Yveaux
http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJx9kctOwzAQRff9ilm2labyI69aYlHRLFiURxuxRaZx24jUKbELVMC_MwkNikBi5znXYx-P3wcA66p8LXK_gwtIqLRVbh52ptjuPBERdqjbE7ABIYaxglI7D0OGIgxHxBLkoQJnbG7qHuYRCqkgN84XVvuisr1QBAoKp9dPzZq6a_N8LiKUXMGBKuOhNHZLt1NfJJs2KVBGCl5M7c7nSd7yGOVU0ZP2e23zhvP2moBhQMJDZw661r6qRzAGuLxZLGbXcwUMlukdfACHVZoBjJuOBEPyWRnrqvpq3lMOI5Lokux0ML0sCjAm63tdHs2vKBYYk9qtPpWVJrXJZEJTFBIeT964dn4Mk38kBayyZTpbfPslCU6FgvTN1xqcP242BKcSOZM_U7PHfSvAgrj9CBYg5_yvQ9g5fH4BuWqGBg

I wanted to write some text but need to run!

[Yveaux]

@Damme OK, then RUN! :racehorse:
I'm interested to read your accompanying text....
Please also export the diagram as SVG and include it in the discussion....

[Damme]

@Yveaux
not much to say, there is room for 5 extra commands before needing to change protocol version. but I haven't put much thought in future backward compatibility.. the stream art I think OTA developer has to take a look at. I haven't had time to learn that yet. but the more space for payload the better. I have tested stream to send a small image once. but now then I think about it I don't care if the package is set or req.

[hek]

@Yveaux said:

It contains all the routing information of a packet. The ack-flags are used for 'reliable' transmission and it contains the length of the payload (the actual MySensors data) following.
This header will completely be replaced by the RadioHead library, when implemented.

Th ack flags is actually not replaced by the radiohead library (it only takes care of acking the first hop). We still need to implement this part in the MySensors library.

Also note you've increased payload-length field. This is only interesting if we use radios with supporting payloads larger than 32 bytes. So the question is if we just should allow the least common divisor here or not.

Here is a proposal reflecting the discussions in the other thread. The datatype gets one extra bit (its crowded now).

http://interactive.blockdiag.com/packetdiag/?compression=deflate&src=eJxtkE9LxDAQxe_7Keaoh2D_pGk3ICJYEASFRfAgsoRm2AZDUpPsShG_uxNoF0Fze7_3mHmTrw3A4O2n0WmEa-hIOq9xP6I5jIlI1axozWyLDaGCtRIuHvvnl6fdA9z3t3f97hKsigmWR6GOlY2Ev7GITmNYU6VgVf3PMI0xGaeS8S7HKs4qIWEKPnmqDCcMkazstKwuyVGz9UqDRXdII_G6krQqXQX8ONKsTGjPokAN75lwCSaugtqiG8I8JdQ3GQhWbyVolRSkeUJCvGC8zXNd9AGG0VgNRueCvGNNc3aWeCOYoKUnZY-4MsFZ-6vvK11z_u_6bfP9A4PMcVE

[Yveaux]

@hek My 5 cents (more like 2 Euros...):

• Is there a reason why you draw this as one monotonic packet? Don't you like the idea of nested structures?
• When using nested structures and defining commands, a command number has to be stored to indicate the data following the header. This command can differentiate between set-command and req-command, hence the set/request bit is not required.
• The payload length should be part of the network header. In fact, the payload length shouldn't have to be stored at all as the nRF24L01+ already stores it in its header. For radio's which don't pass it in their native header (or don't have a native header at all) it can be added to the network header.
• I would like to get rid of the whole isack/reqack idea... I think it should be handled by the network layer, when enabled, either between two nodes or end-to-end. It should be a configuration of the network connection and should operate at a lower level then MySensors. If RadioHead doesn't offer end-to-end (which I think it does; have a look at RHReliableDatagram) then a separate layer between MySensors and RadioHead should handle this. A solid communications library should provide send-and-forget methods; a user (sensor node) should not have to keep track whether its message got delivered or not. Btw. RadioHead uses message ID's to recognize when an ack to a message arrives.
• The encrypted-flag might need more states than just yes/no -- but that's for the future. Good to mention it here, though.
• Encryption, like end-to-end acknowledgement or (de)fragmentation, should also be handled on a lower level. Maybe move it to the network header?

[hek]

@Yveaux said:

Don't you like the idea of nested structures?

Yes, I do. Hmm.. Ok... it might be good to keep the command-filed for future things we might not have anticipated. Agree on keeping it.

Actually thought some time about skipping the length field also. The length information could be handled differently in the library. But what should we do with the 5 bits that frees up? :)

I totally missed @Damme post a couple of hours ago where he combined command and version in one byte. Might be a good thing. This would allow 16 different commands and versions.

And yes, the best thing would be if the transportation layer would handle end-to-end acking.

[hek]

So... how about something like this packed?

typedef struct
{
	uint8_t last;            	 // 8 bit - Id of last node this message passed
	uint8_t sender;          	 // 8 bit - Id of sender node (origin)
	uint8_t destination;     	 // 8 bit - Id of destination node
	uint8_t ack; 	        	 // 1 bit - Request an ack - Indicator that receiver should send an ack back.
								  // 1 bit - Is ack messsage - Indicator that this is the actual ack message.
								  // 6 bit - Reserved for future use. Perhaps encryption?
	char header[MAX_MESSAGE_LENGTH-4];

} transportation;

typedef struct
{
	uint8_t command;     // 8 bit - Command type
	char commandPayload[MAX_MESSAGE_LENGTH - sizeof(transportation) - 1];
} header;

typedef struct
{
	uint8_t sensorId;            // 8 bit - child sensor id
	uint8_t sensorType; 		 // 8 bit - sensor type
	uint8_t valueType_dataType;  // 5 bit - value type
	                             // 3 bit - data type

	union {
		uint8_t bValue;
		unsigned long ulValue;
		long lValue;
		unsigned int uiValue;
		int iValue;
		struct { 
			float fValue;
			uint8_t fPrecision;   // Number of decimals when serializing
		};
		char data[MAX_MESSAGE_LENGTH - sizeof(header) - 4];
	};

} commandSetReq;

But this would only allow 32 different valueTypes per sensorType. But it should be enough. The worst case "Node" would require 22 unique values including the 10 V_CONFIG/V_VAR.

[Yveaux]

@hek said:

version_command

Almost there ;-)

• I think we could drop the protocol version number and use a full byte for command type.
  When, in the future, a command gets extended we can just reserve a new command-number for it. All existing applications will only recognize the old command-number and ignore the new one. As @zeph already pointed out this allows for more efficient usage of the 8 bits and a new version will not immediately disqualify an onld node completely as it will stil listen & respond to old commands.
• It might be better to not include the remaining part of the message in a structure (e.g. header[MAX_MESSAGE_LENGTH-4], commandPayload[MAX_MESSAGE_LENGTH - sizeof(transportation) - 1] as this will make it very inefficient to store single instances of it (e.g. declaring a local instance of transportation or header will always clame a full message size)
• As a second though, it might be best to declare everything as union's, so the structures will overlap eachother (will become one badass strusture then....)
• I think I'll dive a bit deeper into the RadioHead library to understand what is already possible with it and to be able to exchange some ideas with @kolaf

Edit @hek you were right about the RadioHead library:

RHRouter (and therefore RHMesh) use reliable hop-to-hop delivery of messages using hop-to-hop acknowledgements, but not end-to-end acknowledgements.

[Zeph]

We should also consider interoperability with the the MQTT delivery concepts of "at most once", "at least once" and "exactly once" so the interface is smooth.

There are 4 dataflows where this interoperability could might be at least considered, I think.

controller to MQTT
MQTT to node
node to MQTT
MQTT to controller

[Zeph]

I think it's important that if we use structs and unions here, we understand that this is just a convenience for discussion, not the specification. For discussion we can assume that the structs will have 1 byte alignment for themselves and for all members, and will use little endianness.

What really matters to the spec is that the bytes in the order they go OTA be exactly the same for all senders and all receivers.

Those bytes might be interpreted in place as a struct (via static cast of a pointer into a byte buffer for example) IN SOME CASES, depending on CPU and on compiler and its options. In other cases it might be necessary to pack and unpack dynamically to avoid alignment and endianness issues.

Just a caveat to watch out for.

[Zeph]

@Yveaux said:

• I think we could drop the protocol version number and use a full byte for command type.
  When, in the future, a command gets extended we can just reserve a new command-number for it. All existing applications will only recognize the old command-number and ignore the new one. As @zeph already pointed out this allows for more efficient usage of the 8 bits and a new version will not immediately disqualify an onld node completely as it will stil listen & respond to old commands.

The value of having defined groups of command associated with a given protocol version is that you can get a pretty good idea of what a node will understand (or send). The first approach was to have some fixed number of bits for a protocol id, and some number for command-within-protocol (like 8+8 or 5+3 or whatever). I'm glad you see the advantages of adding command codes sequentially instead, especially if we are using one byte.

I was suggesting that a given protocol version define a contiguous range of commands (plus some universal ones at the beginning of the codes). One caveat was that the "current protocol version" would be allowed to add new commands at the end of the sequence - the "last command allowed in this version" doesn't get frozen until a version is superceded by another one. Any given node need only understand the universal commands plus the range corresponding to the protocol version it was written to. So each version has the common universal commands plus a non-overlapping range of command specific to this version.

The disadvantage is that you may have to redefine new command codes for commonly needed commands like "set value" every time you move to the next version. So in the first version, command 4 might be set (part of range 3..9). Then in the next version with commands (10..17) we have to define a new "set value" within that range, maybe code 11. Nodes implementing the second version only understand commands 0..2 (universal) and 10..17, they have no need to understand version one commands 3..9. A gateway would not expect a version 2 node to understand command 6. The hub could ask the node for the start and end of the range of commands it can accept.

===

One variant, which you are suggesting I think, is to ignore versoin based commadn code sub-ranges and just keep all commands allowable to all nodes. This has the advantage that "Set value" command 4 (in the above example) can continue to be used "forever", and not need to be reallocated code 11 to fit within the new version's range.

But it may over time cause fragmentation or bloat - either we carry around the code to handle all commands (eg: 1.3 and 1.4 both), or the set of commands allowed once you switch to a new protocol version is non-contiguous (it contains some but not all commands from the older protocol version). I actually considered this, and the possibility of having the node respond to "what commands can you handle" with a bitmap instead of a start and end value, but decided that was getting too complicated. That was just my judgement call tho.

(If you can correctly anticpate this kind of command in the "universal commands" that are part of all versions, the issue goes away. In the first appraoch, maybe "set value" is command 2 and globally available, not needing to be allocated a code in each versions's code range) But sometimes your concept changes more than you anticipated and "set value" might need to become a new command type in newer protocols).

Deciding among these options depends somewhat on one's philosophy of where configuration information should be stored - in the nodes or in the hub. There are many answers to this question at different granularities. If you accept the concept of storing much config on the hub, then another approach emerges:

• each node can use a basic common protocol to report on first connecting what version of the full protocol it implements
• that per-node protocol version is stored in the hub
• the hub has bitmaps or tables of what command codes work for each protocol version it understands
• if the hub understands more than one version, it can communicate with nodes using any of those versions

In this case, you don't make every packet include an explicit (fixed bits allocated) or implicit (code range) protocol version. But the hub could still know which commands work and don't work for any given node.