💬 Serial Protocol - 2.x
-
This thread contains comments for the article "Serial Protocol - 2.x" posted on MySensors.org.
-
I've been gone for a while, but this seems like a nice next step. Feel free to move these comments to the forum if that make more sense. If it's too late for this round, perhaps they can be useful for the next revision.
-
I still think the controller (or if need be, the gateway) should do any neccessary conversion between metric and imperial units, rather than pushing it down to every node; it seems like some kind of violation of partitioning to push that UI customization concept (which in many controllers the user can toggle freely or even display both ways) down to the node level, where it adds unneeded complication. But I'll let that go.
-
It would be good to augment this with a systematic description of the underlying value type (int32, float32, string), and implied units if appropriate (potentially two units, for metric and imperial). For example, is wind speed in m/s or km/hr (assuming metric)? Is pressure in mmHg or hPA?
-
Aside from the type and implied units, there are a few hints given already - what 0 and 1 mean, an enumerated list of valid string values, color given a strings of hex digits, position as a multivalued structured string, etc. Keep those as notes in augmented docs as described in #1, and add more a needed (eg: IR commands)
2A) Is there any logic to when enumerated values are encoded as integers, and when they are encoded as strings from a controlled vocabulary? Could we get by with one or the other, rather than this hybrid? -
How do the semicolons internal to V_POSITION interact with the semicolons in the message field structure? If the same character is used for both, some note about that is needed. Or use commas in the former (substructured payloads) and semicolons in the former (overall message structure).
-
I'm glad to see that a distinction has been made between sensed light levels in Lux and uncalibrated light levels. Generalizing, I can see four types of possible reporting of such things. 1: uncalibrated results in arbitrary units (eg: ADC values of a LDR+resitor voltage divider), 2: scaled results 0-100%, 3: reporting in predefined units understood by both sides, like Lux, 4: reporting in uninterpreted units which can only be displayed but are not parsed. In the first case, I think the only thing to define for a given V_variable is whether brighter should be reported as higher or lower. This standard chooses 0-100% for V_LIGHT_LEVEL. Could't that be reported as V_PERCENTAGE instead? That is, use V_PERCENTAGE for any sensor reporting in scaled percentages.
-
If the protocol is going to allow for both uncalibrated and calibrated values (the latter in implied units), as with light levels, then I think it makes sense for the value with implied units to be the more specific variable; that is, V_LIGHT_LEVEL should be in lux, and V_LEVEL (or V_PERCENTAGE) would be used for the generic values, without units.
5A) V_LEVEL currently seems a little convoluted. It appears that the implied units depend upon which S_sensor the variable is associated with, which kind of breaks the pattern that units (if any) are implied by V_variable. And for S_DUST, S_AIR_QUALITY) there are no units. So for a calibrated light level in lux OR an uncalibrated dust level, use V_LEVEL but for an uncalibrated light level use V_LIGHT_LEVEL. V_LEVEL seems to be a grab bag of conceptually unrelated magnitude values, some with per-S_sensor implied units and some without. This could use rethinking. How about if V_LEVEL was used for any values which do not have implicit units (see #1 above), and V_PERCENTAGE was used if the value scales to 0..100%. Other V_variables could have pre-specified implicit units (metric and imperial).
5B) V_UNIT_PREFIX is really more of a suffix than a prefix. But this can work well with 5A above. If a value is reported as V_LEVEL then there are no interpreted or parsed units involved and the controller is not expected to be able to interpret or convert units. However, V_UNIT_SUFFIX could optionally supply a free text string with can be displayed after the value. So if you are reading a LDR and reporting the raw ADC value (or inverted value to ge the direction right), use only V_LEVEL. If you are reporting something in units not understood by the system, then use V_LEVEL and describe the units in V_UNIT_SUFFIX. (And if you are reporting percentage, use V_PERCENTAGE).-
Semantically, there are two kinds of outputs: set-current-value which sets a new state to be retained, and trigger/command which performs some action once. It's kind of like edges and levels in hardware logic. Setting the same value twice is harmless, but issuing the same command twice may have different effects than sending it once. Most of the outputs are about set-current-value. However, some are commands - like sending an IR code.
6A) What do V_UP, V_DOWN, and V_STOP mean? Unless they contain payloads to configure a (retained) movement speed or something, they seem more like commands/triggers rather than set-current-value. In that case, it makes more sense to have a V_SHADES variable which is set to 0,1,2 for stop, up and down (which are mutually exclusive), than to have three variables which could be set in parallel. Likewise with V_SCENE_ON and _OFF. These should be handled with one V_variable, much as sending a IR command. This is also easier to extend for an additional command to the same device - just define additional values to action associations, rather than needing to add a new V_variable.
6B) (A shade controller which could set/report percentage opening could use V_PERCENTAGE) -
Likewise there are two kinds of inputs: report-current-value and trigger-event. Mostly this protocol is about report-values. However the trigger functionality comes up in terms of V_TRIPPED, which is often implemented as a trigger. It should be made explicit what resets a triggered state: some systems use autoreset where after one report a device goes back to untriggered (trigger/event); other systems will keep reporting triggered forever until separately reset by some other means (report-current-state). We should be clear which mode this protocol expects.
It's good to see the ongoing refinement of MySensors and related initiatives! These are definitely meant to be helpful, not as criticism! Your work is much appreciated.
-
-
@Zeph
Thx for this "in depth analysis". I think it will be difficult to break compatibility, if there aren't enough complaints. :)My wish for the future would be, to be able to plug your own handling of the messages on the gateway. I would use this for writing an MQTT-controller with integrated gateway. No need for a controller anymore and the mapping to MQTT would hide the MySensors internals, no more cryptic topics... :)
-
I've been gone for a while, but this seems like a nice next step. Feel free to move these comments to the forum if that make more sense. If it's too late for this round, perhaps they can be useful for the next revision.
-
I still think the controller (or if need be, the gateway) should do any neccessary conversion between metric and imperial units, rather than pushing it down to every node; it seems like some kind of violation of partitioning to push that UI customization concept (which in many controllers the user can toggle freely or even display both ways) down to the node level, where it adds unneeded complication. But I'll let that go.
-
It would be good to augment this with a systematic description of the underlying value type (int32, float32, string), and implied units if appropriate (potentially two units, for metric and imperial). For example, is wind speed in m/s or km/hr (assuming metric)? Is pressure in mmHg or hPA?
-
Aside from the type and implied units, there are a few hints given already - what 0 and 1 mean, an enumerated list of valid string values, color given a strings of hex digits, position as a multivalued structured string, etc. Keep those as notes in augmented docs as described in #1, and add more a needed (eg: IR commands)
2A) Is there any logic to when enumerated values are encoded as integers, and when they are encoded as strings from a controlled vocabulary? Could we get by with one or the other, rather than this hybrid? -
How do the semicolons internal to V_POSITION interact with the semicolons in the message field structure? If the same character is used for both, some note about that is needed. Or use commas in the former (substructured payloads) and semicolons in the former (overall message structure).
-
I'm glad to see that a distinction has been made between sensed light levels in Lux and uncalibrated light levels. Generalizing, I can see four types of possible reporting of such things. 1: uncalibrated results in arbitrary units (eg: ADC values of a LDR+resitor voltage divider), 2: scaled results 0-100%, 3: reporting in predefined units understood by both sides, like Lux, 4: reporting in uninterpreted units which can only be displayed but are not parsed. In the first case, I think the only thing to define for a given V_variable is whether brighter should be reported as higher or lower. This standard chooses 0-100% for V_LIGHT_LEVEL. Could't that be reported as V_PERCENTAGE instead? That is, use V_PERCENTAGE for any sensor reporting in scaled percentages.
-
If the protocol is going to allow for both uncalibrated and calibrated values (the latter in implied units), as with light levels, then I think it makes sense for the value with implied units to be the more specific variable; that is, V_LIGHT_LEVEL should be in lux, and V_LEVEL (or V_PERCENTAGE) would be used for the generic values, without units.
5A) V_LEVEL currently seems a little convoluted. It appears that the implied units depend upon which S_sensor the variable is associated with, which kind of breaks the pattern that units (if any) are implied by V_variable. And for S_DUST, S_AIR_QUALITY) there are no units. So for a calibrated light level in lux OR an uncalibrated dust level, use V_LEVEL but for an uncalibrated light level use V_LIGHT_LEVEL. V_LEVEL seems to be a grab bag of conceptually unrelated magnitude values, some with per-S_sensor implied units and some without. This could use rethinking. How about if V_LEVEL was used for any values which do not have implicit units (see #1 above), and V_PERCENTAGE was used if the value scales to 0..100%. Other V_variables could have pre-specified implicit units (metric and imperial).
5B) V_UNIT_PREFIX is really more of a suffix than a prefix. But this can work well with 5A above. If a value is reported as V_LEVEL then there are no interpreted or parsed units involved and the controller is not expected to be able to interpret or convert units. However, V_UNIT_SUFFIX could optionally supply a free text string with can be displayed after the value. So if you are reading a LDR and reporting the raw ADC value (or inverted value to ge the direction right), use only V_LEVEL. If you are reporting something in units not understood by the system, then use V_LEVEL and describe the units in V_UNIT_SUFFIX. (And if you are reporting percentage, use V_PERCENTAGE).-
Semantically, there are two kinds of outputs: set-current-value which sets a new state to be retained, and trigger/command which performs some action once. It's kind of like edges and levels in hardware logic. Setting the same value twice is harmless, but issuing the same command twice may have different effects than sending it once. Most of the outputs are about set-current-value. However, some are commands - like sending an IR code.
6A) What do V_UP, V_DOWN, and V_STOP mean? Unless they contain payloads to configure a (retained) movement speed or something, they seem more like commands/triggers rather than set-current-value. In that case, it makes more sense to have a V_SHADES variable which is set to 0,1,2 for stop, up and down (which are mutually exclusive), than to have three variables which could be set in parallel. Likewise with V_SCENE_ON and _OFF. These should be handled with one V_variable, much as sending a IR command. This is also easier to extend for an additional command to the same device - just define additional values to action associations, rather than needing to add a new V_variable.
6B) (A shade controller which could set/report percentage opening could use V_PERCENTAGE) -
Likewise there are two kinds of inputs: report-current-value and trigger-event. Mostly this protocol is about report-values. However the trigger functionality comes up in terms of V_TRIPPED, which is often implemented as a trigger. It should be made explicit what resets a triggered state: some systems use autoreset where after one report a device goes back to untriggered (trigger/event); other systems will keep reporting triggered forever until separately reset by some other means (report-current-state). We should be clear which mode this protocol expects.
It's good to see the ongoing refinement of MySensors and related initiatives! These are definitely meant to be helpful, not as criticism! Your work is much appreciated.
Thank you for the comments about the protocol. We have started to look at a new binary protocol for the future, and I think that your input is valuable.
However this will break compatibility completely, so we have to get plugin developers on board as well, when we are confident enough about the base layout of the protocol. As they have to be ready with support for a new protocol from day 1, when we release it.
(the protocol draft is on a Google document, can't remember the link right now, but perhaps @hek has it in his favorites :))
-
-
Thank you for the comments about the protocol. We have started to look at a new binary protocol for the future, and I think that your input is valuable.
However this will break compatibility completely, so we have to get plugin developers on board as well, when we are confident enough about the base layout of the protocol. As they have to be ready with support for a new protocol from day 1, when we release it.
(the protocol draft is on a Google document, can't remember the link right now, but perhaps @hek has it in his favorites :))
@tbowmo I made some related comments some while back when 2.0 was being developed, but then I disappeared from MySensors for many months (life does that). So I understand that it's too late for changes to that, and I am only planting seeds for the future, accepting that many will not sprout soon, or ever. The current spec is already a good effort.
The comments about more specific documentation can still be implemented, tho.
Perhaps I can help with comments on the binary protocol. The problem is that it has to map nearly 1:1 into the serial protocol, right? Or not?
I continue to be impressed with the quality of this community.
-
As the binary protocol is breaking things anyways, we could break it so it becomes right :) So it doesn't have to map 1:1 with the old protocol.
I think that we had the ASCII protocol so much in the heads, when we started looking at a future (binary) protocol it was made 1:1.. But it's not finalized at all yet, so it's still very open for comments :)
-
Is there any chances that binary protocol will support sending of multiple sensor values per one message?
I'm talking about nodes that (for example) sending several binary switch sensor values with 1-byte value. Even if one set/req message contains 5 fields (5 bytes, currently Sensor id, Var type, Ack-thing (request ack / is ack), Payload type (integer, float, string, binary), Payload) it's possible to pack at least 4-5 results into single 32-byte message. This will reduce the load on the radio network 5 times. Also this will reduce gateway's load. It's also possible to remove duplicating fields and further reduce protocol.
For example, some of my nodes sending 5 temperatures from ds18b20 sensors. Packing all 5 temps into single message (2 bytes for temp and 1 byte for sensor id) will reduce traffic 5 times.
Packing/unpacking can be done at gateway side.
-
the protocol spec is still in very early draft.. That said, I would really like to keep things as simple as possible on the node side, in order to conserve flash / ram space as much as possible..
But let's see, it might be a year before we have something ready.. And it will probably be mysensors 3.0 that will get the new protocol, as it will probably break compatibility in every way possible :D
-
the protocol spec is still in very early draft.. That said, I would really like to keep things as simple as possible on the node side, in order to conserve flash / ram space as much as possible..
But let's see, it might be a year before we have something ready.. And it will probably be mysensors 3.0 that will get the new protocol, as it will probably break compatibility in every way possible :D
@tbowmo I agree with keeping things simple to implement, particularly on the Node & Gateway side. And it doesn't hurt that this would likely also keep things simpler to implement on the Controller side (ie: simpler plug-ins/adapters/drivers to write, which is a positive even when resources are not so tight.).
We might be able to nevertheless implement some kind of multi-value packet concept tho. I added a comment to the 3.0 doc about a simple and low overhead way to allow homogeneous arrays. That might or might not be worthwhile.
Robosensors's use case would involve some kind of tagged sub-unit approach, with multiple id/value pairs, yet another concept worth considering.
-
Regarding V_POSITION and using semi-colons inside the payload, that will in my view break the API message structure. At least home assistant/pymysensors will not work with this. I'll build for and recommend users of that controller to use comma instead of semi-colon to split info inside the payload for V_POSITION.
-
This is just what I've been looking for. I do however have a suggestion. I'm building a solar heater (water) for my shop and need to keep track of at least 3 temps (solar panel, storage tank, and inside shop), 1 water pressure and 1 water flow meter (for solar panel Pump) . There doesn't seem to be V_ or S_ variables for direct water pressure or flow. Would you please add something for them.
-
@mfalkvidd said:
@musthafa is there a reason S_BINARY can't be used for those cases?
Thanks. Didn't look at it. Solved my problem.
-
Where can I find the structure for the payload of the internal command I_LOG_MESSAGE ? e.g. payload "TSF:MSG:SEND,211-0-220-220,s=211,c=1,t=23,pt=2,l=2,sg=0,ft=0,st=OK:69"
I know I'm sending a value from node 211 to node 220 with the value 69. What does c=1, t=23, pt=2, l=2, sg=0,ft=0 mean? -
Where can I find the structure for the payload of the internal command I_LOG_MESSAGE ? e.g. payload "TSF:MSG:SEND,211-0-220-220,s=211,c=1,t=23,pt=2,l=2,sg=0,ft=0,st=OK:69"
I know I'm sending a value from node 211 to node 220 with the value 69. What does c=1, t=23, pt=2, l=2, sg=0,ft=0 mean?
