💬 Serial Protocol - 2.x

[hek]

This thread contains comments for the article "Serial Protocol - 2.x" posted on MySensors.org.

[Zeph]

I've been gone for a while, but this seems like a nice next step. Feel free to move these comments to the forum if that make more sense. If it's too late for this round, perhaps they can be useful for the next revision.

1. I still think the controller (or if need be, the gateway) should do any neccessary conversion between metric and imperial units, rather than pushing it down to every node; it seems like some kind of violation of partitioning to push that UI customization concept (which in many controllers the user can toggle freely or even display both ways) down to the node level, where it adds unneeded complication. But I'll let that go.

2. It would be good to augment this with a systematic description of the underlying value type (int32, float32, string), and implied units if appropriate (potentially two units, for metric and imperial). For example, is wind speed in m/s or km/hr (assuming metric)? Is pressure in mmHg or hPA?

3. Aside from the type and implied units, there are a few hints given already - what 0 and 1 mean, an enumerated list of valid string values, color given a strings of hex digits, position as a multivalued structured string, etc. Keep those as notes in augmented docs as described in #1, and add more a needed (eg: IR commands)
   2A) Is there any logic to when enumerated values are encoded as integers, and when they are encoded as strings from a controlled vocabulary? Could we get by with one or the other, rather than this hybrid?

4. How do the semicolons internal to V_POSITION interact with the semicolons in the message field structure? If the same character is used for both, some note about that is needed. Or use commas in the former (substructured payloads) and semicolons in the former (overall message structure).

5. I'm glad to see that a distinction has been made between sensed light levels in Lux and uncalibrated light levels. Generalizing, I can see four types of possible reporting of such things. 1: uncalibrated results in arbitrary units (eg: ADC values of a LDR+resitor voltage divider), 2: scaled results 0-100%, 3: reporting in predefined units understood by both sides, like Lux, 4: reporting in uninterpreted units which can only be displayed but are not parsed. In the first case, I think the only thing to define for a given V_variable is whether brighter should be reported as higher or lower. This standard chooses 0-100% for V_LIGHT_LEVEL. Could't that be reported as V_PERCENTAGE instead? That is, use V_PERCENTAGE for any sensor reporting in scaled percentages.

6. If the protocol is going to allow for both uncalibrated and calibrated values (the latter in implied units), as with light levels, then I think it makes sense for the value with implied units to be the more specific variable; that is, V_LIGHT_LEVEL should be in lux, and V_LEVEL (or V_PERCENTAGE) would be used for the generic values, without units.

5A) V_LEVEL currently seems a little convoluted. It appears that the implied units depend upon which S_sensor the variable is associated with, which kind of breaks the pattern that units (if any) are implied by V_variable. And for S_DUST, S_AIR_QUALITY) there are no units. So for a calibrated light level in lux OR an uncalibrated dust level, use V_LEVEL but for an uncalibrated light level use V_LIGHT_LEVEL. V_LEVEL seems to be a grab bag of conceptually unrelated magnitude values, some with per-S_sensor implied units and some without. This could use rethinking. How about if V_LEVEL was used for any values which do not have implicit units (see #1 above), and V_PERCENTAGE was used if the value scales to 0..100%. Other V_variables could have pre-specified implicit units (metric and imperial).
5B) V_UNIT_PREFIX is really more of a suffix than a prefix. But this can work well with 5A above. If a value is reported as V_LEVEL then there are no interpreted or parsed units involved and the controller is not expected to be able to interpret or convert units. However, V_UNIT_SUFFIX could optionally supply a free text string with can be displayed after the value. So if you are reading a LDR and reporting the raw ADC value (or inverted value to ge the direction right), use only V_LEVEL. If you are reporting something in units not understood by the system, then use V_LEVEL and describe the units in V_UNIT_SUFFIX. (And if you are reporting percentage, use V_PERCENTAGE).

1. Semantically, there are two kinds of outputs: set-current-value which sets a new state to be retained, and trigger/command which performs some action once. It's kind of like edges and levels in hardware logic. Setting the same value twice is harmless, but issuing the same command twice may have different effects than sending it once. Most of the outputs are about set-current-value. However, some are commands - like sending an IR code.
   6A) What do V_UP, V_DOWN, and V_STOP mean? Unless they contain payloads to configure a (retained) movement speed or something, they seem more like commands/triggers rather than set-current-value. In that case, it makes more sense to have a V_SHADES variable which is set to 0,1,2 for stop, up and down (which are mutually exclusive), than to have three variables which could be set in parallel. Likewise with V_SCENE_ON and _OFF. These should be handled with one V_variable, much as sending a IR command. This is also easier to extend for an additional command to the same device - just define additional values to action associations, rather than needing to add a new V_variable.
   6B) (A shade controller which could set/report percentage opening could use V_PERCENTAGE)

2. Likewise there are two kinds of inputs: report-current-value and trigger-event. Mostly this protocol is about report-values. However the trigger functionality comes up in terms of V_TRIPPED, which is often implemented as a trigger. It should be made explicit what resets a triggered state: some systems use autoreset where after one report a device goes back to untriggered (trigger/event); other systems will keep reporting triggered forever until separately reset by some other means (report-current-state). We should be clear which mode this protocol expects.

It's good to see the ongoing refinement of MySensors and related initiatives! These are definitely meant to be helpful, not as criticism! Your work is much appreciated.

[FotoFieber]

@Zeph
Thx for this "in depth analysis". I think it will be difficult to break compatibility, if there aren't enough complaints.

My wish for the future would be, to be able to plug your own handling of the messages on the gateway. I would use this for writing an MQTT-controller with integrated gateway. No need for a controller anymore and the mapping to MQTT would hide the MySensors internals, no more cryptic topics...

[tbowmo]

@Zeph

Thank you for the comments about the protocol. We have started to look at a new binary protocol for the future, and I think that your input is valuable.

However this will break compatibility completely, so we have to get plugin developers on board as well, when we are confident enough about the base layout of the protocol. As they have to be ready with support for a new protocol from day 1, when we release it.

(the protocol draft is on a Google document, can't remember the link right now, but perhaps @hek has it in his favorites :))

[hek]

Not much in it yet:
https://docs.google.com/document/d/1A853QT1mQhjJCmCKW1-zIylFOQTHYCNG2Sz22CGd8Qg/edit#heading=h.a87wwm5smzb

...and of course we can open it up for more editors who likes helping out writing technical documentation.

[Zeph]

@tbowmo I made some related comments some while back when 2.0 was being developed, but then I disappeared from MySensors for many months (life does that). So I understand that it's too late for changes to that, and I am only planting seeds for the future, accepting that many will not sprout soon, or ever. The current spec is already a good effort.

The comments about more specific documentation can still be implemented, tho.

Perhaps I can help with comments on the binary protocol. The problem is that it has to map nearly 1:1 into the serial protocol, right? Or not?

I continue to be impressed with the quality of this community.

[tbowmo]

@Zeph

As the binary protocol is breaking things anyways, we could break it so it becomes right So it doesn't have to map 1:1 with the old protocol.

I think that we had the ASCII protocol so much in the heads, when we started looking at a future (binary) protocol it was made 1:1.. But it's not finalized at all yet, so it's still very open for comments

[mfalkvidd]

There has been some discussions on making it possible to use redundant gateways. Serious use of home automation will need the communication to be working all the time. Maybe that's something to consider.

[robosensor]

Is there any chances that binary protocol will support sending of multiple sensor values per one message?

I'm talking about nodes that (for example) sending several binary switch sensor values with 1-byte value. Even if one set/req message contains 5 fields (5 bytes, currently Sensor id, Var type, Ack-thing (request ack / is ack), Payload type (integer, float, string, binary), Payload) it's possible to pack at least 4-5 results into single 32-byte message. This will reduce the load on the radio network 5 times. Also this will reduce gateway's load. It's also possible to remove duplicating fields and further reduce protocol.

For example, some of my nodes sending 5 temperatures from ds18b20 sensors. Packing all 5 temps into single message (2 bytes for temp and 1 byte for sensor id) will reduce traffic 5 times.

Packing/unpacking can be done at gateway side.

[tbowmo]

@robosensor

the protocol spec is still in very early draft.. That said, I would really like to keep things as simple as possible on the node side, in order to conserve flash / ram space as much as possible..

But let's see, it might be a year before we have something ready.. And it will probably be mysensors 3.0 that will get the new protocol, as it will probably break compatibility in every way possible

[Zeph]

@tbowmo I agree with keeping things simple to implement, particularly on the Node & Gateway side. And it doesn't hurt that this would likely also keep things simpler to implement on the Controller side (ie: simpler plug-ins/adapters/drivers to write, which is a positive even when resources are not so tight.).

We might be able to nevertheless implement some kind of multi-value packet concept tho. I added a comment to the 3.0 doc about a simple and low overhead way to allow homogeneous arrays. That might or might not be worthwhile.

Robosensors's use case would involve some kind of tagged sub-unit approach, with multiple id/value pairs, yet another concept worth considering.

[martinhjelmare]

Regarding V_POSITION and using semi-colons inside the payload, that will in my view break the API message structure. At least home assistant/pymysensors will not work with this. I'll build for and recommend users of that controller to use comma instead of semi-colon to split info inside the payload for V_POSITION.

[RWoerz]

This is just what I've been looking for. I do however have a suggestion. I'm building a solar heater (water) for my shop and need to keep track of at least 3 temps (solar panel, storage tank, and inside shop), 1 water pressure and 1 water flow meter (for solar panel Pump) . There doesn't seem to be V_ or S_ variables for direct water pressure or flow. Would you please add something for them.

[RWoerz]

Sorry I found V_FLOW 34 Flow of water (in meter) S_WATER. What exactly does "in meter" mean?

[musthafa]

Can we have S_SWITCH for Wall switches, door bell switches etc ?

[mfalkvidd]

@musthafa is there a reason S_BINARY can't be used for those cases?

[musthafa]

@mfalkvidd said:

@musthafa is there a reason S_BINARY can't be used for those cases?

Thanks. Didn't look at it. Solved my problem.

[mfalkvidd]

@musthafa great! Thanks for reporting back.

[gvorster]

Where can I find the structure for the payload of the internal command I_LOG_MESSAGE ? e.g. payload "TSF:MSG:SEND,211-0-220-220,s=211,c=1,t=23,pt=2,l=2,sg=0,ft=0,st=OK:69"
I know I'm sending a value from node 211 to node 220 with the value 69. What does c=1, t=23, pt=2, l=2, sg=0,ft=0 mean?

[mfalkvidd]

@gvorster see https://forum.mysensors.org/post/43373

[gvorster]

@mfalkvidd said in Serial Protocol - 2.x:

@gvorster see https://forum.mysensors.org/post/43373

Thanks, got it now.

[siod]

how can I handle sendheartbeat() messages in my controller? What payload will be sent? 1 or 0, True or False...?

ps I´m using openhab

[gvorster]

@siod said in Serial Protocol - 2.x:

how can I handle sendheartbeat() messages in my controller? What payload will be sent? 1 or 0, True or False...?

ps I´m using openhab

I just tested this using a Serial gateway.

Sending sendHeartbeat() from node displays this on my Serial gateway:

...
214;255;3;0;22;73
214;255;3;0;22;108
214;255;3;0;22;143
...

3=internal message
0=nack
22=I_HEARTBEAT_RESPONSE

In MyTransport.cpp I found

uint32_t transportGetHeartbeat(void)
{
	return transportTimeInState();
}

---

uint32_t transportTimeInState(void)
{
	return hwMillis() - _transportSM.stateEnter;
}

So the payload is some elapsed time in milliseconds.

[siod]

@gvorster Thank you for explanation!

One more thing: I wanted to implement a sendHeartbeat() into a Repeater node (which should never sleep!), where should I put this function and how? When putting it just into the loop it is of course spamming my gateway with heartbeat messages...Is it possible anyway?

[gvorster]

@siod said in Serial Protocol - 2.x:

@gvorster Thank you for explanation!

One more thing: I wanted to implement a sendHeartbeat() into a Repeater node (which should never sleep!), where should I put this function and how? When putting it just into the loop it is of course spamming my gateway with heartbeat messages...Is it possible anyway?

There are many examples code how to do this. One Timer library I use myself is this https://playground.arduino.cc/Code/Timer

e.g. for a repeater you could use this:

#include "Timer.h"

Timer t;

void setup() {
    t.every(60000, sendImAlive);
}

void loop() {
    t.update();
}

void sendImAlive() {
    sendHeartbeat();
}

[shabba]

What is the best for a push button? I have a button to open a lock. The button cannot lock the lock however. So binary is not really a proper option?

[hek]

ATM scene controller is probably the best option.

https://github.com/mysensors/MySensors/issues/724

[dmurphy]

S_SPRINKLER is defined as having V_STATUS and V_TRIPPED - no V_ARMED....but V_ARMED says it is in S_SPRINKLER...Which way should I code this up for my controller??

[jjk]

Hi there,
do you know that, you've built a node like a 100 times and I think it's all routine and then something that completely bluffs you? I happened to me with a simple Temp/Hum node (DHT22) that keeps sending me messages that I can't seem to figure out: 43;255;3;0;33;300000 and 43;255;3;0;32;500 - They seem to show up in the frequency that other nodes send a heartbeat signal, but no heartbeat is sent from this node. The regular DHT22 datapoints are sent as normal. Anyone knows what this means?

[gohan]

Are you using smart sleep?

[jjk]

yes, I do - like on every other DHT node as well, but never had that...?! Plus, I can't seem to figure out what the messages mean.

[gohan]

I assume the 300000 is the sleep time you set while the 500 is the smart sleep embedded 500ms wait that lets the controller know the node is about to go to sleep but it is going to wait for 500ms for queued commands to be sent to the node

[jjk]

hm, you might be right on the 300000 as that#s indeed the smartsleep wait time I'm setting. Not sure about the 500, but certainly possible (is that documented somewhere?) I'm wondering, though, why this node is sending this info, but not the other ones?! I don't think I have changed anything t the sketch other than the node-id and node name...

[gohan]

500ms is the default setting when using smartsleep

[jjk]

o.k., fair enough, that's a great explanation! - I'm still struggling, though, with why this node is sending this (but no heartbeat), but none of the previous ones I did...

[gohan]

Try reflash the older node, maybe you updated library or else. Why do you need heatbeat if the node is sending every 5 minutes?

[jjk]

@gohan fair question, I don't need it, it's just still in the code and besides, the 5 min message from smartsleep seems to show up every two mins (the heartbeaat rate)
And yes, I did indeed update the library, so maybe that's why. Turns out, the bigger problem is that these message seem to cause an exception in the openhab mysensor binding, so at best I'd like to figure out how to stop the node from sending them... Will do some digging

[gohan]

Are you sure it is the smartsleep? Since you are debugging, use https://www.mysensors.org/build/parser to decode messages to a more human readable format

[jjk]

yes, I am sure now: https://forum.mysensors.org/topic/8933/mysensors-2-2-0-released/27

Looks like the only way to eliminate the messages is to not use smartsleep

[gohan]

You may need to get in touch with openhab binding developer to fix that

[jjk]

@gohan yes, have done that. Thanks for your time and support!

[itbeyond]

I am investigating a problem I have found with I think openHAB but I am not sure if this issue is not related to the Gateway code of 2.3.0.
When a node performs a requestTime() I am seeing this reported:

2018-10-09 10:47:13.337 [INFO ] [rs.internal.gateway.MySensorsGateway] - I_TIME request received from 2, answering...

Looks perfect and the node receives the time however I was curious as to why sometimes the node would receive GMT and at other times my local timezone GMT+8. I decided to run the gateway via MYSContoller so I could monitor and log the traffic and I have found the problem. When the Gateway receives the request it responds twice within a few ms of each other, for what reason I do not know but the first response send the local time zone response (GMT+8) and the second is a GMT response. The logs from MYSController for the above report are as follows:

318857 9/10/2018 10:47:19 RX 2 - Irrigation Ctrl (2) INTERNAL C_INTERNAL NO I_TIME
318858 9/10/2018 10:47:19 TX 2 - Irrigation Ctrl (2) INTERNAL C_INTERNAL NO I_TIME 1539082039
318859 9/10/2018 10:47:19 TX 2 - Irrigation Ctrl (2) INTERNAL C_INTERNAL NO I_TIME "1539053233

"
Note the quotes where picked up in the copy and paste (I did this several times) so it looks like the body was changed to a string type but I also caught this in the MYSController Debug:
9/10/2018 10:47:19 RX 2;255;3;0;1;
9/10/2018 10:47:19 INFO Reply time request
9/10/2018 10:47:19 TX 2;255;3;0;1;1539082039
9/10/2018 10:47:19 FWD 2;255;3;0;1;1539053233
9/10/2018 10:47:19 TX 2;255;3;0;1;1539053233

So I am curious as to where the FWD comes from and why the response body has changed between the TX, FWD & TX?

[tonbor]

Is there a type to present in mysensors for solar radiation w/m2. Got a veml6075 sensor. And domoticz has a solar radiation sensor type.