2.0 Discussion: Units, sensor types and protocol

[hek]

Ok, I've had a solutions bubbling in my head over night. It will add one byte in the MySensors Req/Set Header but solves a lot of the discussions we've had earlier in other threads as well. Its also replaces both INTERNAL and PRESENTATION command.

--- Moved to the first post ---

[Yveaux]

@hek said:

Where does this fit?
V_DEW_POINT - Degrees Celsius <int or float>

Either it doesn't fit in (let the controller calculate it) or place under S_HUMIDITY.

I'm not completely convinced we need to store it this way and use another byte for each req/set message (which are most common...). I'll let it rest for a while so I might diss your solution later ;-)

[Damme]

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated. (my conclusion - I dont like it)

[hek]

@Damme said:

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated.

Read my post again and explain how you would distinguish a thermometer from a uv sensor on the controller side without it?

[Yveaux]

@hek said:

distinguish a thermometer from a uv sensor on the controller side

IMHO this all depends on the controller used, as I already explained. I'm not familiar with Vera, but e.g. for OpenHAB/MQTT the actual type of a sensor is defined by the OpenHAB configuration. It just receives a value and you need to tell it that node x sensor y is a temperature sensor.
Possibly Vera and other systems are more of a plug-and-play kind and are able to add a sensor depending on its type automatically.

I (for a long time) agree with @Damme but I can imagine there are systems that do require this information.

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

[Damme]

@hek I've been thinking and thinking and even might have an other solution.. but I'll wait with that.

so every piece of data has a S_type and V_type.. V_ must be a fixed unique table (not starting from 0 on every S_type)

I'm more acceptable now than 10 minutes ago.. :)

[Damme]

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

I think for the controllers view it would be best for the node to send every message with sensor id, sensor type, value type and payload. Noone needs to remember anything :) The way back to the node the rest can be omitted, as long as sensor id is there.

[hek]

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

Yes, it would save a byte. But it could be very useful for a simpler controller (which can determine sensor type from each and every message) and when displaying sniffing data ;).

This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

[Damme]

@hek S_node could include stuff as
Reboot
ClearEEprom
Read EEprom byte
Set EEprom byte
besides stream data types.
(to be discussed if any of them are useful or not and if they should be hardcoded or not)

[Yveaux]

@hek said:

when displaying sniffing data

Yeay yeah, blame it on the sniffer :facepunch:
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

This means just 1 bit is needed for a SET/REQ flag

Looks like the protocol discussion is shifting towards this topic. Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

[hek]

@Yveaux said:

Yeay yeah, blame it on the sniffer
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

:) We're talking about a total of just 8 bytes header here. And if we remove routing just 5 bytes brimful with useful information about the payload.

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

:question: I do not understand this.

[Yveaux]

@hek said:

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.
I do not understand this.

Probably because I didn't understand you then...
I interpret your post as using a single message structure for either set or req messages. A single bit is used to distinguish between the two, right?

[hek]

@Yveaux said:

A single bit is used to distinguish between the two, right?

Yes.

[Yveaux]

@hek What I meant was to just have two separate commands for set & req (like it is now in 1.3 & 1.4b) but use the same messgae structure to send these messages. The command value will be used to differentiate between set & req.
No need to reserve a bit in the message this way.
Hope its clear now.... :dart:

[Zeph]

@hek
First off - I think this taxonomy of sensactuator values is getting better each time, and I appreciate your mixture of creativity and receptivity as the leader in this effort. Good work.

This is not dissing, more like reflecting on what I see more than making judgements.

The S_ and V_ enums (controlled vocabularies) are becoming relatively more orthogonal rather than confusing people by appearing to cover similar conceptual territory twice but in different way, as the earlier versions inherited from Vera tended to do. So for example, S_TEMPERATURE has V_LEVEL rather than V_TEMPERATURE.

One result is that to understand what a transmitted value (number) represents, you would now need to look at both the S_ code and the V_code (as a good thing). The "dimension" and "units" concepts for understanding a V_LEVEL value depend on the S_ code associated with the variable. In a sense there would now be a two-byte combined expansion of the concepts often formerly described in the one byte V_ code. (So instead of V_TEMPERATURE, there is V_LEVEL+S_TEMPERATURE and you have to look at both to see that the number is temperature in degrees C).I think this is getting cleaner.

The semantics or meaning of a variable are still somewhat distributed between the S and V codes. For example V_LEVEL and V_ANGLE have very little semantic content, relying almost entirely on the S code to provide that, but V_ARMED vs V_TRIPPED do have semantic content. This is just an observation; I'm still letting it sink in.

A big question is the relationship of node children with S_ code - both are called sensors but they need not be the same thing. For this discussion a "Child" is an addressable subunit of a node, numbered 0,1,2... and each child can contain a set of variables. An S code is a one byte code associated with each variable (along with a V code). In an earlier discussion where I was questioning the value of the "Sensor" level of the heirarchy you made the point that there could be value in grouping several variables that are logically tied to "the same thing". The Child concept does that, it provides an addressable container for one or more variables.

At issue is whether there is a fixed relationship beween the Child and the S_ code, whether all variables within the same Child container must have the same S_ code or not.

• If all variables in a given Child must have the same S_ code, then as somebody else pointed out this could be sent to the hub once during node integration and stored there (associated with the given node/child), rather than being transmitted with each value. But you cannot include variables for temperature and humidity in the same Child, even if logically that's what you want to convey (say you have two DHT-11's on the same node, you can't just make them two Children each with a temp and humidity).

• If each variable within a given Child can have its own S code, then the S code and V code become a two byte descriptor of the variable, and Child is a flexible container used to represent "these variables go together" and only that.

Which of these relationships between a Child (addressable subunit of a node) and S code (one byte code that along with V code helps interpret a given variable) did you have in mind?

[Damme]

@hek All these changes are more 2.0 than 1.4.. :)

Edit;
Btw, All these extra bytes S_ and V_ to describe a payload is good for mqtt also.
And this is why we need some more command types (i.e. Stream for OTA update etc. more space for payload smaller header)

[Zeph]

@hek

I see the edited version with MIN, MAX and RESET's added. This takes the orthogonality of V and S codes even further, and it seems like a good direction.

And it's good to see a start on better addressing one of the complicated aspects of this framework = accumulation periods (for Max/Min/Average/Total).

But is V_RESET a variable or a command, conceptually? Is is something sent to the node (probably) or reported by the hub (probably not)?

This brings up another broad conceptual issue in creating a taxonomny like this - the difference between values and command/events. Many software frameworks have both concepts, for example class member variables and class funtions/events. (In electrical terms. this is somewhat like the "Level" vs "Edge or Trigger" distinction).

In the current taxonomy, command/event and value are kind of intermingled. I think it will be important to document which is which.

There are a couple of approaches. One is to define a list of commands (eg; C_RESET) distince from values (V_LEVEL); the other is to have a well defined way of using variables as pseudo-commands.

For one possible example of the latter, one could say that for any variable defined as a command variable:

• when the variable goes from 0 to 1, the node will take the defined action
• the hub will send the node a "Set" command to set the value to 1; if it was 0, then the action is taken
• Setting to 0 has no effect
• the node will set the value to 0 when the action is completed (may be immediate)
• the hub can query the current value of the variable to see if there is any action still pending or not

(This might be exactly what you had in mind, or you might have had a variation in mind - I'm just saying that it should be clearly defined and documented).

Then we could say that V_RESET is a command variable and thus shares the standard dynamics, where the specific "action" is to restart the accumulation period. And there could be other command variables, each documented as such.

(The Event concept is similar to a command in that it's a transition rather than a level, but used as reports from the node to the hub. Not addressed yet)

[Zeph]

One characteristic of the MySensors system is that nodes are not systematically aware of time. That's good in that it simplifies the node firmware, and it's a limitation in terms of dealing with accumulation periods.

I'm speaking of "accumulation" periods rather than "reporting" periods, because a node may "report" many times in an accumulation period, for example every 10 seconds it may report the accumulated rainfall since midnight.

And there are several kinds of accumulation periods: in particular rolling and fixed start. Fixed start might report accumulation (sum, average, min, max) since a given time like midnight. Rolling might report the last hour or last 5 minutes from now. (I mentioned a rain sensor which does it's own "rain event" splitting based on no-rain intervals).

In some cases, accumulation periods may not be under our control - for example reading an integrated weather station. At best a node could report the accumulation period.

In other cases, we can control the accumulation period, for example the node's sketch could itself count KWH or rain, or it could average temperature or speed or direction, taking readings every N seconds.

One more concept: how to handle a Reset. The simplest approach for a hub-commanded reset would be for the hub to be sure it has all desired recent value reports before sending a Reset command (so the time gap between last report and Reset is short enough that nothing signficant was missed, eg: mm of rain). A more synchronous approach would be for the node to send one last set of accumulated values immediately before zeroing it's counter.

But what about a manually triggered reset (button pushed)? The sensor node could in that case want to let the hub know that it happened.

---

One way to unify these in a simple and consistent way would be to report V_ACCUM_TIME - the number of seconds since the accumulation period began. That's the period over which values were averaged, summed, or min/max'd. The time would increase until a reset, go to zero and increase again. This handles controllable and not controllable accumulation periods, fixed and rolling accumulation periods, hub triggered and manually triggered resets.

If an accumulation period is reset, the node could potentially send a last report with the ending V_ACCUM_TIME before the reset and then the following report would have time since starting the new period.

If the time is just based on millis() then with the Arduino Pro Micro the time accuracy would be nominally about 0.5% (they have ceramic resonators rather than crystals and that's a typical spec). That's probably good enough, tho it's possible to do better if it mattered.

---

Where this seems to be going is that any quantity variable could potentially be expanded into an accumulation variable - which means that in addition to (say) V_LEVEL, there is V_LEVEL_MAX, V_LEVEL_MIN, V_LEVEL_TIME, and either V_LEVEL_AVG or V_LEVEL_SUM. This "accumulation enhancement" could optionally be added to temp, humidity, wind, power, fluid/gas flow, whatever - in a defined and consistent way.

[hek]

@Zeph

FYI: Sleeping sensors (who also can be woken by a interrupt on a pin) will have a hard time keeping time without clock module or by fetching time from server at each wakeup from pin trigger.

Good idea about a STATUS for RAIN sensor. It will be added and indicates if rain is ongoing.
Will also change to ACCUMULATED for all accumulated values. Average for these makes no sense.

So.. I see now that there is need for more than one reset command. We'll need RESET_ACCUMELATED, RESET_LEVEL, RESET_PERCENTAGE. They should also have their respective MAX, MIN, AVERAGE. Could you please give me a use case for V_LEVEL_SUM.

Just to clarify for anyone getting a bit scared of all these new variables. Using them for your own sensor is of course optional!

I feel it's time to move the new stuff to the first post to keep it in focus.

[Damme]

I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible

If I wanted node could send
S_MOTION V_TRIPPED 1 (motion tripped)
S_MOTION V_ARMED 0 (no longer armed)
S_MOTION V_LEVEL 24 (motion quality 24 of 255) (probobly animal)

controller decides S_MOTION V_RESET 1

(order not thought of at all, needs to be fixed)

 #DEFINE S_DOOR	1
 #DEFINE S_MOTION	2
 #DEFINE S_SMOKE	3
 #DEFINE S_BINARY	4
 #DEFINE S_DIMMABLE	5
 #DEFINE S_WINDOW_COVER	6
 #DEFINE S_THERMOMETER	7
 #DEFINE S_HUMIDITY	8
 #DEFINE S_BAROMETER	9
 #DEFINE S_WIND	10
 #DEFINE S_RAIN	11
 #DEFINE S_UV	12
 #DEFINE S_WEIGHT_SCALE	13
 #DEFINE S_POWER	14
 #DEFINE S_HEATER	15
 #DEFINE S_DISTANCE	16
 #DEFINE S_LIGHT_SENSOR	17
 #DEFINE S_NODE	18
 #DEFINE S_LOCK	19
 #DEFINE S_IR	20
 #DEFINE S_WATER_METER	21
 #DEFINE S_AIR_QUALITY	22
 #DEFINE S_CUSTOM	23
 #DEFINE S_DUST	24
 #DEFINE S_PH	25
 #DEFINE S_SCENE_CONTROLLER	26
 #DEFINE S_NODE	255
 
 #DEFINE V_CONFIG1	1
 #DEFINE V_CONFIG2	2
 #DEFINE V_CONFIG3	3
 #DEFINE V_CONFIG4	4
 #DEFINE V_CONFIG5	5
 #DEFINE V_VAR1	6
 #DEFINE V_VAR2	7
 #DEFINE V_VAR3	8
 #DEFINE V_VAR4	9
 #DEFINE V_VAR5	10
 #DEFINE V_ARMED	11
 #DEFINE V_STATUS	12
 #DEFINE V_WATT	13
 #DEFINE V_PERCENTAGE	14
 #DEFINE V_STOP	15
 #DEFINE V_LEVEL	16
 #DEFINE V_MAX	17
 #DEFINE V_MIN	18
 #DEFINE V_RESET	19
 #DEFINE V_DEW_POINT	20
 #DEFINE V_MODE	21
 #DEFINE V_ANGLE	22
 #DEFINE V_RATE	23
 #DEFINE V_VOLTS	24
 #DEFINE V_AMPS	25
 #DEFINE V_PRESENTATION	26
 #DEFINE V_BATTERY_LEVEL	27
 #DEFINE V_RESET	28
 #DEFINE V_TIME	29
 #DEFINE V_ID	30
 #DEFINE V_LOG_MESSAGE	31
 #DEFINE V_SKETCH_NAME	32
 #DEFINE V_SKETCH_VERSION	33
 #DEFINE V_FIND_PARENT	34
 #DEFINE V_CHILDREN	35
 #DEFINE V_VERSION	36
 #DEFINE V_INCLUSION_MODE	37
 #DEFINE V_GATEWAY_READY	38
 #DEFINE V_STATUS	39
 #DEFINE V_IR_SEND	40
 #DEFINE V_IR_RECEIVE	41