2.0 Discussion: Units, sensor types and protocol

[Zeph]

@hek said:

V_TEMPERATURE - Degrees celsius <int or float>
V_HUMIDITY - Humidity percentage 0-100% <int or float>

Can any of the values be int or float? Or just these two?

V_DIMMER - Percentage 0-100% (used for lights, blinds and curtains)

The core concept here is V_PERCENT. In one home automation system there might be only one included concept, the Dimmer, and you'd have to use that for blinds as well as dimmers. In another home automation system there might be separate UI widgets for blinds and light dimmers. The sensactuator node's code should not need to change either way. Mapping a given V_PERCENT variable to UI concepts of blinds or dimmers is conceptually part of the configuration for a given (version of a) home automation system.

V_FORECAST Whether forecast. One of "stable", "sunny", "cloudy", "unstable", "thunderstorm" or "unknown"

Difficult one. That is very definitely not the controlled vocabulary used by all forecasting sources, and mapping into this list is going to be difficult. "Scattered showers" and "partly cloudy" and "light rain" and "foggy" are not going to be easily represented. What about stable/sunny versus stable/rainy? Are stable/unstable mutually exclusive choices with the others, or optional additional attributes for any of the other choices?

Maybe this has to be an open vocabulary with a text string? What text they will get depends on their source.

V_RAIN - mm
V_WIND - m/s (average wind speed during last report period)
V_GUST - m/s (highest wind speed during since last report)
V_WATT - Watt <int>
V_KWH - kWh <int or float>

What is the report period and what resets it? Last minute, last hour, last 24 hours, since midnight, since the reset button was pushed, or using automatic episode detection (I was just looking at a rain sensor which uses 8 hours of no rain to separate rain accumulation events). Without some knowledge of the reporting period these values cannot be meaningfully interpreted.

V_RAIN could be a forever incrementing total if the host wants to do its own deltas and define it's own report periods (maybe several of them).

V_WIND is a bit more difficult. If we had guaranteed delivery, we could just average since the last report from the node. Of course, some weather stations may do their own averaging, or the node could average many readings. But how does the host know?

From my perspective these "time period sum or average" values would ideally involve:

• an OTA format which loses no useful information available from the sensors or peripherals
• a flexible gateway (or adapter) which reformulates into time units meaningful to the HA controller software
  Let me make up an example. Suppose the sensor reported wind speed averages over 5 minute periods and the HA controller wanted to report it over one hour periods. The gateway could average 12 reports - for either the last 60 minutes, or for on-the-hour, as needed. Or "new rainfall" could be reported in 10 minute increments, but accumulated since midnight by the gateway, if that's what the controller prefers. Different controllers might use different accumulation periods; and what is sent to a controller could vary from what was uploaded to a cloud service. The node should not be adapting to the controller, it should send all available and useful info and the gateway or adapter should adjust - sort of like degrees F but more complicated because it might involve keeping running totals for averaging.

V_ANGLE - radians <float or int>

Here's one of those "generic vs specific" ones. Like V_PERCENT suggested above, V_ANGLE has no implications for what is being measured, only the units. It could be wind direction, it could be the plantation shutter position, it could be the solar panel tilt. How does the host know?

(This is the kind of thing that prompted my suggestions on another thread to separate the concept of units from what is being measured.)

An int for radians doesn't make much sense (unless we have a scaling factor).
I think degrees is a reasonable compromise - it's more precise than any sensor we're going to afford (or than the noise in the data itself would make moot), so it loses no meaningful accuracy. And virtually no sensor is going to report in radians. Some report in degrees. If it reports in 16 compass points or such, that can be easily converted to degrees, without even using floating point.

V_WEIGHT - kg <int, float>
V_DISTANCE - Cm <int, float>
V_IMPEDANCE - Ohm

See other message about intermediate calculations and the true conceptual thing being measured, (with impedance as an example). What does a distance of 240cm mean?

V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed
V_TRIPPED - Tripped status of a security sensor. 1=Tripped, 0=Untripped

Here the underlying concept is "V_BINARY", with a variety of associated labels for 1 and 0 based on what is being reported or controlled. On/Off, Open/Closed, Armed/Unarmed, Tripped/Untripped, Locked/Unlocked, Enabled/Disabled, etc. Do we create an ever expanding set of binary options as different variable types? Or is it functionally one variable type (or a small number with definably different dynamics from each other) with additional attributes (like the labels used for 1 and 0)?

V_SCENE_ON - Scene number <int>
V_SCENE_OFF - Scene number <int>

Now we are conflating events or commands with statuses or values. We turn on a light by setting it to 1 (or 100%) and we turn it off by setting it to 0 (or 0%). This leverages the same "set value to" underlaying functionality, and allows a simple and consistent query concept (what is the current value of the light?) What is the advantage of treating a Scene differently?

Scenes are a bit confusing, because they may or may not be mutually exclusive. They are neither obviously "checkboxes" or "radio buttons".

V_HEATER_MODE - Heater mode. One of "Off", "HeatOn", "CoolOn", or "AutoChangeOver"

How do we set the themostat? What about Fan?

V_HEATER_STATUS - 1=On, 0=Off. Heater power switch

What about CoolOn mode? If it's cooling, is V_HEATER_STATUS on or off?
.

V_LIGHT_LEVEL - 0-100% <int>

Is this a sensor (alternative to V_Lux) or an actuator (different than V_DIMMER/V_PERCENT above)?

V_UP - command which modes blinds ful up. Use DIMMER to specify percentage
V_DOWN - command which moved blinds to full down. Use DIMMER to specify percentage

Use V_DIMMER (or V_PERCENT) instead of or in addition to this? Why not just set to 100% instead of having a "V_UP" command?

V_STOP - stop blinds in the middle of motion.

This is a bit tricky too - it's time dependent and latency becomes important, if you want to set a given position manually. So maybe it's just about canceling any previous command whereever the blinds are, eg: for safety. I'm not really sure how this is used in a script.

V_LOCK_STATUS - 1=Locker, 0 = Unlocked

Again, is this different than other binary set/query dynamics, or just different labels?

V_DUST_LEVEL - ug/m3

One of the issues I mentioned in the other thread is uncalibrated values. That comes up with some sensors. For example, the sensor's values may not be convertable to ug/m3, it might measure particle count without mass, or size without density to compute mass, or only particles within a given range. (Light is like this too, by the way, as a sensor's true output is not lux, but lux-within-a-given-wavelength-range-with_weighting, but we skip past that).

V_VOLTS - Volt <int or float>
V_AMPS - Ampere <int or float>

Discussed in another message. OK if the thing we are really measuring IS voltage or current.

Anyway, this is moving forward despite my questions, and it's meeting my desire to have each of these defined more meaningfully. Thanks!

[Zeph]

I see four concepts involved in describing variables types here.

1. Simplest is format, like int or float (or what size of int, signed or not). OK, that's getting separated out and is pretty well understood.

2. Another is what I'm going to call dimension - what kind of physical or abstract thing is being measured, like temperature or position or speed or particle density.

3. A related one is units.

The units usually imply the dimension, so it becomes very tempting to conflate dimension and units into just the latter. However sometimes we don't know or need to know standard units. For example, we might have an uncalibrated light dependent resistor, for which we know only two things: we get readings from 0 to 1023, and higher is brighter (or lower is brighter) - but we don't know the lux and don't have the info to compute it. In this case we DO know the dimension (brightness) and the direction. We could record values over time and graph them, we could know when it's getting brighter or dimmer and when it peaks, we could empirically set meanngful thresholds ("when value > 350 at night the lights are still on") - all without knowing the value in Lux. If we cannot report brightness in lux, should we ignore the readings,, lie about the lux, or have a concept of uncalibrated values whose demension and direction are known but whose units are not?

Also, the same units may be used for conceptually different things - like temperature, dew point, and wind chill. So units don't always imply the dimension.

4. Meaning or semantics. Take angle - is it a wind direction or a servo position or what? Even something like temperature gets a bit confused - we are all familiar with air temp, but concepts like temp at 3 meters and at grould level come up. Or input and output temperature for a heating system. Or measured temperature versus thermostat setting. Or how about absolute vs relative humidity?

Now when we come up with a defined list of variable types, it's all too easy to mix and match among these concepts when deciding where to lump and where to split )(and how to name and describe). Avoiding a hodgepodge of concepts is the challenge.

Let me give a small example. Here's one way to slice things:with just dimension and units:

 V_SPEED m/s
 V_ANGLE degrees

And here's a different one, mixing in semantics:

V_WIND m/s
V_WIND_DIRECTION "from direction" in degrees cw from true north

We currently don't have either, we have V_ANGLE and V_WIND, some from conceptual framework. Not criticising, just observing. I'm very sympathetic to how this happens.

Also notice that angle for wind also has an implicit or explicit orientation (where is 0 degrees/radians and which way increases), which is missing from the abstraction of angle.

The problem with combining semantics is that your controlled vocabulary of types can get so large.

My best solution so far is to handle dimension as a mutually exclusive type choice (choose one), and semantics as a tag or attribute (choose zero or more of many). If the semantics are not understood by a given system ("what's manifold water temperature mean, I've not been programmed to understand that?" then at least it understands the temperature dimension and the units. This doesn't make it all easy, but it seems to make it more tractable than just having one mutually exclusive list for to represent all aspects (except format #1 above)

[Yveaux]

@Zeph Most of the concepts and ideas you touch have already been mentioned in the posts above... Did you read them?

[hek]

Ok, I've had a solutions bubbling in my head over night. It will add one byte in the MySensors Req/Set Header but solves a lot of the discussions we've had earlier in other threads as well. Its also replaces both INTERNAL and PRESENTATION command.

--- Moved to the first post ---

[Yveaux]

@hek said:

Where does this fit?
V_DEW_POINT - Degrees Celsius <int or float>

Either it doesn't fit in (let the controller calculate it) or place under S_HUMIDITY.

I'm not completely convinced we need to store it this way and use another byte for each req/set message (which are most common...). I'll let it rest for a while so I might diss your solution later ;-)

[Damme]

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated. (my conclusion - I dont like it)

[hek]

@Damme said:

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated.

Read my post again and explain how you would distinguish a thermometer from a uv sensor on the controller side without it?

[Yveaux]

@hek said:

distinguish a thermometer from a uv sensor on the controller side

IMHO this all depends on the controller used, as I already explained. I'm not familiar with Vera, but e.g. for OpenHAB/MQTT the actual type of a sensor is defined by the OpenHAB configuration. It just receives a value and you need to tell it that node x sensor y is a temperature sensor.
Possibly Vera and other systems are more of a plug-and-play kind and are able to add a sensor depending on its type automatically.

I (for a long time) agree with @Damme but I can imagine there are systems that do require this information.

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

[Damme]

@hek I've been thinking and thinking and even might have an other solution.. but I'll wait with that.

so every piece of data has a S_type and V_type.. V_ must be a fixed unique table (not starting from 0 on every S_type)

I'm more acceptable now than 10 minutes ago.. :)

[Damme]

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

I think for the controllers view it would be best for the node to send every message with sensor id, sensor type, value type and payload. Noone needs to remember anything :) The way back to the node the rest can be omitted, as long as sensor id is there.

[hek]

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

Yes, it would save a byte. But it could be very useful for a simpler controller (which can determine sensor type from each and every message) and when displaying sniffing data ;).

This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

[Damme]

@hek S_node could include stuff as
Reboot
ClearEEprom
Read EEprom byte
Set EEprom byte
besides stream data types.
(to be discussed if any of them are useful or not and if they should be hardcoded or not)

[Yveaux]

@hek said:

when displaying sniffing data

Yeay yeah, blame it on the sniffer :facepunch:
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

This means just 1 bit is needed for a SET/REQ flag

Looks like the protocol discussion is shifting towards this topic. Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

[hek]

@Yveaux said:

Yeay yeah, blame it on the sniffer
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

:) We're talking about a total of just 8 bytes header here. And if we remove routing just 5 bytes brimful with useful information about the payload.

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

:question: I do not understand this.

[Yveaux]

@hek said:

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.
I do not understand this.

Probably because I didn't understand you then...
I interpret your post as using a single message structure for either set or req messages. A single bit is used to distinguish between the two, right?

[hek]

@Yveaux said:

A single bit is used to distinguish between the two, right?

Yes.

[Yveaux]

@hek What I meant was to just have two separate commands for set & req (like it is now in 1.3 & 1.4b) but use the same messgae structure to send these messages. The command value will be used to differentiate between set & req.
No need to reserve a bit in the message this way.
Hope its clear now.... :dart:

[Zeph]

@hek
First off - I think this taxonomy of sensactuator values is getting better each time, and I appreciate your mixture of creativity and receptivity as the leader in this effort. Good work.

This is not dissing, more like reflecting on what I see more than making judgements.

The S_ and V_ enums (controlled vocabularies) are becoming relatively more orthogonal rather than confusing people by appearing to cover similar conceptual territory twice but in different way, as the earlier versions inherited from Vera tended to do. So for example, S_TEMPERATURE has V_LEVEL rather than V_TEMPERATURE.

One result is that to understand what a transmitted value (number) represents, you would now need to look at both the S_ code and the V_code (as a good thing). The "dimension" and "units" concepts for understanding a V_LEVEL value depend on the S_ code associated with the variable. In a sense there would now be a two-byte combined expansion of the concepts often formerly described in the one byte V_ code. (So instead of V_TEMPERATURE, there is V_LEVEL+S_TEMPERATURE and you have to look at both to see that the number is temperature in degrees C).I think this is getting cleaner.

The semantics or meaning of a variable are still somewhat distributed between the S and V codes. For example V_LEVEL and V_ANGLE have very little semantic content, relying almost entirely on the S code to provide that, but V_ARMED vs V_TRIPPED do have semantic content. This is just an observation; I'm still letting it sink in.

A big question is the relationship of node children with S_ code - both are called sensors but they need not be the same thing. For this discussion a "Child" is an addressable subunit of a node, numbered 0,1,2... and each child can contain a set of variables. An S code is a one byte code associated with each variable (along with a V code). In an earlier discussion where I was questioning the value of the "Sensor" level of the heirarchy you made the point that there could be value in grouping several variables that are logically tied to "the same thing". The Child concept does that, it provides an addressable container for one or more variables.

At issue is whether there is a fixed relationship beween the Child and the S_ code, whether all variables within the same Child container must have the same S_ code or not.

• If all variables in a given Child must have the same S_ code, then as somebody else pointed out this could be sent to the hub once during node integration and stored there (associated with the given node/child), rather than being transmitted with each value. But you cannot include variables for temperature and humidity in the same Child, even if logically that's what you want to convey (say you have two DHT-11's on the same node, you can't just make them two Children each with a temp and humidity).

• If each variable within a given Child can have its own S code, then the S code and V code become a two byte descriptor of the variable, and Child is a flexible container used to represent "these variables go together" and only that.

Which of these relationships between a Child (addressable subunit of a node) and S code (one byte code that along with V code helps interpret a given variable) did you have in mind?

[Damme]

@hek All these changes are more 2.0 than 1.4.. :)

Edit;
Btw, All these extra bytes S_ and V_ to describe a payload is good for mqtt also.
And this is why we need some more command types (i.e. Stream for OTA update etc. more space for payload smaller header)

[Zeph]

@hek

I see the edited version with MIN, MAX and RESET's added. This takes the orthogonality of V and S codes even further, and it seems like a good direction.

And it's good to see a start on better addressing one of the complicated aspects of this framework = accumulation periods (for Max/Min/Average/Total).

But is V_RESET a variable or a command, conceptually? Is is something sent to the node (probably) or reported by the hub (probably not)?

This brings up another broad conceptual issue in creating a taxonomny like this - the difference between values and command/events. Many software frameworks have both concepts, for example class member variables and class funtions/events. (In electrical terms. this is somewhat like the "Level" vs "Edge or Trigger" distinction).

In the current taxonomy, command/event and value are kind of intermingled. I think it will be important to document which is which.

There are a couple of approaches. One is to define a list of commands (eg; C_RESET) distince from values (V_LEVEL); the other is to have a well defined way of using variables as pseudo-commands.

For one possible example of the latter, one could say that for any variable defined as a command variable:

• when the variable goes from 0 to 1, the node will take the defined action
• the hub will send the node a "Set" command to set the value to 1; if it was 0, then the action is taken
• Setting to 0 has no effect
• the node will set the value to 0 when the action is completed (may be immediate)
• the hub can query the current value of the variable to see if there is any action still pending or not

(This might be exactly what you had in mind, or you might have had a variation in mind - I'm just saying that it should be clearly defined and documented).

Then we could say that V_RESET is a command variable and thus shares the standard dynamics, where the specific "action" is to restart the accumulation period. And there could be other command variables, each documented as such.

(The Event concept is similar to a command in that it's a transition rather than a level, but used as reports from the node to the hub. Not addressed yet)