2.0 Discussion: Units, sensor types and protocol

Damme

@hek I know, ist more 'need moisture' or not :) Hmm... V_Moisture, (relative) 0-255 / 0-100% / True | False

Yveaux

@hek Can we also pass battery level as just a sensor value? There's no real need to pass this as an internal message, is it?

Zeph

First off, this is a very positive step forward!

Now some details
@hek said:
> I think the impedance-variable comes from the RFXTRX plugin. And it is related to body fat measurements for body scales.

I think that we need to be clear about the core concept of what we intend to measure.

Resistance or impedance generally don't make much sense, as they are usually not the conceptual thing being measured but an intermediate value used in calculating it.

This is also true of Volts and Amps, except that sometimes we really do want volts and amps (for power monitoring).

But consider a Light Dependent Resistor in a bridge circuit.
We could report the value as ADC readings 0..1023.
Or we could convert that to volts 0..3.3 or 0..5.0 or whatever.
Or we could convert that to LDR resistance (knowing the other resistor value) in ohms.
Or we could do some calcs and try to report the result in Lux.

Since the conceptual thing we are trying to measure is light intensity, the best option would be to report it in Lux if possible. Not as ohms, volts, or ADC readings.

So any time we see something reporting as resistance or impedance, we should ask if that's the core concept we need to report, or an incompletely processed proxy that will be hard for the server to understand.

Likewise with volts or amps - are we really trying to report temperature or humidty or what? If we are monitoring a battery or the AC mains, then maybe volts it exactly right, but we need to think it out first.

So who in a home automation context cares about resistance or impedance as the core thing being measured?

Zeph

@hek said:

V_TEMPERATURE - Degrees celsius <int or float>
V_HUMIDITY - Humidity percentage 0-100% <int or float>

Can any of the values be int or float? Or just these two?

V_DIMMER - Percentage 0-100% (used for lights, blinds and curtains)

The core concept here is V_PERCENT. In one home automation system there might be only one included concept, the Dimmer, and you'd have to use that for blinds as well as dimmers. In another home automation system there might be separate UI widgets for blinds and light dimmers. The sensactuator node's code should not need to change either way. Mapping a given V_PERCENT variable to UI concepts of blinds or dimmers is conceptually part of the configuration for a given (version of a) home automation system.

V_FORECAST Whether forecast. One of "stable", "sunny", "cloudy", "unstable", "thunderstorm" or "unknown"

Difficult one. That is very definitely not the controlled vocabulary used by all forecasting sources, and mapping into this list is going to be difficult. "Scattered showers" and "partly cloudy" and "light rain" and "foggy" are not going to be easily represented. What about stable/sunny versus stable/rainy? Are stable/unstable mutually exclusive choices with the others, or optional additional attributes for any of the other choices?

Maybe this has to be an open vocabulary with a text string? What text they will get depends on their source.

V_RAIN - mm
V_WIND - m/s (average wind speed during last report period)
V_GUST - m/s (highest wind speed during since last report)
V_WATT - Watt <int>
V_KWH - kWh <int or float>

What is the report period and what resets it? Last minute, last hour, last 24 hours, since midnight, since the reset button was pushed, or using automatic episode detection (I was just looking at a rain sensor which uses 8 hours of no rain to separate rain accumulation events). Without some knowledge of the reporting period these values cannot be meaningfully interpreted.

V_RAIN could be a forever incrementing total if the host wants to do its own deltas and define it's own report periods (maybe several of them).

V_WIND is a bit more difficult. If we had guaranteed delivery, we could just average since the last report from the node. Of course, some weather stations may do their own averaging, or the node could average many readings. But how does the host know?

From my perspective these "time period sum or average" values would ideally involve:

an OTA format which loses no useful information available from the sensors or peripherals
a flexible gateway (or adapter) which reformulates into time units meaningful to the HA controller software
Let me make up an example. Suppose the sensor reported wind speed averages over 5 minute periods and the HA controller wanted to report it over one hour periods. The gateway could average 12 reports - for either the last 60 minutes, or for on-the-hour, as needed. Or "new rainfall" could be reported in 10 minute increments, but accumulated since midnight by the gateway, if that's what the controller prefers. Different controllers might use different accumulation periods; and what is sent to a controller could vary from what was uploaded to a cloud service. The node should not be adapting to the controller, it should send all available and useful info and the gateway or adapter should adjust - sort of like degrees F but more complicated because it might involve keeping running totals for averaging.

V_ANGLE - radians <float or int>

Here's one of those "generic vs specific" ones. Like V_PERCENT suggested above, V_ANGLE has no implications for what is being measured, only the units. It could be wind direction, it could be the plantation shutter position, it could be the solar panel tilt. How does the host know?

(This is the kind of thing that prompted my suggestions on another thread to separate the concept of units from what is being measured.)

An int for radians doesn't make much sense (unless we have a scaling factor).
I think degrees is a reasonable compromise - it's more precise than any sensor we're going to afford (or than the noise in the data itself would make moot), so it loses no meaningful accuracy. And virtually no sensor is going to report in radians. Some report in degrees. If it reports in 16 compass points or such, that can be easily converted to degrees, without even using floating point.

V_WEIGHT - kg <int, float>
V_DISTANCE - Cm <int, float>
V_IMPEDANCE - Ohm

See other message about intermediate calculations and the true conceptual thing being measured, (with impedance as an example). What does a distance of 240cm mean?

V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed
V_TRIPPED - Tripped status of a security sensor. 1=Tripped, 0=Untripped

Here the underlying concept is "V_BINARY", with a variety of associated labels for 1 and 0 based on what is being reported or controlled. On/Off, Open/Closed, Armed/Unarmed, Tripped/Untripped, Locked/Unlocked, Enabled/Disabled, etc. Do we create an ever expanding set of binary options as different variable types? Or is it functionally one variable type (or a small number with definably different dynamics from each other) with additional attributes (like the labels used for 1 and 0)?

V_SCENE_ON - Scene number <int>
V_SCENE_OFF - Scene number <int>

Now we are conflating events or commands with statuses or values. We turn on a light by setting it to 1 (or 100%) and we turn it off by setting it to 0 (or 0%). This leverages the same "set value to" underlaying functionality, and allows a simple and consistent query concept (what is the current value of the light?) What is the advantage of treating a Scene differently?

Scenes are a bit confusing, because they may or may not be mutually exclusive. They are neither obviously "checkboxes" or "radio buttons".

V_HEATER_MODE - Heater mode. One of "Off", "HeatOn", "CoolOn", or "AutoChangeOver"

How do we set the themostat? What about Fan?

V_HEATER_STATUS - 1=On, 0=Off. Heater power switch

What about CoolOn mode? If it's cooling, is V_HEATER_STATUS on or off?
.

V_LIGHT_LEVEL - 0-100% <int>

Is this a sensor (alternative to V_Lux) or an actuator (different than V_DIMMER/V_PERCENT above)?

V_UP - command which modes blinds ful up. Use DIMMER to specify percentage
V_DOWN - command which moved blinds to full down. Use DIMMER to specify percentage

Use V_DIMMER (or V_PERCENT) instead of or in addition to this? Why not just set to 100% instead of having a "V_UP" command?

V_STOP - stop blinds in the middle of motion.

This is a bit tricky too - it's time dependent and latency becomes important, if you want to set a given position manually. So maybe it's just about canceling any previous command whereever the blinds are, eg: for safety. I'm not really sure how this is used in a script.

V_LOCK_STATUS - 1=Locker, 0 = Unlocked

Again, is this different than other binary set/query dynamics, or just different labels?

V_DUST_LEVEL - ug/m3

One of the issues I mentioned in the other thread is uncalibrated values. That comes up with some sensors. For example, the sensor's values may not be convertable to ug/m3, it might measure particle count without mass, or size without density to compute mass, or only particles within a given range. (Light is like this too, by the way, as a sensor's true output is not lux, but lux-within-a-given-wavelength-range-with_weighting, but we skip past that).

V_VOLTS - Volt <int or float>
V_AMPS - Ampere <int or float>

Discussed in another message. OK if the thing we are really measuring IS voltage or current.

Anyway, this is moving forward despite my questions, and it's meeting my desire to have each of these defined more meaningfully. Thanks!

Zeph

I see four concepts involved in describing variables types here.

Simplest is format, like int or float (or what size of int, signed or not). OK, that's getting separated out and is pretty well understood.
Another is what I'm going to call dimension - what kind of physical or abstract thing is being measured, like temperature or position or speed or particle density.
A related one is units.

The units usually imply the dimension, so it becomes very tempting to conflate dimension and units into just the latter. However sometimes we don't know or need to know standard units. For example, we might have an uncalibrated light dependent resistor, for which we know only two things: we get readings from 0 to 1023, and higher is brighter (or lower is brighter) - but we don't know the lux and don't have the info to compute it. In this case we DO know the dimension (brightness) and the direction. We could record values over time and graph them, we could know when it's getting brighter or dimmer and when it peaks, we could empirically set meanngful thresholds ("when value > 350 at night the lights are still on") - all without knowing the value in Lux. If we cannot report brightness in lux, should we ignore the readings,, lie about the lux, or have a concept of uncalibrated values whose demension and direction are known but whose units are not?

Also, the same units may be used for conceptually different things - like temperature, dew point, and wind chill. So units don't always imply the dimension.

Meaning or semantics. Take angle - is it a wind direction or a servo position or what? Even something like temperature gets a bit confused - we are all familiar with air temp, but concepts like temp at 3 meters and at grould level come up. Or input and output temperature for a heating system. Or measured temperature versus thermostat setting. Or how about absolute vs relative humidity?

Now when we come up with a defined list of variable types, it's all too easy to mix and match among these concepts when deciding where to lump and where to split )(and how to name and describe). Avoiding a hodgepodge of concepts is the challenge.

Let me give a small example. Here's one way to slice things:with just dimension and units:

 V_SPEED m/s
 V_ANGLE degrees

And here's a different one, mixing in semantics:

V_WIND m/s
V_WIND_DIRECTION "from direction" in degrees cw from true north

We currently don't have either, we have V_ANGLE and V_WIND, some from conceptual framework. Not criticising, just observing. I'm very sympathetic to how this happens.

Also notice that angle for wind also has an implicit or explicit orientation (where is 0 degrees/radians and which way increases), which is missing from the abstraction of angle.

The problem with combining semantics is that your controlled vocabulary of types can get so large.

My best solution so far is to handle dimension as a mutually exclusive type choice (choose one), and semantics as a tag or attribute (choose zero or more of many). If the semantics are not understood by a given system ("what's manifold water temperature mean, I've not been programmed to understand that?" then at least it understands the temperature dimension and the units. This doesn't make it all easy, but it seems to make it more tractable than just having one mutually exclusive list for to represent all aspects (except format #1 above)

Yveaux

@Zeph Most of the concepts and ideas you touch have already been mentioned in the posts above... Did you read them?

hek

Ok, I've had a solutions bubbling in my head over night. It will add one byte in the MySensors Req/Set Header but solves a lot of the discussions we've had earlier in other threads as well. Its also replaces both INTERNAL and PRESENTATION command.

--- Moved to the first post ---

Yveaux

@hek said:

Where does this fit?
V_DEW_POINT - Degrees Celsius <int or float>

Either it doesn't fit in (let the controller calculate it) or place under S_HUMIDITY.

I'm not completely convinced we need to store it this way and use another byte for each req/set message (which are most common...). I'll let it rest for a while so I might diss your solution later ;-)

Damme

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated. (my conclusion - I dont like it)

hek

@Damme said:

Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated.

Read my post again and explain how you would distinguish a thermometer from a uv sensor on the controller side without it?

Yveaux

@hek said:

distinguish a thermometer from a uv sensor on the controller side

IMHO this all depends on the controller used, as I already explained. I'm not familiar with Vera, but e.g. for OpenHAB/MQTT the actual type of a sensor is defined by the OpenHAB configuration. It just receives a value and you need to tell it that node x sensor y is a temperature sensor.
Possibly Vera and other systems are more of a plug-and-play kind and are able to add a sensor depending on its type automatically.

I (for a long time) agree with @Damme but I can imagine there are systems that do require this information.

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

Damme

@hek I've been thinking and thinking and even might have an other solution.. but I'll wait with that.

so every piece of data has a S_type and V_type.. V_ must be a fixed unique table (not starting from 0 on every S_type)

I'm more acceptable now than 10 minutes ago.. :)

Damme

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

I think for the controllers view it would be best for the node to send every message with sensor id, sensor type, value type and payload. Noone needs to remember anything :) The way back to the node the rest can be omitted, as long as sensor id is there.

hek

@Yveaux said:

[dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

Yes, it would save a byte. But it could be very useful for a simpler controller (which can determine sensor type from each and every message) and when displaying sniffing data ;).

This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

Damme

@hek S_node could include stuff as
Reboot
ClearEEprom
Read EEprom byte
Set EEprom byte
besides stream data types.
(to be discussed if any of them are useful or not and if they should be hardcoded or not)

Yveaux

@hek said:

when displaying sniffing data

Yeay yeah, blame it on the sniffer :facepunch:
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

This means just 1 bit is needed for a SET/REQ flag

Looks like the protocol discussion is shifting towards this topic. Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

hek

@Yveaux said:

Yeay yeah, blame it on the sniffer
Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

:) We're talking about a total of just 8 bytes header here. And if we remove routing just 5 bytes brimful with useful information about the payload.

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

:question: I do not understand this.

Yveaux

@hek said:

Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.
I do not understand this.

Probably because I didn't understand you then...
I interpret your post as using a single message structure for either set or req messages. A single bit is used to distinguish between the two, right?

hek

@Yveaux said:

A single bit is used to distinguish between the two, right?

Yes.

Yveaux

@hek What I meant was to just have two separate commands for set & req (like it is now in 1.3 & 1.4b) but use the same messgae structure to send these messages. The command value will be used to differentiate between set & req.
No need to reserve a bit in the message this way.
Hope its clear now.... :dart:

2.0 Discussion: Units, sensor types and protocol

19

11.7k

11.2k

113.0k