2.0 Discussion: Units, sensor types and protocol


  • Admin

    To simplify sketches even further we've discussed and decided to let sensors send in standard SI units and let controller convert them to the appropriate format.

    This will also remove the need for a unit-config exchange with controller at sensor startup.

    The new over-the-air messages has changed quite much with a more logical structure that can handle extension and even a completely different network layer if needed.

    The ongoing work can be found here:
    https://github.com/henrikekblad/Arduino/blob/development/libraries/MySensors/MyMessage.h

    Will not bring this into development until it actually compiles.

    Feel free to give feedback and report any missed sensor-types.

    In parallell we should probably discuss the serial-protocol for 2.0. Is it time for json perhaps?


  • Hero Member

    How about

    V_RELAY_STATUS 1=Active, 0=Deactive



  • @HEK I'll bite:

    Use under scores consistently ? eg V_RAINRATE and V_LIGHT_LEVEL

    The units to be used here can be clearer:
    V_VOLTAGE ---> V_VOLTS
    V_CURRENT ---> V_AMPS

    Where the base unit is not used - should the name reflect that? eg V_DISTANCE --> V_DISTANCE_CM

    Electrical impedance and resistance are not the same thing- suspect resistance is intended here?
    V_IMPEDANCE --> V_RESISTANCE

    HUM is a bit vague - suggest: V_HUM --> V_HUMIDITY

    V_DIRECTION may be confused with an angle or up/down. Could use V_COMPASS_POINT, but that may be confused with magnetic or true north!
    V_LIGHT could be confused with V_LIGHT_LEVEL. Could use V_LIGHT_STATUS
    V_TEMP while often used, can be confused with a temporary variable? Could use V_TEMPERATURE
    V_VOLUME could be confused with audio volume? Could use V_CUBIC_MEASURE
    V_HEATER_MODE for V_HEATER?

    Could add V_DEW_POINT

    STATUS always be used for binary values eg on/off, locked/unlocked?
    V_LOCK_STATUS
    V_LIGHT_STATUS
    V_HEATER_STATUS


  • Hero Member

    @hek could we pleas add a V_pH. Or V_PH_LEVEL


  • Mod

    @hek said:

    V_HUM - Humidity percentage 0-100% <int>

    No reason to limit to int, can also be float.
    Why do you mention the data types for some and not for most? Mysensors doesn't really care....


  • Mod

    @hek said:

    V_WIND - m/s (average wind speed during last hour)
    V_GUST - m/s (highest wind speed during last hour)

    Why average/highest over last hour? My weather station reports wind speed every minute and I like to see the minute detail instead of averaging everything over an hour.


  • Mod

    @hek said:

    Do we need another one for lux?

    Yes please.


  • Mod

    @hek said:

    V_DUST_LEVEL - ug/m3

    g/m3 if you standardize on SI
    Same for distance BTW. Might be more...


  • Mod

    @hek said:

    V_DIRECTION

    Maybe just return angle ( in radians and call it V_ANGLE) then textual indication. This way higher resolutions can also be supported.

    Sorry for all the separate messages. Works more convenient on a tablet...


  • Mod

    I would like a way to configure a sensor from the gateway, to e.g calibrate a sensor or store some ID. Maybe a V_CONFIG can be added, where childid indicates the actual variable written. Or define a few V_CONFIG0,V_CONFIG1 etc.
    What do you think?


  • Admin

    @Yveaux said:

    g/m3 if you standardize on SI
    Same for distance BTW. Might be more...

    I choose to scale the SI units to something relevant for thing getting measured. E.g. g/m3 would give a VERY small small value for dust.

    Distance is mainly for the distance sensor now, which only has a small range.

    Rain - measure in meters would also render very small numbers.

    So, until we have a scaling factor in the protocol or something similar we'll have select a proper scaling at the source and agree on it between sensor and controller.


  • Admin

    @Yveaux said:

    V_CONFIG0,V_CONFIG1

    A few config-placeholders would be a nice.


  • Admin

    @gregl said:

    V_PH_LEVEL

    Added.


  • Admin

    @Yveaux said:

    Why average/highest over last hour? My weather station reports wind speed every minute and I like to see the minute detail instead of averaging everything over an hour.

    I'll change the definition to "over the last report period".



  • V_ANGLE sounds good. With a bit of (floating point) maths, all can be calculated.


  • Admin

    @ServiceXp said:

    V_RELAY_STATUS 1=Active, 0=Deactive

    Not sure what differs this form V_LIGHT_STATUS.
    Maybe we need a better word than LIGHT that describes a thing-that-can-be-turned-on-or-off?


  • Mod

    @hek said:

    So, until we have a scaling factor in the protocol or something similar we'll have select a proper scaling at the source and agree on it between sensor and controller.

    When values are sent as floats it doesn't matter. Scaling is implicitly store in the floating point value when sent as binary float. When transfered as ASCII you're right, but that isn't required with 1.4 anymore...


  • Mod

    @hek said:

    a thing-that-can-be-turned-on-or-of

    Hmmmm.... Let me think.... a switch ? 😉


  • Admin

    @a-lurker said:

    Electrical impedance and resistance are not the same thing- suspect resistance is intended here?
    V_IMPEDANCE --> V_RESISTANCE

    I think the impedance-variable comes from the RFXTRX plugin. And it is related to body fat measurements for body scales.

    http://code.mios.com/trac/mios_rfxtrx/browser/trunk/L_RFXtrx.lua

    If I understand it correctly Impedace is used for AC and resistance for DC. Not sure what would be most useful in this setup.


  • Admin

    @Yveaux said:

    When transfered as ASCII you're right, but that isn't required with 1.4 anymore.

    Most users still use the serial protocol. Which means strings like "0.000003423" must be generated and transferred to the controller from gateway.


  • Admin

    @Yveaux said:

    Hmmmm.... Let me think.... a switch ?

    Haha.. yeah.. 👍


  • Code Contributor

    @hek Moisture or Water detecded... Might be 2 different ?
    Thinking different applications: Moisture in ground. (I.e. Water needed? Ground dry?) And Water level, hmm, if you have a fluid level meter to detect overfill. or leakage. Fluid flow


  • Mod

    @hek said:

    strings like "0.000003423" must be generated and transferred

    Ok, but is this a problem? You could even pass it as 3.423e-6....
    You don't want this string on air if you can transfer the same thing in 4 bytes, that's true.
    I only want to state that if you transfer it purely in SI units, without scaling, you always know what the actual value is.


  • Mod

    I'm still in doubt if we really need all these different value types...
    In principle a (light) switch, water level sensor, burglar alarm, heater status <fill in your own binary switch> all just return or take 0 or 1. V_BINARY should suffice for all of them.
    For other values you see the struggle between defining them as bare units (V_AMPS, V_VOLTS, V_CUBIC_MEASURE) and function (V_FLOW, V_WEIGHT, V_HUMIDITY).
    Can't we just live with transfering values by specifying their units only and leave the interpretation of it to the controller (e.g. V_AMPS, V_BINARY, V_PERCENTAGE) ?
    I'm not familiar with Vera, but e.g. the OpenHAB/MQTT combination doesn't care at all what type of value is exchanged. It's up to the configuration of OpenHAB how a value is interpreted.


  • Admin

    @Damme said:

    Moisture or Water detecded... Might be 2 different ?

    For water detected I would probably use V_TRIPPED.

    Hmm.. Soil Moisture.. I have no idea how to represent this. Help me! The ebay sensors we're using isn't super accurate and comes uncalibrated.


  • Code Contributor

    @hek I know, ist more 'need moisture' or not 🙂 Hmm... V_Moisture, (relative) 0-255 / 0-100% / True | False


  • Mod

    @hek Can we also pass battery level as just a sensor value? There's no real need to pass this as an internal message, is it?


  • Hero Member

    First off, this is a very positive step forward!

    Now some details
    @hek said:
    > I think the impedance-variable comes from the RFXTRX plugin. And it is related to body fat measurements for body scales.

    I think that we need to be clear about the core concept of what we intend to measure.

    Resistance or impedance generally don't make much sense, as they are usually not the conceptual thing being measured but an intermediate value used in calculating it.

    This is also true of Volts and Amps, except that sometimes we really do want volts and amps (for power monitoring).

    But consider a Light Dependent Resistor in a bridge circuit.
    We could report the value as ADC readings 0..1023.
    Or we could convert that to volts 0..3.3 or 0..5.0 or whatever.
    Or we could convert that to LDR resistance (knowing the other resistor value) in ohms.
    Or we could do some calcs and try to report the result in Lux.

    Since the conceptual thing we are trying to measure is light intensity, the best option would be to report it in Lux if possible. Not as ohms, volts, or ADC readings.

    So any time we see something reporting as resistance or impedance, we should ask if that's the core concept we need to report, or an incompletely processed proxy that will be hard for the server to understand.

    Likewise with volts or amps - are we really trying to report temperature or humidty or what? If we are monitoring a battery or the AC mains, then maybe volts it exactly right, but we need to think it out first.

    So who in a home automation context cares about resistance or impedance as the core thing being measured?


  • Hero Member

    @hek said:

    V_TEMPERATURE - Degrees celsius <int or float>
    V_HUMIDITY - Humidity percentage 0-100% <int or float>

    Can any of the values be int or float? Or just these two?

    V_DIMMER - Percentage 0-100% (used for lights, blinds and curtains)

    The core concept here is V_PERCENT. In one home automation system there might be only one included concept, the Dimmer, and you'd have to use that for blinds as well as dimmers. In another home automation system there might be separate UI widgets for blinds and light dimmers. The sensactuator node's code should not need to change either way. Mapping a given V_PERCENT variable to UI concepts of blinds or dimmers is conceptually part of the configuration for a given (version of a) home automation system.

    V_FORECAST Whether forecast. One of "stable", "sunny", "cloudy", "unstable", "thunderstorm" or "unknown"

    Difficult one. That is very definitely not the controlled vocabulary used by all forecasting sources, and mapping into this list is going to be difficult. "Scattered showers" and "partly cloudy" and "light rain" and "foggy" are not going to be easily represented. What about stable/sunny versus stable/rainy? Are stable/unstable mutually exclusive choices with the others, or optional additional attributes for any of the other choices?

    Maybe this has to be an open vocabulary with a text string? What text they will get depends on their source.

    V_RAIN - mm
    V_WIND - m/s (average wind speed during last report period)
    V_GUST - m/s (highest wind speed during since last report)
    V_WATT - Watt <int>
    V_KWH - kWh <int or float>

    What is the report period and what resets it? Last minute, last hour, last 24 hours, since midnight, since the reset button was pushed, or using automatic episode detection (I was just looking at a rain sensor which uses 8 hours of no rain to separate rain accumulation events). Without some knowledge of the reporting period these values cannot be meaningfully interpreted.

    V_RAIN could be a forever incrementing total if the host wants to do its own deltas and define it's own report periods (maybe several of them).

    V_WIND is a bit more difficult. If we had guaranteed delivery, we could just average since the last report from the node. Of course, some weather stations may do their own averaging, or the node could average many readings. But how does the host know?

    From my perspective these "time period sum or average" values would ideally involve:

    • an OTA format which loses no useful information available from the sensors or peripherals
    • a flexible gateway (or adapter) which reformulates into time units meaningful to the HA controller software
      Let me make up an example. Suppose the sensor reported wind speed averages over 5 minute periods and the HA controller wanted to report it over one hour periods. The gateway could average 12 reports - for either the last 60 minutes, or for on-the-hour, as needed. Or "new rainfall" could be reported in 10 minute increments, but accumulated since midnight by the gateway, if that's what the controller prefers. Different controllers might use different accumulation periods; and what is sent to a controller could vary from what was uploaded to a cloud service. The node should not be adapting to the controller, it should send all available and useful info and the gateway or adapter should adjust - sort of like degrees F but more complicated because it might involve keeping running totals for averaging.

    V_ANGLE - radians <float or int>

    Here's one of those "generic vs specific" ones. Like V_PERCENT suggested above, V_ANGLE has no implications for what is being measured, only the units. It could be wind direction, it could be the plantation shutter position, it could be the solar panel tilt. How does the host know?

    (This is the kind of thing that prompted my suggestions on another thread to separate the concept of units from what is being measured.)

    An int for radians doesn't make much sense (unless we have a scaling factor).
    I think degrees is a reasonable compromise - it's more precise than any sensor we're going to afford (or than the noise in the data itself would make moot), so it loses no meaningful accuracy. And virtually no sensor is going to report in radians. Some report in degrees. If it reports in 16 compass points or such, that can be easily converted to degrees, without even using floating point.

    V_WEIGHT - kg <int, float>
    V_DISTANCE - Cm <int, float>
    V_IMPEDANCE - Ohm

    See other message about intermediate calculations and the true conceptual thing being measured, (with impedance as an example). What does a distance of 240cm mean?

    V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed
    V_TRIPPED - Tripped status of a security sensor. 1=Tripped, 0=Untripped

    Here the underlying concept is "V_BINARY", with a variety of associated labels for 1 and 0 based on what is being reported or controlled. On/Off, Open/Closed, Armed/Unarmed, Tripped/Untripped, Locked/Unlocked, Enabled/Disabled, etc. Do we create an ever expanding set of binary options as different variable types? Or is it functionally one variable type (or a small number with definably different dynamics from each other) with additional attributes (like the labels used for 1 and 0)?

    V_SCENE_ON - Scene number <int>
    V_SCENE_OFF - Scene number <int>

    Now we are conflating events or commands with statuses or values. We turn on a light by setting it to 1 (or 100%) and we turn it off by setting it to 0 (or 0%). This leverages the same "set value to" underlaying functionality, and allows a simple and consistent query concept (what is the current value of the light?) What is the advantage of treating a Scene differently?

    Scenes are a bit confusing, because they may or may not be mutually exclusive. They are neither obviously "checkboxes" or "radio buttons".

    V_HEATER_MODE - Heater mode. One of "Off", "HeatOn", "CoolOn", or "AutoChangeOver"

    How do we set the themostat? What about Fan?

    V_HEATER_STATUS - 1=On, 0=Off. Heater power switch

    What about CoolOn mode? If it's cooling, is V_HEATER_STATUS on or off?
    .

    V_LIGHT_LEVEL - 0-100% <int>

    Is this a sensor (alternative to V_Lux) or an actuator (different than V_DIMMER/V_PERCENT above)?

    V_UP - command which modes blinds ful up. Use DIMMER to specify percentage
    V_DOWN - command which moved blinds to full down. Use DIMMER to specify percentage

    Use V_DIMMER (or V_PERCENT) instead of or in addition to this? Why not just set to 100% instead of having a "V_UP" command?

    V_STOP - stop blinds in the middle of motion.

    This is a bit tricky too - it's time dependent and latency becomes important, if you want to set a given position manually. So maybe it's just about canceling any previous command whereever the blinds are, eg: for safety. I'm not really sure how this is used in a script.

    V_LOCK_STATUS - 1=Locker, 0 = Unlocked

    Again, is this different than other binary set/query dynamics, or just different labels?

    V_DUST_LEVEL - ug/m3

    One of the issues I mentioned in the other thread is uncalibrated values. That comes up with some sensors. For example, the sensor's values may not be convertable to ug/m3, it might measure particle count without mass, or size without density to compute mass, or only particles within a given range. (Light is like this too, by the way, as a sensor's true output is not lux, but lux-within-a-given-wavelength-range-with_weighting, but we skip past that).

    V_VOLTS - Volt <int or float>
    V_AMPS - Ampere <int or float>

    Discussed in another message. OK if the thing we are really measuring IS voltage or current.

    Anyway, this is moving forward despite my questions, and it's meeting my desire to have each of these defined more meaningfully. Thanks!


  • Hero Member

    I see four concepts involved in describing variables types here.

    1. Simplest is format, like int or float (or what size of int, signed or not). OK, that's getting separated out and is pretty well understood.

    2. Another is what I'm going to call dimension - what kind of physical or abstract thing is being measured, like temperature or position or speed or particle density.

    3. A related one is units.

    The units usually imply the dimension, so it becomes very tempting to conflate dimension and units into just the latter. However sometimes we don't know or need to know standard units. For example, we might have an uncalibrated light dependent resistor, for which we know only two things: we get readings from 0 to 1023, and higher is brighter (or lower is brighter) - but we don't know the lux and don't have the info to compute it. In this case we DO know the dimension (brightness) and the direction. We could record values over time and graph them, we could know when it's getting brighter or dimmer and when it peaks, we could empirically set meanngful thresholds ("when value > 350 at night the lights are still on") - all without knowing the value in Lux. If we cannot report brightness in lux, should we ignore the readings,, lie about the lux, or have a concept of uncalibrated values whose demension and direction are known but whose units are not?

    Also, the same units may be used for conceptually different things - like temperature, dew point, and wind chill. So units don't always imply the dimension.

    1. Meaning or semantics. Take angle - is it a wind direction or a servo position or what? Even something like temperature gets a bit confused - we are all familiar with air temp, but concepts like temp at 3 meters and at grould level come up. Or input and output temperature for a heating system. Or measured temperature versus thermostat setting. Or how about absolute vs relative humidity?

    Now when we come up with a defined list of variable types, it's all too easy to mix and match among these concepts when deciding where to lump and where to split )(and how to name and describe). Avoiding a hodgepodge of concepts is the challenge.

    Let me give a small example. Here's one way to slice things:with just dimension and units:

     V_SPEED m/s
     V_ANGLE degrees
    

    And here's a different one, mixing in semantics:

    V_WIND m/s
    V_WIND_DIRECTION "from direction" in degrees cw from true north
    

    We currently don't have either, we have V_ANGLE and V_WIND, some from conceptual framework. Not criticising, just observing. I'm very sympathetic to how this happens.

    Also notice that angle for wind also has an implicit or explicit orientation (where is 0 degrees/radians and which way increases), which is missing from the abstraction of angle.

    The problem with combining semantics is that your controlled vocabulary of types can get so large.

    My best solution so far is to handle dimension as a mutually exclusive type choice (choose one), and semantics as a tag or attribute (choose zero or more of many). If the semantics are not understood by a given system ("what's manifold water temperature mean, I've not been programmed to understand that?" then at least it understands the temperature dimension and the units. This doesn't make it all easy, but it seems to make it more tractable than just having one mutually exclusive list for to represent all aspects (except format #1 above)


  • Mod

    @Zeph Most of the concepts and ideas you touch have already been mentioned in the posts above... Did you read them?


  • Admin

    Ok, I've had a solutions bubbling in my head over night. It will add one byte in the MySensors Req/Set Header but solves a lot of the discussions we've had earlier in other threads as well. Its also replaces both INTERNAL and PRESENTATION command.

    --- Moved to the first post ---


  • Mod

    @hek said:

    Where does this fit?
    V_DEW_POINT - Degrees Celsius <int or float>

    Either it doesn't fit in (let the controller calculate it) or place under S_HUMIDITY.

    I'm not completely convinced we need to store it this way and use another byte for each req/set message (which are most common...). I'll let it rest for a while so I might diss your solution later 😉


  • Code Contributor

    Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated. (my conclusion - I dont like it)


  • Admin

    @Damme said:

    Could someone please explain to me why we have S_ types at all? I really dont get it.. I Only think it makes things more complicated.

    Read my post again and explain how you would distinguish a thermometer from a uv sensor on the controller side without it?


  • Mod

    @hek said:

    distinguish a thermometer from a uv sensor on the controller side

    IMHO this all depends on the controller used, as I already explained. I'm not familiar with Vera, but e.g. for OpenHAB/MQTT the actual type of a sensor is defined by the OpenHAB configuration. It just receives a value and you need to tell it that node x sensor y is a temperature sensor.
    Possibly Vera and other systems are more of a plug-and-play kind and are able to add a sensor depending on its type automatically.

    I (for a long time) agree with @Damme but I can imagine there are systems that do require this information.

    [dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]


  • Code Contributor

    @hek I've been thinking and thinking and even might have an other solution.. but I'll wait with that.

    so every piece of data has a S_type and V_type.. V_ must be a fixed unique table (not starting from 0 on every S_type)

    I'm more acceptable now than 10 minutes ago.. 🙂


  • Code Contributor

    @Yveaux said:

    [dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

    I think for the controllers view it would be best for the node to send every message with sensor id, sensor type, value type and payload. Noone needs to remember anything 🙂 The way back to the node the rest can be omitted, as long as sensor id is there.


  • Admin

    @Yveaux said:

    [dissing start] Did you think about sending the sensor type only once, durig presentation? There really is no need to send it with each set/req message as it is static for the duration of the connection. For Vera and the like this probably means you should buffer these values in the gateway... [dissing end]

    Yes, it would save a byte. But it could be very useful for a simpler controller (which can determine sensor type from each and every message) and when displaying sniffing data ;).

    This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.


  • Code Contributor

    @hek S_node could include stuff as
    Reboot
    ClearEEprom
    Read EEprom byte
    Set EEprom byte
    besides stream data types.
    (to be discussed if any of them are useful or not and if they should be hardcoded or not)


  • Mod

    @hek said:

    when displaying sniffing data

    Yeay yeah, blame it on the sniffer 👊
    Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

    This means just 1 bit is needed for a SET/REQ flag

    Looks like the protocol discussion is shifting towards this topic. Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.


  • Admin

    @Yveaux said:

    Yeay yeah, blame it on the sniffer
    Making stuff 'easier' to sniff can never be a reason to always send one extra byte...

    🙂 We're talking about a total of just 8 bytes header here. And if we remove routing just 5 bytes brimful with useful information about the payload.

    Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.

    ❓ I do not understand this.


  • Mod

    @hek said:

    Better not reserve a bit to indicate in a set/req message which of the two it is, but use different message indicator values. The actual message structures can still be the same.
    I do not understand this.

    Probably because I didn't understand you then...
    I interpret your post as using a single message structure for either set or req messages. A single bit is used to distinguish between the two, right?


  • Admin

    @Yveaux said:

    A single bit is used to distinguish between the two, right?

    Yes.


  • Mod

    @hek What I meant was to just have two separate commands for set & req (like it is now in 1.3 & 1.4b) but use the same messgae structure to send these messages. The command value will be used to differentiate between set & req.
    No need to reserve a bit in the message this way.
    Hope its clear now.... 🎯


  • Hero Member

    @hek
    First off - I think this taxonomy of sensactuator values is getting better each time, and I appreciate your mixture of creativity and receptivity as the leader in this effort. Good work.

    This is not dissing, more like reflecting on what I see more than making judgements.

    The S_ and V_ enums (controlled vocabularies) are becoming relatively more orthogonal rather than confusing people by appearing to cover similar conceptual territory twice but in different way, as the earlier versions inherited from Vera tended to do. So for example, S_TEMPERATURE has V_LEVEL rather than V_TEMPERATURE.

    One result is that to understand what a transmitted value (number) represents, you would now need to look at both the S_ code and the V_code (as a good thing). The "dimension" and "units" concepts for understanding a V_LEVEL value depend on the S_ code associated with the variable. In a sense there would now be a two-byte combined expansion of the concepts often formerly described in the one byte V_ code. (So instead of V_TEMPERATURE, there is V_LEVEL+S_TEMPERATURE and you have to look at both to see that the number is temperature in degrees C).I think this is getting cleaner.

    The semantics or meaning of a variable are still somewhat distributed between the S and V codes. For example V_LEVEL and V_ANGLE have very little semantic content, relying almost entirely on the S code to provide that, but V_ARMED vs V_TRIPPED do have semantic content. This is just an observation; I'm still letting it sink in.

    A big question is the relationship of node children with S_ code - both are called sensors but they need not be the same thing. For this discussion a "Child" is an addressable subunit of a node, numbered 0,1,2... and each child can contain a set of variables. An S code is a one byte code associated with each variable (along with a V code). In an earlier discussion where I was questioning the value of the "Sensor" level of the heirarchy you made the point that there could be value in grouping several variables that are logically tied to "the same thing". The Child concept does that, it provides an addressable container for one or more variables.

    At issue is whether there is a fixed relationship beween the Child and the S_ code, whether all variables within the same Child container must have the same S_ code or not.

    • If all variables in a given Child must have the same S_ code, then as somebody else pointed out this could be sent to the hub once during node integration and stored there (associated with the given node/child), rather than being transmitted with each value. But you cannot include variables for temperature and humidity in the same Child, even if logically that's what you want to convey (say you have two DHT-11's on the same node, you can't just make them two Children each with a temp and humidity).

    • If each variable within a given Child can have its own S code, then the S code and V code become a two byte descriptor of the variable, and Child is a flexible container used to represent "these variables go together" and only that.

    Which of these relationships between a Child (addressable subunit of a node) and S code (one byte code that along with V code helps interpret a given variable) did you have in mind?


  • Code Contributor

    @hek All these changes are more 2.0 than 1.4.. 🙂

    Edit;
    Btw, All these extra bytes S_ and V_ to describe a payload is good for mqtt also.
    And this is why we need some more command types (i.e. Stream for OTA update etc. more space for payload smaller header)


  • Hero Member

    @hek

    I see the edited version with MIN, MAX and RESET's added. This takes the orthogonality of V and S codes even further, and it seems like a good direction.

    And it's good to see a start on better addressing one of the complicated aspects of this framework = accumulation periods (for Max/Min/Average/Total).

    But is V_RESET a variable or a command, conceptually? Is is something sent to the node (probably) or reported by the hub (probably not)?

    This brings up another broad conceptual issue in creating a taxonomny like this - the difference between values and command/events. Many software frameworks have both concepts, for example class member variables and class funtions/events. (In electrical terms. this is somewhat like the "Level" vs "Edge or Trigger" distinction).

    In the current taxonomy, command/event and value are kind of intermingled. I think it will be important to document which is which.

    There are a couple of approaches. One is to define a list of commands (eg; C_RESET) distince from values (V_LEVEL); the other is to have a well defined way of using variables as pseudo-commands.

    For one possible example of the latter, one could say that for any variable defined as a command variable:

    • when the variable goes from 0 to 1, the node will take the defined action
    • the hub will send the node a "Set" command to set the value to 1; if it was 0, then the action is taken
    • Setting to 0 has no effect
    • the node will set the value to 0 when the action is completed (may be immediate)
    • the hub can query the current value of the variable to see if there is any action still pending or not

    (This might be exactly what you had in mind, or you might have had a variation in mind - I'm just saying that it should be clearly defined and documented).

    Then we could say that V_RESET is a command variable and thus shares the standard dynamics, where the specific "action" is to restart the accumulation period. And there could be other command variables, each documented as such.

    (The Event concept is similar to a command in that it's a transition rather than a level, but used as reports from the node to the hub. Not addressed yet)


  • Hero Member

    One characteristic of the MySensors system is that nodes are not systematically aware of time. That's good in that it simplifies the node firmware, and it's a limitation in terms of dealing with accumulation periods.

    I'm speaking of "accumulation" periods rather than "reporting" periods, because a node may "report" many times in an accumulation period, for example every 10 seconds it may report the accumulated rainfall since midnight.

    And there are several kinds of accumulation periods: in particular rolling and fixed start. Fixed start might report accumulation (sum, average, min, max) since a given time like midnight. Rolling might report the last hour or last 5 minutes from now. (I mentioned a rain sensor which does it's own "rain event" splitting based on no-rain intervals).

    In some cases, accumulation periods may not be under our control - for example reading an integrated weather station. At best a node could report the accumulation period.

    In other cases, we can control the accumulation period, for example the node's sketch could itself count KWH or rain, or it could average temperature or speed or direction, taking readings every N seconds.

    One more concept: how to handle a Reset. The simplest approach for a hub-commanded reset would be for the hub to be sure it has all desired recent value reports before sending a Reset command (so the time gap between last report and Reset is short enough that nothing signficant was missed, eg: mm of rain). A more synchronous approach would be for the node to send one last set of accumulated values immediately before zeroing it's counter.

    But what about a manually triggered reset (button pushed)? The sensor node could in that case want to let the hub know that it happened.


    One way to unify these in a simple and consistent way would be to report V_ACCUM_TIME - the number of seconds since the accumulation period began. That's the period over which values were averaged, summed, or min/max'd. The time would increase until a reset, go to zero and increase again. This handles controllable and not controllable accumulation periods, fixed and rolling accumulation periods, hub triggered and manually triggered resets.

    If an accumulation period is reset, the node could potentially send a last report with the ending V_ACCUM_TIME before the reset and then the following report would have time since starting the new period.

    If the time is just based on millis() then with the Arduino Pro Micro the time accuracy would be nominally about 0.5% (they have ceramic resonators rather than crystals and that's a typical spec). That's probably good enough, tho it's possible to do better if it mattered.


    Where this seems to be going is that any quantity variable could potentially be expanded into an accumulation variable - which means that in addition to (say) V_LEVEL, there is V_LEVEL_MAX, V_LEVEL_MIN, V_LEVEL_TIME, and either V_LEVEL_AVG or V_LEVEL_SUM. This "accumulation enhancement" could optionally be added to temp, humidity, wind, power, fluid/gas flow, whatever - in a defined and consistent way.


  • Admin

    @Zeph

    FYI: Sleeping sensors (who also can be woken by a interrupt on a pin) will have a hard time keeping time without clock module or by fetching time from server at each wakeup from pin trigger.

    Good idea about a STATUS for RAIN sensor. It will be added and indicates if rain is ongoing.
    Will also change to ACCUMULATED for all accumulated values. Average for these makes no sense.

    So.. I see now that there is need for more than one reset command. We'll need RESET_ACCUMELATED, RESET_LEVEL, RESET_PERCENTAGE. They should also have their respective MAX, MIN, AVERAGE. Could you please give me a use case for V_LEVEL_SUM.

    Just to clarify for anyone getting a bit scared of all these new variables. Using them for your own sensor is of course optional!

    I feel it's time to move the new stuff to the first post to keep it in focus.


  • Code Contributor

    I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible

    If I wanted node could send
    S_MOTION V_TRIPPED 1 (motion tripped)
    S_MOTION V_ARMED 0 (no longer armed)
    S_MOTION V_LEVEL 24 (motion quality 24 of 255) (probobly animal)

    controller decides S_MOTION V_RESET 1

    (order not thought of at all, needs to be fixed)

     #DEFINE S_DOOR	1
     #DEFINE S_MOTION	2
     #DEFINE S_SMOKE	3
     #DEFINE S_BINARY	4
     #DEFINE S_DIMMABLE	5
     #DEFINE S_WINDOW_COVER	6
     #DEFINE S_THERMOMETER	7
     #DEFINE S_HUMIDITY	8
     #DEFINE S_BAROMETER	9
     #DEFINE S_WIND	10
     #DEFINE S_RAIN	11
     #DEFINE S_UV	12
     #DEFINE S_WEIGHT_SCALE	13
     #DEFINE S_POWER	14
     #DEFINE S_HEATER	15
     #DEFINE S_DISTANCE	16
     #DEFINE S_LIGHT_SENSOR	17
     #DEFINE S_NODE	18
     #DEFINE S_LOCK	19
     #DEFINE S_IR	20
     #DEFINE S_WATER_METER	21
     #DEFINE S_AIR_QUALITY	22
     #DEFINE S_CUSTOM	23
     #DEFINE S_DUST	24
     #DEFINE S_PH	25
     #DEFINE S_SCENE_CONTROLLER	26
     #DEFINE S_NODE	255
     
     #DEFINE V_CONFIG1	1
     #DEFINE V_CONFIG2	2
     #DEFINE V_CONFIG3	3
     #DEFINE V_CONFIG4	4
     #DEFINE V_CONFIG5	5
     #DEFINE V_VAR1	6
     #DEFINE V_VAR2	7
     #DEFINE V_VAR3	8
     #DEFINE V_VAR4	9
     #DEFINE V_VAR5	10
     #DEFINE V_ARMED	11
     #DEFINE V_STATUS	12
     #DEFINE V_WATT	13
     #DEFINE V_PERCENTAGE	14
     #DEFINE V_STOP	15
     #DEFINE V_LEVEL	16
     #DEFINE V_MAX	17
     #DEFINE V_MIN	18
     #DEFINE V_RESET	19
     #DEFINE V_DEW_POINT	20
     #DEFINE V_MODE	21
     #DEFINE V_ANGLE	22
     #DEFINE V_RATE	23
     #DEFINE V_VOLTS	24
     #DEFINE V_AMPS	25
     #DEFINE V_PRESENTATION	26
     #DEFINE V_BATTERY_LEVEL	27
     #DEFINE V_RESET	28
     #DEFINE V_TIME	29
     #DEFINE V_ID	30
     #DEFINE V_LOG_MESSAGE	31
     #DEFINE V_SKETCH_NAME	32
     #DEFINE V_SKETCH_VERSION	33
     #DEFINE V_FIND_PARENT	34
     #DEFINE V_CHILDREN	35
     #DEFINE V_VERSION	36
     #DEFINE V_INCLUSION_MODE	37
     #DEFINE V_GATEWAY_READY	38
     #DEFINE V_STATUS	39
     #DEFINE V_IR_SEND	40
     #DEFINE V_IR_RECEIVE	41

  • Hero Member

    @hek said:

    @Zeph

    FYI: Sleeping sensors (who also can be woken by a interrupt on a pin) will have a hard time keeping time without clock module or by fetching time from server at each wakeup from pin trigger.

    Sleeping nodes may have a hard time averaging in some cases too.. But your point is well taken and it is a real concern.

    One of the sleep libraries calibrate the RC watchdog timer against the main clock, so it can make a good guess for adjusting milliseconds upon each wakeup. That could help for a simple sleep, but not for waking up from an interrupt.

    So the problem child is a node for which all the following apply:

    • sleeps
    • is awoken by interrupts at unpredictable intervals
    • has no local RTC
    • has accumulated variables

    One option would just be to ignore the accumulated time feature for the subset of nodes in that category; for them, you have to fall back to where would be be if I had never spoken up about tracking time. For other nodes, the tracking & reporting of accumulation time can be an enhancement, as described in an earlier message.

    But suppose we DO want to go further and determine accumulation time for all nodes, even the above.

    (Sigh. I acknowledged that there was a simplicity advantage to not dealing with time, and that just using the local node's millis() was the simplest but not only way to handle time; and that does handle most cases but not the above)

    System Time approach 1: Master Clock Broadcast to nodes

    The hub will have the master clock, using its millis(). Periodically it will broadcast the value of this to all nodes. A node could also request a time be sent. We could apply a crude delta to adjust this according to number of hops and typical transmission time, or not - we don't need anywhere close to millisecond accuracy. Each node would maintain a delta between its own millisecond clock and the broadcast millis and adjust that every time it receives that broadcast. (Smart nodes could even use a computed time scaling factor to adjust for differences in their clock vs the master). When for any reason has a need to estimate the current time (including but not limited to the case of having been powered up from sleeping by an interrupt AND needing to repost accumulations with more precision than it can estimate), the node can either wait for the next broadcast, or request one.

    System Time approach 2: Track accumulation times in the hub

    Whenever an accumulation period is started, the node will send a message to the hub to that effect (telling it which variable is involved and an incrementing reset count). This could be triggered by a commanded or manual reset, or by the automatic reset of an external weather station or service, or whatever. Then it would send the reset count (V_RESET_COUNT) rather than the accumulated time along with accumulation based values. The hub can look at its current time, and at the recorded time of the last reported reset of that variable, to compute for itself the elapsed time. The reset count makes sure both sides are sync'd as to which reset they are talking about. Both the "Starting accumulation period #n for (variable)" and the "reporting accumulated value for (variable)" messages travel from node to hub, and their travel times will roughly cancel out, at least well enough.

    The former approach is more powerful. It better supports rolling averages. And it would provide a flexible "MySensors network time" framework for future features, like reporting event times or periods more accurately.

    For example, I've considered putting two motion detectors at angles and with masks such that one could usually detect which way somebody went through a doorway by the trigger times, or at different place in a hallway. I'm not proposing to support that directly at this time, but when it comes up, if each node had the option of knowing the hub's master clock time to within tens of milliseconds or better, that could be a useful base upon which to build.

    [Total aside. My ambition for another nRF + Arduino system is more precise than this, possibly even enough to do PWM of 60 Hz mains powered lights without a local zero crossing detector - not yet determined. That's why I connect the nRF24L01+'s IRQ output to Arduino pin 8 = ICP1, even if I'm not using interrupt driven nRF code. If the nRF's receive interrupt is enabled, this allows the ATMega328p to get a very accurate time of reception using Timer 1 if it wants, even in polling mode without interrupts. And if I ever do want to interrupt the ATMega from the nRF, I can use the capture interrupt, while leaving INT0 and INT1 on pins 3 & 4 for other uses. The above first option is a smaller, simpler version of that design, adapted for the needs of a sensor network]


  • Hero Member

    @hek said:

    Could you please give me a use case for V_LEVEL_SUM.

    • "snapshot in time" values like temperature and humidity and watts logically accumulate as (averages. min, and max) over the given period as their end product. Tuesday the average temperature was 22.3 degrees.

    • "incrementing total" values like KWH, liquid flow, and rainfall logically accumulate as total or (sum) over the given period as their primary end product: Tuesday a total of 20mm of rain fell. (IFF you know the time as well, you can calculate an average as a secondary product)

    These two types of values have somewhat different dynamics.

    Of course if you report a total AND an elapsed time, the hub can calculate averages for itself, but not min and max. This calculation can be done at the hub even for temperature, tho total degrees C is not meaningful and total mm rain is.

    So IF we can rely on S_LEVEL_ELAPSED and S_LEVEL_SUM both always being available (and the latter not overflowing integer formats), we could omit S_LEVEL_AVG. But in the original proposal without a time framework, the node should report S_LEVEL_SUM for rain and S_LEVEL_AVG for temp.


  • Hero Member

    EDIT - there are some good concepts in the next few posts from me (IMHO obviously), but some may turn out to be sidestepped by a more radical change described in response
    http://forum.mysensors.org/topic/304/unit-standardization-1-4/57,
    so you might want to just read that first.


    @hek said:

    ###S_WIND - Wind sensor
    V_LEVEL - Wind level in m/s (average wind speed during last report period)
    V_LEVEL_MIN - Min wind level
    V_LEVEL_MAX - Max wind level (gust)
    V_LEVEL_AVERAGE - Average wind level-
    V_LEVEL_RESET - Reset max/min value
    V_ANGLE - radians <float or int>

    First, i still want to convince you that V_ANGLE in "degrees clockwise from true north" is the proper measure here, not radians. (and you need to define if this points to where the wind comes from or where it's moving towards, so is a west wind 90 degrees or 270 degrees - or translate to radians if you want to see ugly)

    Look, I'm a math person (or was) and I totally get that e, national logs and exponents, pi and radians are more fundamentally defined values than degrees or using base 10 for decibels or logs. But the point is not scientific purity (which are are not really close to anyway) but pragmatic utility in the field of sensor networks. And degrees simply work better in this context of conveying direction, especially if you allow int representation. Enough on that for now.

    Second, notice that you really only have two real measurements here: speed and direction. Those are what I call snapshot values - if you sample them at a given time, you get a value, which can later rise or fall.

    The former has additional derived or calculated sub-values, and those subvalues follow a pattern representing "simple time accumulation statistics" and also common to many other snapshot variable types, like humidity.

    The thing is - direction is also a snapshot value. If you are going to report average wind over the past hour, you can also report average direction.

    Alas, angle differs from continuum measurements in that it wraps, so average/min/max have to be computed carefully and are only meaningful if the source stays within a reasonable range; and sum makes no sense.

    But any continuum variable is a potential candidate for accumulation-period based derived sub-variables. I picture that there will be a single "accumulation" library which can be applied to any int or float variable, rather than that the temperature library will have accumulation code and the humidity library will have its own accumulation code.

    So in the hierarchy I'd put V_LEVEL_MIN, etc one level indented under V_LEVEL rather than make it look like a parallel and independent measurement.


    Let's also be clear that there are two reasons to have accumulation period based derived variables. One is that the node could be getting values from a device which does its own accumulation of some sort (eg: a weather station or some rain gauges). The other is that the device may sample the instantaneous variables more often than it transmits them. If we transmitted every measurement, the hub could potentially do its own min/max/average/sum calculations. No big deal there, but the hub could also accumulate by multiple arbitrary periods, like hour of the day AND since sunrise AND past 10 minutes AND past hour AND since Oct 1st (which is the start of the water year as used by scientists around here). So the most flexible thing would be to report every measurement to the hub and let it handle aggregation any way (as many ways) as it wants. But that could be bandwidth heavy, or might impose too much work on simple hubs (or controllers they report to).

    So for one or more of these reasons, sometimes it may make sense to have the sketch running in a node handle accumulation sub-variables. If it reports often enough in a well considered format, the hub or controller or cloud based analytical software can still do aggregation over some of these alternative time periods.


  • Hero Member

    @hek said:

    ###S_BINARY - Light (or other) on/off actuator
    V_STATUS - 1 - turn on, 0 = turn off

    ###S_DIMMABLE - Dimmable actuator
    V_STATUS - 1 - turn on, 0 = turn off
    V_PERCENTAGE - Dimmer level 0-100%

    OK, let's define what happens if you get more than one type of command (probably just making explicit what we are thinking anyway).

    S_DIMMABLE:
    V_STATUS = 1 is the same as V_PERCENTAGE = 100, either has same effect
    V_STATUS = 0 is the same as V_PERCENTAGE = 0, either has same effect

    If you set V_STATUS = 1 and then query V_PERCENTAGE it should return 100% (or 0 => 0%)
    If you set V_PERCENTAGE = 100 and query V_STATUS it should return 1

    OK?

    The only question is setting V_PERCENTAGE = 1..99 and querying V_STATUS.
    Option 1: Return -1 for "neither fully on nor fully off"
    Option 2: Return 1 for "at least partially on" (ie: anything above 0%)


    OK, what about a Fan with setting Off, Low and High, and another fan with settings "Off, Low, Medium and High?
    We could map those to text "OFF, LOW, MED, HIGH"
    Or we could map them to 0,1,2 where 2=HIGH for the first fan, and 0,1,2,3 where 3 = HIGH for the second

    I'm going to suggest that we primarily map to V_PERCENTAGE, but perhaps also support the latter option as an expansion of binary 0..1 into multi step value 0..N_STEPS.

    All fans:
        OFF = 0%
        HIGH = 100%
    Fan 1:
        Low = 50%
    Fan 2:
        Low = 33%
        Med = 67%
    

    If you are sent a V_PERCENTAGE, set the fan to the closest speed to the given percentage. 100% is always the highest speed, 0% is always OFF, in between will approximate. This extends to 4 level fans. And it works for a heater or motor with multiple speeds as well. If you query V_PERCENTAGE the actual value chosen from the list will be returned, not the requested value (ie: requesting 45% but reporting 50% in the case of fan 1).

    The above intermediate percentages are the default. However the node is empowered to set other values. We all know of fans or other devices where the speeds are not evenly spaced. I have a heater which has 600, 900 and 1500 watt settings (both elements). The Low and Medium setpoints could be 40% and 60%. Otherwise it works as described above.

    And there could be an integer V_SET_STEP (or something), corresponding to V_BINARY* but ranging from 0 to N_STEPS where N_STEPS is per-variable defined. I'm not attached to this, but it's an option, It would work in the obvious way. So V_SET_STEP of 1 for Fan1 would set it to Low=50% and V_SET_STEP of 1 for fan2 would set it to Low=33% (and V_SET_STEP of 1 for the heater example would set it to Low=40%. SET_STEP of 0 is always off, and SET_STEP of N_STEPS is always HIGH or 100%


    For completeness, I will mention that there's another way to define this for V_DIMMABLE. It could be required to memorize the last set percentage, and then V_STATUS = 0 means turn to 0% but remember last non-zero percentat, and V_STATUS = 1 could mean return to last non-zero percentage.

    Tho I don't favor this dynamcic, it too could work. However I strongly favor defining which way we expect things to work rather than leaving it ambiguous!


  • Hero Member

    @Damme said:

    I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible

    As described at the time I write, the proposed taxonomy of values isn't really as heirarchical as it's presented.
    You can also parse the meaning from V code to S code:

    Ah, we have a V_LEVEL value, that means it represents a one dimensions continuum
    But what does that continuum represents?
    Oh, I see the variable is also tagged with S_TEMPERATURE
    That means the continuum is temperature in degrees C
    

    That is, the V code tells you something of the structural concept, and the S code adds some degree of semantics (like dimension and unit).

    So you could also invert the displayed hierarchy and show which S codes are available for a given V code


  • Hero Member

    Hold the presses.

    I've happily engaged in the collaborative process of cleaning up the variable taxonomies of V 1.4b here.

    But we keep coming to conceptual tricky parts. Some of that is inherent, but in thinking deeper about it, i've come to the conclusion that a good part of what makes it hard to resolve cleanly is that we are trying to cram too much information into too few bits of variable metadata which has to be included in every packet.

    So rather than reforming S and V codes, I'm thinking a more radical approach is needed, to make the system both more flexible and simpler. Part of that comes from identifying what metadata about a variable is static and what needs to be dynamic, and then moving the static part out of the normal OTA operations. That turns out to make the taxonomy much more tractable and flexible while simplifying the node firmware.

    Guess what - even child ID turns out to be metadata in this approach, rather than addressing.

    Here's the sketch of the new approach.

    http://forum.mysensors.org/topic/308/another-way-of-organizing-variables

    It's sufficiently different that I expect it may encounter resistance - from the natural and appropriate fear of breaking too much (I share that concern). But it just may turn out to be sufficiently attractive to be worth it. Have a look. And now would be the time to consider such a break - with RadioHead and OTA programming support also in the air.


  • Code Contributor

    @hek said:

    This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

    I switched the bootloader code between using stream and using simple internal messages - at the end doesn't matter as long as it's simple enough to dis-samble and assemble a packet without a lot of additional code overhead.


  • Admin

    @ToSa

    Sorry, forget I brought this up. We'll keep the STREAM command as-is. Skyped you about it yesterday 😉


  • Code Contributor

    @Zeph said:

    But suppose we DO want to go further and determine accumulation time for all nodes, even the above.
    ...
    System Time approach 1: Master Clock Broadcast to nodes
    ...
    System Time approach 2: Track accumulation times in the hub

    I'm in favor of approach 1 as well - and an optional addition to the MySensors class to keep track of time. Only specific sensor types would use this feature - interesting mainly for actuators rather than sensors:

    Let's assume you use a relay to turn on/off your garden watering while you are on vacation. The network breaks (controller or gateway or whatever node needed to relay messages).

    • If the sensor node itself knows about the time and the schedule for watering, it can turn on and off the watering itself. If there is an issue with the communication, the time would not be 100% accurate over time but still good enough.
    • If the sensor node relies completely on the controller for time and (worst case) the communication breaks just between the start watering and stop watering command you can imagine what happens...

    Knowing about the time heavily reduces the single points of failure. The protocol already covers the "request time" communication so this should be relatively easy to implement...

    • controller sends a time broadcast every e.g. 15 minutes
    • only the nodes interested in time care about it and update their internal time
    • when a node needs the current time it calculates it based on the last timestamp received, the millis() when it was received and the current millis...

    Doesn't solve the sleep issue (as long as you don't want to submit a separate "get time request" every time you wake up) but at least for sensors that are not battery powered (relay nodes in most cases are not) it's a feasible approach.

    This is probably worth a separate thread 🙂


  • Admin

    @ToSa, @Zeph

    Yes, and to extend this even further it would be great to be able to distribute/push simple rules all the way out to the nodes. It might be enough with simple scheduler-commands to start with but this could easily support even advances scenarios. The question is where to set the cuttingpoint...

    The most extreme solution would be to allow a graphical rule engine on the controller which automatically generates a sketch, compiles it and sends it out OTA to the node.

    Definitely worth discussing in a new thread...


  • Hero Member

    @hek said:

    Yes, and to extend this even further it would be great to be able to distribute/push simple rules all the way out to the nodes. It might be enough with simple scheduler-commands to start with but this could easily support even advances scenarios. The question is where to set the cuttingpoint...

    For the level of protocol under current discussion, the cuttingpoint could just be giving nodes an optional sense of time (using broadcast packets to keep in sync with the master as needed). And making use of that time as part of the (optional) accumulation enhancement for some variables.

    What else they do with that time at higher layers would continue to evolve in the future. We don't have to bite it all off now.



  • V_R_PERCENTAGE - Red component % <int>
    V_G_PERCENTAGE - Green component % <int>
    V_B_PERCENTAGE - Blue component % <int>
    V_W_PERCENTAGE - White component % <int>

    Just a hundred steps might be to low, especially for green in the lower end which the eye sees much brighter.
    Please consider floats or per thousand instead.


  • Admin

    @Rasmus-Eneman

    Ok! Didn't know that. But couldn't the sensor code adjust this with some sort of color curve?



  • @hek
    That would probably do it, however one would need to be defined and standardized so that all behaves the same. One more clever than me needs to propose how the curve would look tough.


  • Plugin Developer

    @hek @Rasmus-Eneman

    S_RGBW - RGBW Light
    V_STATUS - 1 - turn on, 0 = turn off
    V_R_PERCENTAGE - Red component % <int>
    V_G_PERCENTAGE - Green component % <int>
    V_B_PERCENTAGE - Blue component % <int>
    V_W_PERCENTAGE - White component % <int>

    Ain't it better to split up S_RGBW to two different types. W is already present in full R, G, and B. For example i have implemented Philips Hue and there is a difference in fixture types some do RGB some only W. If an user would implement such with mysensors the same option can be possible. An S_RGBW declaration for a W only would indicate RGB capabilities and maybe cause confusion.

    Just a hundred steps might be to low, especially for green in the lower end which the eye sees much brighter.

    Agree, but thousands could be too much (which never is the case of course), maybe 0-255 would be sufficient and widely used (all though not as precise in steps as with thousands)? Also the W_PERCENTAGE should be left out because this is full R,G,B or similar calibrated R,G,B levels (which is quite difficult to do at every possible level) with the end result it then in fact is a brightness level instead of a white component.

    An addition could be S_HSB/S_HSV with float,float,float (0.0-360.0, 0.0-100.0, 0.0-100.0)? Also i think the color presentation should be or calibrated on the device or in the controller. Even some of those Philips Hue's fixtures are often controller side calibrated (better said color index modified after color selection)....


  • Admin

    I actually have RGBW led strips mounted around our house. W is indeed its own component (Warm White in my case). So it will stick 😉

    (Warm White using the special W-component)
    20130825_211259.jpg

    Cold white (RGB activated)
    20130825_211337.jpg

    Crazy
    20130825_211203.jpg


  • Admin

    @John

    But switching to LEVEL and using 0-255 would be ok (if we need it ;)).


  • Plugin Developer

    @hek
    Ahhh... understood :)... Splitting those up will not be an option then because it could then create a numerous amount of options....

    A S_HSB would be nice though, even if it would only be ints....:
    V_STATUS - 1 - turn on, 0 = turn off
    V_H_DEGREE - Degree (0-360)<int>
    V_S_PERCENTAGE - Saturation % <int>
    V_B_PERCENTAGE/V_V_PERCENTAGE - Brightness/Value % <int>
    V_WATT - Watt <int>
    V_WATT_MAX - Max watt value
    V_WATT_MIN - Min watt value
    V_WATT_AVERAGE - Min watt value
    V_WATT_RESET - Reset max/min value

    I must say it is a personal request :$....

    [Edit]

    But switching to LEVEL and using 0-255 would be ok (if we need it 😉).

    It would create space to add the HSB/HSV option 😉


  • Hero Member

    What's wrong with setting lights at 1.475% Is percentage limited to integer?

    Let's be very explicit about that. Is the idea that all values accept only integers unless they say they have the option of using floats? If so that should be made clear in the list.


    Another thing that needs to be more clear is how to handle multiple V codes for the same sensacturator (like the RGB or HSV lights).

    • Is a sensor expected to report all of them?
    • Is an actuator expected to respond to all of them?
    • How do both sides stay in sync about subsetting?

    By the way, there is a distinction to be made between RGB lights (three blendable primaries able to synthesize many colors in a predictable and fairly standard way) and multichannel control of N lights of arbitrary colors. RGBW is another thing entirely. The W could be cool or warm white. For that matter the fourth color could be yellow or violet. It's not a color space.

    If the RGB channels blend I'd implement it as an RGB light plus a monochrome light, because there is no RGBW color space. If they don't blend I'd implement it as 4 monochrome channels, the same way I'd implement a Yellow and Green pair that didn't blend.


  • Plugin Developer

    @Zeph

    What's wrong with setting lights at 1.475% Is percentage limited to integer?

    It just not represents any absolute bit value for a specific color, it is an approximate. If for example you have a fixture which does 255 steps and you want to set it at step 40. you will need to send 15.682.....%. But what if you would send 15.770... ? Due to the nature of floats it could be rounded down or up to 40 or 41 and especially in the lower steps of a color the difference would be seen if this is a mix of on of the RGB values, and even more as all the RGB values are send in percentages. A percentage would only be usable (But even then it's still an approximate) with 32 bit or analogue blend RGB fixtures. There ain't a lot of these.

    @zeph said:

    If the RGB channels blend I'd implement it as an RGB light plus a monochrome light, because there is no RGBW color space. If they don't blend I'd implement it as 4 monochrome channels, the same way I'd implement a Yellow and Green pair that didn't blend.

    I do agree with you, and you're totally right, but:

    @john said:

    Splitting those up will not be an option then because it could then create a numerous amount of options....

    In other words it could eat up the possible S_TYPE's

    @hek
    Please disregard this one, it wouldn't :"it would create space to add the HSB/HSV option".
    But, couldn't it be an option to add the HSB/HSV variables to the S_RGBW? because if the presentation stays you would only have to send the vars that are supported, and the type S_RGBW could be renamed to something like S_LIGHT_COLOR?


  • Admin

    The question is where the actual calculation is done between RGB and HSB/HSV. Should it be done on the Arduino side or controller? Nothing stops you from making a fancy HSB/HSV GUI on your controller but the actual values send to the actuator is RGB. But my knowledge of color scales is a bit limited and I'm probably missing something?


  • Plugin Developer

    @hek

    It depends, if the HSB scale is purely in integers you will have about 3.600.000 options (including brightness steps) which is much less then 16 bit 0,0,0 - 255,255,255 scale. But this scale ain't specific in int's it is an implementation chosen scale. If choosing for ints, yeah keep the RGB, if going to floats, add it next to the RGB.

    probably missing something?

    The main difference is that with pure 16 bit you will have whole bit numbers and with degrees there is more then just go from 359 to 360/0. you could use 359,00 to 359,99 which gives an extra 990.000 extra red combinations (including saturation and brightness) with absolute 16 bit you stick with whole ints/bytes limiting the possible fixtures light high "resolution" capabilities.

    Also because of the nature of HSB with higher color resolution possibilities you will always have the possibility to find the closest 16 bit neighbor or even the exact bit value because of higher to lower "spectrum" resolution.

    With the above if you would leave out the brightness you would have a higher color resolution 255,254,255 is one step lower green form maximum, you can't go to 254.9 right? in degrees i think green is 120, you could go to 119.99999. because the degrees are not fixed to a whole as a byte is.

    Because of more steps calibrating un-calibrated fixtures/leds (for example with the green mentioned earlier) can be more precise. Also HSB is more human perspective then fixed bits.

    Maybe the above shines a light on the difference?

    RGB is just a brightness setting of a specific color, HSB is pure color selection, intensity setting, and the brightness of these.

    Personally:
    HSB is nice to have, but from personal needs i have enough convertion functions (hsb->rgb and vice, hsb -> kelvin, etc...) to convert.

    Should it be done on the Arduino side or controller?

    I would let the controller do the actual conversions and let the device spec it's capabilities.


  • Hero Member

    @hek said:

    S_RGBW - RGBW Light
    V_STATUS - 1 - turn on, 0 = turn off
    V_R_PERCENTAGE - Red component % <int>
    V_G_PERCENTAGE - Green component % <int>
    V_B_PERCENTAGE - Blue component % <int>
    V_W_PERCENTAGE - White component % <int>
    V_WATT - Watt <int>
    V_WATT_MAX - Max watt value
    V_WATT_MIN - Min watt value
    V_WATT_AVERAGE - Min watt value
    V_WATT_RESET - Reset max/min values

    One of the areas where the spec is very unclear is when a variable is writable and where it's only reportable (read only).

    For example, can I set the watts to 20 above (or for S_DIMMABLE) in order to tell the light how much power I want it to use? Seems like a valuable option. How am I supposed to know?

    What if we made the write/report/trigger function explicit:

    S_RGBW - RGBW Light

    W_STATUS - 1 - turn on (100%), 0 = turn off (0%)
    W_R_PERCENTAGE - Red component % <int>
    W_G_PERCENTAGE - Green component % <int>
    W_B_PERCENTAGE - Blue component % <int>
    W_W_PERCENTAGE - White component % <int>
    R_WATT - Watt <int> - current instantaneous or short term average value
    R_WATT_MAX - Max watt value since last reset
    R_WATT_MIN - Min watt value since last reset
    R_WATT_AVERAGE - Min watt value since last reset
    T_WATT_RESET - Reset max/min/avg value
    

    W variables can be written to; the current value can also be queried
    R variables can only be reported (automatically or by query)
    T variables are triggers, writing to them triggers some action

    This gives guidance to both the sketch writer for the node, as well the controller adapter writer in knowing what they can do with each variable a node presents.

    The actual numeric values for R, W and T codes could be interspersed or non-overlapping ranges could be reserved for each (eg: 1-100 for R, 101-200 for W and 201-255 for T)


  • Code Contributor

    @Zeph it all depends on how skilled programmer and how nerdy you are. if you want to set a lamp power with V_WATT great, thats nerdy! but in not, just dont read any thing from v_watt. .. there is no such thing as a read only variable, all depends on if you listen to incoming messages of that type or not.... I think you over-think this a bit. The atmega328 is a rather small mcu and wont be able to magically do tons of stuff. how to know what is what, well every home automation system should have a manual 🙂 Write it down.


  • Hero Member

    @John

    I can see a case for making HSB the norm.

    Think of light types like a class heirarchy, with added functionality

    S_BINARY

    V_STATUS (1=100%, 0 =0%)
    

    S_DIMMABLE

    V_STATUS (1=100%, 0 =0%)
    V_PERCENTAGE (0.0% - 100.0%)
    

    S_HSB

    V_STATUS (1=100%, 0 =0%)
    V_PERCENTAGE (0.0% - 100.0%)
    V_HUE (0.0 - 360.0)
    V_SATURATION (0.0% - 100.0%)
    

    If you set V_STATUS to 0, or V_PERCENTAGE to 0%, any of these lights turn fully off.
    If you set V_PERCENTAGE to 100%, S_DIMMABLE will go to 100% (V_STATUS=1)
    and S_HSV will go to 100%, in the current hue and saturation
    If you set V_PERCENTAGE to 50%, likewise.

    I think one could make the case that it's more human compatible to go to "yellow, 25%" than to figure out what the RGB values would be. (The user interface could translate yellow to a hue number)

    One aspect HSB is that there are a couple of ways of associating colors with angles, so we would need to specify which is our standard, so 40 degrees is the same hue for all nodes. I like the rainbow style.


    @hek
    I do also get the point that HSB -> RGB conversions could be done on the controller, passing the lower level RGB values to the node. There is an attraction to that as well.

    This cannot be fully resolved without also considering the question of light curves or gamma factors, which has been raised. The controller needs to know whether or not the node implements light curves for RGB channels, so it can avoid doing that twice; and also it in the changes how the controller should convert HSB to RGB (the stock conversions assume RGB are linear).


  • Hero Member

    @Damme said:

    Write it down

    That's what I'm trying to do.

    We could CLEARLY write down that V_WATT is intended only to report values and cannot be set, while V_RESET is a trigger, and V_R_PERCENTAGE is settable - so different implementors don't make different and incompatible guesses about what these mean.

    We do not want the situation where one sketch writer interprets V_WATT to be a way to control how much power the lamp will use and another sketch write interprets it as reporting only. And then controller interface writers also guess differently about what it means. That's chaos.

    One step in that direction would be naming them according to the intention - writable, reportable, or trigger.

    I've done a LOT of implementing from specifications, as many others have. Having a clear specification that avoids huge ambiguities (is this something you set or something that reports is huge) is not over-thinking - it pays off big time.


  • Mod

    @hek said:

    Crazy

    You didn't dare posting a red version, did you? 💃


  • Plugin Developer

    @Zeph

    I can see a case for making HSB the norm.

    I'm not ;). it is just a possible addition of possible value types to the S_RGBW type and if not possible like you stated a S_HSB and to keep clarity V_PERCENTAGE should then be V_BRIGHTNESS. Or both percentages (not possible) or both naming as it is.

    One aspect HSB is that there are a couple of ways of associating colors with angles

    Off topic, which ones? i only know one, are you sure your not mixing HSB with HSL?

    On topic:
    I'm fine with only sending RGB if HSB ain't possible to be combined with S_RGBW or added with S_HSB/S_HSV if it is thought of as a logical addition.


  • Hero Member

    @John
    You are probably way ahead of me in terms of formal color spaces. I'm just a color user.

    I like using a modified Hue, such the the "rainbow" described here:

    https://github.com/FastLED/FastLED/wiki/FastLED-HSV-Colors


  • Plugin Developer

    @zeph said

    You are probably way ahead of me in terms of formal color spaces

    Not really, i'm no color expert, But did a lot with colors lately because i use this to influence our living environment

    Ok got it, Hue with rainbow style is variable due to the amount of colors (real life circumstance like angel etc) on that page an almost perfect rainbow angle and lighting situation is used.

    I agree with you, but that's user implementation on the hardware, and something mysensors can not control. You would still need the HSB parameters 😉 The implementation of rainbow or spectrum should be done on the controller<->end device side and the end user should know about it (or at least the developer of that specific controller/hardware implementation), not protocol based, because the parameters are still the same.


  • Hero Member

    @Hek globally I don't understand the min/max/average, are they managed by the sensor or the controller ? if it is the sensor in case of reboot it should get it back from the gateway/controller and should be resettable maybe to have a year/month/week/day min/max. For me that should be controller side to handle this logic...

    Now for my favorite playground 😉 it needs the calibration value clearly first, then for the discussion some projects (AirPi) avoid the calibration pitfall by sending back the voltage... If we want to have a concentration value from the sensor that is ok.

     S_AIR_QUALITY - Air Quality Sensor
     V_LEVEL - ? We need to describe this better
     V_LEVEL_RESET - Reset max/min value
    
    @hek Hmm.. Soil Moisture.. I have no idea how to represent this. Help me! The ebay sensors we're using isn't super accurate and comes uncalibrated.
    
    @damme   Hmm... V_Moisture, (relative) 0-255 / 0-100% / True | False
    

    On both I agree, I have some imaginatronix and yes you need a margin of error... but when it rains it rains... this is where a calibration value will be usefull at least to have several sensors showing the same value at the same environment time... I you want a logic then the true false can be good but not enough... on my imagitronix I have an empty value of 20% and a watered value of 100%, sometimes after the rain it is 80%..., where would be the limit ?

    Also, I am thinking of a leaf wetness (we had a lot here this year... and diseases that goes with it) that could be mapped to soil sensors [Article on values(http://www.decagon.com/education/do-you-measure-leaf-wetness/)


  • Admin

    @epierre said:

    globally I don't understand the min/max/average, are they managed by the sensor or the controller ? if it is the sensor in case of reboot it should get it back from the gateway/controller and should be resettable maybe to have a year/month/week/day min/max. For me that should be controller side to handle this logic...

    The MAX/MIN/AVERAGE are optional to use. My idea was that sensors would manage them. They can choose to store it in eeprom or throw away values when reboot occurs.
    The idea originates from GUST for wind which really is a max value over a certain time. This is just a generalization.
    Usecase: Sensor might not report every wind value to controller but just every 10 minute. But internally it can calculate average/max/min for every 10 seconds without pushing this.



  • Please leave HSV/HSL to the controller. Why complicate the node more that necessary?
    Using LEVEL would be good, 8 bit per color is good, it's also super easy for the node to scale down if only supporting 7 or 6 bits and it's perfect for those digital LED strips.

    Please have a pure RGB without W, most strips and RGB leds don't have a W diode so that would need to be simulated with different result in different nodes.


  • Plugin Developer

    @Rasmus-Eneman

    Please leave HSV/HSL to the controller. Why complicate the node more that necessary?

    That's why i asked for an addition if it would over complicate stuff (and if there is room for it) ;). And well, yeah you're right, S_RGBW with HSB would be complicating things more mixing capabilities with the node naming. There is always S_CUSTOM with V_VAR_1,V_VAR_2,V_VAR_3 combination.

    I agree with leaving out the W. But could be a nice addition for those non calibrated fixtures where you have to supply calibrated values in the node to get "proper" white levels.


  • Hero Member

    @hek ok I agree. Can we have too an alert that could be sent back to the gateway, or like an armed mode for the air sensors ?

    Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...


  • Mod

    @epierre In 1.4 a binary represetation of a float value can be exchanged with the library. This allows to report e.g. + or - infinite where normally floating point values are sent. Maybe this will serve your purpose?


  • Hero Member

    If you do have a strip with RGB and W LEDs (rare), this can be handled as an RGB light and a DIMMER.


    A lot of our discussions really hinge on how much smarts to put where in the architecture. My overall philosophy would be to keep the node simple and move functionality to it only when the same thing cannot be easily done centrally. Reasons to move smarts into a node could include:

    • Uses local calibration that central doesn't have
    • Needs updating more often than reports go to central (eg: wind max)
    • Timing, eg: safety features that need to act fast
    • Reliability of data links, eg: safety features that should not depend on RF robustness
    • Providing some logically consistent interface available to all controllers
    • Providing some functionality which cannot be as easily implemented in a controller (eg: closed source)

    The latter two would be one-off custom coding at this time, tho @hek has mentioned the idea of pushing some rules down to the node level in a more automated way someday.


    This probably argues more for using RGB rather than HSV at the node level. What I liked about the HSB was the unification of on/off, dimmable, and HSB functionality as a nested set of expanding functionality. So a script or rule that sets the brightness could work on a DIMMABLE or a HSB light by setting the V_PERCENTAGE value (or V_BRIGHTNESS as someone suggested). But that could be done at the central level.


    I've used the term "central" because I still have the vision that a "hub" or "gateway" could implement some features, rather than having the smarts only in the controller or in the sensactuator nodes. So for example, it could be possible for the gateway to convert HSB to RGB such that the controller doesn't have to implement it. I know that approach is not currently popular so you don't need to register general disapproval. This will either come to seem a worthwhile additional open in coming years or not.


  • Hero Member

    @epierre said:

    Can we have too an alert that could be sent back to the gateway, or like an armed mode for the air sensors ?

    Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...

    Let's not bring infinity into this...

    How about just returning "the maximum reportable value" (finite). If the node is capable of detecting "above measurable" conditions, have it define the "maximum reportable" as 1 higher than it's internal "maximum measurable".

    The idea is to keep a simple scale of results, where for example we can consistently compute "max/min over 24 hours", or set a trigger on "value >= threshold" without needing to deal with infinity math. Some controller software may not have any concept of infinity, so using it brings in unnecessary incompatibilities.

    This allows the controller to define the alert as it wishes. For example, suppose a sensor reports uncalibrated values of 25 to 173, and the node uses 174 for a detected "over-range". The controller can implement a trigger or event at "value > 160" to test for high values (without having to test for infinite values too). Or it could test for "value >=174" if it specifically wants to test for overrange (which could be way over the max measured value or just barely above it).


  • Admin

    @epierre said:

    Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...

    LOL


  • Hero Member

    @Zeph I don't share the same view, since we allow logic in the sensor for min/max interval, why not doing it too for the gas level ? all the more that the limit of the sensor is not a "business logic" but an internal logic linked by the physical capacity of the sensor itself (and which varies with each sensor type...).

    My point was more that if you monitor LPG NO2 / NO3 or smoke that could be really harmfull we should provide safeguards anyway... not a big deal, more a safety net...


  • Admin

    Ok,

    I will change create two device-types S_RGB and S_RGBW and use a level instead of percentage for each color component (0-255).


  • Hero Member

    For my sound sketch :

    S_SOUND - Sound sensor

    V_LEVEL - sound level in db
    V_LEVEL_MAX - Max sound level
    V_LEVEL_MIN - Min sound level
    V_LEVEL_AVERAGE - Average sound level
    V_LEVEL_RESET - Reset max/min value


  • Hero Member

    For my vibration sketch :

    S_VIBRATION -Vibration sensor

    V_LEVEL - vibration level in Hertz
    V_LEVEL_MAX - Max vibration level
    V_LEVEL_MIN - Min vibration level
    V_LEVEL_AVERAGE - Average vibration level
    V_LEVEL_RESET - Reset max/min value


  • Hero Member

    @zeph the smoke sensor is an Air Quality sensor in fact so we coud decribe it this way too.
    @hek what about the V_GAS_TYPE proposed below ? that would limit extensing the S and V no ?

    S_AIR_QUALITY - Air Quality Sensor

    V_GAS_TYPE - gas type reported by the sensor
    V_LEVEL - gas concentration in ppm/ppb
    V_LEVEL_MAX - Max gas concentration level
    V_LEVEL_MIN - Min gas concentration level
    V_LEVEL_AVERAGE - Average gas concentration level
    V_LEVEL_RESET - Reset max/min value
    V_STATUS - Tripped status of a gas sensor. 1=Tripped, 0=Untripped
    V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed


  • Admin

    @epierre said:

    S_SOUND - Sound sensor

    Wouldn't it be good to have a uncalibrated variable for S_SOUND also? Like proposed for S_LIGHT_SENSOR. Reported as:

    V_PERCENTAGE - 0-100%


  • Hero Member

    @hek hopefully all smartphone now have a calibrated microphone that could help calibrate this... what 'Ive not worked out enough is to know if the store sensor is suitable for this...

    also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?


  • Admin

    @epierre said:

    also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?

    Yeah, something like (1023-analogRead(ANALOG_PIN))/10.23


  • Hero Member

    @hek hmmm... what about if this is a 3.3V or a 5V arduino board too ? no ? ?


  • Hero Member

    Coming from the Can one child-sensor-id have multiple values thread I had a look at the initial post and I am wondering if this is what you can call "the latest proposal".

    The rest of this thread seems to be more about which variables but what I am wondering is how all this would be implemented. Looking at the initial post I see:

    New header would be:

    ChildSensorId | ChildSensorType | ValueType | Payload

    My assumption is that there is still a NodeID in the header, so actually:

    NodeId | ChildSensorId | ChildSensorType | ValueType | Payload

    If so, I think there is too much freedom/overhead in the first three fields.

    My expectations as an user is that every sensor, or better "box" is one node and implements one type of sensor with one or more values. In this sense I do not see the use of the ChildSensorId field, so why not use:

    NodeId | SensorType | ValueType | Payload


 

428
Online

7.7k
Users

8.6k
Topics

92.5k
Posts