2.0 Discussion: Units, sensor types and protocol
-
@epierre said:
Can we have too an alert that could be sent back to the gateway, or like an armed mode for the air sensors ?
Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...
Let's not bring infinity into this...
How about just returning "the maximum reportable value" (finite). If the node is capable of detecting "above measurable" conditions, have it define the "maximum reportable" as 1 higher than it's internal "maximum measurable".
The idea is to keep a simple scale of results, where for example we can consistently compute "max/min over 24 hours", or set a trigger on "value >= threshold" without needing to deal with infinity math. Some controller software may not have any concept of infinity, so using it brings in unnecessary incompatibilities.
This allows the controller to define the alert as it wishes. For example, suppose a sensor reports uncalibrated values of 25 to 173, and the node uses 174 for a detected "over-range". The controller can implement a trigger or event at "value > 160" to test for high values (without having to test for infinite values too). Or it could test for "value >=174" if it specifically wants to test for overrange (which could be way over the max measured value or just barely above it).
@Zeph I don't share the same view, since we allow logic in the sensor for min/max interval, why not doing it too for the gas level ? all the more that the limit of the sensor is not a "business logic" but an internal logic linked by the physical capacity of the sensor itself (and which varies with each sensor type...).
My point was more that if you monitor LPG NO2 / NO3 or smoke that could be really harmfull we should provide safeguards anyway... not a big deal, more a safety net...
-
I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible
If I wanted node could send
S_MOTION V_TRIPPED 1 (motion tripped)
S_MOTION V_ARMED 0 (no longer armed)
S_MOTION V_LEVEL 24 (motion quality 24 of 255) (probobly animal)controller decides S_MOTION V_RESET 1
(order not thought of at all, needs to be fixed)
#DEFINE S_DOOR 1 #DEFINE S_MOTION 2 #DEFINE S_SMOKE 3 #DEFINE S_BINARY 4 #DEFINE S_DIMMABLE 5 #DEFINE S_WINDOW_COVER 6 #DEFINE S_THERMOMETER 7 #DEFINE S_HUMIDITY 8 #DEFINE S_BAROMETER 9 #DEFINE S_WIND 10 #DEFINE S_RAIN 11 #DEFINE S_UV 12 #DEFINE S_WEIGHT_SCALE 13 #DEFINE S_POWER 14 #DEFINE S_HEATER 15 #DEFINE S_DISTANCE 16 #DEFINE S_LIGHT_SENSOR 17 #DEFINE S_NODE 18 #DEFINE S_LOCK 19 #DEFINE S_IR 20 #DEFINE S_WATER_METER 21 #DEFINE S_AIR_QUALITY 22 #DEFINE S_CUSTOM 23 #DEFINE S_DUST 24 #DEFINE S_PH 25 #DEFINE S_SCENE_CONTROLLER 26 #DEFINE S_NODE 255 #DEFINE V_CONFIG1 1 #DEFINE V_CONFIG2 2 #DEFINE V_CONFIG3 3 #DEFINE V_CONFIG4 4 #DEFINE V_CONFIG5 5 #DEFINE V_VAR1 6 #DEFINE V_VAR2 7 #DEFINE V_VAR3 8 #DEFINE V_VAR4 9 #DEFINE V_VAR5 10 #DEFINE V_ARMED 11 #DEFINE V_STATUS 12 #DEFINE V_WATT 13 #DEFINE V_PERCENTAGE 14 #DEFINE V_STOP 15 #DEFINE V_LEVEL 16 #DEFINE V_MAX 17 #DEFINE V_MIN 18 #DEFINE V_RESET 19 #DEFINE V_DEW_POINT 20 #DEFINE V_MODE 21 #DEFINE V_ANGLE 22 #DEFINE V_RATE 23 #DEFINE V_VOLTS 24 #DEFINE V_AMPS 25 #DEFINE V_PRESENTATION 26 #DEFINE V_BATTERY_LEVEL 27 #DEFINE V_RESET 28 #DEFINE V_TIME 29 #DEFINE V_ID 30 #DEFINE V_LOG_MESSAGE 31 #DEFINE V_SKETCH_NAME 32 #DEFINE V_SKETCH_VERSION 33 #DEFINE V_FIND_PARENT 34 #DEFINE V_CHILDREN 35 #DEFINE V_VERSION 36 #DEFINE V_INCLUSION_MODE 37 #DEFINE V_GATEWAY_READY 38 #DEFINE V_STATUS 39 #DEFINE V_IR_SEND 40 #DEFINE V_IR_RECEIVE 41For my sound sketch :
S_SOUND - Sound sensor
V_LEVEL - sound level in db
V_LEVEL_MAX - Max sound level
V_LEVEL_MIN - Min sound level
V_LEVEL_AVERAGE - Average sound level
V_LEVEL_RESET - Reset max/min valuez-wave - Vera -> Domoticz
rfx - Domoticz <- MyDomoAtHome <- Imperihome
mysensors -> mysensors-gw -> Domoticz -
For my sound sketch :
S_SOUND - Sound sensor
V_LEVEL - sound level in db
V_LEVEL_MAX - Max sound level
V_LEVEL_MIN - Min sound level
V_LEVEL_AVERAGE - Average sound level
V_LEVEL_RESET - Reset max/min valueFor my vibration sketch :
S_VIBRATION -Vibration sensor
V_LEVEL - vibration level in Hertz
V_LEVEL_MAX - Max vibration level
V_LEVEL_MIN - Min vibration level
V_LEVEL_AVERAGE - Average vibration level
V_LEVEL_RESET - Reset max/min value -
@zeph the smoke sensor is an Air Quality sensor in fact so we coud decribe it this way too.
@hek what about the V_GAS_TYPE proposed below ? that would limit extensing the S and V no ?S_AIR_QUALITY - Air Quality Sensor
V_GAS_TYPE - gas type reported by the sensor
V_LEVEL - gas concentration in ppm/ppb
V_LEVEL_MAX - Max gas concentration level
V_LEVEL_MIN - Min gas concentration level
V_LEVEL_AVERAGE - Average gas concentration level
V_LEVEL_RESET - Reset max/min value
V_STATUS - Tripped status of a gas sensor. 1=Tripped, 0=Untripped
V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed -
For my sound sketch :
S_SOUND - Sound sensor
V_LEVEL - sound level in db
V_LEVEL_MAX - Max sound level
V_LEVEL_MIN - Min sound level
V_LEVEL_AVERAGE - Average sound level
V_LEVEL_RESET - Reset max/min value -
@epierre said:
S_SOUND - Sound sensor
Wouldn't it be good to have a uncalibrated variable for S_SOUND also? Like proposed for S_LIGHT_SENSOR. Reported as:
V_PERCENTAGE - 0-100%
@hek hopefully all smartphone now have a calibrated microphone that could help calibrate this... what 'Ive not worked out enough is to know if the store sensor is suitable for this...
also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?
z-wave - Vera -> Domoticz
rfx - Domoticz <- MyDomoAtHome <- Imperihome
mysensors -> mysensors-gw -> Domoticz -
@hek hopefully all smartphone now have a calibrated microphone that could help calibrate this... what 'Ive not worked out enough is to know if the store sensor is suitable for this...
also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?
-
@epierre said:
also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?
Yeah, something like (1023-analogRead(ANALOG_PIN))/10.23
-
Coming from the Can one child-sensor-id have multiple values thread I had a look at the initial post and I am wondering if this is what you can call "the latest proposal".
The rest of this thread seems to be more about which variables but what I am wondering is how all this would be implemented. Looking at the initial post I see:
New header would be:
ChildSensorId | ChildSensorType | ValueType | Payload
My assumption is that there is still a NodeID in the header, so actually:
NodeId | ChildSensorId | ChildSensorType | ValueType | Payload
If so, I think there is too much freedom/overhead in the first three fields.
My expectations as an user is that every sensor, or better "box" is one node and implements one type of sensor with one or more values. In this sense I do not see the use of the ChildSensorId field, so why not use:
NodeId | SensorType | ValueType | Payload
-
-
what if you want to report multiple temperatures from one node?
Does not sound to logical to me and I would be surprised if more than 1% of the implemented sensors have "double values", so the first question is if we want to carry an field for exceptions.
Multiple temperatures/LEDs/Blinds can still be implemented by adding an additional ValueType V_LEVEL_SECONDARY but a much cleaner approach would be to use another NodeId.
Let's also have a look at this from the perspective on how to represent things at the Controller:
- Is it logical to have two values of the same type reported for a node?
- How does the user know which temperature/LED/blind is what?
-
what if you want to report multiple temperatures from one node?
Does not sound to logical to me and I would be surprised if more than 1% of the implemented sensors have "double values", so the first question is if we want to carry an field for exceptions.
Multiple temperatures/LEDs/Blinds can still be implemented by adding an additional ValueType V_LEVEL_SECONDARY but a much cleaner approach would be to use another NodeId.
Let's also have a look at this from the perspective on how to represent things at the Controller:
- Is it logical to have two values of the same type reported for a node?
- How does the user know which temperature/LED/blind is what?
@daulagari said:
Let's also have a look at this from the perspective on how to represent things at the Controller:
- Is it logical to have two values of the same type reported for a node?
- How does the user know which temperature/LED/blind is what?
This seems to depend on how you conceptualize the system. In @hek 's proposal and the current system, the core concept of an endpoint is the child aka "sensor". (I'm going to cal this the child/sensor; elsewhere it's often called a "sensor' which as you say can be confusing, but what we mean is the conceptual element addressed using a child id within a node)
The node is a device with an RF transceiver (addressed by node id which is part of the transceiver address), which contains and makes accessible one or more child/sensors, each identified by the node id and child id. The controller receives reports from child/sensors and sends commands to child/sensors. The node is just part of the transportation system.
The controller is configured to know what each child/sensor consists of..
In that context, your first question could be split:
Is it logical to have two values of the same type reported for a child/sensor?
Is it logical to have more than one child/sensor supported by a single transceiver+uC (ie: node)?@hek made the point a while back that a child/sensor can be thought of as a semantic container for one (or sometimes more) closely related values. A node can contain one or more of those.
Yes, there would be an alternative, in skipping the child/sensor container layer. One approach would be to support only a single sensor per node, but there is clearly interest in multi-sensor nodes. Another would be to dissolve the child layer of this heirarchy use rules something like:
- a node can report on (or accept control for) more than one sensor, with more than one value
- no multi-sensor node can have more than one sensor which reports the same kind of value
- the controller will be responsible for logically grouping related sensor/control values within a node
I think one could get by with this for now, but I'm not sure it's a clean conceptual architecture to continue to expand for the future. And I wonder if it puts too much onto the controller - not a big deal if there was only one controller type supported (eg: Vera only), but with an expanding list of controllers, any functionality (like conceptual grouping of the values within a node into "sensor" containers) which is moved to the controller, requires that functionality to be programmed into every controller's architecture in whatever languages that requires - a lot of parallel maintenance to keep in sync for any changes. The less functionality is moved into the controllers, the easier it will be to maintain.
- How does the user know which temperature/LED/blind is what?
Exactly. Is this configured per value, or per child/sensor? (If no sensor ever had more than one value, this would be moot).
Currently the "what" is configured per child/sensor.
Suppose next year one wanted to measure the effectiveness of a swamp cooler with intake temperature and output temperature. Does that need to require two nodes, because the architecture was designed such that no node can report two temperatures?
If the goal is just to create an architecture which can get by for most systems today, there's not need for a big rethinking or the labor of rewriting the code - version 1.4 already does that. I think the point of this thread is to define a flexible conceptual base that's likely to be more cleanly extensible for a few years of growth. I think that hek's child/sensor concept is somewhat helpful in creating that flexibility. Perhaps there's an even better approach to serve that function, but just omitting the child/sensor container level doesn't currently strike me as such.
-
I would agree, I rely heavily on the child/sensor container level. I typically build nodes that poll multiple types of sensors.
-
Look at how other systems do this (for those that do not agree with the current setup).
Multi sensors are used all over the place.
The very well known temp/humid/lux/motion sensors but also tilt sensors combined with a tamper alarm.
And when it comes down to using more of the same sensors on one node: i am going to build a sensor that has two IR sensors for 2 directions. -
@Zeph , @ServiceXp, @marceltrapman Thanks for the feedback.
I would agree, I rely heavily on the child/sensor container level. I typically build nodes that poll multiple types of sensors.
and
The very well known temp/humid/lux/motion sensors but also tilt sensors combined with a tamper alarm.
Both are still perfectly possible.
And when it comes down to using more of the same sensors on one node: i am going to build a sensor that has two IR sensors for 2 directions.
That is what I think is the "less than 1% of the implemented sensors"
Note that the current proposal:
NodeId | ChildSensorId | ChildSensorType | ValueType | Payload
has one more field than the 1.4 protocol.
My proposal is basically to merge the ChildSensorId and ChildSensorType into one, being the ChildSensorType, so:
NodeId | ChildSensorType | ValueType | Payload
If the goal is just to create an architecture which can get by for most systems today, there's not need for a big rethinking or the labor of rewriting the code - version 1.4 already does that. I think the point of this thread is to define a flexible conceptual base that's likely to be more cleanly extensible for a few years of growth.
See above, the proposal adds an additional field and yes, that gives additional flexibility but with the price of one additional byte (or at least field) for every message and additional implementation problems for Controllers on how to represent this.
My question still is how a controller should represent this. Most controllers work with devices that can have one or multiple values. Device can map to node but how to map the remaining fields?
-
@Zeph , @ServiceXp, @marceltrapman Thanks for the feedback.
I would agree, I rely heavily on the child/sensor container level. I typically build nodes that poll multiple types of sensors.
and
The very well known temp/humid/lux/motion sensors but also tilt sensors combined with a tamper alarm.
Both are still perfectly possible.
And when it comes down to using more of the same sensors on one node: i am going to build a sensor that has two IR sensors for 2 directions.
That is what I think is the "less than 1% of the implemented sensors"
Note that the current proposal:
NodeId | ChildSensorId | ChildSensorType | ValueType | Payload
has one more field than the 1.4 protocol.
My proposal is basically to merge the ChildSensorId and ChildSensorType into one, being the ChildSensorType, so:
NodeId | ChildSensorType | ValueType | Payload
If the goal is just to create an architecture which can get by for most systems today, there's not need for a big rethinking or the labor of rewriting the code - version 1.4 already does that. I think the point of this thread is to define a flexible conceptual base that's likely to be more cleanly extensible for a few years of growth.
See above, the proposal adds an additional field and yes, that gives additional flexibility but with the price of one additional byte (or at least field) for every message and additional implementation problems for Controllers on how to represent this.
My question still is how a controller should represent this. Most controllers work with devices that can have one or multiple values. Device can map to node but how to map the remaining fields?
@daulagari said:
My proposal is basically to merge the ChildSensorId and ChildSensorType into one, being the ChildSensorType, so:
OK. That touches on one of my critiques of the direction it's going. (Maybe "critique" would sound stronger than I intend, I'm not disparaging the current direction, just suggesting another one).
I think that ChildSensorID is an appropriate concept for identification and addressing (ie: included in packets), while "ChildSensorType" is static metadata which should be collected during configuration but does not need to be passed in every packet. So I would simplify in the other direction - get rid of ChildSensorType in the packet addressing heirarchy, rather than getting rid of ChildSensorID.
This idea is not currently popular, in large part I think because the metadata describing a given ChildSensor would have to be collected/stored by each Controller plugin, as the currently favored architecture is to have a very thin gateway node which just passes packets between controller plugins and the OTA packets. My alternative vision would be to have a thicker gateway which could, for example, store the metadata for each ChildSensor (eg: ChildSensorType, but expandable) gathered from nodes during config, and provide it to the controller plugins.
With this approach there is no barrier to having multiple children of the same type (but with different ChildIDs) because the concept of addressing & identity have not been conflated with the ideas of types and metadata. In my preferred alternatives, the normal OTA packet for routine reports and commands omits the semantics and includes only addressing/identity, while the initial configuration passes semantics, types and names that are relatively static and do not change per-packet. Moving ChildSensorType from per-packet addressing to static metadata would be one of the implications of this design principle. (If you apply this to values, then we also include a numeric ValueIndex in the packet and move the ValueType info to the metadata.)
(I went even further and suggested omitting the ChildID from the regular OTA packet by having only NodeID | ValueIndex for addressing, and treating ChildID as static metadata which can be associated with each ValueIndex. Thus for example we could say that ValueIndex 3 contains a temperature value associated with child 2, and ValueIndex 4 contains a humidty value also associated with child 2. The central system (thick gateway or controller plugin) would use saved metadata to know which values come from or go to a given child ID container, but the OTA packet doesn't have to include that every time. Not surprisingly, this was a bit too radical a revamping and has not gained traction. And that's OK too, I'm only one voice with only part of the picture here).
Just to be complete, EncodingFormat (eg: 1, 2, or 4 byte integer or unsigned interger, or 4 byte float, or float encoded a text string etc) could be either config time metadata if it's always the same, or could be included in the OTA packet if it changes dynamically. I can see a case for either design.
I'm mentioning this because you seem interested in looking into deeper changes to the architecture, but I'm not pushing these concepts here now - my ideas on such things have been heard and a few have influenced the direction hek is going, but the overall design concept has not been persuasive enough. That's fair.
The system here is pretty good already, so it's not critical to make such changes. So rather the push MySensor too much, I've been (intermittantly) working on my own protocol ideas (at the spec level) which incorporates these and other concepts, and which could co-exist with MySensors. However my tentative design has additional design goals including high bandwidth distribution of pixel control data and medium high bandwidth collection of rapidly changing "sensor" data (eg: joysticks used for low latency pattern. It would be able to handle the MySensors use case of low bandwidth sensor networking as a less challenging subset. However my design is an off-and-on project on paper so far, while MySensors is a working system, so I give a lot of weight to the latter.
-
Yes, we should set the goals high. But we need a realistic plan using the limited development resources we actually have where people should be able to pitch in when they have some time over between family and work.
I've seen so many Open Source (and workplace) projects die due to grand unrealistic plans where there just isn't enough resources or time to realize them. The key here is doing things in small graspable steps.
So we have a few things to consider and prioritize.
- Hardware requirements on the gateway. Today we have chosen to keep this as simple as possible but this might change in the future and nothing stops anyone from creating a gateway directly on RPI or ATMega2560 which could handle more advance MQTT stuff ot handle the id-handout.
- Support for multiple radios. Ongoing work started by @ToSa with a hanging PR (were have you been hanging lately BTW ;) )
- Keep plugin developers happy with a low threshold to create a plugin. We should keep ourselves from making too large serial protocol changes (too often) or we'll lose many supported controllers along the way.
- Support more sensor types with examples. This thread is the playground for creating a more complete set of supported sensors. In the spirit of this I've spend the last weeks trying to create more fun examples. The sensor-examples (and controller support) is the key factor to grow a larger audience (and hopefully more developers).
- Support for thing like encryption and simple OTA firmware updates.
The more people helping out the faster we'll get there. :D
-
Yes, we should set the goals high. But we need a realistic plan using the limited development resources we actually have where people should be able to pitch in when they have some time over between family and work.
I've seen so many Open Source (and workplace) projects die due to grand unrealistic plans where there just isn't enough resources or time to realize them. The key here is doing things in small graspable steps.
So we have a few things to consider and prioritize.
- Hardware requirements on the gateway. Today we have chosen to keep this as simple as possible but this might change in the future and nothing stops anyone from creating a gateway directly on RPI or ATMega2560 which could handle more advance MQTT stuff ot handle the id-handout.
- Support for multiple radios. Ongoing work started by @ToSa with a hanging PR (were have you been hanging lately BTW ;) )
- Keep plugin developers happy with a low threshold to create a plugin. We should keep ourselves from making too large serial protocol changes (too often) or we'll lose many supported controllers along the way.
- Support more sensor types with examples. This thread is the playground for creating a more complete set of supported sensors. In the spirit of this I've spend the last weeks trying to create more fun examples. The sensor-examples (and controller support) is the key factor to grow a larger audience (and hopefully more developers).
- Support for thing like encryption and simple OTA firmware updates.
The more people helping out the faster we'll get there. :D
@hek said:
Keep plugin developers happy with a low threshold to create a plugin. We should keep ourselves from making too large serial protocol changes (too often) or we'll lose many supported controllers along the way.
I do not mean the following to insist on any changes - I accept that there is a balance to be had between changing things for the future and making today's changes as small as possible, and that somebody has to make that judgement call.
But just to explain where I have been coming from and why I have bothered explaining the alternative approach (clean separation between dynamic addressing/identity and static metadata) - it's because of exactly your point quoted above. If we run into limitations in the architecture a year from now and wish we'd thought it out better, it's going to be a lot harder to fix things because there will be a lot more plugins already written by then. It could be better to bite off that transition now when many HA system interfaces to MySensors are not yet written, and the ones that do exist are still in early development with the original developers highly involved. A year or two from now, more interfaces will exist, and some of the early implementers may have moved on to other foci.
The concrete of decisions will be setting into the stone of legacy code over the next year. Anything substantial that isn't in the next revision of the protocol - being discussed in this thread - is likely to never happen (lots of small tweaks may happen of course). The cusp of going from 95% Vera users with almost all architecture-tied code from a single developer, to a broad array of HA system interfaces written by many groups, is now, and it's going to change the degree of flexibility. This may be the last chance to do a deep rethinking.
The other point was that I was suggesting how to simplify the task of plugin writers, by have a single codebase within the MySensors universe (eg: Arduino C++) for as many of the tasks as possible. That lowers the bar some, and reduces the amount of parallel development of doing the same thing in multiple plugins (like assigning addresses).
OK, I wanted to be understood that my purpose was very much in line with the quoted desiderata, not opposed to it - but I'll be cooperating with the direction @hek chooses.
