2.0 Discussion: Units, sensor types and protocol

[hek]

@Zeph

FYI: Sleeping sensors (who also can be woken by a interrupt on a pin) will have a hard time keeping time without clock module or by fetching time from server at each wakeup from pin trigger.

Good idea about a STATUS for RAIN sensor. It will be added and indicates if rain is ongoing.
Will also change to ACCUMULATED for all accumulated values. Average for these makes no sense.

So.. I see now that there is need for more than one reset command. We'll need RESET_ACCUMELATED, RESET_LEVEL, RESET_PERCENTAGE. They should also have their respective MAX, MIN, AVERAGE. Could you please give me a use case for V_LEVEL_SUM.

Just to clarify for anyone getting a bit scared of all these new variables. Using them for your own sensor is of course optional!

I feel it's time to move the new stuff to the first post to keep it in focus.

[Damme]

I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible

If I wanted node could send
S_MOTION V_TRIPPED 1 (motion tripped)
S_MOTION V_ARMED 0 (no longer armed)
S_MOTION V_LEVEL 24 (motion quality 24 of 255) (probobly animal)

controller decides S_MOTION V_RESET 1

(order not thought of at all, needs to be fixed)

 #DEFINE S_DOOR	1
 #DEFINE S_MOTION	2
 #DEFINE S_SMOKE	3
 #DEFINE S_BINARY	4
 #DEFINE S_DIMMABLE	5
 #DEFINE S_WINDOW_COVER	6
 #DEFINE S_THERMOMETER	7
 #DEFINE S_HUMIDITY	8
 #DEFINE S_BAROMETER	9
 #DEFINE S_WIND	10
 #DEFINE S_RAIN	11
 #DEFINE S_UV	12
 #DEFINE S_WEIGHT_SCALE	13
 #DEFINE S_POWER	14
 #DEFINE S_HEATER	15
 #DEFINE S_DISTANCE	16
 #DEFINE S_LIGHT_SENSOR	17
 #DEFINE S_NODE	18
 #DEFINE S_LOCK	19
 #DEFINE S_IR	20
 #DEFINE S_WATER_METER	21
 #DEFINE S_AIR_QUALITY	22
 #DEFINE S_CUSTOM	23
 #DEFINE S_DUST	24
 #DEFINE S_PH	25
 #DEFINE S_SCENE_CONTROLLER	26
 #DEFINE S_NODE	255
 
 #DEFINE V_CONFIG1	1
 #DEFINE V_CONFIG2	2
 #DEFINE V_CONFIG3	3
 #DEFINE V_CONFIG4	4
 #DEFINE V_CONFIG5	5
 #DEFINE V_VAR1	6
 #DEFINE V_VAR2	7
 #DEFINE V_VAR3	8
 #DEFINE V_VAR4	9
 #DEFINE V_VAR5	10
 #DEFINE V_ARMED	11
 #DEFINE V_STATUS	12
 #DEFINE V_WATT	13
 #DEFINE V_PERCENTAGE	14
 #DEFINE V_STOP	15
 #DEFINE V_LEVEL	16
 #DEFINE V_MAX	17
 #DEFINE V_MIN	18
 #DEFINE V_RESET	19
 #DEFINE V_DEW_POINT	20
 #DEFINE V_MODE	21
 #DEFINE V_ANGLE	22
 #DEFINE V_RATE	23
 #DEFINE V_VOLTS	24
 #DEFINE V_AMPS	25
 #DEFINE V_PRESENTATION	26
 #DEFINE V_BATTERY_LEVEL	27
 #DEFINE V_RESET	28
 #DEFINE V_TIME	29
 #DEFINE V_ID	30
 #DEFINE V_LOG_MESSAGE	31
 #DEFINE V_SKETCH_NAME	32
 #DEFINE V_SKETCH_VERSION	33
 #DEFINE V_FIND_PARENT	34
 #DEFINE V_CHILDREN	35
 #DEFINE V_VERSION	36
 #DEFINE V_INCLUSION_MODE	37
 #DEFINE V_GATEWAY_READY	38
 #DEFINE V_STATUS	39
 #DEFINE V_IR_SEND	40
 #DEFINE V_IR_RECEIVE	41

[Zeph]

@hek said:

@Zeph

FYI: Sleeping sensors (who also can be woken by a interrupt on a pin) will have a hard time keeping time without clock module or by fetching time from server at each wakeup from pin trigger.

Sleeping nodes may have a hard time averaging in some cases too.. But your point is well taken and it is a real concern.

One of the sleep libraries calibrate the RC watchdog timer against the main clock, so it can make a good guess for adjusting milliseconds upon each wakeup. That could help for a simple sleep, but not for waking up from an interrupt.

So the problem child is a node for which all the following apply:

• sleeps
• is awoken by interrupts at unpredictable intervals
• has no local RTC
• has accumulated variables

One option would just be to ignore the accumulated time feature for the subset of nodes in that category; for them, you have to fall back to where would be be if I had never spoken up about tracking time. For other nodes, the tracking & reporting of accumulation time can be an enhancement, as described in an earlier message.

But suppose we DO want to go further and determine accumulation time for all nodes, even the above.

(Sigh. I acknowledged that there was a simplicity advantage to not dealing with time, and that just using the local node's millis() was the simplest but not only way to handle time; and that does handle most cases but not the above)

System Time approach 1: Master Clock Broadcast to nodes

The hub will have the master clock, using its millis(). Periodically it will broadcast the value of this to all nodes. A node could also request a time be sent. We could apply a crude delta to adjust this according to number of hops and typical transmission time, or not - we don't need anywhere close to millisecond accuracy. Each node would maintain a delta between its own millisecond clock and the broadcast millis and adjust that every time it receives that broadcast. (Smart nodes could even use a computed time scaling factor to adjust for differences in their clock vs the master). When for any reason has a need to estimate the current time (including but not limited to the case of having been powered up from sleeping by an interrupt AND needing to repost accumulations with more precision than it can estimate), the node can either wait for the next broadcast, or request one.

System Time approach 2: Track accumulation times in the hub

Whenever an accumulation period is started, the node will send a message to the hub to that effect (telling it which variable is involved and an incrementing reset count). This could be triggered by a commanded or manual reset, or by the automatic reset of an external weather station or service, or whatever. Then it would send the reset count (V_RESET_COUNT) rather than the accumulated time along with accumulation based values. The hub can look at its current time, and at the recorded time of the last reported reset of that variable, to compute for itself the elapsed time. The reset count makes sure both sides are sync'd as to which reset they are talking about. Both the "Starting accumulation period #n for (variable)" and the "reporting accumulated value for (variable)" messages travel from node to hub, and their travel times will roughly cancel out, at least well enough.

The former approach is more powerful. It better supports rolling averages. And it would provide a flexible "MySensors network time" framework for future features, like reporting event times or periods more accurately.

For example, I've considered putting two motion detectors at angles and with masks such that one could usually detect which way somebody went through a doorway by the trigger times, or at different place in a hallway. I'm not proposing to support that directly at this time, but when it comes up, if each node had the option of knowing the hub's master clock time to within tens of milliseconds or better, that could be a useful base upon which to build.

[Total aside. My ambition for another nRF + Arduino system is more precise than this, possibly even enough to do PWM of 60 Hz mains powered lights without a local zero crossing detector - not yet determined. That's why I connect the nRF24L01+'s IRQ output to Arduino pin 8 = ICP1, even if I'm not using interrupt driven nRF code. If the nRF's receive interrupt is enabled, this allows the ATMega328p to get a very accurate time of reception using Timer 1 if it wants, even in polling mode without interrupts. And if I ever do want to interrupt the ATMega from the nRF, I can use the capture interrupt, while leaving INT0 and INT1 on pins 3 & 4 for other uses. The above first option is a smaller, simpler version of that design, adapted for the needs of a sensor network]

[Zeph]

@hek said:

Could you please give me a use case for V_LEVEL_SUM.

• "snapshot in time" values like temperature and humidity and watts logically accumulate as (averages. min, and max) over the given period as their end product. Tuesday the average temperature was 22.3 degrees.

• "incrementing total" values like KWH, liquid flow, and rainfall logically accumulate as total or (sum) over the given period as their primary end product: Tuesday a total of 20mm of rain fell. (IFF you know the time as well, you can calculate an average as a secondary product)

These two types of values have somewhat different dynamics.

Of course if you report a total AND an elapsed time, the hub can calculate averages for itself, but not min and max. This calculation can be done at the hub even for temperature, tho total degrees C is not meaningful and total mm rain is.

So IF we can rely on S_LEVEL_ELAPSED and S_LEVEL_SUM both always being available (and the latter not overflowing integer formats), we could omit S_LEVEL_AVG. But in the original proposal without a time framework, the node should report S_LEVEL_SUM for rain and S_LEVEL_AVG for temp.

[Zeph]

EDIT - there are some good concepts in the next few posts from me (IMHO obviously), but some may turn out to be sidestepped by a more radical change described in response
http://forum.mysensors.org/topic/304/unit-standardization-1-4/57,
so you might want to just read that first.

---

@hek said:

###S_WIND - Wind sensor
V_LEVEL - Wind level in m/s (average wind speed during last report period)
V_LEVEL_MIN - Min wind level
V_LEVEL_MAX - Max wind level (gust)
V_LEVEL_AVERAGE - Average wind level-
V_LEVEL_RESET - Reset max/min value
V_ANGLE - radians <float or int>

First, i still want to convince you that V_ANGLE in "degrees clockwise from true north" is the proper measure here, not radians. (and you need to define if this points to where the wind comes from or where it's moving towards, so is a west wind 90 degrees or 270 degrees - or translate to radians if you want to see ugly)

Look, I'm a math person (or was) and I totally get that e, national logs and exponents, pi and radians are more fundamentally defined values than degrees or using base 10 for decibels or logs. But the point is not scientific purity (which are are not really close to anyway) but pragmatic utility in the field of sensor networks. And degrees simply work better in this context of conveying direction, especially if you allow int representation. Enough on that for now.

Second, notice that you really only have two real measurements here: speed and direction. Those are what I call snapshot values - if you sample them at a given time, you get a value, which can later rise or fall.

The former has additional derived or calculated sub-values, and those subvalues follow a pattern representing "simple time accumulation statistics" and also common to many other snapshot variable types, like humidity.

The thing is - direction is also a snapshot value. If you are going to report average wind over the past hour, you can also report average direction.

Alas, angle differs from continuum measurements in that it wraps, so average/min/max have to be computed carefully and are only meaningful if the source stays within a reasonable range; and sum makes no sense.

But any continuum variable is a potential candidate for accumulation-period based derived sub-variables. I picture that there will be a single "accumulation" library which can be applied to any int or float variable, rather than that the temperature library will have accumulation code and the humidity library will have its own accumulation code.

So in the hierarchy I'd put V_LEVEL_MIN, etc one level indented under V_LEVEL rather than make it look like a parallel and independent measurement.

---

Let's also be clear that there are two reasons to have accumulation period based derived variables. One is that the node could be getting values from a device which does its own accumulation of some sort (eg: a weather station or some rain gauges). The other is that the device may sample the instantaneous variables more often than it transmits them. If we transmitted every measurement, the hub could potentially do its own min/max/average/sum calculations. No big deal there, but the hub could also accumulate by multiple arbitrary periods, like hour of the day AND since sunrise AND past 10 minutes AND past hour AND since Oct 1st (which is the start of the water year as used by scientists around here). So the most flexible thing would be to report every measurement to the hub and let it handle aggregation any way (as many ways) as it wants. But that could be bandwidth heavy, or might impose too much work on simple hubs (or controllers they report to).

So for one or more of these reasons, sometimes it may make sense to have the sketch running in a node handle accumulation sub-variables. If it reports often enough in a well considered format, the hub or controller or cloud based analytical software can still do aggregation over some of these alternative time periods.

[Zeph]

@hek said:

###S_BINARY - Light (or other) on/off actuator
V_STATUS - 1 - turn on, 0 = turn off

###S_DIMMABLE - Dimmable actuator
V_STATUS - 1 - turn on, 0 = turn off
V_PERCENTAGE - Dimmer level 0-100%

OK, let's define what happens if you get more than one type of command (probably just making explicit what we are thinking anyway).

S_DIMMABLE:
V_STATUS = 1 is the same as V_PERCENTAGE = 100, either has same effect
V_STATUS = 0 is the same as V_PERCENTAGE = 0, either has same effect

If you set V_STATUS = 1 and then query V_PERCENTAGE it should return 100% (or 0 => 0%)
If you set V_PERCENTAGE = 100 and query V_STATUS it should return 1

OK?

The only question is setting V_PERCENTAGE = 1..99 and querying V_STATUS.
Option 1: Return -1 for "neither fully on nor fully off"
Option 2: Return 1 for "at least partially on" (ie: anything above 0%)

---

OK, what about a Fan with setting Off, Low and High, and another fan with settings "Off, Low, Medium and High?
We could map those to text "OFF, LOW, MED, HIGH"
Or we could map them to 0,1,2 where 2=HIGH for the first fan, and 0,1,2,3 where 3 = HIGH for the second

I'm going to suggest that we primarily map to V_PERCENTAGE, but perhaps also support the latter option as an expansion of binary 0..1 into multi step value 0..N_STEPS.

All fans:
    OFF = 0%
    HIGH = 100%
Fan 1:
    Low = 50%
Fan 2:
    Low = 33%
    Med = 67%

If you are sent a V_PERCENTAGE, set the fan to the closest speed to the given percentage. 100% is always the highest speed, 0% is always OFF, in between will approximate. This extends to 4 level fans. And it works for a heater or motor with multiple speeds as well. If you query V_PERCENTAGE the actual value chosen from the list will be returned, not the requested value (ie: requesting 45% but reporting 50% in the case of fan 1).

The above intermediate percentages are the default. However the node is empowered to set other values. We all know of fans or other devices where the speeds are not evenly spaced. I have a heater which has 600, 900 and 1500 watt settings (both elements). The Low and Medium setpoints could be 40% and 60%. Otherwise it works as described above.

And there could be an integer V_SET_STEP (or something), corresponding to V_BINARY* but ranging from 0 to N_STEPS where N_STEPS is per-variable defined. I'm not attached to this, but it's an option, It would work in the obvious way. So V_SET_STEP of 1 for Fan1 would set it to Low=50% and V_SET_STEP of 1 for fan2 would set it to Low=33% (and V_SET_STEP of 1 for the heater example would set it to Low=40%. SET_STEP of 0 is always off, and SET_STEP of N_STEPS is always HIGH or 100%

---

For completeness, I will mention that there's another way to define this for V_DIMMABLE. It could be required to memorize the last set percentage, and then V_STATUS = 0 means turn to 0% but remember last non-zero percentat, and V_STATUS = 1 could mean return to last non-zero percentage.

Tho I don't favor this dynamcic, it too could work. However I strongly favor defining which way we expect things to work rather than leaving it ambiguous!

[Zeph]

@Damme said:

I dont know why we have to limit some S_type to a number of V_types. Why not just let a S_type have 'all' V_types... User can be more flexible

As described at the time I write, the proposed taxonomy of values isn't really as heirarchical as it's presented.
You can also parse the meaning from V code to S code:

Ah, we have a V_LEVEL value, that means it represents a one dimensions continuum
But what does that continuum represents?
Oh, I see the variable is also tagged with S_TEMPERATURE
That means the continuum is temperature in degrees C

That is, the V code tells you something of the structural concept, and the S code adds some degree of semantics (like dimension and unit).

So you could also invert the displayed hierarchy and show which S codes are available for a given V code

[Zeph]

Hold the presses.

I've happily engaged in the collaborative process of cleaning up the variable taxonomies of V 1.4b here.

But we keep coming to conceptual tricky parts. Some of that is inherent, but in thinking deeper about it, i've come to the conclusion that a good part of what makes it hard to resolve cleanly is that we are trying to cram too much information into too few bits of variable metadata which has to be included in every packet.

So rather than reforming S and V codes, I'm thinking a more radical approach is needed, to make the system both more flexible and simpler. Part of that comes from identifying what metadata about a variable is static and what needs to be dynamic, and then moving the static part out of the normal OTA operations. That turns out to make the taxonomy much more tractable and flexible while simplifying the node firmware.

Guess what - even child ID turns out to be metadata in this approach, rather than addressing.

Here's the sketch of the new approach.

http://forum.mysensors.org/topic/308/another-way-of-organizing-variables

It's sufficiently different that I expect it may encounter resistance - from the natural and appropriate fear of breaking too much (I share that concern). But it just may turn out to be sufficiently attractive to be worth it. Have a look. And now would be the time to consider such a break - with RadioHead and OTA programming support also in the air.

[ToSa]

@hek said:

This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

I switched the bootloader code between using stream and using simple internal messages - at the end doesn't matter as long as it's simple enough to dis-samble and assemble a packet without a lot of additional code overhead.

[hek]

@ToSa

Sorry, forget I brought this up. We'll keep the STREAM command as-is. Skyped you about it yesterday

[ToSa]

@Zeph said:

But suppose we DO want to go further and determine accumulation time for all nodes, even the above.
...
System Time approach 1: Master Clock Broadcast to nodes
...
System Time approach 2: Track accumulation times in the hub

I'm in favor of approach 1 as well - and an optional addition to the MySensors class to keep track of time. Only specific sensor types would use this feature - interesting mainly for actuators rather than sensors:

Let's assume you use a relay to turn on/off your garden watering while you are on vacation. The network breaks (controller or gateway or whatever node needed to relay messages).

• If the sensor node itself knows about the time and the schedule for watering, it can turn on and off the watering itself. If there is an issue with the communication, the time would not be 100% accurate over time but still good enough.
• If the sensor node relies completely on the controller for time and (worst case) the communication breaks just between the start watering and stop watering command you can imagine what happens...

Knowing about the time heavily reduces the single points of failure. The protocol already covers the "request time" communication so this should be relatively easy to implement...

• controller sends a time broadcast every e.g. 15 minutes
• only the nodes interested in time care about it and update their internal time
• when a node needs the current time it calculates it based on the last timestamp received, the millis() when it was received and the current millis...

Doesn't solve the sleep issue (as long as you don't want to submit a separate "get time request" every time you wake up) but at least for sensors that are not battery powered (relay nodes in most cases are not) it's a feasible approach.

This is probably worth a separate thread

[hek]

@ToSa, @Zeph

Yes, and to extend this even further it would be great to be able to distribute/push simple rules all the way out to the nodes. It might be enough with simple scheduler-commands to start with but this could easily support even advances scenarios. The question is where to set the cuttingpoint...

The most extreme solution would be to allow a graphical rule engine on the controller which automatically generates a sketch, compiles it and sends it out OTA to the node.

Definitely worth discussing in a new thread...

[Zeph]

@hek said:

Yes, and to extend this even further it would be great to be able to distribute/push simple rules all the way out to the nodes. It might be enough with simple scheduler-commands to start with but this could easily support even advances scenarios. The question is where to set the cuttingpoint...

For the level of protocol under current discussion, the cuttingpoint could just be giving nodes an optional sense of time (using broadcast packets to keep in sync with the master as needed). And making use of that time as part of the (optional) accumulation enhancement for some variables.

What else they do with that time at higher layers would continue to evolve in the future. We don't have to bite it all off now.

[Rasmus Eneman]

V_R_PERCENTAGE - Red component % <int>
V_G_PERCENTAGE - Green component % <int>
V_B_PERCENTAGE - Blue component % <int>
V_W_PERCENTAGE - White component % <int>

Just a hundred steps might be to low, especially for green in the lower end which the eye sees much brighter.
Please consider floats or per thousand instead.

[hek]

@Rasmus-Eneman

Ok! Didn't know that. But couldn't the sensor code adjust this with some sort of color curve?

[Rasmus Eneman]

@hek
That would probably do it, however one would need to be defined and standardized so that all behaves the same. One more clever than me needs to propose how the curve would look tough.

[John]

@hek @Rasmus-Eneman

S_RGBW - RGBW Light
V_STATUS - 1 - turn on, 0 = turn off
V_R_PERCENTAGE - Red component % <int>
V_G_PERCENTAGE - Green component % <int>
V_B_PERCENTAGE - Blue component % <int>
V_W_PERCENTAGE - White component % <int>

Ain't it better to split up S_RGBW to two different types. W is already present in full R, G, and B. For example i have implemented Philips Hue and there is a difference in fixture types some do RGB some only W. If an user would implement such with mysensors the same option can be possible. An S_RGBW declaration for a W only would indicate RGB capabilities and maybe cause confusion.

Just a hundred steps might be to low, especially for green in the lower end which the eye sees much brighter.

Agree, but thousands could be too much (which never is the case of course), maybe 0-255 would be sufficient and widely used (all though not as precise in steps as with thousands)? Also the W_PERCENTAGE should be left out because this is full R,G,B or similar calibrated R,G,B levels (which is quite difficult to do at every possible level) with the end result it then in fact is a brightness level instead of a white component.

An addition could be S_HSB/S_HSV with float,float,float (0.0-360.0, 0.0-100.0, 0.0-100.0)? Also i think the color presentation should be or calibrated on the device or in the controller. Even some of those Philips Hue's fixtures are often controller side calibrated (better said color index modified after color selection)....

[hek]

I actually have RGBW led strips mounted around our house. W is indeed its own component (Warm White in my case). So it will stick

(Warm White using the special W-component)

Cold white (RGB activated)

Crazy

[hek]

@John

But switching to LEVEL and using 0-255 would be ok (if we need it ;)).

[John]

@hek
Ahhh... understood :)... Splitting those up will not be an option then because it could then create a numerous amount of options....

A S_HSB would be nice though, even if it would only be ints....:
V_STATUS - 1 - turn on, 0 = turn off
V_H_DEGREE - Degree (0-360)<int>
V_S_PERCENTAGE - Saturation % <int>
V_B_PERCENTAGE/V_V_PERCENTAGE - Brightness/Value % <int>
V_WATT - Watt <int>
V_WATT_MAX - Max watt value
V_WATT_MIN - Min watt value
V_WATT_AVERAGE - Min watt value
V_WATT_RESET - Reset max/min value

I must say it is a personal request :$....

[Edit]

But switching to LEVEL and using 0-255 would be ok (if we need it ).

It would create space to add the HSB/HSV option

[Zeph]

What's wrong with setting lights at 1.475% Is percentage limited to integer?

Let's be very explicit about that. Is the idea that all values accept only integers unless they say they have the option of using floats? If so that should be made clear in the list.

---

Another thing that needs to be more clear is how to handle multiple V codes for the same sensacturator (like the RGB or HSV lights).

• Is a sensor expected to report all of them?
• Is an actuator expected to respond to all of them?
• How do both sides stay in sync about subsetting?

By the way, there is a distinction to be made between RGB lights (three blendable primaries able to synthesize many colors in a predictable and fairly standard way) and multichannel control of N lights of arbitrary colors. RGBW is another thing entirely. The W could be cool or warm white. For that matter the fourth color could be yellow or violet. It's not a color space.

If the RGB channels blend I'd implement it as an RGB light plus a monochrome light, because there is no RGBW color space. If they don't blend I'd implement it as 4 monochrome channels, the same way I'd implement a Yellow and Green pair that didn't blend.

[John]

@Zeph

What's wrong with setting lights at 1.475% Is percentage limited to integer?

It just not represents any absolute bit value for a specific color, it is an approximate. If for example you have a fixture which does 255 steps and you want to set it at step 40. you will need to send 15.682.....%. But what if you would send 15.770... ? Due to the nature of floats it could be rounded down or up to 40 or 41 and especially in the lower steps of a color the difference would be seen if this is a mix of on of the RGB values, and even more as all the RGB values are send in percentages. A percentage would only be usable (But even then it's still an approximate) with 32 bit or analogue blend RGB fixtures. There ain't a lot of these.

@zeph said:

If the RGB channels blend I'd implement it as an RGB light plus a monochrome light, because there is no RGBW color space. If they don't blend I'd implement it as 4 monochrome channels, the same way I'd implement a Yellow and Green pair that didn't blend.

I do agree with you, and you're totally right, but:

@john said:

Splitting those up will not be an option then because it could then create a numerous amount of options....

In other words it could eat up the possible S_TYPE's

@hek
Please disregard this one, it wouldn't :"it would create space to add the HSB/HSV option".
But, couldn't it be an option to add the HSB/HSV variables to the S_RGBW? because if the presentation stays you would only have to send the vars that are supported, and the type S_RGBW could be renamed to something like S_LIGHT_COLOR?

[hek]

The question is where the actual calculation is done between RGB and HSB/HSV. Should it be done on the Arduino side or controller? Nothing stops you from making a fancy HSB/HSV GUI on your controller but the actual values send to the actuator is RGB. But my knowledge of color scales is a bit limited and I'm probably missing something?

[John]

@hek

It depends, if the HSB scale is purely in integers you will have about 3.600.000 options (including brightness steps) which is much less then 16 bit 0,0,0 - 255,255,255 scale. But this scale ain't specific in int's it is an implementation chosen scale. If choosing for ints, yeah keep the RGB, if going to floats, add it next to the RGB.

probably missing something?

The main difference is that with pure 16 bit you will have whole bit numbers and with degrees there is more then just go from 359 to 360/0. you could use 359,00 to 359,99 which gives an extra 990.000 extra red combinations (including saturation and brightness) with absolute 16 bit you stick with whole ints/bytes limiting the possible fixtures light high "resolution" capabilities.

Also because of the nature of HSB with higher color resolution possibilities you will always have the possibility to find the closest 16 bit neighbor or even the exact bit value because of higher to lower "spectrum" resolution.

With the above if you would leave out the brightness you would have a higher color resolution 255,254,255 is one step lower green form maximum, you can't go to 254.9 right? in degrees i think green is 120, you could go to 119.99999. because the degrees are not fixed to a whole as a byte is.

Because of more steps calibrating un-calibrated fixtures/leds (for example with the green mentioned earlier) can be more precise. Also HSB is more human perspective then fixed bits.

Maybe the above shines a light on the difference?

RGB is just a brightness setting of a specific color, HSB is pure color selection, intensity setting, and the brightness of these.

Personally:
HSB is nice to have, but from personal needs i have enough convertion functions (hsb->rgb and vice, hsb -> kelvin, etc...) to convert.

Should it be done on the Arduino side or controller?

I would let the controller do the actual conversions and let the device spec it's capabilities.

[Zeph]

@hek said:

S_RGBW - RGBW Light
V_STATUS - 1 - turn on, 0 = turn off
V_R_PERCENTAGE - Red component % <int>
V_G_PERCENTAGE - Green component % <int>
V_B_PERCENTAGE - Blue component % <int>
V_W_PERCENTAGE - White component % <int>
V_WATT - Watt <int>
V_WATT_MAX - Max watt value
V_WATT_MIN - Min watt value
V_WATT_AVERAGE - Min watt value
V_WATT_RESET - Reset max/min values

One of the areas where the spec is very unclear is when a variable is writable and where it's only reportable (read only).

For example, can I set the watts to 20 above (or for S_DIMMABLE) in order to tell the light how much power I want it to use? Seems like a valuable option. How am I supposed to know?

What if we made the write/report/trigger function explicit:

S_RGBW - RGBW Light

W_STATUS - 1 - turn on (100%), 0 = turn off (0%)
W_R_PERCENTAGE - Red component % <int>
W_G_PERCENTAGE - Green component % <int>
W_B_PERCENTAGE - Blue component % <int>
W_W_PERCENTAGE - White component % <int>
R_WATT - Watt <int> - current instantaneous or short term average value
R_WATT_MAX - Max watt value since last reset
R_WATT_MIN - Min watt value since last reset
R_WATT_AVERAGE - Min watt value since last reset
T_WATT_RESET - Reset max/min/avg value

W variables can be written to; the current value can also be queried
R variables can only be reported (automatically or by query)
T variables are triggers, writing to them triggers some action

This gives guidance to both the sketch writer for the node, as well the controller adapter writer in knowing what they can do with each variable a node presents.

The actual numeric values for R, W and T codes could be interspersed or non-overlapping ranges could be reserved for each (eg: 1-100 for R, 101-200 for W and 201-255 for T)

[Damme]

@Zeph it all depends on how skilled programmer and how nerdy you are. if you want to set a lamp power with V_WATT great, thats nerdy! but in not, just dont read any thing from v_watt. .. there is no such thing as a read only variable, all depends on if you listen to incoming messages of that type or not.... I think you over-think this a bit. The atmega328 is a rather small mcu and wont be able to magically do tons of stuff. how to know what is what, well every home automation system should have a manual Write it down.

[Zeph]

@John

I can see a case for making HSB the norm.

Think of light types like a class heirarchy, with added functionality

S_BINARY

V_STATUS (1=100%, 0 =0%)

S_DIMMABLE

V_STATUS (1=100%, 0 =0%)
V_PERCENTAGE (0.0% - 100.0%)

S_HSB

V_STATUS (1=100%, 0 =0%)
V_PERCENTAGE (0.0% - 100.0%)
V_HUE (0.0 - 360.0)
V_SATURATION (0.0% - 100.0%)

If you set V_STATUS to 0, or V_PERCENTAGE to 0%, any of these lights turn fully off.
If you set V_PERCENTAGE to 100%, S_DIMMABLE will go to 100% (V_STATUS=1)
and S_HSV will go to 100%, in the current hue and saturation
If you set V_PERCENTAGE to 50%, likewise.

I think one could make the case that it's more human compatible to go to "yellow, 25%" than to figure out what the RGB values would be. (The user interface could translate yellow to a hue number)

One aspect HSB is that there are a couple of ways of associating colors with angles, so we would need to specify which is our standard, so 40 degrees is the same hue for all nodes. I like the rainbow style.

---

@hek
I do also get the point that HSB -> RGB conversions could be done on the controller, passing the lower level RGB values to the node. There is an attraction to that as well.

This cannot be fully resolved without also considering the question of light curves or gamma factors, which has been raised. The controller needs to know whether or not the node implements light curves for RGB channels, so it can avoid doing that twice; and also it in the changes how the controller should convert HSB to RGB (the stock conversions assume RGB are linear).

[Zeph]

@Damme said:

Write it down

That's what I'm trying to do.

We could CLEARLY write down that V_WATT is intended only to report values and cannot be set, while V_RESET is a trigger, and V_R_PERCENTAGE is settable - so different implementors don't make different and incompatible guesses about what these mean.

We do not want the situation where one sketch writer interprets V_WATT to be a way to control how much power the lamp will use and another sketch write interprets it as reporting only. And then controller interface writers also guess differently about what it means. That's chaos.

One step in that direction would be naming them according to the intention - writable, reportable, or trigger.

I've done a LOT of implementing from specifications, as many others have. Having a clear specification that avoids huge ambiguities (is this something you set or something that reports is huge) is not over-thinking - it pays off big time.

[Yveaux]

@hek said:

Crazy

You didn't dare posting a red version, did you?

[John]

@Zeph

I can see a case for making HSB the norm.

I'm not ;). it is just a possible addition of possible value types to the S_RGBW type and if not possible like you stated a S_HSB and to keep clarity V_PERCENTAGE should then be V_BRIGHTNESS. Or both percentages (not possible) or both naming as it is.

One aspect HSB is that there are a couple of ways of associating colors with angles

Off topic, which ones? i only know one, are you sure your not mixing HSB with HSL?

On topic:
I'm fine with only sending RGB if HSB ain't possible to be combined with S_RGBW or added with S_HSB/S_HSV if it is thought of as a logical addition.

[Zeph]

@John
You are probably way ahead of me in terms of formal color spaces. I'm just a color user.

I like using a modified Hue, such the the "rainbow" described here:

https://github.com/FastLED/FastLED/wiki/FastLED-HSV-Colors

[John]

@zeph said

You are probably way ahead of me in terms of formal color spaces

Not really, i'm no color expert, But did a lot with colors lately because i use this to influence our living environment

Ok got it, Hue with rainbow style is variable due to the amount of colors (real life circumstance like angel etc) on that page an almost perfect rainbow angle and lighting situation is used.

I agree with you, but that's user implementation on the hardware, and something mysensors can not control. You would still need the HSB parameters The implementation of rainbow or spectrum should be done on the controller<->end device side and the end user should know about it (or at least the developer of that specific controller/hardware implementation), not protocol based, because the parameters are still the same.

[epierre]

@Hek globally I don't understand the min/max/average, are they managed by the sensor or the controller ? if it is the sensor in case of reboot it should get it back from the gateway/controller and should be resettable maybe to have a year/month/week/day min/max. For me that should be controller side to handle this logic...

Now for my favorite playground it needs the calibration value clearly first, then for the discussion some projects (AirPi) avoid the calibration pitfall by sending back the voltage... If we want to have a concentration value from the sensor that is ok.

 S_AIR_QUALITY - Air Quality Sensor
 V_LEVEL - ? We need to describe this better
 V_LEVEL_RESET - Reset max/min value

@hek Hmm.. Soil Moisture.. I have no idea how to represent this. Help me! The ebay sensors we're using isn't super accurate and comes uncalibrated.

@damme   Hmm... V_Moisture, (relative) 0-255 / 0-100% / True | False

On both I agree, I have some imaginatronix and yes you need a margin of error... but when it rains it rains... this is where a calibration value will be usefull at least to have several sensors showing the same value at the same environment time... I you want a logic then the true false can be good but not enough... on my imagitronix I have an empty value of 20% and a watered value of 100%, sometimes after the rain it is 80%..., where would be the limit ?

Also, I am thinking of a leaf wetness (we had a lot here this year... and diseases that goes with it) that could be mapped to soil sensors [Article on values(http://www.decagon.com/education/do-you-measure-leaf-wetness/)

[hek]

@epierre said:

globally I don't understand the min/max/average, are they managed by the sensor or the controller ? if it is the sensor in case of reboot it should get it back from the gateway/controller and should be resettable maybe to have a year/month/week/day min/max. For me that should be controller side to handle this logic...

The MAX/MIN/AVERAGE are optional to use. My idea was that sensors would manage them. They can choose to store it in eeprom or throw away values when reboot occurs.
The idea originates from GUST for wind which really is a max value over a certain time. This is just a generalization.
Usecase: Sensor might not report every wind value to controller but just every 10 minute. But internally it can calculate average/max/min for every 10 seconds without pushing this.

[Rasmus Eneman]

Please leave HSV/HSL to the controller. Why complicate the node more that necessary?
Using LEVEL would be good, 8 bit per color is good, it's also super easy for the node to scale down if only supporting 7 or 6 bits and it's perfect for those digital LED strips.

Please have a pure RGB without W, most strips and RGB leds don't have a W diode so that would need to be simulated with different result in different nodes.

[John]

@Rasmus-Eneman

Please leave HSV/HSL to the controller. Why complicate the node more that necessary?

That's why i asked for an addition if it would over complicate stuff (and if there is room for it) ;). And well, yeah you're right, S_RGBW with HSB would be complicating things more mixing capabilities with the node naming. There is always S_CUSTOM with V_VAR_1,V_VAR_2,V_VAR_3 combination.

I agree with leaving out the W. But could be a nice addition for those non calibrated fixtures where you have to supply calibrated values in the node to get "proper" white levels.

[epierre]

@hek ok I agree. Can we have too an alert that could be sent back to the gateway, or like an armed mode for the air sensors ?

Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...

[Yveaux]

@epierre In 1.4 a binary represetation of a float value can be exchanged with the library. This allows to report e.g. + or - infinite where normally floating point values are sent. Maybe this will serve your purpose?

[Zeph]

If you do have a strip with RGB and W LEDs (rare), this can be handled as an RGB light and a DIMMER.

---

A lot of our discussions really hinge on how much smarts to put where in the architecture. My overall philosophy would be to keep the node simple and move functionality to it only when the same thing cannot be easily done centrally. Reasons to move smarts into a node could include:

• Uses local calibration that central doesn't have
• Needs updating more often than reports go to central (eg: wind max)
• Timing, eg: safety features that need to act fast
• Reliability of data links, eg: safety features that should not depend on RF robustness
• Providing some logically consistent interface available to all controllers
• Providing some functionality which cannot be as easily implemented in a controller (eg: closed source)

The latter two would be one-off custom coding at this time, tho @hek has mentioned the idea of pushing some rules down to the node level in a more automated way someday.

---

This probably argues more for using RGB rather than HSV at the node level. What I liked about the HSB was the unification of on/off, dimmable, and HSB functionality as a nested set of expanding functionality. So a script or rule that sets the brightness could work on a DIMMABLE or a HSB light by setting the V_PERCENTAGE value (or V_BRIGHTNESS as someone suggested). But that could be done at the central level.

---

I've used the term "central" because I still have the vision that a "hub" or "gateway" could implement some features, rather than having the smarts only in the controller or in the sensactuator nodes. So for example, it could be possible for the gateway to convert HSB to RGB such that the controller doesn't have to implement it. I know that approach is not currently popular so you don't need to register general disapproval. This will either come to seem a worthwhile additional open in coming years or not.

[Zeph]

@epierre said:

Can we have too an alert that could be sent back to the gateway, or like an armed mode for the air sensors ?

Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...

Let's not bring infinity into this...

How about just returning "the maximum reportable value" (finite). If the node is capable of detecting "above measurable" conditions, have it define the "maximum reportable" as 1 higher than it's internal "maximum measurable".

The idea is to keep a simple scale of results, where for example we can consistently compute "max/min over 24 hours", or set a trigger on "value >= threshold" without needing to deal with infinity math. Some controller software may not have any concept of infinity, so using it brings in unnecessary incompatibilities.

This allows the controller to define the alert as it wishes. For example, suppose a sensor reports uncalibrated values of 25 to 173, and the node uses 174 for a detected "over-range". The controller can implement a trigger or event at "value > 160" to test for high values (without having to test for infinite values too). Or it could test for "value >=174" if it specifically wants to test for overrange (which could be way over the max measured value or just barely above it).

[hek]

@epierre said:

Today, one of my daughter broke a bottle of perfume, saturating the air and making the sensor go to its max. Even if the sensor is not calibrated, on some kind that (IMHO) must raise an alert for I cannot say infinite or over the maximum value...

LOL

[epierre]

@Zeph I don't share the same view, since we allow logic in the sensor for min/max interval, why not doing it too for the gas level ? all the more that the limit of the sensor is not a "business logic" but an internal logic linked by the physical capacity of the sensor itself (and which varies with each sensor type...).

My point was more that if you monitor LPG NO2 / NO3 or smoke that could be really harmfull we should provide safeguards anyway... not a big deal, more a safety net...

[hek]

Ok,

I will change create two device-types S_RGB and S_RGBW and use a level instead of percentage for each color component (0-255).

[epierre]

For my sound sketch :

S_SOUND - Sound sensor

V_LEVEL - sound level in db
V_LEVEL_MAX - Max sound level
V_LEVEL_MIN - Min sound level
V_LEVEL_AVERAGE - Average sound level
V_LEVEL_RESET - Reset max/min value

[epierre]

For my vibration sketch :

S_VIBRATION -Vibration sensor

V_LEVEL - vibration level in Hertz
V_LEVEL_MAX - Max vibration level
V_LEVEL_MIN - Min vibration level
V_LEVEL_AVERAGE - Average vibration level
V_LEVEL_RESET - Reset max/min value

[epierre]

@zeph the smoke sensor is an Air Quality sensor in fact so we coud decribe it this way too.
@hek what about the V_GAS_TYPE proposed below ? that would limit extensing the S and V no ?

S_AIR_QUALITY - Air Quality Sensor

V_GAS_TYPE - gas type reported by the sensor
V_LEVEL - gas concentration in ppm/ppb
V_LEVEL_MAX - Max gas concentration level
V_LEVEL_MIN - Min gas concentration level
V_LEVEL_AVERAGE - Average gas concentration level
V_LEVEL_RESET - Reset max/min value
V_STATUS - Tripped status of a gas sensor. 1=Tripped, 0=Untripped
V_ARMED - Armed status of a security sensor. 1=Armed, 0=Bypassed

[hek]

@epierre said:

S_SOUND - Sound sensor

Wouldn't it be good to have a uncalibrated variable for S_SOUND also? Like proposed for S_LIGHT_SENSOR. Reported as:

V_PERCENTAGE - 0-100%

[epierre]

@hek hopefully all smartphone now have a calibrated microphone that could help calibrate this... what 'Ive not worked out enough is to know if the store sensor is suitable for this...

also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?

[hek]

@epierre said:

also I don't know what the semantic of "100% sound" would mean... max sensor capabilities ?

Yeah, something like (1023-analogRead(ANALOG_PIN))/10.23

[epierre]

@hek hmmm... what about if this is a 3.3V or a 5V arduino board too ? no ? ?

[daulagari]

Coming from the Can one child-sensor-id have multiple values thread I had a look at the initial post and I am wondering if this is what you can call "the latest proposal".

The rest of this thread seems to be more about which variables but what I am wondering is how all this would be implemented. Looking at the initial post I see:

New header would be:

ChildSensorId | ChildSensorType | ValueType | Payload

My assumption is that there is still a NodeID in the header, so actually:

NodeId | ChildSensorId | ChildSensorType | ValueType | Payload

If so, I think there is too much freedom/overhead in the first three fields.

My expectations as an user is that every sensor, or better "box" is one node and implements one type of sensor with one or more values. In this sense I do not see the use of the ChildSensorId field, so why not use:

NodeId | SensorType | ValueType | Payload

[hek]

Hmm... what if you want to report multiple temperatures from one node?

[Yveaux]

@hek said:

Hmm... what if you want to report multiple temperatures from one node?

Or for actuators, control multiple LEDs/Blinds etc. from a single node...

I think a few bits could at least be saved by limiting the amount of ChildId's per node.

[daulagari]

what if you want to report multiple temperatures from one node?

Does not sound to logical to me and I would be surprised if more than 1% of the implemented sensors have "double values", so the first question is if we want to carry an field for exceptions.

Multiple temperatures/LEDs/Blinds can still be implemented by adding an additional ValueType V_LEVEL_SECONDARY but a much cleaner approach would be to use another NodeId.

Let's also have a look at this from the perspective on how to represent things at the Controller:

• Is it logical to have two values of the same type reported for a node?
• How does the user know which temperature/LED/blind is what?

[Zeph]

@daulagari said:

Let's also have a look at this from the perspective on how to represent things at the Controller:

• Is it logical to have two values of the same type reported for a node?
• How does the user know which temperature/LED/blind is what?

This seems to depend on how you conceptualize the system. In @hek 's proposal and the current system, the core concept of an endpoint is the child aka "sensor". (I'm going to cal this the child/sensor; elsewhere it's often called a "sensor' which as you say can be confusing, but what we mean is the conceptual element addressed using a child id within a node)

The node is a device with an RF transceiver (addressed by node id which is part of the transceiver address), which contains and makes accessible one or more child/sensors, each identified by the node id and child id. The controller receives reports from child/sensors and sends commands to child/sensors. The node is just part of the transportation system.

The controller is configured to know what each child/sensor consists of..

In that context, your first question could be split:
Is it logical to have two values of the same type reported for a child/sensor?
Is it logical to have more than one child/sensor supported by a single transceiver+uC (ie: node)?

@hek made the point a while back that a child/sensor can be thought of as a semantic container for one (or sometimes more) closely related values. A node can contain one or more of those.

Yes, there would be an alternative, in skipping the child/sensor container layer. One approach would be to support only a single sensor per node, but there is clearly interest in multi-sensor nodes. Another would be to dissolve the child layer of this heirarchy use rules something like:

• a node can report on (or accept control for) more than one sensor, with more than one value
• no multi-sensor node can have more than one sensor which reports the same kind of value
• the controller will be responsible for logically grouping related sensor/control values within a node

I think one could get by with this for now, but I'm not sure it's a clean conceptual architecture to continue to expand for the future. And I wonder if it puts too much onto the controller - not a big deal if there was only one controller type supported (eg: Vera only), but with an expanding list of controllers, any functionality (like conceptual grouping of the values within a node into "sensor" containers) which is moved to the controller, requires that functionality to be programmed into every controller's architecture in whatever languages that requires - a lot of parallel maintenance to keep in sync for any changes. The less functionality is moved into the controllers, the easier it will be to maintain.

• How does the user know which temperature/LED/blind is what?

Exactly. Is this configured per value, or per child/sensor? (If no sensor ever had more than one value, this would be moot).

Currently the "what" is configured per child/sensor.

Suppose next year one wanted to measure the effectiveness of a swamp cooler with intake temperature and output temperature. Does that need to require two nodes, because the architecture was designed such that no node can report two temperatures?

If the goal is just to create an architecture which can get by for most systems today, there's not need for a big rethinking or the labor of rewriting the code - version 1.4 already does that. I think the point of this thread is to define a flexible conceptual base that's likely to be more cleanly extensible for a few years of growth. I think that hek's child/sensor concept is somewhat helpful in creating that flexibility. Perhaps there's an even better approach to serve that function, but just omitting the child/sensor container level doesn't currently strike me as such.

[RJ_Make]

I would agree, I rely heavily on the child/sensor container level. I typically build nodes that poll multiple types of sensors.

[marceltrapman]

Look at how other systems do this (for those that do not agree with the current setup).

Multi sensors are used all over the place.

The very well known temp/humid/lux/motion sensors but also tilt sensors combined with a tamper alarm.
And when it comes down to using more of the same sensors on one node: i am going to build a sensor that has two IR sensors for 2 directions.

[daulagari]

@Zeph , @ServiceXp, @marceltrapman Thanks for the feedback.

I would agree, I rely heavily on the child/sensor container level. I typically build nodes that poll multiple types of sensors.

and

The very well known temp/humid/lux/motion sensors but also tilt sensors combined with a tamper alarm.

Both are still perfectly possible.

And when it comes down to using more of the same sensors on one node: i am going to build a sensor that has two IR sensors for 2 directions.

That is what I think is the "less than 1% of the implemented sensors"

Note that the current proposal:

NodeId | ChildSensorId | ChildSensorType | ValueType | Payload

has one more field than the 1.4 protocol.

My proposal is basically to merge the ChildSensorId and ChildSensorType into one, being the ChildSensorType, so:

NodeId | ChildSensorType | ValueType | Payload

If the goal is just to create an architecture which can get by for most systems today, there's not need for a big rethinking or the labor of rewriting the code - version 1.4 already does that. I think the point of this thread is to define a flexible conceptual base that's likely to be more cleanly extensible for a few years of growth.

See above, the proposal adds an additional field and yes, that gives additional flexibility but with the price of one additional byte (or at least field) for every message and additional implementation problems for Controllers on how to represent this.

My question still is how a controller should represent this. Most controllers work with devices that can have one or multiple values. Device can map to node but how to map the remaining fields?

[Zeph]

@daulagari said:

My proposal is basically to merge the ChildSensorId and ChildSensorType into one, being the ChildSensorType, so:

OK. That touches on one of my critiques of the direction it's going. (Maybe "critique" would sound stronger than I intend, I'm not disparaging the current direction, just suggesting another one).

I think that ChildSensorID is an appropriate concept for identification and addressing (ie: included in packets), while "ChildSensorType" is static metadata which should be collected during configuration but does not need to be passed in every packet. So I would simplify in the other direction - get rid of ChildSensorType in the packet addressing heirarchy, rather than getting rid of ChildSensorID.

This idea is not currently popular, in large part I think because the metadata describing a given ChildSensor would have to be collected/stored by each Controller plugin, as the currently favored architecture is to have a very thin gateway node which just passes packets between controller plugins and the OTA packets. My alternative vision would be to have a thicker gateway which could, for example, store the metadata for each ChildSensor (eg: ChildSensorType, but expandable) gathered from nodes during config, and provide it to the controller plugins.

With this approach there is no barrier to having multiple children of the same type (but with different ChildIDs) because the concept of addressing & identity have not been conflated with the ideas of types and metadata. In my preferred alternatives, the normal OTA packet for routine reports and commands omits the semantics and includes only addressing/identity, while the initial configuration passes semantics, types and names that are relatively static and do not change per-packet. Moving ChildSensorType from per-packet addressing to static metadata would be one of the implications of this design principle. (If you apply this to values, then we also include a numeric ValueIndex in the packet and move the ValueType info to the metadata.)

(I went even further and suggested omitting the ChildID from the regular OTA packet by having only NodeID | ValueIndex for addressing, and treating ChildID as static metadata which can be associated with each ValueIndex. Thus for example we could say that ValueIndex 3 contains a temperature value associated with child 2, and ValueIndex 4 contains a humidty value also associated with child 2. The central system (thick gateway or controller plugin) would use saved metadata to know which values come from or go to a given child ID container, but the OTA packet doesn't have to include that every time. Not surprisingly, this was a bit too radical a revamping and has not gained traction. And that's OK too, I'm only one voice with only part of the picture here).

Just to be complete, EncodingFormat (eg: 1, 2, or 4 byte integer or unsigned interger, or 4 byte float, or float encoded a text string etc) could be either config time metadata if it's always the same, or could be included in the OTA packet if it changes dynamically. I can see a case for either design.

I'm mentioning this because you seem interested in looking into deeper changes to the architecture, but I'm not pushing these concepts here now - my ideas on such things have been heard and a few have influenced the direction hek is going, but the overall design concept has not been persuasive enough. That's fair.

The system here is pretty good already, so it's not critical to make such changes. So rather the push MySensor too much, I've been (intermittantly) working on my own protocol ideas (at the spec level) which incorporates these and other concepts, and which could co-exist with MySensors. However my tentative design has additional design goals including high bandwidth distribution of pixel control data and medium high bandwidth collection of rapidly changing "sensor" data (eg: joysticks used for low latency pattern. It would be able to handle the MySensors use case of low bandwidth sensor networking as a less challenging subset. However my design is an off-and-on project on paper so far, while MySensors is a working system, so I give a lot of weight to the latter.

[hek]

Yes, we should set the goals high. But we need a realistic plan using the limited development resources we actually have where people should be able to pitch in when they have some time over between family and work.

I've seen so many Open Source (and workplace) projects die due to grand unrealistic plans where there just isn't enough resources or time to realize them. The key here is doing things in small graspable steps.

So we have a few things to consider and prioritize.

1. Hardware requirements on the gateway. Today we have chosen to keep this as simple as possible but this might change in the future and nothing stops anyone from creating a gateway directly on RPI or ATMega2560 which could handle more advance MQTT stuff ot handle the id-handout.
2. Support for multiple radios. Ongoing work started by @ToSa with a hanging PR (were have you been hanging lately BTW )
3. Keep plugin developers happy with a low threshold to create a plugin. We should keep ourselves from making too large serial protocol changes (too often) or we'll lose many supported controllers along the way.
4. Support more sensor types with examples. This thread is the playground for creating a more complete set of supported sensors. In the spirit of this I've spend the last weeks trying to create more fun examples. The sensor-examples (and controller support) is the key factor to grow a larger audience (and hopefully more developers).
5. Support for thing like encryption and simple OTA firmware updates.

The more people helping out the faster we'll get there.

[Zeph]

@hek said:

Keep plugin developers happy with a low threshold to create a plugin. We should keep ourselves from making too large serial protocol changes (too often) or we'll lose many supported controllers along the way.

I do not mean the following to insist on any changes - I accept that there is a balance to be had between changing things for the future and making today's changes as small as possible, and that somebody has to make that judgement call.

But just to explain where I have been coming from and why I have bothered explaining the alternative approach (clean separation between dynamic addressing/identity and static metadata) - it's because of exactly your point quoted above. If we run into limitations in the architecture a year from now and wish we'd thought it out better, it's going to be a lot harder to fix things because there will be a lot more plugins already written by then. It could be better to bite off that transition now when many HA system interfaces to MySensors are not yet written, and the ones that do exist are still in early development with the original developers highly involved. A year or two from now, more interfaces will exist, and some of the early implementers may have moved on to other foci.

The concrete of decisions will be setting into the stone of legacy code over the next year. Anything substantial that isn't in the next revision of the protocol - being discussed in this thread - is likely to never happen (lots of small tweaks may happen of course). The cusp of going from 95% Vera users with almost all architecture-tied code from a single developer, to a broad array of HA system interfaces written by many groups, is now, and it's going to change the degree of flexibility. This may be the last chance to do a deep rethinking.

The other point was that I was suggesting how to simplify the task of plugin writers, by have a single codebase within the MySensors universe (eg: Arduino C++) for as many of the tasks as possible. That lowers the bar some, and reduces the amount of parallel development of doing the same thing in multiple plugins (like assigning addresses).

OK, I wanted to be understood that my purpose was very much in line with the quoted desiderata, not opposed to it - but I'll be cooperating with the direction @hek chooses.

[ntruchsess]

@Zeph said:

I think that ChildSensorID is an appropriate concept for identification and addressing (ie: included in packets), while "ChildSensorType" is static metadata which should be collected during configuration but does not need to be passed in every packet. So I would simplify in the other direction - get rid of ChildSensorType in the packet addressing heirarchy, rather than getting rid of ChildSensorID.

100% agree. Sending ChildSensorType is just redundant static metadata. Just having implement a MySensor-conroller for fhem I noticed there's no way to request Presentation-messages from Sensors so a Controller after Sensor startup, so the controller has no way to verify it's persistent 'static' metadata without a user going to restart the sensors. This actually is a point that needs to improove.
But the argument a controller would have to be 'thin' (in terms of not being statefull in respect to sensor details) is a no-arg. You cannot do automation without knowing details about the controlled items.

• Norbert

[hek]

Yep ChildSensorType is redundant.
If thin things like the Arduino based MQTT gateway would manage without it we could probably remove it from the normal payload.

Request for metadata is a good idea. Would require sensor to keep this available at all time in memory or eeprom when controller requests it.

[hek]

Update first post to reflect ChildSensorType changes. Should probably rename this thread also.

[ntruchsess]

@hek said:

Request for metadata is a good idea. Would require sensor to keep this available at all time in memory.

This is allready in memory, just requires a move of send-presentation-messages code into own method that is called from begin and in response to 'request-metadata'.

[ntruchsess]

@hek said:

If thin things like the Arduino based MQTT gateway would manage without it we could probably remove it from the normal payload.

As MQTT-protocoll is not stateless (it does Quality-of-service with message-storage and redelivery - though the existing gateway code doesn't support this yet) the MQTT-gateway cannot be a thin thing anyway. I wouldn't mind if it doesn't run on Uno due to memory-constraints, there's the Mega2560 or even the DUE which seems to be affordable as you only need a single gateway per install.

[hek]

First post updated.

[tbowmo]

what about a "RS232" sensor?

The TV in my living room has a serial port to control it (Can do everything I can with the remote) and also request status from the TV (eg. powerstate, channel being watched etc). Could be cool to include that into some kind of domotica solution

/ Thomas

[hek]

This would probably qualify for a special command-type as it's not a "normal" sensor.

Today I've squeezed in scene controller and ir-commands among the "sensors" but they should probably also have their own commands. Thanks for the input!

[Anticimex]

Perhaps rs232 and ir can be fused together as one type? Both are by definition serial protocols. Each with a protocol and symbol rate specific to the appliance at hand.

[hek]

I know too little about rs332 and IR to make a suitable command structure at the moment.

[Anticimex]

Think "uart"

[tbowmo]

Think that, for UART it should be enough to send characters (or strings) to the sensor, and send back replies to the master (If anything comes in at the serial port)

I asume that if one needs to use serial to communicate with some remote device, one will setup baudrate etc. for that particular device in the arduino sketch. So no need to use initialisation commands for the serial stuff.

For IR it's a bit more complicated, because there is no single protocol definition like serial. for IR it could be RC5, RC6, something else, it can be different carrier frequences etc. for each brand. And it can be send to multiple devices, using the same IR "blaster"

/ Thomas

[Anticimex]

@Thomas-Bowman-Mørch well, yes. But I am more thinking a "protocol". No matter what serial interface the sensor has externally, any serial interface implement some form of protocol for managing handshaking, flow control, etc. And considering the limitation in the data packets in RF24, we cannot expect to be able to implement a "stream".
By having a protocol, the sensor node can ensure it collects enough data to be able to satisfy it's external device, i.e. caching data packets for making a larger continuous serial transmission. Or reading a larger continuous transmission and caching it from the device, before starting to transmitting it to the controller. If transmission starts too early, data sent from the device, could be lost (depending on HW/SW solution) as serial protocols do not have to implement flow control (and rarely does).
Both IR and UART share the same caveats so in that respect, I think it makes good sense to figure out a command type that can satisfy both types as the needs are similar. And then it potentially can also be applied to other serial protocols (CAN for instance).

[lunarok]

Hi,

Not very usefull post for you here, but I want to say I'm very pleased to see this discussion and the news I can read here or what appears in dev on the github.

About the presentation of sub-type accepted. After getting an almost complete controller for Jeedom, we have difficulties to "predict" things to automate and make it very simple for user to use the plugin. Like for exemple, create commands. For relays and dimmer it was easy, but after when we look for servo or some extra type it's impossible to know for now what we can create.
So reading, the sensor will tell it in presentation it's just perfect.

Same for the presentation during node of lib used, that's just better than during the sensors, make sense.

I read also on the protocol page, that there is a way for a node to request data from gateway/controller. Can this point be clarified, maybe it needs also the sensor/node to say what it can request ? That's also for automation of autolink. For us (Jeedom plugin) and I think for other, if we know the sensor is able to request some data, we can automaticly create the information item and the user will have only to map it to the equipment it wants to point. I don't know if this is still included/planned as I'm aware many of the "smart" part must be on the controller side, but with beautiful things like Scene Controller and LCD display, I can imagine there is a need ?

I read also the OTA process will change a lot. So, is it better waiting 2.0 to include it on our side ? I mean, for the controller part, will this change a lot of things ?

And last point, there is actually a internal message type Reboot. This one is indicated only working for OTA bootloader. Is that true ?
If so, is it possible in 2.0 to include a way to request a complete presentation of an already started sensor. (that's why I ask for the Reboot message, simpler to request a reboot so the node will present itself at the same type)
Sometimes it's easier when migrating for exemple or reinstall, and we know the ID of sensor to ask it for presentating itself than going there to unplug it.

That's my thoughs after getting a little more in deep of protocol for the controler/plugin

Cédric

[aliasdoc]

Hi, I think we could integrate an uuid system in protocol, like this one https://code.google.com/p/tinkerit/wiki/TrueRandom .

[hek]

@lunarok said:

If so, is it possible in 2.0 to include a way to request a complete presentation of an already started sensor. (that's why I ask for the Reboot message, simpler to request a reboot so the node will present itself at the same type)
Sometimes it's easier when migrating for exemple or reinstall, and we know the ID of sensor to ask it for presentating itself than going there to unplug it.

True, reboot would send presentation again. But another option is to save the presentations in the api. Still a bit unclear which option I will choose.

@aliasdoc
What do you mean the random number should be used for?

[nneeoo]

Hi. I would like my nodes measuring various values (temperature, humidity, ...) to send those values at specific times. I don't want to include the RTC on the node, so would like the node to request a sleep period from the controller. I can see there is no such possibility with the version 1.4 of the protocol. What do you think about including such thing?

[hek]

You can use VAR1-5 for this in 1.4.

2.0 will have both VAR (used mostly for pushing custom data) and CONFIG (for custom node configuration).

[BulldogLowell]

Right now you can use the Time.h library and you don't need an RTC.

Just sync the time with MySensors call...

gw.requestTime(receiveTime);

with the function:

void receiveTime(unsigned long controllerTime) 
{
  Serial.print("Time value received: ");
  Serial.println(controllerTime);
  RTC.set(controllerTime);
}

[nneeoo]

@hek said:

You can use VAR1-5 for this in 1.4.

2.0 will have both VAR (used mostly for pushing custom data) and CONFIG (for custom node configuration).

Yeah, good point. I somehow missed these general purpose variables. It might be good to add a dedicated type for it if you'll find it useful. If not I can live with the VAR for sure ;). Thanks.

[nneeoo]

@BulldogLowell said:

Right now you can use the Time.h library and you don't need an RTC.

Just sync the time with MySensors call...

gw.requestTime(receiveTime);

with the function:

void receiveTime(unsigned long controllerTime) 
{
  Serial.print("Time value received: ");
  Serial.println(controllerTime);
  RTC.set(controllerTime);
}

Thanks for pointing out this library. However I don't want to track the time on the node, but would like just request the time for which the node should sleep from the controller. I think it makes more sense to have the logic in controller.

[BulldogLowell]

@nneeoo

Oh, you are requesting the duration of the sleep... not the moment at which you wish it to sleep.

I thought you were wanting it to transmit data at certain times...

[nneeoo]

@BulldogLowell said:

@nneeoo

Oh, you are requesting the duration of the sleep... not the moment at which you wish it to sleep.

I thought you were wanting it to transmit data at certain times...

Well, yes. I want the node to transmit at certain time. Since the controller should know when that time is, it has to be able to calculate for how long should the node sleep, when the node asks the controller. I hope you get my point.

[BulldogLowell]

@nneeoo

yes, so you can use Henrik's advice and store a variable which is the time interval in seconds, minutes or even milliseconds.

pass that variable into a function that uses that time as the interval to transmit your data from the sensor.

something like this... pseudo code

int intervalFromController; //in minutes

void setup()
{

}

void loop()
{
  gw.begin(getVariables, RADIO_ID, false);
  //your other stuff
  updateSensorData(intervalFromController);
}

void getVariables(const MyMessage &message) //
{
  if (message.sensor == CHILD_ID)
  {
    if (message.type == V_VAR1)
    {
      // get your V_VAR1
    }
  }
}
updateSensorData(int interval)
{
  static unsigned long lastTransmitTime;
  if (millis() - lastTransmitTime >= interval *60000UL)
  {
    //collect and transmit sensor data
   //request sensor interval
  lastTransmitTime += interval * 60000UL;
  }  
}

[ToSa]

@lunarok said:

And last point, there is actually a internal message type Reboot. This one is indicated only working for OTA bootloader. Is that true ?

Reset and OTA bootloader are independent - but you need to set the fuse bits of the MCU to enable the watchdog for the software reset to work. This is done automatically when flashing the OTA bootloader but you can do that as well in a separate step keeping the existing bootloader (e.g. optiboot).

[aliasdoc]

@hek said:

What do you mean the random number should be used for?

I am currently trying to develop a bridge between mysensors and home kit managed with RPi and for this purpose I need an uuid different of the id used in the my sensors library and I thought the above library is a good start for the generation of a uuid (almost unique).

[Zeph]

If there is interest in using somewhat of a standard rather than rolling our own (and I think a case could be made either way), it would be worth reviewing the IoTDB effort which is also trying to describe the semantics of sensor/actuator control. For example:

https://iotdb.org/pub/iot-attribute.html

[hek]

@Zeph said:

https://iotdb.org/pub/iot-attribute.html

Thanks interesting reading.

[daulagari]

Yes, indeed interesting reading also in the light of controllers.

[lunarok]

Hi,

Please to see this post. As our plugin for Jeedom is complet now, I have some feedback to share.
First, is there a date and status of version 2 for now ?

Here my thoughs :

• about units, I agree with the fact that it's lightier to have the Arduino not taking care of it, but instead use some conversion on the controller side

• about value types, it will be better to have really type and not a notion of unit sometime here too like for power get only consuption and not kwh, really going all the way and give the unit control to the controller

• about value types and sensor types, after writing the plugin, still weird for me to have a sensor presentation giving some clue on what type it is, but need to wait for some value to be sure. I mean power sensor for exemple that can get some different type of values, this is not really clean I think. Or the weather sensor, in one case it will send a value type that is numeric, the other one is text (forecast)

• will it not be better to have some mechanism when it's an actuator to deliver the type of message it is accepting ? for exemple a heater, there is the off/on but also a mode type. How is it possible to know that the sensor is accepting all or only on/off ?

So, for me, something like after a presentation being able on the controller to say what type of data and what commands we can say, will be the best. And important too, what data it can be requesting. I mean, now with the scene_controller to extend functions, we need to get data from the automation software too. Imagine to display some informations from outside the mySensors network.
Maybe by completing the sensor_type and merging with value_types, removing the unit notion inside the value. And so during presentation we know all about the data it can send. And in second step during presentation, declare the actuator commands.

One other point important and asked by some users, is it possible to integrate some security in the protocol ? As soon you talk about actuator accepting command from anyone, that's unsecure. Key given during inclusion by gateway and use for encode the data ?

Thanks for reading

Cédric

[Anticimex]

About security, I'm looking into that. I am working on a concept involving key exchange and signing using an external circuit.

[lunarok]

Yes for security can be also a first presentation requiring a signing. Like what Puppet is doing, agents first present themself to the master, master is waiting for someone to accept a request, and after it gives the certificate to the client and then the client tell him verything about itself (facts in this view can be compared to the sketch and sensors presentation)

But I don't know if it's easy with arduino this type of exchange.
Other way can be a simple key we put in sketchs, and using the same on every of our home, and use it in the lib to encode the data.

[Anticimex]

I am working on a security protocol, and have posted my take on things in the security thread. I will weigh in on this thread once I have verified my design, but so far, only two new message types should be needed, one to request security capabilities and once for capabilities and a nonce. I do not think software based security is a suitable solution due to memory constraints so I am going for a hardware based solution with a pre-shared key. I will publish more concrete examples once I have verified them.

[lunarok]

Hi,

Is there any move on the V2 ? Can we see the actual status and what it is going to look like ?
Going to a V2 for the Jeedom controller, will like to see it coming with mySensors v2 changes.
Precisly will like to have a status about :

• reboot of node without needing a special bootloader (inside the lib will be the best)
• sending libversion from presentation including for gateway (this is include for nodes, I don't know for gateway)
• possible of sending V_type used during presentation, no need to wait to send data
• possible of sending a short desc name for each sensors created (can be helpfull when you create many sensors with same type and the difference is not only the order)
• possible to send the power source of node, I don't know inside battery or else. This will be very helpful with battery/plug or battery/solar source for exemple. So we can know how much the battery is full but also if the sensor is actually on battery.
• getConfig to be used for any paramteres instead of units, like this the node can request parameters from the conrtoler (think about a global sketch for a switch that can talk to a relay node, by the controller side you set which node it's controling)

Can we have a wiki page maybe with an actual status of where is the V2 ? And is there a dev version we can try ?

[hek]

@lunarok said:

Hi,

Is there any move on the V2 ? Can we see the actual status and what it is going to look like ?

There has been some delays due to workload. But i hope to finish it eventually.

• reboot of node without needing a special bootloader (inside the lib will be the best)

Not sure it is possible to reboot an arduino without watchdog enabled (anyone knows any tricks?).

• sending libversion from presentation including for gateway (this is include for nodes, I don't know for gateway)

Yep, possible to get version from gateway today.

• possible of sending V_type used during presentation, no need to wait to send data

There will be a different setup.

• possible of sending a short desc name for each sensors created (can be helpfull when you create many sensors with same type and the difference is not only the order)

Good idea! I'll take that into consideration.

• possible to send the power source of node, I don't know inside battery or else. This will be very helpful with battery/plug or battery/solar source for exemple. So we can know how much the battery is full but also if the sensor is actually on battery.

Thats also a good idea. I'll have to think about how it could be incorporated.

• getConfig to be used for any paramteres instead of units, like this the node can request parameters from the conrtoler (think about a global sketch for a switch that can talk to a relay node, by the controller side you set which node it's controling)

Config/settings will be a bit different as well.

Can we have a wiki page maybe with an actual status of where is the V2 ? And is there a dev version we can try ?

Hmm.. my github account (henrikekblad) will contain the source until I feel it's worth trying out by the community. But there is a part of it I'm waiting for a c++ guru at work helping out with (advanced c++ templating which is a bit over my level of c++ knowledge). Those will hopefully make callbacks (incoming messages) usage awesome.

The plan is the gateway will be able to accept/push json objects instead of the semicolon separated parameters, Much depends on how memory demanding it becomes.

[axillent]

can we avoid float?
instead of float a multiplied integer can be used
for example 34.6 C temperature can be presented as 346 (no dot)

a single float operation increase HEX size significantly

@hek what about idea we discussed to have a network with multiple gateways?
what about adding support for the 2.0 for the nodes be able to have a different BASE address per each gateway?
this way the future network will allows to have 255 sensors per EACH gateway, not per NETWORK

[hek]

@axillent

Yes, configurable base address would be handy if we choose to create an ESP-gateway. It is already supported today but you have to hard code it into MyConfig.h. Would be neat to have this configurable over WiFi.

Using OTA transmitted floats is optional. I haven't looked so deeply on how/if this will be supported yet in the upcoming version.

[tekka]

Any plans for encryption?

[Anticimex]

This post is deleted!