Generalizing MySensors

Zeph

As best I understand it, MySensors was originally developed around the conceptual framing of devices as used by Vera/MiCasaVerde/MIOS; has been remapped by some adapters to work with other systems; and is at the stage where the semantic gaps are becoming obvious enough to raise the question of rethinking the internal conceptual framework to be more general and less Vera specific.

Let's start from the bottom of the hierarchy - what we could call "variables" - each containing a value which may vary over time.

**Format: **
The values could be

floating point or integer (of various sizes), or
chosen from an enumerated set of possible values (a common case being binary options like on/off or true/false or armed/unarmed etc)
a text string

What it measures (or controls):
In the case of a numeric value, this could be temperature or relative humidity or wind speed or water depth or brightness or rate of flow of electricity or water or accumulated flow of electricity or water or loudness of sound, or angle of a servo, etc. Sometimes there may be no conventional dimension or interpretation.

Units and scaling
There may be a translation to a standard unit of measure like degrees C or F, or percent humidity, or feet or meters, or mph or kph or m/s, or lux, or db or angular degrees or radians, etc. Sometimes we may know the dimension being measured (brightness) but have no translation to standard units (not calibrated in lux, just an arbitrary and perhaps non-linear value) so this is optional. And sometimes there may be a scaling factor needed to get to some standard unit, from the value reported.

The dimension and the units are often conflated, but of course a value in degrees C could be a temperature reading or a dewpoint, two different dimensions which use the same units.

That's the basic level - a set of distributed variables measured from the real world (or computed from measurements), or controlling some action in the real world. Then we built on top of that.

Grouping of related measurements
We might wish to associate several variables with a given device or sensor. For example a weather station might have temp and humidity and wind speed and wind direction, etc. Or a flow meter might have immediate flow (gph or kw) as well as accumulated flow (gals or kwh).

This is where some things go a little wonky, because there isn't always one obvious way to logically group variables, and there may be little guidance as to the best practicies OUTSIDE OF matching the way a particular controller has chosen to conceptually organize things.

From the top down, MySensors groups sensactuator variables by the wireless node which reports or uses them. This has a lot of practical sense to it, but in some sense it's a transport or addressing concept rather than a sensor concept. Suppose you had some one wire sensors strung around the property. As you add, remove and move wireless nodes around over time, you might find it more convenient to connect the (relatively long) wiring to different wireless nodes for reporting to the controller. In this case, the actual one-wire sensor would not change in any logical way (it's still measuring the same thing in the same location) but the values are reported through a different path, a different wireless node. So grouping of values by wireless node should be done lightly - it's not a primary logical grouping based on what is being measured or controlled, but more of "the current address and transport path".

The child id is also part of the addressing and transport path.

From the bottom up, MySensors group sensactuator variables into Sensors, each of which has a sensor type. This is one of the confusing areas, since the types of sensors seems to overlap semantically with the types of variables which might be associated with it.

In trying to sort through this, I think the Sensor type mostly represents "which user interface object in the Vera system should be used to display and accept user input" for a group of variables". So if you have one or more related sensor variables which can logically be associated with a given Vera UI object, you may use that sensor type. Or you can do some more work and add a new sensor type to Vera, and use that.

The problem is - another controller may not choose the same set of UI objects. I think that "sensor type" may be better conceptualized not at the MySensors generalized node level, but as a mapping of the generalized sensor nodes into the Vera system.

In this approach (which is being brainstormed, and is not fully figured out), the sensor node would not need to understand Sensor Type (tho it would understand Variable formats, dimensions, scaling, units and names). Instead the gateway to Vera would map variables into Vera related groupings, and the gateway to another controller would map the variables into the groupings which makes sense to that system.

Suppose that one controller had a user interface element which displayed weather related information graphically, always using metric values. Another controller has a "temp and humidity" user interface object but no weather station, tho it does have a separate flow concept to which wind speed could be mapped; the second one allows the user to select metric or imperial units. The first controller might display wind direction in degrees clockwise from true north, the second might use NNW type notation.

The same physical sensors and associated variables (at the node level) might be grouped into "sensor UI objects" differently in the two controllers. But the nodes shouldn't care whether wind speed is shown inside the same box with temperature (a "sensor" UI element) or in separate boxes, or even omitted from a controller which doesn't have a target for some measurement. They just report values for variables, and the gateways alone handle grouping variables into user interface elements as used by the controller they support.

In this model, more than one controller could be used with the same sensor network, so long as they don't issue conflicting commands. The sensors might always report temperature in C, and each controller + gateway would translate if it so needs. The sensors might report wind direciton in degrees and each controller + gateway would translate to compass points if so needed. If one of the controllers doesn't have a way to display "microcontroller temperature" or "H Bridge back-emf" then it can omit that variable, without precluding another controller from accepting such variables.

Knowledge about the controller and its concepts would be removed from the wireless node code, and concentrated in the gateway to that controller. Nodes just report (or accept) one or multiple sensacturator variables. The gateway could package up groups of sensactuator variables as needed for the specific type of controller it serves. Suppose a given controller had a "temp & humidity" object but no weather station object. As described above, you could map wind speed into some other controller element for now. But then when a weather station "sensor type" with additional variables is defined next year, the gateway could regroup sensor variables into that package, without any changes in the wireless nodes.

Splitting the gateway and the hub

The concept of a gateway may need to be separated from the concept of the wireless hub.

The wireless hub understands how to communicate with the wireless nodes, up through the protocol level of transferring sensactuator variable values back and forth. Like the nodes, it doesn't need to know what controller(s) are attached or their concepts, it just moves values. It could use a different radio chip, or RS-422.

The gateway function translates from this general format to a specific controller's needs. (Cloud storage could be another type of "controller", by the way). The gateway function doesn't have to worry about ack and routing tables for the nRF24L01+.

There is a general API between the MySensors wireless hub code and the MySensors gateways. They can run on the same processor (or not, if there is a transport option for the API).

When you add a new controller, you write a new gateway, but can use the same wireless hub code.

And a message manager like a MQTT broker could be inserted between the wireless hub and one or more gateways.

That's the architecture which appeals to me. If it has enough appeal to others, including and especially hek, then maybe that will be version 1.5 or 2.0 of MySensors.

This is where I was generally heading before discovering MySensors (which is WAY ahead of me in terms of practical implementation).

marceltrapman

@Zeph Wow that is some writing. To be honest I have not read it all because it is way too much. Apologies for that.

Having said that, do I understand correct that you are trying to start a discussion about splitting up the gateway from the wireless protocol/hardware?

Damme

Good writing! me myself is writing a MQTT broker instead of serialgateway / ethernet gateway, Right now I'm waiting for new hardware so the development has stopped.

One thing I really want is better ack support. I.E. I send on to a relay, and I request ack. I really think this should be the gateway to really see that the message got away and recieved, it not - retry (or report fail back).

I also think we need to start thinging about MySensors 2.0 and it should not be VERA based protocol.

Yveaux

@Damme totally agree! Guaranteed message delivery is key for any decent (sensor) network and if this requires storing of messages on the node, relay or gateway then so be it!
I would like to see it built in the my sensors library and if not, then I'll start working on a derived class/fork in the very near future...

marceltrapman

I don't have the knowledge that some of you guys have so here are my 2 questions:

Guaranteed message delivery: isn't that what @hek is working towards?

How is this done with, for example, the z-wave protocol?

From my understanding z-wave (or possibly all Domotica stuff) needs a controller to send, receive and store messages and the gateway takes care of delivering them.

When we have a Vera (for z-wave) or any other hardware controller the two are integrated but, with more generic hardware like a Raspi or (in my case) a Mac, the software acts as a controller.

But maybe I am wrong...

hek

@Zeph
I really like that we have these kinds of discussion for the upcoming versions!

Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).

Couldn't agree more on unit and scaling. Just sending standard SI units from sensors would be preferred. But this requires more logic on the controller plugin side. Which is fine for me now when things has settled a bit. But when the project was young I wanted to keep change-frequency and coding/logic on the plugin side to a minimum as I hate the freakin LUA coding (with minimum debug possibilities on Vera). Here the work would be to document the proper unit for supported variables.

I don't fully understand the wireless hub/gateway. Maybe just a wording thing.
For me a driver concept would fit better where we create transport drivers for NRF24L01/Serial/Other Radio Model/Ethernet.
So the current serial-gateway would be a simple combination of Serial + NRF24L01 drivers.

@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users (it is already a pretty steep learning curve for the electronics/programmer beginners). We also have to think about how to attract more controller plugin developers. This requires stable versions to keep things in sync on controller side.

There will never be any guaranteed delivery in any network as nodes can be offline for long periods of time. The big question is how much we can afford to buffer (time and memory vise). And where buffering should take place. Here I've seen different opinions. Some like the MySensor library to be slim with acking (taking care of resend for specially important messages themselves) while others like to have a more advanced resend functionality build into the library. We could probably combine these requirements by being smart (driver architecture or by providing plugins like you proposed).
@marceltrapman
In the z-wave world the controller is buffering and resending commands a few times with pretty long delays in between (1-5 seconds) over the z-wave network before giving up. After this the node is considered DEAD by some controllers (which means no more resends occur until it gets replies) others just show a problem status for the device. The z-wave nodes tries to find other routes to the destination but will eventually give up. As I understand the buffering in nodes is minimal (if any at all?) and the resend logic is handled on a higher level.

A general comment: As as always we do appreciate contribution of both code and ideas to the project! The problem for me right now is that there is a lot of opinions/ideas but not much help implementing them. This needs to change if we want this project to evolve faster.

So.... Start these kinds of discussions. Make concept classes. Discuss. Draw sketches of your ideas and post them here. Code. Create pull requests on github. Document things you find unclear. All is welcome!

My god... this was a long answer.

Yveaux

@hek

@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users

maybe my comment sounded quite like 'let's start a revolution' but that never was my intention.
Creating a subclass or forking would just be too experiment with buffering and 'guaranteed delivery'.
Of course you can never guarantee delivery, and for most sensor /actuators missing a delivery deadline of a few seconds counts as a loss too (imagine your light turning on 30seconds after you flip the switch).
I can however not accept a solution which just discards the message after a few failed attempts, so I'm happy where the act mechanism is going, but I'm still investigating the (im) possibilities..

gbfromhb

This post is deleted!

John

To give my two cents, I think the current setup is quite ok. On the hardware side you want as less as possible traveled through the air with as less power consumption as possible.

controller: Software/End user.
gateway and further: mysensors.

To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup. I can only agree with the fact that the api should care less about it's controller. And the sensors (library) should care more about data types. For example one byte extra explaining the payload's datatype so the controller can create the correct datatype for itself or further dispatching of the data. Also the sensors should, in my opinion also never report metric OR imperial, it always should be one standardized result over all the sensors and let the "controller" correct these.

Also about the grouping, and discard if i did not understand correctly, should never be done by the hardware supplier, in this case mysensors. It should always happen at controller level.

I do agree with the guaranteed message delivery, but to have it done within the hardware brings some complications, buffer size not to big to have enough room for incoming messages, etc... I do believe that re-sending queues should be handled on the controller side and not on the gateway side. This gives more flexibility in choices if particular messages can eventually be discarded or not. If not QOS should be implemented on the gateway.

Also the gateway should not care about any transport protocols used out there. It should only care about the transport layer of the data with the payload to be as small as possible to keep the gateway doing as less ass possible and keep precious time free to do what it needs to do, be an hardware gateway. Nothing more and nothing less. Using one standardized protocol over tcp and serial looks like enough for me personally and let my controller (which packs more power/cycles to do other stuff then the gateway) convert these to the required protocol.

But these are just my two cents, let the gateway be nothing more then a gateway and any form of data handling/grouping/etc.. should be done by the controller. A controller should meet gateway requirements, not vice verse.

Zeph

@hek said:

Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).

One of the fundamental ideas in the original post is separating the controller's UI based semantic groupings from the sensactuator nodes, so that the code running in nodes is not Vera specific (nor specific to any other controller).. That way the sensor network could be controller agnostic, and translation to controller concepts would be done by the gateway to a given controller type (not yet sure if this is a plugin or driver, in your terms).

What I hear from you is that there may also be some inherent semantic grouping of related sensactuator variables - not dependent on the controller and it's UI objects, but deriving from the physical world being sensed and controlled by the nodes. So if a sensor node were to have two DHT-22's (which can be separated by a fairly long cable), there would still be value in associating temp1 & hum1 as separate from temp2 & hum2.

I have two tentative responses to that.

This inherent grouping might be handled by keeping the MySensors concept of a child id within a node.. Child 1 might have Temp and Humidity values, and child 2 might also have temp and Humidity values. That could still make sense. The thing to avoid is implementing controller-specific Sensor type at the node level.
The gateway to a given controller could be flexibly able to group arbitrary sensactuator variables into a given controller-specific object (with a controller-specific type). It might be common to associate sensactuator variables from the same child of the same node with a given controller object, but that need not be a limitation.

One concept that might be useful is giving each child a network unique text name. The gateway configuration might refer to that child by name; this would be translated by the wireless hub into a node id and child id as transport concepts.

Zeph

@John said:

To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup0.

I did not fully understand your post yet, John. I think you were largely agreeing, tho at times you appeared to be thinking you were ratifying how it works now. Let me try to clarify. I'm not radically changing your diagram, but am suggesting splitting the "gateway" functionality and partitioning some of the knowledge between the ends of that spectrum.

Let's start with some roles.

Controller - one of several possible programs which were written independently of MySensors and which are intended to control or at least record values from a set of sensors and actuators. See the thread on Open Source Home Automation for a list of examples. Vera is one of these. Recording in the cloud is also in this category.

Sensor/Actuator node - an addressable microcontroller which interfaces with the physical world using sensors and actuators, sends measurements towards the controllers and accepts commands from the controllers (both of these can be done through transferring new values for defined variables). In some networks, like MySensors, these nodes may also relay messages between nodes that cannot communicate well directly.

What I'm advocating is that sensor/actuator node should not need to know or care about the conceptualizations unique to each of the varied controllers. Their code should reflect the nature of the sensors and actuators, and the nature of the network (eg: a nRF24L01+ based wireless networking stack) they use, period.

Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.

Between the sensor/actuator node and the controller there are two distinct functions - the network hub and the adapter for a given controller.

The network hub is part of the sensor network architecture; it handles the centralized end of keeping the sensor network functioning. It understands basically the same concepts that a sensor node understands - packets and acks and routing and addressing and transporting various types of variables via packets.

The adapter function (which I earlier called a "gateway", I'm withdrawing that term as ambiguous) translates between the concepts and communication protocols of a given controller, and the more general concepts of the sensor network. It has to understand the API provided by the network hub software, and it has to understand the controller it was written for. An adapter written for Vera would understand the Vera defined Sensor Types, and would group sensor level variables into Vera type Sensors as needed.

In the current MySensors implementation, the adapter function may reside partly in Lua code & configuration within Vera, and partly in a "gateway node" which also contains the network hub functionality.

I think the current architecture is actually very cool. All designs are tradeoffs, optimizing one aspect at the expense of another. The current design was optimized for one point in the evolution of the MySensors network, when it was well adapted to Vera. However MySensors is at the point where it's poised to support multiple controllers, and the architecture or design may need to evolve to handle that well. Hence this discussion.

I don't aim to change version 1.4, or to distract hek from the wonderful work being done on the existing network. I hope to constructively influence future versions, and I hope that having a vision for where future versions may go will be useful at times when choosing alternatives in intermediate releases.

John

@Zeph

Ok, now I understand it more by giving the gateway the term adapter, and if I understand correct, the hub would be the gateway as i had in my previous post. I think we do agree that the current architecture. But i have to disagree a little bit about the libraries design, if this is the design you meant.

All though it looks like it has been adapted for the vera, in which i partly agree, the current protocol used is, in what i believe, portable enough to be used in all Home Automation projects which support custom adapters. I just can not agree in mysensors should support multiple kind of controllers. Multiple kind of controllers should support mysensors.

If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?

Is it a piece of hardware between the hub and controller?
Is it a piece of software which needs to be coexisting on the network hub?
A combination of software in the controller and software in the hub?
Is it a piece of software build into the controller?

My personal view with the above are the next:

This would create extra costs, so i do not think this is what it is, unless someone specific chooses for this solution.
I think this also would not be the case because this causes or a lot of forks or mysensors can less focus on the core and must provide libraries for multiple controllers.
This would be completely not maintainable, this means coders would have to write code and for their controller software and for mysensors hardware. I personally think this is a maintenance nightmare, when the controller or hub evolves they must combine different versions of different kind of software which should work together.
This would in my opinion be the best case. And i will explain why i believe this:

MySensors should focus only on their sensor networking. Supplying vera plugins is off course a choice, but the fault lies in the fact that the MySensors API has a bit extra for the vera. This should not have been done, but i understand why it is. But, to explain:
The controller should include the adapter, a controller should be adaptable. A base conception of a controller then should be:

End user interface <-> Internal data handling (Controller) <-> hardware protocol to Internal data protocol and vice verse (Adapter) <-> MySensors Hub <-> etc...

If it ain't possible to install/expand/etc... a controller to include an adapter to translate the hardware protocol used into the internal data handling protocol, it is a bad controller design case and exposes the controllers shortcomings.

I understand the need for implementing multiple controllers via shared or other types of adapter software with MySensors, and i think it is mainly because a lot of home automation software development is not evolving or adapting that fast enough to support an evolving hardware protocol or the software has shortcomings. And because they are not that fast or flexible, the hardware supplier should be responsible to make sure they are. And i believe this should not be the case.

My view on this is because my software works with a "drivers" system of which is capable of supporting multiple versions of a specific piece of hardware versions used protocols. This gives the end user a choice of what they would like to use. In my personal view i can not agree with having a shared adapter setup with combined controller and MySensors implementation if not implemented on the controller. But again, I do understand the need for these kind of adapters, but, should MySensors be responsible for this?

Well ,that was a big one... I hope I made it more clear, and yes, i did agree with you for the biggest part.

hek

@Zeph said:

Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.

For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.
With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.

One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.

char variables[] = {V_FORECAST, V_TEMP, V_PRESURE}
gw.present(<id>, S_BAROMETER , variables);

One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.

John

@hek

char variables[] = {V_FORECAST, V_TEMP, V_PRESURE}
gw.present(<id>, S_BAROMETER , variables);   
One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.

It should be possible because the presentation is in order in the sensors because of the char array:

This can possibly create V_FORECAST, V_TEMP, V_PRESURE, V_TEMP

Where the controller map these to for example: V_FORECAST_1, V_TEMP_2 V_PRESURE_3, V_TEMP_4.

The only but of course is that there then should be an extra parameter used which defines which position in this array has been send...

[Edit]It is on the sensor, and not if[/Edit]

Zeph

@hek said:

For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.

So the sensor type can serve as a hint as to some of the variables a child might have. But of course there are so many possible conbinations that it's easy for a "off the shelf" sensor type to have variables that a real sensactuator doesn't support or vice versa. And two different controllers may have different sets of variables they associate with a Barometer type (or whatever). So...

With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.

One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.

Yes, that is exactly the type of reframing I am supporting. It's open ended, with no assumptions about how any given controller will operate. If your actuator has Back EMF sending, then the sensactuator node can report it, whether or not the controller you are currently using anticipated that. If you have a weather station with "rain intensity" (vibration sensor measuring the intensity of impact on a metal plate), you can create such a variable and report it.

The functional component I am not calling the "adapter" needs to map the actual sensors and actuator variables in the sensor network, to concepts in the home automation controller it was written to adapt. In the adapter's configuration, one might indicate that (node 5)(child 2)(temp) and (node 5)(child 2)(humid) variables will be packaged as part of a SimpleEnvironmentMonitor object "LivingRoomMonitor" as it's "t1" and "h1" members. (I'm making up a controller example here). For another controller, the adapter might group sensactuator variables differently - including ignoring some or making use of some the first controller ignored.

That's a big part of what the adapter does - it maps and groups (and scales/translates if necessary) the MySensors sensactuator variables into the objects and terminology of a given controller. Since there may be a lot of commonality in this function across controllers, it's possible that the adapters for different controllers would share code and configuration files.

I think we are converging.

Zeph

@John said:

If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?

Is it a piece of hardware between the hub and controller?

Is it a piece of software which needs to be coexisting on the network hub?

A combination of software in the controller and software in the hub?

Is it a piece of software build into the controller?

The adapter in my description so far is a functional component that handles certain aspects of the data flow. It could potentially be implemented in any of those forms, while providing that translating and mapping functionality.

As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.

John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.

On the other hand, somebody who is fluent in that controller's language and system has to implement the whole thing. As mentioned in my previous response to hek, there may be a lot of common functionality across adapters (mapping MySensors sensactuator variables to controller, along with creating/editing/storing that configuration) - which may need to keep being re-implemented in a new environment/language/etc. Not insurmountable, but a factor.

Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.

But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.

However, none of that should require any change to the actual sensactuator nodes.

John

@Zeph said:

As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.

Next to the fact i'm also new to MySensors, and currently building native support for it in my system, the above is also my observation. The only thing i'm seeing when i look at the gateway/master node, is that i'm only seeing MySensors specific protocol with only some functionality added to make it more compatible with the Vera. If the choice was made to support an other home automation system i think we would still see all the same MySensors specific protocol code and output, but then with some additions made to it for that other system. The current protocol setup can cause the confusion that the current protocol is Vera specific, while i think it's not, it has some added functionality. But if this ain't the case @hek should stand me correct of course.

John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.

Well, i personally think that is the only solution. It is impossible to generate common objects. JSON objects are not common, well the objects are, but the data they represent are not, same for xml, mqtt etc... And that's of course where the problem resides. And something you mention further on, if i understood correctly.

Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.

And now is the question, how would the adapter publish the simple syntax file to the controller. Or even better, how would the controller interpret the output from the adapter? Then we still need a plugin/driver system to understand this data presentation.

But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.

I completely understand, i hope, what you are trying to say. But then there are some personal believes that oppose. I think every component, software or hardware, should serve it's purpose. A hub should only be a hub. At the moment you put software extending logic in the hardware controller, for example the hub. The hub ain't a single purpose anymore and becomes part of the software ecosystem. Who will then be responsible for this ecosystem mix? There now already is a dependency called Vera, if Vera changes, the gateway needs to change, even when there is no MySensors protocol change needed at all. By having dependencies with your product you need to spend time on projects that are not part of the core which is MySensors hardware.

The questions then also are: If there is a part put on the Arduino then

Who is responsible for the part of the code for controller compatibility?
Why limiting the code for controller compatibility to "most" of them?
If you use a controller which is not one of the "most". Who then creates the compatibility, other code on the Arduino or Controller?
By mentioning "most" you are already ruling out the others.

The current protocol setup does allow it to be used with i think every controller, as long there is an adapter on the controller. And this makes me understand the need for standardization or more compatibility on for example the hub. But, this would be fighting a losing battle.

However, none of that should require any change to the actual sensactuator nodes.

Of course

I personally believe the current protocol implementation is good, not too much data over the air and serial/network controller. The less data there is the more time can be spend to sensor sleeping and have the controller do other stuff... The lesser the better.

I do absolutely agree there should be a syntax file. But i think this should be limited to MySensors protocol structure and data presentation. We do now know what is what if we take a look at the source. It would be nice of course if we would not have to.

hek

@John said:

The current protocol setup can cause the confusion that the current protocol is Vera specific, while i think it's not, it has some added functionality. But if this ain't the case @hek should stand me correct of course.

The serial over-the-air and serial protocols has been designed to be vera independent. The variables/sensor types is of course inspired by how vera has arranged their devices. But as I said earlier. If a controller gets meta information on how sensors is arranged it can choose to use this information or not.

We can conclude the following from this discussion (as I see it):

We need some sort of grouping mechanism. Today this is called "sensor" which is identified by a sensorId (0-254). All sensor-groups has a sensor-type which is a hint to controller. I can't right now see any alternative to the this (more that providing even more information like I proposed below). An adapter or controller-plugin needs this to prepare/create/decide how and what type of data it should present and group together. This is of course optional. The controller can just throw this away and wait for the first sensordata to arrive.
We should move unit information/conversion back to the controller. All sensor data is send as SI-units. This will make the sketches even simpler but it puts some extra workload on the plugin developers. I'll discuss this further with @marceltrapman and @john who is the first to port/create 1.4 plugins for their respective favorite controllers.

@Zeph I need to see a more concrete use case for moving more logic to to the Arduino-gw-side in a adapter setup. The Arduino MCU is very limited and we need to keep the code really tight there.
I suggest you help out building a plugin for your favorite controller (OpenHAB and Domoticz has a lot of requests). You will more easily see missing details in the meta information you receive from the sensor network and will hopefully come back with even more helpful insights.

Zeph

I think this is discussion is converging, as I've said. Good points made and taken all around.

There seems to be agreement that the sensactuator nodes should be controller agnostic.

The largest area where they are not is in choosing a "sensor type" which is Vera specific. That's what started this rethinking for me - in trying to annotate the MySensor libary's variable and sensor type enums (in the lost forum posts), I wasn't sure how to describe what a "sensor type" really represented. I tried to find some independent meaning to the Sensor type that I could document, but failed. If it says the sensor type is heater, what does that imply? Low/Medium/High or a thermostat or what? It is controlling a heater or just monitoring one, or both? The conclusion was that type means nothing more or less than "you may be able to fit this 'sensor' into what Vera calls a heater", meaning there may be a user interface object in Vera into whose displays and user controls you can reasonably well map the variables and behavior of this sensor, and the icons displayed for that object in Vera may more or less fit the functionality of this sensor. There was no Vera-independent semantics to the sensor-type.

However, hek suggests that this is just a hint which can be ignored, so one could say that it's at worst a small and meaningless overhead for some controllers, and at best makes Vera easier to interface with. Not a show stopper to keep it around for now, since it can be ignored. It might be more clear to call this data "Vera-hint" than "sensor type", tho. Other controllers could try to discern something from the Vera-hint, or ignore it.

That said, moving the metric/imperial conversion to a centalized location seems like a good move, and hek's suggestion of letting senseactuator nodes specify what variables a given child device ('sensor') implements, seems like an important move in the right direction. I think that any controller could benefit from getting a list of what variables a "sensor" (an addressable node+child-id grouping of variables) generates and accepts, even Vera. So this goes a long way towards addressing my concerns, whether or not the sensactuator nodes continue to present a vera-hint as well.

John feels strongly that all the adapter functionality should be implemented entirely within each home automation controller. I've never argued against that option, only suggested keeping open some other possibilities as well. However, if no need to implement some or all of the adaptation outside the controller ever comes up, that would of course be fine with me.

hek makes the point that the gateway (wireless hub) does not have room or performance to implement very much. Given that I've never suggested that the gateway be forced to implement any adapter functionality, we are not exactly disagreeing (our views are not incompatible). However that brings up a more general point, in that I would not design the architecture at any deep level to assume that the gateway will will continue to be limited to an ATMega328p @16MHz and 2K of RAM.

I DO see reason to design for the Arduino Pro Micro for the sensactuator nodes: There can be many of them, and the APM is very cheap, it's small, and it can be very low power to accommodate battery power (and it's Arduino compatible so it can inherit a rich set of device libraries). And it's adequate for most sensactuator tasks; in most cases more power would be wasted anyway. I don't see much on the near term horizon that fill the sensactuator node niche much better than the APM class device for MySensors. (Only possible exception: if the nRF51822 became more widely used in the form factors and price point the APM+nrf24L01+ occupy).

But the gateway needs to be on full time and wired to the controller, so it can be wall powered and even a bit physically larger if need be, and since there is only one of them it could be a bit more expensive. Arduino Mega clones are $15 - no faster but 4x the RAM and more peripherals. A teensy 3.1 has 64K of RAM and runs 32 bit ARM code using the Arduino IDE and library interfaces for $17. Arduino Due clones with an ARM and 96 KB can be had for $20. ARM based STM32F103 with 20K RAM, 72 MHz processor and lots of IO are available on eBay for under $7. Or you can get a ST32F411 Nucleo for about $10 with floating point ARM (they can use the Arduino-like mBed system). The RPi is $35, the BBB is $55 now, and there are several other GHz class ARM devices under $80 like the cubieboards (plus shipping; the other prices include shipping BTW). Given all that, I'm going to be very surprised if two years from now most of the MySensors wireless gateways/hubs are still running ATMega328p processors.

So while I'm not proposing now to force all gateways to be larger than an ATMega328p, I'm also not assuming in designing an architecture that the gateway hardware can never do more than that chip could do - at least in some configurations. If we had to spend an extra $5-15 dollars for a much more capable (single) gateway in order to interface with a given controller we wanted to run, that would not be a limiting factor for most of us.

AND - John may be right - maybe there will never come a case where it makes any sense to implement the adaptation functions anywhere except on the controller (eg: inside a Vera or a PC running home automation software, etc). Time will tell. If I differ some in the degree of flexibility I suggest, I am nevertheless respecting the other views which have been well stated here, and have no certainty that such flexibility will be needed. We seem to be agreeing on the fundamentals.

Rasmus Eneman

I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.

So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.

So, I would prefer if SensorType described what a human would classify a sensor node as.

I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display
setting and nothing else.

hek

@Rasmus-Eneman said:

S_TEMP would be S_THERMOMETER

Sounds very reasonable.

Damme

I just wanted to make a pointer to http://forum.mysensors.org/topic/260/mysensors-mqtt-gateway , Might be a combined discussion about the future of the protocol, I am thinking about the address-layout. Using Vera-style or just ignore some parts of the address.

I have not been reading everything in this discussion, I haven't had time yet. But if you come to any conclusion it would be goot to write them down!

John

@Zeph

The largest area where they are not is in choosing a "sensor type" which is Vera specific.......There was no Vera-independent semantics to the sensor-type.......Other controllers could try to discern something from the Vera-hint, or ignore it.

I totally agree with you about the sensor type limitations. It would certainly be nice if the Vera would provide an interface where sensor types can be extended. But i'm not well known enough with the Vera to know if this is the case or not. All though, as you mention, that the sensor type is just a hint which could be ignored, it does give some extra information about the sensor produced variable type values. And where the total sensor types table is concerned it mixes up sensor type semantics. As i read it now in your post it then appears this mix up is because of the Vera and I better understand the protocol part of the discussion. Thanks for this clarification (I shouldn't be doing multiple things at the same time).

As example and thoughts spin off.
In your example you are referring to heater, where as i understand an heater could be a collection of possible sensors and actuators, like temperature and switches. In this example it would be best to have an N_HEATER (heater node) sensor node with a S_TEMP sensor and for example a S_SWITCH actuator. from which produces V_TEMP and V_ON/V_OFF. But even then taken this as an example you still do not know if it is only monitoring, etc... which is only something you can assume depending on the reported sensor types. And it would even be better if the user can define the node him/her self of course.

John feels strongly that all the adapter functionality should be implemented entirely within each home automation controller. I've never argued against that option, only suggested keeping open some other possibilities as well. However, if no need to implement some or all of the adaptation outside the controller ever comes up, that would of course be fine with me.

Yes i do, and certainly with the open sourced, or plugin capable automation systems. But, i am open for possibilities where there are protocol based implementations possible on the hardware instead of controller protocol type. And yes this would acquire more capable hub hardware. It would be nice if there where more protocol based solutions like offering an Vera hub, MQTT hub, JSON Hub.

Zeph

@Rasmus-Eneman said:

I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.

No disagreement there, the question is where and how to specify that.

So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.

Looking through the sensor types with a perspective like that is what started me on this thread (initially in the messages that got lost).

As you say, a measurement of "temperature" could mean many things. It could be a simple themometer, or it could be a dewpoint, or it could be a daily max from a weather station or it could be the intake or outlet temp from a heater or cooler device, or it could be a themostat setting - all in degrees C.

One approach would be to define a new "sensor type" (as you use it, I'm going to use a different terminology for this concept below) for every type of temperature variable; basically invert the current heirarchy and define many sensors types for every variable type. Put another way, the variable could be the addressable unit, and one of its attributes could be a sensor type from the list defined for that type of variable. So we find that there is a V_TEMPERATURE variable, and we ask it "what kind of V_TEMPERATURE are you?" and it responds that its a "S_TEMP_HIGHLIMIT" or a "S_TEMP_FREEAIR" or some other S_TEMP_* type of temperature.

HOWEVER, hek has made the point that there is utility to defining the a container for multiple related variables (what's called a sensor in the current system, and addressed with a node id and child id). The current concept of the "Sensor Type" is associated with that group or container. So you might have a S_WEATHERSTATION which (potentially) contains V_TEMPERATURE and V_RELHUMIDITY and V_WINDSPEED etc variables. (Multiple variable types for a given sensor type, in this way of looking at it).

The problem with the container is that real world sensors vary a lot - one weather station will include a UV sensor and another won't, or one will measure peak wind speeds and another will not, or one includes both inside and outside temps, etc. So it's hard to pre-specify what all the variables a S_WEATHERSTATION may have, and if you do, most actual ones will be pretty sparsely populated. And then we look at a home automation controller - which of those measurements will their nearest analog to the S_WEATHERSTATION implement? Probably not exactly the same as all other controllers.

I think the closest we've come to sorting through this inherent complexity is something like this:

There are Containers which are individually addressable and may report more than one measurement (or accept more than one setting). All of the measurements within a given container are physically or logically related to each other. (This is the thing addressable with a node address and child id).
Each measurement (or setting) associated with a container has a V_* variable type which says what is measured, and if possible specifies the standard units or enumerated states.
Perhaps each measurement can also have a "semantic" attribute, chosen from a list specific to the variable type (eg: V_TEMPERATURE can have VS_TEMP_FREEAIR or VS_TEMP_DEWPOIT etc). (VS=Variable Subtype or Variable Semantics). This is where I'd put what you were calling a sensor type - as a variable subtype attribute.
The central authority can ask each node about the containers (children) it has. It can ask each child what variables it reports or accepts. It can find the V_* variable type and the VS_* subtypes for each variable.
The node can also report a hint from the S_ list for each container (child). This hint does not limit what variables the container may report or accept, and it may or may not match well with the way any given controller lumps and splits things - but the hint can be ignored.
There has to be some configuration for each controller, which maps variables do defined by the nodes, into variable and objects which make sense for the given controller. (I was calling this the adapter function). This mapping is not required to make a one to one association between addressable variable containers ("children" or "sensors") and the objects in that controller's universe.

An example. Suppose a node reports outdoor temp & humidity and wind speed and direction and UV index, all associated with a single measurement location and reported as variables in the same child container. And there's a controller which doesn't have a "weather station" object to group all of those variables. But it does have an "environment monitor" object which combines temp, humidity and noise level. The adapter function could map the node's temp and humidity to the temp and humidity of the controller "environment monitor" object (leaving noise level unreported), and find some other controller object(s) for wind speed and UV index - or ignore them. (There could be another parallel "controller" like a cloud recording system which does make use of them even if the given home automation controller does not).

So, I would prefer if SensorType described what a human would classify a sensor node as.

If we associate SensorType with a multi-variable container, then there are many mix-and-match varieties of these, as perceived by a human, not all of which are going to be implemented in any given controller. If we mean "variable subtype" as described above, then no problem.

I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display setting and nothing else.

I think we are coming to some agreement about that part (single set of standard units for reporting, rather than pushing the user display preferences out to each node). But even that's not quite as simple as saying "SI" and walking away. I've seen wind speed reported in m/s and kph for example. And pressure has multiple possible units. As do light or sound. Even temp has C and K. I've seen fluid flow in cubic meters/hour or liters/min. So I think we should explicitly specify what units each measurement should use (if it can) and that we should look primarily to SI when possible in defining those expectations.

And we need to accommodate uncalibrated reports - like brightness reported based on an uncalibrated and non-linear sensor which nevertheless has higher (or lower) values for brighter light. We may not be able to translate to lux, but the HA controller could nevertheless use empirically set thresholds for triggers or conditions.

Zeph

@John said:

In your example you are referring to heater, where as i understand an heater could be a collection of possible sensors and actuators, like temperature and switches. In this example it would be best to have an N_HEATER (heater node) sensor node with a S_TEMP sensor and for example a S_SWITCH actuator. from which produces V_TEMP and V_ON/V_OFF. But even then taken this as an example you still do not know if it is only monitoring, etc... which is only something you can assume depending on the reported sensor types. And it would even be better if the user can define the node him/her self of course.

I see that the difficulty of organizing this complexity and diversity is becoming clear to us all.

I would disagree about N_HEATER, ie: a semantic type for a node. I see the node as a transport mechanism, which may contain one or more "variable containers" (or "sensors" where a sensor can have multiple measurements which are closely associated). So for example a node type I'm thinking to put in many rooms would have:

DHT-11 or DHT-22 for temp and humidity (former so cheap all would have at least that)
LDR or digital sensor for brightness (likewise)
PIR motion detector
IR remote control decoder
IR LED
visible LED for signalling
possibly a puzzer or small speaker
battery voltage sensor (divider)
I'm considering sonar range finder in some cases
possibly a door switch

This is one node (I hope). And the physical sensors would be lumped and split into child containers for variables (logical "sensors"). It does not matter to the central controller whether temp and humidity came from one physical sensor or two - the reported variables will be aggregated into several logical groupings of related measurements.

I an live with each container (child "sensor") having a "sensor type" as a semantic hint. But I don't see the node itself as having a semantic type - it's function is addressing and transport. I don't see that there's even any "hint" to be associated with the above node which would be of use to the controller.

Going a step further, if the timing gets too messy, or I want motion sensing and IR decoding in different locations in the room, I might split up these devices into two nodes in some rooms, rather than one. As I see it, that should make no difference to the controller software per se. In the "adapter" function, some local configuration (addressing) would change, but the same set of variables would be reported.

(And here I thought this thread was about cooked)

Damme

I'll try to write my opions about this, First of : Do Not over complicate stuff!

Today we have presentation values and sensor_type values, those two tables are not compatable (Dont understand why). I think this should be only ONE list off things, used both for presentation and value type. We just put more stuff in it..

IF someone needs more complicated stuff it is easy to do in the node code; Just send an empty request to the node and you can get whatever you want in return, I use this code here to get latest relay state from the node if the controller forgets what the last one was. (makes more sense also because saving the value at the controller might be far off the value from the node)

void incomingMessage(const MyMessage &msg) {
	if (msg.type==V_LIGHT) {  // We dont care about other stuff than V_LIGHT , this could be anything!
		if (strlen(msg.getString())==0) { // is there any payload?
			// NO = Get the latest state, construct a message and send.
			gw.send(message.setSensor(msg.sensor).setType(V_LIGHT).set(digitalRead(msg.sensor-1+RELAY_1)?RELAY_ON:RELAY_OFF)); 
		} else { 
			// Here we could check is it 0 / 1 ? or something else, could be 'STOP_60' and we could program a timer to shut down the relay in 60 minutes.
			digitalWrite(msg.sensor-1+RELAY_1, msg.getBool()?RELAY_ON:RELAY_OFF);
			gw.saveState(msg.sensor, msg.getBool());
		}
	} 
}

The current form of address in my mqtt broker is 'prefix/nodeid/sendorid/v_type' -> 'MyMQTT/074/002/Light'

In this example client can send 0/1 or nothing to the broker, if nothing it will publish the latest state back to client. but payload can be anything.

marceltrapman

@hek said:

We should move unit information/conversion back to the controller. All sensor data is send as SI-units. This will make the sketches even simpler but it puts some extra workload on the plugin developers. I'll discuss this further with @marceltrapman and @john who is the first to port/create 1.4 plugins for their respective favorite controllers.

So, what you mean is that we send information based on the metric system (if any) and receive information about temperature etc. SI based with the actual conversion solely happening in the controller?

Basically any system is fine with me as long as I know about it

John

@hek said:

We should move unit information/conversion back to the controller. All sensor data is send as SI-units. This will make the sketches even simpler but it puts some extra workload on the plugin developers. I'll discuss this further with @marceltrapman and @john who is the first to port/create 1.4 plugins for their respective favorite controllers.

I could not agree more. And, to be honest, it would even require less work then keeping track per sensor node. Having a sensor network wide setting is much more comfortable. And nonetheless this solution means less cycles spend on sensor node side saving power.

@zeph said:

I would disagree about N_HEATER

Me too, it was solely an example which could be used as a group definition to make it clear what this sensor would do in remark of sensor-node typing and should be done controller side. In the following up example you provide you mention a nice multi-purpose sensor node which clearly produces a "User knows best" case which is absolutely true in case of the flexibility provided with MySensors and is the perfect example why there should be done as much as possible done on the controller and let the hardware do nothing more then deliver only the most basic types and their data.

I an live with each container (child "sensor") having a "sensor type" as a semantic hint. But I don't see the node itself as having a semantic type - it's function is addressing and transport. I don't see that there's even any "hint" to be associated with the above node which would be of use to the controller.

I totally agree until the "hint" part. I personally find it very useful if the table was more organized in the type specifics. I'm seeing device types and sensor/actuators in the same table, which agreeable can cause confusion. I personally would leave out the device types which should be user definable.

In my personal opinion the most optimal solution with the tables would be still provide two tables, but one with sensor/actuator types, and one with datatypes and let the user decide in the controller what a sensor-node device represents. A door/window sensor could be replaced with a reed switch/hall effect to detect open/close, which could be integrated in one sensor type defining an open/close (0/1,true/false) state. This would i think be the simplest solution.
And if the controller is smart enough, group specific sensor types together.

@hek said:

char variables[] = {V_FORECAST, V_TEMP, V_PRESURE, V_TEMP}
gw.present(<id>, S_BAROMETER , variables);

I think this could be possible and helps generalizing MySensors by decoupling address and type. But this would require some extra work on the controller side. Is it possible that the position of the sensor presentation in the array can be the sensor's address? It would also require some extra work for the programmer of the sensor node where he/she needs to remember which sensor is at which position.
The controller would see based on the char array's character position that V_FORECAST is address 0, V_TEMP = 1, V_PRESSURE = 2, and V_TEMP = 3. Because the sensor developer knows which V_TEMP sensor is for what. The first is for outside, and the second is for inside, he/she can define this on the controller.
In this way you would decouple the the sensor type and address definition. Or is this making it too complex?

Zeph

We continue to mostly agree.

Warning: this turned out to be very long indeed, and I need to move on - and I didn't want to lose this thinking and describing, so here it is. This should probably be condensed into a document.

@John said:

@hek said:
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE, V_TEMP}
gw.present(<id>, S_BAROMETER , variables);
I think this could be possible and helps generalizing MySensors by decoupling address and type. But this would require some extra work on the controller side. Is it possible that the position of the sensor presentation in the array can be the sensor's address?

We are talking about multiple levels of hierarchy here. In the current system, there is:

node (no semantic typing)
"sensor" or child (with a S_* type)
variable (with a V_* type)

The array hek was using as an example (of possible future approaches) was for variables within a "sensor", not sensors within a node.

I find the above heirarchy still useful, but I think we are asking for trouble when we label the middle level "sensor". Structurally it looks like a group of one or more logically or physically related variables which are reported or assigned (where the reports measure the real world and the assignments may change the real world).

What do we want to call a "sensor" (apart from the current usage)? Is a DHT-11 one sensor or two? Is a light sensor which also can also report internal temperature two sensors? If we go with the "those are multiple sensors" then is there any distinction between a "sensor" and a "variable" or is there a one to one mapping?

For purposes of this discussion, I'm going to go with "SensorGroup" for that middle level of the hierarchy (currently "sensor"). Where I say "SensorGroup" you can read "the addressable container formerly known as a 'sensor' "

The MySensors network can have one or more end nodes
A node can have one or more SensorGroups, each with a child id by which it can be addressed
*. A SensorGroup can have one or more Variables.

The purpose of a SensorGroup is to provide an addressable container for related measurements and controls (represented as variables). So far, not very different from MySensors 1.4 and earlier, except the name.

Where we begin to evolve the design is in hek's suggestion that a SensorGroup could enumerate the variables it contains. Currently, the central control doesn't really know what variable types any given SensorGroup might report. It might take a wild guess based on the presentation type of the SensorGroup (the current S_* enums), but we've had lots of examples of how flawed any such assumptions might be. So heck proposed (very much in line with the rethinking going on here) that a SensorGroup might report what Variables it contains.

@hek said:

char variables[] = {V_FORECAST, V_TEMP, V_PRESURE, V_TEMP}
gw.present(<id>, S_BAROMETER , variables);

This is very interesting because there are two V_TEMP values in the list. The current addressing hierarchy is node/SensorGroup/variable_type (where SensorGroup is represented by a numeric child id). A given SensorGroup should have only one V_TEMP variable, because we otherwise have no way to distinguish "which V_TEMP".

So the above implies another numeric address component - the variable id (or index into that array). You could now identify the first V_TEMP as (node address)/(child id)/1 and the second as (node address)/(child id)/3 (if they are zero based like the array (note 1)).

So: (node address)/(child id)/(variable idx) is the implied new addressing scheme. And passing the array in hek's example does two things: it reports how many variables the given SensorGroup has, and it tells the type of each. So the controller (or some adapter function) knows that variable 2 in that SensorGroup is type V_PRESSURE.

This is very much along the lines that I have been thinking as well (for a different wireless network design).

If we follow this concept, the world actually starts to sort out. The "atoms" in this chemistry are Variables - they can be reported as measurements or set to cause an action or configure a device. These can be combined into logical SensorGroups. And SensorGroups are combined into Nodes for addressing.

Now we can talk about the flavor of those atoms, the Variables. That is, a given Variable can have various attributes. We have one major attribute already - the V_* types. Those give the variable's values some semantic interpretation - like interpreting the numeric value 45 as meaning 45 degrees Celsius. Each V_* type would imply a standard unit to be used (if meaningful).

Another obvious attribute of a given Variable is the format or structure: eg: one/two/four byte signed/unsigned int, or floating point, or string, or boolean. For some purposes like weather in degrees C, a one byte signed int could be sufficient for V_TEMP, but for a reflow oven we might need two bytes or a float, while still being semantically V_TEMP.

I'm going to add another attribute: UNCALIBRATED. This is optional, but if attached to a Variable we know that it is NOT reporting in the standard units. (If it's not attached, then standard units should be used for reporting). For example an uncalibrated LDR might report values which are not in lux and probably very non-linear. However, the value should follow the same slope as the standard units - if a higher number means "brighter" in the standard units, then the uncalibrated values should also rise for brighter. So we can set thresholds for "brighter than" even if we don't have a lux value.

Earlier examples have pointed out that degrees C could be used for a thermostat or a temperature sensor, and for free air temperature or contact-with-device temperature or even dewpoint. Even air temp could be "intake temp" or "exhaust temp". Some of these concepts could be reported as additional attributes associated with a given Variable.

Another attribute would be "name". Gee - if the variable has a meaningful name, at least a human might more easily set up the mappings and configuration files. Knowing that node 7, child 1, variable 2 is a V_TEMP and is named "IntakeTempB" might be useful. A simple extension would be to allow a short "Description" attribute as well.

One of the ongoing areas for clarifying is when to define additional attributes (classes and the members of each), and when to just let the name.and/or description serve. This is a classic question of controlled vocabularies (pre-defined enums) vs free tagging (just a string or set of strings). If done right, a mixture of these can work well.

At this point one might feel this is getting way too complicated, but really I think it's coming into focus. You can always omit most of these attributes (except structure and V_* type perhaps), and you will default to something like the current situation - except that it has the hooks to do more in a consistent way when you are ready.

I will note that these attributes can be stored in PROGMEM and need not be very large, really, for most nodes. The predefined attributes might be as simple as two bytes (vocabulary and term) to allow up to 256 vocabs with up to 256 terms each). If you run out of PROGMEM, omit the less valuable attributes.

AND - note that these attributes can also be associated with a given Variable in the configuration of the adapter function. For example, there could be a simple text file which links node 7/child 3/var 2 to a set of attributes (encoded as a text string with simple delims). The advantage of putting them into the node is that they will aid in self-documentation and discovery and reduce the effort and errors of configuration, but conceptually the attributes (except structure) are not needed until they get to the controller adapter, so they could be reported over the air when a node joins (or is queried), or they could be pulled from a file or other memory in the central system.

(Yes the controller adapter can be implemented as a plugin of the controller).

What I picture is that in the sensactuator node code, one would typically define the types of variables within each child SensorGroup statically, and that would include storing the attributes in PROGMEM as a byte array.

A node would only automatically report the minimum when joining the network. The hub could query it to find out:

 Node name (optional)
 number of SensorGroups
 for each SensorGroup:
     sensor type hint (optional) (S_*)
     Sensor name (optional)
     number of variables
     for each variable:
          variable structure (size and signedness, or string or bool, etc)
          basic variable type (V_*)
          variable name (optional)
          optional attributes (sequence of vocab/term bytes)  (optional)

Maybe add optional "Description" to some of these levels?

It would not do this querying with a single packet! For example there might be a separate query for each variable name, and for each variable's list of optional attributes. And the hub would not be required to query for and use the optional parts. The required parts would tell it everything it knows in the current MySensors architecture.

And finally (for now), this is one of the places where I could see a smart wireless hub doing much of the legwork. Probably using a Mega or Due or Teensy 3.x (or maybe even a RPi). This device could handle querying new sensactuator nodes to create a copy of all this informaion in local RAM. Or reading some of it from config files (eg: SD card) and combining that with the bare skeleton reported over the air. In either case, the adapter to a given controller would just have all this info (the hierarchy of SensorGroups and Variable with all names and attributes. That info would be part of the API which the adapter gets to use in accessing the sensor network.

OR - a dumb wireless hub could let a smart adapter (eg: running on the home automation controller) handle all this polling for names and attributes. That has implications but could be the best solution in some cases.

(note 1: I might start numbering the variables within a sensor node with 1, reserving 0 for other purposes)

John

@Zeph

The array hek was using as an example (of possible future approaches) was for variables within a "sensor", not sensors within a node.

Yes you are right, i should have used the correct naming.

This is very interesting because there are two V_TEMP values in the list.

I should have better displayed that the second V_TEMP was an addition i made to the remark of what heck posted, my excuse for this. It also does not have to mean there should be an extra address parameter for the specific enum type. Currently, if I've understand correctly the variable type is being send, this then could be replaced with the variable char array position. This because the proposed present already has defined which variable is at what position.

Let's say it is possible to do the below

#define ID 1 /// Barometer child id
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE, V_TEMP} /// Last temp added by me (john in addition to heck's example)
gw.present(ID, S_BAROMETER , variables);

The sendVariable would then look like this:

v_temp_var_at_pos_1 = analogRead(somePin * (somecalculations)); //// Read an attached temp sensor
v_temp_var_at_pos_3 = analogRead(someOtherPin * (somecalculations)); //// Read another attached temp sensor.
gw.sendVariable(ID, 1, v_temp_var_at_pos_1);
gw.sendVariable(ID, 3, v_temp_var_at_pos_3);

Because the controller knows at what position what variable is send, there would be no need to alter the sendVariable to much. This is then completely different from the current variable update notification because it then has to be clear it is targeting a position instead of type presentation.

I hope i clarified my post.

Each V* type would imply a standard unit to be used

And each standardized unit like the V_TEMP example should also stay with a fixed datatype float will be fine because of decimals.

UNCALIBRATED

Would be more like: The reported value is not the value you would expect because when your thermometer shows 12°C i'm reporting 14°C so you need to calibrate me. And not reporting a different unit.

Damme

I am sorry to say this, but I do not really see the gain in this quite complex way?

Still I don't even see the idea of having to present a sensor type and a sensor variable.
Everyone agrees on the following:

The gateway talks to one or many nodes
The node can have one or many sensors

Today we use:

The node has a name (I_SKETCH_NAME) and a version (I_SKETCH_VERSION)
We present every sensor with a ID, an S_TYPE.
We send data with a ID and a V_TYPE.

You are talking about:

The Node present one or many S_TYPE with one or many V_TYPE. (In this setup, The node might handle the ID itself?)
The Node (still?) sends one Value with sensorID and V_TYPE. Here we are missing S_TYPE. (longer address - smaller payload) (How do we handle multiple S_TYPE? How do we separate one from another? And if we have multiple v-temp in one sensor? witch is what and who know about it? v_temp1 v_temp2?)

No, NO! This just messes things up imo. I already think its messed up already with this S_ and V_ allocations. We should stick with ONLY . One type-array. And let controller configuration handle groupings...
OR if someone wants this more complicated way of work, he can program it in the node code himself by constructing a payload with more stuff in it, example ;
Today we use ID and V_TYPE in the address field, no one forces anyone to do this. The user can program an ID and S_TYPE in the address instead and the payload could be constructed as "v_type:VALUE"...

Me myself is using the sketchname for keeping track of what type of node (sensor group in your view) it is. But this could be sensor ID0 reserved for this. Everyone is free to program whatever they want! You dont even have to use V_TYPE.. You could use this as an adress as well, so you can actually address 256x256 sensors on every node..

Today you could even present every ID with a S_TYPE (id 1+2+3+4 = S_WEATHER) and then just send data with ID and V_TYPE.. Let the controller keep track of eveything, this will pose a problem for me tough in my MQTT broker. I dont have the space of storing this type of data. So I'll be might start publishing the Presentation data, This data is just discarded of today. But because I know people using stuff are full of ideas they will work around this problem.

I dont think the progress in here is actually leading for a simple sensor network at all. I see tons of problems from a controller and gateway programmer's view. Sorry guys..

//Damme

John

@Damme

Hi,

The node might handle the ID itself?

I used the reference on the top of this page: http://www.mysensors.org/build/sensor_api

What i'm trying to put in the example is for a S_TYPE to supply multiple V_TYPE'S and the possibility to have the same V_TYPES based on fixed positions defined by the example char array. Just because the current setup is working with S_TYPES and V_TYPES.

I have not thought about multiple S_TYPE's of the same kind... good point..... But would this not just send multiple presentations like:
gw.sendSensorPresentation(ID1, S_DOOR);
gw.sendSensorPresentation(ID2, S_MOTION);

Or am i wrong... ?

I agree absolutely with what your saying and that everything that can be done with data handling like the metrics and grouping etc. should be done on the controller.

The current setup where there is just one group level is from what i think quite clever, of course on controller side it can be discarded, but it gives a nice multiple single purpose, multi sensor setup. An example with S_TYPE "Door":

With the possibility to announce in advance what kind of V_TYPES it can send, lets say: V_LOCK_STATUS at position 0 in the char array and V_OPEN_STATUS at position 1 in the char array (just made up). With the possibility to announce in this way it is possible to auto create this "door" device and will of course be a one time request.
When handling the V_LOCK_STATUS data on controller side when mysensors would send, as current is the case, the V_LOCK_STATUS it can send the array position of this V_LOCK_STATUS. Because the controller knows in advance that position 0 is V_LOCK_STATUS it can bound to that.

Yes it does complicate stuff more on controller side, in my case it would be a simple extra mapping. On the hardware side it would require the programmer to keep track of it's V_TYPE's position.

The driver i'm currently writing does not care how a "device" is defined. It can handle a sensor node as a device where the S_TYPES are groups which can contain multiple V_TYPES. Or it can create a "device" based on the S_TYPE which contains the V_TYPES. My example builds further on @hek 's example, which would give me the opportunity and i think also others to automatically create multi-V_TYPE S_TYPE groups.

Me myself is using the sketchname for keeping track of what type of node (sensor group in your view) it is. But this could be sensor ID0 reserved for this. Everyone is free to program whatever they want! You dont even have to use V_TYPE.. You could use this as an adress as well, so you can actually address 256x256 sensors on every node..

Today you could even present every ID with a S_TYPE (id 1+2+3+4 = S_WEATHER) and then just send data with ID and V_TYPE.. Let the controller keep track of eveything, this will pose a problem for me tough in my MQTT broker. I dont have the space of storing this type of data. So I'll be might start publishing the Presentation data, This data is just discarded of today. But because I know people using stuff are full of ideas they will work around this problem.

With my example I tried to keep the current multi table setup in honor and it currently is the default way to build the sensor network. I think @hek 's example does help to advance the MySensors protocol. And at the moment people are inventing their own idea's with code modifications and change the protocol data position definitions, in which of course they are free to do, they break the default compatibility.

P.S.
I also think this thread could better me renamed to "Advancing MySensors" then "Generalizing MySensors" because it is so extremely difficult or even impossible to generalize at this current moment with all those hundreds of protocols, data- and sensor definitions used. Also with MQTT there is no generalization because everyone is free to create their own hierarchy.

But, eventually, i will be fine with any implementation, as long is i know about it :).

Generalizing MySensors

Suggested Topics

About the architecture: thin vs thick gateways
Development • architecture • • Zeph

Arduino Celebrates 10 years (Malmö/Sweden)
Announcements • • hek

Saving last known good state, but not in EEPROM
Development • • OldSurferDude

[mysensors] Not a valid message: invalid literal for int() with base 10: '\x00\x000'
Development • • kiesel

Sensebender Gateway (SAMD21)
Development • • pikim

Counting Incoming and Outgoing Messages from a Gateway
Development • • Trand

13
Online

11.4k
Users

11.1k
Topics

112.7k
Posts

Generalizing MySensors

Suggested Topics

About the architecture: thin vs thick gateways Development • architecture • • Zeph

Arduino Celebrates 10 years (Malmö/Sweden) Announcements • • hek

Saving last known good state, but not in EEPROM Development • • OldSurferDude

[mysensors] Not a valid message: invalid literal for int() with base 10: '\x00\x000' Development • • kiesel

Sensebender Gateway (SAMD21) Development • • pikim

Counting Incoming and Outgoing Messages from a Gateway Development • • Trand

13Online

11.4kUsers

11.1kTopics

112.7kPosts

About the architecture: thin vs thick gateways
Development • architecture • • Zeph

Arduino Celebrates 10 years (Malmö/Sweden)
Announcements • • hek

Saving last known good state, but not in EEPROM
Development • • OldSurferDude

[mysensors] Not a valid message: invalid literal for int() with base 10: '\x00\x000'
Development • • kiesel

Sensebender Gateway (SAMD21)
Development • • pikim

Counting Incoming and Outgoing Messages from a Gateway
Development • • Trand

13
Online

11.4k
Users

11.1k
Topics

112.7k
Posts