Generalizing MySensors
-
I don't have the knowledge that some of you guys have so here are my 2 questions:
Guaranteed message delivery: isn't that what @hek is working towards?
How is this done with, for example, the z-wave protocol?
From my understanding z-wave (or possibly all Domotica stuff) needs a controller to send, receive and store messages and the gateway takes care of delivering them.
When we have a Vera (for z-wave) or any other hardware controller the two are integrated but, with more generic hardware like a Raspi or (in my case) a Mac, the software acts as a controller.
But maybe I am wrong...
-
@Zeph
I really like that we have these kinds of discussion for the upcoming versions!Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).
Couldn't agree more on unit and scaling. Just sending standard SI units from sensors would be preferred. But this requires more logic on the controller plugin side. Which is fine for me now when things has settled a bit. But when the project was young I wanted to keep change-frequency and coding/logic on the plugin side to a minimum as I hate the freakin LUA coding (with minimum debug possibilities on Vera). Here the work would be to document the proper unit for supported variables.
I don't fully understand the wireless hub/gateway. Maybe just a wording thing.
For me a driver concept would fit better where we create transport drivers for NRF24L01/Serial/Other Radio Model/Ethernet.
So the current serial-gateway would be a simple combination of Serial + NRF24L01 drivers.@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users (it is already a pretty steep learning curve for the electronics/programmer beginners). We also have to think about how to attract more controller plugin developers. This requires stable versions to keep things in sync on controller side.
There will never be any guaranteed delivery in any network as nodes can be offline for long periods of time. The big question is how much we can afford to buffer (time and memory vise). And where buffering should take place. Here I've seen different opinions. Some like the MySensor library to be slim with acking (taking care of resend for specially important messages themselves) while others like to have a more advanced resend functionality build into the library. We could probably combine these requirements by being smart (driver architecture or by providing plugins like you proposed).
@marceltrapman
In the z-wave world the controller is buffering and resending commands a few times with pretty long delays in between (1-5 seconds) over the z-wave network before giving up. After this the node is considered DEAD by some controllers (which means no more resends occur until it gets replies) others just show a problem status for the device. The z-wave nodes tries to find other routes to the destination but will eventually give up. As I understand the buffering in nodes is minimal (if any at all?) and the resend logic is handled on a higher level.A general comment: As as always we do appreciate contribution of both code and ideas to the project! The problem for me right now is that there is a lot of opinions/ideas but not much help implementing them. This needs to change if we want this project to evolve faster.
So.... Start these kinds of discussions. Make concept classes. Discuss. Draw sketches of your ideas and post them here. Code. Create pull requests on github. Document things you find unclear. All is welcome!
My god... this was a long answer.
-
@Zeph
I really like that we have these kinds of discussion for the upcoming versions!Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).
Couldn't agree more on unit and scaling. Just sending standard SI units from sensors would be preferred. But this requires more logic on the controller plugin side. Which is fine for me now when things has settled a bit. But when the project was young I wanted to keep change-frequency and coding/logic on the plugin side to a minimum as I hate the freakin LUA coding (with minimum debug possibilities on Vera). Here the work would be to document the proper unit for supported variables.
I don't fully understand the wireless hub/gateway. Maybe just a wording thing.
For me a driver concept would fit better where we create transport drivers for NRF24L01/Serial/Other Radio Model/Ethernet.
So the current serial-gateway would be a simple combination of Serial + NRF24L01 drivers.@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users (it is already a pretty steep learning curve for the electronics/programmer beginners). We also have to think about how to attract more controller plugin developers. This requires stable versions to keep things in sync on controller side.
There will never be any guaranteed delivery in any network as nodes can be offline for long periods of time. The big question is how much we can afford to buffer (time and memory vise). And where buffering should take place. Here I've seen different opinions. Some like the MySensor library to be slim with acking (taking care of resend for specially important messages themselves) while others like to have a more advanced resend functionality build into the library. We could probably combine these requirements by being smart (driver architecture or by providing plugins like you proposed).
@marceltrapman
In the z-wave world the controller is buffering and resending commands a few times with pretty long delays in between (1-5 seconds) over the z-wave network before giving up. After this the node is considered DEAD by some controllers (which means no more resends occur until it gets replies) others just show a problem status for the device. The z-wave nodes tries to find other routes to the destination but will eventually give up. As I understand the buffering in nodes is minimal (if any at all?) and the resend logic is handled on a higher level.A general comment: As as always we do appreciate contribution of both code and ideas to the project! The problem for me right now is that there is a lot of opinions/ideas but not much help implementing them. This needs to change if we want this project to evolve faster.
So.... Start these kinds of discussions. Make concept classes. Discuss. Draw sketches of your ideas and post them here. Code. Create pull requests on github. Document things you find unclear. All is welcome!
My god... this was a long answer.
@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users
maybe my comment sounded quite like 'let's start a revolution' but that never was my intention.
Creating a subclass or forking would just be too experiment with buffering and 'guaranteed delivery'.
Of course you can never guarantee delivery, and for most sensor /actuators missing a delivery deadline of a few seconds counts as a loss too (imagine your light turning on 30seconds after you flip the switch).
I can however not accept a solution which just discards the message after a few failed attempts, so I'm happy where the act mechanism is going, but I'm still investigating the (im) possibilities.. -
As best I understand it, MySensors was originally developed around the conceptual framing of devices as used by Vera/MiCasaVerde/MIOS; has been remapped by some adapters to work with other systems; and is at the stage where the semantic gaps are becoming obvious enough to raise the question of rethinking the internal conceptual framework to be more general and less Vera specific.
Let's start from the bottom of the hierarchy - what we could call "variables" - each containing a value which may vary over time.
**Format: **
The values could be- floating point or integer (of various sizes), or
- chosen from an enumerated set of possible values (a common case being binary options like on/off or true/false or armed/unarmed etc)
- a text string
What it measures (or controls):
In the case of a numeric value, this could be temperature or relative humidity or wind speed or water depth or brightness or rate of flow of electricity or water or accumulated flow of electricity or water or loudness of sound, or angle of a servo, etc. Sometimes there may be no conventional dimension or interpretation.Units and scaling
There may be a translation to a standard unit of measure like degrees C or F, or percent humidity, or feet or meters, or mph or kph or m/s, or lux, or db or angular degrees or radians, etc. Sometimes we may know the dimension being measured (brightness) but have no translation to standard units (not calibrated in lux, just an arbitrary and perhaps non-linear value) so this is optional. And sometimes there may be a scaling factor needed to get to some standard unit, from the value reported.The dimension and the units are often conflated, but of course a value in degrees C could be a temperature reading or a dewpoint, two different dimensions which use the same units.
That's the basic level - a set of distributed variables measured from the real world (or computed from measurements), or controlling some action in the real world. Then we built on top of that.
Grouping of related measurements
We might wish to associate several variables with a given device or sensor. For example a weather station might have temp and humidity and wind speed and wind direction, etc. Or a flow meter might have immediate flow (gph or kw) as well as accumulated flow (gals or kwh).This is where some things go a little wonky, because there isn't always one obvious way to logically group variables, and there may be little guidance as to the best practicies OUTSIDE OF matching the way a particular controller has chosen to conceptually organize things.
From the top down, MySensors groups sensactuator variables by the wireless node which reports or uses them. This has a lot of practical sense to it, but in some sense it's a transport or addressing concept rather than a sensor concept. Suppose you had some one wire sensors strung around the property. As you add, remove and move wireless nodes around over time, you might find it more convenient to connect the (relatively long) wiring to different wireless nodes for reporting to the controller. In this case, the actual one-wire sensor would not change in any logical way (it's still measuring the same thing in the same location) but the values are reported through a different path, a different wireless node. So grouping of values by wireless node should be done lightly - it's not a primary logical grouping based on what is being measured or controlled, but more of "the current address and transport path".
The child id is also part of the addressing and transport path.
From the bottom up, MySensors group sensactuator variables into Sensors, each of which has a sensor type. This is one of the confusing areas, since the types of sensors seems to overlap semantically with the types of variables which might be associated with it.
In trying to sort through this, I think the Sensor type mostly represents "which user interface object in the Vera system should be used to display and accept user input" for a group of variables". So if you have one or more related sensor variables which can logically be associated with a given Vera UI object, you may use that sensor type. Or you can do some more work and add a new sensor type to Vera, and use that.
The problem is - another controller may not choose the same set of UI objects. I think that "sensor type" may be better conceptualized not at the MySensors generalized node level, but as a mapping of the generalized sensor nodes into the Vera system.
In this approach (which is being brainstormed, and is not fully figured out), the sensor node would not need to understand Sensor Type (tho it would understand Variable formats, dimensions, scaling, units and names). Instead the gateway to Vera would map variables into Vera related groupings, and the gateway to another controller would map the variables into the groupings which makes sense to that system.
Suppose that one controller had a user interface element which displayed weather related information graphically, always using metric values. Another controller has a "temp and humidity" user interface object but no weather station, tho it does have a separate flow concept to which wind speed could be mapped; the second one allows the user to select metric or imperial units. The first controller might display wind direction in degrees clockwise from true north, the second might use NNW type notation.
The same physical sensors and associated variables (at the node level) might be grouped into "sensor UI objects" differently in the two controllers. But the nodes shouldn't care whether wind speed is shown inside the same box with temperature (a "sensor" UI element) or in separate boxes, or even omitted from a controller which doesn't have a target for some measurement. They just report values for variables, and the gateways alone handle grouping variables into user interface elements as used by the controller they support.
In this model, more than one controller could be used with the same sensor network, so long as they don't issue conflicting commands. The sensors might always report temperature in C, and each controller + gateway would translate if it so needs. The sensors might report wind direciton in degrees and each controller + gateway would translate to compass points if so needed. If one of the controllers doesn't have a way to display "microcontroller temperature" or "H Bridge back-emf" then it can omit that variable, without precluding another controller from accepting such variables.
Knowledge about the controller and its concepts would be removed from the wireless node code, and concentrated in the gateway to that controller. Nodes just report (or accept) one or multiple sensacturator variables. The gateway could package up groups of sensactuator variables as needed for the specific type of controller it serves. Suppose a given controller had a "temp & humidity" object but no weather station object. As described above, you could map wind speed into some other controller element for now. But then when a weather station "sensor type" with additional variables is defined next year, the gateway could regroup sensor variables into that package, without any changes in the wireless nodes.
Splitting the gateway and the hub
The concept of a gateway may need to be separated from the concept of the wireless hub.
The wireless hub understands how to communicate with the wireless nodes, up through the protocol level of transferring sensactuator variable values back and forth. Like the nodes, it doesn't need to know what controller(s) are attached or their concepts, it just moves values. It could use a different radio chip, or RS-422.
The gateway function translates from this general format to a specific controller's needs. (Cloud storage could be another type of "controller", by the way). The gateway function doesn't have to worry about ack and routing tables for the nRF24L01+.
There is a general API between the MySensors wireless hub code and the MySensors gateways. They can run on the same processor (or not, if there is a transport option for the API).
When you add a new controller, you write a new gateway, but can use the same wireless hub code.
And a message manager like a MQTT broker could be inserted between the wireless hub and one or more gateways.
That's the architecture which appeals to me. If it has enough appeal to others, including and especially hek, then maybe that will be version 1.5 or 2.0 of MySensors.
This is where I was generally heading before discovering MySensors (which is WAY ahead of me in terms of practical implementation).
-
To give my two cents, I think the current setup is quite ok. On the hardware side you want as less as possible traveled through the air with as less power consumption as possible.
controller: Software/End user.
gateway and further: mysensors.To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup. I can only agree with the fact that the api should care less about it's controller. And the sensors (library) should care more about data types. For example one byte extra explaining the payload's datatype so the controller can create the correct datatype for itself or further dispatching of the data. Also the sensors should, in my opinion also never report metric OR imperial, it always should be one standardized result over all the sensors and let the "controller" correct these.
Also about the grouping, and discard if i did not understand correctly, should never be done by the hardware supplier, in this case mysensors. It should always happen at controller level.
I do agree with the guaranteed message delivery, but to have it done within the hardware brings some complications, buffer size not to big to have enough room for incoming messages, etc... I do believe that re-sending queues should be handled on the controller side and not on the gateway side. This gives more flexibility in choices if particular messages can eventually be discarded or not. If not QOS should be implemented on the gateway.
Also the gateway should not care about any transport protocols used out there. It should only care about the transport layer of the data with the payload to be as small as possible to keep the gateway doing as less ass possible and keep precious time free to do what it needs to do, be an hardware gateway. Nothing more and nothing less. Using one standardized protocol over tcp and serial looks like enough for me personally and let my controller (which packs more power/cycles to do other stuff then the gateway) convert these to the required protocol.
But these are just my two cents, let the gateway be nothing more then a gateway and any form of data handling/grouping/etc.. should be done by the controller. A controller should meet gateway requirements, not vice verse.
-
@Zeph
I really like that we have these kinds of discussion for the upcoming versions!Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).
Couldn't agree more on unit and scaling. Just sending standard SI units from sensors would be preferred. But this requires more logic on the controller plugin side. Which is fine for me now when things has settled a bit. But when the project was young I wanted to keep change-frequency and coding/logic on the plugin side to a minimum as I hate the freakin LUA coding (with minimum debug possibilities on Vera). Here the work would be to document the proper unit for supported variables.
I don't fully understand the wireless hub/gateway. Maybe just a wording thing.
For me a driver concept would fit better where we create transport drivers for NRF24L01/Serial/Other Radio Model/Ethernet.
So the current serial-gateway would be a simple combination of Serial + NRF24L01 drivers.@Yveaux, to tear the project into several competing forks will probably not benefit the MySensors community as things will just get more complex for the end users (it is already a pretty steep learning curve for the electronics/programmer beginners). We also have to think about how to attract more controller plugin developers. This requires stable versions to keep things in sync on controller side.
There will never be any guaranteed delivery in any network as nodes can be offline for long periods of time. The big question is how much we can afford to buffer (time and memory vise). And where buffering should take place. Here I've seen different opinions. Some like the MySensor library to be slim with acking (taking care of resend for specially important messages themselves) while others like to have a more advanced resend functionality build into the library. We could probably combine these requirements by being smart (driver architecture or by providing plugins like you proposed).
@marceltrapman
In the z-wave world the controller is buffering and resending commands a few times with pretty long delays in between (1-5 seconds) over the z-wave network before giving up. After this the node is considered DEAD by some controllers (which means no more resends occur until it gets replies) others just show a problem status for the device. The z-wave nodes tries to find other routes to the destination but will eventually give up. As I understand the buffering in nodes is minimal (if any at all?) and the resend logic is handled on a higher level.A general comment: As as always we do appreciate contribution of both code and ideas to the project! The problem for me right now is that there is a lot of opinions/ideas but not much help implementing them. This needs to change if we want this project to evolve faster.
So.... Start these kinds of discussions. Make concept classes. Discuss. Draw sketches of your ideas and post them here. Code. Create pull requests on github. Document things you find unclear. All is welcome!
My god... this was a long answer.
@hek said:
Grouping. I'm not fully convinced we can live without any grouping at all. Say for example we have two barometers attached to one node. They both report temperature, pressure and altitude. The reported variables need to be kept together in some kind of logical group not to mix values between sensors (one could be outside while the other is mounted on the inside of the house).
One of the fundamental ideas in the original post is separating the controller's UI based semantic groupings from the sensactuator nodes, so that the code running in nodes is not Vera specific (nor specific to any other controller).. That way the sensor network could be controller agnostic, and translation to controller concepts would be done by the gateway to a given controller type (not yet sure if this is a plugin or driver, in your terms).
What I hear from you is that there may also be some inherent semantic grouping of related sensactuator variables - not dependent on the controller and it's UI objects, but deriving from the physical world being sensed and controlled by the nodes. So if a sensor node were to have two DHT-22's (which can be separated by a fairly long cable), there would still be value in associating temp1 & hum1 as separate from temp2 & hum2.
I have two tentative responses to that.
-
This inherent grouping might be handled by keeping the MySensors concept of a child id within a node.. Child 1 might have Temp and Humidity values, and child 2 might also have temp and Humidity values. That could still make sense. The thing to avoid is implementing controller-specific Sensor type at the node level.
-
The gateway to a given controller could be flexibly able to group arbitrary sensactuator variables into a given controller-specific object (with a controller-specific type). It might be common to associate sensactuator variables from the same child of the same node with a given controller object, but that need not be a limitation.
One concept that might be useful is giving each child a network unique text name. The gateway configuration might refer to that child by name; this would be translated by the wireless hub into a node id and child id as transport concepts.
-
-
To give my two cents, I think the current setup is quite ok. On the hardware side you want as less as possible traveled through the air with as less power consumption as possible.
controller: Software/End user.
gateway and further: mysensors.To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup. I can only agree with the fact that the api should care less about it's controller. And the sensors (library) should care more about data types. For example one byte extra explaining the payload's datatype so the controller can create the correct datatype for itself or further dispatching of the data. Also the sensors should, in my opinion also never report metric OR imperial, it always should be one standardized result over all the sensors and let the "controller" correct these.
Also about the grouping, and discard if i did not understand correctly, should never be done by the hardware supplier, in this case mysensors. It should always happen at controller level.
I do agree with the guaranteed message delivery, but to have it done within the hardware brings some complications, buffer size not to big to have enough room for incoming messages, etc... I do believe that re-sending queues should be handled on the controller side and not on the gateway side. This gives more flexibility in choices if particular messages can eventually be discarded or not. If not QOS should be implemented on the gateway.
Also the gateway should not care about any transport protocols used out there. It should only care about the transport layer of the data with the payload to be as small as possible to keep the gateway doing as less ass possible and keep precious time free to do what it needs to do, be an hardware gateway. Nothing more and nothing less. Using one standardized protocol over tcp and serial looks like enough for me personally and let my controller (which packs more power/cycles to do other stuff then the gateway) convert these to the required protocol.
But these are just my two cents, let the gateway be nothing more then a gateway and any form of data handling/grouping/etc.. should be done by the controller. A controller should meet gateway requirements, not vice verse.
@John said:
To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup0.
I did not fully understand your post yet, John. I think you were largely agreeing, tho at times you appeared to be thinking you were ratifying how it works now. Let me try to clarify. I'm not radically changing your diagram, but am suggesting splitting the "gateway" functionality and partitioning some of the knowledge between the ends of that spectrum.
Let's start with some roles.
Controller - one of several possible programs which were written independently of MySensors and which are intended to control or at least record values from a set of sensors and actuators. See the thread on Open Source Home Automation for a list of examples. Vera is one of these. Recording in the cloud is also in this category.
Sensor/Actuator node - an addressable microcontroller which interfaces with the physical world using sensors and actuators, sends measurements towards the controllers and accepts commands from the controllers (both of these can be done through transferring new values for defined variables). In some networks, like MySensors, these nodes may also relay messages between nodes that cannot communicate well directly.
What I'm advocating is that sensor/actuator node should not need to know or care about the conceptualizations unique to each of the varied controllers. Their code should reflect the nature of the sensors and actuators, and the nature of the network (eg: a nRF24L01+ based wireless networking stack) they use, period.
Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.
Between the sensor/actuator node and the controller there are two distinct functions - the network hub and the adapter for a given controller.
The network hub is part of the sensor network architecture; it handles the centralized end of keeping the sensor network functioning. It understands basically the same concepts that a sensor node understands - packets and acks and routing and addressing and transporting various types of variables via packets.
The adapter function (which I earlier called a "gateway", I'm withdrawing that term as ambiguous) translates between the concepts and communication protocols of a given controller, and the more general concepts of the sensor network. It has to understand the API provided by the network hub software, and it has to understand the controller it was written for. An adapter written for Vera would understand the Vera defined Sensor Types, and would group sensor level variables into Vera type Sensors as needed.
In the current MySensors implementation, the adapter function may reside partly in Lua code & configuration within Vera, and partly in a "gateway node" which also contains the network hub functionality.
I think the current architecture is actually very cool. All designs are tradeoffs, optimizing one aspect at the expense of another. The current design was optimized for one point in the evolution of the MySensors network, when it was well adapted to Vera. However MySensors is at the point where it's poised to support multiple controllers, and the architecture or design may need to evolve to handle that well. Hence this discussion.
I don't aim to change version 1.4, or to distract hek from the wonderful work being done on the existing network. I hope to constructively influence future versions, and I hope that having a vision for where future versions may go will be useful at times when choosing alternatives in intermediate releases.
-
@John said:
To think as it is right now: controller <-> gateway <-> sensor node <-> sensor is a very nice setup0.
I did not fully understand your post yet, John. I think you were largely agreeing, tho at times you appeared to be thinking you were ratifying how it works now. Let me try to clarify. I'm not radically changing your diagram, but am suggesting splitting the "gateway" functionality and partitioning some of the knowledge between the ends of that spectrum.
Let's start with some roles.
Controller - one of several possible programs which were written independently of MySensors and which are intended to control or at least record values from a set of sensors and actuators. See the thread on Open Source Home Automation for a list of examples. Vera is one of these. Recording in the cloud is also in this category.
Sensor/Actuator node - an addressable microcontroller which interfaces with the physical world using sensors and actuators, sends measurements towards the controllers and accepts commands from the controllers (both of these can be done through transferring new values for defined variables). In some networks, like MySensors, these nodes may also relay messages between nodes that cannot communicate well directly.
What I'm advocating is that sensor/actuator node should not need to know or care about the conceptualizations unique to each of the varied controllers. Their code should reflect the nature of the sensors and actuators, and the nature of the network (eg: a nRF24L01+ based wireless networking stack) they use, period.
Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.
Between the sensor/actuator node and the controller there are two distinct functions - the network hub and the adapter for a given controller.
The network hub is part of the sensor network architecture; it handles the centralized end of keeping the sensor network functioning. It understands basically the same concepts that a sensor node understands - packets and acks and routing and addressing and transporting various types of variables via packets.
The adapter function (which I earlier called a "gateway", I'm withdrawing that term as ambiguous) translates between the concepts and communication protocols of a given controller, and the more general concepts of the sensor network. It has to understand the API provided by the network hub software, and it has to understand the controller it was written for. An adapter written for Vera would understand the Vera defined Sensor Types, and would group sensor level variables into Vera type Sensors as needed.
In the current MySensors implementation, the adapter function may reside partly in Lua code & configuration within Vera, and partly in a "gateway node" which also contains the network hub functionality.
I think the current architecture is actually very cool. All designs are tradeoffs, optimizing one aspect at the expense of another. The current design was optimized for one point in the evolution of the MySensors network, when it was well adapted to Vera. However MySensors is at the point where it's poised to support multiple controllers, and the architecture or design may need to evolve to handle that well. Hence this discussion.
I don't aim to change version 1.4, or to distract hek from the wonderful work being done on the existing network. I hope to constructively influence future versions, and I hope that having a vision for where future versions may go will be useful at times when choosing alternatives in intermediate releases.
Ok, now I understand it more by giving the gateway the term adapter, and if I understand correct, the hub would be the gateway as i had in my previous post. I think we do agree that the current architecture. But i have to disagree a little bit about the libraries design, if this is the design you meant.
All though it looks like it has been adapted for the vera, in which i partly agree, the current protocol used is, in what i believe, portable enough to be used in all Home Automation projects which support custom adapters. I just can not agree in mysensors should support multiple kind of controllers. Multiple kind of controllers should support mysensors.
If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?
- Is it a piece of hardware between the hub and controller?
- Is it a piece of software which needs to be coexisting on the network hub?
- A combination of software in the controller and software in the hub?
- Is it a piece of software build into the controller?
My personal view with the above are the next:
- This would create extra costs, so i do not think this is what it is, unless someone specific chooses for this solution.
- I think this also would not be the case because this causes or a lot of forks or mysensors can less focus on the core and must provide libraries for multiple controllers.
- This would be completely not maintainable, this means coders would have to write code and for their controller software and for mysensors hardware. I personally think this is a maintenance nightmare, when the controller or hub evolves they must combine different versions of different kind of software which should work together.
- This would in my opinion be the best case. And i will explain why i believe this:
MySensors should focus only on their sensor networking. Supplying vera plugins is off course a choice, but the fault lies in the fact that the MySensors API has a bit extra for the vera. This should not have been done, but i understand why it is. But, to explain:
The controller should include the adapter, a controller should be adaptable. A base conception of a controller then should be:End user interface <-> Internal data handling (Controller) <-> hardware protocol to Internal data protocol and vice verse (Adapter) <-> MySensors Hub <-> etc...
If it ain't possible to install/expand/etc... a controller to include an adapter to translate the hardware protocol used into the internal data handling protocol, it is a bad controller design case and exposes the controllers shortcomings.
I understand the need for implementing multiple controllers via shared or other types of adapter software with MySensors, and i think it is mainly because a lot of home automation software development is not evolving or adapting that fast enough to support an evolving hardware protocol or the software has shortcomings. And because they are not that fast or flexible, the hardware supplier should be responsible to make sure they are. And i believe this should not be the case.
My view on this is because my software works with a "drivers" system of which is capable of supporting multiple versions of a specific piece of hardware versions used protocols. This gives the end user a choice of what they would like to use. In my personal view i can not agree with having a shared adapter setup with combined controller and MySensors implementation if not implemented on the controller. But again, I do understand the need for these kind of adapters, but, should MySensors be responsible for this?
Well ,that was a big one... I hope I made it more clear, and yes, i did agree with you for the biggest part.
-
@Zeph said:
Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.
For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.
With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE} gw.present(<id>, S_BAROMETER , variables);One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.
-
@Zeph said:
Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.
For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.
With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE} gw.present(<id>, S_BAROMETER , variables);One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE} gw.present(<id>, S_BAROMETER , variables);One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.
It should be possible because the presentation is in order in the sensors because of the char array:
This can possibly create V_FORECAST, V_TEMP, V_PRESURE, V_TEMP
Where the controller map these to for example: V_FORECAST_1, V_TEMP_2 V_PRESURE_3, V_TEMP_4.
The only but of course is that there then should be an extra parameter used which defines which position in this array has been send...
[Edit]It is on the sensor, and not if[/Edit]
-
@Zeph said:
Concretely, a node could have multiple "child" devices which functionally group related sensors and actuators (accepting hek's point) - as in the child id of MySensors. This is part of the addressing and transporting functionality of the sensor network. And they would understand variables and variable types - integer, floating point, text or enumerated values which can be sent or received. But they would not have knowledge of a 'Sensor Type' which is specific to a given Controller and meaningless to another controller.
For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.
With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.
char variables[] = {V_FORECAST, V_TEMP, V_PRESURE} gw.present(<id>, S_BAROMETER , variables);One problem with todays implementation (because child/group is part of addressing) is that you can only have i.e. one V_TEMP per group/child.
@hek said:
For a child sensor (or group) today you can hint the controller what the group represent sendPresentation()/present(). The controller can ignore this if it has no use for this information. But it might be useful for some to minimize the manual work on controller side to detect how to display and or group the device. The problem is that there is no way we can force a sensor to send in all its variables.. so some sensor might wait a long time before they actually send their first altitude or whatever.
So the sensor type can serve as a hint as to some of the variables a child might have. But of course there are so many possible conbinations that it's easy for a "off the shelf" sensor type to have variables that a real sensactuator doesn't support or vice versa. And two different controllers may have different sets of variables they associate with a Barometer type (or whatever). So...
With a proper presentation (of sensor groups) when sensors starts we have a way of prepare things on controller side.
Maybe present() is not a good name and we could perhaps add some new properties to this concept to make it more useful.One idea could be to include the variables controller can expect from the group. This way the hint could make more sense for other controllers.
Yes, that is exactly the type of reframing I am supporting. It's open ended, with no assumptions about how any given controller will operate. If your actuator has Back EMF sending, then the sensactuator node can report it, whether or not the controller you are currently using anticipated that. If you have a weather station with "rain intensity" (vibration sensor measuring the intensity of impact on a metal plate), you can create such a variable and report it.
The functional component I am not calling the "adapter" needs to map the actual sensors and actuator variables in the sensor network, to concepts in the home automation controller it was written to adapt. In the adapter's configuration, one might indicate that (node 5)(child 2)(temp) and (node 5)(child 2)(humid) variables will be packaged as part of a SimpleEnvironmentMonitor object "LivingRoomMonitor" as it's "t1" and "h1" members. (I'm making up a controller example here). For another controller, the adapter might group sensactuator variables differently - including ignoring some or making use of some the first controller ignored.
That's a big part of what the adapter does - it maps and groups (and scales/translates if necessary) the MySensors sensactuator variables into the objects and terminology of a given controller. Since there may be a lot of commonality in this function across controllers, it's possible that the adapters for different controllers would share code and configuration files.
I think we are converging.
-
Ok, now I understand it more by giving the gateway the term adapter, and if I understand correct, the hub would be the gateway as i had in my previous post. I think we do agree that the current architecture. But i have to disagree a little bit about the libraries design, if this is the design you meant.
All though it looks like it has been adapted for the vera, in which i partly agree, the current protocol used is, in what i believe, portable enough to be used in all Home Automation projects which support custom adapters. I just can not agree in mysensors should support multiple kind of controllers. Multiple kind of controllers should support mysensors.
If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?
- Is it a piece of hardware between the hub and controller?
- Is it a piece of software which needs to be coexisting on the network hub?
- A combination of software in the controller and software in the hub?
- Is it a piece of software build into the controller?
My personal view with the above are the next:
- This would create extra costs, so i do not think this is what it is, unless someone specific chooses for this solution.
- I think this also would not be the case because this causes or a lot of forks or mysensors can less focus on the core and must provide libraries for multiple controllers.
- This would be completely not maintainable, this means coders would have to write code and for their controller software and for mysensors hardware. I personally think this is a maintenance nightmare, when the controller or hub evolves they must combine different versions of different kind of software which should work together.
- This would in my opinion be the best case. And i will explain why i believe this:
MySensors should focus only on their sensor networking. Supplying vera plugins is off course a choice, but the fault lies in the fact that the MySensors API has a bit extra for the vera. This should not have been done, but i understand why it is. But, to explain:
The controller should include the adapter, a controller should be adaptable. A base conception of a controller then should be:End user interface <-> Internal data handling (Controller) <-> hardware protocol to Internal data protocol and vice verse (Adapter) <-> MySensors Hub <-> etc...
If it ain't possible to install/expand/etc... a controller to include an adapter to translate the hardware protocol used into the internal data handling protocol, it is a bad controller design case and exposes the controllers shortcomings.
I understand the need for implementing multiple controllers via shared or other types of adapter software with MySensors, and i think it is mainly because a lot of home automation software development is not evolving or adapting that fast enough to support an evolving hardware protocol or the software has shortcomings. And because they are not that fast or flexible, the hardware supplier should be responsible to make sure they are. And i believe this should not be the case.
My view on this is because my software works with a "drivers" system of which is capable of supporting multiple versions of a specific piece of hardware versions used protocols. This gives the end user a choice of what they would like to use. In my personal view i can not agree with having a shared adapter setup with combined controller and MySensors implementation if not implemented on the controller. But again, I do understand the need for these kind of adapters, but, should MySensors be responsible for this?
Well ,that was a big one... I hope I made it more clear, and yes, i did agree with you for the biggest part.
@John said:
If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?
- Is it a piece of hardware between the hub and controller?
- Is it a piece of software which needs to be coexisting on the network hub?
- A combination of software in the controller and software in the hub?
- Is it a piece of software build into the controller?
The adapter in my description so far is a functional component that handles certain aspects of the data flow. It could potentially be implemented in any of those forms, while providing that translating and mapping functionality.
As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.
John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.
On the other hand, somebody who is fluent in that controller's language and system has to implement the whole thing. As mentioned in my previous response to hek, there may be a lot of common functionality across adapters (mapping MySensors sensactuator variables to controller, along with creating/editing/storing that configuration) - which may need to keep being re-implemented in a new environment/language/etc. Not insurmountable, but a factor.
Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.
But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.
However, none of that should require any change to the actual sensactuator nodes.
-
@John said:
If i understand correct, the adapter should be the translating proxy between the network hub (mysensors protocol) and the controller (automation software). The only problem i have with this approach is what is a correct adapter type?
- Is it a piece of hardware between the hub and controller?
- Is it a piece of software which needs to be coexisting on the network hub?
- A combination of software in the controller and software in the hub?
- Is it a piece of software build into the controller?
The adapter in my description so far is a functional component that handles certain aspects of the data flow. It could potentially be implemented in any of those forms, while providing that translating and mapping functionality.
As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.
John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.
On the other hand, somebody who is fluent in that controller's language and system has to implement the whole thing. As mentioned in my previous response to hek, there may be a lot of common functionality across adapters (mapping MySensors sensactuator variables to controller, along with creating/editing/storing that configuration) - which may need to keep being re-implemented in a new environment/language/etc. Not insurmountable, but a factor.
Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.
But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.
However, none of that should require any change to the actual sensactuator nodes.
@Zeph said:
As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.
Next to the fact i'm also new to MySensors, and currently building native support for it in my system, the above is also my observation. The only thing i'm seeing when i look at the gateway/master node, is that i'm only seeing MySensors specific protocol with only some functionality added to make it more compatible with the Vera. If the choice was made to support an other home automation system i think we would still see all the same MySensors specific protocol code and output, but then with some additions made to it for that other system. The current protocol setup can cause the confusion that the current protocol is Vera specific, while i think it's not, it has some added functionality. But if this ain't the case @hek should stand me correct of course.
John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.
Well, i personally think that is the only solution. It is impossible to generate common objects. JSON objects are not common, well the objects are, but the data they represent are not, same for xml, mqtt etc... And that's of course where the problem resides. And something you mention further on, if i understood correctly.
Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.
And now is the question, how would the adapter publish the simple syntax file to the controller. Or even better, how would the controller interpret the output from the adapter? Then we still need a plugin/driver system to understand this data presentation.
But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.
I completely understand, i hope, what you are trying to say. But then there are some personal believes that oppose. I think every component, software or hardware, should serve it's purpose. A hub should only be a hub. At the moment you put software extending logic in the hardware controller, for example the hub. The hub ain't a single purpose anymore and becomes part of the software ecosystem. Who will then be responsible for this ecosystem mix? There now already is a dependency called Vera, if Vera changes, the gateway needs to change, even when there is no MySensors protocol change needed at all. By having dependencies with your product you need to spend time on projects that are not part of the core which is MySensors hardware.
The questions then also are: If there is a part put on the Arduino then
- Who is responsible for the part of the code for controller compatibility?
- Why limiting the code for controller compatibility to "most" of them?
- If you use a controller which is not one of the "most". Who then creates the compatibility, other code on the Arduino or Controller?
- By mentioning "most" you are already ruling out the others.
The current protocol setup does allow it to be used with i think every controller, as long there is an adapter on the controller. And this makes me understand the need for standardization or more compatibility on for example the hub. But, this would be fighting a losing battle.
However, none of that should require any change to the actual sensactuator nodes.
Of course :)
I personally believe the current protocol implementation is good, not too much data over the air and serial/network controller. The less data there is the more time can be spend to sensor sleeping and have the controller do other stuff... The lesser the better.
I do absolutely agree there should be a syntax file. But i think this should be limited to MySensors protocol structure and data presentation. We do now know what is what if we take a look at the source. It would be nice of course if we would not have to.
My Domotica project: http://www.pidome.org
-
@Zeph said:
As best I understand it, currently the adapter function for Vera is a combination of Lua scripting on Vera and some code on the network master node (which also contains the code for the network hub function). I'm still new to MySensors tho and the current implementation is still coming into focus for me.
Next to the fact i'm also new to MySensors, and currently building native support for it in my system, the above is also my observation. The only thing i'm seeing when i look at the gateway/master node, is that i'm only seeing MySensors specific protocol with only some functionality added to make it more compatible with the Vera. If the choice was made to support an other home automation system i think we would still see all the same MySensors specific protocol code and output, but then with some additions made to it for that other system. The current protocol setup can cause the confusion that the current protocol is Vera specific, while i think it's not, it has some added functionality. But if this ain't the case @hek should stand me correct of course.
John, I think you advocate that the adaptor function be implemented entirely within the controller's system. Often a controller has some kind of plugin or driver system for adding such things. I think that writing a controller-specific plugin or driver which includes the adapter functionality may sometimes be a good solution.
Well, i personally think that is the only solution. It is impossible to generate common objects. JSON objects are not common, well the objects are, but the data they represent are not, same for xml, mqtt etc... And that's of course where the problem resides. And something you mention further on, if i understood correctly.
Somewhere there needs to be configuration which maps MySensor variables into controller objects, members, and concepts, and this need is common across adapters. I could imagine a simple syntax text file for example, which would be fairly consistent across controllers and their adapters.
And now is the question, how would the adapter publish the simple syntax file to the controller. Or even better, how would the controller interpret the output from the adapter? Then we still need a plugin/driver system to understand this data presentation.
But in honestly, I'm not that far along in understanding the details, so while I can foresee some of the general principles that would influence a decision, I'm not weighing in heavily about which implementation of the adapter function will turn out to work best, for a given controller, or across "most" controllers. That's why I like the flexibility implied by the 4 alternatives. If you start by trying to implement a plug in or driver for a given home automation controller and it looks like your work would be simplified by putting some of the adapter logic into an Arduino (perhaps the same one that serves as wireless network hub), that would be an option to consider.
I completely understand, i hope, what you are trying to say. But then there are some personal believes that oppose. I think every component, software or hardware, should serve it's purpose. A hub should only be a hub. At the moment you put software extending logic in the hardware controller, for example the hub. The hub ain't a single purpose anymore and becomes part of the software ecosystem. Who will then be responsible for this ecosystem mix? There now already is a dependency called Vera, if Vera changes, the gateway needs to change, even when there is no MySensors protocol change needed at all. By having dependencies with your product you need to spend time on projects that are not part of the core which is MySensors hardware.
The questions then also are: If there is a part put on the Arduino then
- Who is responsible for the part of the code for controller compatibility?
- Why limiting the code for controller compatibility to "most" of them?
- If you use a controller which is not one of the "most". Who then creates the compatibility, other code on the Arduino or Controller?
- By mentioning "most" you are already ruling out the others.
The current protocol setup does allow it to be used with i think every controller, as long there is an adapter on the controller. And this makes me understand the need for standardization or more compatibility on for example the hub. But, this would be fighting a losing battle.
However, none of that should require any change to the actual sensactuator nodes.
Of course :)
I personally believe the current protocol implementation is good, not too much data over the air and serial/network controller. The less data there is the more time can be spend to sensor sleeping and have the controller do other stuff... The lesser the better.
I do absolutely agree there should be a syntax file. But i think this should be limited to MySensors protocol structure and data presentation. We do now know what is what if we take a look at the source. It would be nice of course if we would not have to.
@John said:
The current protocol setup can cause the confusion that the current protocol is Vera specific, while i think it's not, it has some added functionality. But if this ain't the case @hek should stand me correct of course.
The serial over-the-air and serial protocols has been designed to be vera independent. The variables/sensor types is of course inspired by how vera has arranged their devices. But as I said earlier. If a controller gets meta information on how sensors is arranged it can choose to use this information or not.
We can conclude the following from this discussion (as I see it):
- We need some sort of grouping mechanism. Today this is called "sensor" which is identified by a sensorId (0-254). All sensor-groups has a sensor-type which is a hint to controller. I can't right now see any alternative to the this (more that providing even more information like I proposed below). An adapter or controller-plugin needs this to prepare/create/decide how and what type of data it should present and group together. This is of course optional. The controller can just throw this away and wait for the first sensordata to arrive.
- We should move unit information/conversion back to the controller. All sensor data is send as SI-units. This will make the sketches even simpler but it puts some extra workload on the plugin developers. I'll discuss this further with @marceltrapman and @john who is the first to port/create 1.4 plugins for their respective favorite controllers.
@Zeph I need to see a more concrete use case for moving more logic to to the Arduino-gw-side in a adapter setup. The Arduino MCU is very limited and we need to keep the code really tight there.
I suggest you help out building a plugin for your favorite controller (OpenHAB and Domoticz has a lot of requests). You will more easily see missing details in the meta information you receive from the sensor network and will hopefully come back with even more helpful insights. -
I think this is discussion is converging, as I've said. Good points made and taken all around.
There seems to be agreement that the sensactuator nodes should be controller agnostic.
The largest area where they are not is in choosing a "sensor type" which is Vera specific. That's what started this rethinking for me - in trying to annotate the MySensor libary's variable and sensor type enums (in the lost forum posts), I wasn't sure how to describe what a "sensor type" really represented. I tried to find some independent meaning to the Sensor type that I could document, but failed. If it says the sensor type is heater, what does that imply? Low/Medium/High or a thermostat or what? It is controlling a heater or just monitoring one, or both? The conclusion was that type means nothing more or less than "you may be able to fit this 'sensor' into what Vera calls a heater", meaning there may be a user interface object in Vera into whose displays and user controls you can reasonably well map the variables and behavior of this sensor, and the icons displayed for that object in Vera may more or less fit the functionality of this sensor. There was no Vera-independent semantics to the sensor-type.
However, hek suggests that this is just a hint which can be ignored, so one could say that it's at worst a small and meaningless overhead for some controllers, and at best makes Vera easier to interface with. Not a show stopper to keep it around for now, since it can be ignored. It might be more clear to call this data "Vera-hint" than "sensor type", tho. Other controllers could try to discern something from the Vera-hint, or ignore it.
That said, moving the metric/imperial conversion to a centalized location seems like a good move, and hek's suggestion of letting senseactuator nodes specify what variables a given child device ('sensor') implements, seems like an important move in the right direction. I think that any controller could benefit from getting a list of what variables a "sensor" (an addressable node+child-id grouping of variables) generates and accepts, even Vera. So this goes a long way towards addressing my concerns, whether or not the sensactuator nodes continue to present a vera-hint as well.
John feels strongly that all the adapter functionality should be implemented entirely within each home automation controller. I've never argued against that option, only suggested keeping open some other possibilities as well. However, if no need to implement some or all of the adaptation outside the controller ever comes up, that would of course be fine with me.
hek makes the point that the gateway (wireless hub) does not have room or performance to implement very much. Given that I've never suggested that the gateway be forced to implement any adapter functionality, we are not exactly disagreeing (our views are not incompatible). However that brings up a more general point, in that I would not design the architecture at any deep level to assume that the gateway will will continue to be limited to an ATMega328p @16MHz and 2K of RAM.
I DO see reason to design for the Arduino Pro Micro for the sensactuator nodes: There can be many of them, and the APM is very cheap, it's small, and it can be very low power to accommodate battery power (and it's Arduino compatible so it can inherit a rich set of device libraries). And it's adequate for most sensactuator tasks; in most cases more power would be wasted anyway. I don't see much on the near term horizon that fill the sensactuator node niche much better than the APM class device for MySensors. (Only possible exception: if the nRF51822 became more widely used in the form factors and price point the APM+nrf24L01+ occupy).
But the gateway needs to be on full time and wired to the controller, so it can be wall powered and even a bit physically larger if need be, and since there is only one of them it could be a bit more expensive. Arduino Mega clones are $15 - no faster but 4x the RAM and more peripherals. A teensy 3.1 has 64K of RAM and runs 32 bit ARM code using the Arduino IDE and library interfaces for $17. Arduino Due clones with an ARM and 96 KB can be had for $20. ARM based STM32F103 with 20K RAM, 72 MHz processor and lots of IO are available on eBay for under $7. Or you can get a ST32F411 Nucleo for about $10 with floating point ARM (they can use the Arduino-like mBed system). The RPi is $35, the BBB is $55 now, and there are several other GHz class ARM devices under $80 like the cubieboards (plus shipping; the other prices include shipping BTW). Given all that, I'm going to be very surprised if two years from now most of the MySensors wireless gateways/hubs are still running ATMega328p processors.
So while I'm not proposing now to force all gateways to be larger than an ATMega328p, I'm also not assuming in designing an architecture that the gateway hardware can never do more than that chip could do - at least in some configurations. If we had to spend an extra $5-15 dollars for a much more capable (single) gateway in order to interface with a given controller we wanted to run, that would not be a limiting factor for most of us.
AND - John may be right - maybe there will never come a case where it makes any sense to implement the adaptation functions anywhere except on the controller (eg: inside a Vera or a PC running home automation software, etc). Time will tell. If I differ some in the degree of flexibility I suggest, I am nevertheless respecting the other views which have been well stated here, and have no certainty that such flexibility will be needed. We seem to be agreeing on the fundamentals.
-
I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.So, I would prefer if SensorType described what a human would classify a sensor node as.
I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display
setting and nothing else. -
I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.So, I would prefer if SensorType described what a human would classify a sensor node as.
I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display
setting and nothing else. -
As best I understand it, MySensors was originally developed around the conceptual framing of devices as used by Vera/MiCasaVerde/MIOS; has been remapped by some adapters to work with other systems; and is at the stage where the semantic gaps are becoming obvious enough to raise the question of rethinking the internal conceptual framework to be more general and less Vera specific.
Let's start from the bottom of the hierarchy - what we could call "variables" - each containing a value which may vary over time.
**Format: **
The values could be- floating point or integer (of various sizes), or
- chosen from an enumerated set of possible values (a common case being binary options like on/off or true/false or armed/unarmed etc)
- a text string
What it measures (or controls):
In the case of a numeric value, this could be temperature or relative humidity or wind speed or water depth or brightness or rate of flow of electricity or water or accumulated flow of electricity or water or loudness of sound, or angle of a servo, etc. Sometimes there may be no conventional dimension or interpretation.Units and scaling
There may be a translation to a standard unit of measure like degrees C or F, or percent humidity, or feet or meters, or mph or kph or m/s, or lux, or db or angular degrees or radians, etc. Sometimes we may know the dimension being measured (brightness) but have no translation to standard units (not calibrated in lux, just an arbitrary and perhaps non-linear value) so this is optional. And sometimes there may be a scaling factor needed to get to some standard unit, from the value reported.The dimension and the units are often conflated, but of course a value in degrees C could be a temperature reading or a dewpoint, two different dimensions which use the same units.
That's the basic level - a set of distributed variables measured from the real world (or computed from measurements), or controlling some action in the real world. Then we built on top of that.
Grouping of related measurements
We might wish to associate several variables with a given device or sensor. For example a weather station might have temp and humidity and wind speed and wind direction, etc. Or a flow meter might have immediate flow (gph or kw) as well as accumulated flow (gals or kwh).This is where some things go a little wonky, because there isn't always one obvious way to logically group variables, and there may be little guidance as to the best practicies OUTSIDE OF matching the way a particular controller has chosen to conceptually organize things.
From the top down, MySensors groups sensactuator variables by the wireless node which reports or uses them. This has a lot of practical sense to it, but in some sense it's a transport or addressing concept rather than a sensor concept. Suppose you had some one wire sensors strung around the property. As you add, remove and move wireless nodes around over time, you might find it more convenient to connect the (relatively long) wiring to different wireless nodes for reporting to the controller. In this case, the actual one-wire sensor would not change in any logical way (it's still measuring the same thing in the same location) but the values are reported through a different path, a different wireless node. So grouping of values by wireless node should be done lightly - it's not a primary logical grouping based on what is being measured or controlled, but more of "the current address and transport path".
The child id is also part of the addressing and transport path.
From the bottom up, MySensors group sensactuator variables into Sensors, each of which has a sensor type. This is one of the confusing areas, since the types of sensors seems to overlap semantically with the types of variables which might be associated with it.
In trying to sort through this, I think the Sensor type mostly represents "which user interface object in the Vera system should be used to display and accept user input" for a group of variables". So if you have one or more related sensor variables which can logically be associated with a given Vera UI object, you may use that sensor type. Or you can do some more work and add a new sensor type to Vera, and use that.
The problem is - another controller may not choose the same set of UI objects. I think that "sensor type" may be better conceptualized not at the MySensors generalized node level, but as a mapping of the generalized sensor nodes into the Vera system.
In this approach (which is being brainstormed, and is not fully figured out), the sensor node would not need to understand Sensor Type (tho it would understand Variable formats, dimensions, scaling, units and names). Instead the gateway to Vera would map variables into Vera related groupings, and the gateway to another controller would map the variables into the groupings which makes sense to that system.
Suppose that one controller had a user interface element which displayed weather related information graphically, always using metric values. Another controller has a "temp and humidity" user interface object but no weather station, tho it does have a separate flow concept to which wind speed could be mapped; the second one allows the user to select metric or imperial units. The first controller might display wind direction in degrees clockwise from true north, the second might use NNW type notation.
The same physical sensors and associated variables (at the node level) might be grouped into "sensor UI objects" differently in the two controllers. But the nodes shouldn't care whether wind speed is shown inside the same box with temperature (a "sensor" UI element) or in separate boxes, or even omitted from a controller which doesn't have a target for some measurement. They just report values for variables, and the gateways alone handle grouping variables into user interface elements as used by the controller they support.
In this model, more than one controller could be used with the same sensor network, so long as they don't issue conflicting commands. The sensors might always report temperature in C, and each controller + gateway would translate if it so needs. The sensors might report wind direciton in degrees and each controller + gateway would translate to compass points if so needed. If one of the controllers doesn't have a way to display "microcontroller temperature" or "H Bridge back-emf" then it can omit that variable, without precluding another controller from accepting such variables.
Knowledge about the controller and its concepts would be removed from the wireless node code, and concentrated in the gateway to that controller. Nodes just report (or accept) one or multiple sensacturator variables. The gateway could package up groups of sensactuator variables as needed for the specific type of controller it serves. Suppose a given controller had a "temp & humidity" object but no weather station object. As described above, you could map wind speed into some other controller element for now. But then when a weather station "sensor type" with additional variables is defined next year, the gateway could regroup sensor variables into that package, without any changes in the wireless nodes.
Splitting the gateway and the hub
The concept of a gateway may need to be separated from the concept of the wireless hub.
The wireless hub understands how to communicate with the wireless nodes, up through the protocol level of transferring sensactuator variable values back and forth. Like the nodes, it doesn't need to know what controller(s) are attached or their concepts, it just moves values. It could use a different radio chip, or RS-422.
The gateway function translates from this general format to a specific controller's needs. (Cloud storage could be another type of "controller", by the way). The gateway function doesn't have to worry about ack and routing tables for the nRF24L01+.
There is a general API between the MySensors wireless hub code and the MySensors gateways. They can run on the same processor (or not, if there is a transport option for the API).
When you add a new controller, you write a new gateway, but can use the same wireless hub code.
And a message manager like a MQTT broker could be inserted between the wireless hub and one or more gateways.
That's the architecture which appeals to me. If it has enough appeal to others, including and especially hek, then maybe that will be version 1.5 or 2.0 of MySensors.
This is where I was generally heading before discovering MySensors (which is WAY ahead of me in terms of practical implementation).
I just wanted to make a pointer to http://forum.mysensors.org/topic/260/mysensors-mqtt-gateway , Might be a combined discussion about the future of the protocol, I am thinking about the address-layout. Using Vera-style or just ignore some parts of the address.
I have not been reading everything in this discussion, I haven't had time yet. But if you come to any conclusion it would be goot to write them down! :)
-
I think this is discussion is converging, as I've said. Good points made and taken all around.
There seems to be agreement that the sensactuator nodes should be controller agnostic.
The largest area where they are not is in choosing a "sensor type" which is Vera specific. That's what started this rethinking for me - in trying to annotate the MySensor libary's variable and sensor type enums (in the lost forum posts), I wasn't sure how to describe what a "sensor type" really represented. I tried to find some independent meaning to the Sensor type that I could document, but failed. If it says the sensor type is heater, what does that imply? Low/Medium/High or a thermostat or what? It is controlling a heater or just monitoring one, or both? The conclusion was that type means nothing more or less than "you may be able to fit this 'sensor' into what Vera calls a heater", meaning there may be a user interface object in Vera into whose displays and user controls you can reasonably well map the variables and behavior of this sensor, and the icons displayed for that object in Vera may more or less fit the functionality of this sensor. There was no Vera-independent semantics to the sensor-type.
However, hek suggests that this is just a hint which can be ignored, so one could say that it's at worst a small and meaningless overhead for some controllers, and at best makes Vera easier to interface with. Not a show stopper to keep it around for now, since it can be ignored. It might be more clear to call this data "Vera-hint" than "sensor type", tho. Other controllers could try to discern something from the Vera-hint, or ignore it.
That said, moving the metric/imperial conversion to a centalized location seems like a good move, and hek's suggestion of letting senseactuator nodes specify what variables a given child device ('sensor') implements, seems like an important move in the right direction. I think that any controller could benefit from getting a list of what variables a "sensor" (an addressable node+child-id grouping of variables) generates and accepts, even Vera. So this goes a long way towards addressing my concerns, whether or not the sensactuator nodes continue to present a vera-hint as well.
John feels strongly that all the adapter functionality should be implemented entirely within each home automation controller. I've never argued against that option, only suggested keeping open some other possibilities as well. However, if no need to implement some or all of the adaptation outside the controller ever comes up, that would of course be fine with me.
hek makes the point that the gateway (wireless hub) does not have room or performance to implement very much. Given that I've never suggested that the gateway be forced to implement any adapter functionality, we are not exactly disagreeing (our views are not incompatible). However that brings up a more general point, in that I would not design the architecture at any deep level to assume that the gateway will will continue to be limited to an ATMega328p @16MHz and 2K of RAM.
I DO see reason to design for the Arduino Pro Micro for the sensactuator nodes: There can be many of them, and the APM is very cheap, it's small, and it can be very low power to accommodate battery power (and it's Arduino compatible so it can inherit a rich set of device libraries). And it's adequate for most sensactuator tasks; in most cases more power would be wasted anyway. I don't see much on the near term horizon that fill the sensactuator node niche much better than the APM class device for MySensors. (Only possible exception: if the nRF51822 became more widely used in the form factors and price point the APM+nrf24L01+ occupy).
But the gateway needs to be on full time and wired to the controller, so it can be wall powered and even a bit physically larger if need be, and since there is only one of them it could be a bit more expensive. Arduino Mega clones are $15 - no faster but 4x the RAM and more peripherals. A teensy 3.1 has 64K of RAM and runs 32 bit ARM code using the Arduino IDE and library interfaces for $17. Arduino Due clones with an ARM and 96 KB can be had for $20. ARM based STM32F103 with 20K RAM, 72 MHz processor and lots of IO are available on eBay for under $7. Or you can get a ST32F411 Nucleo for about $10 with floating point ARM (they can use the Arduino-like mBed system). The RPi is $35, the BBB is $55 now, and there are several other GHz class ARM devices under $80 like the cubieboards (plus shipping; the other prices include shipping BTW). Given all that, I'm going to be very surprised if two years from now most of the MySensors wireless gateways/hubs are still running ATMega328p processors.
So while I'm not proposing now to force all gateways to be larger than an ATMega328p, I'm also not assuming in designing an architecture that the gateway hardware can never do more than that chip could do - at least in some configurations. If we had to spend an extra $5-15 dollars for a much more capable (single) gateway in order to interface with a given controller we wanted to run, that would not be a limiting factor for most of us.
AND - John may be right - maybe there will never come a case where it makes any sense to implement the adaptation functions anywhere except on the controller (eg: inside a Vera or a PC running home automation software, etc). Time will tell. If I differ some in the degree of flexibility I suggest, I am nevertheless respecting the other views which have been well stated here, and have no certainty that such flexibility will be needed. We seem to be agreeing on the fundamentals.
The largest area where they are not is in choosing a "sensor type" which is Vera specific.......There was no Vera-independent semantics to the sensor-type.......Other controllers could try to discern something from the Vera-hint, or ignore it.
I totally agree with you about the sensor type limitations. It would certainly be nice if the Vera would provide an interface where sensor types can be extended. But i'm not well known enough with the Vera to know if this is the case or not. All though, as you mention, that the sensor type is just a hint which could be ignored, it does give some extra information about the sensor produced variable type values. And where the total sensor types table is concerned it mixes up sensor type semantics. As i read it now in your post it then appears this mix up is because of the Vera and I better understand the protocol part of the discussion. Thanks for this clarification (I shouldn't be doing multiple things at the same time).
As example and thoughts spin off.
In your example you are referring to heater, where as i understand an heater could be a collection of possible sensors and actuators, like temperature and switches. In this example it would be best to have an N_HEATER (heater node) sensor node with a S_TEMP sensor and for example a S_SWITCH actuator. from which produces V_TEMP and V_ON/V_OFF. But even then taken this as an example you still do not know if it is only monitoring, etc... which is only something you can assume depending on the reported sensor types. And it would even be better if the user can define the node him/her self of course.John feels strongly that all the adapter functionality should be implemented entirely within each home automation controller. I've never argued against that option, only suggested keeping open some other possibilities as well. However, if no need to implement some or all of the adaptation outside the controller ever comes up, that would of course be fine with me.
Yes i do, and certainly with the open sourced, or plugin capable automation systems. But, i am open for possibilities where there are protocol based implementations possible on the hardware instead of controller protocol type. And yes this would acquire more capable hub hardware. It would be nice if there where more protocol based solutions like offering an Vera hub, MQTT hub, JSON Hub.
-
I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.So, I would prefer if SensorType described what a human would classify a sensor node as.
I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display
setting and nothing else.@Rasmus-Eneman said:
I have thought a bit about this, not from a MySensors perspective but as a Controller perspective as I'm developing one.
For me SensorType can be interesting (however it will need to be remodeled a bit). If I get a temperature from the sensor,
it's probably a Thermometer, but it may be the inner core temperature or something else. If I get a unit that supports turning
on or off it may be a Lamp, but it may also be a bunch of other things. To create the best experience for the user I would
need to know not just that it's a temperature reading, but from where.No disagreement there, the question is where and how to specify that.
So for this to work S_TEMP would be S_THERMOMETER. Then I know that a V_TEMP is a environment temperature
reading and not a inner core temperature. Obviously all of the SensorTypes would have to be looked over if they need
similar changes.Looking through the sensor types with a perspective like that is what started me on this thread (initially in the messages that got lost).
As you say, a measurement of "temperature" could mean many things. It could be a simple themometer, or it could be a dewpoint, or it could be a daily max from a weather station or it could be the intake or outlet temp from a heater or cooler device, or it could be a themostat setting - all in degrees C.
One approach would be to define a new "sensor type" (as you use it, I'm going to use a different terminology for this concept below) for every type of temperature variable; basically invert the current heirarchy and define many sensors types for every variable type. Put another way, the variable could be the addressable unit, and one of its attributes could be a sensor type from the list defined for that type of variable. So we find that there is a V_TEMPERATURE variable, and we ask it "what kind of V_TEMPERATURE are you?" and it responds that its a "S_TEMP_HIGHLIMIT" or a "S_TEMP_FREEAIR" or some other S_TEMP_* type of temperature.
HOWEVER, hek has made the point that there is utility to defining the a container for multiple related variables (what's called a sensor in the current system, and addressed with a node id and child id). The current concept of the "Sensor Type" is associated with that group or container. So you might have a S_WEATHERSTATION which (potentially) contains V_TEMPERATURE and V_RELHUMIDITY and V_WINDSPEED etc variables. (Multiple variable types for a given sensor type, in this way of looking at it).
The problem with the container is that real world sensors vary a lot - one weather station will include a UV sensor and another won't, or one will measure peak wind speeds and another will not, or one includes both inside and outside temps, etc. So it's hard to pre-specify what all the variables a S_WEATHERSTATION may have, and if you do, most actual ones will be pretty sparsely populated. And then we look at a home automation controller - which of those measurements will their nearest analog to the S_WEATHERSTATION implement? Probably not exactly the same as all other controllers.
I think the closest we've come to sorting through this inherent complexity is something like this:
-
There are Containers which are individually addressable and may report more than one measurement (or accept more than one setting). All of the measurements within a given container are physically or logically related to each other. (This is the thing addressable with a node address and child id).
-
Each measurement (or setting) associated with a container has a V_* variable type which says what is measured, and if possible specifies the standard units or enumerated states.
-
Perhaps each measurement can also have a "semantic" attribute, chosen from a list specific to the variable type (eg: V_TEMPERATURE can have VS_TEMP_FREEAIR or VS_TEMP_DEWPOIT etc). (VS=Variable Subtype or Variable Semantics). This is where I'd put what you were calling a sensor type - as a variable subtype attribute.
-
The central authority can ask each node about the containers (children) it has. It can ask each child what variables it reports or accepts. It can find the V_* variable type and the VS_* subtypes for each variable.
-
The node can also report a hint from the S_ list for each container (child). This hint does not limit what variables the container may report or accept, and it may or may not match well with the way any given controller lumps and splits things - but the hint can be ignored.
-
There has to be some configuration for each controller, which maps variables do defined by the nodes, into variable and objects which make sense for the given controller. (I was calling this the adapter function). This mapping is not required to make a one to one association between addressable variable containers ("children" or "sensors") and the objects in that controller's universe.
An example. Suppose a node reports outdoor temp & humidity and wind speed and direction and UV index, all associated with a single measurement location and reported as variables in the same child container. And there's a controller which doesn't have a "weather station" object to group all of those variables. But it does have an "environment monitor" object which combines temp, humidity and noise level. The adapter function could map the node's temp and humidity to the temp and humidity of the controller "environment monitor" object (leaving noise level unreported), and find some other controller object(s) for wind speed and UV index - or ignore them. (There could be another parallel "controller" like a cloud recording system which does make use of them even if the given home automation controller does not).
So, I would prefer if SensorType described what a human would classify a sensor node as.
If we associate SensorType with a multi-variable container, then there are many mix-and-match varieties of these, as perceived by a human, not all of which are going to be implemented in any given controller. If we mean "variable subtype" as described above, then no problem.
I would also prefer if all units reported were in SI units. If a user want to display the as something else, that is a display setting and nothing else.
I think we are coming to some agreement about that part (single set of standard units for reporting, rather than pushing the user display preferences out to each node). But even that's not quite as simple as saying "SI" and walking away. I've seen wind speed reported in m/s and kph for example. And pressure has multiple possible units. As do light or sound. Even temp has C and K. I've seen fluid flow in cubic meters/hour or liters/min. So I think we should explicitly specify what units each measurement should use (if it can) and that we should look primarily to SI when possible in defining those expectations.
And we need to accommodate uncalibrated reports - like brightness reported based on an uncalibrated and non-linear sensor which nevertheless has higher (or lower) values for brighter light. We may not be able to translate to lux, but the HA controller could nevertheless use empirically set thresholds for triggers or conditions.
-
