Another way of organizing variables

[Zeph]

@hek said:

Regarding keeping meta data on controller rather than transfer it with every message. ... There are of course both advantages and disadvantages with this approach.
If you are communicating directly between sensors in your network (without involving a controller) the meta data certainly makes a difference.

Let's explore that.

As I see it, the core data transfer functionality of the sensor network consists of sending the current values of variables to and from sensactuator nodes. Each node has some number of such variables.

Suppose there's are two variables containing the current temperature and humidity from a given DHT-11 (one of several sensors in that node). How are those specific variables "addressed" or identified as distinct from other variables in the same node?

In the latest proposal, the "identity" of those variables might be:

Node = 6
Child = 2, 
S code = S_TEMPERATURE, 
V code = V_LEVEL.

and

Node - 6
Child = 2
S code = S_HUMIDITY
V code = V_PERCENTAGE

If you change the S code or V code (or child), you are no longer referring to the same variable. So S code and V code are both metadata AND part of the addressing/identity for that variable. (And you can't have two variables of the same type in the same child).

In the new proposal, addressing or identity might be:

 Node = 6
 VariableIdx = 6

and

 Node = 6
 VariableIdx = 7

This is pure addressing. Metadata (S and V codes, or enhancements of them) is kept separate and transmitted only once during network joining (if at all).

OK?

Suppose that node 3 wants to display that temp and humidity, and let's use the currently proposed S and V codes as the metadata for apples to apples.

Now, how would one node know how to communicate directly with another node, without involving the hub/controller/gateway? Node 3 would not inherently know that node 6 exists, or which child within node 6 has the required temperature and humidity - unless you program that information into node 3. And/or program node 6 to send its information to node 3.

Case 1: Node 3 knows that the temp to display has identity/address:

 Node 6
 Child 1
 S code S_TEMPERATURE
 V code V_LEVEL

Case 2: node 3 knows that the temp to display has address:

 Node 6
 VariableIdx 0

and it could just as well continue to know that for that variable has metadata:

 S code S_TEMPERATURE
 V code V_LEVEL

In either case node 3 can know the S and V code, tho in the former those are part of the address and in the latter they are metadata.

Or if node 6 just sends its reports to node 3, it could send metadata with each packet or separately. We can examine that case too, but I don't see it as any show stopper.

Also when implementing simple The Arduino MQTT gateway (with limited resources) would also have a hard time without it.

Could you explain that a bit more? Are you saying that MQTT cannot route well using only node id and variable idx, and that it needs to be able to interpret S and V codes for each packet in flight? If anything I'd think the simpler addressing without S and V would take less resources but I'm not familiar with the internals so I'm guessing.

[hek]

@Zeph said:

How is that?

:thumbsup: Much better :)

Yes, a variableindex is a classical solution for minimizing storage. I've used googles protocol buffer a few times which is a excellent solution for packing structured data. But I'm not fully convinced we should increase the complexity for a byte of saved data which very well could help out in other cases.

Now, how would one node know how to communicate directly with another node, without involving the hub/controller/gateway? Node 3 would not inherently know that node 6 exists, or which child within node 6 has the required temperature and humidity - unless you program that information into node 3. And/or program node 6 to send its information to node 3.

Many people use "hardcoded" node ids and direct communication without any controller/gateway involved. To have them accept presentations and create/store variable indexes would probably be too much effort.

We could also (in the future) support some kind of find-node-with-certain-sensor-attached broadcast message. To locate all temperature sensors in the radio network.

So what I'm saying is that we must be able to handle a gateway/controller-less setup also without too much fuss.

[Zeph]

Now, how would one node know how to communicate directly with another node, without involving the hub/controller/gateway? Node 3 would not inherently know that node 6 exists, or which child within node 6 has the required temperature and humidity - unless you program that information into node 3. And/or program node 6 to send its information to node 3.

@hek said:

Many people use "hardcoded" node ids and direct communication without any controller/gateway involved. To have them accept presentations and create/store variable indexes would probably be too much effort.

But if you are now hard coding the node id and child id, you could just as easily be hard coding the node ID and variable idx instead, no?

You only have to accept presentations if you want to be dynamically configured. If you want to hard code the V code and S code (metadata) you can do that in either case.

So what I'm saying is that we must be able to handle a gateway/controller-less setup also without too much fuss.

I may still be missing something because it seems like we are reducing the fuss by treating V code, S code and (if needed Child/group ID) as pure metadata rather than as both metadata and address/identity.

Or perhaps I'm talking about a little deeper restructuring than I'm yet conveyed to you, and the emergent simplicity is obscured by not yet seeing what falls out.

I'm open to either possibility. Let's first assume that I'm missing something.

Any further thoughts about the MQTT question? I do want to be compatible with that, so your concern about that is something I want to understand and address.

[hek]

@Zeph said:

Any further thoughts about the MQTT question? I do want to be compatible with that, so your concern about that is something I want to understand and address.

@Damme would probably have to give his POV when it comes to the simple Arduino MQTT gateway and variable indexes.

[Damme]

@hek @Zeph
No I don't like the idea that the gateway should remember stuff about the sensors. If should be just a gateway. So it is not only about MQTT gateway but serial / ethernet gateway too. We could have a meta transportation command to transport your meta data from node via gateway to controller that handles all the magic stuff. But this will only lead to yet another byte used up in the set/req. which doesn't actually matter. for most value types 20bytes is enough for almost anything.. :)
And why everyone is so afraid of missing one byte or 3 or 6 is weird, The discussion about the protocol is to have different commands. there some commands have more header data about the payload and other might have none (just transportation header + payload)..
And yes ofc we could add isp flash and stuff but this will make the gateway not being a gateway any longer...

And the mqtt address is at the moment nodeid/sensorid/datatype. next version will have nodeid/sensorid/sensortype(text)/valuetype(text) and because I know at the controller I have the ability to use # at sensor id so I could get data via only names S_TEMP/V_VALUE ie.. and IF I am going to store anything at all is is the NODE NAME. and this will probably fill the available memory up. ofc we could go with a more expensive MCU also.. but like I already said , The gateway should be simple. It would be nice to be able to address something with MQTT/#/base_wash/#/S_Humid/V_Value. .. and maybe skip number ass together. Only to identify the mqtt address with only number, sure that would work, but it would look like a mess. :) if we have predefined types, just send those with the payload and we know what w are dealing with!

[Zeph]

@Damme said:

No I don't like the idea that the gateway should remember stuff about the sensors. If should be just a gateway.

Hmmm. Seems like the gateway or the controller HAVE to remember stuff about the sensors in any case. Otherwise how do they know the address/identity of the temperature and humidity that the living room DHT-11 provide?

All we currently have to remember is that those sensors have address/identity:

 Node ID
 Child ID
 S Code
 V Code

If you fail to remember any of that, you cannot identify or address those variables, right?

Now imagine for a moment that while the gateway or controller must remember all four of those items, they can identify/address the given variables OTA with only two (ie: the latter two values do not need to be sent in every packet referencing the given variable). There's nothing new to remember that you were not already remembering - but you just don't need to send all of your remembered data in each packet.

(And you are remembering a variable idx rather than a child id, but to the gateway either one is just an arbitrary 8 bit number it has been told to associate with the given variable).

What am I missing here? Thanks for your patience.

[John]

I will take my controller as an example:

Node ID, The address of the node
Child ID The sensor on this node
S Code one of the possible more variables send on behave of the sensor node
V code the variable type of the send value

I HAVE to remember those so it can be linked together, so if i receive a value it has to be linked to a var type, from which i can say is possible to only send while presenting..

On my controller side it is possible to leave the V Code out (but a device then MUST present itself), but all the others should stay.

If there would be an arbitrary number to identify it it should be decoded again to match it to the node id, with the sensor id and s code. But that then would be in my controllers case.

[Zeph]

Let me be clear about something. The proposal is largely about cleanly separating static metadata about a variable from the identity/addressing of that variable, not about saving bytes in the packet.

Compared to the the current proposed with S and V codes as both metadata and address/identity, the proposed alternative of transferring those static values during setup rather than with each packet does have the small advantage of saving two bytes. But that's not the motivation or main payoff.

Let me give an example of where it might pay off. Suppose that after a while we realized that it would be useful to pass brightness values which are not expressed in Lux because we have no way to calibrate a given sensor. The current S and V codes only allow brightness to be expressed in Lux. So we could add a new S code for non-Lux brightness. And a new S code for non-dB sound loudness. And a new code for any uncalibrated measurement which is just like the normal measurment but not available in standard units.

But suppose instead we wanted just one bit of semantics: the values reported for this sensor are not in the standard units, but they do measure the same thing. Then this bit of information could be equally associated with any variable of any type, without adding new codes.

We could add that bit to the OTA packet format so that the node would convey that bit of semantics every time it reports a brightness (is this calibrated to standard units or not?). Depending on where that bit was inserted into the packet format, all nodes might need to be modified to use the new packet layout, even if they are calibrated.

Or, with the separated metadata approach, a node could convey that new information just once during setup. The OTA Set command format doesn't change at all. And a node which uses only calibrated unit variables doesn't change at all (the default remains "calibrated"). A node that does have uncalibrated veriables is modified to pass that flag during setup. Done.

---

I see this as a need-to-know design issue, not unlike using protected member variables in a class is a design issue. You should design the users of a class so that they do not need to directly understand the implementation of the class, so that you can later change that implementation without cascading effects on all the users of that class.

Likewise the mechanism for sending the dynamic current value of a variable from point A to point B should be cleanly partitioned from the static metadata needed to describe and interpret that value, so that you can later modify or add to the static metadata without implications for the simple transporting of a dynamic current value. There is a clean need-to-know partitioning between the concepts of static metadata about a variable and dynamic current values, which I strongly believe will pay off in many ways as the design continues to evolve in months and years to come.

I believe that conflating static metadata with the addressing needed to identify a given variable is going to be an ongoing source of semantic and structural confusion. Any later tweaks to the metadata to address newly recognized shortcomings or add functionality will have cascading effects in parts of the protocol (eg variable value transport) which logically would not need to change.

At least that's my current design sense.

[Zeph]

Here's an example of the kind of metadata that might be passed from a node when it joins the network (using the approach advocated in this thread).

(The term "variable" is a little too generic and easily confused with other variables. I'm going to start calling these things Network Accessible Variables or NAVs.)

NUM_NAV = number of NAVs implemented by this node (addressed 0..NUM_NAV-1)

and then for each NAV there would be a metadata sequence of 3 or more non-zero bytes terminated with a zero byte:

S code (1..255) chosen from S_LIST
V code (1..255) chosen from V_LIST
Child Id (1.255) grouping container
Zero or more A codes (each 1..255) chosen from A_LIST
0 byte terminating the metadata for this NAV

The optional attribute codes in the A_LIST could include concepts like "uncalibrated units" or "WaterTemperature" that modify the interpretation of the values reported or set for the associated NAV. The absence of a given A code in the metadata means "use the default interpretation" - like "reported in standard units" or "free air temperature".

Any A code that is not understood can be ignored.

Is that hard to describe, understand or implement? Is it a large amount of data to store, transport or remember?

---

Example metadata definition in the node:

uint8_t mymeta[] = {
     3, // number of NAVs
     S_TEMPERATURE, V_LEVEL, 1, 0,    // NAV 0 metadata
     S_HUMIDITY, V_PERCENTAGE, 1, 0,    // NAV 1 metadata
     S_BRIGHTNESS, V_LEVEL, 2, A_UNCALIBRATED, 0    // NAV 2 metadata
};

Here's another node with two DHT-11's (they can communicate for some distance over wire so one might be inside and one outside, or intake and exhaust of a swamp cooler or whatever)

uint8_t mymeta[] = {
     4,     // NUM_NAV
     S_TEMPERATURE, V_LEVEL, 1, 0,  // NAV 0, first DHT-11
     S_HUMIDITY, V_PERCENTAGE, 1, 0, // NAV 1
     S_TEMPERATURE, V_LEVEL, 2, 0,  // NAV 2, second DHT-11
     S_HUMIDITY, V_PERCENTAGE, 2, 0
};

Note that the "child id" groups the variables from each DHT-11 together (as each pair represent the highly related temp and humidity at the same location). This reflects the use of child-id as a container for related variables as @hek pointed out me to a few days ago.

---

I'm not proposing that this be the final version; it's just for illustration of concepts and how they can be very simple.

[Zeph]

@Damme said:

No I don't like the idea that the gateway should remember stuff about the sensors. If should be just a gateway.

I have thought some more about this, and I think there are multiple viable ways to organize a sensor network.

• I understand that your preferred model is that all of the smarts of the wireless sensor network per se exists in the nodes themselves - and the "API" is a contract between the end nodes and the HA controller software. Any gateway is just a transport mechanism, as transparent as possible. The "gateway" node is just a switchboard for packets,along with the minimal additional functionality to make things work (like address assignment).

• I have been conceptualizing the API contract more as being between the "gateway" node and the HA Controller; the gateway makes use of sensor nodes to accomplish its task. This expanded concept of a "gateway" would allow the gateway to do more controlling,, optimizing, translating, and enhancing.

As an example, the enhanced gateway could know all the variables available from each connected sensacturator node via saving some startup metadata. It could hide some version changes from the HA controller.

Suppose we had added "decimal point scaling" metadata, so that a node could report that its MHT-22 measured a temp of 29.1 degrees as the integer 291 with a metadata scaling factor of 1 (divide by 10 to get the standard units). Versus sending the value as a 4 byte float with the hint that it's meaningful to one decimal place. The point is it could report both of those to the HA controller as 29.1 degrees, doing the translation itself. This allows more details of the OTA protocol to be transparent to the HA Controller - information hiding. And that means that when there are several HA controller adapters (eg: plug-ins) in several languages from several authors, the WSN architecture can evolve without creating unneccessary dependencies within each HA controller plug-in. That gateway hides any aspect of the WSN that the HA controller plug-in doesn't logically need to understand.

Of course some level of major changes to the WSN's internal architecture or protocols will have to be reflected in the HA controllers' plug-in, but the more the enhanced gateway hides unnecessary details, the less often that will happen.

Another thing to node is that the enhanced gateway could answer some queries about a battery powered node from the HA controller plug-in, even between wakup times. Like metadata queries (what variables does it have, using which S and V codes), or the most recent value reported and when. Or it could queue up commands, to be passed to the node when it powers on and contacts the enhanced gateway.

Note that this "enhanced gateway" conceptualization can still include the same transparent switching functionality; it just leaves open the idea that there is also a translation layer between the switch and the WSN API as seen by the various HA controller plug-ins.

I find this model of an enhanced gateway very attractive. The details of what it does and doesn't do can evolve, but it has flexibility to do things which the nodes cannot. The main 'downside' is that when a WSN system gets large, the gateway may need to cost another $10 for more capable hardware, mainly for more RAM tho more program memory and speed usually comes with that too. But I'm going to predict that within a couple of years most gateways will be running on cheap ARM chips rather than AVRs anyway, especially for larger WSN networks. If your 30 nodes have cost you $800, upgrading the hub for $10 to make the programming and maintenance is going to be an easy choice. (or use a RPi directly for a bit more cost).

AND - there's also something to be said for keeping the gateway node's functionality minimal, as you envision it. I'm not discounting that.

The two approaches just trade off complexity and flexibility in different areas of the design. We sweep the complexity from one corner of the building to another, based on where we intuit that will bring the best results down the road.

By this time next year, I think we'll be seeing some of the way these appraoches pan out. At this time so much is about intuition of which path will turn out better, rather than solid deductive logic, so it's a judgment all rather than being "right" vs "wrong".

[Rasmus Eneman]

I really like this proposal.

What it would give us:

• Multiple variables of the same type (like two temperatures)
• The possibility for human description of a variable (core temp, air temp, engine temp)
• The possibility to describe the values even if they don't exist (Say if temperature didn't exist we could describe it as a LEVEL with the unit °C
• The possibility to describe the range of values (I.E. This temp may be between 0-25°C)

This doesn't need to be handled by the gateway, it could be moved to the controller which usually are a high-performance device. And if it isn't (doesn't have room to store the configuration) it may ask the node for it every time it receives a value from the node. In this configuration the gateway would just be a translator between nrf24 and USB, or nrf24 and Ethernet or whatever.

[Zeph]

@hek - I'm trying to follow your advice about communicating more simply

Here is how I understand the essential nature of the current system

[Zeph]

Here is another conceptual approach.

[Zeph]

In the current conceptual architecture, I am beginning to understand why there is resistance to the gateway ("dumb switching gateway") retaining or understanding anything about nodes other than their addresses. It is conceived as a simple nearly-transparent channel between the Nodes and the Plug-in.

• The plug-in knows a lot about nodes, but knows as little as possible about the gateway; at most the gateway might assist encoding as the plug-in tells it.
• A feature like binary OTA value encoding to the nodes would need to be handled by the plug-in and the nodes

In the alternative conceptual architecture, the plug-in communicates with the gateway ("smart gateway as proxy"), which in turn makes use of the nodes to provide bidirectonal services to the plug-in.

• The plug-in know about the WSN and its gateway as proxy, but knows as little as possible about how the gateway and nodes communicate.
• All the plug-in knows about nodes is what the WSN gateway wants or needs it to know
• A feature like binary OTA value encoding to the nodes only affects the gateway and the nodes, and is hidden from the plug-in

Neither approach is "right". They have tradeoffs. I'm currently suggesting that the alternative approach may prove advantageous in the course of time, but I have no proof of that.

---

An example implication (tradeoff):

• In the current approach, sending only binary values OTA to nodes would mean changing every controller plug-in to somehow know the "payload type" needed for each node, and use an enhanced serial format (etc) to tell the dumb gateway how to binary encode the value. Until all plug-ins are updated, we can't support the new all-binary format. No one team is likely to maintain all plug-ins or stay in sync.

• In the alternative approach, changing the OTA binary encoding would not require any changes to the various controller plug-ins, at that would be a private thing between the gateway and the node (the node would probably inform the gateway when it joined the network).

Suppose a new sketch for a particular node extended the calibrated range of a sensor (eg: sound, light..) or actuator and needed to use 2 bytes OTA rather than one. The controller plug-in would never need to know. The smart gateway would automatically adapt the next time the revised node joined the network, so its code and config would not need to change.

[Zeph]

About Plug-ins.

I see the weakest link in maintaining and developing the not-just-Vera MySensors system over years to come as being the per-controller plug-ins.

I think the core system can be made very solid - all programmed in C++ for known hardware where we can pretty much "control the vertical and horizontal" (sorry, old US TV reference). We have a lot of control, and the whole community is potentially available to review things like the 1.4 upgrade.

But the plug-ins fragment the community's resources. Writing or maintaining a plug-in requires not just general MySensors knowledge but also developer-level knowledge of a given HA controller and it's plug-in architecture. Plugins may often use a different language than C++, and run on different hardware and OS's. Inherently, only a fraction of the community is likely to have the knowledge and time (and incentive) to work on any given plug-in. So it's likely that some plug-ins are going to get stale, and be slow to get updates - compared to the core system.

Given that prospect, my tendency is to make the task of the plug-in relatively simple, and stable with as few changes as possible even as we evolve MySensors per se to new versions. The more we can hide logically unneccessary information about the nodes and network from the plug-ins, the less our future development of the core will require that every plug-in writer "keep up" with it. (Or the less we get fragmented by maintaining and backporting to old versions of MySensors in order to support the plug-ins that haven't been updated yet).

Just my judgement call, not a fact.

[daulagari]

@Zeph What I see on your architecture pictures are three/four levels/components:

• Nodes
• The Gateway
• Home Automation Controller X/MySensors Plug-in

I agree with @Damme that it is better to keep the gateway simple but also share your concern of the plug-ins taking up valuable community's resources.

Why not take the plug-in (partly) outside the Home Automation Controller X and make it a component between the Home Automation Controller X and the Gateway, let me call it "Home Automation Abstraction Layer" (HAAL) or "MySensors Abstraction Layer" (MAL).

The HAAL/MAL should:

• Have a clear defined interface towards the Home Automation Controller
• Do as much as possible that is common for the different Home Automation Controllers
• Be able to do all communication with the Gateway/Nodes
• Be preferably a plain-C module so that it can be linked with the Home Automation Controller or linked with the Gateway code

For the interface towards the Home Automation Controller we could have a look at existing interfaces like the OpenZwave interface, the ZigBee Home Automation standard or even a new defined interface as long as it is good documented and using it you do not have to known anything about the MySensor internals.

[Zeph]

@daulagari said:

Why not take the plug-in (partly) outside the Home Automation Controller X and make it a component between the Home Automation Controller X and the Gateway

That's kind of what I proposed a while back - with the provision that I suggested that one option could include running on the same hardware as the gateway.

The idea was indeed to abstract the MySensors wsn.

Thanks for the pointers to OpenZwave and Zigbee as possible interface candidates (or inspirations).. That will take more study.

Does anybody know how widely these interfaces are already implemented in Home Automation control software? For example, if we had an OpenZWave interface to the MySensors WSN, would we be able to easily connect may open and/or closed source HM contollers to it?