Another way of organizing variables

[Zeph]

Let me be clear about something. The proposal is largely about cleanly separating static metadata about a variable from the identity/addressing of that variable, not about saving bytes in the packet.

Compared to the the current proposed with S and V codes as both metadata and address/identity, the proposed alternative of transferring those static values during setup rather than with each packet does have the small advantage of saving two bytes. But that's not the motivation or main payoff.

Let me give an example of where it might pay off. Suppose that after a while we realized that it would be useful to pass brightness values which are not expressed in Lux because we have no way to calibrate a given sensor. The current S and V codes only allow brightness to be expressed in Lux. So we could add a new S code for non-Lux brightness. And a new S code for non-dB sound loudness. And a new code for any uncalibrated measurement which is just like the normal measurment but not available in standard units.

But suppose instead we wanted just one bit of semantics: the values reported for this sensor are not in the standard units, but they do measure the same thing. Then this bit of information could be equally associated with any variable of any type, without adding new codes.

We could add that bit to the OTA packet format so that the node would convey that bit of semantics every time it reports a brightness (is this calibrated to standard units or not?). Depending on where that bit was inserted into the packet format, all nodes might need to be modified to use the new packet layout, even if they are calibrated.

Or, with the separated metadata approach, a node could convey that new information just once during setup. The OTA Set command format doesn't change at all. And a node which uses only calibrated unit variables doesn't change at all (the default remains "calibrated"). A node that does have uncalibrated veriables is modified to pass that flag during setup. Done.

---

I see this as a need-to-know design issue, not unlike using protected member variables in a class is a design issue. You should design the users of a class so that they do not need to directly understand the implementation of the class, so that you can later change that implementation without cascading effects on all the users of that class.

Likewise the mechanism for sending the dynamic current value of a variable from point A to point B should be cleanly partitioned from the static metadata needed to describe and interpret that value, so that you can later modify or add to the static metadata without implications for the simple transporting of a dynamic current value. There is a clean need-to-know partitioning between the concepts of static metadata about a variable and dynamic current values, which I strongly believe will pay off in many ways as the design continues to evolve in months and years to come.

I believe that conflating static metadata with the addressing needed to identify a given variable is going to be an ongoing source of semantic and structural confusion. Any later tweaks to the metadata to address newly recognized shortcomings or add functionality will have cascading effects in parts of the protocol (eg variable value transport) which logically would not need to change.

At least that's my current design sense.

[Zeph]

Here's an example of the kind of metadata that might be passed from a node when it joins the network (using the approach advocated in this thread).

(The term "variable" is a little too generic and easily confused with other variables. I'm going to start calling these things Network Accessible Variables or NAVs.)

NUM_NAV = number of NAVs implemented by this node (addressed 0..NUM_NAV-1)

and then for each NAV there would be a metadata sequence of 3 or more non-zero bytes terminated with a zero byte:

S code (1..255) chosen from S_LIST
V code (1..255) chosen from V_LIST
Child Id (1.255) grouping container
Zero or more A codes (each 1..255) chosen from A_LIST
0 byte terminating the metadata for this NAV

The optional attribute codes in the A_LIST could include concepts like "uncalibrated units" or "WaterTemperature" that modify the interpretation of the values reported or set for the associated NAV. The absence of a given A code in the metadata means "use the default interpretation" - like "reported in standard units" or "free air temperature".

Any A code that is not understood can be ignored.

Is that hard to describe, understand or implement? Is it a large amount of data to store, transport or remember?

---

Example metadata definition in the node:

uint8_t mymeta[] = {
     3, // number of NAVs
     S_TEMPERATURE, V_LEVEL, 1, 0,    // NAV 0 metadata
     S_HUMIDITY, V_PERCENTAGE, 1, 0,    // NAV 1 metadata
     S_BRIGHTNESS, V_LEVEL, 2, A_UNCALIBRATED, 0    // NAV 2 metadata
};

Here's another node with two DHT-11's (they can communicate for some distance over wire so one might be inside and one outside, or intake and exhaust of a swamp cooler or whatever)

uint8_t mymeta[] = {
     4,     // NUM_NAV
     S_TEMPERATURE, V_LEVEL, 1, 0,  // NAV 0, first DHT-11
     S_HUMIDITY, V_PERCENTAGE, 1, 0, // NAV 1
     S_TEMPERATURE, V_LEVEL, 2, 0,  // NAV 2, second DHT-11
     S_HUMIDITY, V_PERCENTAGE, 2, 0
};

Note that the "child id" groups the variables from each DHT-11 together (as each pair represent the highly related temp and humidity at the same location). This reflects the use of child-id as a container for related variables as @hek pointed out me to a few days ago.

---

I'm not proposing that this be the final version; it's just for illustration of concepts and how they can be very simple.

[Zeph]

@Damme said:

No I don't like the idea that the gateway should remember stuff about the sensors. If should be just a gateway.

I have thought some more about this, and I think there are multiple viable ways to organize a sensor network.

• I understand that your preferred model is that all of the smarts of the wireless sensor network per se exists in the nodes themselves - and the "API" is a contract between the end nodes and the HA controller software. Any gateway is just a transport mechanism, as transparent as possible. The "gateway" node is just a switchboard for packets,along with the minimal additional functionality to make things work (like address assignment).

• I have been conceptualizing the API contract more as being between the "gateway" node and the HA Controller; the gateway makes use of sensor nodes to accomplish its task. This expanded concept of a "gateway" would allow the gateway to do more controlling,, optimizing, translating, and enhancing.

As an example, the enhanced gateway could know all the variables available from each connected sensacturator node via saving some startup metadata. It could hide some version changes from the HA controller.

Suppose we had added "decimal point scaling" metadata, so that a node could report that its MHT-22 measured a temp of 29.1 degrees as the integer 291 with a metadata scaling factor of 1 (divide by 10 to get the standard units). Versus sending the value as a 4 byte float with the hint that it's meaningful to one decimal place. The point is it could report both of those to the HA controller as 29.1 degrees, doing the translation itself. This allows more details of the OTA protocol to be transparent to the HA Controller - information hiding. And that means that when there are several HA controller adapters (eg: plug-ins) in several languages from several authors, the WSN architecture can evolve without creating unneccessary dependencies within each HA controller plug-in. That gateway hides any aspect of the WSN that the HA controller plug-in doesn't logically need to understand.

Of course some level of major changes to the WSN's internal architecture or protocols will have to be reflected in the HA controllers' plug-in, but the more the enhanced gateway hides unnecessary details, the less often that will happen.

Another thing to node is that the enhanced gateway could answer some queries about a battery powered node from the HA controller plug-in, even between wakup times. Like metadata queries (what variables does it have, using which S and V codes), or the most recent value reported and when. Or it could queue up commands, to be passed to the node when it powers on and contacts the enhanced gateway.

Note that this "enhanced gateway" conceptualization can still include the same transparent switching functionality; it just leaves open the idea that there is also a translation layer between the switch and the WSN API as seen by the various HA controller plug-ins.

I find this model of an enhanced gateway very attractive. The details of what it does and doesn't do can evolve, but it has flexibility to do things which the nodes cannot. The main 'downside' is that when a WSN system gets large, the gateway may need to cost another $10 for more capable hardware, mainly for more RAM tho more program memory and speed usually comes with that too. But I'm going to predict that within a couple of years most gateways will be running on cheap ARM chips rather than AVRs anyway, especially for larger WSN networks. If your 30 nodes have cost you $800, upgrading the hub for $10 to make the programming and maintenance is going to be an easy choice. (or use a RPi directly for a bit more cost).

AND - there's also something to be said for keeping the gateway node's functionality minimal, as you envision it. I'm not discounting that.

The two approaches just trade off complexity and flexibility in different areas of the design. We sweep the complexity from one corner of the building to another, based on where we intuit that will bring the best results down the road.

By this time next year, I think we'll be seeing some of the way these appraoches pan out. At this time so much is about intuition of which path will turn out better, rather than solid deductive logic, so it's a judgment all rather than being "right" vs "wrong".

[Rasmus Eneman]

I really like this proposal.

What it would give us:

• Multiple variables of the same type (like two temperatures)
• The possibility for human description of a variable (core temp, air temp, engine temp)
• The possibility to describe the values even if they don't exist (Say if temperature didn't exist we could describe it as a LEVEL with the unit °C
• The possibility to describe the range of values (I.E. This temp may be between 0-25°C)

This doesn't need to be handled by the gateway, it could be moved to the controller which usually are a high-performance device. And if it isn't (doesn't have room to store the configuration) it may ask the node for it every time it receives a value from the node. In this configuration the gateway would just be a translator between nrf24 and USB, or nrf24 and Ethernet or whatever.

[Zeph]

@hek - I'm trying to follow your advice about communicating more simply

Here is how I understand the essential nature of the current system

[Zeph]

Here is another conceptual approach.

[Zeph]

In the current conceptual architecture, I am beginning to understand why there is resistance to the gateway ("dumb switching gateway") retaining or understanding anything about nodes other than their addresses. It is conceived as a simple nearly-transparent channel between the Nodes and the Plug-in.

• The plug-in knows a lot about nodes, but knows as little as possible about the gateway; at most the gateway might assist encoding as the plug-in tells it.
• A feature like binary OTA value encoding to the nodes would need to be handled by the plug-in and the nodes

In the alternative conceptual architecture, the plug-in communicates with the gateway ("smart gateway as proxy"), which in turn makes use of the nodes to provide bidirectonal services to the plug-in.

• The plug-in know about the WSN and its gateway as proxy, but knows as little as possible about how the gateway and nodes communicate.
• All the plug-in knows about nodes is what the WSN gateway wants or needs it to know
• A feature like binary OTA value encoding to the nodes only affects the gateway and the nodes, and is hidden from the plug-in

Neither approach is "right". They have tradeoffs. I'm currently suggesting that the alternative approach may prove advantageous in the course of time, but I have no proof of that.

---

An example implication (tradeoff):

• In the current approach, sending only binary values OTA to nodes would mean changing every controller plug-in to somehow know the "payload type" needed for each node, and use an enhanced serial format (etc) to tell the dumb gateway how to binary encode the value. Until all plug-ins are updated, we can't support the new all-binary format. No one team is likely to maintain all plug-ins or stay in sync.

• In the alternative approach, changing the OTA binary encoding would not require any changes to the various controller plug-ins, at that would be a private thing between the gateway and the node (the node would probably inform the gateway when it joined the network).

Suppose a new sketch for a particular node extended the calibrated range of a sensor (eg: sound, light..) or actuator and needed to use 2 bytes OTA rather than one. The controller plug-in would never need to know. The smart gateway would automatically adapt the next time the revised node joined the network, so its code and config would not need to change.

[Zeph]

About Plug-ins.

I see the weakest link in maintaining and developing the not-just-Vera MySensors system over years to come as being the per-controller plug-ins.

I think the core system can be made very solid - all programmed in C++ for known hardware where we can pretty much "control the vertical and horizontal" (sorry, old US TV reference). We have a lot of control, and the whole community is potentially available to review things like the 1.4 upgrade.

But the plug-ins fragment the community's resources. Writing or maintaining a plug-in requires not just general MySensors knowledge but also developer-level knowledge of a given HA controller and it's plug-in architecture. Plugins may often use a different language than C++, and run on different hardware and OS's. Inherently, only a fraction of the community is likely to have the knowledge and time (and incentive) to work on any given plug-in. So it's likely that some plug-ins are going to get stale, and be slow to get updates - compared to the core system.

Given that prospect, my tendency is to make the task of the plug-in relatively simple, and stable with as few changes as possible even as we evolve MySensors per se to new versions. The more we can hide logically unneccessary information about the nodes and network from the plug-ins, the less our future development of the core will require that every plug-in writer "keep up" with it. (Or the less we get fragmented by maintaining and backporting to old versions of MySensors in order to support the plug-ins that haven't been updated yet).

Just my judgement call, not a fact.

[daulagari]

@Zeph What I see on your architecture pictures are three/four levels/components:

• Nodes
• The Gateway
• Home Automation Controller X/MySensors Plug-in

I agree with @Damme that it is better to keep the gateway simple but also share your concern of the plug-ins taking up valuable community's resources.

Why not take the plug-in (partly) outside the Home Automation Controller X and make it a component between the Home Automation Controller X and the Gateway, let me call it "Home Automation Abstraction Layer" (HAAL) or "MySensors Abstraction Layer" (MAL).

The HAAL/MAL should:

• Have a clear defined interface towards the Home Automation Controller
• Do as much as possible that is common for the different Home Automation Controllers
• Be able to do all communication with the Gateway/Nodes
• Be preferably a plain-C module so that it can be linked with the Home Automation Controller or linked with the Gateway code

For the interface towards the Home Automation Controller we could have a look at existing interfaces like the OpenZwave interface, the ZigBee Home Automation standard or even a new defined interface as long as it is good documented and using it you do not have to known anything about the MySensor internals.

[Zeph]

@daulagari said:

Why not take the plug-in (partly) outside the Home Automation Controller X and make it a component between the Home Automation Controller X and the Gateway

That's kind of what I proposed a while back - with the provision that I suggested that one option could include running on the same hardware as the gateway.

The idea was indeed to abstract the MySensors wsn.

Thanks for the pointers to OpenZwave and Zigbee as possible interface candidates (or inspirations).. That will take more study.

Does anybody know how widely these interfaces are already implemented in Home Automation control software? For example, if we had an OpenZWave interface to the MySensors WSN, would we be able to easily connect may open and/or closed source HM contollers to it?