Porting MySensors to work with the RadioHead library

[hek]

@kolaf

Memory is usually not a big problem. When I turn off debug messages a normal actuator (e.g. DimmableLED) produces the following.

Sketch uses 11,868 bytes (36%) of program storage space. Maximum is 32,256 bytes.  
Global variables use 423 bytes (20%) of dynamic memory, leaving 1,625 bytes for local     variables. Maximum is 2,048 bytes.

But if you start combining sensors using external libraries such as OneWire, LEDDisplay I guess it adds up. The RadioHead library is very well written and would probably not add much overhead.

Inclusion mode is only handled between gateway and controller so it would not be affected.
Gateway would probably not need much changed. But the MyMessage class would need some tweaking lifting out the routing part.
RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

[marceltrapman]

@kolaf said:

Why are you guys so worried about RAM usage? Could you please explain to me what makes the library so RAM intensive? Maybe I don't understand all the things that the library does...

For 2 reasons:

1. I am one of those people that believe that small is better (== more efficient).
2. I want to combine sensors. Combining, for example, MySensors with a clock (RTClib) and lcd (ST7735) is already a big challenge...

[kolaf]

I definitely get your point. For reference, this is the result after compiling a simple clients using the RadioHead library on a moteino.

Sketch uses 10,486 bytes (32%) of program storage space. Maximum is 32,256 bytes.
Global variables use 980 bytes (47%) of dynamic memory, leaving 1,068 bytes for local variables. Maximum is 2,048 bytes.

So I agree that the library requires a significant amount of memory, and probably RAM as well. Still, for my use where I want to use quite simple sensors, which I read either directly or maybe using OneWire I believe the amount of space should be sufficient for my needs. As always, there is a trade-off between convenience and usability. I hope, though, that combining the two libraries would not equal the sum of the memory requirements since we get rid of another radio library :-)

I'm not dead set on doing this, so if you mean that this is not worth the trouble for the above reasons, I am happy to roll my own for my simple needs. I was just thinking that the amount of work I would spend on doing my own would be larger than the amount of work required to adapt MySensors to the other radio library.

[hek]

I would definitely be interested in your findings. This types of things would be interesting for the next (1.5/2.0?) version of the library where we also will look closer look at the protocol again.
I must prepare documentations and new example sketches for the 1.4 release so my time is somewhat limited right now. But I always have time for questions if you run into problems when doing your tests.

[kolaf]

@hek Great, I will give it a shot, then.

I have started playing around with the source code and I'm beginning to think that the only thing that we need to keep from the MySensors part of the library is the message format (mostly), the inclusion mode with automatic node ID assignment, as well as the API on both the sensor side and the server side.

Route discovery, acknowledgements, and CRC is handled by the radio library and can therefore be removed from MySensors.

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

1. I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?

2. Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Anyway, what I am proposing to do as a start is the following:

1. Change the packet format to remove the CRC.
2. Remove the logic that handles acknowledgements. Whether this is a smart thing to do depends on whether the acknowledgements in your library are "application level" or "link level" acknowledgements. If they are "application level" (i.e. end-to-end) acknowledgements, then I should better leave them in.
3. Remove any references to routing nodes (this is handled by the new radio library)
4. Remove everything related to learning, storing, and restoring routes. I think storing of routes is not supported in the new radio library, but is this a big deal? It allows for a more dynamic network, and other sensor nodes expected to shutdown often? It is possible to put the radio (at least some of the radios) in very low power modes so that it would not have to completely shut off the node. I guess route discovery can be a bit power consuming since it involves a broadcast that visits every node once.
5. Replace radio initialisation, send, and receive mechanisms to use the new library. The initial version of this will support just a single radio type (RFM69*) until I have figured out the solution to problem number two.
6. Modify the sleeping routines.
7. I will leave the automatic node ID discovery as is in the hopes that I will solve problem number one.

Broadly speaking this requires a small change in MyMessage, and significant changes in MySensors and possibly MyGateway. The upside is that the server side and library API should remain unchanged so that it can benefit from all the existing examples and server side scripts to get things integrated with home control systems.

Until now I have spent a few hours starting to rewrite the radio specific portions of the code. It is slow going since I have to read up on both the new radio library and how MySensors is supposed to work :-). A few updated examples on the library usage would be helpful, but not critical.

Comments are very welcome.

[hek]

First of all, I hope you are looking at the 1.4 version of the library
https://github.com/mysensors/Arduino/tree/development/libraries/MySensors
(the reason for asking is that you are talking about CRC which has been removed)

@kolaf said:

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?

The temporary id (while a sensor requests a new unique id) is 255. Therefore you shouldn't include more than one new sensor at the time. This is usually not a problem.

Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Ok, I've tried to simplify things for the end user as much as possible (often none-programmers). Have this in mind when you think of a setup with the drivers. The less boilerplate code in the examples the better :)

There are both end-to-end acks and node-node acks .The later uses the build in hardware acking of the nrf-chip.

[kolaf]

I am looking at the development branch of the github repository. My reference to the CRC probably comes from reading the documentation, not from reading the source code :-).

Based on your comments about the acknowledgements I assume that the code I see to handle ack messages relates to the end to end acknowledgements. In that case I will leave them in.

I definitely agree that you don't want boilerplate code. My goal is to reach a point where your radio choice is simply a parameter to the constructor. I might have to have the radio choice as a define in order to filter out unnecessary code, though.

As a disclaimer I should also mention that my C++ is a bit rusty, so I hope someone has time to review my changes at some point. Also, do not expect any significant results for many weeks to a few months.

[hek]

@kolaf

No worries. Don't tell me about rusty c++ knowledge. This is the first c++ program in 10+ years. But it´s slowly getting back... and why I left it for other languages long ago ;)

[kolaf]

Until I get everything set up and working I am committing stuff to a branch in my own repository. It is located here if anyone wants to see the progress :-).

https://github.com/kolaf/mysensors

Currently it is just a big mess...

[Zeph]

I've been lightly following the RF69 family of wireless sensors (jeenode (modified), moteino, anarduino) and noticed the appearance of the hardware agnostic radiohead library, so I'll be very interested in your progress.

MySensors and the underlying nRF library is fairly well optimized to take advantage of the nRF24l01+, eg: ESB. It will be interesting to see how the more hardware abstracted radiohead library works in this role.

If I understand correctly (always a question), the current MySensors library uses a hybrid between hardware link-level ack and end to end software ack. A packet from hub directly to leaf node uses ESB hardware for the ack. A packet from hub to repeater to leaf node involves "end to end" ack from the repeater to the hub, but ESB from the repeater to the leaf node. In other words, it's the next to last node in the sequence which handles network ack (and with a direct link with no relay, the "next to last node" is actually the originator). @hek and other more knowledgable folks can correct if this is wrong.

RadioHead would presumably do network acks from the leaf node back to the hub. (a more pure network ack, rather than a hybrid using ESB for the final link).

[kolaf]

As far as I have been able to figure out the RF69 radio uses link level acknowledgements, i.e. an acknowledgement per hop through the network. When using the RMesh manager which I am basing my port on (since this will handle everything related to routing for me) the initial send message function returns after the first of acknowledgement has been received, and the receive function of the final receiving end sends a link level acknowledgement.

When doing the port I am therefore leaving in the end-to-end acknowledgements and replacing anything related to the original RF24 acknowledgements.

Status update:
I have completed an initial rewrite of everything except the MQTTGateway code and I have reached a point where everything compiles correctly. This is with a hardcoded RF69 driver, which means that we do not get the radio agnostic version just yet. However, there is more to coding than just compiling, so I will now start the tedious task of getting this thing to actually run. I spent five minutes this morning testing it and nothing seemed to happen. I got one message from the gateway over the serial link looking something like this: "0;0;3;2;#@" where the two last symbols in the string were some weird ASCII characters. I was expecting to see some kind of initialisation message here as per the serial gateway source code.

I will continue testing and fixing over the next few days, I hope. Once I have everything running I will probably build a few sketches of my own before I start adding support for the other radio drivers.

[Damme]

@kolaf that sound to me as a sram overflow. There are ways to see in runtime how much free sram there is in runtime. I think there is a commented out version of it in the code..
last lines in mysensor.cpp

int MySensor::freeRam (void) {
  extern int __heap_start, *__brkval;
  int v;
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}

[kolaf]

@Damme Thanks for your input. I have had a chance to debug a bit more this evening and it turned out that I was not initialising the radio correctly.

I now have a version which is working with the RF 69 radio soldered to my Arduinos. Currently it should be easy to change the radio driver by changing the driver that is initialised and changing the appropriate include files.

I have achieved a successful communication from a sensor implementation to the serial gateway implementation where it requests a node ID. This is using the RHMesh manager implementation which handles a dynamic mesh network in the background.

I have not tested return communication yet because am frankly not quite sure what the correct command would be to assign a node ID. I guess all figure this out tomorrow, unless someone cares to give me an example?

With the debug enabled the gateway and simple sensor (simply sending a message every five seconds) amounts to the following when building..

Gateway:

Sketch uses 22,674 bytes (70%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,424 bytes (69%) of dynamic memory, leaving 624 bytes for local variables. Maximum is 2,048 bytes.

Sensor:

Sketch uses 19,854 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,141 bytes (55%) of dynamic memory, leaving 907 bytes for local variables. Maximum is 2,048 bytes.

[hek]

Great, sounds like your doing real progress.

[Zeph]

@kolaf
How big is the flash and RAM footprint of an equivalently limited nRF24L01+ implementation? The RadioHead library sounds heavier.

---

As I'm understanding you, a node is using the RadioHead RF69 mesh network to send a request to the hub for a MySensors node id. Doesn't the mesh network have its own equivalent node id? How are you integrating those two? (eg: does your node use id 255 temporarily within the RadioHead addressing, then change it a hub assigned ID, or...?)

I'm not really clear enough about the logical layering of the two networks, and how you are mixing and matching the layers.

[kolaf]

@Damme @Zeph I took a quick stab at dumping the free run by uncommenting the serial.write command in debugPrint. I didn't get anything, so I will look into it later.

Anyway, I was finally able to send a node ID response from the serial gateway back to the sensor, and is responded with a bunch of interesting data :-). This means that every thing seems to work, and I think the implementation is in a state where I can actually use it to build sensors. The challenge now is to have the self-discipline to generalise this so that others may use it to ;).

Based on all your comments and worries about flash and RAM space, I have been thinking that maybe my approach was not the best. I have ripped out everything related to routing so that I rely completely on the RadioHead for anything related to the link and network level. This means that we get support for quite a complex dynamic sensor networks at the cost of reduced available memory for useful stuff. May be a hybrid approach would have been better, relying on the reliable datagram manager from the library and using the mysensors routing functionality. Perhaps this would yield a smaller footprint, I don't know.

As for node addressing, there is a function in the new radio library that allows me to set the node address. I can therefore use the existing node address discovery to set everything up. I've included the message sequence I get when performing this initialisation below. The first line is the node assignment sent from the gateway to the sensor. There is some extra debug garbage in the printout which I had to add when trying to get this to work.

255;255;4;0;3;1
0;0;3;9;send: 0-255 s=255,c=3,t=4,pt=0,l=1,st=302:1
0;0;3;9;sent: 0-255 s=255,c=3,t=4,pt=0,l=1,st=295:1
0;0;3;9;Available
0;0;3;9;read: 1 s=255,c=3,t=11,pt=0,l=11:Dust Sensor
1;255;3;11;Dust Sensor
0;0;3;9;Available
0;0;3;9;read: 1 s=255,c=3,t=12,pt=0,l=3:1.1
1;255;3;12;1.1
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=0,t=0,pt=0,l=15:1.4b1 (18848a2)
1;3;0;0;1.4b1 (18848a2)
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=1,t=16,pt=2,l=2:1
1;3;1;16;1
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=1,t=16,pt=2,l=2:1
1;3;1;16;1

I'm wondering since we now have a working prototype version, perhaps I could be allowed to push my branch to your repository? Of course we don't merge anything yet, but it will allow others more easy access to it for further testing. By only changing a few lines to account for different radios the current version should support most of the moteino and anarduino variations with our different radio chipsets, I think.

[kolaf]

I'm getting a bunch of errors when I compile the mqtt gateway source related to the initial list of constants. The error messages look like this:

D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:14:15: error: variable 'broker' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char broker[] PROGMEM = MQTT_BROKER_PREFIX;
               ^
D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:16:12: error: variable 'S_0' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char S_0[] PROGMEM = "Temperature"; //V_TEMP
            ^
D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:17:12: error: variable 'S_1' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char S_1[] PROGMEM = "Humidity"; //V_HUM

My current solution is to just delete these two files since I only care about the serial gateway, but it would be good to have everything working :-).

[kolaf]

I am also happy to report that I just tried to compile the secret knock sensor without any difficulties :-)

Sketch uses 22,074 bytes (68%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,241 bytes (60%) of dynamic memory, leaving 807 bytes for local variables. Maximum is 2,048 bytes.

[Damme]

@kolaf What version of arduino IDE do you use? you should really use 1.5+
(But I'll fix so it works in older version)

[kolaf]

@Damme said:

@kolaf What version of arduino IDE do you use? you should really use 1.5+
(But I'll fix so it works in older version)

1.5.7