Porting MySensors to work with the RadioHead library

kolaf

Hi guys,

I have received my Moteinos and anarduinos with RFM69HW radios. The anarduino webpage recommends a radio library called RadioHead which supports multiple radio tips and provides a common API for communication. http://www.airspayce.com/mikem/arduino/RadioHead/

It even supports quite complex things such as mesh networks. It currently supports the following radios:

RH_RF22 Works with Hope-RF RF22B and RF23B based transceivers, and compatible chips and modules, including the RFM22B transceiver module. Supports GFSK, FSK and OOK. Access to other chip features such as on-chip temperature measurement, analog-digital converter, transmitter power control etc is also provided.
RH_RF69 Works with Hope-RF RF69B based radio modules, such as the RFM69 module, (as used on the excellent Moteino and Moteino-USB boards from LowPowerLab http://lowpowerlab.com/moteino/) and compatible chips and modules such as RFM69W, RFM69HW, RFM69CW, RFM69HCW (Semtech SX1231, SX1231H). Also works with Anarduino MiniWireless -CW and -HW boards http://www.anarduino.com/miniwireless/ including the marvellous high powered MinWireless-HW (with 20dBm output for excelent range). Supports GFSK, FSK.
RH_NRF24 Works with Nordic nRF24 based 2.4GHz radio modules, such as nRF24L01 and others. Also works with Hope-RF RFM73 and compatible devices (such as BK2423). nRF24L01 and RFM73 can interoperate with each other.
RH_NRF905 Works with Nordic nRF905 based 433/868/915 MHz radio modules.
RH_RF95 Works with Semtech SX1276/77/78 and HopeRF RFM95/96/97/98 and other similar LoRa capable radios. Supports Long Range (LoRa) with spread spectrum frequency hopping, large payloads etc. FSK/GFSK/OOK modes are not (yet) supported.
RH_ASK Works with a range of inexpensive ASK (amplitude shift keying) RF transceivers such as RX-B1 (also known as ST-RX04-ASK) receiver; TX-C1 transmitter and DR3100 transceiver; FS1000A/XY-MK-5V transceiver; HopeRF RFM83C / RFM85. Supports ASK (OOK).
RH_Serial Works with RS232, RS422, RS485, RS488 and other point-to-point and multidropped serial connections, or with TTL serial UARTs such as those on Arduino and many other processors, or with data radios with a serial port interface. RH_Serial23 provides packetization and error detection over any hardware or virtual serial connection.
RH_TCP For use with simulated sketches compiled and running on Linux. Works with tools/etherSimulator.pl to pass messages between simulated sketches, allowing testing of Manager classes on Linux and without need for real radios or other transport hardware.
``

Seeing as the radio that is currently supported by MySensors is also supported by RadioHead I was wondering if anyone was interested in helping me porting MySensors to work with the RadioHead library. This will greatly increase the versatility of the MySensors library and magically allow it to work with a range of different radios :-).

I have briefly looked at the MySensors source code and it looks like most of the changes have to be done in MySensors.h and .cpp files.

I am willing to take a stab at it, but as everyone else I am limited on time. Still, it looks like it should not be a very difficult task. As discussed in another thread, it should be sufficient either to subclass the new radio library and replace a bunch of function calls, or drop the subclassing altogether and simply use the radio library as a regular included library.

The obvious difficulty comes soon as MySensors requires a function that is not available in the new radio library. However, seeing as MySensors is a pretty high-level library, and the API support from radioHead seems quite good, I suspect it should be possible to get around any such problems.

Anyway, I'm thinking of creating a branch of the library and spending a few hours trying to port it and get some basic functionality to work. Is anyone else is any value in this it would be great to have some help :-)

hek

Great. Let me know if you need help understanding protocols and code.

marceltrapman

@kolaf Sounds good but since a couple of my ideas already hit the ceiling of ATMEGA memory do you have any idea of what the additional overhead will be (if any)?

kolaf

Honestly, no, I have no idea. What kind of ideas are you talking about? I think anything related to the networking can be handled by the radio library, and the rest is simply a messaging protocol, correct? I see that you have included a bunch of other libraries in the repository, but I assume that these are just to keep everything in one place when developing, i.e. you don't have to include everything, just the things you need.

I must admit that I have not spent much time looking at your library so I might be jumping in too deep :-)

kolaf

I have spent a few more minutes looking at the MySensors source code and I think the following needs to be done if I have understood things correctly.

We need to change SetUpRadio,SendRoute, Process, and internalSleep to use different library commands from the new radio library. In addition, if we use the mesh mode from RadioHead I think we can get rid of both the repeater mode and the child routes since this will be handled by the radio library.

This is in MySensors.cpp. I assume something similar has to be done in the Gateway code. I'm not yet sure what inclusion mode is, but assume this has something to do with learning the routes, in which case we can get rid of this as well.

Is there any specific reason why the library is a subclass of the radio library? Do you access many of the internal systems of the radio library, or is it just for convenience? I'm not very familiar with the internals of the RadioHead library, so I think it would be easiest if we could simply include the library as normal instead of doing a sub-class. What do you think?

Damme

I think we need to think about choosing a bitter atmega for this.. Its a bit too bad that atmega havn't made a atmega628p which is pin compatible with 328 but with 64KB of program storage instead.. :) so the next up is 644 which arduino has no boards for. but there is sanguino.cc - https://code.google.com/p/sanguino/ though.

Next up would be Arduino Mega 2560 with what it feels like 'Unlimited ram!'
They are not That expensive today since china made its own ch340 usb - serial chip (Why this was the expensive component blows my mind :P) ATmega2560-16AU CH340G MEGA 256 you can find for ~10EUR / 13.50USD..

kolaf

Why are you guys so worried about RAM usage? Could you please explain to me what makes the library so RAM intensive? Maybe I don't understand all the things that the library does...

hek

@kolaf

Memory is usually not a big problem. When I turn off debug messages a normal actuator (e.g. DimmableLED) produces the following.

Sketch uses 11,868 bytes (36%) of program storage space. Maximum is 32,256 bytes.  
Global variables use 423 bytes (20%) of dynamic memory, leaving 1,625 bytes for local     variables. Maximum is 2,048 bytes.

But if you start combining sensors using external libraries such as OneWire, LEDDisplay I guess it adds up. The RadioHead library is very well written and would probably not add much overhead.

Inclusion mode is only handled between gateway and controller so it would not be affected.
Gateway would probably not need much changed. But the MyMessage class would need some tweaking lifting out the routing part.
RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

marceltrapman

@kolaf said:

Why are you guys so worried about RAM usage? Could you please explain to me what makes the library so RAM intensive? Maybe I don't understand all the things that the library does...

For 2 reasons:

I am one of those people that believe that small is better (== more efficient).
I want to combine sensors. Combining, for example, MySensors with a clock (RTClib) and lcd (ST7735) is already a big challenge...

kolaf

I definitely get your point. For reference, this is the result after compiling a simple clients using the RadioHead library on a moteino.

Sketch uses 10,486 bytes (32%) of program storage space. Maximum is 32,256 bytes.
Global variables use 980 bytes (47%) of dynamic memory, leaving 1,068 bytes for local variables. Maximum is 2,048 bytes.

So I agree that the library requires a significant amount of memory, and probably RAM as well. Still, for my use where I want to use quite simple sensors, which I read either directly or maybe using OneWire I believe the amount of space should be sufficient for my needs. As always, there is a trade-off between convenience and usability. I hope, though, that combining the two libraries would not equal the sum of the memory requirements since we get rid of another radio library :-)

I'm not dead set on doing this, so if you mean that this is not worth the trouble for the above reasons, I am happy to roll my own for my simple needs. I was just thinking that the amount of work I would spend on doing my own would be larger than the amount of work required to adapt MySensors to the other radio library.

hek

I would definitely be interested in your findings. This types of things would be interesting for the next (1.5/2.0?) version of the library where we also will look closer look at the protocol again.
I must prepare documentations and new example sketches for the 1.4 release so my time is somewhat limited right now. But I always have time for questions if you run into problems when doing your tests.

kolaf

@hek Great, I will give it a shot, then.

I have started playing around with the source code and I'm beginning to think that the only thing that we need to keep from the MySensors part of the library is the message format (mostly), the inclusion mode with automatic node ID assignment, as well as the API on both the sensor side and the server side.

Route discovery, acknowledgements, and CRC is handled by the radio library and can therefore be removed from MySensors.

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?
Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Anyway, what I am proposing to do as a start is the following:

Change the packet format to remove the CRC.
Remove the logic that handles acknowledgements. Whether this is a smart thing to do depends on whether the acknowledgements in your library are "application level" or "link level" acknowledgements. If they are "application level" (i.e. end-to-end) acknowledgements, then I should better leave them in.
Remove any references to routing nodes (this is handled by the new radio library)
Remove everything related to learning, storing, and restoring routes. I think storing of routes is not supported in the new radio library, but is this a big deal? It allows for a more dynamic network, and other sensor nodes expected to shutdown often? It is possible to put the radio (at least some of the radios) in very low power modes so that it would not have to completely shut off the node. I guess route discovery can be a bit power consuming since it involves a broadcast that visits every node once.
Replace radio initialisation, send, and receive mechanisms to use the new library. The initial version of this will support just a single radio type (RFM69*) until I have figured out the solution to problem number two.
Modify the sleeping routines.
I will leave the automatic node ID discovery as is in the hopes that I will solve problem number one.

Broadly speaking this requires a small change in MyMessage, and significant changes in MySensors and possibly MyGateway. The upside is that the server side and library API should remain unchanged so that it can benefit from all the existing examples and server side scripts to get things integrated with home control systems.

Until now I have spent a few hours starting to rewrite the radio specific portions of the code. It is slow going since I have to read up on both the new radio library and how MySensors is supposed to work :-). A few updated examples on the library usage would be helpful, but not critical.

Comments are very welcome.

hek

First of all, I hope you are looking at the 1.4 version of the library
https://github.com/mysensors/Arduino/tree/development/libraries/MySensors
(the reason for asking is that you are talking about CRC which has been removed)

@kolaf said:

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?

The temporary id (while a sensor requests a new unique id) is 255. Therefore you shouldn't include more than one new sensor at the time. This is usually not a problem.

Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Ok, I've tried to simplify things for the end user as much as possible (often none-programmers). Have this in mind when you think of a setup with the drivers. The less boilerplate code in the examples the better :)

There are both end-to-end acks and node-node acks .The later uses the build in hardware acking of the nrf-chip.

kolaf

I am looking at the development branch of the github repository. My reference to the CRC probably comes from reading the documentation, not from reading the source code :-).

Based on your comments about the acknowledgements I assume that the code I see to handle ack messages relates to the end to end acknowledgements. In that case I will leave them in.

I definitely agree that you don't want boilerplate code. My goal is to reach a point where your radio choice is simply a parameter to the constructor. I might have to have the radio choice as a define in order to filter out unnecessary code, though.

As a disclaimer I should also mention that my C++ is a bit rusty, so I hope someone has time to review my changes at some point. Also, do not expect any significant results for many weeks to a few months.

hek

@kolaf

No worries. Don't tell me about rusty c++ knowledge. This is the first c++ program in 10+ years. But it´s slowly getting back... and why I left it for other languages long ago ;)

kolaf

Until I get everything set up and working I am committing stuff to a branch in my own repository. It is located here if anyone wants to see the progress :-).

https://github.com/kolaf/mysensors

Currently it is just a big mess...

Zeph

I've been lightly following the RF69 family of wireless sensors (jeenode (modified), moteino, anarduino) and noticed the appearance of the hardware agnostic radiohead library, so I'll be very interested in your progress.

MySensors and the underlying nRF library is fairly well optimized to take advantage of the nRF24l01+, eg: ESB. It will be interesting to see how the more hardware abstracted radiohead library works in this role.

If I understand correctly (always a question), the current MySensors library uses a hybrid between hardware link-level ack and end to end software ack. A packet from hub directly to leaf node uses ESB hardware for the ack. A packet from hub to repeater to leaf node involves "end to end" ack from the repeater to the hub, but ESB from the repeater to the leaf node. In other words, it's the next to last node in the sequence which handles network ack (and with a direct link with no relay, the "next to last node" is actually the originator). @hek and other more knowledgable folks can correct if this is wrong.

RadioHead would presumably do network acks from the leaf node back to the hub. (a more pure network ack, rather than a hybrid using ESB for the final link).

kolaf

As far as I have been able to figure out the RF69 radio uses link level acknowledgements, i.e. an acknowledgement per hop through the network. When using the RMesh manager which I am basing my port on (since this will handle everything related to routing for me) the initial send message function returns after the first of acknowledgement has been received, and the receive function of the final receiving end sends a link level acknowledgement.

When doing the port I am therefore leaving in the end-to-end acknowledgements and replacing anything related to the original RF24 acknowledgements.

Status update:
I have completed an initial rewrite of everything except the MQTTGateway code and I have reached a point where everything compiles correctly. This is with a hardcoded RF69 driver, which means that we do not get the radio agnostic version just yet. However, there is more to coding than just compiling, so I will now start the tedious task of getting this thing to actually run. I spent five minutes this morning testing it and nothing seemed to happen. I got one message from the gateway over the serial link looking something like this: "0;0;3;2;#@" where the two last symbols in the string were some weird ASCII characters. I was expecting to see some kind of initialisation message here as per the serial gateway source code.

I will continue testing and fixing over the next few days, I hope. Once I have everything running I will probably build a few sketches of my own before I start adding support for the other radio drivers.

Damme

@kolaf that sound to me as a sram overflow. There are ways to see in runtime how much free sram there is in runtime. I think there is a commented out version of it in the code..
last lines in mysensor.cpp

int MySensor::freeRam (void) {
  extern int __heap_start, *__brkval;
  int v;
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}

kolaf

@Damme Thanks for your input. I have had a chance to debug a bit more this evening and it turned out that I was not initialising the radio correctly.

I now have a version which is working with the RF 69 radio soldered to my Arduinos. Currently it should be easy to change the radio driver by changing the driver that is initialised and changing the appropriate include files.

I have achieved a successful communication from a sensor implementation to the serial gateway implementation where it requests a node ID. This is using the RHMesh manager implementation which handles a dynamic mesh network in the background.

I have not tested return communication yet because am frankly not quite sure what the correct command would be to assign a node ID. I guess all figure this out tomorrow, unless someone cares to give me an example?

With the debug enabled the gateway and simple sensor (simply sending a message every five seconds) amounts to the following when building..

Gateway:

Sketch uses 22,674 bytes (70%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,424 bytes (69%) of dynamic memory, leaving 624 bytes for local variables. Maximum is 2,048 bytes.

Sensor:

Sketch uses 19,854 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,141 bytes (55%) of dynamic memory, leaving 907 bytes for local variables. Maximum is 2,048 bytes.

Porting MySensors to work with the RadioHead library

14

11.7k

11.2k

113.0k