Porting MySensors to work with the RadioHead library

[hek]

I would definitely be interested in your findings. This types of things would be interesting for the next (1.5/2.0?) version of the library where we also will look closer look at the protocol again.
I must prepare documentations and new example sketches for the 1.4 release so my time is somewhat limited right now. But I always have time for questions if you run into problems when doing your tests.

[kolaf]

@hek Great, I will give it a shot, then.

I have started playing around with the source code and I'm beginning to think that the only thing that we need to keep from the MySensors part of the library is the message format (mostly), the inclusion mode with automatic node ID assignment, as well as the API on both the sensor side and the server side.

Route discovery, acknowledgements, and CRC is handled by the radio library and can therefore be removed from MySensors.

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

1. I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?

2. Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Anyway, what I am proposing to do as a start is the following:

1. Change the packet format to remove the CRC.
2. Remove the logic that handles acknowledgements. Whether this is a smart thing to do depends on whether the acknowledgements in your library are "application level" or "link level" acknowledgements. If they are "application level" (i.e. end-to-end) acknowledgements, then I should better leave them in.
3. Remove any references to routing nodes (this is handled by the new radio library)
4. Remove everything related to learning, storing, and restoring routes. I think storing of routes is not supported in the new radio library, but is this a big deal? It allows for a more dynamic network, and other sensor nodes expected to shutdown often? It is possible to put the radio (at least some of the radios) in very low power modes so that it would not have to completely shut off the node. I guess route discovery can be a bit power consuming since it involves a broadcast that visits every node once.
5. Replace radio initialisation, send, and receive mechanisms to use the new library. The initial version of this will support just a single radio type (RFM69*) until I have figured out the solution to problem number two.
6. Modify the sleeping routines.
7. I will leave the automatic node ID discovery as is in the hopes that I will solve problem number one.

Broadly speaking this requires a small change in MyMessage, and significant changes in MySensors and possibly MyGateway. The upside is that the server side and library API should remain unchanged so that it can benefit from all the existing examples and server side scripts to get things integrated with home control systems.

Until now I have spent a few hours starting to rewrite the radio specific portions of the code. It is slow going since I have to read up on both the new radio library and how MySensors is supposed to work :-). A few updated examples on the library usage would be helpful, but not critical.

Comments are very welcome.

[hek]

First of all, I hope you are looking at the 1.4 version of the library
https://github.com/mysensors/Arduino/tree/development/libraries/MySensors
(the reason for asking is that you are talking about CRC which has been removed)

@kolaf said:

Most of this looks to be pretty straightforward, but I am faced with two immediate challenges.

I'm not sure whether the new radio library handles replacing node IDs. It looks like you set a default node ID upon initialisation, and then later replace this when you get a response back from the server with an updated ID. I assume the node ID used in the MySensors is identical to the one used by the radio library for addressing, routing, et cetera. Related to this, how do you know on the Gateway/server side which device you are communicating with? If the ID is dynamically assigned you will just have to infer it from the sensor type which seems quite unreliable. How is this handled?

The temporary id (while a sensor requests a new unique id) is 255. Therefore you shouldn't include more than one new sensor at the time. This is usually not a problem.

Each radio type supported by the new library requires a different driver, which in turn requires different include statements. I must admit that I am not sure how to solve this in a dynamic fashion. There is a global driver class from which all the drivers are derived, so setting up the pointer should be relatively easy. The difficulty comes when you tried to reduce the size of the application by including only the relevant headers. There are also some driver specific options in the initialisation such as setting output power, frequency, and modem modulation. I think most of this is exposed through a unified API, but there might be some peculiarities I have to deal with.

Ok, I've tried to simplify things for the end user as much as possible (often none-programmers). Have this in mind when you think of a setup with the drivers. The less boilerplate code in the examples the better

There are both end-to-end acks and node-node acks .The later uses the build in hardware acking of the nrf-chip.

[kolaf]

I am looking at the development branch of the github repository. My reference to the CRC probably comes from reading the documentation, not from reading the source code :-).

Based on your comments about the acknowledgements I assume that the code I see to handle ack messages relates to the end to end acknowledgements. In that case I will leave them in.

I definitely agree that you don't want boilerplate code. My goal is to reach a point where your radio choice is simply a parameter to the constructor. I might have to have the radio choice as a define in order to filter out unnecessary code, though.

As a disclaimer I should also mention that my C++ is a bit rusty, so I hope someone has time to review my changes at some point. Also, do not expect any significant results for many weeks to a few months.

[hek]

@kolaf

No worries. Don't tell me about rusty c++ knowledge. This is the first c++ program in 10+ years. But it´s slowly getting back... and why I left it for other languages long ago

[kolaf]

Until I get everything set up and working I am committing stuff to a branch in my own repository. It is located here if anyone wants to see the progress :-).

https://github.com/kolaf/mysensors

Currently it is just a big mess...

[Zeph]

I've been lightly following the RF69 family of wireless sensors (jeenode (modified), moteino, anarduino) and noticed the appearance of the hardware agnostic radiohead library, so I'll be very interested in your progress.

MySensors and the underlying nRF library is fairly well optimized to take advantage of the nRF24l01+, eg: ESB. It will be interesting to see how the more hardware abstracted radiohead library works in this role.

If I understand correctly (always a question), the current MySensors library uses a hybrid between hardware link-level ack and end to end software ack. A packet from hub directly to leaf node uses ESB hardware for the ack. A packet from hub to repeater to leaf node involves "end to end" ack from the repeater to the hub, but ESB from the repeater to the leaf node. In other words, it's the next to last node in the sequence which handles network ack (and with a direct link with no relay, the "next to last node" is actually the originator). @hek and other more knowledgable folks can correct if this is wrong.

RadioHead would presumably do network acks from the leaf node back to the hub. (a more pure network ack, rather than a hybrid using ESB for the final link).

[kolaf]

As far as I have been able to figure out the RF69 radio uses link level acknowledgements, i.e. an acknowledgement per hop through the network. When using the RMesh manager which I am basing my port on (since this will handle everything related to routing for me) the initial send message function returns after the first of acknowledgement has been received, and the receive function of the final receiving end sends a link level acknowledgement.

When doing the port I am therefore leaving in the end-to-end acknowledgements and replacing anything related to the original RF24 acknowledgements.

Status update:
I have completed an initial rewrite of everything except the MQTTGateway code and I have reached a point where everything compiles correctly. This is with a hardcoded RF69 driver, which means that we do not get the radio agnostic version just yet. However, there is more to coding than just compiling, so I will now start the tedious task of getting this thing to actually run. I spent five minutes this morning testing it and nothing seemed to happen. I got one message from the gateway over the serial link looking something like this: "0;0;3;2;#@" where the two last symbols in the string were some weird ASCII characters. I was expecting to see some kind of initialisation message here as per the serial gateway source code.

I will continue testing and fixing over the next few days, I hope. Once I have everything running I will probably build a few sketches of my own before I start adding support for the other radio drivers.

[Damme]

@kolaf that sound to me as a sram overflow. There are ways to see in runtime how much free sram there is in runtime. I think there is a commented out version of it in the code..
last lines in mysensor.cpp

int MySensor::freeRam (void) {
  extern int __heap_start, *__brkval;
  int v;
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}

[kolaf]

@Damme Thanks for your input. I have had a chance to debug a bit more this evening and it turned out that I was not initialising the radio correctly.

I now have a version which is working with the RF 69 radio soldered to my Arduinos. Currently it should be easy to change the radio driver by changing the driver that is initialised and changing the appropriate include files.

I have achieved a successful communication from a sensor implementation to the serial gateway implementation where it requests a node ID. This is using the RHMesh manager implementation which handles a dynamic mesh network in the background.

I have not tested return communication yet because am frankly not quite sure what the correct command would be to assign a node ID. I guess all figure this out tomorrow, unless someone cares to give me an example?

With the debug enabled the gateway and simple sensor (simply sending a message every five seconds) amounts to the following when building..

Gateway:

Sketch uses 22,674 bytes (70%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,424 bytes (69%) of dynamic memory, leaving 624 bytes for local variables. Maximum is 2,048 bytes.

Sensor:

Sketch uses 19,854 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,141 bytes (55%) of dynamic memory, leaving 907 bytes for local variables. Maximum is 2,048 bytes.

[hek]

Great, sounds like your doing real progress.

[Zeph]

@kolaf
How big is the flash and RAM footprint of an equivalently limited nRF24L01+ implementation? The RadioHead library sounds heavier.

---

As I'm understanding you, a node is using the RadioHead RF69 mesh network to send a request to the hub for a MySensors node id. Doesn't the mesh network have its own equivalent node id? How are you integrating those two? (eg: does your node use id 255 temporarily within the RadioHead addressing, then change it a hub assigned ID, or...?)

I'm not really clear enough about the logical layering of the two networks, and how you are mixing and matching the layers.

[kolaf]

@Damme @Zeph I took a quick stab at dumping the free run by uncommenting the serial.write command in debugPrint. I didn't get anything, so I will look into it later.

Anyway, I was finally able to send a node ID response from the serial gateway back to the sensor, and is responded with a bunch of interesting data :-). This means that every thing seems to work, and I think the implementation is in a state where I can actually use it to build sensors. The challenge now is to have the self-discipline to generalise this so that others may use it to ;).

Based on all your comments and worries about flash and RAM space, I have been thinking that maybe my approach was not the best. I have ripped out everything related to routing so that I rely completely on the RadioHead for anything related to the link and network level. This means that we get support for quite a complex dynamic sensor networks at the cost of reduced available memory for useful stuff. May be a hybrid approach would have been better, relying on the reliable datagram manager from the library and using the mysensors routing functionality. Perhaps this would yield a smaller footprint, I don't know.

As for node addressing, there is a function in the new radio library that allows me to set the node address. I can therefore use the existing node address discovery to set everything up. I've included the message sequence I get when performing this initialisation below. The first line is the node assignment sent from the gateway to the sensor. There is some extra debug garbage in the printout which I had to add when trying to get this to work.

255;255;4;0;3;1
0;0;3;9;send: 0-255 s=255,c=3,t=4,pt=0,l=1,st=302:1
0;0;3;9;sent: 0-255 s=255,c=3,t=4,pt=0,l=1,st=295:1
0;0;3;9;Available
0;0;3;9;read: 1 s=255,c=3,t=11,pt=0,l=11:Dust Sensor
1;255;3;11;Dust Sensor
0;0;3;9;Available
0;0;3;9;read: 1 s=255,c=3,t=12,pt=0,l=3:1.1
1;255;3;12;1.1
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=0,t=0,pt=0,l=15:1.4b1 (18848a2)
1;3;0;0;1.4b1 (18848a2)
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=1,t=16,pt=2,l=2:1
1;3;1;16;1
0;0;3;9;Available
0;0;3;9;read: 1 s=3,c=1,t=16,pt=2,l=2:1
1;3;1;16;1

I'm wondering since we now have a working prototype version, perhaps I could be allowed to push my branch to your repository? Of course we don't merge anything yet, but it will allow others more easy access to it for further testing. By only changing a few lines to account for different radios the current version should support most of the moteino and anarduino variations with our different radio chipsets, I think.

[kolaf]

I'm getting a bunch of errors when I compile the mqtt gateway source related to the initial list of constants. The error messages look like this:

D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:14:15: error: variable 'broker' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char broker[] PROGMEM = MQTT_BROKER_PREFIX;
               ^
D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:16:12: error: variable 'S_0' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char S_0[] PROGMEM = "Temperature"; //V_TEMP
            ^
D:\Home control\Arduino\libraries\MySensors\MyMQTT.cpp:17:12: error: variable 'S_1' must be const in order to be put into read-only section by means of '__attribute__((progmem))'
 char S_1[] PROGMEM = "Humidity"; //V_HUM

My current solution is to just delete these two files since I only care about the serial gateway, but it would be good to have everything working :-).

[kolaf]

I am also happy to report that I just tried to compile the secret knock sensor without any difficulties

Sketch uses 22,074 bytes (68%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,241 bytes (60%) of dynamic memory, leaving 807 bytes for local variables. Maximum is 2,048 bytes.

[Damme]

@kolaf What version of arduino IDE do you use? you should really use 1.5+
(But I'll fix so it works in older version)

[kolaf]

@Damme said:

@kolaf What version of arduino IDE do you use? you should really use 1.5+
(But I'll fix so it works in older version)

1.5.7

[kolaf]

I have been thinking on how to extend the library to support different radios. As far as I can see there are two options.

1. Have a generic radio driver pointer (of the superclass type RHGenericDriver) and initialise this depending on an input parameter to the Gateway constructor. The trouble with this approach is that we have to static_cast every access to this variable to its appropriate subclass in order to access radio specific functions to set transmit power, et cetera. We also have to include a bunch of header files so that everything is available during runtime. This obviously increases footprint and complexity for the maintainer.

2. Use a define in the sensor application and Gateway application to determine which radio is to be used, e.g. "#define DRH_RF69" to select the radio I am using. With an include #ifdef directives in the header and my sensor source files to include only the necessary driver headers and choose the correct set of initialisation functions for the radio. Most of this would go into the setupRadio function.

I think that the second approach is the easiest to implement. It also requires a single addition to any client application, namely the correct define at the top of the configurations header file. We can leave this as a default radio value, and the knowledgeable user can change this if he uses a different chipset. Does this make sense?

[hek]

@kolaf

Tried to open a chat (see the icon at the bottom of the screen)

[Yveaux]

@kolaf said:

I think that the second approach is the easiest to implement

Agree. Pointers to instances makes more sense when radios have to be switched dynamically or when multiple radios have to be supported.
For this application having a #define in a configuration file makes perfect sense.

[Yveaux]

@kolaf said:

Sketch uses 22,074 bytes (68%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,241 bytes (60%) of dynamic memory, leaving 807 bytes for local variables. Maximum is 2,048 bytes.

These are very acceptable numbers IMHO.
No need to worry about flash/ram usage!

[hek]

@Yveaux said:

Pointers to instances makes more sense when radios have to be switched dynamically or when multiple radios have to be supported.

Yes, defines certainly make sense now! But it would be cool to have gateway-nodes between different radio models in the future

[Yveaux]

@hek Maybe, but I have yet to see a valid usecase for this. You can always use 2 Arduino's running the default serial gateway and tie them together
I haven't looked into the radiohead interface yet (have to admit you guys got me curious, though) but isn't there simply one base class defining the interface and every radio derives from it? That would definately make things simple!

[hek]

@Yveaux said:

Maybe, but I have yet to see a valid usecase for this

Yeah, the use case is a bit strained and theoretical. But you know how it is... eventually someone want to try to mix different brand radio units.

[Yveaux]

@hek Nah, f*ck 'em

[Damme]

@kolaf I was unable to repreduce those errors, When trying to fix it to const the compiler moved the array-table to sram instead >.< I hate progmem (and Harvard architecture)

[kolaf]

@Yveaux said:

I haven't looked into the radiohead interface yet (have to admit you guys got me curious, though) but isn't there simply one base class defining the interface and every radio derives from it? That would definately make things simple!

Unfortunately it is not as simple as that. Each radio driver has a constructor which is completely different from the others, and they have different concepts.

For instance, the RF 69 library requires an interrupt pin and a select pin, while the NRF 24 library uses a select pin and an enable pin. They also have completely different notions of frequency versus channel, modem modulation versus data rate, and so forth.

I think we should expand the setupRadio function to configure some useful defaults for every radio, and then we can expose the specific radio driver so that the power user can set more complex options such as a different modulation, data rate, channel, et cetera.

Make sense?

[Yveaux]

@kolaf said:

I think we should expand the setupRadio function to configure some useful defaults for every radio

Can't you wrap the radio setup and some more stuff in a class for every radio type, derived from a single interface class?
I hope in the actual mesh layer etc. you don't have to distinguish between different radio types anymore....

[kolaf]

@Yveaux said:

@kolaf said:

I think we should expand the setupRadio function to configure some useful defaults for every radio

Can't you wrap the radio setup and some more stuff in a class for every radio type, derived from a single interface class?
I hope in the actual mesh layer etc. you don't have to distinguish between different radio types anymore....

I guess I could do that, but it sounds like more work :-). For now I will stick to a define section for each radio that configures the defaults. After all, most of the radio settings for the original version of the library also relies on defines.

I think the RF 69 configuration of the library is ready for testing by others, and I have also verified that the NRF 24 version of the library compiles. I do not have any hardware for this one, so maybe someone else wants to test to see if it works? I have forked the original repository, and my current progress can be found in the following branch (Radiohead_port):

https://github.com/kolaf/Arduino/tree/radiohead_port

[kolaf]

If you want to test, be aware that the default pins for the constructor set up to match the RF 69 radio. You have to specify the select and enable pins in the constructor explicitly.

[Yveaux]

I just had a quick look at the RadioHead implementation.
All drivers are derived from RHGenericDriver (from http://www.airspayce.com/mikem/arduino/RadioHead/classRHGenericDriver.html

If you pass an instance of RHGenericDriver in e.g. the c'tor of MySensors you're done.... But maybe I had a too-quick look

[kolaf]

It is not that simple because all the different specific drivers implement a bunch of their own functions which are not virtualised. This means that you need to static cast or dynamic cast the driver every time you want to use it which is a real pain.

On the other hand, each driver seems to come with a relatively intelligent set of defaults, so I think it is quite easy to cater for the generic case, and then we can expose the internals for those who really want to get into the specifics. Still, if you want to try to generalise it in a different way, feel free

[Zeph]

@kolaf said:

It is not that simple because all the different specific drivers implement a bunch of their own functions which are not virtualised. This means that you need to static cast or dynamic cast the driver every time you want to use it which is a real pain.

Are those driver-specific functions needed during normal operation, or are they only for non-default setup? (eg: frequency, bandwidth, etc)

[Zeph]

What about packet size? I'm guessing that the idea is to use the 32 byte limit of the nRF24L01+ for all radios (even if they have larger packet options) right?

(This approach is frequently called "least common denominator", tho the actual mathematical concept being misreferenced is actually the greatest common denominator)

[Zeph]

I don't think the flash and RAM footprint is a deal breaker in this application, but it could be a concern for some multi-sensor nodes which include several sensor libraries. So I was wondering how much heavier it is, but not saying (yet) that it's too large.

Remember to save room for a possible crypto library, if we do a more secure version.

My own use case is unusual. I am using nRF24L01+ for christmas light control, which involves a high bandwidth transmission from hub to nodes (and uses a fair amount of RAM on the nodes for buffers). I've been thinking that I could have the Christmas light display nodes also transmit sensor data to a MySensors network periodically,since they use the same radio and processor. That isn't critical tho. I could just use separate nodes on different channels for the two applications.

[Yveaux]

@Zeph said:

I'm guessing that the idea is to use the 32 byte limit of the nRF24L01+ for all radios (even if they have larger packet options)

Why? MySensors can easily handle payloads with different sizes. Only when you start mixing different radios in one network ( hek apparently thinks in this direction) you have to handle fragmentation.

[Zeph]

@hek said:
Yes, defines certainly make sense now! But it would be cool to have gateway-nodes between different radio models in the future

@Yveaux said:

Maybe, but I have yet to see a valid usecase for this. You can always use 2 Arduino's running the default serial gateway and tie them together

The RF69 has some nice possibilities for range using 433 or 868/915 MHz, albeit with less bandwidth than the nRF24L01+ at 2.4 GHz. That makes the former attractive for low bandwidth applications like a sensor network. I like the latter for some other projects that make use of the higher bandwidth, but it could be nice to include MySensor network as well using the same radio. That's what brought me here. So I could envision using both radios - RF69 for pure low bandwidth sensors and nRF for combined nodes.

@Yveaux, is there really a way to tie two hubs together for MySensors.

[Zeph]

@Yveaux said:

@Zeph said:

I'm guessing that the idea is to use the 32 byte limit of the nRF24L01+ for all radios (even if they have larger packet options)

Why? MySensors can easily handle payloads with different sizes. Only when you start mixing different radios in one network ( hek apparently thinks in this direction) you have to handle fragmentation.

Yeah, but handling fragmentation might add a lot of complication and code. If the network was designed around "expect all radios to handle at least 32 byte packets" it could be lighter and simpler.

And this is not just about mixing radios in one system; if one writes a sensor which makes use of more than 32 byte packets because it's initially using the RF69, then it would not port well to another system using a non-fragmentation nRF implementation.

[Yveaux]

@Zeph said:

is there really a way to tie two hubs together for MySensors.

Maybe it won't work out of the box, but in principe the serial format for send/receive is identical. I think that just cross-connecting the serial ports of two gateways will get you close to a working solution. When different speeds have to bridged, of (de)fragmentation is required the whole story changes...

Btw. if you want different networks, why not just use a different gateway for each network? Or is Vera not able to handle multiple gateways?

[Zeph]

One other consideration - there is a thread on Security. if MySensors were to migrate towards multi-radio support (the nRF and the RF69 being the most attractive options), then it might be tempting to design any security layer to make use of the latter's built in AES encryption, with its strengths (fast with little code) and weakneses (less flexible than a software implementation). That is, use AES in software on the nRF to do the same approach as is done in hardware with the RF69.

This would not be required (each radio could use it's own security approach, or the same software security could be used for all of them), just somewhat attractive to keep down the number of separate implementation while also making use of the hardware.

[Yveaux]

@Zeph said:

Yeah, but handling fragmentation might add a lot of complication and code. If the network was designed around "expect all radios to handle at least 32 byte packets" it could be lighter and simpler.

I'm certainly not in favour of implementing fragmentation in MySensors, and if it would be added it should be handled at a lower level anyway.

And this is not just about mixing radios in one system; if one writes a sensor which makes use of more than 32 byte packets because it's initially using the RF69, then it would not port well to another system using a non-fragmentation nRF implementation.

All the examples that come with the MySensors library use a lot less than the maximum MySensors payload (which is less than 32 bytes due to the header btw). I don't have the actual figures, but I think the largest payload is sent in the sensor presentation (the version string).
For radio-portability it would indeed be benificial to have a minimum payload size defined.

[Zeph]

@Yveaux said:

Maybe it won't work out of the box, but in principe the serial format for send/receive is identical. I think that just cross-connecting the serial ports of two gateways will get you close to a working solution. When different speeds have to bridged, of (de)fragmentation is required the whole story changes...

I've been thinking of using a APM as a "smart peripheral" to a node (for Christmas lighting, because driving some pixel light strings requires turning off interrupts for long periods). A 5V APM is as little as $2.30 shipped, so I could make a light controller intermediary with a more asynchronous interface (so the main node processor can be interrupted).

Something similar might also work with two MySensor radio hubs, especially given the relatively low bandwidth. One radio hub might be an APM connected to the main hub.

If the home automation controller will accept multiple gateways, tho, you may be right that it would be the simpler solution.

[Yveaux]

@Zeph said:

If the home automation controller will accept multiple gateways

I currently use an MQTT broker as middleware - I can install a zillion gateways if I have to this way...

[Zeph]

@Damme said:

I think we need to think about choosing a bitter atmega for this..
Next up would be Arduino Mega 2560 with what it feels like 'Unlimited ram!'
They are not That expensive today since china made its own ch340 usb - serial chip (Why this was the expensive component blows my mind :P) ATmega2560-16AU CH340G MEGA 256 you can find for ~10EUR / 13.50USD..

Also note the ATMega2560 small form factor offering: http://www.ebay.com/itm/261541870545 which is similar to an Arduino Pro Mini but with the larger processor. And it has even more pins broken out than the Arduino Mega2560.

[kolaf]

@Zeph said:

@kolaf said:

It is not that simple because all the different specific drivers implement a bunch of their own functions which are not virtualised. This means that you need to static cast or dynamic cast the driver every time you want to use it which is a real pain.

Are those driver-specific functions needed during normal operation, or are they only for non-default setup? (eg: frequency, bandwidth, etc)

Just for setup. Runtime operation is handled by the manager and is independent of the radio driver.

[kolaf]

@Zeph And you even get the "mini mega" presoldered with the radio chip of your choice for $20

https://lowpowerlab.com/shop/index.php?route=moteinomega

[Zeph]

@kolaf said:

And you even get the "mini mega" presoldered with the radio chip of your choice for $20

Well, it's actually another $6-7 for the radio, $1 for pins, and optionally $3 for extra flash. But it's still a good deal.

It's nice to have more ATMega1284 based Arduino styles options available. The official Arduino folks went for the ATMega1280/2560 instead for their high end AVR based system, and they have their advantages, but the extra RAM on the 1284 is probably more useful than the extra Flash memory.

For our nodes, the Moteino Mega probably has more than enough IO. Compared to the Arduino Pro Micor, it basically has 10 additional digital pins (plus all 8 analog pins can be used as digital, not just 6 of them). I2c is done with digital IO pins rather than analog pins. And it has another serial port.

The ATMega2560-core (I posted an ebay link a couple msgs back) has a huge number of IO pins, including many that were not available in the Arduino Mega board - eg: the pins needed for doing SPI master mode with the additional USARTS. There are applications that might be useful for, but for sensor network nodes we would rarely need so many I/Os.

(I'll stop there - these messages on other boards and chip might be a good candidate for a moderator to move to another thread)

[kolaf]

I took a look at the freeRam function to see if memory was part of my radio performance issues. However, are not able to make sense of the readings. It reports consistently -152 which seems a bit weird to me. Can anyone help me understand what this means, and whether it is a problem?

[Yveaux]

@kolaf can you show us the internals of this function?

[kolaf]

It is the function that is included in the bottom of the radio library main file.

return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);

I'm guessing that a negative value here is not a good sign. I have been battling with a very poor radio performance (10 cm range) and I think it is related to memory usage. If a switch from the mesh manager to the reliable datagram manager I suddenly get much better range, as well as heaps of free memory. I have started a thread on this on the Radiohead forum, so let's see if some bright guy there can help me.

In the meantime I have two options: use only point to point links, or reimplement the multihop routing existing in the original MySensors library. For the moment I'll hold off to see what the other guys can tell me.

[Yveaux]

@kolaf Please find a detailed explanation of the function here: http://jeelabs.org/2011/05/22/atmega-memory-use/

[kolaf]

I've turned off debug messages for the serial Gateway, and I now have 600 bytes free RAM when running. It should be even more on the sensor node, but I haven't checked. I hope this is sufficient to build relatively complex sensors without any problems.

I'm still having problems with the mesh code giving the very poor range, but I have now ruled out that this is caused by lack of memory. Maybe there is a bug in Radiohead? The saga continues...

[kolaf]

Time for a short status update:

I'm still working on trying to figure out why I get very poor radio range (10 cm) when using the automatic mesh functionality of Radiohead. I get no such issues when using simple point-to-point messages without the routing network layer.

Through trial and error I have narrowed it down to possibly a problem with the serial gateway code. I am able to communicate successfully between two implementations of a sensor using gw.send and gw.process, but as soon as I change the type from MySensors to MyGateway in one node things stop working.

I will be able to do more testing on this during the next few days. In the meantime, do you have any suggestions as to what can be different in MyGateway to stop things from working? The three things that come to my mind is its use of interrupts for inclusion mode, memory problems from increased functionality (unlikely), or possibly something to do with the serial communication.

Could you please explain to me what the purpose of inclusion mode is? I cannot seem to find any code where this mode actually does anything useful

I also don't quite understand how the serial communication works. It appears to be some kind of event handler tied to the serial port, but where and how is this done? Could this have impact on the issues I'm seeing, perhaps it is blocking for some time (although I find that hard to believe)?

[hek]

@kolaf said:

Could you please explain to me what the purpose of inclusion mode is? I cannot seem to find any code where this mode actually does anything useful

The gateway is just indicating when inclusion mode is active by flashing the LEDs. You can also enable/disable this mode by sending a message to controller. Noting more is done on the gateway side.
The purpose is to prohibit unwanted device includes. You must actively enable the inclusion mode. And during this time all newly presented sensors will be added on the controller side.

It is of course optional to use this functionality from a controller implementers point of view. It was necessary to provide this functionality when implementing support for the Vera controller as it tends to restarts every time you add a new device programatically. And you wouldn't want that to happen just by presenting a new sensor in your radio network.

I doubt the interrupts would affect radio in this case. They are only triggered when pushing the inclusion button (attached at pin 3). Just check you didn't connect radio IRQ-pin to 3 which I did once with funny results.

I also don't quite understand how the serial communication works. It appears to be some kind of event handler tied to the serial port, but where and how is this done? Could this have impact on the issues I'm seeing, perhaps it is blocking for some time (although I find that hard to believe)?

Really doubt this is you problem.

Gateway is always assigned radio-address 0. Does this perhaps have a special meaning in RadioHead?

[kolaf]

The gateway address 0 is not a problem, when I communicate between two sensors directly they have the addresses 0 and 1 without issue.

For the moteino and anarduino the RF 69 radio is always connected to interrupt pin 2, so I agree with you that this should not be an issue.

I'm also considering that I might be doing something different in the constructor since the sensor and gateway uses different constructors, but I also find this hard to believe. All radio initialisation is done inside setUpRadio which is common for both classes.

Do you have a pointer to somewhere that explains the serial event handling?

[hek]

@kolaf said:

Do you have a pointer to somewhere that explains the serial event handling?

http://arduino.cc/en/Tutorial/SerialEvent

[kolaf]

@kolaf said:

Do you have a pointer to somewhere that explains the serial event handling?

Nevermind, I found it

Regarding addresses, the only dress with a special meaning in Radiohead is 255 which is broadcast. This should not be an issue for my experiments since amusing fixed node IDs.

[kolaf]

I think I just found my problem!!

I was looking at the header file for MyGateway and I suddenly noticed that I have a bogus default value for transmit power (-14 instead of 14). This explains nicely why my range suddenly gets limited when running the Gateway. Can't wait until I get home to test this :-).

Still, experience shows it is never as easy as you think, so I won't know until tonight.

[Zeph]

OT but ... what's the point of SerialEvent()?

From the description it's called between loop() calls if there is serial input data, and you should check for more than one byte of data.

void loop() {
      // do stuff
}

void serialEvent() {
    while (Serial.available()) {
        char inChar = (char)Serial.read(); 
        // use it
    }
}

So wouldn't adding this to the end (or start) of loop do the same thing?

void loop() {
      // do stuff

      while (Serial.available()) {
         char inChar = (char)Serial.read(); 
         // use it
      }
}

Except this would work on the Leonardo etc, so there must be advantage of serialEvent that I'm missing.

[Damme]

@Zeph arduino specific.. http://www.seeedstudio.com/wiki/Where_is_Main_Function

[kolaf]

Good news everyone!

My hunch was correct and I now finally have the MySensors library working correctly through my entire house (one hop covering 20 m through two timber walls) using the mesh network manager from Radiohead. I have not tried multiple hops yet, but I have no reason to believe that that should not work since I'm using the standard Radiohead library (famous last words?).

I have pushed my current progress of the fully working version to my fork of mysensors. My ongoing task now is to merge in the current development branch and deal with all the conflicts. There are a lot of conflicts since I have changed a lot of functions signatures in my code. The next task is to try to generalise everything so that it works correctly with the radio you guys use with some decent defaults. My plan as we have discussed earlier is to have a minimal constructor and initialisation routine covers the basic functionality for every radio we want to support and exports the radio driver from the MySensors object so that power users can access this directly and tweak the settings for the radio if they want to.

I have my first my sense and node working with the new code :-). It is a simple distance sensor with a 10 cm limit which gives a binary output detected/not detected. I use this to detect a hand waving in front of the sensor and send a message to the gateway every time a hand is detected and disappears again. This is piped through a modified version of the PC mqtt Gateway I have borrowed from zephy and modified to support communication over the serial port (not just sockets). This again is connected to openhab where I use a rule based on what has been described in another thread to toggle a relay on and off and control my light :-).

When I enter my room, wave my hand, light on. When I leave my room, wave my hand, light off. Unfortunately I have to have this powered by the mains power system since I need it on all the time to have the response I want from the wave detector. If I sleep I might miss a wave, which means I will have to stand and wave for several seconds for something to happen.

[Yveaux]

@kolaf said:

My plan as we have discussed earlier is to have a minimal constructor and initialisation routine covers the basic functionality for every radio we want to support and exports the radio driver from the MySensors object so that power users can access this directly and tweak the settings for the radio if they want to.

On a second thought, I prefer to not construct the radio instance in the MySensors library and expose its ptr for 'power users', but to create the radio instance from the sketch and pass it along with the MySensors c'tor or begin-method.
This is in line with how the RadioHead library handles radios and more flexible when e.g. support for new radios is added to the RadioHead library (the MySensors library does not have to be modified then).
I also don't like all the #ifdef's in the MySensors library for all the different radio types. This just pollutes the code...

As an example I wrote down how the radio instantiation and passing it to MySensors could look like in a sketch:

#include <SPI.h>
#include <RH_NRF24.h>
#include <MySensor.h>  

RH_NRF24 nrf24;
MySensor gw;

void setup() 
{
  nrf24.init();
  // I'm a power user, so let's change the channel ;-)
  nrf24.setChannel(1);
  // Start MySensors and pass it the radio to use
  gw.begin( nrf24 );
}

I think this notation is clean and easy to understand for novice users.
How about it?

modified version of the PC mqtt Gateway I have borrowed from zephy

From Yveaux, I suppose?

[kolaf]

@Yveaux said:

I think this notation is clean and easy to understand for novice users.
How about it?

I support that wholeheartedly. The reason for my initial suggestion was the desire from the library side to be easy to initialise. Having the user initialise the radio separately removes a bunch of configuration complexity from the radio set up. The only trouble I foresee is that the typically are some standard commands required to initialise the different radios property. For instance, for my radio I have to set the frequency and transmit power. This will be identical for every sensor I will build, so it would be good to have some kind of default initialisation for the radios. I don't know what is required to set up the RF 24 radios (or was it RF 22)?

modified version of the PC mqtt Gateway I have borrowed from zephy

From Yveaux, I suppose?

Sorry, I just remembered that it was a username with a weird combination of letters

[Yveaux]

@kolaf said:

a weird combination of letters

It's French
Kinda....

[Yveaux]

@kolaf said:

The only trouble I foresee is that the typically are some standard commands required to initialise the different radios property. For instance, for my radio I have to set the frequency and transmit power. This will be identical for every sensor I will build, so it would be

We should choose & define some sane defaults (like the current defaults).
I think most of the users will just continue using the nRF24's, so this radio can be used in the examples.

It will make the transtion rather painless, IMHO

[kolaf]

The trouble is that it cannot be solved just by using defines defaults. For my case I have to explicitly call

driver.setTXPower(14)
driver.setFrequency(868)

Every time I initialise the driver. This is a pain. Perhaps we should have a separate initialisation routine that enters default values for each driver which is called in the constructor? Or maybe we call the driver specific initialisation on the object after we have constructed it?

MyGateway gw(driver);
gw.initialise_RF69()
gw.begin();

I must admit that this does not seem very elegant...

[Yveaux]

@kolaf From my head, you're using the RF69 driver, right?
Well, this applies to any driver....
Looking in RH_RF69.cpp, RH_RF69::init(), it explicitly sets the TxPower to 13 as last statement of the init procedure.
If this default is changed into a #define which can e.g. be overwritten with your own defaults we're also done, e.g.

In e.g. RH_RF69.cpp

#ifndef RF69_DEFAULT_TXPOWER
#define RF69_DEFAULT_TXPOWER (13)
#endif

and then ofcourse in RH_RF69::init():

    setTxPower(RF69_DEFAULT_TXPOWER); 

If we now allow a file with our own MySensors defaults to be included before the default definition of RF69_DEFAULT_TXPOWER, it has precedence over the RadioHead's defaults and we're done.

This requires modifying RadioHead a little, but I can't imagine the maintainers would have a problem with implementing a mechanism for compile-time configurable default values.

[Yveaux]

I just built my first nRF24L01+ setup using the RadioHead library running the nrf24_reliable_datagram_client and nrf24_reliable_datagram_server sketches on two Uno's.

Works like a charm, after I disconnected the interrupt line to the nRF24.
Don't know why yet (possibly an interrupt handler is 'secretly' installed by the library), but it took me some time to figure out...

Also requires the driver to be constructed as RH_NRF24 driver(9, 10) when using the MySensors connection-scheme.

[kolaf]

Great to hear, I'm looking forward to seeing whether my implementation will work with your radio and whether there is any benefit from it, or just a big cost. I know that from my radio the radio library needs to know the interrupt pin, so definitely installs an interrupt handler. I shouldn't think it would do that for every driver, but I might very well be wrong. Let me know if you need any help getting my work in progress to work for you, though I suspect that you are knowledgeable enough to manage it yourself

I agree that your solution to the configurations pretty elegant, and in fact for my specific case I don't think we need to change the defaults since the only reason I have to do my own initialisation is because I have the high-powered version. It makes sense to leave the RF 69 driver defaults as is, and I will just have to deal with it

Still, I am a bit reluctant to make changes to the Radiohead library. The major reason is of course that it will be difficult to upgrade the library as new releases are published (which they are quite often, apparently). The new releases are only distributed using zip files as far as I can see, which means that we would need a manual merge job every time we want to upgrade the library. It would be easier if they had a git repository for the driver, then we could to a large extent handle merging in new versions automatically. I guess I could check on their Google group whether this is an option. Doing a manual merge is of course also possible, and maybe it turns out that this is the best option after all.

Another option which is sort of a compromise is to have the main constructor detect which type of driver it is supplied with. I guess it is possible to do some kind of type checking to figure out which class it is an instance of. We can then build our own default initialisation routines (by all means based on defined values) which are called by the constructor automatically without the need for user interaction. The trouble is, obviously, but this might overwrite whatever initialisation is you did yourself before instantiating the class. I guess this can be sold with an additional parameter to turn this on and off, but then it becomes messy again.

[Yveaux]

@kolaf said:

Another option which is sort of a compromise is to have the main constructor detect which type of driver it is supplied with.

I'm afraid this isn't an option as the linker will have to link code for all radios in then, got every radio configuration....

[kolaf]

The trouble with initialising the radio separately by the user is that it increases the complexity, which I think is one of @HEK's major points with the library. It definitely looks like the easiest way to support multiple radios, but the question is, is it good enough?

[Yveaux]

@kolaf I understand @hek 's reservations.
Maybe we can add a function to MySensors that returns a static instance to the radio, all configured to use with MySensors.

#include <MySensor.h>  
MySensor gw;

void setup() 
{
  // Start MySensors and pass it the radio to use. Mysensors manages the instance
  // and configures the radio with defaults for us.
  // 'power users' can instantiate their own radio and configure it to their liking
  gw.begin( gw.createRadio() );
}

How 'bout that?

[hek]

@Yveaux @kolaf

Now we're talking!

Really appreciate you're effort on RadioHead while still keeping it simple for the end-user.

I'm finalizing the documentation for 1.4 so we can release it and continue the work on RadioHead and the new protocol changes. As @Damme said somewhere, this might be enough for a 2.0 version :).

[kolaf]

@Yveaux said:

How 'bout that?

That makes sense. I will need some help to build the default radio configuration that you use, and I can start redoing the initialisation stuff.

[kolaf]

I have made some progress on the proposed initialisation routines and I am at the point where something works. The current initialisation for my sensor looks like this:

#include <MySensor.h>
#include <RH_RF69.h>
#include <SPI.h>

RH_RF69 driver;

MySensor gw;
void setup()  
{  
  
  if(gw.setRadio(&driver)) {
	driver.setFrequency(868);
	driver.setTxPower(14);
	gw.begin();
  }
}

I stumbled upon a few snags on the way. For instance, when you initialise the manager (after giving it the driver) it sets up a bunch of defaults. This means that we have to override these defaults after the manager is initialised. This is why I split this out in a separate function setRadio. This function initialises the manager and returns true if this is successful. You may then set up your optional parameters and call begin.

A great benefit of having the radio include in the source file instead of in the library is that we now do not have to distribute it inside the MySensors library. We can now use a regular Radiohead installation in the Arduino environment. I have pushed this to my repository, let me know what you think.

https://github.com/kolaf/Arduino/tree/radiohead_port

[kolaf]

The next challenge is sleeping. The different drivers sleep using different function calls.

[Yveaux]

@kolaf said:

For instance, when you initialise the manager (after giving it the driver) it sets up a bunch of defaults.

Yes, I see it calls init() from the driver, which sucks...
I'll guess you want gw.setradio to initialize the manager then (which in turn initializes the driver)? This way you'll be able to modify the driver's parameters before MySensors starts. Not a very elegant way... (but I also don't have a better solution at the moment...)

[kolaf]

@Yveaux Exactly.

[Yveaux]

@kolaf Have you checked initialization of the managers for any radio communication taking place? If it does, your solution will not work as it will be executed using the default radio parameters...

[kolaf]

@Yveaux I don't think there is any radio communication, it just initialises the radio chip and confirms that it is able to communicate with it. Otherwise you would have had real problems with trying to start every radio at the same time

[Yveaux]

@Yveaux said:

Works like a charm, after I disconnected the interrupt line to the nRF24.

I went through the RadioHead code, looking for enabled INT0 interrupts, interrupts handlers etc. in nRF24 code.
Found none, so tested again today, with the interrupt line connected.

Now it also works with interrupt connected...

Don't know what went wrong before.

[kolaf]

@Yveaux Good to hear that it is working out for you. I'm very curious to see whether you can get my version of the library working with your radio.

I have posted on the Radiohead group to see whether they can implement a generic sleep function in the driver or in the manager. This would make it much easier for us to put everything to sleep when required. Short of that the only reasonable solution I can see to fix this is to create a sleep function in the sensor source code and pass this as a parameter to the library. Alternatively we can build a different sleep functions in the library, but I guess this will cause the same problems as you talked about earlier with bringing in a lot of code we do not need.

[Yveaux]

@kolaf said:

I have posted on the Radiohead group to see whether they can implement a generic sleep function in the driver or in the manager

Ah great! Let's see how flexible they are

Btw. I'm not totally dismissive of the idea of making small changes to the RadioHead library when it makes our life easier to get a clean interface for MySensors users. Indeed, we have to sync changes (which are almost on a daily basis right now, but that will settle over time) but when the changes to the original library are relatively small this won't be too hard.

Anyway, let's not give up yet and see if we can use the vanilla library.

[Yveaux]

I created a github repo containing the RadioHead sources, including change history. The code is commit using a script, so future updates can easily be added.

IMHO having the RadioHead library sourcecode under version control is a requrement for inclusion in MySensors and if we ever decide to make small changes to the library we can now easily fork from the original sourcecode.

Get it at https://github.com/Yveaux/RadioHead

[hek]

@kolaf

Agree on keeping the radio specific sleeping in each driver is the best option.

[Zeph]

Are we talking here about

(1) each radio driver having its own full sleep library to put the processor to sleep (potentially in addition to the sleep library used by the main sketch), or

(2) just each radio driver having its own function to put its radio into low power mode with/without IRQ?

[kolaf]

Just (2)

[Zeph]

Good. Let's call it prepare-to-sleep or power-down or something rather than sleep.

[kolaf]

@Zeph Fair enough.

Since I'm not getting any response on their mailing list, perhaps we should just go ahead and implement something. My plan is to create a virtualised powerdown in RHGenericDriver and then implement a reasonable default at least for the two radios we use. We can then expand this as we gain more experience. I know that for the RF69 driver this should be no more than five minutes work. Does anyone want to take care of your radios?

[kolaf]

If it works out, we can send them a patch

[kolaf]

Scratch my previous messages. I took a new look at the documentation and there appears to be a function setMode which can set the radio to TX, RX, and idle. I think the idle mode is as close as you come to sleeping, and I know that several of the drivers have a function called setIdleMode which you can use to define how the radio sleeps. This means that we can initialise the sleep function in the same way as we initialise the the other radio specific parameters (e.g. power and frequency), and then simply set the radio to the generic idle mode when sleeping.

[Yveaux]

Just a small update on my effort to get RadioHead & MySensors working with nRF24L01+...
I managed to get the node configured correctly using the MySensors 'defaults'.
I see data getting sent over the air (as my sniffer captures it). The only data on air, captured by Wireshark, is:

ff:7e:01:00:ff:7e:00:52:00:01:01:00
ff:7e:02:00:ff:7e:00:53:00:01:01:00
ff:7e:03:00:ff:7e:00:54:00:01:01:00
ff:7e:04:00:ff:7e:00:55:00:01:01:00
etc.

Roughly 4 seconds apart.
I have to dive deeper into the protocol , but I suspect this is the node sending broadcasting requests to find a parent.
The gateway doesn't acknowledge these broadcasts yet...
Fixed a few small issues in the nRF24 driver along the way (see https://github.com/Yveaux/RadioHead/commits/MySensors/RadioHead/RH_NRF24.cpp)
Seems like I'm almost there!

Some things I miss in the library:

• Normally the nRF24 changes its destination address when sending to the destination's node address. The current driver/RadioHead implementation just always broadcasts and the header of the data determines which node the data is really for. This means any node within range will always have to process all messages coming in to determine if they are meant for it or not. From a generic driver point-of-view I understand this approach, but it ignores powerful functionality of the nRF24. As the nRF24 is still the main radio target for the MySensors library we might have to think about a solution.
• The library does not use the auto-acknowledge feature of the nRF24 but instead seems to retry on application level. Another nice feauture of the nRF24 which remains unused...

[kolaf]

I would guess it is a arp request. Weird that it does not use the radio properly, I think both addressing and retransmission works well with rf69.

[hek]

Using auto-ack would be impossible (i think) using the in radioheads mesh-setup where every message basically is a broadcast.

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

[kolaf]

radiohead uses direct messaging, with an arp protocol and route discovery to build local routing tables, much as original mysensors. At least as far as I understand it.

Do you use the same library in both the sensor and gateway? It won't work of you mix network layer protocols.

[kolaf]

@kolaf with hop by hop acknowledgements

[Yveaux]

@Yveaux said:

ff:7e:01:00:ff:7e:00:52:00:01:01:00

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

[hek]

Yep, a 10 byte header is a lot.

[kolaf]

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

[Yveaux]

@kolaf Just because of the size of the header?
I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...