Porting MySensors to work with the RadioHead library

[kolaf]

I think I just found my problem!!

I was looking at the header file for MyGateway and I suddenly noticed that I have a bogus default value for transmit power (-14 instead of 14). This explains nicely why my range suddenly gets limited when running the Gateway. Can't wait until I get home to test this :-).

Still, experience shows it is never as easy as you think, so I won't know until tonight.

[Zeph]

OT but ... what's the point of SerialEvent()?

From the description it's called between loop() calls if there is serial input data, and you should check for more than one byte of data.

void loop() {
      // do stuff
}

void serialEvent() {
    while (Serial.available()) {
        char inChar = (char)Serial.read(); 
        // use it
    }
}

So wouldn't adding this to the end (or start) of loop do the same thing?

void loop() {
      // do stuff

      while (Serial.available()) {
         char inChar = (char)Serial.read(); 
         // use it
      }
}

Except this would work on the Leonardo etc, so there must be advantage of serialEvent that I'm missing.

[Damme]

@Zeph arduino specific.. http://www.seeedstudio.com/wiki/Where_is_Main_Function

[kolaf]

Good news everyone!

My hunch was correct and I now finally have the MySensors library working correctly through my entire house (one hop covering 20 m through two timber walls) using the mesh network manager from Radiohead. I have not tried multiple hops yet, but I have no reason to believe that that should not work since I'm using the standard Radiohead library (famous last words?).

I have pushed my current progress of the fully working version to my fork of mysensors. My ongoing task now is to merge in the current development branch and deal with all the conflicts. There are a lot of conflicts since I have changed a lot of functions signatures in my code. The next task is to try to generalise everything so that it works correctly with the radio you guys use with some decent defaults. My plan as we have discussed earlier is to have a minimal constructor and initialisation routine covers the basic functionality for every radio we want to support and exports the radio driver from the MySensors object so that power users can access this directly and tweak the settings for the radio if they want to.

I have my first my sense and node working with the new code :-). It is a simple distance sensor with a 10 cm limit which gives a binary output detected/not detected. I use this to detect a hand waving in front of the sensor and send a message to the gateway every time a hand is detected and disappears again. This is piped through a modified version of the PC mqtt Gateway I have borrowed from zephy and modified to support communication over the serial port (not just sockets). This again is connected to openhab where I use a rule based on what has been described in another thread to toggle a relay on and off and control my light :-).

When I enter my room, wave my hand, light on. When I leave my room, wave my hand, light off. Unfortunately I have to have this powered by the mains power system since I need it on all the time to have the response I want from the wave detector. If I sleep I might miss a wave, which means I will have to stand and wave for several seconds for something to happen.

[Yveaux]

@kolaf said:

My plan as we have discussed earlier is to have a minimal constructor and initialisation routine covers the basic functionality for every radio we want to support and exports the radio driver from the MySensors object so that power users can access this directly and tweak the settings for the radio if they want to.

On a second thought, I prefer to not construct the radio instance in the MySensors library and expose its ptr for 'power users', but to create the radio instance from the sketch and pass it along with the MySensors c'tor or begin-method.
This is in line with how the RadioHead library handles radios and more flexible when e.g. support for new radios is added to the RadioHead library (the MySensors library does not have to be modified then).
I also don't like all the #ifdef's in the MySensors library for all the different radio types. This just pollutes the code...

As an example I wrote down how the radio instantiation and passing it to MySensors could look like in a sketch:

#include <SPI.h>
#include <RH_NRF24.h>
#include <MySensor.h>  

RH_NRF24 nrf24;
MySensor gw;

void setup() 
{
  nrf24.init();
  // I'm a power user, so let's change the channel ;-)
  nrf24.setChannel(1);
  // Start MySensors and pass it the radio to use
  gw.begin( nrf24 );
}

I think this notation is clean and easy to understand for novice users.
How about it?

modified version of the PC mqtt Gateway I have borrowed from zephy

From Yveaux, I suppose?

[kolaf]

@Yveaux said:

I think this notation is clean and easy to understand for novice users.
How about it?

I support that wholeheartedly. The reason for my initial suggestion was the desire from the library side to be easy to initialise. Having the user initialise the radio separately removes a bunch of configuration complexity from the radio set up. The only trouble I foresee is that the typically are some standard commands required to initialise the different radios property. For instance, for my radio I have to set the frequency and transmit power. This will be identical for every sensor I will build, so it would be good to have some kind of default initialisation for the radios. I don't know what is required to set up the RF 24 radios (or was it RF 22)?

modified version of the PC mqtt Gateway I have borrowed from zephy

From Yveaux, I suppose?

Sorry, I just remembered that it was a username with a weird combination of letters

[Yveaux]

@kolaf said:

a weird combination of letters

It's French
Kinda....

[Yveaux]

@kolaf said:

The only trouble I foresee is that the typically are some standard commands required to initialise the different radios property. For instance, for my radio I have to set the frequency and transmit power. This will be identical for every sensor I will build, so it would be

We should choose & define some sane defaults (like the current defaults).
I think most of the users will just continue using the nRF24's, so this radio can be used in the examples.

It will make the transtion rather painless, IMHO

[kolaf]

The trouble is that it cannot be solved just by using defines defaults. For my case I have to explicitly call

driver.setTXPower(14)
driver.setFrequency(868)

Every time I initialise the driver. This is a pain. Perhaps we should have a separate initialisation routine that enters default values for each driver which is called in the constructor? Or maybe we call the driver specific initialisation on the object after we have constructed it?

MyGateway gw(driver);
gw.initialise_RF69()
gw.begin();

I must admit that this does not seem very elegant...

[Yveaux]

@kolaf From my head, you're using the RF69 driver, right?
Well, this applies to any driver....
Looking in RH_RF69.cpp, RH_RF69::init(), it explicitly sets the TxPower to 13 as last statement of the init procedure.
If this default is changed into a #define which can e.g. be overwritten with your own defaults we're also done, e.g.

In e.g. RH_RF69.cpp

#ifndef RF69_DEFAULT_TXPOWER
#define RF69_DEFAULT_TXPOWER (13)
#endif

and then ofcourse in RH_RF69::init():

    setTxPower(RF69_DEFAULT_TXPOWER); 

If we now allow a file with our own MySensors defaults to be included before the default definition of RF69_DEFAULT_TXPOWER, it has precedence over the RadioHead's defaults and we're done.

This requires modifying RadioHead a little, but I can't imagine the maintainers would have a problem with implementing a mechanism for compile-time configurable default values.

[Yveaux]

I just built my first nRF24L01+ setup using the RadioHead library running the nrf24_reliable_datagram_client and nrf24_reliable_datagram_server sketches on two Uno's.

Works like a charm, after I disconnected the interrupt line to the nRF24.
Don't know why yet (possibly an interrupt handler is 'secretly' installed by the library), but it took me some time to figure out...

Also requires the driver to be constructed as RH_NRF24 driver(9, 10) when using the MySensors connection-scheme.

[kolaf]

Great to hear, I'm looking forward to seeing whether my implementation will work with your radio and whether there is any benefit from it, or just a big cost. I know that from my radio the radio library needs to know the interrupt pin, so definitely installs an interrupt handler. I shouldn't think it would do that for every driver, but I might very well be wrong. Let me know if you need any help getting my work in progress to work for you, though I suspect that you are knowledgeable enough to manage it yourself

I agree that your solution to the configurations pretty elegant, and in fact for my specific case I don't think we need to change the defaults since the only reason I have to do my own initialisation is because I have the high-powered version. It makes sense to leave the RF 69 driver defaults as is, and I will just have to deal with it

Still, I am a bit reluctant to make changes to the Radiohead library. The major reason is of course that it will be difficult to upgrade the library as new releases are published (which they are quite often, apparently). The new releases are only distributed using zip files as far as I can see, which means that we would need a manual merge job every time we want to upgrade the library. It would be easier if they had a git repository for the driver, then we could to a large extent handle merging in new versions automatically. I guess I could check on their Google group whether this is an option. Doing a manual merge is of course also possible, and maybe it turns out that this is the best option after all.

Another option which is sort of a compromise is to have the main constructor detect which type of driver it is supplied with. I guess it is possible to do some kind of type checking to figure out which class it is an instance of. We can then build our own default initialisation routines (by all means based on defined values) which are called by the constructor automatically without the need for user interaction. The trouble is, obviously, but this might overwrite whatever initialisation is you did yourself before instantiating the class. I guess this can be sold with an additional parameter to turn this on and off, but then it becomes messy again.

[Yveaux]

@kolaf said:

Another option which is sort of a compromise is to have the main constructor detect which type of driver it is supplied with.

I'm afraid this isn't an option as the linker will have to link code for all radios in then, got every radio configuration....

[kolaf]

The trouble with initialising the radio separately by the user is that it increases the complexity, which I think is one of @HEK's major points with the library. It definitely looks like the easiest way to support multiple radios, but the question is, is it good enough?

[Yveaux]

@kolaf I understand @hek 's reservations.
Maybe we can add a function to MySensors that returns a static instance to the radio, all configured to use with MySensors.

#include <MySensor.h>  
MySensor gw;

void setup() 
{
  // Start MySensors and pass it the radio to use. Mysensors manages the instance
  // and configures the radio with defaults for us.
  // 'power users' can instantiate their own radio and configure it to their liking
  gw.begin( gw.createRadio() );
}

How 'bout that?

[hek]

@Yveaux @kolaf

Now we're talking!

Really appreciate you're effort on RadioHead while still keeping it simple for the end-user.

I'm finalizing the documentation for 1.4 so we can release it and continue the work on RadioHead and the new protocol changes. As @Damme said somewhere, this might be enough for a 2.0 version :).

[kolaf]

@Yveaux said:

How 'bout that?

That makes sense. I will need some help to build the default radio configuration that you use, and I can start redoing the initialisation stuff.

[kolaf]

I have made some progress on the proposed initialisation routines and I am at the point where something works. The current initialisation for my sensor looks like this:

#include <MySensor.h>
#include <RH_RF69.h>
#include <SPI.h>

RH_RF69 driver;

MySensor gw;
void setup()  
{  
  
  if(gw.setRadio(&driver)) {
	driver.setFrequency(868);
	driver.setTxPower(14);
	gw.begin();
  }
}

I stumbled upon a few snags on the way. For instance, when you initialise the manager (after giving it the driver) it sets up a bunch of defaults. This means that we have to override these defaults after the manager is initialised. This is why I split this out in a separate function setRadio. This function initialises the manager and returns true if this is successful. You may then set up your optional parameters and call begin.

A great benefit of having the radio include in the source file instead of in the library is that we now do not have to distribute it inside the MySensors library. We can now use a regular Radiohead installation in the Arduino environment. I have pushed this to my repository, let me know what you think.

https://github.com/kolaf/Arduino/tree/radiohead_port

[kolaf]

The next challenge is sleeping. The different drivers sleep using different function calls.

[Yveaux]

@kolaf said:

For instance, when you initialise the manager (after giving it the driver) it sets up a bunch of defaults.

Yes, I see it calls init() from the driver, which sucks...
I'll guess you want gw.setradio to initialize the manager then (which in turn initializes the driver)? This way you'll be able to modify the driver's parameters before MySensors starts. Not a very elegant way... (but I also don't have a better solution at the moment...)

[kolaf]

@Yveaux Exactly.

[Yveaux]

@kolaf Have you checked initialization of the managers for any radio communication taking place? If it does, your solution will not work as it will be executed using the default radio parameters...

[kolaf]

@Yveaux I don't think there is any radio communication, it just initialises the radio chip and confirms that it is able to communicate with it. Otherwise you would have had real problems with trying to start every radio at the same time

[Yveaux]

@Yveaux said:

Works like a charm, after I disconnected the interrupt line to the nRF24.

I went through the RadioHead code, looking for enabled INT0 interrupts, interrupts handlers etc. in nRF24 code.
Found none, so tested again today, with the interrupt line connected.

Now it also works with interrupt connected...

Don't know what went wrong before.

[kolaf]

@Yveaux Good to hear that it is working out for you. I'm very curious to see whether you can get my version of the library working with your radio.

I have posted on the Radiohead group to see whether they can implement a generic sleep function in the driver or in the manager. This would make it much easier for us to put everything to sleep when required. Short of that the only reasonable solution I can see to fix this is to create a sleep function in the sensor source code and pass this as a parameter to the library. Alternatively we can build a different sleep functions in the library, but I guess this will cause the same problems as you talked about earlier with bringing in a lot of code we do not need.

[Yveaux]

@kolaf said:

I have posted on the Radiohead group to see whether they can implement a generic sleep function in the driver or in the manager

Ah great! Let's see how flexible they are

Btw. I'm not totally dismissive of the idea of making small changes to the RadioHead library when it makes our life easier to get a clean interface for MySensors users. Indeed, we have to sync changes (which are almost on a daily basis right now, but that will settle over time) but when the changes to the original library are relatively small this won't be too hard.

Anyway, let's not give up yet and see if we can use the vanilla library.

[Yveaux]

I created a github repo containing the RadioHead sources, including change history. The code is commit using a script, so future updates can easily be added.

IMHO having the RadioHead library sourcecode under version control is a requrement for inclusion in MySensors and if we ever decide to make small changes to the library we can now easily fork from the original sourcecode.

Get it at https://github.com/Yveaux/RadioHead

[hek]

@kolaf

Agree on keeping the radio specific sleeping in each driver is the best option.

[Zeph]

Are we talking here about

(1) each radio driver having its own full sleep library to put the processor to sleep (potentially in addition to the sleep library used by the main sketch), or

(2) just each radio driver having its own function to put its radio into low power mode with/without IRQ?

[kolaf]

Just (2)

[Zeph]

Good. Let's call it prepare-to-sleep or power-down or something rather than sleep.

[kolaf]

@Zeph Fair enough.

Since I'm not getting any response on their mailing list, perhaps we should just go ahead and implement something. My plan is to create a virtualised powerdown in RHGenericDriver and then implement a reasonable default at least for the two radios we use. We can then expand this as we gain more experience. I know that for the RF69 driver this should be no more than five minutes work. Does anyone want to take care of your radios?

[kolaf]

If it works out, we can send them a patch

[kolaf]

Scratch my previous messages. I took a new look at the documentation and there appears to be a function setMode which can set the radio to TX, RX, and idle. I think the idle mode is as close as you come to sleeping, and I know that several of the drivers have a function called setIdleMode which you can use to define how the radio sleeps. This means that we can initialise the sleep function in the same way as we initialise the the other radio specific parameters (e.g. power and frequency), and then simply set the radio to the generic idle mode when sleeping.

[Yveaux]

Just a small update on my effort to get RadioHead & MySensors working with nRF24L01+...
I managed to get the node configured correctly using the MySensors 'defaults'.
I see data getting sent over the air (as my sniffer captures it). The only data on air, captured by Wireshark, is:

ff:7e:01:00:ff:7e:00:52:00:01:01:00
ff:7e:02:00:ff:7e:00:53:00:01:01:00
ff:7e:03:00:ff:7e:00:54:00:01:01:00
ff:7e:04:00:ff:7e:00:55:00:01:01:00
etc.

Roughly 4 seconds apart.
I have to dive deeper into the protocol , but I suspect this is the node sending broadcasting requests to find a parent.
The gateway doesn't acknowledge these broadcasts yet...
Fixed a few small issues in the nRF24 driver along the way (see https://github.com/Yveaux/RadioHead/commits/MySensors/RadioHead/RH_NRF24.cpp)
Seems like I'm almost there!

Some things I miss in the library:

• Normally the nRF24 changes its destination address when sending to the destination's node address. The current driver/RadioHead implementation just always broadcasts and the header of the data determines which node the data is really for. This means any node within range will always have to process all messages coming in to determine if they are meant for it or not. From a generic driver point-of-view I understand this approach, but it ignores powerful functionality of the nRF24. As the nRF24 is still the main radio target for the MySensors library we might have to think about a solution.
• The library does not use the auto-acknowledge feature of the nRF24 but instead seems to retry on application level. Another nice feauture of the nRF24 which remains unused...

[kolaf]

I would guess it is a arp request. Weird that it does not use the radio properly, I think both addressing and retransmission works well with rf69.

[hek]

Using auto-ack would be impossible (i think) using the in radioheads mesh-setup where every message basically is a broadcast.

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

[kolaf]

radiohead uses direct messaging, with an arp protocol and route discovery to build local routing tables, much as original mysensors. At least as far as I understand it.

Do you use the same library in both the sensor and gateway? It won't work of you mix network layer protocols.

[kolaf]

@kolaf with hop by hop acknowledgements

[Yveaux]

@Yveaux said:

ff:7e:01:00:ff:7e:00:52:00:01:01:00

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

[hek]

Yep, a 10 byte header is a lot.

[kolaf]

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

[Yveaux]

@kolaf Just because of the size of the header?
I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

[Yveaux]

@kolaf said:

I'm very curious to see whether you can get my version of the library working with your radio.

Got it to work with nRF24!
Still some issues every now and then and I have to verify the radio config but messages are exchanged between node & gateway, using RHMesh.

Can't believe I ran into the same trap again as with the CRC8 calculation (remember @hek ?)
sizeof(message) in MySensor::sendWrite is 33 bytes, which is one byte more than nRF24 can send, causing all transmissions to fail...

See my changes here: https://github.com/Yveaux/Arduino/commits/radiohead_port

[kolaf]

@Yveaux maybe I could give you write access to my repository. Will be much easier if we are working on the same thing, less hassle with keeping up to date.

[Yveaux]

@kolaf OK, fine, but it's working now so I don't plan on much changes anymore.
Could you merge all 1.4 changes to your fork? I feel it's quite running behind.
Next I'll focus on a Wireshark dissector, as I'm not convinced everything on air is needed/correct...
It would be great BTW if the sniffer would work with the Radiohead drivers, so it supports different radios....

[kolaf]

I pulled in the latest eight commits, merged, and pushed. I think I should be completely up-to-date with the development branch.

[kolaf]

@Yveaux I think you have to submit a pull request for me to merge your changes in.

[Yveaux]

@kolaf I think I just merged everything in your repo... This is all a bit new to me

[kolaf]

@Yveaux Got it, thanks.

[ToSa]

@hek said:

RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

That's beneficial in terms of available EEPROM space for other purposes - but would be interesting to see the shit-storm after a power-outage when all nodes power-up at the same time and need to rebuild the entire network tree from scratch?

[ToSa]

@hek said:

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

@Kolaf said:

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

@Yveaux said:

I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

Just a thought (I did NOT review the entire RH code) - from the RH documentation it sounds like we could use the RH Drivers without using any of the RH Managers - keeping routing etc. within MySensors responsibility.
Just checking as it appears that @kolaf and @Yveaux have spent significant time reviewing the code and I would want to understand if that's an option or a totally stupid idea

[kolaf]

@ToSa It is definitely not a stupid idea, it is a variation of what I suggested previously which you quoted in your post (using the reliable datagram manager instead). It is a question of the abstraction level we want to use. Personally I prefer to use a library where everything is built in and that has widespread use. This gives us much more features for "free", and depending on the user base also a lot more testing. However, it turns out that not everything is perfect. The header is too big, and the library is perhaps not as efficient as it could be.

This boils down to whether we should do things more efficiently ourselves, or try to fix the library, either officially or unofficially. It looks like the maintainer of the project is very open to suggestions. In fact, I just saw a post where he said he had implemented a generic powerdown mode for all the drivers that I suggested a few days ago. My guess is that if we work with him we could get a lean and mean radio library which could benefit both the MySensors project and the Arduino community as a whole. There appears to be people here with quite good radio knowledge, so think this could be a very powerful combination. Still, it is a question of time, effort, and priorities...

[Yveaux]

@kolaf I totally agree with you.
The MySensors library currently has a solid nrf24 implementation and routing works fine (though not fully mesh) so there's no direct need to switch driver and routing layer.
This is a roadmap item on which we should continue working, IMO together with the RadioHead development. The RadioHead library has only recently been developed (few months old or so) and is already very mature looking. This is very promising for the future and there's no use in developing/maintaining 2 nearly identical libraries.
I think the current flaws (e.g. code & message size) can be improved with our help making it a solid base for MySensors.

[ToSa]

Thanks @kolaf @Yveaux.

makes sense - I thought "just" using the drivers at least as an interim step could give us a head-start but it appears that the majority of the "issues" with RH need to be fixed on the driver level anyways so we would still need to work these first with the RH team...

@Yveaux said:

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

I think the addressing is not that bad: the TO/FROM is actually the next/last used in MySensors - only for a single-hop communication these are the same as SOURCE/DEST. With that FROM/SOURCE/DEST in RH == last/sender/destination in MySensors. The TO (next) is the one that is unnecessary because it's part of the nRF24 header - as this is not the case for all radios, this should be adjusted in the NRF24 specific code in RH and shouldn't be that hard to do.

The ID fields are used to determine if a packet is a duplicate (once for the hop and once end-to-end). I'm wondering how this is done in MySensors today. For the single hop that's probably part of the nRF24 internals (auto-acknowledge / auto-resend) and maybe it could be avoided on the driver level in RH but probably needs some tweaking of ReliableDatagram as well. For e2e I don't think there is an equivalent in MySensors today and it might actually be a valuable add rather than a waste of a byte what could happen without it: a node receives a packet and sends an ack - the ack does not make it to the sender and therefore the sender submits the same packet again - the node receives the package once again and thinks it's a new packet... think about a command that toggles a light - woulc toggle twice and go off->on->off instead of off->on.

The FLAGS bytes appear to be a total waste of space.

• Looking at the first FLAGS byte it's actually defined on the driver level already but not used in Driver or Datagram but only in ReliableDatagram and the only flag set is RH_FLAGS_ACK - a full byte for a single bit worth of information... The two options would be to either remove that byte and mark ACK packets differently or to just make use of the remaining 7 bits for other purposes. I would prefer using the 7 bits - protocol version (3 bits) is definitely a good fit - command (3 bits) might be the other. Using the lower four bits of FLAGS for application layer purposes is even foreseen in RH (and could be easily changed to use 6 bits for app layer in RHGenericDriver.h)
• The benefit of the second FLAGS byte (from the Router) is totally unclear to me. It seems like the neither RHMesh nor RHRouter code uses it at all even if it's defined on that level so nothing else should use it either... worth a discussion with the RH team - and either remove it completely or use it again for app layer specific stuff (MySensors type?). I'm surprised that they don't use these FLAGS instead of the first byte of RHMesh messages to determine route discovery vs. app date vs. ...

Assuming that we can get rid of the "TO" and the "driver level ID" and make use of the FLAGS fields as mentioned above the header would be:

• RHDatagram: FROM:FLAGS (covering MyMessage last / ack / version / command)
• RHRouter: DEST:SOURCE:HOPS:ID:FLAGS (covering MyMessage destination / sender / type)
• RHMesh: one byte to determine "application data" - tbc if this can be moved to the FLAGS instead consuming less than a full byte. at least the MyMessage "payload type" can be included in that byte
• MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

This would leave us with 2+5+1=8 bytes instead of the current 7 bytes for the header (to be fair 8 instead of 6 due to the removed sensor field). That's not as bad as 14!

[hek]

@ToSa said:

MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

Yes, that makes sense.

[hek]

Reply from Mike:

Some of your contributors have the wrong idea about the headers.

There are 4 header bytes used by drivers, RHDatagram and RHReliableDatagram

TO
FROM
ID
FLAGS

These are the hop-to-hop headers. In fact they are present in the payload (but
effectively unused) even if you use the drivers directly.

RHRouter and RHMesh (if you use them) add

DEST
SOURCE
HOPS
ID
FLAGS

these are end-to-end headers and are not necessarily the same as the hop-to-
hop headers. In the general case they will be different values.

In my view all these header are all necessary.

BTW, new version 1.33 supports sleep mode for RH_RF69, RH_RF22, RH_NRF24,
RH_RF24, RH_RF95 drivers.

Cheers.

[ToSa]

@hek said:

Reply from Mike:

these are end-to-end headers and are not necessarily the same as the hop-to-
hop headers. In the general case they will be different values.

yep

In my view all these header are all necessary.

Well, debatable at least for the FLAGS (or the size of the FLAGS).

Three questions (@Mike, if you are reading here, just reply directly, otherwise I'll contact you via the RH channels...):

• is the RH library itself using the end-to-end FLAGS? I couldn't find any reference other than setting it to 0 (when using RHMesh).
• is the RH library itself using the hop FLAGS for anything other than identifying ACK packets (RH_FLAGS_ACK / RH_FLAGS_NONE)?
• for the hop ID and TO - if the specific RF chip supports this internally (as part of the preamble/header the chip adds to the on-air packet), why duplicating this wasting valuable payload size - the driver for the specific RF chip could handle it internally and make it look the same for the outside world using the RHGenericDriver interface without changes... I'll give it a try and let you know.

[Yveaux]

@hek Thanks for getting Mike 'on board'! Good move!

[kolaf]

Great work guys. I'm glad to see some enthusiasm for this as it will allow me to utilise the platform and not have to reinvent the wheel

[ToSa]

Test compilation comparing code size with current MySensors 1.4 radio library and with radiohead. The radiohead integration to MySensors is not optimized (pushing full MySensors header through even if the radiohead header has the same content etc.) and the radiohead library provides some benefits (not just a 1:1 replacament) - so it's expected to be larger. Based on the two examples below it's right now about 4-5k flash and ~800 ram for global variables.

=> let's see how far we can get that down reducing the duplication of header data etc.

**SerialGateway with MySensors: **
Sketch uses 17,566 bytes (57%) of program storage space. Maximum is 30,720 bytes.
Global variables use 688 bytes (33%) of dynamic memory, leaving 1,360 bytes for local variables. Maximum is 2,048 bytes.

SerialGateway with RadioHead:
Sketch uses 22,242 bytes (72%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,455 bytes (71%) of dynamic memory, leaving 593 bytes for local variables. Maximum is 2,048 bytes.

**DallasTemp with MySensors: **
Sketch uses 20,288 bytes (66%) of program storage space. Maximum is 30,720 bytes.
Global variables use 551 bytes (26%) of dynamic memory, leaving 1,497 bytes for local variables. Maximum is 2,048 bytes.

DallasTemp with RadioHead:
Sketch uses 24,592 bytes (80%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,319 bytes (64%) of dynamic memory, leaving 729 bytes for local variables. Maximum is 2,048 bytes.

[Damme]

@ToSa Wonder what the runtime values are...

[Yveaux]

@ToSa said:

SerialGateway with RadioHead:
Sketch uses 22,242 bytes (72%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,455 bytes (71%) of dynamic memory, leaving 593 bytes for local variables. Maximum is 2,048 bytes.

What radio/Arduino IDE did you compile for?

For Serial Gateway, nrf24, Arduino 1.5.7 I get:
Sketch uses 20.962 bytes (64%) of program storage space. Maximum is 32.256 bytes.
Global variables use 1.471 bytes (71%) of dynamic memory, leaving 577 bytes for local variables. Maximum is 2.048 bytes.

@Damme

Wonder what the runtime values are...

Currently MySensor::setRadio creates an instance of RHMesh at runtime, so RAM usage is even worse...

[Damme]

This post is deleted!

[ToSa]

@Yveaux said:

What radio/Arduino IDE did you compile for?

nRF24 / 1.5.6-r2
DEBUG turned on - your values appear to be with DEBUG turned off

I did a few changes starting to reduce the header so I can't rerun with DEBUG off and compare right now...

[kolaf]

A small digression from the main topic here, but I just tested the mesh functionality of the library. I took a laptop with a serial gateway outside of the range of my sensor and observed that it stopped working. I then went back a few metres and powered up a battery powered sensor which basically just sent hello every second and processed messages. I was able to significantly extend my range, experiencing only a few seconds of interrupt in the ping flow.

Moving back and powering down the intermediate node, the direct route was reestablished within a new couple of seconds and everything was working correctly again I think I will have much fun playing with this.

[Yveaux]

@kolaf said:

I was unable to significantly extend my range, experiencing only a few seconds of interrupt in the ping flow.

Shouldn't that be "I was able"?
I want to be happy with you, but not sure now

[kolaf]

You are most certainly correct, I have edited my original post. I'm suffering a bit from tendinitis, so I'm using speech recognition when I write. Sometimes it recognises almost exactly what I say

[Zeph]

@Yveaux said:

The MySensors library currently has a solid nrf24 implementation and routing works fine (though not fully mesh) so there's no direct need to switch driver and routing layer.

What are the differences between the MySensors protocol and RH in terms of mesh dynamics?

I'm guessing that RH is more dynamicly configured, but I'd like to understand more.

[Yveaux]

@kolaf ah, great to see your results then!
I read some mixed results on the Radiohead mailing list regarding mesh, so good to see it's working for you.
BTW I'm in the progress of writing wireshark directors for Radiohead which can be used with the nrf24 sniffer. It'll give us better insight on Radiohead's performance and functioning.

[Zeph]

It makes sense to migrate the MySensors last,sender and destination bytes to RH from,source,dest.

I do not think it makes sense to move type (V_code) or command into the RH FLAGs. Typically when such space is reserved for the protocol in the header but not yet fully used, it's subject to change as the protocal evolves. If part of the goal is to take advantage of the maintenance and ongoing development of RH, we don't want to set up a conflict - because other users are not going to use those FLAGs our way anyway.

Basically, Radio Head should be concerned with delivering a payload (set of bytes) to a node. Everything specific to just the MySensors application should be in the payload (as seen by RH). That includes:

• child id ("sensor")
• command
• V_code ("type")
• version (if needed)
• ack - is this still needed??

If we want to remain compatible with other RH users (ie: their development), rather than re-allocating bytes or bits in the RH header to our own purposes, we would be putting our MySensors' header in the RH payload, but omitting from our header the redundant fields which are handled by RH:

• last,
• sender,
• destination

The RH payload would include the reduced MySensors header plus the MySensors payload.

[ToSa]

@Zeph said:

I do not think it makes sense to move type (V_code) or command into the RH FLAGs. Typically when such space is reserved for the protocol in the header but not yet fully used, it's subject to change as the protocal evolves. If part of the goal is to take advantage of the maintenance and ongoing development of RH, we don't want to set up a conflict - because other users are not going to use those FLAGs our way anyway.

From the RH documentation / the source code:

• for Datagram FLAGS:
  A bitmask of flags. The most significant 4 bits are reserved for use by RadioHead. The least significant 4 bits are reserved for applications.
• for Router/Mesh FLAGS:
  Optional flags for use by subclasses or application layer

Not setting up a conflict with the RH code will be the biggest issue: the two benefits of RH are the multi radio support and the more advanced Mesh topology - which would be worth a couple of additional bytes in flash and ram - but the other reason for the big overhead is that several advanced features of the nRF24 chip are not used because they are not available in all supported chips.

Two examples:

• the multi-pipe capability of nRF24 would allow to filter traffic to only current address and broadcast and avoid that any other traffic ever reaches the MCU (only slightly less code but wondering if that helps with battery powered nodes).
• the ReliableDiagram processing in code essentially duplicates capabilities that are build into the chipset as well (auto-acknowledge / autoresubmit)

I know you are interested in getting RF69 supported but the question is how much negative impact for all the other users (with current nRF24 setup) is acceptable. That's like using the RF69 and implementing an AES encryption in code rather than using the build-in capabilities of the chip...

[ToSa]

Didn't want to sound too negative yesterday...

• I'm still looking at the RH library but changes will be major and will not necessarily be in line with the RH purpose so Mike might not want to merge them into the main codebase: mainly defining what driver is used at compile time of the library (similar to MyConfig) which would then allow to e.g. "bypass" the majority of ReliableDatagram etc.
• what we should definitely do is splitting application layer and network layer: no longer inherit MySensor from RF24 but create an RF24 instance at runtime / leave application layer message handling in MySensors / move network layer message handling to a separate class (the "driver" for RF24). This will make it way easier to switch radios in the future.

[kolaf]

@ToSa said:

Didn't want to sound too negative yesterday...

• I'm still looking at the RH library but changes will be major and will not necessarily be in line with the RH purpose so Mike might not want to merge them into the main codebase: mainly defining what driver is used at compile time of the library (similar to MyConfig) which would then allow to e.g. "bypass" the majority of ReliableDatagram etc.

Is it not sufficient that we define the radio at compile time in the sensors and the gateway as we have done now?

[ToSa]

@kolaf said:

Is it not sufficient that we define the radio at compile time in the sensors and the gateway as we have done now?

I'm talking about changing how the library is compiled and essentially having a separate ReliableDiagram class that bypasses manual handling of ACKs etc. and instead leaves reliable delivery to the radio itself. Mesh and Router are derived from the ReliableDatagram class - therefore we would need to define which radio to use prior to library compilation - not just prior to sketch compilation.

[Zeph]

@ToSa said:

the two benefits of RH are the multi radio support and the more advanced Mesh topology

Could someone summarize the differences between the current MySensors mesh and the RadioHead mesh, and why the latter is preferable for us?

[hek]

@Zeph

RH is a true mesh where every node can communicate with any other directly (if in range). All nodes acts as repeaters.

MySensors forms nodes forms a star network where every node has a parent and potentially a few child sensors. The gateway node is the "center". Every node always tries to find the closest (least number of hops) route to gateway.

         GW
         / \  
        A   D
       / \
      B   C

If B want to communicate with C messages has to pass A in the MySensors case. In RH C and B will communicate directly if the can hear each other.

RH does not use the nifty features NRF24L01 offers which offloads mcu (pipes/addresses) nor auto ack (impossible in RH setup). So nodes in range has to process every message it can "hear". And this can potentially be a lot.

[Zeph]

@hek
Thanks, that was very helpful, and a good explanation.

That helps explain the overhead in the RH library mesh. It sounds cool for some purposes, but it also sounds like possible overkill for the wireless sensor network. Allowing B and C to communicate directly is not a primary use case. The MySensors is purpose-built for the sensor network with central gateway use case, and is pretty lean thereby. As I see it, even allowing B to send to C via A is not a primary use case, but sort of falls out "for free" given the routing approach that allows repeating nodes.

Aside - can B send a message to itself via A?

In my case, I'm mainly focusing on multi-sensor nodes, so RAM or Flash could be in short supply with a larger radio library. Worse still, I want to be able to incorporate MySensors functionality into nodes which also have non MySensors functions (eg: xmas light control also usng nRF24L01+), so space is even more at a premium.

So while the RH option is interesting, and a lighter weight RH derivative may pan out, I hope the MySensor WSN radio layer will continue to provide a "lean and mean" alternative into the future.

[Zeph]

@hek said:

RH does not use the nifty features NRF24L01 offers which offloads mcu (pipes/addresses) nor auto ack (impossible in RH setup). So nodes in range has to process every message it can "hear". And this can potentially be a lot.

So MySensors can use both individual node addresses as well a broadcast, while RH uses only broadcast (at the radio level)? I understand why RH needs to do this, and I wonder if the receive FIFO ever overruns with unneeded packets.

I wonder if MySensors gets much savings from using the auto-ack of ESB. If you already have network ack code anyway, how much code & ram is saved by sometimes using ESB (one hop away) and sometimes not? (I do get that a star network with no repeater nodes could use only ESB autoack and save some code).

[hek]

@Zeph said:

So MySensors can use both individual node addresses as well a broadcast, while RH uses only broadcast (at the radio level)?

Yes.

I think end-to-end ack is very important. The inter-node ack is only used as a trigger re-routing today (if communication fails 3 times to parent node .. the node tries to find another parent).

[kolaf]

Could someone please explain to me how the direct messaging of the NRF24 works? In my mind every radio network is a broadcast network on the physical layer, and it is up to the receiving radio to determine whether the packet is relevant or not. How is this different from your radios? Is it perhaps that your radios have a hardware address so that the filtering is done on the hardware level, while Radiohead uses a software address, so each packet has to be explicitly processed? I have no idea, I'm just guessing here...

[hek]

@kolaf

Yes,

NRF chip offer 6 reading "pipes". In practice it acts as a hw filter for addresses between 0-255.

I use this to have one broadcast address (255) which all repeating nodes and gateway listens to and each sensor also listen to its own address.

[kolaf]

Looking at the documentation for the RF69 (page 41) it talks about sync word recognition. This is a programmable word that apparently can be used as the node's hardware address, similarly to the NRF24. Perhaps it is possible to expand the Radiohead drivers to utilise this functionality where it is available for filtering incoming packets?

As for the dynamic mesh, this is a feature I find especially useful for one of my dream applications. We have horses, and it would be really fun to place a radio module with a accelerometer, and maybe GPS receiver, on each horse to track their movements. This could also be combined with a breakable wire in the harness which would let us know if the radio had been ripped off of the horse. Having a dynamic mesh will greatly increase the range of flexibility of such a monitoring network. I know that this is somewhat outside of the normal use case, but how fun wouldn't it be to build this? :-).

Personally I'm also a bit interested in direct sensor to sensor configurations to allow for tightly coupled control systems without the need for a central controller in the loop (all the time). I know, we have resource limits so that I can probably not have everything, but one can dream...

[kolaf]

Nevermind the sink word recognition, this appears to function as some kind of network ID has to be the same for both the sender and receiver. However, there is also an optional address byte which is considered by the radio before the packet enters the FIFO queue. Page 55 of http://www.hoperf.com/upload/rfchip/RF69-V1.2.pdf

[Yveaux]

@kolaf said:

an optional address byte

The trouble is you have to distinguish between broadcasts and direct messages, so you need to listen at 2 addresses at a time, so to say.

[hek]

@kolaf said:

As for the dynamic mesh, this is a feature I find especially useful for one of my dream applications.

Yep, both pros and cons.

A nice feature you can do with MySensors is to set a static parent (when calling gw.begin()). This could be useful for a presence sensor e.g. mounted on a car set statically to contact a repeater node mounted near the garage. The car sensor won't try to find or contact any other node than the garage one.

[Zeph]

@kolaf said:

As for the dynamic mesh, this is a feature I find especially useful for one of my dream applications. We have horses, and it would be really fun to place a radio module with a accelerometer, and maybe GPS receiver, on each horse to track their movements. This could also be combined with a breakable wire in the harness which would let us know if the radio had been ripped off of the horse. Having a dynamic mesh will greatly increase the range of flexibility of such a monitoring network.

Just brainstorming, but what if we consider adding the concept of a roving node to the MySensors network, without the overhead of a full mesh? Your use case doesn't require that all nodes be meshed with each other, just that a class of leaf nodes be able to move around.

If I understand, a gateway or repeater node will forward a packet from anybody, if it recognizes the destination (as a node id in its routing table). Suppose you had coverage of your corral (or whatever) via gateway and various repeaters. In theory (perhaps naive theory) the roving node could report to the gateway by broadcasting a message picked up by whatever repeater is nearest; being addressed to node 0, the repeater knows what to do with it.

There would be no network ack and no way for the gateway to send to the roving node as described so far. That is, unless this special "rover packet" caused each relay to rewrite that entry of its routing table as it was passed to the gateway, to enable an updated path back to the roving node. If your horses run around among many nodes often, EEPROM wear could be a concern, unless there was a small "routing override table" in RAM to handle roving nodes.

One part I'm not clear about is how to avoid two or more nearby repeaters both forwarding this broadcast packet from the roving node. That might be tricky if it often causes OTA collisions. It could also complicate the back route logic, IF that was implemented. If we could avoid or gracefully recover from collisions, the gateway and controller may be able to deal with multiple copies of the same packet (not unlike the MQTT level 1 "deliver at least once").

And - it might be technically infeasible to enhance MySensors networking to support roving nodes. But in that case I'm sure I'll learn something from the manner in which it's shot down

[kolaf]

@Yveaux said:

@kolaf said:

an optional address byte

The trouble is you have to distinguish between broadcasts and direct messages, so you need to listen at 2 addresses at a time, so to say.

There is a separate optional broadcast address byte. I'm not sure how this is used, but I'm guessing that the radio checks against both addresses before deciding what to do with the message. That should solve the broadcast problem.

[hek]

@Zeph @kolaf

The rovering node will be able to send its message (after a few failed attempts) in the current solution.
If even faster parent-search is required you can change the #define SEARCH_FAILURES in MySensor.h.

[kolaf]

@hek Just to be clear, I'm not dependent on the mesh functionality. I think I could be very happy with the current MySensors functionality with the added part of supporting my radio

[hek]

@kolaf said:

There is a separate optional broadcast address byte. I'm not sure how this is used, but I'm guessing that the radio checks against both addresses before deciding what to do with the message. That should solve the broadcast problem.

That's good!

[Zeph]

@kolaf said:

There is a separate optional broadcast address byte.

Aha, I see that on page 57 now.

I think that would handle it! (I see no need for 6 address-filtered pipes, two addresses will do).

Yes, it would be nice if the RH library could take advantage of address filtering for those radios which support it. I'm not so sure auto-ack is important tho.

[Yveaux]

So, anyone still developing on RadioHead integration, or did all of you just give up?

I spend some time on writing Wireshark dissectors for RadioHead and integration of MySensors, to be used with the sniffer (http://forum.mysensors.org/topic/242/wireless-nrf24l01-sniffer-for-mysensors)
Code is almost final (still have to test dessection of routing tables).

Just as a preview, here's a screenshot to hopefully get you enthusiastic again!

[hek]

@Yveaux

Looks good @Yveaux!

[Yveaux]

I'm still a bit in doubt how sleeping nodes fit in when using RadioHead.
It has no notion of sleeping nodes and just expects all nodes to be available anytime.
When a route is e.g. discovered which uses a node that is about to go to sleep, routing will fail, but probably the route will then be rediscovered and finally settle using a node that is continuously powered.
How long this will take and the likelyhood of such scenarios (depends ofcourse on the amount of nodes and how often/when they are awake) is still unclear to me.
Hopefully I can analyze some real-world behavior using the sniffer.

[ToSa]

@Yveaux said:

So, anyone still developing on RadioHead integration, or did all of you just give up?

no progress on my end - but that's rather related to the fact that this was my first week back in the office after vacation...

I think we should do one thing first: decouple the "network layer" from the "application layer" in MySensors:

• not having MySensors derived from RF24
• have a "generic" interface between MySensors and the radio

That way it would be rather easy to make use of other radios and other network topologies - could be RH or something else. I've taken a shot at that (a lot of this based on what Kolaf and Yveaux did adjusting MySensors to RH) and will provide a link once it compiles and is tested and I hope I can keep the overhead small enough to be acceptable for the gained flexibility.

[kolaf]

It is great to see that something is still happening. I wonder if I can build a packet sniffer for the RF69 radios? I guess it shouldn't be that difficult since they receive everything that is sent...

The reason for my lack of progress late the has several reasons.

1. I have a version of the library networks for my simplifications. This means that I can start building small sensors instead of spending all my time on the library
2. My kids think I spent way too much time on this project already
3. I'm waiting for some kind of consensus to emerge.

My hope has been to get this new radio library thing to a point where the community would adopt it so that I would not have to do any specific maintenance to keep my copy up to date with the official version. Based on this very long discussion thread there seems to be interest among you guys to achieve this, so I hope we will be able to get there at some point. However, we need to agree on how it is to be done, how much of the Radiohead library we want to use (or maybe even use a different library?). This obviously depends mostly on resource constraints considerations, and I'm not the best to judge this since I have not used MySensors extensively.

It is great to see effort being spent on trying to reduce the footprint, and as an experiment it makes sense to do this. Once we have seen how small it is possible to make it, then we can perhaps agree on what features we want (everything, or just the device interface, maybe with RHDatagram manager class (without acknowledgements)).

I'm holding back on developing any more this until we have reached some kind of consensus since I do not really have time to play around too much without being a relatively sure that the results will be useful.

[ToSa]

@ToSa said:

I think we should do one thing first: decouple the "network layer" from the "application layer" in MySensors:

• not having MySensors derived from RF24
• have a "generic" interface between MySensors and the radio

Initial version - tested with a simple DallasTemperatureSensor setup successfully here
Note that this fork/branch does not include the "1.4 update 1" changes for nodeID / CRLF.

Overhead is limited (~150b flash / ~20b ram):
Sketch uses 20,426 bytes (66%) of program storage space. Maximum is 30,720 bytes.
Global variables use 575 bytes (28%) of dynamic memory, leaving 1,473 bytes for local variables. Maximum is 2,048 bytes.

[kolaf]

Care to share any details on which parts you took from where? Is it basically the Radiohead drivers with the MySensors network layer? Or did you make more effort on trying to split everything up to a larger extent

[ToSa]

@kolaf it's only splitting the MySensor application layer form the underlying network layer. Still using the MySensor tree network topology and the RF24 driver. It's adding the framework to allow others to easily integrate other topologies (e.g. mesh) and other radio modules (e.g. RF69). The interface expects a reliable submission - no matter if that's achieved by software (like radiohead) or hardware (nRF24).
The MyDriver header file includes a short description of the routines and it should be fairly easy to attach it to radiohead. The main benefit is that the original MySensor tree setup still works as before if wished.