Porting MySensors to work with the RadioHead library

[Yveaux]

@kolaf Just because of the size of the header?
I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

[Yveaux]

@kolaf said:

I'm very curious to see whether you can get my version of the library working with your radio.

Got it to work with nRF24!
Still some issues every now and then and I have to verify the radio config but messages are exchanged between node & gateway, using RHMesh.

Can't believe I ran into the same trap again as with the CRC8 calculation (remember @hek ?)
sizeof(message) in MySensor::sendWrite is 33 bytes, which is one byte more than nRF24 can send, causing all transmissions to fail...

See my changes here: https://github.com/Yveaux/Arduino/commits/radiohead_port

[kolaf]

@Yveaux maybe I could give you write access to my repository. Will be much easier if we are working on the same thing, less hassle with keeping up to date.

[Yveaux]

@kolaf OK, fine, but it's working now so I don't plan on much changes anymore.
Could you merge all 1.4 changes to your fork? I feel it's quite running behind.
Next I'll focus on a Wireshark dissector, as I'm not convinced everything on air is needed/correct...
It would be great BTW if the sniffer would work with the Radiohead drivers, so it supports different radios....

[kolaf]

I pulled in the latest eight commits, merged, and pushed. I think I should be completely up-to-date with the development branch.

[kolaf]

@Yveaux I think you have to submit a pull request for me to merge your changes in.

[Yveaux]

@kolaf I think I just merged everything in your repo... This is all a bit new to me ;-)

[kolaf]

@Yveaux Got it, thanks.

[ToSa]

@hek said:

RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

That's beneficial in terms of available EEPROM space for other purposes - but would be interesting to see the shit-storm after a power-outage when all nodes power-up at the same time and need to rebuild the entire network tree from scratch?

[ToSa]

@hek said:

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

@Kolaf said:

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

@Yveaux said:

I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

Just a thought (I did NOT review the entire RH code) - from the RH documentation it sounds like we could use the RH Drivers without using any of the RH Managers - keeping routing etc. within MySensors responsibility.
Just checking as it appears that @kolaf and @Yveaux have spent significant time reviewing the code and I would want to understand if that's an option or a totally stupid idea :)

[kolaf]

@ToSa It is definitely not a stupid idea, it is a variation of what I suggested previously which you quoted in your post (using the reliable datagram manager instead). It is a question of the abstraction level we want to use. Personally I prefer to use a library where everything is built in and that has widespread use. This gives us much more features for "free", and depending on the user base also a lot more testing. However, it turns out that not everything is perfect. The header is too big, and the library is perhaps not as efficient as it could be.

This boils down to whether we should do things more efficiently ourselves, or try to fix the library, either officially or unofficially. It looks like the maintainer of the project is very open to suggestions. In fact, I just saw a post where he said he had implemented a generic powerdown mode for all the drivers that I suggested a few days ago. My guess is that if we work with him we could get a lean and mean radio library which could benefit both the MySensors project and the Arduino community as a whole. There appears to be people here with quite good radio knowledge, so think this could be a very powerful combination. Still, it is a question of time, effort, and priorities...

[Yveaux]

@kolaf I totally agree with you.
The MySensors library currently has a solid nrf24 implementation and routing works fine (though not fully mesh) so there's no direct need to switch driver and routing layer.
This is a roadmap item on which we should continue working, IMO together with the RadioHead development. The RadioHead library has only recently been developed (few months old or so) and is already very mature looking. This is very promising for the future and there's no use in developing/maintaining 2 nearly identical libraries.
I think the current flaws (e.g. code & message size) can be improved with our help making it a solid base for MySensors.

[ToSa]

Thanks @kolaf @Yveaux.

makes sense - I thought "just" using the drivers at least as an interim step could give us a head-start but it appears that the majority of the "issues" with RH need to be fixed on the driver level anyways so we would still need to work these first with the RH team...

@Yveaux said:

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

I think the addressing is not that bad: the TO/FROM is actually the next/last used in MySensors - only for a single-hop communication these are the same as SOURCE/DEST. With that FROM/SOURCE/DEST in RH == last/sender/destination in MySensors. The TO (next) is the one that is unnecessary because it's part of the nRF24 header - as this is not the case for all radios, this should be adjusted in the NRF24 specific code in RH and shouldn't be that hard to do.

The ID fields are used to determine if a packet is a duplicate (once for the hop and once end-to-end). I'm wondering how this is done in MySensors today. For the single hop that's probably part of the nRF24 internals (auto-acknowledge / auto-resend) and maybe it could be avoided on the driver level in RH but probably needs some tweaking of ReliableDatagram as well. For e2e I don't think there is an equivalent in MySensors today and it might actually be a valuable add rather than a waste of a byte :) what could happen without it: a node receives a packet and sends an ack - the ack does not make it to the sender and therefore the sender submits the same packet again - the node receives the package once again and thinks it's a new packet... think about a command that toggles a light - woulc toggle twice and go off->on->off instead of off->on.

The FLAGS bytes appear to be a total waste of space.

• Looking at the first FLAGS byte it's actually defined on the driver level already but not used in Driver or Datagram but only in ReliableDatagram and the only flag set is RH_FLAGS_ACK - a full byte for a single bit worth of information... The two options would be to either remove that byte and mark ACK packets differently or to just make use of the remaining 7 bits for other purposes. I would prefer using the 7 bits - protocol version (3 bits) is definitely a good fit - command (3 bits) might be the other. Using the lower four bits of FLAGS for application layer purposes is even foreseen in RH (and could be easily changed to use 6 bits for app layer in RHGenericDriver.h)
• The benefit of the second FLAGS byte (from the Router) is totally unclear to me. It seems like the neither RHMesh nor RHRouter code uses it at all even if it's defined on that level so nothing else should use it either... worth a discussion with the RH team - and either remove it completely or use it again for app layer specific stuff (MySensors type?). I'm surprised that they don't use these FLAGS instead of the first byte of RHMesh messages to determine route discovery vs. app date vs. ...

Assuming that we can get rid of the "TO" and the "driver level ID" and make use of the FLAGS fields as mentioned above the header would be:

• RHDatagram: FROM:FLAGS (covering MyMessage last / ack / version / command)
• RHRouter: DEST:SOURCE:HOPS:ID:FLAGS (covering MyMessage destination / sender / type)
• RHMesh: one byte to determine "application data" - tbc if this can be moved to the FLAGS instead consuming less than a full byte. at least the MyMessage "payload type" can be included in that byte
• MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

This would leave us with 2+5+1=8 bytes instead of the current 7 bytes for the header (to be fair 8 instead of 6 due to the removed sensor field). That's not as bad as 14!

[hek]

@ToSa said:

MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

Yes, that makes sense.

[hek]

Reply from Mike:

Some of your contributors have the wrong idea about the headers.

There are 4 header bytes used by drivers, RHDatagram and RHReliableDatagram

TO
FROM
ID
FLAGS

These are the hop-to-hop headers. In fact they are present in the payload (but
effectively unused) even if you use the drivers directly.

RHRouter and RHMesh (if you use them) add

DEST
SOURCE
HOPS
ID
FLAGS

these are end-to-end headers and are not necessarily the same as the hop-to-
hop headers. In the general case they will be different values.

In my view all these header are all necessary.

BTW, new version 1.33 supports sleep mode for RH_RF69, RH_RF22, RH_NRF24,
RH_RF24, RH_RF95 drivers.

Cheers.

[ToSa]

@hek said:

Reply from Mike:

these are end-to-end headers and are not necessarily the same as the hop-to-
hop headers. In the general case they will be different values.

yep :)

In my view all these header are all necessary.

Well, debatable at least for the FLAGS (or the size of the FLAGS).

Three questions (@Mike, if you are reading here, just reply directly, otherwise I'll contact you via the RH channels...):

• is the RH library itself using the end-to-end FLAGS? I couldn't find any reference other than setting it to 0 (when using RHMesh).
• is the RH library itself using the hop FLAGS for anything other than identifying ACK packets (RH_FLAGS_ACK / RH_FLAGS_NONE)?
• for the hop ID and TO - if the specific RF chip supports this internally (as part of the preamble/header the chip adds to the on-air packet), why duplicating this wasting valuable payload size - the driver for the specific RF chip could handle it internally and make it look the same for the outside world using the RHGenericDriver interface without changes... I'll give it a try and let you know.

[Yveaux]

@hek Thanks for getting Mike 'on board'! Good move!

[kolaf]

Great work guys. I'm glad to see some enthusiasm for this as it will allow me to utilise the platform and not have to reinvent the wheel :-)

[ToSa]

Test compilation comparing code size with current MySensors 1.4 radio library and with radiohead. The radiohead integration to MySensors is not optimized (pushing full MySensors header through even if the radiohead header has the same content etc.) and the radiohead library provides some benefits (not just a 1:1 replacament) - so it's expected to be larger. Based on the two examples below it's right now about 4-5k flash and ~800 ram for global variables.

=> let's see how far we can get that down reducing the duplication of header data etc.

**SerialGateway with MySensors: **
Sketch uses 17,566 bytes (57%) of program storage space. Maximum is 30,720 bytes.
Global variables use 688 bytes (33%) of dynamic memory, leaving 1,360 bytes for local variables. Maximum is 2,048 bytes.

SerialGateway with RadioHead:
Sketch uses 22,242 bytes (72%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,455 bytes (71%) of dynamic memory, leaving 593 bytes for local variables. Maximum is 2,048 bytes.

**DallasTemp with MySensors: **
Sketch uses 20,288 bytes (66%) of program storage space. Maximum is 30,720 bytes.
Global variables use 551 bytes (26%) of dynamic memory, leaving 1,497 bytes for local variables. Maximum is 2,048 bytes.

DallasTemp with RadioHead:
Sketch uses 24,592 bytes (80%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,319 bytes (64%) of dynamic memory, leaving 729 bytes for local variables. Maximum is 2,048 bytes.

[Damme]

@ToSa Wonder what the runtime values are...