Porting MySensors to work with the RadioHead library

[kolaf]

I would guess it is a arp request. Weird that it does not use the radio properly, I think both addressing and retransmission works well with rf69.

[hek]

Using auto-ack would be impossible (i think) using the in radioheads mesh-setup where every message basically is a broadcast.

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

[kolaf]

radiohead uses direct messaging, with an arp protocol and route discovery to build local routing tables, much as original mysensors. At least as far as I understand it.

Do you use the same library in both the sensor and gateway? It won't work of you mix network layer protocols.

[kolaf]

@kolaf with hop by hop acknowledgements

[Yveaux]

@Yveaux said:

ff:7e:01:00:ff:7e:00:52:00:01:01:00

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

[hek]

Yep, a 10 byte header is a lot.

[kolaf]

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

[Yveaux]

@kolaf Just because of the size of the header?
I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

[Yveaux]

@kolaf said:

I'm very curious to see whether you can get my version of the library working with your radio.

Got it to work with nRF24!
Still some issues every now and then and I have to verify the radio config but messages are exchanged between node & gateway, using RHMesh.

Can't believe I ran into the same trap again as with the CRC8 calculation (remember @hek ?)
sizeof(message) in MySensor::sendWrite is 33 bytes, which is one byte more than nRF24 can send, causing all transmissions to fail...

See my changes here: https://github.com/Yveaux/Arduino/commits/radiohead_port

[kolaf]

@Yveaux maybe I could give you write access to my repository. Will be much easier if we are working on the same thing, less hassle with keeping up to date.

[Yveaux]

@kolaf OK, fine, but it's working now so I don't plan on much changes anymore.
Could you merge all 1.4 changes to your fork? I feel it's quite running behind.
Next I'll focus on a Wireshark dissector, as I'm not convinced everything on air is needed/correct...
It would be great BTW if the sniffer would work with the Radiohead drivers, so it supports different radios....

[kolaf]

I pulled in the latest eight commits, merged, and pushed. I think I should be completely up-to-date with the development branch.

[kolaf]

@Yveaux I think you have to submit a pull request for me to merge your changes in.

[Yveaux]

@kolaf I think I just merged everything in your repo... This is all a bit new to me ;-)

[kolaf]

@Yveaux Got it, thanks.

[ToSa]

@hek said:

RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

That's beneficial in terms of available EEPROM space for other purposes - but would be interesting to see the shit-storm after a power-outage when all nodes power-up at the same time and need to rebuild the entire network tree from scratch?

[ToSa]

@hek said:

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

@Kolaf said:

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

@Yveaux said:

I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

Just a thought (I did NOT review the entire RH code) - from the RH documentation it sounds like we could use the RH Drivers without using any of the RH Managers - keeping routing etc. within MySensors responsibility.
Just checking as it appears that @kolaf and @Yveaux have spent significant time reviewing the code and I would want to understand if that's an option or a totally stupid idea :)

[kolaf]

@ToSa It is definitely not a stupid idea, it is a variation of what I suggested previously which you quoted in your post (using the reliable datagram manager instead). It is a question of the abstraction level we want to use. Personally I prefer to use a library where everything is built in and that has widespread use. This gives us much more features for "free", and depending on the user base also a lot more testing. However, it turns out that not everything is perfect. The header is too big, and the library is perhaps not as efficient as it could be.

This boils down to whether we should do things more efficiently ourselves, or try to fix the library, either officially or unofficially. It looks like the maintainer of the project is very open to suggestions. In fact, I just saw a post where he said he had implemented a generic powerdown mode for all the drivers that I suggested a few days ago. My guess is that if we work with him we could get a lean and mean radio library which could benefit both the MySensors project and the Arduino community as a whole. There appears to be people here with quite good radio knowledge, so think this could be a very powerful combination. Still, it is a question of time, effort, and priorities...

[Yveaux]

@kolaf I totally agree with you.
The MySensors library currently has a solid nrf24 implementation and routing works fine (though not fully mesh) so there's no direct need to switch driver and routing layer.
This is a roadmap item on which we should continue working, IMO together with the RadioHead development. The RadioHead library has only recently been developed (few months old or so) and is already very mature looking. This is very promising for the future and there's no use in developing/maintaining 2 nearly identical libraries.
I think the current flaws (e.g. code & message size) can be improved with our help making it a solid base for MySensors.

[ToSa]

Thanks @kolaf @Yveaux.

makes sense - I thought "just" using the drivers at least as an interim step could give us a head-start but it appears that the majority of the "issues" with RH need to be fixed on the driver level anyways so we would still need to work these first with the RH team...

@Yveaux said:

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

I think the addressing is not that bad: the TO/FROM is actually the next/last used in MySensors - only for a single-hop communication these are the same as SOURCE/DEST. With that FROM/SOURCE/DEST in RH == last/sender/destination in MySensors. The TO (next) is the one that is unnecessary because it's part of the nRF24 header - as this is not the case for all radios, this should be adjusted in the NRF24 specific code in RH and shouldn't be that hard to do.

The ID fields are used to determine if a packet is a duplicate (once for the hop and once end-to-end). I'm wondering how this is done in MySensors today. For the single hop that's probably part of the nRF24 internals (auto-acknowledge / auto-resend) and maybe it could be avoided on the driver level in RH but probably needs some tweaking of ReliableDatagram as well. For e2e I don't think there is an equivalent in MySensors today and it might actually be a valuable add rather than a waste of a byte :) what could happen without it: a node receives a packet and sends an ack - the ack does not make it to the sender and therefore the sender submits the same packet again - the node receives the package once again and thinks it's a new packet... think about a command that toggles a light - woulc toggle twice and go off->on->off instead of off->on.

The FLAGS bytes appear to be a total waste of space.

• Looking at the first FLAGS byte it's actually defined on the driver level already but not used in Driver or Datagram but only in ReliableDatagram and the only flag set is RH_FLAGS_ACK - a full byte for a single bit worth of information... The two options would be to either remove that byte and mark ACK packets differently or to just make use of the remaining 7 bits for other purposes. I would prefer using the 7 bits - protocol version (3 bits) is definitely a good fit - command (3 bits) might be the other. Using the lower four bits of FLAGS for application layer purposes is even foreseen in RH (and could be easily changed to use 6 bits for app layer in RHGenericDriver.h)
• The benefit of the second FLAGS byte (from the Router) is totally unclear to me. It seems like the neither RHMesh nor RHRouter code uses it at all even if it's defined on that level so nothing else should use it either... worth a discussion with the RH team - and either remove it completely or use it again for app layer specific stuff (MySensors type?). I'm surprised that they don't use these FLAGS instead of the first byte of RHMesh messages to determine route discovery vs. app date vs. ...

Assuming that we can get rid of the "TO" and the "driver level ID" and make use of the FLAGS fields as mentioned above the header would be:

• RHDatagram: FROM:FLAGS (covering MyMessage last / ack / version / command)
• RHRouter: DEST:SOURCE:HOPS:ID:FLAGS (covering MyMessage destination / sender / type)
• RHMesh: one byte to determine "application data" - tbc if this can be moved to the FLAGS instead consuming less than a full byte. at least the MyMessage "payload type" can be included in that byte
• MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

This would leave us with 2+5+1=8 bytes instead of the current 7 bytes for the header (to be fair 8 instead of 6 due to the removed sensor field). That's not as bad as 14!