Okay, reporting in to the discussion thread from the github PR.

My setup is a bit different from @virtualmkr, since I don't have any N2N traffic in my network, all logic is handled by the controller (HomeAssistant). Summing up the details, here's what I've learned so far that affects packet loss on my setup.

1. Time to switch from receiving to sending. With a repeater (e.g. 328p @12Mhz) buffering RX packets before sending, this takes a while. Suggested solutions: - max out MY_RF24_SET_ARD register value, implement exponential backoff when sending. (How to actually implement it in mysensors code - that's a big question)

2. Interference from wifi/zigbee (including sidelobes!). This is pretty easy to solve, if you don't have too many neighbors. My DIY usb<-->nRF24L01 dongles can work by sweeping all available channels looking for carrier frequency. It works like a poor man's 2.4Ghz spectrum analyzer. Once looking at the picture, I moved my WiFi to 6 & 11 channels, and used the lower part of the spectrum for mysensors & zigbee, making enough space in-between for the sidelobes to fit in.

3. Switching several actuators in one node very fast. I have LED dimmers that have 4 channels. If I switch them all at once in home assistant - some packets are lost. My guess is that once 3 packets are inside the RX FIFO of the nRF24 subsequent packets are lost. And since the gateway is an STM32 it can send those waaay faster. Maxing out MY_RF24_SET_ARD register value may help, I still have to try it.

Hardware issues:

1. Crappy antennas/design on modules, suffering from 'magic finger' problem. I've written about these in my blog. Can be fixed with some manual labor: https://ncrmnt.org/2021/01/11/nrf24l01-manually-calibrating-the-antenna-with-mysensors-and-homeassistant/

2. Weather. Wet snow kills 2.4Ghz signals, if you're operating outside.

3. Antenna direction. Especially if the gateway has a PA+LNA module and an external SMA antenna.

@virtualmkr

This is a SN7440 clone from Soviet times, right?

Exactly. Still have a handful of those sitting somewhere in the closet.

A wait() is not a good idea because it introduces recursion of unknown deep, depending from the users receive() implementation.

Yep, I thought the same. Something like (ARD * 15 / 2) seems like a good delay.

But exponential backoff is also not a good idea, because this can become a long delay of a second or more which blocks all time the main loop. But a short random delay of some 10th ms (like mentioned by @skywatch) worked for me in my setup.

Exponential backoff would be a nice feature for NodeManager, but that would only take care to deliver sensor data to the uplink (unless we ack data back all the time). This wouldn't help if the traffic jam occurs somewhere further up.

Another idea I thought of, would be listening if any carrier is present. (nRF24L01+ has that feature) and sending any packets only once there's no carrier + adding a random delay. This would hopefully avoid collisions, even with other things, like WiFi.

@virtualmkr I have your sketches a quick look. Seems like you have debug enabled on the gateway. Esp8266 has to deal with WiFi stack &mqtt handling despite running on pretty high speed. This may introduce a lot of delays.
And my wild guess would be that that is enough for m328p to chew up.

I'll set up my 'staging' gateway in a radio noisy environment and give your test a spin this weekend.

@virtualmkr Great work. Meanwhile, I have updated my homeassistant installation and set up a second, 'staging' network with the modules I have around. I think I can arrange remote access to this setup later, if that's needed.

It seems to me that the issue I've been experiencing is partially related to HomeAssistant's way of working with mysensors actuators. If I 'gang-switch' a bunch of lights I see the following in the log: (note the 7;X;1;1;2;1)

Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 8
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;3;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;4;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;5;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;6;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;7;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;8;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 8: value_type 2, value = 1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 8: value_type 3, value = 100
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 2, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 3, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 7;253;1;0;37;-45
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 253
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 7;253;1;0;37;-45
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 253
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 2, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 3, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 253: value_type 37, value = -45

HomeAssistant requests acks for all packets and dumps them all at once to the gateway. With half-duplex nodes this is just asking for trouble. I'm diving right now into hass component code to see if I can add a little extra debugging.

I gave a the HAL code a more detailed review, so I think there's a possibility to implement something like a simple 'collision avoidance' using RPD register. First, here's some info from the datasheet:

Now, let's do some calculations.
We have ~42 byte packets max (1 + 5 + 2 + 32 + 2). These take 42/250000.0 * 1000 * 1000 = 168 uS of radio time at 250000. With that in mind, I'd try something like this.

LOCAL bool RF24_sendMessage(const uint8_t recipient, const void *buf, const uint8_t len,
                            const bool noACK)
{
int retry = 5;
while (retry--) {
   RF24_stopListening();
   if (RF24_testRPD()) { //Something was talking on the radio, we have to wait for a while
     RF24_startListening(); //Start listening again and wait.
     delay_us(180 * 2);  // Delay enough time to chew up at least 2 radio packets at 250Kbps
   }

But that would be RF24-specific. Another idea is making the delays dependent on NODE_ID and implement something like a simple bus arbiter, so that nodes with lower NODE_ID have more priority. I will only be able to give this a shot on the weekend, so feel free to try it out.

I gave a the HAL code a more detailed review, so I think there's a possibility to implement something like a simple 'collision avoidance' using RPD register. First, here's some info from the datasheet:

Now, let's do some calculations.
We have ~42 byte packets max (1 + 5 + 2 + 32 + 2). These take 42/250000.0 * 1000 * 1000 = 168 uS of radio time at 250000. With that in mind, I'd try something like this.

LOCAL bool RF24_sendMessage(const uint8_t recipient, const void *buf, const uint8_t len,
                            const bool noACK)
{
int retry = 5;
while (retry--) {
   RF24_stopListening();
   if (RF24_testRPD()) { //Something was talking on the radio, we have to wait for a while
     RF24_startListening(); //Start listening again and wait.
     delay_us(180 * 2);  // Delay enough time to chew up at least 2 radio packets at 250Kbps
   }

But that would be RF24-specific. Another idea is making the delays dependent on NODE_ID and implement something like a simple bus arbiter, so that nodes with lower NODE_ID have more priority. I will only be able to give this a shot on the weekend, so feel free to try it out.

I see 2 ways and one hacky way:

• Agressive buffering with some delays, e.g. only switch modes after N ms after last packet is received, so that repeaters absorb all data bursts. Perhaps even TX buffering.
• Adjust HomeAssistant controller code: It should not request ack, but instead expect devices to send back the new state some time soon after flipping state (and retry those packets)
• The hacky way: Make HomeAssistant wait for the ack, before sending the next command for the node. Perhaps the easiest, but we'll have to bother some of the devs working on that integration.

@virtualmkr Thanks for the nice pictures. With that in mind I'm pretty much sure I'm getting a clash with the acks going back the the gw somewhere on the way, especially when a repeater or two is involved.

@virtualmkr Great work. Meanwhile, I have updated my homeassistant installation and set up a second, 'staging' network with the modules I have around. I think I can arrange remote access to this setup later, if that's needed.

It seems to me that the issue I've been experiencing is partially related to HomeAssistant's way of working with mysensors actuators. If I 'gang-switch' a bunch of lights I see the following in the log: (note the 7;X;1;1;2;1)

Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 8
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;3;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;4;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;5;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;6;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;7;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [mysensors.transport] Sending 7;8;1;1;2;1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 8: value_type 2, value = 1
Mar 08 19:14:20 bladeling hass[26390]: 2021-03-08 19:14:20 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 8: value_type 3, value = 100
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;2;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 2, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 3, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 4;2;1;0;3;0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 4 child 2
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 7;253;1;0;37;-45
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 253
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [mysensors.transport] Receiving 7;253;1;0;37;-45
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.gateway] Node update: node 7 child 253
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 2, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Strip Wisp 4 2: value_type 3, value = 0
Mar 08 19:14:21 bladeling hass[26390]: 2021-03-08 19:14:21 DEBUG (MainThread) [homeassistant.components.mysensors.device] Entity update: Red Wisp 7 253: value_type 37, value = -45

HomeAssistant requests acks for all packets and dumps them all at once to the gateway. With half-duplex nodes this is just asking for trouble. I'm diving right now into hass component code to see if I can add a little extra debugging.

@virtualmkr I have your sketches a quick look. Seems like you have debug enabled on the gateway. Esp8266 has to deal with WiFi stack &mqtt handling despite running on pretty high speed. This may introduce a lot of delays.
And my wild guess would be that that is enough for m328p to chew up.

I'll set up my 'staging' gateway in a radio noisy environment and give your test a spin this weekend.

@virtualmkr

This is a SN7440 clone from Soviet times, right?

Exactly. Still have a handful of those sitting somewhere in the closet.

A wait() is not a good idea because it introduces recursion of unknown deep, depending from the users receive() implementation.

Yep, I thought the same. Something like (ARD * 15 / 2) seems like a good delay.

But exponential backoff is also not a good idea, because this can become a long delay of a second or more which blocks all time the main loop. But a short random delay of some 10th ms (like mentioned by @skywatch) worked for me in my setup.

Exponential backoff would be a nice feature for NodeManager, but that would only take care to deliver sensor data to the uplink (unless we ack data back all the time). This wouldn't help if the traffic jam occurs somewhere further up.

Another idea I thought of, would be listening if any carrier is present. (nRF24L01+ has that feature) and sending any packets only once there's no carrier + adding a random delay. This would hopefully avoid collisions, even with other things, like WiFi.

Okay, reporting in to the discussion thread from the github PR.

My setup is a bit different from @virtualmkr, since I don't have any N2N traffic in my network, all logic is handled by the controller (HomeAssistant). Summing up the details, here's what I've learned so far that affects packet loss on my setup.

1. Time to switch from receiving to sending. With a repeater (e.g. 328p @12Mhz) buffering RX packets before sending, this takes a while. Suggested solutions: - max out MY_RF24_SET_ARD register value, implement exponential backoff when sending. (How to actually implement it in mysensors code - that's a big question)

2. Interference from wifi/zigbee (including sidelobes!). This is pretty easy to solve, if you don't have too many neighbors. My DIY usb<-->nRF24L01 dongles can work by sweeping all available channels looking for carrier frequency. It works like a poor man's 2.4Ghz spectrum analyzer. Once looking at the picture, I moved my WiFi to 6 & 11 channels, and used the lower part of the spectrum for mysensors & zigbee, making enough space in-between for the sidelobes to fit in.

3. Switching several actuators in one node very fast. I have LED dimmers that have 4 channels. If I switch them all at once in home assistant - some packets are lost. My guess is that once 3 packets are inside the RX FIFO of the nRF24 subsequent packets are lost. And since the gateway is an STM32 it can send those waaay faster. Maxing out MY_RF24_SET_ARD register value may help, I still have to try it.

Hardware issues:

1. Crappy antennas/design on modules, suffering from 'magic finger' problem. I've written about these in my blog. Can be fixed with some manual labor: https://ncrmnt.org/2021/01/11/nrf24l01-manually-calibrating-the-antenna-with-mysensors-and-homeassistant/

2. Weather. Wet snow kills 2.4Ghz signals, if you're operating outside.

3. Antenna direction. Especially if the gateway has a PA+LNA module and an external SMA antenna.