nRF24L01+ Communication Failure: Root Cause and “Solution”

[Yveaux]

@ben036 I would certainly like to have a look at your design files.

You could give Ebyte modules a try : https://forum.mysensors.org/topic/9668/cdebyte-s-new-nrf24-modules-are-great-and-cheap
I have very good experience with these, and are probably using authentic nrf24's

[Ben036]

@yveaux

I've uploaded our Eagle PCB files as well as a few PDFs/PNGs of the schematic/layout for quick viewing.

> > > Files Here < < <

Thanks for the tip on the other nRF24 board to try.

[odritter]

@yveaux My statement is based on the assumption that everything can interfere with your desired signal. It's just a matter of how much. It's very possible that genuine nRF boards are less susceptible to the interference we have shown. However, all the nRF boards we have are suspected clones so I do not have a way to direct compare clone vs genuine (unless we purchase from the Ebyte link you provided).

I have provided a link below to a public application note I came across that provides some background information on sources of interference and how it impacts your desired signal/receiver. Unfortunately it gets very technical at times and assumes a large amount of base RF knowledge so its usefulness may be limited. Nonetheless, I thought I would share if anyone was interested.

Testing and Troubleshooting Digital RF Communications Receiver Designs

The nRF uses a sampled IF receiver as described in section 1.2.2 and similar
to what is shown in figure 4. Section 3.2.2 covers some interfering signals but mostly talks about more complex modulation schemes than the nRF uses.

I went back and did an additional experiment using only our Gateway and the Software Defined Radio (SDR) to measure interference on something other than our sensor PCB.

My setup:
Gateway=RaspberryPi 3 with cabled nRF24L01+ hanging out in free space
SDR=USRP B210 with 2.4GHz Vertical antenna
Gateway and SDR placed about 2 feet apart in the middle of a room

Gateway transmitted 16 packets for both graphs. The only difference between the two graphs: upper graph constantly polls TX status where the lower graph uses the IRQ line to wait for completion. The physical hardware setup was unchanged between the two measurements.

TX Status Polling

IRQ Wait

These data Eye patterns were taken with a single nRF board (in contrast, the Eye patterns included in the original post are a composite of 10 different nRF boards)

[odritter]

@skywatch said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

@odritter Thanks for the clarification! - I learn something new (again)... :)

My question was based on the fact that if enough energy is present in the spi signal to imprint on the Tx output, then it would be even worse for the Rx side as the levels of signal received would be much lower than anything transmitted.

It certainly is possible the Rx side could be even worse. However, it strongly depends on where the interference is coming in. For instance, if the interference happened before the Low Noise Amplifier (LNA) then it would very likely be just as bad or even worse. However, if the interference happened after the LNA it may not have anywhere near as much impact since the signal should be much larger. Though, if the interference impacted the digitization (not really shown in the block diagram explicitly) then it would likely have a significant impact.

Thinking about the TX side of things, the interference is very likely occurring before the modulator (to the left of the PA) since I can "see" the SPI interference with the SDR after it has been demodulated. I have included the nRF block diagram from the datasheet below for reference.

[alowhum]

@yveaux said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

@odritter What puzzles me is that the interference in image 6 apparently isn't always present (see image 7), although the behavior of the MySensors stack is always identical (see image 8 and the matching code snippet 1).
Do you see a pattern when it is present and when not?

Also, switching to using the nRF24 interrupt line will break MySensors for a lot of (existing) boards, that don't have the IRQ line connected.
So, if we decide to add a silent period to the stack, we also need a non-IRQ implementation based on e.g. a delay.

Indeed, the RF Nano has no connected IRQ pins (and uses the NRF24 non-plus version..).

Although it may be unrelated, it's fascinating that this is another instance of MySensors being 'too fast', and 'needing a delay' to work better.

[Yveaux]

@alowhum said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

MySensors being 'too fast', and 'needing a delay' to work better.

Let's not jump to conclusions too fast!

So far there's only measurement data of one case where reducing SPI traffic during TX appears to reduce noise; this makes it very likely that the issue is hardware related (SPI lines run directly below the nRF24 board, the nRF24 board is placed next to the Pro Mini SPI lines (pins 10-13) and nRF24 modules are of questionable origin).
The fact that increasing the distance between nRF24 and board by jumper wire improves the transmission success rate is a clear indication this issue is hardware related.

Still, reducing potential disturbances during transmission and reception is important, especially if all MySensors users can potentially benefit from it with no or minimal penalty.

The MySensors stack however needs to support different hardware variations, with and without IRQ line connected.
The TX case with IRQ connected is trivial to solve, when IRQ is not connected is less trivial IMHO.
When using nRF24 with hardware ACK the actual TX time is not known upfront due to potential retries; we can only calculate a lower and upper bound.
Always silencing SPI communication for the upper time bound makes the node slow, shorter wait times require polling the nRF24 over SPI, thus introducing potential disturbance. The patch by @tekka reduces the polling rate and limits the upper bound (the number of hardware ACK retries), still it delays the TX time compared to the latest 2.3 release implementation.
Another solution could be limiting the SPI clock rate during the TX polling time, which also potentially improves RX polling.

As always, there is no one size fits all solution ;-)

@odritter MySensors Core team has a number of SDRs and genuine nRF24's at their disposal. Do you have a description how to derive the eye pattern so others can replicate your setup?

[alowhum]

@Yveaux Point taken.

I also agree that it should still work on non-IRQ scenario's, as that's pretty much my scenario now that I switched to the RF Nano.

Out of curiosity: what would be 'slow' in this scenario? Are we talking milliseconds? Hundreds?

And could this issue be related in any way?

[Yveaux]

@alowhum said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

Out of curiosity: what would be 'slow' in this scenario? Are we talking milliseconds? Hundreds?

As always; it depends on a number of things, e.g. radio bitrate, maximum number of retries, message length. I would have to do some calculations to get an indication.

And could this issue be related in any way?

No. I suspect #941 and #1134 are related.

[scalz]

@alowhum
for older nodes, it would need retrocompatibility, but imho, when you get started in MySensors, it's bad hardware choice to not connect the radio irq signal. It's also mentioned in MySensors doc it's better to use it.

Why someone wouldn't like better communication for his HA??
to be precise why someone in 2019 would like to use nrf24+low 328p ? :japanese_ogre: :smile:

With irq events instead of always-polling, you get queue management for messages. And also smoother processing, better states sync, more reactive just in time, not delayed or at the wrong moment, using resources only when needed.

So,to start out on the right foot , I would strongly advice people/newcomers to connect irq when possible, and for new nodes (it's mandatory for rfm69).

[odritter]

@yveaux I can certainly provide details for others to replicate. For completeness I am including details beyond just Eye diagram creation. See below

My setup:

• nRF24L01+ (clone)
  ** #define MY_RF24_PA_LEVEL RF24_PA_HIGH
  ** #define MY_RF24_DATARATE RF24_250KBPS
  ** #define MY_RF24_CHANNEL 30
  ** #define MY_RF24_IRQ_PIN 4
• SDR: USRP B210 with 2.4GHz vertical antenna
  ** 5MHz bandwidth centered at 2.43GHz (for channel 30)
  ** Eye diagrams require oversampling to build a meaningful eye. I would recommend a minimum 10x oversampling if possible (5MHz BW at 250KBPS yields 20x)
• Processing:
  ** Capture waveform with desired transmission (my setup can't trigger on anything specific, so I looked at signal power to trim down the record)
  ** Detect and remove carrier frequency offset - I used "0xAAA8" as a sync word
  ** Perform FM demodulation normalizing to expected frequency shift (±160kHz at 250KBPS)
  ** Find the center of the eye - this is done by looking for the location with the highest correlation to the sync word (or can be manually found)
  ** To build the Eye diagram, "chop" up the waveform at the crossing points which are located at ±(0.5*OversampleRate) samples from Eye center (±10 samples in my case)
  ** Overlay resulting waveforms on top of one another to form the Eye diagram

See image below for demonstration. Dashed red lines are where the waveform should be chopped:

I used LabVIEW with the Modulation Toolkit to perform my measurements. I can provide my code if helps though I suspect you probably use GNU Radio. I can provide additional details if something isn't clear or I left something out.

What metric do you currently use to measure link quality?

[Yveaux]

@odritter said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

I suspect you probably use GNU Radio.

That's correct. To be frank, I was hoping for a simple gnu radio script that I could just run to get the pattern out. There is manual work and LabVIEW involved (which I do not have access to) so it would take some experimentation to get the same results.

Thanks for your description though!

At the moment I don't have a way to measure signal quality accurately for nrf24. Of course there's the transmission retry counter, but I found that in general it either requires a few retries at most or goes up to the maximum and fails.

[alowhum]

@scalz said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

Why someone wouldn't like better communication for his HA??
to be precise why someone in 2019 would like to use nrf24+low 328p ?

While I get your point, in my case there were a number of reasons:

• A Chinese Nano is 2 euro. A good radio is 3 euros. An expansion board that connects them is 1,50 euro. That's a total of 6,50 for the 'starting point' of all my nodes. And then it uses a old fashioned mini USB connector. If I upgrade to a Nano with micro-USB connector costs increase at least a euro or the RobotDyn version.
• An RF Nano is 3,5 euros, and a basic Nano Expansion Board is 1 euro. That's a total of 4 euros.

That's almost half the price. And if you use a lot of them in workshops, it adds up.

But wait, there's more.

• The RF Nano + board forms a smaller package so it can fit in a smaller housing. It's all-in-one nature means it's less complicated (I give workshops)
• It's less fragile (eByte modules are thin and stick out).
• The RF Nano has a micro USB connector which everybody already has cables for. And the cables are ubiquitous and cheap.

I understand the RF Nano version has less range (doesn't support 250KBPS), doesn't have its IRQ pin connected and is 'fake'. But... it works just fine.

In case you were asking "why not use NRF52/4": it's not beginner friendly.
In case you were asking "why not RFM69": there is no cheap beginner friendly version (preferably one where you can just connect Dupont wires too, with lots of 5V and GND pins).

[alowhum]

@yveaux said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

No. I suspect #941 and #1134 are related.

https://github.com/mysensors/MySensors/issues/941
https://github.com/mysensors/MySensors/issues/1134

Ah, interesting. Makes sense.

[Yveaux]

@odritter I rebuilt my SDR/GnuRadio setup and did a few initial measurements:

Although the on-air data is clearly visible, the samplerate (2.4MHz) to create a usable eye pattern is just too low I'm afraid.
My setup is a cheap RTL2832U RTL-SDR dongle, with BT-281 downconverter, as described here: http://blog.cyberexplorer.me/2014/01/sniffing-and-decoding-nrf24l01-and.html , no match for your USRP B210 :white_frowning_face:

[boozz]

@scalz

That would mean IRQ on pin 2 (pro-mini or nano)?
Could be a wire-bridge for a lot of my boards I think... :thinking_face:

[Yveaux]

@boozz said in nRF24L01+ Communication Failure: Root Cause and “Solution”:

That would mean IRQ on pin 2 (pro-mini or nano)?

2 or 3, and it should work for other interrupt pins too (yet untested).

[boozz]

@yveaux

Ok, great! I presume it requires me to set this pin somewhere in the #define section? Could you hint me what variable is used for that in the config section? I guess the MY_RFM69_IRQ_PIN is not the proper one.

About the other interrupt pins: That's an interesting one. I only know of pin 2 and 3 being the interrupt pins on a pro-mini / nano. Is it possible to use other pins for that? If so, how to achieve that?

[mfalkvidd]

@boozz MY_RF24_IRQ_PIN

[boozz]

@mfalkvidd
Thanks!
I guessed something like that, but couldn't find it in the API section

[Sergio Rius]

I've been having range issues so I gave this approach a try. I've been testing the @odritter solution along with the @tekka improvements and while the no interrupt code is totally operative, the proposed is not. At least on my network. I'll be reviewing it as I think it may be some bug on the coding.

That's:

	do {
#ifdef MY_RF24_IRQ_PIN
		RF24_DEBUG(PSTR("RF24:TXM:IRQ_PIN_WAIT\n"));
		RF24_status = hwDigitalRead(MY_RF24_IRQ_PIN);
	} while (RF24_status && timeout--);
        // timeout value after successful TX on 16Mhz AVR ~ 65500, i.e. msg is transmitted after ~36 loop cycles
        RF24_status = RF24_getStatus();
#else
		delayMicroseconds(MINIMAL_AIR_TIME_US + len * AIR_TIME_BYTE_US);
		RF24_status = RF24_getStatus();
	} while (!(RF24_status & (_BV(RF24_MAX_RT) | _BV(RF24_TX_DS))) && timeout--);
#endif