@evb Hello. I am the maker of the Canique MK2 boards. I started just like you with the website from Andreas Rohner, desoldering the LED and desoldering the voltage regulator from such a chinese board until I realized this is the wrong way round.
As far as I recall, I've seen voltage regulators consume something in the order of 100uA when reverse powered. So that might be a hint.
The external crystal should not be drawing that much current. Using a 16MHz oscillator current can go as low as 4uA with watchdog enabled - in theory and on boards built with minimum consumption as a design goal.

To your question regarding the SCK pin: yes, if it is connected to a LED every clock pulse on the SCK pin (when SPI is enabled) will make the LED draw current.

You also have these kinds of troubles (SPI drawing too much current when active or inactive) with chinese boards having a BME280 on them for example. If you have high quality standards, the stock chinese boards won't fit your needs.

Here I am introducing part of my project that I've been working on for the last couple of years (since 2016/2017).

Meet the temperature/humidity sensor (picture showing a prerelease state - the sensor will be upgraded soon):

• +5 years battery lifetime (single AA alkaline battery) when sending every 30 seconds. (calculations rather suggest 7.5 years) Everything is highly optimized for low consumption. Almost no public library is used (most libraries don't offer superior quality).

• 400-800m range using 868 MHz carrier frequency

• <2 ppm RTC @ room temperature. Algorithmic temperature compensation is done to reduce RTC offset.

Meet the heat control system supporting Vaillant 7-8-9 interface (again a prelease version):

LEDs and button on the back are hidden in the picture.
The device is able to control the water temperature in Vaillant boilers (and other manufacturers using the same interface). It isn't just using on/off switching. It sets the water temperature between 30°C and 90°C.

• 5-28V input voltage
• output range 3V-24V
• ~6mV output resolution
• Automatic calibration to heating system voltage
• Automatic detection of stable or oscillating voltage at startup
• Many error checks to detect unconnected wire and the like
• Radio: 868 MHz
• Automatic off state when no radio message is received for 5 minutes
• expected power consumption in operating mode < 0.1W
• expected power consumption in standby mode < 0.05W
• button to switch between modes - device can be "disconnected" from heating system by pressing the button (gives control back to the heating system)

The 3rd part of the whole story is the gateway. There is no picture of the gateway yet. It's a complete mini-computer though (no ESPxxx and the like), offering ethernet and an MQTT API.
It's equipped with 4 cpu cores, 512MB RAM, and an external 868MHz antenna.

@tssk There is a name for this noise: it is called "coil whine". In German "Spulenfiepen". It stems from the inductor - I wouldn't know any way to get rid of it.
My mainboard or power supply on my PC creates similar noise when the CPU goes in certain doze modes.

@Puneit-Thukral You're mixing up the sensor node and the Vaillant heat control system. The minimum input voltage is 5V for the heat control system (only). This has nothing to do with the used MCU, but the 7-8-9 Vaillant interface usually works with 24V. So usually the input voltage is 24V anyway.
The interface for the Vaillant heat system is usually as follows:
24V input => roughly 11V to 17V output on the CTL pin to regulate the boiler temperature
If you used 5V on the input you'd only be able to regulate up to max. 5V.
The Vaillant heat control system (it's the second picture, the one with the box) does not run from batteries. It gets power directly from the Vaillant (or compatible) heating system.

The single AA battery temperature/humidity node runs from a single 0.8V-1.6V battery. You can see the battery in the picture - in this case it's an Eneloop although I'd recommend Alkaline batteries for devices that run for years. To run from a single battery the node has a boost circuit integrated that boosts the voltage to 1.85V.
On all devices (sensor nodes, heating control, gateway) I'm using a variant of ARM Cortex M0+ or Cortex M0. (Eventually it will all be Cortex M0+). This 32 Bit architecture is superior to Atmega328P (the typical MCU used in Arduinos) in every respect (be it power efficiency, pure performance, sleep current with wakeup timer, price to performance/flash size ratio, peripherals quality and quantity, etc etc.). Just 1 example: the used ADC is 12 Bit with hardware oversampling support (vs 10 Bit without oversampling on Atmega328)

@monte I'll sum it up:
Android App - might be open source in the future
Gateway - not open source, but open
Server Code (cloud) - no
Sensor Node - no
Heat Control - no

In the future the Android App might become open source. Anybody can build a UI, because the data needed for that is available from the gateway.

I wish I'd use my own design for the gateway but to finance that, I'd have to sell thousands of these devices because it's a very time intensive task requiring specialists. Currently the gateway is running on an Orange Pi Zero board with a custom hat. This is why it's the most expensive part. The communication Orange Pi <-> Hat is via USART.

This is what a Bosch freezer looks like when measured over 24 hours. It is set to cool down to -18°C. Measurement is done in the top drawer (afaik the coldest one)

Canique Climat mounted outside with 2 screws. Venting holes on the left and downside.

Uses a brand new onboard temperature/humidity sensor.

@NeverDie said in RFM69 sensitivity vs packet loss:

How is it that you're increasing your receive sensitivity?

When I say "sensitivity setting" I mean the RSSI threshold setting in register 0x29 as posted above.

@NeverDie said in RFM69 sensitivity vs packet loss:

RxReady just means that the RSSI Threshold has been met and that the SX1231 can start listening for a preamble. See figures on page 39 of the SX1231 datasheet

The bold part of your statement is plain wrong. RxReady means that enough preamble (or any other data like noise) has been received to read RSSI and optionally tune AGC/AFC. [This is why -if you set the wrong RSSI Threshold- you permanently get RxReady interrupts.]
The preamble is there to help adjust the AGC/AFC. That's the purpose of the preamble. So your preamble must be sent before the RxReady interrupt, not after it like you claim. Of course you can send the preamble after the RxReady interrupt (like you can pour soup over your salad) but then you have a potentially wrong AGC/AFC setting. It's just not meant to be.

@NeverDie said in RFM69 sensitivity vs packet loss:

It doesn't mean that that a matching preamble has been found, nor does it necessarily mean that a matching preamble will be found in the time allowed.

This is true. RxReady will also fire on noise. The formulas given on the page you refer to more often than not suggest ~14 bits of data for RxReady to fire (with AGC enabled, AFC disabled). This data can stem from noise. This is why it is important to keep the RSSI threshold above the noise floor. I'm not sure whether you got me right regarding my goal: I'm not trying to improve sensitivity. I'm trying to reduce packet loss to a minimum.

@NeverDie said in RFM69 sensitivity vs packet loss:

In packet mode, there's a fixed window between rxready and timing out in which to receive a valid packet, so a stream of rxready events could just mean a stream of timed-out receive windows triggered by ambient noise, not a series of received preambles.

Are you referring to RegRxTimeout2 (register 0x2B)? By default it is not set.
And iirc I only get this series of RxReady events if I restart the RX loop. Otherwise RxReady will stay set.

I have gathered data in a LibreOffice table...

Sensitivity vs RX Ready at different bit rates

The gist of it is:
@19.2 kbps I was able to achieve -105 dBm with low noise (low number of false RX Ready interrupts)

@25 kbps -104 dBm
@4.8 kbps -110 dBm
@40 kbps -99 dBm
@76.8 kbps -99 dBm

@alexelite I guess you'd have to reduce sensitivity to below -90 dBm, maybe -85 dBm, for things to work while TV is on.

Just to give a few numbers...

Condition: Bitrate 76800 bps / RX BW 125000

I could measure the following numbers:

Sensitivity | number of RX Ready per minute
-101 dBm | 4500
-100,5 dBm | 1300
-100 dBm | 1100
-99 dBm | 800

If set to -99 dBm, packet loss is small.
But today evening, the number of RX Ready events went up significantly. Packet loss went up too.

Reason: I switched on my television. Distance to Gateway ~30cm.
So a TV in the vicinity of the Gateway has a tremendous effect on sensitivity.

I ran a similar test some days ago with a sensitivity of -90 dBm and did not notice any packet loss then.

I am biased since I run that website but I can recommend https://www.canique.com/boost
I've never heard it making noise.

@Yveaux alas I can't due to lack of time. But maybe somenbody has spare time to tackle this.

@tssk There is a name for this noise: it is called "coil whine". In German "Spulenfiepen". It stems from the inductor - I wouldn't know any way to get rid of it.
My mainboard or power supply on my PC creates similar noise when the CPU goes in certain doze modes.

@Yveaux

There are 2 points I'm trying to make:

1. the sensitivity setting (register 0x29) is very important. Setting it too low makes AGC (and AFC, but it's rarely used) completely useless. One could even think of setting AGC manually instead.
2. The RFM69 chip is not smart enough to restart the RX procedure at the right point in time. It has some interrupts that tell you at which stage of the RX procedure it is currently. If you don't use these interrupts but follow the approach "I'll just wait for a Payload Ready IRQ" you will face packet loss.

There are multiple ways I can think of to fix this:

a) use the internal timeout setting (register 0x2B) of the RFM69 chip to set a timeout for the payload ready interrupt + listen to the RFM69 timeout interrupt (DIO0, DIO3 or DIO4 can be used) in your microcontroller. As soon as the timeout interrupt fires, the RX loop has to be restarted via register 0x3D (RegPacketConfig2).

b) same as above but more sophisticated: Instead of using the RFM69 internal timeout a timer of the MCU can be used for the task and more fine grained control is achievable via these interrupts (in the given order):
RxReady, SyncAddress, Fifolevel (you'd have to set Fifothreshold to some sane value like 3 bytes first), PayloadReady.
According to the bit time (called Tbit in the RFM69 documentation) every interrupt must occur within a certain time frame. E.g. if your SyncAddress is 2 bytes long (standard), the timeout must be at least 2x8xTbit time. That's the time needed to receive the SyncAddress after RxReady has fired. But it's a bit more complicated than that, because afaik RxReady can fire while still receiving the preamble (e.g. 5 byte preamble, but RxReady firing after 2 bytes). This is dependent on whether AFC/AGC are used or not and explaind in the documentation.
If there is a timeout detected at any stage -> RxRestart like in a).

Additionally [valid for a) + b)] the sensitivity has to be tuned in such a way, that there are not thousands and thousands of false preambles within a short time. Or to put it in other words: The sensitivity must be above the noise level.

The advantage of b) over a) is that the RX loop can be aborted at an earlier stage without having to wait the full "timeout" time.

Let's have a look @ some other setup.
Location: Vienna, close to center.
Acks: Not used.
Receiver: Orange Pi Zero with custom Gateway Hat with noise filtering (using mains power) with RFM69HW
Sender: Custom battery powered nodes with RFM69W
Same Radio settings as above, but sensitivity: -110 dBm

I have measured packet loss from 5 sender devices over a time window of ~ 16,5 hours.

Device | Received | Missed | Packet Loss %
A | 2000 | 28 | 1,38%
B | 1978 | 27 | 1,35%
C | 2002 | 45 | 2,20%
D | 1970 | 26 | 1,30%
E | 2000 | 25 | 1,23%

A 2nd test showed similar results ranging from 0,91% to 1,74%.

I repeated the same test, this time, with a battery powered node as receiver - just to make sure it has nothing to do with noise on the power line.
The results were even slightly worse: ranging from 1,3% to 2,41% packet loss.

Using the gateway as receiver again, I changed radio settings (BR 76,8 kbps, BW 125kHz, Whitening, changing sync word etc) but the figures were still not better.

So what happend when reducing sensitivity to -90 dBm? Still no substantial improvement.
Because RFM69 libraries are broken by design in a 2nd manner. It has to do with the way the transceiver works when listening for packets. It goes through multiple stages.
The problem is: If you only wait for "Payload Ready" the receiver can be caught in the wrong stage, just at the very moment when your sender is transmitting the message - in this case the receiver will miss the packet, because it was waiting for the end of the packet of the "wrong" sender.

This is the reason for the packet loss.
After fixing this error by introducing timeouts, I have achieved the following figure:

Received | Missed | Packet Loss %
2745 | 7 | 0,25%

Same environment, same receiver, one the the same senders, same message frequency.
Before I had missed packets every hour. At maximum I would have 1 hour without missed packets. Now 5-6 hours can pass without a single missed packet. All figures are without ACKs/retransmissions.