Thanks a lot! that's the only thing I haven't tried... I assumed that since the raspberry pi was working fine, the internal power regulator was providing enough juice to the module through the gpio pins...

This is quite weird, I replaced the power supply, everything started worked fine. So I said, let's try the old power supply again, it should be broken... Everything is working fine too... This is quite insane.

Also, I found https://www.raspberrypi.org/forums/viewtopic.php?t=190584 and I ran that in my rpi, and I get 0x0 with both power supplies

# vcgencmd get_throttled
throttled=0x0

According to https://forum.libreelec.tv/thread/17860-how-to-interpret-rpi-vcgencmd-get-throttled/, 0x0 means all is fine with the power supply, so not sure what is going on...

I will leave it running for some hours and see what happens...

@skywatch Yeah, I have spent a lot of time fighting those as well, however in my case, I have designed a special PCB, which connects directly to the GPIO pins in the rPi, which should avoid these issues:

As per the power supply, I have been using a PSU from the same people I buy the rPi from, which is a 5VCC, 3.1A, for months until this started happening. After @Yveaux suggestion, I tried my pixel charger, which should deliver 5V - 3A, and as I said, it worked fine for about 10 hours straight, but then stopped working again. Furthermore, the rPi is not reporting undervoltage or throttling, so not sure if it is the PSU at all... I don't want to take appart the PSU, but if there are no more ideas on what else might be, I guess I'll have no choice...

@skywatch Ahhh, makes sense now thanks!

@ejlane I got the specifications from the PCB manufacturer (Attaching below), and the copper thickness is 35 microns...
I managed to get the module with the antenna pin right next to the connector, so the trace length would be just 4mm, for what I was able to find, my case for a single sided PCB would be a "Coplanar wave guide" and plugging all of those values in a calculator I found in KiCAD, it seems that I am fine with a 1 mm wide trace, with a 0.25mm gap with the GND around it.

Hope I got it right! Thank you for your suggestions!

@ejlane @Yveaux @skywatch So, I have found exactly what was going on, and had nothing to do with the hardware... Basically, one of my nodes was jamming the radio channel

How I discovered the issue:
There are 2 more flags, besides MY_DEBUG which are MY_DEBUG_VERBOSE_RFM69 and MY_DEBUG_VERBOSE_TRANSPORT_HAL which I couldn't find a way to add, so I did it by editing the configure script and where it says

if [[ ${debug} == "enable" ]]; then
    CPPFLAGS="-DMY_DEBUG $CPPFLAGS"
fi

I changed it for

if [[ ${debug} == "enable" ]]; then
    CPPFLAGS="-DMY_DEBUG -DMY_DEBUG_VERBOSE_RFM69 -DMY_DEBUG_VERBOSE_TRANSPORT_HAL $CPPFLAGS"
fi

Then re-ran the script and recompiled the mysgw binary. The important flag in my case was MY_DEBUG_VERBOSE_RFM69.

When mysgw was running, and everything was working fine, I was seeing lines with DEBUG RFM69:CSMA:RSSI=-102 popping in once or twice (with different values for RSSI) and everything continued normally. Then, when the whole thing was wedged and nothing worked, I noticed that message pretty much going on forever

Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-53
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-53

And on and on and on.... forever... Clearly there was a loop somewhere for whatever reason printing this message. As it turns out, there's a function that the RFM69 driver calls before sending a message https://github.com/mysensors/MySensors/blob/2e00bf6a10f76d6aaa1999e12313237bc3edabd3/hal/transport/RFM69/driver/new/RFM69_new.cpp#L369-L375 that pretty much checks if there is noise in the RF channel, before actually sending anything...
I was doing some modifications to one node I have, where I replaced the arduino with another one, which I thought was a 3.3v one, burned it as so, but it was a 5v. For whatever reason (maybe the different clock speed? dunno...) when doing that, the node will send garbage through the radio... I have no idea how it manages to init the radio... doesn't really matter, the thing was sending something in the same channel constantly...

And that is why the gateway was receiving data from other nodes, but was never able to send back... this other node was preventing it because it would never shut up... I have now replaced the arduino with a 3.3v one, checked in the serial monitor after burning and re-installed, and everything is back to normal and working...

Thank you all for your help and suggestions, I will nevertheless build the new PCB with the suggested changes and replace the ones I am currently using.

@mfalkvidd Totally missed that! sweet !

@ejlane @Yveaux @skywatch So, I have found exactly what was going on, and had nothing to do with the hardware... Basically, one of my nodes was jamming the radio channel

How I discovered the issue:
There are 2 more flags, besides MY_DEBUG which are MY_DEBUG_VERBOSE_RFM69 and MY_DEBUG_VERBOSE_TRANSPORT_HAL which I couldn't find a way to add, so I did it by editing the configure script and where it says

if [[ ${debug} == "enable" ]]; then
    CPPFLAGS="-DMY_DEBUG $CPPFLAGS"
fi

I changed it for

if [[ ${debug} == "enable" ]]; then
    CPPFLAGS="-DMY_DEBUG -DMY_DEBUG_VERBOSE_RFM69 -DMY_DEBUG_VERBOSE_TRANSPORT_HAL $CPPFLAGS"
fi

Then re-ran the script and recompiled the mysgw binary. The important flag in my case was MY_DEBUG_VERBOSE_RFM69.

When mysgw was running, and everything was working fine, I was seeing lines with DEBUG RFM69:CSMA:RSSI=-102 popping in once or twice (with different values for RSSI) and everything continued normally. Then, when the whole thing was wedged and nothing worked, I noticed that message pretty much going on forever

Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-53
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-52
Sep 09 18:37:11 DEBUG RFM69:CSMA:RSSI=-53

And on and on and on.... forever... Clearly there was a loop somewhere for whatever reason printing this message. As it turns out, there's a function that the RFM69 driver calls before sending a message https://github.com/mysensors/MySensors/blob/2e00bf6a10f76d6aaa1999e12313237bc3edabd3/hal/transport/RFM69/driver/new/RFM69_new.cpp#L369-L375 that pretty much checks if there is noise in the RF channel, before actually sending anything...
I was doing some modifications to one node I have, where I replaced the arduino with another one, which I thought was a 3.3v one, burned it as so, but it was a 5v. For whatever reason (maybe the different clock speed? dunno...) when doing that, the node will send garbage through the radio... I have no idea how it manages to init the radio... doesn't really matter, the thing was sending something in the same channel constantly...

And that is why the gateway was receiving data from other nodes, but was never able to send back... this other node was preventing it because it would never shut up... I have now replaced the arduino with a 3.3v one, checked in the serial monitor after burning and re-installed, and everything is back to normal and working...

Thank you all for your help and suggestions, I will nevertheless build the new PCB with the suggested changes and replace the ones I am currently using.

@skywatch Ahhh, makes sense now thanks!

@ejlane I got the specifications from the PCB manufacturer (Attaching below), and the copper thickness is 35 microns...
I managed to get the module with the antenna pin right next to the connector, so the trace length would be just 4mm, for what I was able to find, my case for a single sided PCB would be a "Coplanar wave guide" and plugging all of those values in a calculator I found in KiCAD, it seems that I am fine with a 1 mm wide trace, with a 0.25mm gap with the GND around it.

Hope I got it right! Thank you for your suggestions!

@ejlane Well, I am a software developer myself (Not engineer, and certainly not RF engineer) And by looking at my design it is clear that I can barely consider myself an electronic hobbyist

Thanks a lot for your suggestions, This is a very basic one-sided PCB I designed in order to avoid having wires, which I thought was worse... I will redesign the whole thing and will post back (If you have more suggestions, I will be more than pleased to know)

@skywatch I am trying to make this as simple as possible, and to be able to fit inside a rPi case, so I will try to get the trace as short as possible between the module and the SMA for now, and see how it goes. As per your AMS1117 suggestion, I am assuming you mean instead of getting 3.3v pin from the GPIO, use the 5V one and go through an AMS1117 to get 3.3v, right? Also, about the capacitor, may I ask what exactly is the "neg board" ?

@skywatch I found a way to enable flags to get more debug info both for the RFM69 as for the Transport HAL in order to try and get more info on what the software see when it is broken, and the thing has been working fine for 2 days straight now... Go figure... So I am wondering (Attaching the custom PCB design below):

Some questions:

1. I have highlighted the via for the antenna, could it be too long and it was maybe picking up some interference? I am thinking on moving the RFM69 module to be up there, as close as possible to the SMA connector as possible...

2. What do you think about the via thickness? should I make it wider? it is 0.4mm right now

3. Is it ok that the 4 corners for the SMA connector are connected to GND? I believe it is ok, right?

@skywatch Heheh, yeah, a friend gave it to me in a software conference back in 2014, very useful when testing you clamped everything fine a couple of rooms away

it does sound as if something is changing over time, that is why I suspect power supply, cables or connections.

Yeah, it is weird though since the rPi has been in the same table for months... it is even plugged to an APC UPS, so there should be no outages or anything in the main lines messing with the PSU.

I guess you checked for dry joints on the pcb, vut it would be worth checking ecery pin has a firm connection with a multimeter and a spare dupont pin.

Yup, I have 4 PCB's built in the exact same way, 4 RFM69HW and 4 antennas. I have tried each 4 with the same rPi and with a spare one I have... same thing... The only thing I haven't tried yet (which I guess I need to do) is formatting the SD card and starting from scratch... I really don't see why this would fix it, since it has been running the exact same software since... I haven't even updated packages (I have done so since the issues started occurring, to no avail)

Is the radio module connected to 5V on pi or 3.3V? this is another possible area to look at but I need to know this first...

It is connected exactly as described in https://www.mysensors.org/build/raspberry#wiring for the RFM69, so 3.3V.

One thing with this that I am not 100% sure, is that there are 2 flags described in https://www.mysensors.org/apidocs/group__RaspberryPiGateway.html the --my-rfm69-irq-pin and --my-rfm69-cs-pin I am not sure if I should specify a pin for this when wiring as suggested... And if I do, I don't know which pins are CS and IRQ in the RFM69.

I put another question a couple of months ago in https://forum.mysensors.org/topic/11716/having-problems-with-rfm69hw-in-raspberry-pi but the problem eventually went away (not sure why), so it seems they are not needed. (Also I switched from MQTT to Ethernet)

finally, with good dupont cables, remove your pcb from the pi and connect it again using dupont wires 150mm or more away from the pi and try that. I am wondering if the RF path to the antenna being close to the gpio and cpu might perhaps be possible culprit. Is there any screening on the back of the pcb ?

No, this is a 1 sided PCB. I will try your suggestions, hopefully tonight, since cannot right now. Sounds kind of weird that the antenna path would be an issue since it has been working fine for months, with a range that went over my expectations (More than 30m away, behind 4 walls)... One thing I am not sure if I built fine in the PCB, you can see that the RF Connector has the center connection to the antenna path going to the "ANA" in the RFM69 chip, however the 4 angles of the RF connector, are soldered to GND, my understanding is that this is correct, can you confirm?

@skywatch Yeah, I have spent a lot of time fighting those as well, however in my case, I have designed a special PCB, which connects directly to the GPIO pins in the rPi, which should avoid these issues:

As per the power supply, I have been using a PSU from the same people I buy the rPi from, which is a 5VCC, 3.1A, for months until this started happening. After @Yveaux suggestion, I tried my pixel charger, which should deliver 5V - 3A, and as I said, it worked fine for about 10 hours straight, but then stopped working again. Furthermore, the rPi is not reporting undervoltage or throttling, so not sure if it is the PSU at all... I don't want to take appart the PSU, but if there are no more ideas on what else might be, I guess I'll have no choice...

Alright... now the GW has stopped working again... throttled still shows 0x0, and switching to another power supply doesn't fix it

Thanks a lot! that's the only thing I haven't tried... I assumed that since the raspberry pi was working fine, the internal power regulator was providing enough juice to the module through the gpio pins...

This is quite weird, I replaced the power supply, everything started worked fine. So I said, let's try the old power supply again, it should be broken... Everything is working fine too... This is quite insane.

Also, I found https://www.raspberrypi.org/forums/viewtopic.php?t=190584 and I ran that in my rpi, and I get 0x0 with both power supplies

# vcgencmd get_throttled
throttled=0x0

According to https://forum.libreelec.tv/thread/17860-how-to-interpret-rpi-vcgencmd-get-throttled/, 0x0 means all is fine with the power supply, so not sure what is going on...

I will leave it running for some hours and see what happens...

Hey everyone, I have been using a Raspberry Pi 3 as a gateway, and have several sensors around the house built with Arduino. Everything has been working fine for months, until 2 days ago, out of the blue, everything stopped working.

I am using the RFM69W radio in the Arduino nodes, and the RFM69HW in the gateway. For what I can tell, everything is the same between the radio modules, with the difference that the HW variant supports using higher power, so I figured I would use this in the gateway, and the W variant in the nodes, since some are battery powered, in order to save battery. For months, this has been working flawlessly, and suddenly it stopped. It appears to be happening at the gateway, since it is unlikely 10 sensors have all stopped working at the same time. I have tried changing the RFM69HW module, changing the antenna, and even changing the Raspberry Pi altogether... nothing works. I have been using the development branch of mysensors, which has worked fine thus far, and I also have tried using the 2.3.2 release... same issue... Of course, doing a clean build and removing the /etc/mysensors.eeprom between tests...

I have built a simple "ping-pong" node, which I use to test coverage around the house. The arduino code is super simple, it only calls

bool rcv = send(msg.set(seconds), 1);

to get an ACK. If it does it is counted as success, if not, it is counted as failure, it loops endlessly, sleeping 0.5 Seconds between sending, and it will show results in an LCD display, so I can move around the house. This has worked fine when I was deploying the different sensors around the house.

After compiling the mysgw in the rPi, I started and I see this log

root@openhab:/home/pi/MySensors# ./bin/mysgw                                                                                                                                                              
Sep 04 21:13:56 INFO  Config file /etc/mysensors.conf does not exist, creating new file.                                                                                                                  
Sep 04 21:13:56 INFO  Starting gateway...                                                                                                                                                                 
Sep 04 21:13:56 INFO  Protocol version - 2.3.2                                                                                                                                                            
Sep 04 21:13:56 INFO  EEPROM file /etc/mysensors.eeprom does not exist, creating new file.                                                                                                                
Sep 04 21:13:56 DEBUG MCO:BGN:INIT GW,CP=RPNGL--X,FQ=NA,REL=255,VER=2.3.2                                                                                                                                 
Sep 04 21:13:56 DEBUG TSF:LRT:OK                                                                                                                                                                          
Sep 04 21:13:56 DEBUG TSM:INIT                                                                                                                                                                            
Sep 04 21:13:56 DEBUG TSF:WUR:MS=0                                                                                                                                                                        
Sep 04 21:13:56 DEBUG TSM:INIT:TSP OK                                                                                                                                                                     
Sep 04 21:13:56 DEBUG TSM:INIT:GW MODE                                                                                                                                                                    
Sep 04 21:13:56 DEBUG TSM:READY:ID=0,PAR=0,DIS=0                                                                                                                                                          
Sep 04 21:13:56 DEBUG MCO:REG:NOT NEEDED                                                                                                                                                                  
Sep 04 21:13:56 DEBUG Listening for connections on d~:5003                                                                                                                                                
Sep 04 21:13:56 DEBUG MCO:BGN:STP                                                                                                                                                                         
Sep 04 21:13:56 DEBUG MCO:BGN:INIT OK,TSP=1                                                                                                                                                               
Sep 04 21:13:56 DEBUG TSM:READY:NWD REQ                                                                                                                                                                   
Sep 04 21:13:56 DEBUG ?TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=20,pt=0,l=0,sg=0,ft=0,st=OK:

Everything looks good... Now I turn on my ping-pong node, and I see this:

Sep 04 21:14:06 DEBUG TSF:MSG:READ,220-220-255,s=255,c=3,t=7,pt=0,l=0,sg=0:                                                                                                                               
Sep 04 21:14:06 DEBUG TSF:MSG:BC                                                                                                                                                                          
Sep 04 21:14:06 DEBUG TSF:MSG:FPAR REQ,ID=220                                                                                                                                                             
Sep 04 21:14:06 DEBUG TSF:PNG:SEND,TO=0                                                                                                                                                                   
Sep 04 21:14:06 DEBUG TSF:CKU:OK                                                                                                                                                                          
Sep 04 21:14:06 DEBUG TSF:MSG:GWL OK                                                                                                                                                                      
Sep 04 21:14:10 DEBUG !TSF:MSG:SEND,0-0-220-220,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=NACK:0                                                                                                                
Sep 04 21:14:11 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1                                                                                                                               
Sep 04 21:14:11 DEBUG TSF:MSG:PINGED,ID=220,HP=1                                                                                                                                                          
Sep 04 21:14:15 DEBUG !TSF:MSG:SEND,0-0-220-220,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=NACK:1                                                                                                               
Sep 04 21:14:15 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=15,pt=6,l=2,sg=0:0100                                                                                                                            
Sep 04 21:14:19 DEBUG !TSF:MSG:SEND,0-0-220-220,s=255,c=3,t=15,pt=6,l=2,sg=0,ft=0,st=NACK:0100                                                                                                            
Sep 04 21:14:20 DEBUG TSF:MSG:READ,220-220-0,s=255,c=0,t=17,pt=0,l=5,sg=0:2.3.2                                                                                                                           
Sep 04 21:14:21 DEBUG TSF:MSG:READ,220-220-0,s=255,c=0,t=17,pt=0,l=5,sg=0:2.3.2                                                                                                                           
Sep 04 21:14:22 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=6,pt=1,l=1,sg=0:0                                                                                                                                
Sep 04 21:14:24 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=6,pt=1,l=1,sg=0:0                                                                                                                                
Sep 04 21:14:25 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=6,pt=1,l=1,sg=0:0
Sep 04 21:14:28 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=11,pt=0,l=14,sg=0:Ping-Pong Node
Sep 04 21:14:29 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=11,pt=0,l=14,sg=0:Ping-Pong Node
Sep 04 21:14:31 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=11,pt=0,l=14,sg=0:Ping-Pong Node
Sep 04 21:14:32 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=12,pt=0,l=3,sg=0:1.0
Sep 04 21:14:34 DEBUG TSF:MSG:READ,220-220-0,s=255,c=3,t=12,pt=0,l=3,sg=0:1.0
Sep 04 21:14:35 DEBUG TSF:MSG:READ,220-220-0,s=0,c=0,t=36,pt=0,l=0,sg=0:
Sep 04 21:14:36 DEBUG TSF:MSG:READ,220-220-0,s=0,c=0,t=36,pt=0,l=0,sg=0:
Sep 04 21:14:38 DEBUG TSF:MSG:READ,220-220-0,s=0,c=0,t=36,pt=0,l=0,sg=0:

So, the node is sending data just fine, and the GW is receiving data fine, and the problem occurs when the GW attempts to send data back to the node... Now, this could mean the RFM69HW module broke down, or the antenna is broken, or even that something in the rPi GPIO pins or whatever broke down... but being that I have pretty much replaced every single part of it with brand new stuff... brand new rpi, brand new RFM69HW, brand new antenna... and I continue to experience this... Does anyone have ANY suggestion on what to try next? I am a bit clueless here... and any advice is greatly appreciated.

Thanks all,
Franco