MQTTClientGateway broken after upgrade - signature failure

[Anticimex]

@tomkxy I use rf24 with default settings (except some moved io pins) and a PA-enhanced radio on the GW. But getting the rf24 to behave can be tricky. And the larger the message, the trickier it gets. Unfortunately this means it gets trickiest with signing enabled as it makes most messages very large (thus making them more sensitive to rf disturbance). Unfortunately, it is not much I can do about it from a signing perspective. Reducing the message size for signatures and nonces or adding fault tolerance to the security messages compromises security quality, and I already to that with the truncation of the signatures (due to rf24 limitations) so I don't want to "nerf" if further. So the signing solution is quite rf sensitive. But the way I see it, that just serves as a good measure for the overall quality of the rf network. If it works with shorter messages, sooner or later, maybe you add a node that transmits longer messages and start to get issues. With signing enabled, you are forced to root out any lingering rf issues immediately, and is saved from unpleasant surprises later on. But of course I understand the frustration, having experienced it myself several times. But st=fail is not a signing problem, it is a rf problem. So I am afraid I am not the best resource to provide answers. @tekka has made an excellent pull request where he has optimized the rf24 stack significantly. Perhaps applying it could help solve your rf issue: https://github.com/mysensors/Arduino/pull/392

[tomkxy]

@Anticimex I fully agree to your judgement that the problem is not due to the signing as such. However, I also do not believe in the RF24 issue. I tried with a CUSTOM child and using the max payload size available which without signing went through.

I think there might be an issue related to the changed code pathes which is caused by "injecting" signing or a define or whatsoever in my sketch is wrong after the upgrade.

I tried @tekka pull request with the result that nothing arrived at the gateway at all.

Do you know why hardware signing is not supported in the MQTTClientGateway?

[Anticimex]

@tomkxy Both hardware and software signing has no knowledge about MQTT. They only handle signatures of messages passed between gw and nodes. How the gw communicates with controller is irrelevant. Unless MQTT messes up how gw adresses nodes, I cannot see how signing could not work for MQTT. And if it does, it is a bug in MQTT implementation and not signing.

[tomkxy]

@Anticimex I did not want to suggest that it is a bug in signing, I just referred to the comment in the W5100MQTTClientGateway sketch saying "Hardware SHA204 signing is currently not supported" and was wondering whether you know why.
Sorry for bothering you on that. As I said I agree that the problem must somehow be related to transmission.

[hek]

@tomkxy said:

"Hardware SHA204 signing is currently not supported

Hmm.. must be a copy-paste error. I cannot recall any reason why it wouldn't be supported.

[Anticimex]

@tomkxy it does? That's a surprise for me. I see no reason for why it should not be supported. Perhaps the guy who did the initial implementation of it did not have atsha204 on the existing target hw, but I cannot imagine a reason for it not being supported for MQTT. Currently the only target architecture that does not support hw signing is esp I believe. At least I do not think it will work as the low level io looks different and the driver is not adapted to handle it.

[Anticimex]

It is interesting that you could get your own "big" message through, but it could be depending on how it actually look. A nonce is pure random and therefore more sensitive to noise (perhaps, I'm not an expert on rf). And I have no knowledge of MQTT at all either but if it affects signing, I probably need to read up on it. @hek perhaps know if I need to :) I have assumed with signing support that a gw communicates with all nodes the same way, MQTT or not.

[hek]

@Anticimex said:

I have assumed with signing support that a gw communicates with all nodes the same way, MQTT or not.

Yes, correct. The used gateway transport shouldn't affect anything.

[tomkxy]

@hek @Anticimex Ok. I switched to hardware signing. HMACs are being generated. So that is ok. When I yesterday saw the issue with the nonce and the comment in the header I switched to soft signing assuming unsupported hardware signing could have caused that.

@hek what is exactly the define

/ W5100 Ethernet module SPI enable (optional if using a shield/module that manages SPI_EN signal)
#define MY_W5100_SPI_EN 4  ```

for. I am using an Arduino Mega and I remember that with my old sketch I did not need to use Soft SPI. Is this define on a Mega necessary and if yes pin 4 correct?

[hek]

The comment says it all. :) If you have a W5100 module with a SPI-enable pin (not many of them have this) you don't need to enable the Soft-spi for the radio (If I remember correctly it is disabled automatically if the MY_W5100_SPI_EN is defined).

4 is the (default) pin where you should connect it to on your Arduino board. You can change this to fit your setup.

[tomkxy]

@hek Thanks!

After uploading to the Mega I receive intermittent the following error during initialization:

0;255;3;0;9;read register, reg=6, value=0
0;255;3;0;9;Sanity check failed: RF_SETUP register=0 instead of 39, check wiring, replace module or non-P version
0;255;3;0;9;Radio init failed. Check wiring.

Most of the time unplugging the Mega from power it goes away. This happens with two different modulesm one with external antenna and one without. Any idea?

[tomkxy]

@Anticimex I am still digging.... Now I figured the following:

The moment I disable MY_SIGNING_REQUEST_SIGNATURES on the sensor node while still having it on the gateway enabled sensor data is received and properly processed on gateway.

I am now a bit puzzled. Am I right assuming that if MY_SIGNING_REQUEST_SIGNATURES is enabled on gateway that sensors nodes need to sign messages to the gateway?

[Anticimex]

@tomkxy no. GW will only require signatures from nodes that require signatures. Else it would require it from every node.

[tomkxy]

@Anticimex @Hek I hacked now my sketches and tested the following scenarios:

1. sending a full payload from the sensor to the gateway in form of a nonce response
   -> it was received by the gateway and dropped -> ok
2. doing the same as above but from the gateway to a node (the node was in a _process() loop) -> transmission failure

So it seems that the sending side from the gateway to the sensor makes trouble, or the other way round receiving on the sensor node. I have not enough know regarding RF communication but this kind of asymetry seems to be strange.
Any idea what I can try or who might be able to help?

[Anticimex]

@tomkxy it is not totally strange that TX and RX performance differ. You could try to move your node around a bit and see if it is affected by location. Fiddling a bit with the transmission strength could also be a thing to try.

[tomkxy]

@Anticimex I expanded on my previous experiment. I do not require full payload. If I just transmit one byte, most of the transmissions fail.

So I need to investigate the sending node which is a Arduino Mega with an Ethernet shield...

[hek]

If you're using the 3v3 line on the mega, think again. When I tried that, I got very much transmission failures. It's crap. Use a regulator from the 5V rail.

[tomkxy]

@Anticimex @Hek thanks a lot for your support. This is very much appreciated.

I did now the following:

• solder a 4.7uf directly on the radio
• changed to the 5v power rail from the Mega, utilizing a regulator
• moved both nodes farer away

Results are much better but still not as reliable as I would need. One node acting as repeater does not get any response to its "parent" broadcasts (need to investigate this).

After some research I read that the Mega is probably not the best combination due to its current supply.
What would be a suitable alternative to the Mega (>32kb memory) which provides more reliably power?

[tomkxy]

Finally, I have everything working again. Again a big thanks to @hek and @Anticimex .

As lessons learnt, I did the following:

• Solder 4.7uf directly on the radio
• Changed to the 5v power rail from the Mega, utilizing a regulator
• Put a 100 uf between 5v and gnd on the Mega
• Moved both nodes farer away (when I was testing I had same lying side by side which obviously created interferences)
• Switched the RF24 channel utilizing a channel which was not so polluted by all the Wifis around me

[Anticimex]

Great news!