Can't get encryption and signing to work

[vigonotion]

Hey guys,

so encryption and signing seems to be exasperating. I've tried it over and over, read the documentation and some forum posts and tried to make it work.
My goal is to have all my nodes to communicate encrypted and signed, so that nothing else but my nodes can read the sensor data and send sensor data to the Gateway.
I do not have an ATSHA so I want to go with soft signing. What I've done so far:

• Running the personalizer with Soft Signing for the GW and Soft Signing Serial for my sensors
• Tried many different combinations of the Signing and Encryption definitions.
  Currently, I've got these definitions:

/** SIGNING AND ENCRYPTION **/
// Setup Soft Signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES

#define MY_SECURITY_SIMPLE_PASSWD "XXXXXXXXXXXXXXXX"
//#define MY_RF24_ENABLE_ENCRYPTION
/** END SIGNING AND ENCRYPTION **/

I've tried this out with and without Simple Password enabled, but as I understand it correctly, using the Personalizer is more secure so I want to go with the personalizer.

What I get:

1. The gateway accepts everything. A normal node (no signing, no personalization whatsoever) can send data to the gateway
2. The gateway accepts nothing. Neither the node run with the same personalizer sketch/simple passwd nor any other node without security settings.

I can't get behind this security and signing thing as I can't get it running. Can you guys help me?

Thanks in advance
Tom

PS: Hard- and Software:

• Arduino Pro Mini 3V3 and 5V for Sensor Nodes
• ESP8266 for MQTT Gateway
• PlatformIO with MySensors 2.2.0
• NRF24 Antennas (the normal ones)

[Anticimex]

@b1ackra1n hi,
First of all, the simple password flag is mutually exclusive with all other security settings.
Secondly, make sure you set any sketch specific defines prior to including mysensors.h.
Thirdly, there are debugging flags to enable so you get more detailed logs. Right now you provide nothing, so it is impossible to determine what the problem is.

[vigonotion]

1. It doesn't work with SIMPLE_PASSWD disabled either

currently I'm trying it with these settings:

#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES

#define MY_RF24_ENABLE_ENCRYPTION

1. Yeah, defines are all above the include
2. I can't get #define MY_DEBUG_VERBOSE_SIGNING to work, the output is just gibberish, but here are my normal debug prints from the sensor node:

 __  __       ____
|  \/  |_   _/ ___|  ___ _ __  ___  ___  _ __ ___
| |\/| | | | \___ \ / _ \ `_ \/ __|/ _ \| `__/ __|
| |  | | |_| |___| |  __/ | | \__ \  _  | |  \__ \
|_|  |_|\__, |____/ \___|_| |_|___/\___/|_|  |___/
        |___/                      2.2.0

32 MCO:BGN:INIT NODE,CP=RNNNAS-X,VER=2.2.0
124 TSM:INIT
126 TSF:WUR:MS=0
137 TSM:INIT:TSP OK
141 TSM:INIT:STATID=42
145 TSF:SID:OK,ID=42

149 TSM:FPAR
231 TSF:MSG:SEND,42-42-255-255,s=255,c=3,t=7,pt=0,l=0,sg=0,ft=0,st=OK:
376 TSF:MSG:READ,0-0-42,s=255,c=3,t=8,pt=1,l=1,sg=1:0
387 TSF:MSG:FPAR OK,ID=0,D=1
2246 TSM:FPAR:OK
2248 TSM:ID
2250 TSM:ID:OK
2254 TSM:UPL
2260 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=24,pt=1,l=1,sg=0,ft=0,st=OK:1
2312 TSF:MSG:READ,0-0-42,s=255,c=3,t=25,pt=1,l=1,sg=1:1
2322 TSF:MSG:PONG RECV,HP=1
2328 TSM:UPL:OK
2330 TSM:READY:ID=42,PAR=0,DIS=1
2347 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=15,pt=6,l=2,sg=0,ft=0,st=OK:0101
2369 TSF:MSG:READ,0-0-42,s=255,c=3,t=15,pt=6,l=2,sg=0:0101
2392 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=16,pt=0,l=0,sg=0,ft=0,st=OK:
2404 !TSF:MSG:SIGN FAIL
2414 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=16,pt=0,l=0,sg=0,ft=1,st=OK:
2426 !TSF:MSG:SIGN FAIL
2435 TSF:MSG:READ,0-0-42,s=255,c=3,t=17,pt=6,l=25,sg=0:<NONCE>
2609 TSF:MSG:READ,0-0-42,s=255,c=3,t=17,pt=6,l=25,sg=0:<NONCE>
4431 !TSF:MSG:SIGN FAIL
4435 !TSF:MSG:SIGN FAIL
4440 !TSF:MSG:SIGN FAIL
4544 !TSF:MSG:SIGN FAIL
4648 !TSF:MSG:SIGN FAIL
4753 !TSF:MSG:SIGN FAIL
4757 MCO:REG:REQ
4765 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=26,pt=1,l=1,sg=0,ft=7,st=OK:2
4804 TSF:MSG:READ,0-0-42,s=255,c=3,t=16,pt=0,l=0,sg=1:
4851 TSF:MSG:SEND,42-42-0-0,s=255,c=3,t=17,pt=6,l=25,sg=1,ft=0,st=OK:<NONCE>
4886 TSF:MSG:READ,0-0-42,s=255,c=3,t=27,pt=1,l=1,sg=1:1
4896 !TSF:MSG:SIGN VERIFY FAIL
4900 MCO:BGN:STP
XXX - STARTING NODE - XXX
4904 MCO:BGN:INIT OK,TSP=1
4915 MCO:SLP:MS=5,SMS=0,I1=255,M1=255,I2=255,M2=255
4923 TSF:TDI:TSL
4927 MCO:SLP:WUP=-1
4931 TSF:TRI:TSB

4933 !TSF:MSG:SIGN FAIL
4937 !TSF:MSG:SIGN FAIL
4943 !TSF:MSG:SIGN FAIL
4947 !TSF:MSG:SIGN FAIL
4952 MCO:SLP:MS=1000,SMS=0,I1=255,M1=255,I2=255,M2=255
4962 TSF:TDI:TSL
4966 MCO:SLP:WUP=-1
4968 TSF:TRI:TSB
4972 MCO:SLP:MS=5,SMS=0,I1=255,M1=255,I2=255,M2=255
4982 TSF:TDI:TSL
4984 MCO:SLP:WUP=-1
4988 TSF:TRI:TSB
4993 MCO:SLP:MS=1000,SMS=0,I1=255,M1=255,I2=255,M2=255
5003 TSF:TDI:TSL
5005 MCO:SLP:WUP=-1
5009 TSF:TRI:TSB
5013 MCO:SLP:MS=5,SMS=0,I1=255,M1=255,I2=255,M2=255
5023 TSF:TDI:TSL
5025 MCO:SLP:WUP=-1
5029 TSF:TRI:TSB
5033 MCO:SLP:MS=1000,SMS=0,I1=255,M1=255,I2=255,M2=255
5042 TSF:TDI:TSL
5046 MCO:SLP:WUP=-1
5050 TSF:TRI:TSB
5054 MCO:SLP:MS=5,SMS=0,I1=255,M1=255,I2=255,M2=255
5062 TSF:TDI:TSL
5066 MCO:SLP:WUP=-1
5070 TSF:TRI:TSB

and from the gateway:

1093 GWT:TPC:CONNECTING...
1098 GWT:TPC:IP=192.168.1.70
1103 MCO:BGN:STP
1106 MCO:BGN:INIT OK,TSP=1
1111 GWT:TPC:IP=192.168.1.70
1116 GWT:RMQ:MQTT RECONNECT
1138 GWT:RMQ:MQTT CONNECTED
1143 GWT:TPS:TOPIC=mygateway1-out/0/255/0/0/18,MSG SENT
1154 GWT:TPS:TOPIC=mygateway1-out/0/255/3/0/11,MSG SENT
1164 GWT:TPS:TOPIC=mygateway1-out/0/255/3/0/12,MSG SENT
pm open,type:2 0
13651 TSF:MSG:READ,42-42-255,s=255,c=3,t=7,pt=0,l=0,sg=0:

13661 TSF:MSG:BC
13664 TSF:MSG:FPAR REQ,ID=42
13669 TSF:PNG:SEND,TO=0
13674 TSF:CKU:OK
13677 TSF:MSG:GWL OK
14051 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=OK:0
14704 TSF:MSG:READ,42-42-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1
14714 TSF:MSG:PINGED,ID=42,HP=1
14727 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=OK:1
14742 TSF:MSG:READ,42-42-0,s=255,c=3,t=15,pt=6,l=2,sg=0:0101
14755 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=15,pt=6,l=2,sg=0,ft=0,st=OK:0101
15940 TSF:MSG:READ,42-42-0,s=255,c=3,t=26,pt=1,l=1,sg=0:2
15957 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=16,pt=0,l=0,sg=0,ft=0,st=OK:
15981 TSF:MSG:READ,42-42-0,s=255,c=3,t=17,pt=6,l=25,sg=0:<NONCE>
15997 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=27,pt=1,l=1,sg=1,ft=0,st=OK:1
33057 TSF:MSG:READ,42-42-255,s=255,c=3,t=7,pt=0,l=0,sg=0:
33068 TSF:MSG:BC
33071 TSF:MSG:FPAR REQ,ID=42
33076 TSF:PNG:SEND,TO=0
33080 TSF:CKU:OK
33083 TSF:MSG:GWL OK
33409 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=8,pt=1,l=1,sg=1,ft=0,st=OK:0
34109 TSF:MSG:READ,42-42-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1
34120 TSF:MSG:PINGED,ID=42,HP=1
34137 TSF:MSG:SEND,0-0-42-42,s=255,c=3,t=25,pt=1,l=1,sg=1,ft=0,st=OK:1

Still trying to get those signing debug logs to work, but I hope these logs already help

[vigonotion]

@b1ackra1n
Okay got the signing debug prints (sensor node)

79 SGN:PER:OK
112 SGN:INI:BND OK
122 SGN:SKP:MSG CMD=3,TYPE=7
1449 SGN:SKP:MSG CMD=3,TYPE=8
2209 SGN:SKP:MSG CMD=3,TYPE=24
2271 SGN:SKP:MSG CMD=3,TYPE=25
2277 SGN:PRE:SGN REQ
2279 SGN:PRE:WHI NREQ
2283 SGN:SKP:MSG CMD=3,TYPE=15
2293 SGN:PRE:XMT,TO=0
2297 SGN:PRE:WAIT GW
2322 SGN:SKP:MSG CMD=3,TYPE=15
2326 SGN:PRE:SGN REQ,FROM=0

2332 SGN:SKP:MSG CMD=3,TYPE=16
2342 SGN:SGN:NCE REQ,TO=0
2377 SGN:SKP:MSG CMD=3,TYPE=17
2383 SGN:NCE:FROM=0
2387 SGN:BND:NONCE=X
2562 SGN:BND:HMAC=X
2576 SGN:SGN:SGN
2586 SGN:SKP:MSG CMD=3,TYPE=16
2598 SGN:SGN:NCE REQ,TO=0
2672 SGN:SKP:MSG CMD=3,TYPE=17
2678 SGN:NCE:FROM=0
2682 SGN:BND:NONCE=X
2856 SGN:BND:HMAC=X
2871 SGN:SGN:SGN
4888 SGN:SKP:MSG CMD=3,TYPE=16
4900 SGN:SGN:NCE REQ,TO=0
4904 !SGN:SGN:SGN FAIL
4909 SGN:SKP:MSG CMD=3,TYPE=16
4921 SGN:SGN:NCE REQ,TO=0
4927 !SGN:SGN:SGN FAIL
4931 SGN:SKP:MSG CMD=3,TYPE=16
4943 SGN:SGN:NCE REQ,TO=0
4947 !SGN:SGN:SGN FAIL
5052 SGN:SKP:MSG CMD=3,TYPE=16
5154 !SGN:SGN:NCE REQ,TO=0 FAIL
5261 SGN:SKP:MSG CMD=3,TYPE=16
5365 !SGN:SGN:NCE REQ,TO=0 FAIL
5470 SGN:SKP:MSG CMD=3,TYPE=16
5574 !SGN:SGN:NCE REQ,TO=0 FAIL
5580 SGN:SKP:MSG CMD=3,TYPE=26
5683 SGN:SKP:MSG CMD=3,TYPE=7
5789 SGN:SKP:MSG CMD=3,TYPE=17
5795 SGN:NCE:FROM=0
5799 SGN:BND:NONCE=X
5974 SGN:BND:HMAC=X

5992 SGN:SKP:MSG CMD=3,TYPE=17
5998 SGN:NCE:FROM=0
6000 SGN:BND:NONCE=X
6176 SGN:BND:HMAC=X
6195 SGN:SKP:MSG CMD=3,TYPE=17
6199 SGN:NCE:FROM=0
6203 SGN:BND:NONCE=X
6377 SGN:BND:HMAC=X
6633 SGN:SKP:MSG CMD=3,TYPE=8
7786 SGN:SGN:NREQ=0
7845 SGN:SKP:MSG CMD=3,TYPE=25

PS: if anybody else has problems setting those up, you can only use MY_DEBUG or MY_DEBUG_VERBOSE_SIGNING. I had both enabled.

[Anticimex]

@b1ackra1n not really. If you run with personalizer (which by the way without an atsha is not much more secure than the simple password flag). I'd suggest you start with the simple flag first. If personalization has not been executed properly, signing won't work.

[Anticimex]

@b1ackra1n said in Can't get encryption and signing to work:

PS: if anybody else has problems setting those up, you can only use MY_DEBUG or MY_DEBUG_VERBOSE_SIGNING. I had both enabled.

This is also in the documentation: https://www.mysensors.org/apidocs/group__MySigningTroubleshootinggrp.html

Also, you don't need to obfuscate nonce and hmac signatures. They are not secret. Confirming that node and gw agree on these is a good place to start troubleshooting.

[vigonotion]

@anticimex what I've now done is I'm just using the MY_SECURITY_SIMPLE_PASSWD flag.
This is my sensor nodes output:

2154 SGN:PRE:SGN NREQ
2158 SGN:PRE:WHI NREQ
2166 SGN:PRE:XMT,TO=0
2170 SGN:PRE:WAIT GW
2205 SGN:PRE:NSUP

The sensor node can send messages to the GW, but the GW also accepts messages from nodes without signing, which is not yet what I wanted.

I'm now checking if GW and Sensor agree on Nonce and HMAC

[Anticimex]

@b1ackra1n SGN:PRE:NSUP indicate that one of your nodes does not support signing, so your definitions are not correctly set on either the node or the GW.

[vigonotion]

@anticimex I think it's the GW:

11607 SGN:PRE:NSUP,TO=42
11613 SGN:PRE:XMT,TO=42

But I can't find what I've done wrong:

// Enable debug prints to serial monitor
#define MY_DEBUG_VERBOSE_SIGNING

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24

/** SIGNING AND ENCRYPTION **/
// Setup Soft Signing
// #define MY_SIGNING_SOFT
// #define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
// #define MY_SIGNING_REQUEST_SIGNATURES
//
#define MY_SECURITY_SIMPLE_PASSWD "XXXXXXXXXXXXXXXX"
// #define MY_RF24_ENABLE_ENCRYPTION
/** END SIGNING AND ENCRYPTION **/

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set this node's subscribe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-gw"

#define MY_NODE_ID 1

// Enable these if your MQTT broker requires usenrame/password
#define MY_MQTT_USER "XXX"
#define MY_MQTT_PASSWORD "XXX"

// Set WIFI SSID and password
#define MY_ESP8266_SSID "XXX"
#define MY_ESP8266_PASSWORD "XXX"

// Set the hostname for the WiFi Client. This is the hostname
// it will pass to the DHCP server if not static.
#define MY_ESP8266_HOSTNAME "mqtt-sensor-gateway"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
//#define MY_IP_ADDRESS 192,168,178,87

// If using static ip you need to define Gateway and Subnet address as well
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// MQTT broker ip address.
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 1, X

// The MQTT broker port to to open
#define MY_PORT X

#define SN "MySensors MQTT Gateway"
#define SV "1.0"

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <MySensors.h>

void setup()
{
}

void presentation()
{
  // Send the Sketch Info
  sendSketchInfo(SN, SV);
}


void loop()
{

}

[Anticimex]

@b1ackra1n I can't either to be honest. It seems like your gw at some point disable the feature. You could dig down the code to see what the simple password flag actually enables, and add #error statements at the bottom of your gw sketch to validate that the expected flags remain set.
Unfortunately I am knocked out in bed by some influenza so I cannot do effective code digging for the moment.

[vigonotion]

@anticimex I think I got it working. The problem was that i used #define MY_SECURITY_SIMPLE_PASSWD instead of #define MY_SIGNING_SIMPLE_PASSWD (did I read the wrong docs? https://www.mysensors.org/apidocs-beta/group__SigningSettingGrpPub.html#gaedf8ec407fbde609a520ea0d95da2aac)

With that, only nodes with that password are able to communicate. Still needs some testing, but seems to work.

That's for the SIMPLE_PASSWD part. Next step would be getting Soft Signing to work, I will try that now and then send the debug logs.

Thanks already, if everything else fails, I'm now able to use at least the SIMPLE_PASSWD.

[Anticimex]

@b1ackra1n the link points to the beta release. So it is aligned with the development branch, not the master branch, and it looks like you use the official 2.2.0 release, so strip the -beta part of the url. My troubleshooting link also points to the release documentation.

[vigonotion]

Now I got Soft Signing to work, although I'm not yet sure why it works now:

/** SOFT SIGNING **/
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_SOFT
#define MY_SIGNING_REQUEST_SIGNATURES
/** END SOFT SIGNING **/

I think the problem was that "the simple password flag is mutually exclusive" (thanks @Anticimex).
Now everything works as expected!

[Anticimex]

@b1ackra1n good news