Sensor Data Transmission Integrity
-
I've recently been updating my nodes and gateways to use the latest version of MySensors. I've also recently acquired a handful of Sensebender Micros and set one up in the pool shed to test/play and get some outside temperature and humidity readings. A problem I found though is that I was getting some anomalous data readings which completely throws off my time period data - see the 24-hour chart below.
Solar flares perhaps? I ruled that out.
I set up a test last night before turning in for the night and ran it for about 18 hours. I've collected and compared the debug logs of my node with those of the Ethernet and MQTT gateways that I have and found that the node is sending a valid float value at all times but the gateways will sometimes receive something different.
Node Debug Data (stripped to just the sending of one of the sensors)
4935 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:21.6 5480 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:21.2 6891 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:20.7 8331 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:20.4And corresponding receives from the Ethernet Gateway debug log
Jun 04 00:55:20 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:21.6 Jun 04 01:02:35 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=5,l=5,sg=0:1101633946 Jun 04 01:37:38 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:20.7 Jun 04 02:13:52 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:20.4You can see that the data received changed payload type from P_FLOAT32 (pt=7) to P_ULONG32 (pt=5) !!!
Note that the graph above is from data collected by way of the MQTT gateway and as such doesn't specifically correspond to the debug logs above but I do see exactly the same flip of payload type in the MQTT GW log - though at different times from those in the Ethernet GW log
Now I understand that this is going to come down to noise in the radio transmission but it raises some bigger-picture questions for me.
- How can I ensure that the data received by the gateway is specifically what the node sent?
- Would using ACK and having the node confirm if what was sent was received correctly be a valid approach?
- Would implementing message signing help in any way?
I'm at a loss on how best to ensure the integrity of the data being sent and collected. What are your thoughts?
Allen
@alphaHotel signing is intended to solve exactly this problem but perhaps geared more to guard against malicious manipulation of messages. But it would work equally well for messages that unintentionally get modified in transit for any reason.
Encryption would also provide a similar filter for you but has the caveat that you need to enable it on all nodes in the network. The upside of encryption is that it does not require handshaking (at the cost of quality) so it is easier on battery driven nodes and "faster" as sending a message is still just sending a message (as opposed to sending, receiving, sending for signing). -
I've recently been updating my nodes and gateways to use the latest version of MySensors. I've also recently acquired a handful of Sensebender Micros and set one up in the pool shed to test/play and get some outside temperature and humidity readings. A problem I found though is that I was getting some anomalous data readings which completely throws off my time period data - see the 24-hour chart below.
Solar flares perhaps? I ruled that out.
I set up a test last night before turning in for the night and ran it for about 18 hours. I've collected and compared the debug logs of my node with those of the Ethernet and MQTT gateways that I have and found that the node is sending a valid float value at all times but the gateways will sometimes receive something different.
Node Debug Data (stripped to just the sending of one of the sensors)
4935 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:21.6 5480 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:21.2 6891 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:20.7 8331 TSF:MSG:SEND,4-4-0-0,s=1,c=1,t=0,pt=7,l=5,sg=0,ft=0,st=OK:20.4And corresponding receives from the Ethernet Gateway debug log
Jun 04 00:55:20 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:21.6 Jun 04 01:02:35 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=5,l=5,sg=0:1101633946 Jun 04 01:37:38 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:20.7 Jun 04 02:13:52 DEBUG TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:20.4You can see that the data received changed payload type from P_FLOAT32 (pt=7) to P_ULONG32 (pt=5) !!!
Note that the graph above is from data collected by way of the MQTT gateway and as such doesn't specifically correspond to the debug logs above but I do see exactly the same flip of payload type in the MQTT GW log - though at different times from those in the Ethernet GW log
Now I understand that this is going to come down to noise in the radio transmission but it raises some bigger-picture questions for me.
- How can I ensure that the data received by the gateway is specifically what the node sent?
- Would using ACK and having the node confirm if what was sent was received correctly be a valid approach?
- Would implementing message signing help in any way?
I'm at a loss on how best to ensure the integrity of the data being sent and collected. What are your thoughts?
Allen
-
@alphaHotel what radio do you use?
-
@alphaHotel signing is intended to solve exactly this problem but perhaps geared more to guard against malicious manipulation of messages. But it would work equally well for messages that unintentionally get modified in transit for any reason.
Encryption would also provide a similar filter for you but has the caveat that you need to enable it on all nodes in the network. The upside of encryption is that it does not require handshaking (at the cost of quality) so it is easier on battery driven nodes and "faster" as sending a message is still just sending a message (as opposed to sending, receiving, sending for signing).@Anticimex Thanks. I've done some preliminary research into signing and encryption and have struggled a bit with taking it from concept to implementation. I guess I should roll up my sleeves and dig into the how-tos. This is intended to be a battery-powered sensor so perhaps I'll start with encryption first.
-
@Anticimex Thanks. I've done some preliminary research into signing and encryption and have struggled a bit with taking it from concept to implementation. I guess I should roll up my sleeves and dig into the how-tos. This is intended to be a battery-powered sensor so perhaps I'll start with encryption first.
@alphaHotel it comes down to the frequency of transmission. You can use the simple password flags to rapidly test something.
-
@alphaHotel it comes down to the frequency of transmission. You can use the simple password flags to rapidly test something.
@Anticimex I've got it set to measure once per minute at the moment and force transmissions that have no changes every 30 but that's my testing stance. In the end, it'll likely be more of a 5 or 10-minute measurement and forced hourly.
-
@Anticimex I've got it set to measure once per minute at the moment and force transmissions that have no changes every 30 but that's my testing stance. In the end, it'll likely be more of a 5 or 10-minute measurement and forced hourly.
@alphaHotel ok. That could be a bit painful to do signed on batteries.
-
@Yveaux I'm using Nrf24's.
-
@Anticimex Thanks. I've done some preliminary research into signing and encryption and have struggled a bit with taking it from concept to implementation. I guess I should roll up my sleeves and dig into the how-tos. This is intended to be a battery-powered sensor so perhaps I'll start with encryption first.
@alphaHotel have you checked the memory usage? Too high ram usage can cause stack overflows which can cause all kind of mayhem.
-
@alphaHotel have you checked the memory usage? Too high ram usage can cause stack overflows which can cause all kind of mayhem.
@Anticimex I had to turn off MY_DEBUG to add the encryption directives. I also moved everything back to version 2.3.1 as I was still seeing some weird stuff going on like msg-type 169 (unknown) and attempts to route through nodes that don't exist. These things I guess may have been due to high ram usage. I think I was at about 78% w version 2.3.2 and basic encryption enabled but now...
Sketch uses 21542 bytes (70%) of program storage space. Maximum is 30720 bytes. Global variables use 1108 bytes (54%) of dynamic memory, leaving 940 bytes for local variables. Maximum is 2048 bytes.@Yveaux - I don't expect that it would have been high ram usage causing my initial issues as I wasn't using any encryption or signing before. I started seeing these problems when I moved to 2.3.2 but as I had also been dealing with a new sensor on my first Sensebender Micro, I wasn't sure what might be causing the problems. What do you think might have caused the received data type to flip from a float to an unsigned integer?
