MQTT gateway sends duplicate messages to controller?

[willemx]

My MQTT gateways produce lots of duplicate messages. I have two gateways in my network: one with a NRF24 radio and one with a RFM69 radio. Both gateways use a ESP8266 processor board (Wemos D1 mini) and communicate via WiFi with a MQTT broker on a raspberry pi. In my controller (on a Linux machine) I see lots of duplicate messages arriving from the MQTT broker.
Duplicate messages come from both gateways and can originate from all network nodes.
I created a small RFM69 based testnode that periodically sends an incrementing counter and checks for send errors. There are never any send errors, but I do get duplicate messages from this node.
What can be the reason that a gateway should send a message multiple times?

[willemx]

Done some more debugging...
Realised that maybe the MQTT broker can also be the culprit. I modified my controller code to make it inspect the "dup"-flag of received packets. So far I found out that this flag was never set although there were duplicate messages received, so I guess this flag has some other meaning (maybe duplicate messages for different topics?)
Also set the (Mosquitto) broker config to "log_types all", but unfortunately this does not log the message payload so it is very hard to check for duplicate packets in the logfile.
Maybe I should just make my controller ignore duplicate messages...

[mfalkvidd]

@willemx the gateway debug log should show what is happening.

[Yveaux]

@willemx depending on the QoS value of an MQTT publish the same message can arrive multiple times at subscribers. This could be the issue.

[electrik]

For the RFM69 radios this can be caused by the software acknowledgement mechanism. If the sending node doesn't receive the acknowledgement from the gateway, the message is sent again. This will result in double messages on the gateway.

[willemx]

@mfalkvidd : In the gateway log I can indeed see that some messages are sent twice
@Yveaux : I modified my controller to subscribe to the gateway using QoS=2 (was 0), but this made no difference; I keep receiving duplicate packets.
@electrik : Your suggestion could be right. This is consistent with the duplicate messages in the gateway log and the duplicate messages to the controller despite QoS=2. If the node resends the message, it is apparently considered by the gateway to be a 'new' message.
So the question is now why the gateway does not send an acknowledgment for every message. I thought that RFM69 modules were very reliable.

[mfalkvidd]

@willemx the gateway most likely sends acknowledgement for every message. But the node might not hear every acknowledgement, and will therefore resend.

It is impossible for the gateway to differentiate between a resent message and a new message with the same content, so the gateway dutifully forwards both messages to mqtt. The mqtt broker is in the same situation as the gateway: if it gets two messages, it forwards two messages.

The only reliable way to distinguish between original messages and resent messages is to use something similar to a sequence number. MySensors does not use such a thing.

[willemx]

That is a bit disappointing and also confusing :confused:
So the returned boolean of the MySensors send() function is not to be trusted? The docs state that send "Returns true if message reached the first stop on its way to destination". In my test setup there is a direct connection between node and gateway and every call to send() returns true, yet apparently this is no guarantee that the message reached the gateway.

[electrik]

The send() function can be trusted, the message arrives right? Twice even.
I think you should look at optimising the tx levels. https://forum.mysensors.org/topic/11107/help-needed-to-setup-a-rfm69-esp8266-gateaway-on-adafruit-feather-huzzah/30?_=1601132509084
Here is a short description of the approach

[mfalkvidd]

@willemx the documentation is correct. The gateway is not necessarily the next stop. But since you don't have any repeaters, and the node is sending to the gateway, the gateway will be the next stop.

send() returns true if the the sending node received an acknowledgement from the next node. The nodes automatically resend unil they receive the acknowledgement (or a limit is reached), so there is absolutely no guarantee that the message was sent only once.

If you want to read up on the information theory, check Two Generals' Problem especially this part:

The Two Generals' Problem was the first computer communication problem to be proved to be unsolvable.

[willemx]

Thanks @mfalkvidd for the Two Generals theory, that was very enlightening.
So I assume the following scenario is happening in my situation:

1. the sensor node sends a message to the gateway
2. the gateway receives the message and sends acknowledge back to sensor node
3. the gateway relays the message to the MQTT broker
4. the sensor node fails to receive the acknowledgement from the gateway so it resends the message to the gateway
5. the gateway receives the message, thinks it is a new message and also relays this message to the MQTT broker

Not sure if this scenario is correct, but it explains my controller receiving duplicate messages.
It seems as if a gateway cannot discriminate between a new message and a resent message?
If it could, then it should simply ignore resent messages that have already been relayed to the MQTT broker.

[mfalkvidd]

@willemx yes, that is correct.

The most common way to see if messages are new or old is to include a sequence number in the message. MySensors does not do that.

[willemx]

@mfalkvidd If I add a sequence number to all messages, is there a way to include some code in the gateway sketch to filter out duplicate messages before they are relayed to the MQTT broker?
I mean without modifying the MySensors gateway code, but just inserting some kind of hook that gets called by the gateway code on every incoming message.
I could not find such a hook in the library API. There is a receive() function, but I don't think this is meant for this purpose. Ideally there should be a onReceive() function that can do some processing on messages and return a status code to indicate that a message should be discarded.

[mfalkvidd]

@willemx yes that sounds reasonable. Such a hook might exist, but I have not seen any, and I think you would have found it if it existed.

[hlehoux]

Hi all, discovered today this thread while i was searching for a problem of message received twice.

Even though i've been using mySensors for several years, this is the first time i encounter this.

I was not aware there is no guarantee that a message is sent only once....

Unfortunatly, the message is sent to an actuator to tell to "toggle" the light: so if received twice, the effect is not very efficient.

I will have to implement something...

[hlehoux]

@hlehoux BTW, in my case this seems to be related to the battery being very low.