MQTT gateway sends duplicate messages to controller?

[willemx]

@mfalkvidd : In the gateway log I can indeed see that some messages are sent twice
@Yveaux : I modified my controller to subscribe to the gateway using QoS=2 (was 0), but this made no difference; I keep receiving duplicate packets.
@electrik : Your suggestion could be right. This is consistent with the duplicate messages in the gateway log and the duplicate messages to the controller despite QoS=2. If the node resends the message, it is apparently considered by the gateway to be a 'new' message.
So the question is now why the gateway does not send an acknowledgment for every message. I thought that RFM69 modules were very reliable.

[mfalkvidd]

@willemx the gateway most likely sends acknowledgement for every message. But the node might not hear every acknowledgement, and will therefore resend.

It is impossible for the gateway to differentiate between a resent message and a new message with the same content, so the gateway dutifully forwards both messages to mqtt. The mqtt broker is in the same situation as the gateway: if it gets two messages, it forwards two messages.

The only reliable way to distinguish between original messages and resent messages is to use something similar to a sequence number. MySensors does not use such a thing.

[willemx]

That is a bit disappointing and also confusing :confused:
So the returned boolean of the MySensors send() function is not to be trusted? The docs state that send "Returns true if message reached the first stop on its way to destination". In my test setup there is a direct connection between node and gateway and every call to send() returns true, yet apparently this is no guarantee that the message reached the gateway.

[electrik]

The send() function can be trusted, the message arrives right? Twice even.
I think you should look at optimising the tx levels. https://forum.mysensors.org/topic/11107/help-needed-to-setup-a-rfm69-esp8266-gateaway-on-adafruit-feather-huzzah/30?_=1601132509084
Here is a short description of the approach

[mfalkvidd]

@willemx the documentation is correct. The gateway is not necessarily the next stop. But since you don't have any repeaters, and the node is sending to the gateway, the gateway will be the next stop.

send() returns true if the the sending node received an acknowledgement from the next node. The nodes automatically resend unil they receive the acknowledgement (or a limit is reached), so there is absolutely no guarantee that the message was sent only once.

If you want to read up on the information theory, check Two Generals' Problem especially this part:

The Two Generals' Problem was the first computer communication problem to be proved to be unsolvable.

[willemx]

Thanks @mfalkvidd for the Two Generals theory, that was very enlightening.
So I assume the following scenario is happening in my situation:

1. the sensor node sends a message to the gateway
2. the gateway receives the message and sends acknowledge back to sensor node
3. the gateway relays the message to the MQTT broker
4. the sensor node fails to receive the acknowledgement from the gateway so it resends the message to the gateway
5. the gateway receives the message, thinks it is a new message and also relays this message to the MQTT broker

Not sure if this scenario is correct, but it explains my controller receiving duplicate messages.
It seems as if a gateway cannot discriminate between a new message and a resent message?
If it could, then it should simply ignore resent messages that have already been relayed to the MQTT broker.

[mfalkvidd]

@willemx yes, that is correct.

The most common way to see if messages are new or old is to include a sequence number in the message. MySensors does not do that.

[willemx]

@mfalkvidd If I add a sequence number to all messages, is there a way to include some code in the gateway sketch to filter out duplicate messages before they are relayed to the MQTT broker?
I mean without modifying the MySensors gateway code, but just inserting some kind of hook that gets called by the gateway code on every incoming message.
I could not find such a hook in the library API. There is a receive() function, but I don't think this is meant for this purpose. Ideally there should be a onReceive() function that can do some processing on messages and return a status code to indicate that a message should be discarded.

[mfalkvidd]

@willemx yes that sounds reasonable. Such a hook might exist, but I have not seen any, and I think you would have found it if it existed.

[hlehoux]

Hi all, discovered today this thread while i was searching for a problem of message received twice.

Even though i've been using mySensors for several years, this is the first time i encounter this.

I was not aware there is no guarantee that a message is sent only once....

Unfortunatly, the message is sent to an actuator to tell to "toggle" the light: so if received twice, the effect is not very efficient.

I will have to implement something...

[hlehoux]

@hlehoux BTW, in my case this seems to be related to the battery being very low.