Handling ACK from MySensors

[Mihai]

Hi,
Please let me know how to handle ACK from MySensors messages using MQTT? Unfortunately in MySensors seems not to be automatically handled, users have to write a dedicated procedure for this. I think I can do that: set ACK to 1, send message, check the return ACK message and if not received send it again (may be into a 3 times loop). Does HA respond to a message with ACK if requested?
And how to handle the message sent from HA to node? Now I send the message from HA to node to switch a relay and the node send back one message with its new state; if this message is not sent or received properly by HA, the switch on the HA interface doesn't change.
Thank you.

[martinhjelmare]

@Mihai

Ack response is implemented in the serial gateway sketch, in my experience. I haven't tested any MQTT gateway sketches, so I can't say anything there.

Do you think that it's necessary to have an ack from the controller, or is it enough that the gateway sends the ack? I think it's the radio interfaces that are the most at risk for dropped messages. If using a wifi gateway I could see the need for an ack from the controller, but the wired serial and ethernet interfaces should be reliable enough to not need it, I think.

You can set the qos level in home assistant for the MQTT switch platform, for example. This should "guarantee" the message is delivered from home assistant to broker. For the delivery between broker and gateway, if not the same, I guess it depends on the qos level that the gateway set when subscribing to the broker topic.

I don't have any own experience with MQTT, I'm just repeating what I've read.

[Mihai]

@martinhjelmare
Thank you for the answer. I have built a system with several sensors using NRF24 for communication between nodes and gateway, the gateway is connected with Ethernet cable to the network and use MQTT.
Sometimes, in rare cases, the message doesn't reach the other end and this happens in booth ways: nodes messages doesn't show on the controller interface, the commands doesn't trigger the node. Even if they are quite rare cases, they are annoying. I think the problem is on the radio side may be due some intefrerences (I check WiFi signals nearby and set a channel far fro them) or due to many messages in a short time, but I understand that ACK is a checking procedure end-to-end and it is not handled by MySensors libraries, it should be managed by the user code in the node and controller side.
I wil try some tests with QoS and ACK.

[martinhjelmare]

@Mihai

Yeah, I've been thinking about this, and I'm going to implement ack response for the serial gateway in home assistant, but make it optional. That way one could turn it off if wanting to implement it only from gateway.

This will also work for the ethernet implementation I'm working on.

MQTT is relying on the built in support in home assistant, so that's a tricky nut to crack, how to send acks from the controller. I haven't figured out a way there yet. If you do, please share it.

[Mihai]

@martinhjelmare
I found some info about ACK in MySensors here: http://forum.mysensors.org/topic/2189/serial-api-noack-when-sending-with-ack-failed/2
And find somebody code for resend the message in case of sending error: https://github.com/n3roGit/MySensors_n3ro/blob/master/examples/resend/resend.ino
A simplest way to improve the chance of message receiving is to transmit the same data several times even without confirm the delivery and with the drawback of increase the network load.
I have calculate one message duration is about 1ms (maximum message length is 32 bytes=256bits, transmission speed of 250kb/s) and I insert a random pause equivalent with few messages between transmission, so the proposal code into the sketch is a function to be used when delivery is important:

void resend(MyMessage &msg) {
  byte tmin=3;
  byte tmax=9;
  for (byte i=0; i<3 ; i++) {
    send(msg);
    delay(random(tmin,tmax));
  }
}

The delay function is not the best choice, but the overall impact should not be very important because the delay is only few ms.
I plan to use it for PIR, water leak, relays and commands and not for temperature and humidity (even if the temperature value is monitored and a higher value trigger an alarm).

[martinhjelmare]

@Mihai

Could you use wait instead of delay, in the resend function, to be sure not to lose messages?

For switches or binary sensors, I think it could be possible to use a template switch in home assistant to get a real ack from the controller using MQTT.

https://home-assistant.io/components/switch.template/

What do you think?

Edit:

For example for a door reed binary sensor:

1. Make a hidden mqtt binary sensor, with ack set to 1 in topic structure:

binary_sensor:
  platform: mqtt
  state_topic: "mygateway1-out/1/1/1/1/16"
  name: "MQTT Sensor"
  qos: 0
  payload_on: "1"
  payload_off: "0"

1. Make a hidden mqtt switch with ack set to 1 in topic structure:

switch:
  platform: mqtt
  name: "Entrance Switch"
  command_topic: "mygateway1-in/1/1/1/1/16"
  qos: 0
  payload_on: "1"
  payload_off: "0"
  optimistic: true
  retain: false

1. Make a template switch, combining the two components:

switch:
    platform: template
    switches:
        binary_sensor_1:
            friendly_name: 'Entrance door'
            value_template: '{{ states.sensor.binary_sensor_1.state }}'
            turn_on:
                service: switch.turn_on
                entity_id: switch.entrance_switch_1
            turn_off:
                service: switch.turn_off
                entity_id: switch.entrance_switch_1

[martinhjelmare]

Regarding ack in the mysensors library, I was also under the impression that it was not handled by the library. But when I test the RelayWithButtonActuator example connected to a standalone serial gateway, I get ACKs from the gateway, when I push the button.

Maybe someone can confirm this, or clarify that it's a feature?

[Mihai]

@martinhjelmare
Thanks for the idea, I have to check about the wait function, how it works with few messages waiting to be sended again.
The template in HA may work with ACK, I will try them.

[hek]

End-to-end ack is handled by the library. But re-sends are not.

If you request ack from controller (through the serial protocol), the node will automatically send one back.

If you request an ack from the node when sending in a value to controller, the gateway will send ack back to node automatically.

[martinhjelmare]

@hek

Thanks for clarifying!

Although I think the feature that the gateway answers with an ack, could possibly be a bad idea? Since the gateway is not really an endpoint, the message might not have reached the controller, or if the controller also want to acknowledge, two acks will be sent. What do you think about this?

[hek]

Agree that it isn't optimal now when the WiFi gateway option exist (where messages actually could get lost).

We would need updates of the protocol between GW and Controller to support this. We're also drafting a future binary protocol. I'll add a note about this.

[Mihai]

I think the problems may occur if a lot of messages are sent inside the NRF24 network. Sometimes it happened that two (ore more?) nodes (including the gateway) start sending almost in the same time.
One example is group switch: the controller send several messages shortly one after other; the node get the first message, set its relay accordingly and send back to the controller the new state; in this moment the controller should change also the corresponding switch on its interface; but the node may be busy to receive other messages or may be during reply transmission.
For me it happens that the node message doesn't reach the controller, the corresponding switch on the interface doesn't change even if the node get the message from the controller and perform the right command (for example a light on).
I propose several settings on MySensors side that could improve this a little bit, not to solve:

• increase the transmission speed from 250kb/s to 1 or even 2 Mb/s; this will shorter the transmission time by 4 or 8, so decrease the probability of collisions; the drawback is shorter radio range;
• increase the interval between node updates; the examples start with 30s, in my opinion it is not necessary to get temperature of a room so often, may be 1, 3 or 5 min is enough for most situations;
• node send the same message 2 or 3 times at very short random intervals without test anything else, like ACK (still testing here, may be ~20-100ms is ok).

All of those are quite simple actions.
I don't know how to modify the gateway to send the messages to the node as I mentioned - several times, only from the node to the gateway. Any advice (or better - example) could help.
For WiFi gateway, my opinion it that WiFi may not made troubles because its higher speed allow to exchange a lot of messages with the controller via broker, so no bottle neck here.
I have mentioned on the Bug Reports that the gateway act also for its relays if they have the same CHILD_ID as other node, this also could have a negative influence in message sending.
Of course, an automated mechanism end to end will be the best, but here the problem seems to be on the NRF24 network side, so some improvements are needed here.
An automated ACK between nodes and gateway that include resending of the message few more times at short random interval will help a lot.

[Mihai]

I have checked the serial monitor while stress the node with a lot of switch on/off and from time to time I got some errors like:

read: 0-0-10 s=9,c=1,t=2,pt=0,l=2,sg=0:ON
send: 10-10-0-0 s=9,c=�J*�pt=0,l=2,sg=0,st=ok:ON
send: 10-10-0-0 s=9,c=1,t=2,pt=0,l=2,�+�Vt=ok:ON
send: 10-10-0-0 s=9,c=1,t=2,pt=0,l=2,sg=0,st=ok:ON

More info here: http://forum.mysensors.org/topic/3144/multiple-sensor-node-freeze/2
So the missing messages could be related to this: some of them goes out with errors, if they go.

[martinhjelmare]

@Mihai

Nice investigating!

Talking about imrpovements, I don't think sending more messages arbitrarily is wise, when the radio space seems crowded already. I would say that ack and resend when ack absolutely failed, is the only real solution.

[rollercontainer]

There is a second way for end2end verification:
1.) Controller sends command "Relais: ON" -> Gateway -> Node
2.) Controller asks for the state of the Relais -> Gateway -> Node (reads pin and creates answer) -> Gateway -> Controller
3.) If the states doesnt match, the controller has to send the command again.

Requesting a state is done like here:
http://forum.mysensors.org/topic/1467/how-to-deal-with-a-request-for-information-from-controller/19

[martinhjelmare]

@rollercontainer

Hi!

What's the advantage using this method compared to ack?

[rollercontainer]

Endpoint to endpoint, not only from node to gateway. Controllers knows the exact state.

[martinhjelmare]

@rollercontainer

I think it's possible to use ack from controller to node, although I haven't tried this. I know Domoticz uses this. Don't know how effective though.

I think you can simplify your approach even more. The controller sends a command and waits for a state feedback from node. If no feedback is received within certain time limit, the command switch is reset to previous position. This is how it currently works in home assistant. Automatic resend is not implemented though.

[Mihai]

In HomeAssistant, if you use MQTT switch for example, with state and command topic, the command is send to the gateway when requested from the interface, but the switch on the interface change itself when it receive back the status from the gateway. If the message doesn't reach the destination and comes back with the new status, the interface doesn't change the switch state.
I encounter this situation a lot of times. After many tests, my findings show that the messages are lost between the nodes and the gateway, but I never miss a message between the gateway and the controller.
So, while it will be nice and useful to have an end-to-end implementation with ACK and automatically retries, the real problem can be solved if such mechanism will be available between nodes and gateway.