Build retry funtionality into the mysensors library

[Yveaux]

@alowhum said in Build retry funtionality into the mysensors library:

Ideally the send function would try to re-send in the background.

I agree. The send function should be fire and forget, with e.g. a callback to indicate success/failure. The number of retries or delivery timeout should be configurable so the stack can handle the resending in the background.
Most users will probably not be interested in the result because what can you do if delivery eventually failed after many retries?
This however requires buffering messages, increasing RAM usage.

Otherwise the code would block the repeater functionality?

Not necessarily I guess. The stack is handling the send and could also handle repeater function while doing so.

[rozpruwacz]

@mfalkvidd said in Build retry funtionality into the mysensors library:

Do you statically allocate the ram needed to store the transmitted values, or are you able to do it dynamically without causing fragmentation?

For example You have a node with two sensors, then there are two MyMessage objects defined statically.

How much extra ram is required?

The size of MyMessage for every message :)

Is there support for larger messages, such as V_TEXT?
To My knowledge in MySensors there is only one size of MyMessage object which is a union of multiple represenstations of the message content and the size of MyMessage is MAX_MESSAGE_LENGTH. So I don't understand the question :)

How do you handle if the same actuator is changed multiple times before the earlier changes have been reliably delivered?

The last value of the actuator is taken. You can imagine that there are two threads. First is setting the atomic value shared between threads, and second is reading this value and sending it to the gateway. If first thread writes two times before second thread sends first time, the first value is not sent.

My approach is memory hungry if there is a lot of distinct values to send to the gateway. May be it could be improved to reuse single message in the background.

One important thing is that there is a differenc in approach to battery powered (sleeping) nodes and powered nodes.
In powered nodes, there is not problem going to forever loop trying to deliver message to the gateway.
My algorithm is essantial for battery powered nodes, where banging messages in forever loop will drain the battery quickly if for example gateway goes down for a while.

You can look into my code here:
https://github.com/mczerski/MySensorsToolkit - this is the toolkit library with all the logic implemented
https://github.com/mczerski/MyMultiSensor

• this is example use of the toolkit library

[mfalkvidd]

Thanks for explaining. For some use cases (few messages per node) this could be a good solution. If I understand the code correctly, each MyMessage needs 40 bytes. For atmega328 this could be quite limiting (many sketches are tight on ram already), but for devices with more ram 40 bytes per message won't be a problem.

@rozpruwacz said in Build retry funtionality into the mysensors library:

So I don't understand the question

Since you're creating a full instance of the MyMessage, my question doesn't make sense. I thought you were doing something more "clever" to minimize ram usage, hence the question.

[rozpruwacz]

@mfalkvidd I believe that my code have a lot of room for optimisations. From the ram perspective having multiple union structures that can hold also text up to 32 chars (?) is not optimal. One way of optimising the ram usage would be to keep one MyMessage instance, and initialise this message just before sending it with a required value. This way the ram usage would be 1 MyMessage + sum(sizeof(value) for value in sensor_values) where sensor_values are current sensors/actuator readings.

[mfalkvidd]

@rozpruwacz yes that should be sufficient, at least if we only plan to support messages to the gateway (not to other nodes) and ditch repeater support.

I guess some sort of timestamp when the last send attempt occurred, and a retry counter is needed to be stored per message as well?

[rozpruwacz]

@mfalkvidd said in Build retry funtionality into the mysensors library:

I guess some sort of timestamp when the last send attempt occurred, and a retry counter is needed to be stored per message as well?

Yes, could be useful for debugging purposes. In the ideal setup retry counter should be always 0 :)

[mfalkvidd]

@rozpruwacz you use a constant interval between retries? (No exponential backoff)?

[rozpruwacz]

@mfalkvidd said in Build retry funtionality into the mysensors library:

(No exponential backoff)?

its expotential, but no more than hardcoded value (don't remember what value)

[Sergio Rius]

@mfalkvidd said in Build retry funtionality into the mysensors library:

How do you handle if the same actuator is changed multiple times before the earlier changes have been reliably delivered?

How MQTT qos 1 does? I think it should be similar to it.

Also I don't think changes in between should be dropped. That would be like dropouts in a metered system (logged to influx, fEx) and probably do strange things with scenes and group switching.

[rozpruwacz]

@sergio-rius said in Build retry funtionality into the mysensors library:

How MQTT qos 1 does? I think it should be similar to it.
Also I don't think changes in between should be dropped. That would be like dropouts in a metered system (logged to influx, fEx) and probably do strange things with scenes and group switching.

But MQTT broker is not running on the 1kB ram mcu. Having lots of memory available it is not a brain teaser to implement such functionality. For a small mcu You have to make some compromise. In one case droping messages is completely ok, but for other is not. It may turn out that there is no ONE algorithm that will fit all cases ...

[alowhum]

@rozpruwacz said in Build retry funtionality into the mysensors library:

It may turn out that there is no ONE algorithm that will fit all cases ...

Agreed. It seems to me that a small home with about 10 to 15 devices should be able to run on Arduino Nano's and offer a decent level of predictability. If all devices send, on average, one message per minute, then this would mean one message every 70 milliseconds. Most of these devices will be connected to the gateway directly. Let's say half of them require

So in a basic home situation a buffer for two messages would be fine, and three would be a luxury.

If you want to run a large scale sensor network, then it makes sense to upgrade your parts too (bigger antenna on the gateway, more ram on nodes that extend the network).

Retry functionality, in the normal home scenario, probably isn't so much about how many messages are buffered, but about how long people turn on the microwave oven, which disrupts the network. So for me it's about having control over the (exponential) time period that the node keeps retrying. The memory/buffering capacity of my home network is fine, it's just that I don't want to implement this "keep retrying for longer" routine myself.

Which #define values could I already change today to get the nodes to keep retrying for longer than just a few seconds?

[Yveaux]

@alowhum said in Build retry funtionality into the mysensors library:

So in a basic home situation a buffer for two messages would be fine, and three would be a luxury.

Each nrf24 has this luxury of a 3 message hardware buffer :-)

[Sergio Rius]

@rozpruwacz said in Build retry funtionality into the mysensors library:

But MQTT broker is not running on the 1kB ram mcu

I'm not saying to make a mqtt broker run on an arduino. Just picking the process logic as a guideline.
MQTT is not a protocol made for raspberries, it's only that often run on them.

[mfalkvidd]

The main thing mqtt has (that MySensors doesn’t) is ”packet identifier” in mqtt lingo. It is similar to tcp’s sequence number.

Mqtt also assumes that

• the broker (I guess the analogue is the MySensors gateway) has persistent storage, or at least sufficient ram to buffer all messages until they have been marked as delivered
• everything is single hop (no repeaters)
• there is a DUP flag
• clients have sufficient ram to buffer all outgoing messages until they have been marked as delivered, and has timers to retransmit messages

I am not sure how mqtt handles ordering of messages. I think mqtt doesn’t guarantee ordering, regardless of qos level.

[Sergio Rius]

Voilá.
And it maintains a live inventory of clients.
Some points are so difficult to achieve. But I was looking at message identification as a response to the sequence switch problem.

[efix durov]

I am not sure how retrying even more times will help anything