MQTT Broker gateway

[John]

@Damme
Clear!

And how about the internal messages? Are these somewhat compatible with the Serial API? Like this one now has a "I_GATEWAY_READY" internal var, this one complies then with the "Started!\n" message?

Also about the presentation (if the node does it), are these also passed through?

I know it's a bit question banging... But i just want to do it right.

[Damme]

@John No worries! :)
Presentation is On Todo.. Or Rather To think About list is presentation . I dont do anything with that as it is (Just ignores it). I havn't had any good idea for that yet.

Internal is nothing more or less, it connects and set a variable that it is connected. :) But sure, there could be transport for internal messages also.
I do have an experimental version there I can send internal messages to reboot node, reboot gateway and set clear read eeprom values. (over mqtt-protocol) but this s not finnished yet.. Ideas are welcome!

[Damme]

@John About Presentation ; My idea was that next version of the variable system I will either store some bytes in the MQTT gateway about what S_types to V_types, or hopefully the next protocol contains both those so the GW doesnt have to remember. And then add an extra directory to the MQTT path ( MyMQTT/[0-255]/[0-255]/S_[STRING]/V_[STRING]/[VALUE]

[John]

I think maybe it depends on the next protocol implementation. Personally i wouldn't mind if the full path mapping was /[0-255]/[0-255]/etc.. to keep (prog)mem print smaller (use the V_ and S_ array index values). It could leave space for more specific things or some extra gateway specific functions.

it would be nice if the mqtt gateway output was "cloned" to the serial output. Controllers should/could be able to map the S_ and V_ types internally. I'm not aware of the openhab config if that would allow this, or that such a setup would require a lot of coding at openhab side.

[EDIT]
It would be more difficult to read though if a path would only exist of numbers. But could also introduce easier maintainability in your library because you wouldn't have to keep track on type namings.
[/EDIT].

[Damme]

@John That was my first aproach, and I might set a define configuration to choose if the MQTT gateway should translate or not. As you say, the footprint would be much smaller.
Might be next version of the MQTT, slectable address layout :)

[John]

@Damme
That would be really cool :)

It would be nice though if there was a way to identify node id's with AUTO turned on. Maybe i'm missing something or do you have to know up front with auto enabled which node id is going to be assigned?

I'm just asking this because of identifying new/existing nodes in the network. Is it the V_SKETCH_NAME i can refer to?

And i hope some last questions protocol specific:

• Version compatibility is 3.1?
• All session are clean by default?

[Damme]

@John upon node startup it translates gw.sendSketchInfo("Battery Meter", "1.0"); into MyMQTT/20/255/V_SKETCH_NAME and version

so you could have a list as in my first post and it would get filled up with all your sketch names

@John ; Yes and I hope so :) (I might have missed some stuff, I do not take regard of QOS etc)
if its not working,, I'll try to fix it :)

[John]

@Damme
Thanks for all the info. It becomes more clear now, All nodes users add are persistent anyway.

[Damme]

@ToSa I'll update mqtt later with option to not translate V_TYPE into name string but to keep it as a byte, that would save a lot of memory and might solve problem you have. I'll keep you posted

[Zeph]

As I understand it:

The packet on the left would be from the node, the packet on the right would be to the node. Besides swapping sender and destination, the difference is that in 1.4 the gateway does not have the information to binary encode the value, so it sends the value as a string (payload type 0).
EDIT: added alternate MQTT respresentation of V_code as numeric

[Damme]

@Zeph but soon there will be a choise to translate v_ to text string or keep it as a byte value

[Zeph]

At first, I thought MQTT would be a great extension to the functionality, in particular because it might systematize the idea of routing messages to more than one destination - eg: to both a HA controller and to a cloud storage service.

So I saw these possible advantages of MQTT:

• Route to multiple destinations, dynamically updated
• Subscription wildcards to filter for desired subsets
• Possibly easier to build plug-ins for some HA controllers

As I look deeper into it, I'm coming to wonder about that.

• There are lighter weight ways to send to more than one destination
• Does subscription wildcarding meet the filtering needs well (below)?
• Is it easier or not?

I'm wondering what subscription wildcards really get used - how valuable is the type of filtering functionality offered by MQTT for our purposes? The meaingful patterns I would imagine are:

• everything
• everything from one node
• everything from one child of one node
• everything of type V_RAIN from one child of one node (ie: one variable)
• everything of type V_RAIN from anywhere

Are any of these (or other options) in use except "everything"? Do you anticipate that they will be?

Just as "route to multiple destinations" could be lighter, filtering could be done differently as well. For example, suppose you wanted to receive all report of inside temperature (7 sensors in your network) or humidity (5 sensors), but not outside temp or humidity and not other v codes. That might take 12 subscriptions - and they would have to be manually set up, or make use of manually entered metadata tables in the controller plug-in (eg: the plug-in might be configured to know which nodes had temp and/or humidity, and which ones were considered "inside", if it was written to handled that).

I could imagine that even if the subscription filtering wasn't highly useful, MQTT might make it easier to create plug-ins for some HA controllers, than using semicolon separated values (serial text format). Is that the case?

I'm very open to hearing other advantages of MQTT, I'm not against it, just trying to understand the benefits vs cost aspect better.

[Zeph]

The other extension that occurs to me, in the gateway-as-transparent-switch model, is that if we ever wanted to encode binary values OTA from the gateway to the node, the plug-in would have to tell the gateway which payload_type to use. In the Semicolon Separted Values format, that could be another parameter, eg:

<node id>;<child id>;<command>;<ack>;<v_code>;<payload_type>;<value>

Or it could be added to one of the other fields with a comma sub-delimiter, etc.

How would a controller plug in tell the MQTT gateway to use payload type P_UNIT16 to send to node 7?

I'm thinking that it would be done in the value string, rather than encoded into the topic. So maybe the value string goes from "321" to "4,321" or something like that.

[John]

@Damme
What happens if a node does not receive it's address when it requests one (because of failed send). It is not registered within the broker?

[ToSa]

@Damme said:

@ToSa I'll update mqtt later with option to not translate V_TYPE into name string but to keep it as a byte, that would save a lot of memory and might solve problem you have. I'll keep you posted

I could not follow the discussions the last few days - will definitely have a look over the weekend and let you know if that worked. Thanks!

[John]

@Damme
When the MQTT gateway prints information about itself like "started", is this done with node id 0 or 255?

[Damme]

@John 'started' is only a internal log message, I've changed how that part works.

I've also fixed a couple of bugs and will test the changes tomorrow before I publish them.
New functionallity there user can choose of MQTT should translate ID to V_TYPE or not too.

[John]

@Damme
Ok, my plugin will then just try to reconnect.

Ok, i will check that then probably the day after tomorrow, i will make my code take no translation as first and if not found try the naming (internally), because the server takes over the semantics anyway because it also is going to include a MQTT broker service.

Currently the MQTT implementation is being tested with an end user. Maybe he will help test the new possibility without V_TYPE translation when your changes are implemented.

[John]

@Damme
When does exactly the messages like "MyMQTT/[NODEID]/255/V_SKETCH_NAME" and/or "MyMQTT/[NODEID]/255/V_SKETCH_VERSION" appear? is this only when an address is assigned or always when a node is started?

[Damme]

@John then node is started, There was a bug not processing that message which is fixed now.
hasn't been pulled to official yet but can be found at https://github.com/Damme/MQTTGateway/commit/f10843f5efd9c37900624fe57275be11c69e3942