Mozilla Webthings MySensors add-on / adapter

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

I see. The good news is I see the PyMySensors library manages smartsleep, in theory this should be enough to do the trick and receive the message to retrigger the presentation. I will update my nodes and try that.

[alowhum]

@nca78 Great, let me know how that goes.

I also did a test here, and you're right. If a node cannot re-present itself when the gateway restarts, then it will stay 'disconnected'.

I've changed how persistence works. It now always creates a persistence file. When you enable persistence, it will re-create all devices when the add-on is initialized. That would be more in line with what you expect from a feature like this anyway.

I'll do a few more tests and then try to get it approved.

[alowhum]

Version 0.0.5 has been accepted by Mozilla.

Improvements:

• More V_types supported, getting close to full support now.
• Better, more conservative implementation of 'capabilties' from the Mozilla IoT schema.
• If enabled, persistence is used to immediately re-create all MySensors devives upon reboot. Before, persistence was only used to avoid handing out duplicate node ID's. That has now become standard.
• Smarter way of dealing with node removal. It now also removes persistence data.
• Re-discovery process is faster now (asks one node ID per second to present itself). This is on top of the process that is already built into the PyMySensors library that this add-on is based on.

Perhaps it's getting time to add this to the controllers overview, and fill in all the support details? Please contact me about how to do that. // done

[Nca78]

@alowhum I can't seem to add any more my_sensors nodes with this version...

With or without persistance, I see nothing appearing in "Add Thing" view, and not much in the log, only this:
INFO : mysensors: AddonMangerProxy: recv: b'{"messageType":"startPairing","data":{"timeout":60,"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'

[Nca78]

I removed the add-on, I restarted, reinstalled the add on.
It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ? Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view.

2019-05-30 08:23:31.508 INFO   : mysensors: RECREATING DEVICES FROM PERSISTENCE
2019-05-30 08:23:31.510 INFO   : mysensors: -Recreating: BlackCircleSensor
2019-05-30 08:23:31.512 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.518 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.521 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.522 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 1
2019-05-30 08:23:31.525 INFO   : mysensors: new_description = Temperature
2019-05-30 08:23:31.527 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.528 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.530 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.532 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.534 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 2
2019-05-30 08:23:31.536 INFO   : mysensors: new_description = Humidity
2019-05-30 08:23:31.538 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.539 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.541 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.543 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated

Also it seems all files remain when the add-on is removed, I'm not sure if it's a problem with the add-on or with the gateway ? When re-adding the add-on I have a bunch of messages similar to those :

2019-05-30 08:23:20.911 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/crcmod already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.915 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/serial already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.919 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/voluptuous-0.11.5.egg-info already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.929 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/pymysensors-0.18.0.egg-info already exists. Specify --upgrade to force replacement.

[alowhum]

@nca78 So the first issue is gone, right?

The short version:

Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

The long version:

If you disable persistence, it does not remove the persistence file. That file is always used to remember all the node ID's that have been handed out.

If you disable persistence, all that happens is that it no longer re-generates all the things after a restart of the gateway/add-on. If persistence is disabled, nodes will only show up once they present themselves. To speed this up, the add-on goes over the list of stored node ID's and asks all of them, one per second, to re-present themselves. Another way to get them to re-present themselves is to simply restart the node.

If you remove a thing in the Gateway, the latest version (0.0.5) also removes its data from the persistence file. In other words: if you remove a thing, you will most likely have to restart the node if you want it to show up on the 'add things' page again.

"It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

Correct, it should theoretically no longer be in the persistence file.

"Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

Then how do you know it's been recreated? From the log?

Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@nca78 So the first issue is gone, right?

No, not at all. I still can't add any node in MySensors.

Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

I did, it didn't work

"It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

Correct, it should theoretically no longer be in the persistence file.

Well from the log I believe it's still here ?

"Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

Then how do you know it's been recreated? From the log?

Yes, from the log as it's not visible anywhere else. And neither are new nodes. Very frustrating :(

Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.

No I never fiddled directly with the files. I think I will do it now, my guess is the problem is related to those messages I see in log when I remove the add-on:

2019-05-30 09:45:07.158 INFO   : Unloading MySensorsAdapter
2019-05-30 09:45:07.166 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadAdapter","data":{"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'
2019-05-30 09:45:07.179 INFO   : mysensors: Shutting down MySensors adapter
2019-05-30 09:45:07.181 INFO   : mysensors: MySensors adapter was unable to cleanly close PyMySensors
2019-05-30 09:45:07.183 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadPlugin","data":{"pluginId":"mysensors-adapter"}}'
2019-05-30 09:45:10.188 INFO   : Killing mysensors-adapter plugin.

[Nca78]

So connected with SSH and the directory is gone now.
I reinstalled the add-on and I didn't have those messages about files already existing, only success messages.

But I don't get why it still has my configuration for MySensors gateway URL etc. I install the add-on and after install it connects directly to the MySensors gateway.

I don't know what happened also but it seems I had a problem with the gateway too as I didn't have sensor info in it's log. Now I reprogrammed it and I see log when sensor sends and I have the sensor in the add sensor view.

No idea of what happened, but it seems problem was with my own sensors/gateway and not the add-on :)

[alowhum]

@Nca78 Glad to hear it.

I suspect that because part of the gateway had crashed, all the things you thought you did never actually happened. You never really deleted the thing, and hence the persistence data was still there. You never really deleted the add-on, and hence the files were still there.

[alowhum]

Version 0.0.7 has been created, and should hopefully be accepted soon.

It fixes the issue that the user's preference for which capability should be shown centrally was reset every time the add-on restarted.

Internally, the add-on always uses persistence. Whether devices are shown as 'connected' is now an option. You can either immediately show all MySensors nodes as 'connected' upon a restart, or wait until nodes have sent a signal.

// It has been accepted

[alowhum]

Version 0.1.0 has been accepted. Persistence is a bit faster and hopefully more robust. Hopefully it fixes an issue @rmalbers was experiencing.

[alowhum]

Version 0.1.2 has been uploaded. It has a lot of new features:

• Automatic serial port searching. User doesn't need to add a serial port ID manually anymore (but can if they want)
• Support for metric/imperial units. You can set your preference in the add-on settings.
• Support for multipleOf. This should help show a sane amount of decimals on numeric variables.
• Experimental MQTT support based on user feedback.
• Anticipates switch to version 0.9 of the WebThings gateway by implementing title support.

[alowhum]

The new version has been accepted.

[alowhum]

I have uploaded a new version (0.1.4) that should hopefully be accepted in the coming days.

It allows MySensors devices to better work with voice control.

Its main new ability is that it can split up a single MySensors device into several. The reason this is important is that in the webthings gateway you currently cannot change property names, only thing names. By essentially creating many things with only one property, this problem is circumvented.

It also has some small improvements in MySensors V-types support.

[Nca78]

Thanks you @alowhum, is the splitting systematic or done in settings/configured ?

[alowhum]

@nca78 It's a setting, it's optional.

[alowhum]

Version 0.1.4 has now been accepted.

[alowhum]

Version 1.0.0 has been submitted to Mozilla.

• Support for time-outs. After a user-defined period of not receiving any sign of life (data/heartbeat), a node will be set to disconnected.
• Support for the new WebThings thermostat capability.

[alowhum]

It has been accepted.

[Nca78]

Thank you !