Mozilla Webthings MySensors add-on / adapter

[alowhum]

I've uploaded version 0.0.2, hopefully it will be accepted.

Changes are:

• Now supports not just serial, but also ethernet and MQTT. If anyone can test MQTT I would be very grateful.
• Now uses persistence internally - it remembers which node ID's are available better.
• Some minor improvements to S_ and V_ type support. I've tested most of my nodes now. They require a re-think sometimes in how things are presented.

Currently you need to always give names to the sketch and to the children. If this is a big problem for people I might make it so that if there is no name, a generic name will be given.

[Nca78]

Hello @alowhum, thank you for your efforts. Things gateway looks great until now.
I'm testing it only with temp/hum sensors at the moment, but I have a problem, I would like to be able to see the temperature on the "Things" tab, but I can only see the humidity.

When I use the weather add-on, in the parameters I can chose which type of sensor it is, and I have a choice between multilevel sensor (showing humidity level), and temperature.

But with my temp/hum sensors, I only have 2 "multilevel sensor" options

Meaning on Things page I only have choice between humidity and humidity, while the most important information for me would be the temperature.

Is there anything I should double check ? Messages types are ok in the script.

[Edit] After quickly looking at the code on Github and comparing with the source of the weather addon, shouldn't you change :

                if new_sub_type == 0: # V_TEMP
self._type.append('MultiLevelSensor')

to :

                if new_sub_type == 0: # V_TEMP
self._type.append('TemperatureSensor')

This is what they are using here :
https://github.com/mozilla-iot/weather-adapter/blob/master/lib/weather-device.js
on line 39:

this['@type'] = ['TemperatureSensor', 'MultiLevelSensor'];

[alowhum]

@nca78 I have the exact same issue. This is a known problem with the Mozilla Gateway.

But you're also right that this should be a temperature sensor. Good one! I will change it.

[alowhum]

@Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!

Thank you, it's working now, and the value I select in the sensor settings will be the one displayed on the "Things" page so I can have the temperature or humidity as I please.

But I had a lot of pain to get the new behaviour visible in the interface, after update sensors were still showing the 2 multi level sensors values. I had to deleted the sensors (which was also a painful process, interface stayed on the page after clicking the remove button and I had to remove sensors many times to get them removed).
It might just be a problem of cache (I have an obvious one when updating the floorplan), but it was really annoying.

I also can't see any data log, despite having it checked from the beginning after configuring the gateway. Did you do anything specific for that ? Any idea of what I should do to see it ? Same failure with the assistant, whatever I ask, I get the "Sorry, I didn't understand that."

[alowhum]

@nca78 great to hear the temperature is now optimally displayed.

The new code does indeed mean that you have to remove the old device and add it again. This time it will be added with the improved property definitions.

Removing the devices can fail if the MySensors add-on is not loaded / it hasn't detected your devices yet. After an upgrade like this, it's probably best to restart the Raspberry Pi and give it a minute to re-detect all your MySensors devices / restart your MySensors devices so they are properly rediscovered. If it's a cache issue, then that's probably something to take up with the Mozilla team.

The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.

Once the logging feature is enabled, you will see it as an option in the main menu.

I haven't used the assistant, so I don't know about that. That would be something interesting to explore, since I would also like to control MySensors devices with my voice.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.
Once the logging feature is enabled, you will see it as an option in the main menu.

Ok thank you, I see it now, it didn't notice it was there, I was expecting to see something in the "Event's log" menu of the sensor. It makes no sense to me to put a log menu where all sensors are mixed, it should be accessible from the sensor detail page.

One more problem with the MySensors add on. I accidentality unplugged my rpi and reconnected it back. Then the sensors were showing nothing, I had to go and reset them to run presentation and see data again. Is that "normal" behaviour ? The "persistant" checkbox is checked, so I was expecting the sensor definitions to stay memorized when I restart the gateway ?

[alowhum]

When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

Upon receiving the discover request, your devices then call the presentation function. Once a sketch name is received, the device is re-presented to the Gateway by the add-on, and it becomes 'alive' again.

But this presentation function only sends "S_types" to the server. Since S_types can often have multiple V_types, the properties of your device aren't added until they receive data.

In short, you will need to do this for every sketch:

• Make sure an actual sketch name is presented (as in: a string)
• Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
• Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:

MyMessage text_message(STATUS_CHILD_ID, V_TEXT); 

boolean send_all_values = true;

void presentation()
{
  // Send the sketch version information to the gateway and Controller
  sendSketchInfo(F("Name here"), F("1.0"));
 // Present your children here
 present(STATUS_CHILD_ID, S_INFO, F("Status"));

  send_all_values = true;
}

void send_values()
{
  send(text_message.setSensor(STATUS_CHILD_ID).set( F("Hello") ));
}


void loop()
{

  // Send the status of every child to the controller. This will initialise things there.
  if( send_all_values ){
    send_all_values = false;
    send_values();
  }
}

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

In short, you will need to do this for every sketch:

• Make sure an actual sketch name is presented (as in: a string)
• Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
• Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:

Ok thank you. I don't have names for values, nor use send. These are old sensors and are pretty hard to reprogram because of funky headers so I avoided as long as possible, but I guess I'll have to make the effort

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

I've done what you suggest and I still have the same problem with my Sensors.
They are button cell battery powered sensors so they sleep > 99.99% of the time, and I guess they are missing the discovery request. I know it's much easier said than done, but "persistance" should save the description of the registered sensors (in json format I suppose) and reload it, then update structure/definition if it receives an answer from the discovery request.

Object is still registered with the node id in the gateway, it's still in the interface with its values visible (at least, it seems to still be here maybe it's a cache thing ?) so why not restart to update those values when messages from known sensors are received by the MySensors gateway ?

[alowhum]

@Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

I see. The good news is I see the PyMySensors library manages smartsleep, in theory this should be enough to do the trick and receive the message to retrigger the presentation. I will update my nodes and try that.

[alowhum]

@nca78 Great, let me know how that goes.

I also did a test here, and you're right. If a node cannot re-present itself when the gateway restarts, then it will stay 'disconnected'.

I've changed how persistence works. It now always creates a persistence file. When you enable persistence, it will re-create all devices when the add-on is initialized. That would be more in line with what you expect from a feature like this anyway.

I'll do a few more tests and then try to get it approved.

[alowhum]

Version 0.0.5 has been accepted by Mozilla.

Improvements:

• More V_types supported, getting close to full support now.
• Better, more conservative implementation of 'capabilties' from the Mozilla IoT schema.
• If enabled, persistence is used to immediately re-create all MySensors devives upon reboot. Before, persistence was only used to avoid handing out duplicate node ID's. That has now become standard.
• Smarter way of dealing with node removal. It now also removes persistence data.
• Re-discovery process is faster now (asks one node ID per second to present itself). This is on top of the process that is already built into the PyMySensors library that this add-on is based on.

Perhaps it's getting time to add this to the controllers overview, and fill in all the support details? Please contact me about how to do that. // done

[Nca78]

@alowhum I can't seem to add any more my_sensors nodes with this version...

With or without persistance, I see nothing appearing in "Add Thing" view, and not much in the log, only this:
INFO : mysensors: AddonMangerProxy: recv: b'{"messageType":"startPairing","data":{"timeout":60,"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'

[Nca78]

I removed the add-on, I restarted, reinstalled the add on.
It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ? Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view.

2019-05-30 08:23:31.508 INFO   : mysensors: RECREATING DEVICES FROM PERSISTENCE
2019-05-30 08:23:31.510 INFO   : mysensors: -Recreating: BlackCircleSensor
2019-05-30 08:23:31.512 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.518 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.521 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.522 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 1
2019-05-30 08:23:31.525 INFO   : mysensors: new_description = Temperature
2019-05-30 08:23:31.527 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.528 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.530 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.532 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.534 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 2
2019-05-30 08:23:31.536 INFO   : mysensors: new_description = Humidity
2019-05-30 08:23:31.538 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.539 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
2019-05-30 08:23:31.541 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
2019-05-30 08:23:31.543 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated

Also it seems all files remain when the add-on is removed, I'm not sure if it's a problem with the add-on or with the gateway ? When re-adding the add-on I have a bunch of messages similar to those :

2019-05-30 08:23:20.911 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/crcmod already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.915 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/serial already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.919 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/voluptuous-0.11.5.egg-info already exists. Specify --upgrade to force replacement.
2019-05-30 08:23:20.929 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/pymysensors-0.18.0.egg-info already exists. Specify --upgrade to force replacement.

[alowhum]

@nca78 So the first issue is gone, right?

The short version:

Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

The long version:

If you disable persistence, it does not remove the persistence file. That file is always used to remember all the node ID's that have been handed out.

If you disable persistence, all that happens is that it no longer re-generates all the things after a restart of the gateway/add-on. If persistence is disabled, nodes will only show up once they present themselves. To speed this up, the add-on goes over the list of stored node ID's and asks all of them, one per second, to re-present themselves. Another way to get them to re-present themselves is to simply restart the node.

If you remove a thing in the Gateway, the latest version (0.0.5) also removes its data from the persistence file. In other words: if you remove a thing, you will most likely have to restart the node if you want it to show up on the 'add things' page again.

"It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

Correct, it should theoretically no longer be in the persistence file.

"Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

Then how do you know it's been recreated? From the log?

Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@nca78 So the first issue is gone, right?

No, not at all. I still can't add any node in MySensors.

Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

I did, it didn't work

"It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

Correct, it should theoretically no longer be in the persistence file.

Well from the log I believe it's still here ?

"Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

Then how do you know it's been recreated? From the log?

Yes, from the log as it's not visible anywhere else. And neither are new nodes. Very frustrating

Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.

No I never fiddled directly with the files. I think I will do it now, my guess is the problem is related to those messages I see in log when I remove the add-on:

2019-05-30 09:45:07.158 INFO   : Unloading MySensorsAdapter
2019-05-30 09:45:07.166 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadAdapter","data":{"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'
2019-05-30 09:45:07.179 INFO   : mysensors: Shutting down MySensors adapter
2019-05-30 09:45:07.181 INFO   : mysensors: MySensors adapter was unable to cleanly close PyMySensors
2019-05-30 09:45:07.183 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadPlugin","data":{"pluginId":"mysensors-adapter"}}'
2019-05-30 09:45:10.188 INFO   : Killing mysensors-adapter plugin.

[Nca78]

So connected with SSH and the directory is gone now.
I reinstalled the add-on and I didn't have those messages about files already existing, only success messages.

But I don't get why it still has my configuration for MySensors gateway URL etc. I install the add-on and after install it connects directly to the MySensors gateway.

I don't know what happened also but it seems I had a problem with the gateway too as I didn't have sensor info in it's log. Now I reprogrammed it and I see log when sensor sends and I have the sensor in the add sensor view.

No idea of what happened, but it seems problem was with my own sensors/gateway and not the add-on

[alowhum]

@Nca78 Glad to hear it.

I suspect that because part of the gateway had crashed, all the things you thought you did never actually happened. You never really deleted the thing, and hence the persistence data was still there. You never really deleted the add-on, and hence the files were still there.

[alowhum]

Version 0.0.7 has been created, and should hopefully be accepted soon.

It fixes the issue that the user's preference for which capability should be shown centrally was reset every time the add-on restarted.

Internally, the add-on always uses persistence. Whether devices are shown as 'connected' is now an option. You can either immediately show all MySensors nodes as 'connected' upon a restart, or wait until nodes have sent a signal.

// It has been accepted

[alowhum]

Version 0.1.0 has been accepted. Persistence is a bit faster and hopefully more robust. Hopefully it fixes an issue @rmalbers was experiencing.

[alowhum]

Version 0.1.2 has been uploaded. It has a lot of new features:

• Automatic serial port searching. User doesn't need to add a serial port ID manually anymore (but can if they want)
• Support for metric/imperial units. You can set your preference in the add-on settings.
• Support for multipleOf. This should help show a sane amount of decimals on numeric variables.
• Experimental MQTT support based on user feedback.
• Anticipates switch to version 0.9 of the WebThings gateway by implementing title support.

[alowhum]

The new version has been accepted.

[alowhum]

I have uploaded a new version (0.1.4) that should hopefully be accepted in the coming days.

It allows MySensors devices to better work with voice control.

Its main new ability is that it can split up a single MySensors device into several. The reason this is important is that in the webthings gateway you currently cannot change property names, only thing names. By essentially creating many things with only one property, this problem is circumvented.

It also has some small improvements in MySensors V-types support.

[Nca78]

Thanks you @alowhum, is the splitting systematic or done in settings/configured ?

[alowhum]

@nca78 It's a setting, it's optional.

[alowhum]

Version 0.1.4 has now been accepted.

[alowhum]

Version 1.0.0 has been submitted to Mozilla.

• Support for time-outs. After a user-defined period of not receiving any sign of life (data/heartbeat), a node will be set to disconnected.
• Support for the new WebThings thermostat capability.

[alowhum]

It has been accepted.

[Nca78]

Thank you !

[rmalbers]

Thanks, I have the timeout set (to 30 minutes) and it seems to be working, GREAT! In other news, it looks like they added metric to the gateway with the new release last night and all my temp readings reverted to 'Cent', the new default, until I switched it off 'metric', scared me for a second!

[alowhum]

@rmalbers Thanks for suggesting the idea.

@rmalbers said in Mozilla Webthings MySensors add-on / adapter:

In other news, it looks like they added metric to the gateway with the new release last night and all my temp readings reverted to 'Cent', the new default, until I switched it off 'metric', scared me for a second!

Yeah I have to figure out how the add-on can get this value from the gateway instead.

[alowhum]

A bit of a side-note, but I've created a theme (the very first one!). It also adds thermostat setpoint buttons.

[badmannen]

I been searching around in the forum but I cannot find a way how to send the Battery % so the WEBTHINGS will recognize it , can anyone give me a hint?

at the moment I report it in from my battery node with:
sendBatteryLevel(batteryPcnt);

but I assume this does not get forwarded to the controller?
so should I send it as a separate "child node ID , message?
and which one would be most clever?

thank you

[Nca78]

@badmannen said in Mozilla Webthings MySensors add-on / adapter:

so should I send it as a separate "child node ID , message?

This is what I do at the moment

[badmannen]

@nca78 Do you send it as a "DIMMER" to send the percentage?

[Nca78]

@badmannen not even, was working with the humidity percentage so I kept the same for the battery

MyMessage batPercentMsg(CHILD_ID_BATTERY, V_HUM);

[badmannen]

@nca78 aaa ok . perfect . Now I try with dimmer and I will see if it appears like I want otherwise I´ll try your way . thanks !

[alowhum]

Interesting. I'll see if there's an 'official' way to implement battery level.

// Asked a question on how to implement it.

[badmannen]

@alowhum did it your way and it works like a charm.

[badmannen]

I just wish Mozilla could introduce some more settings in their web interface. Just to edit nodes and bring back nodes that you deleted etc, in an easy way. The easy interface is pretty and all but it would be useful with some more settings options etc

[alowhum]

@badmannen said in Mozilla Webthings MySensors add-on / adapter:

@alowhum did it your way and it works like a charm.

I'm not sure what you're talking about?

[badmannen]

@alowhum a sorry, I was answering the person who wrote above

[badmannen]

Oh and another suggestion regarding the "timeout" feature for the nodes would be to make it possible to select the timeout per-node basis . especially since some are running constant and some are on battery etc .

[alowhum]

@badmannen I'd love that feature too, but for now I think it will have to wait until Mozilla implements per-device settings.

Meanwhile, a new version of the add-on (1.0.2) should be available soon.

• Added MQTT topics support
• Cloning can now be better controlled: child ID's above 100 will be cloned out into new things.
• Better support for S_Power's V_Types.

[alowhum]

Version 1.0.6 is up now. Like 1.0.5 it's a bugfix release.

1.0.5

• Fixed a bug that caused the WebThings Gateway to forget capability preferences.

1.0.6.

• It turned out that strings could lose leading zero's because they went through a function designed to remove zero's from floats. This impacted the Candle smart lock if users wanted to set a phone number that didn't start with a +.

[alowhum]

Version 1.0.8 was released a little while ago.

A lot was rewritten under the hood.

It also allows a serial device to also be a 'thing' itself. This is because the Candle Receiver implemented @mfalkvidd 's code to show transmission succes and error rate.

[alowhum]

If you want the latest developments if this addon, please use the Candle Controller instead of the Webthings Gateway. Candle is a fork of Webthings that I also work on. Going forward it's your best bet for updates.

https://www.candlesmarthome.com

[rmalbers]

@alowhum I see that one of the requirements for this new candle webthing gateway is "Sonoff Zigbee 3.0 USB stick". If I don't use any zigbee 'things', do I still need that for an install to work? I have mysensor things and native webthing things only. I lost remote access a month or so ago to my webthing controller (known issue but doesn't appear to be fixed) so I am thinking about switching to this 'new' candle controller. I use remote monitoring when I am away from the house to make sure the furnaces, etc. are working. Also, can I run my old webthing controller and this new candle webthing controller at the same time on my network while I get it going (probably a dumb question but I don't know)? I have another rasp pi I can install the new one on.