Mozilla Webthings MySensors add-on / adapter

[alowhum]

This is an add-on for the excellent Mozilla WebThings Gateway that will allow MySensors devices to work with it.

.

THE MOZILLA WEBTHINGS GATEWAY

This smart home software, made by the people behind Firefox, focuses on being very user friendly. Here are my MySensors devices in the things overview:

You can also log data with it:

.

INSTALLATION

Just select the add-on via the gateway interface. You're done!

.

Q.
My things are not showing up?

When the add-on is installed, you will need to restart your nodes so that they present themselves. Once you've restarted all your nodes, click on the (+) button on the things page to add them to your list of things.

.

Q.
Not all my children have been turned into properties?

Because of the way that things are currently developed, a new child/property will only be added to the device if a 'set' message for it is received. That's because only then does it become clear with which V_TYPE the S_INFO type is populated.

For example, let's say you send a presentation message for a button:

present(RELAY_1, S_BINARY, F("Relay 1"));

This child won't show up as a property until you tell the WebThings Gateway if that button should be in the on of off position. So in your setup function you should also do something like:

send(relaymsg.set(0));

.

Q.
Is type X or Y supported?

The controller page on this website can give you an indication. You can also check how your preferred sensor type is supported by looking at the actual code at the bottom of this file.

.

Q.
Can I make changes to the code?

Yes you can, and help is always welcome. The code can be found here:

https://github.com/createcandle/Webthings-mysensors-adapter

It's based on the excellent work on pyMySensors by Theo Lind, as well as the example add-on created by Michael Stegeman, who works for Mozilla. Thanks to both!

If you want to use the code manually, perhaps to co-develop it, here's how:

• Enable SSH (under settings -> developer) and log into the device as normal.
• Download the code via Github:

cd ~/.mozilla-iot/addons/
git clone https://github.com/createcandle/Webthings-mysensors-adapter.git

Now delete the .tgz file and restart the gateway so it no longer loads the .tgz file.

sudo systemctl restart mozilla-iot-gateway.service

Now you can run the code manually to test it.

python3 main.py

You will see a lot of messages scroll by.

The three most important files are in the pkg folder.

[alowhum]

I've uploaded the BETA version.

• Improved speed thanks to asynchronous handling of incoming messages.
• Now supports most sensors
• Now supports most actuators.

Missing or just partially implemented are:

• Thermostat/HVAC devices. The Gateway does not support them yet.
• IR sender/receiver

Please test it and let me know how it goes :-)

[alowhum]

You can now just clone the Github repository and it should automatically become available in your add-ons list.

What's more, I've created a request to add the adapter to the official list. If it is accepted, then in the future you will be able to just select the add-on from the list in the interface. No need to open a terminal window anymore.

[alowhum]

Great news: the add-on has been accepted. You can now just select the add-on through the interface of the Mozilla Webthings Gateway.

Here is it is after installation:

Once installed you will see the option to change settings:

Currently these settings don't do anything yet. That will be handled in the next release. First in line will be the ethernet gateway option. In theory the MQTT option should also be possible.

[FotoFieber]

@alowhum
Thanks for the integration. I think it is a good pattern to use webthings with mysensors. You can get security with encryption and signing.

The webthings platform offers direct integration for arduino devices (esp8266 and esp32) whicht don't seem adequate to me. They don't offer encryption or signing: webthings (mozilla iot). Maybe, it is a design problem of the webthings api? Or I am to stupid to get it all... :)

[alowhum]

I've uploaded version 0.0.2, hopefully it will be accepted.

Changes are:

• Now supports not just serial, but also ethernet and MQTT. If anyone can test MQTT I would be very grateful.
• Now uses persistence internally - it remembers which node ID's are available better.
• Some minor improvements to S_ and V_ type support. I've tested most of my nodes now. They require a re-think sometimes in how things are presented.

Currently you need to always give names to the sketch and to the children. If this is a big problem for people I might make it so that if there is no name, a generic name will be given.

[Nca78]

Hello @alowhum, thank you for your efforts. Things gateway looks great until now.
I'm testing it only with temp/hum sensors at the moment, but I have a problem, I would like to be able to see the temperature on the "Things" tab, but I can only see the humidity.

When I use the weather add-on, in the parameters I can chose which type of sensor it is, and I have a choice between multilevel sensor (showing humidity level), and temperature.

But with my temp/hum sensors, I only have 2 "multilevel sensor" options

Meaning on Things page I only have choice between humidity and humidity, while the most important information for me would be the temperature.

Is there anything I should double check ? Messages types are ok in the script.

[Edit] After quickly looking at the code on Github and comparing with the source of the weather addon, shouldn't you change :

                if new_sub_type == 0: # V_TEMP
self._type.append('MultiLevelSensor')

to :

                if new_sub_type == 0: # V_TEMP
self._type.append('TemperatureSensor')

This is what they are using here :
https://github.com/mozilla-iot/weather-adapter/blob/master/lib/weather-device.js
on line 39:

this['@type'] = ['TemperatureSensor', 'MultiLevelSensor'];

[alowhum]

@nca78 I have the exact same issue. This is a known problem with the Mozilla Gateway.

But you're also right that this should be a temperature sensor. Good one! I will change it.

[alowhum]

@Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!

Thank you, it's working now, and the value I select in the sensor settings will be the one displayed on the "Things" page so I can have the temperature or humidity as I please.

But I had a lot of pain to get the new behaviour visible in the interface, after update sensors were still showing the 2 multi level sensors values. I had to deleted the sensors (which was also a painful process, interface stayed on the page after clicking the remove button and I had to remove sensors many times to get them removed).
It might just be a problem of cache (I have an obvious one when updating the floorplan), but it was really annoying.

I also can't see any data log, despite having it checked from the beginning after configuring the gateway. Did you do anything specific for that ? Any idea of what I should do to see it ? Same failure with the assistant, whatever I ask, I get the "Sorry, I didn't understand that."

[alowhum]

@nca78 great to hear the temperature is now optimally displayed.

The new code does indeed mean that you have to remove the old device and add it again. This time it will be added with the improved property definitions.

Removing the devices can fail if the MySensors add-on is not loaded / it hasn't detected your devices yet. After an upgrade like this, it's probably best to restart the Raspberry Pi and give it a minute to re-detect all your MySensors devices / restart your MySensors devices so they are properly rediscovered. If it's a cache issue, then that's probably something to take up with the Mozilla team.

The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.

Once the logging feature is enabled, you will see it as an option in the main menu.

I haven't used the assistant, so I don't know about that. That would be something interesting to explore, since I would also like to control MySensors devices with my voice.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.
Once the logging feature is enabled, you will see it as an option in the main menu.

Ok thank you, I see it now, it didn't notice it was there, I was expecting to see something in the "Event's log" menu of the sensor. It makes no sense to me to put a log menu where all sensors are mixed, it should be accessible from the sensor detail page.

One more problem with the MySensors add on. I accidentality unplugged my rpi and reconnected it back. Then the sensors were showing nothing, I had to go and reset them to run presentation and see data again. Is that "normal" behaviour ? The "persistant" checkbox is checked, so I was expecting the sensor definitions to stay memorized when I restart the gateway ?

[alowhum]

When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

Upon receiving the discover request, your devices then call the presentation function. Once a sketch name is received, the device is re-presented to the Gateway by the add-on, and it becomes 'alive' again.

But this presentation function only sends "S_types" to the server. Since S_types can often have multiple V_types, the properties of your device aren't added until they receive data.

In short, you will need to do this for every sketch:

• Make sure an actual sketch name is presented (as in: a string)
• Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
• Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:

MyMessage text_message(STATUS_CHILD_ID, V_TEXT); 

boolean send_all_values = true;

void presentation()
{
  // Send the sketch version information to the gateway and Controller
  sendSketchInfo(F("Name here"), F("1.0"));
 // Present your children here
 present(STATUS_CHILD_ID, S_INFO, F("Status"));

  send_all_values = true;
}

void send_values()
{
  send(text_message.setSensor(STATUS_CHILD_ID).set( F("Hello") ));
}


void loop()
{

  // Send the status of every child to the controller. This will initialise things there.
  if( send_all_values ){
    send_all_values = false;
    send_values();
  }
}

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

In short, you will need to do this for every sketch:

• Make sure an actual sketch name is presented (as in: a string)
• Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
• Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:

Ok thank you. I don't have names for values, nor use send. These are old sensors and are pretty hard to reprogram because of funky headers so I avoided as long as possible, but I guess I'll have to make the effort :D

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

I've done what you suggest and I still have the same problem with my Sensors.
They are button cell battery powered sensors so they sleep > 99.99% of the time, and I guess they are missing the discovery request. I know it's much easier said than done, but "persistance" should save the description of the registered sensors (in json format I suppose) and reload it, then update structure/definition if it receives an answer from the discovery request.

Object is still registered with the node id in the gateway, it's still in the interface with its values visible (at least, it seems to still be here maybe it's a cache thing ?) so why not restart to update those values when messages from known sensors are received by the MySensors gateway ?

[alowhum]

@Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

[Nca78]

@alowhum said in Mozilla Webthings MySensors add-on / adapter:

@Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

I see. The good news is I see the PyMySensors library manages smartsleep, in theory this should be enough to do the trick and receive the message to retrigger the presentation. I will update my nodes and try that.

[alowhum]

@nca78 Great, let me know how that goes.

I also did a test here, and you're right. If a node cannot re-present itself when the gateway restarts, then it will stay 'disconnected'.

I've changed how persistence works. It now always creates a persistence file. When you enable persistence, it will re-create all devices when the add-on is initialized. That would be more in line with what you expect from a feature like this anyway.

I'll do a few more tests and then try to get it approved.

[alowhum]

Version 0.0.5 has been accepted by Mozilla.

Improvements:

• More V_types supported, getting close to full support now.
• Better, more conservative implementation of 'capabilties' from the Mozilla IoT schema.
• If enabled, persistence is used to immediately re-create all MySensors devives upon reboot. Before, persistence was only used to avoid handing out duplicate node ID's. That has now become standard.
• Smarter way of dealing with node removal. It now also removes persistence data.
• Re-discovery process is faster now (asks one node ID per second to present itself). This is on top of the process that is already built into the PyMySensors library that this add-on is based on.

Perhaps it's getting time to add this to the controllers overview, and fill in all the support details? Please contact me about how to do that. // done

[Nca78]

@alowhum I can't seem to add any more my_sensors nodes with this version...

With or without persistance, I see nothing appearing in "Add Thing" view, and not much in the log, only this:
INFO : mysensors: AddonMangerProxy: recv: b'{"messageType":"startPairing","data":{"timeout":60,"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'