Mozilla Webthings MySensors add-on / adapter


  • Plugin Developer

    This is an add-on for the excellent Mozilla WebThings Gateway that will allow MySensors devices to work with it.

    .

    THE MOZILLA WEBTHINGS GATEWAY

    This smart home software, made by the people behind Firefox, focuses on being very user friendly. Here are my MySensors devices in the things overview:

    0_1556012649800_mysensors-things.png

    You can also log data with it:

    0_1556012681000_mysensors-log.png

    .

    INSTALLATION

    Just select the add-on via the gateway interface. You're done!

    .

    Q.
    My things are not showing up?

    When the add-on is installed, you will need to restart your nodes so that they present themselves. Once you've restarted all your nodes, click on the (+) button on the things page to add them to your list of things.

    .

    Q.
    Not all my children have been turned into properties?

    Because of the way that things are currently developed, a new child/property will only be added to the device if a 'set' message for it is received. That's because only then does it become clear with which V_TYPE the S_INFO type is populated.

    For example, let's say you send a presentation message for a button:

    present(RELAY_1, S_BINARY, F("Relay 1"));
    

    This child won't show up as a property until you tell the WebThings Gateway if that button should be in the on of off position. So in your setup function you should also do something like:

    send(relaymsg.set(0));
    

    .

    Q.
    Is type X or Y supported?

    The controller page on this website can give you an indication. You can also check how your preferred sensor type is supported by looking at the actual code at the bottom of this file.

    .

    Q.
    Can I make changes to the code?

    Yes you can, and help is always welcome. The code can be found here:

    https://github.com/createcandle/Webthings-mysensors-adapter

    It's based on the excellent work on pyMySensors by Theo Lind, as well as the example add-on created by Michael Stegeman, who works for Mozilla. Thanks to both!

    If you want to use the code manually, perhaps to co-develop it, here's how:

    • Enable SSH (under settings -> developer) and log into the device as normal.
    • Download the code via Github:
    cd ~/.mozilla-iot/addons/
    git clone https://github.com/createcandle/Webthings-mysensors-adapter.git
    

    Now delete the .tgz file and restart the gateway so it no longer loads the .tgz file.

    sudo systemctl restart mozilla-iot-gateway.service
    

    Now you can run the code manually to test it.

    python3 main.py
    

    You will see a lot of messages scroll by.

    The three most important files are in the pkg folder.


  • Plugin Developer

    I've uploaded the BETA version.

    • Improved speed thanks to asynchronous handling of incoming messages.
    • Now supports most sensors
    • Now supports most actuators.

    Missing or just partially implemented are:

    • Thermostat/HVAC devices. The Gateway does not support them yet.
    • IR sender/receiver

    Please test it and let me know how it goes 🙂


  • Plugin Developer

    You can now just clone the Github repository and it should automatically become available in your add-ons list.

    What's more, I've created a request to add the adapter to the official list. If it is accepted, then in the future you will be able to just select the add-on from the list in the interface. No need to open a terminal window anymore.


  • Plugin Developer

    Great news: the add-on has been accepted. You can now just select the add-on through the interface of the Mozilla Webthings Gateway.

    Here is it is after installation:
    0_1556265796800_adapter-overview.png

    Once installed you will see the option to change settings:
    0_1556265827400_adapter-settings.png

    Currently these settings don't do anything yet. That will be handled in the next release. First in line will be the ethernet gateway option. In theory the MQTT option should also be possible.


  • Hardware Contributor

    @alowhum
    Thanks for the integration. I think it is a good pattern to use webthings with mysensors. You can get security with encryption and signing.

    The webthings platform offers direct integration for arduino devices (esp8266 and esp32) whicht don't seem adequate to me. They don't offer encryption or signing: webthings (mozilla iot). Maybe, it is a design problem of the webthings api? Or I am to stupid to get it all... 🙂


  • Plugin Developer

    I've uploaded version 0.0.2, hopefully it will be accepted.

    Changes are:

    • Now supports not just serial, but also ethernet and MQTT. If anyone can test MQTT I would be very grateful.
    • Now uses persistence internally - it remembers which node ID's are available better.
    • Some minor improvements to S_ and V_ type support. I've tested most of my nodes now. They require a re-think sometimes in how things are presented.

    Currently you need to always give names to the sketch and to the children. If this is a big problem for people I might make it so that if there is no name, a generic name will be given.


  • Hardware Contributor

    Hello @alowhum, thank you for your efforts. Things gateway looks great until now.
    I'm testing it only with temp/hum sensors at the moment, but I have a problem, I would like to be able to see the temperature on the "Things" tab, but I can only see the humidity.

    When I use the weather add-on, in the parameters I can chose which type of sensor it is, and I have a choice between multilevel sensor (showing humidity level), and temperature.
    0_1557643455240_d94da146-58eb-4858-b143-65ce26076f09-image.png

    But with my temp/hum sensors, I only have 2 "multilevel sensor" options
    0_1557643640319_9e10958d-0c86-41ae-a54b-edf8fe7a7872-image.png

    Meaning on Things page I only have choice between humidity and humidity, while the most important information for me would be the temperature.
    0_1557643695744_677c93b3-e2b3-4456-ad59-8b68552e58b4-image.png

    Is there anything I should double check ? Messages types are ok in the script.

    [Edit] After quickly looking at the code on Github and comparing with the source of the weather addon, shouldn't you change :

                    if new_sub_type == 0: # V_TEMP
    self._type.append('MultiLevelSensor')
    

    to :

                    if new_sub_type == 0: # V_TEMP
    self._type.append('TemperatureSensor')
    

    This is what they are using here :
    https://github.com/mozilla-iot/weather-adapter/blob/master/lib/weather-device.js
    on line 39:

    this['@type'] = ['TemperatureSensor', 'MultiLevelSensor'];
    

  • Plugin Developer

    @nca78 I have the exact same issue. This is a known problem with the Mozilla Gateway.

    But you're also right that this should be a temperature sensor. Good one! I will change it.


  • Plugin Developer

    @Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!


  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    @Nca78 Version 0.0.4 has been accepted, and has the change you proposed. Thanks for thinking along!

    Thank you, it's working now, and the value I select in the sensor settings will be the one displayed on the "Things" page so I can have the temperature or humidity as I please.

    But I had a lot of pain to get the new behaviour visible in the interface, after update sensors were still showing the 2 multi level sensors values. I had to deleted the sensors (which was also a painful process, interface stayed on the page after clicking the remove button and I had to remove sensors many times to get them removed).
    It might just be a problem of cache (I have an obvious one when updating the floorplan), but it was really annoying.

    I also can't see any data log, despite having it checked from the beginning after configuring the gateway. Did you do anything specific for that ? Any idea of what I should do to see it ? Same failure with the assistant, whatever I ask, I get the "Sorry, I didn't understand that."


  • Plugin Developer

    @nca78 great to hear the temperature is now optimally displayed.

    The new code does indeed mean that you have to remove the old device and add it again. This time it will be added with the improved property definitions.

    Removing the devices can fail if the MySensors add-on is not loaded / it hasn't detected your devices yet. After an upgrade like this, it's probably best to restart the Raspberry Pi and give it a minute to re-detect all your MySensors devices / restart your MySensors devices so they are properly rediscovered. If it's a cache issue, then that's probably something to take up with the Mozilla team.

    The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.

    Once the logging feature is enabled, you will see it as an option in the main menu.

    I haven't used the assistant, so I don't know about that. That would be something interesting to explore, since I would also like to control MySensors devices with my voice.


  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    The data log should work just fine. At the moment you need to enable the log function under settings, because it's a beta feature. it will probably be a normal feature in the next release of the WebThings Gateway.
    Once the logging feature is enabled, you will see it as an option in the main menu.

    Ok thank you, I see it now, it didn't notice it was there, I was expecting to see something in the "Event's log" menu of the sensor. It makes no sense to me to put a log menu where all sensors are mixed, it should be accessible from the sensor detail page.

    One more problem with the MySensors add on. I accidentality unplugged my rpi and reconnected it back. Then the sensors were showing nothing, I had to go and reset them to run presentation and see data again. Is that "normal" behaviour ? The "persistant" checkbox is checked, so I was expecting the sensor definitions to stay memorized when I restart the gateway ?


  • Plugin Developer

    When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

    Upon receiving the discover request, your devices then call the presentation function. Once a sketch name is received, the device is re-presented to the Gateway by the add-on, and it becomes 'alive' again.

    But this presentation function only sends "S_types" to the server. Since S_types can often have multiple V_types, the properties of your device aren't added until they receive data.

    In short, you will need to do this for every sketch:

    • Make sure an actual sketch name is presented (as in: a string)
    • Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
    • Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:
    
    MyMessage text_message(STATUS_CHILD_ID, V_TEXT); 
    
    boolean send_all_values = true;
    
    void presentation()
    {
      // Send the sketch version information to the gateway and Controller
      sendSketchInfo(F("Name here"), F("1.0"));
     // Present your children here
     present(STATUS_CHILD_ID, S_INFO, F("Status"));
    
      send_all_values = true;
    }
    
    void send_values()
    {
      send(text_message.setSensor(STATUS_CHILD_ID).set( F("Hello") ));
    }
    
    
    void loop()
    {
    
      // Send the status of every child to the controller. This will initialise things there.
      if( send_all_values ){
        send_all_values = false;
        send_values();
      }
    }
    
    

  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    In short, you will need to do this for every sketch:

    • Make sure an actual sketch name is presented (as in: a string)
    • Make sure that in your presentation function your children are also given a name. This is not mandatory, but it's highly recommended, since it's the only way to name a property.
    • Everytime the presentation function is called, you should also try to re-initiaise all properties by issueing a send() for all children that sends the current value. I usually do this:

    Ok thank you. I don't have names for values, nor use send. These are old sensors and are pretty hard to reprogram because of funky headers so I avoided as long as possible, but I guess I'll have to make the effort 😄


  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    When the Pi is reset a discovery request is sent to the ID's of all remembered devices (this is the only thing the persistence is used for at the moment). There is a delay between each device, so as not to make them all respond at the same time.

    I've done what you suggest and I still have the same problem with my Sensors.
    They are button cell battery powered sensors so they sleep > 99.99% of the time, and I guess they are missing the discovery request. I know it's much easier said than done, but "persistance" should save the description of the registered sensors (in json format I suppose) and reload it, then update structure/definition if it receives an answer from the discovery request.

    Object is still registered with the node id in the gateway, it's still in the interface with its values visible (at least, it seems to still be here maybe it's a cache thing ?) so why not restart to update those values when messages from known sensors are received by the MySensors gateway ?


  • Plugin Developer

    @Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

    Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.


  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    @Nca78 I hear you. Technically this would be possible, as all the values are actually stored in a JSON file already. Basing everything of this persistence file was actually the way the first version of the add-on worked. But then I learnt of the issues with it. The PyMySensors library recommended working based on incoming messages instead of relying on the persistence JSON file.

    Currently the persistence option is just there to make sure that two nodes don't get handed the same node ID from the controller, which is something it seems can happen if persistence is disabled. Perhaps a 'best of both worlds' version is possible where the persistence file is also used somehow. I'll think about it.

    I see. The good news is I see the PyMySensors library manages smartsleep, in theory this should be enough to do the trick and receive the message to retrigger the presentation. I will update my nodes and try that.


  • Plugin Developer

    @nca78 Great, let me know how that goes.

    I also did a test here, and you're right. If a node cannot re-present itself when the gateway restarts, then it will stay 'disconnected'.

    I've changed how persistence works. It now always creates a persistence file. When you enable persistence, it will re-create all devices when the add-on is initialized. That would be more in line with what you expect from a feature like this anyway.

    I'll do a few more tests and then try to get it approved.


  • Plugin Developer

    Version 0.0.5 has been accepted by Mozilla.

    Improvements:

    • More V_types supported, getting close to full support now.
    • Better, more conservative implementation of 'capabilties' from the Mozilla IoT schema.
    • If enabled, persistence is used to immediately re-create all MySensors devives upon reboot. Before, persistence was only used to avoid handing out duplicate node ID's. That has now become standard.
    • Smarter way of dealing with node removal. It now also removes persistence data.
    • Re-discovery process is faster now (asks one node ID per second to present itself). This is on top of the process that is already built into the PyMySensors library that this add-on is based on.

    Perhaps it's getting time to add this to the controllers overview, and fill in all the support details? Please contact me about how to do that. // done


  • Hardware Contributor

    @alowhum I can't seem to add any more my_sensors nodes with this version...

    With or without persistance, I see nothing appearing in "Add Thing" view, and not much in the log, only this:
    INFO : mysensors: AddonMangerProxy: recv: b'{"messageType":"startPairing","data":{"timeout":60,"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'


  • Hardware Contributor

    I removed the add-on, I restarted, reinstalled the add on.
    It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ? Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view.

    2019-05-30 08:23:31.508 INFO   : mysensors: RECREATING DEVICES FROM PERSISTENCE
    2019-05-30 08:23:31.510 INFO   : mysensors: -Recreating: BlackCircleSensor
    2019-05-30 08:23:31.512 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.518 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
    2019-05-30 08:23:31.521 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.522 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 1
    2019-05-30 08:23:31.525 INFO   : mysensors: new_description = Temperature
    2019-05-30 08:23:31.527 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.528 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.530 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
    2019-05-30 08:23:31.532 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.534 INFO   : mysensors: + DEVICE.ADD_CHILD with child_id: 2
    2019-05-30 08:23:31.536 INFO   : mysensors: new_description = Humidity
    2019-05-30 08:23:31.538 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.539 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    2019-05-30 08:23:31.541 INFO   : mysensors: AddonManagerProxy: handle_device_added: MySensors_33
    2019-05-30 08:23:31.543 INFO   : mysensors: AddonManagerProxy: Failed to send message: Nanomsg library was terminated
    

    Also it seems all files remain when the add-on is removed, I'm not sure if it's a problem with the add-on or with the gateway ? When re-adding the add-on I have a bunch of messages similar to those :

    2019-05-30 08:23:20.911 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/crcmod already exists. Specify --upgrade to force replacement.
    2019-05-30 08:23:20.915 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/serial already exists. Specify --upgrade to force replacement.
    2019-05-30 08:23:20.919 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/voluptuous-0.11.5.egg-info already exists. Specify --upgrade to force replacement.
    2019-05-30 08:23:20.929 INFO   : mysensors: Target directory /home/pi/.mozilla-iot/addons/mysensors-adapter/lib/pymysensors-0.18.0.egg-info already exists. Specify --upgrade to force replacement.
    

  • Plugin Developer

    @nca78 So the first issue is gone, right?

    The short version:

    Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

    The long version:

    If you disable persistence, it does not remove the persistence file. That file is always used to remember all the node ID's that have been handed out.

    If you disable persistence, all that happens is that it no longer re-generates all the things after a restart of the gateway/add-on. If persistence is disabled, nodes will only show up once they present themselves. To speed this up, the add-on goes over the list of stored node ID's and asks all of them, one per second, to re-present themselves. Another way to get them to re-present themselves is to simply restart the node.

    If you remove a thing in the Gateway, the latest version (0.0.5) also removes its data from the persistence file. In other words: if you remove a thing, you will most likely have to restart the node if you want it to show up on the 'add things' page again.

    "It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

    Correct, it should theoretically no longer be in the persistence file.

    "Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

    Then how do you know it's been recreated? From the log?

    Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.


  • Hardware Contributor

    @alowhum said in Mozilla Webthings MySensors add-on / adapter:

    @nca78 So the first issue is gone, right?

    No, not at all. I still can't add any node in MySensors.

    Your log shows this error: Nanomsg library was terminated, which indicates that the Gateway component that the add-on communicates with has crashed. Your best bet is to restart the WebThings Gateway/Raspberry Pi.

    I did, it didn't work

    "It's recreating the sensor from Persistance, but that sensor was removed from the thing detail page before, so it should not have any persistance left ?"

    Correct, it should theoretically no longer be in the persistence file.

    Well from the log I believe it's still here ?

    "Also the recreated thing/sensor is not visible anywhere: not in add things, not in things view."

    Then how do you know it's been recreated? From the log?

    Yes, from the log as it's not visible anywhere else. And neither are new nodes. Very frustrating 😞

    Strange that the add-on files are still there after removing the add-on. I'm not even sure those files should have ever been there. Did you do a git clone at some point? Perhaps the crashed part of the Gateway also resulted in the add-on not really being removed.

    No I never fiddled directly with the files. I think I will do it now, my guess is the problem is related to those messages I see in log when I remove the add-on:

    2019-05-30 09:45:07.158 INFO   : Unloading MySensorsAdapter
    2019-05-30 09:45:07.166 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadAdapter","data":{"adapterId":"mysensors-adapter","pluginId":"mysensors-adapter"}}'
    2019-05-30 09:45:07.179 INFO   : mysensors: Shutting down MySensors adapter
    2019-05-30 09:45:07.181 INFO   : mysensors: MySensors adapter was unable to cleanly close PyMySensors
    2019-05-30 09:45:07.183 INFO   : mysensors: AddonMangerProxy: recv: b'{"messageType":"unloadPlugin","data":{"pluginId":"mysensors-adapter"}}'
    2019-05-30 09:45:10.188 INFO   : Killing mysensors-adapter plugin.
    

  • Hardware Contributor

    So connected with SSH and the directory is gone now.
    I reinstalled the add-on and I didn't have those messages about files already existing, only success messages.

    But I don't get why it still has my configuration for MySensors gateway URL etc. I install the add-on and after install it connects directly to the MySensors gateway.

    I don't know what happened also but it seems I had a problem with the gateway too as I didn't have sensor info in it's log. Now I reprogrammed it and I see log when sensor sends and I have the sensor in the add sensor view.

    No idea of what happened, but it seems problem was with my own sensors/gateway and not the add-on 🙂


  • Plugin Developer

    @Nca78 Glad to hear it.

    I suspect that because part of the gateway had crashed, all the things you thought you did never actually happened. You never really deleted the thing, and hence the persistence data was still there. You never really deleted the add-on, and hence the files were still there.


  • Plugin Developer

    Version 0.0.7 has been created, and should hopefully be accepted soon.

    It fixes the issue that the user's preference for which capability should be shown centrally was reset every time the add-on restarted.

    Internally, the add-on always uses persistence. Whether devices are shown as 'connected' is now an option. You can either immediately show all MySensors nodes as 'connected' upon a restart, or wait until nodes have sent a signal.

    // It has been accepted


  • Plugin Developer

    Version 0.1.0 has been accepted. Persistence is a bit faster and hopefully more robust. Hopefully it fixes an issue @rmalbers was experiencing.


 

246
Online

8.6k
Users

9.4k
Topics

99.0k
Posts