Best posts made by rakeshpai

Disclaimer: This is a proof-of-concept. It doesn't work. Things will change, and will break. In fact, I'm sure many things are broken right now. It would be pretty stupid to use this for anything important. I'm just putting this in the public to gauge interest. You have been warned.

Link to the firmware generator.

If you are like me, you want to put approximately half a million MySensors nodes around the house. You've probably already built the first few nodes, so it feels like you are most of the way there. You love the soldering and the building. However, the coding isn't exactly fun. It requires reading through docs, then trying and failing, and rinse and repeat until you get it right.

This tool doesn't solve that problem yet, but it aims to. It aims to provide a simple browser-based UI for configuring your network, and then generating the firmware for the node automatically, using whatever's the best-practice for writing the firmware. This tool writes your code for you!

The good:

• You don't have to write code! This web page does it for you!
• Security built in by default.
• Uses the awesome NodeManager by @user2684 under the hood.
• The web-page, once loaded, works completely offline. You do not need an Internet connection to use this tool! (Try it!)
• No sign up, no registration. In fact, there's no server, or no central database. Your data doesn't even leave your browser.
• There's no server, so there's no network communication and everything's local, so it's blazing fast!
• That bears repeating: The firmware is generated inside your browser, without using any cloud that will rain down on you. Isn't it awesome what browsers can do these days?

The bad:

• Needs a pretty modern browser. Tested on Chrome and Firefox. Untested on IE.
• Doesn't work on mobile devices (yet). Might work in Chrome/Android, but it's untested. Other browsers would need a server-side component to get them to behave.
• I have Google Analytics on the site, just to gawk at some graphs. All GA data is anonymous.

The ugly:

• Well, it looks ugly.
• The code it generates hasn't been tested. This is just a proof of concept. DO NOT depend on this.
• The code, especially the security bits, require review. In fact, I'm pretty sure I've done it wrong.
• They say, 'if you aren't ashamed of it, you've released too late'. I'm definitely releasing too early.

With that summary out of the way, I'd love it if you could poke around and let me know what you think. You don't need to download or install anything - it's just a web page - so just click the link above. I'm trying to gauge interest in the community for this, to decide if it's worth pursuing.

Would you like me to spend time on making this better?

The icon to download firmware is on the top-right of the page, and you can see it on the pages for either the nodes you create, or on the page for the gateway. You can create nodes by selecting your network in the left hand side navigation, and clicking on the + icon at the top right of the page.

The firmware is downloaded as a zip, which contains a SecurityPersonalizer with your security settings, and the main sketch which runs on your node. Currently, the firmware might not even compile.

Hope this gives an idea of what I have in mind, even if it doesn't actually work.

All manner of suggestions and feedback is welcome.

Long time lurker, first time poster.

I've implemented a node.js module which you can include in your node projects, that does the dirty work of communicating with the serial gateway, and exposes a (hopefully) simple API to interact with your mysensors network.

Find the project on Github or NPM.

I've tried this with a Serial gateway, and it works fine. I haven't been able to test it with a Ethernet / WiFi gateway, but it should work fine with them as well.

Please feel free to use this in your code, if you like, and let me know if you have any suggestions/feedback (or post an issue on Github).

I just discovered this: https://github.com/arduino/arduino-create-agent

It's a project by the folks at Arduino, so it's as official as it gets. It's an app that you run locally, that runs in the background, puts an icon in the system tray, and starts a WebSocket server, with which a browser can connect to the local USB devices. It looks like it has been built for the Arduino Web Editor, and is in more-or-less active development.

I just spent under 15 mins with it, and I was able to talk to it from a CLI tool over websockets, and get it to list the available serial ports successfully. This is very encouraging.

Does anyone have any opinion about / experience with this? They have binaries for the three big desktop OSs. It's written in Go, so it's very clean in terms of deployment - it's just a single binary with no dependencies.

It looks like they enforce security by having a config.ini file with a list of allowed origins (domains) that are allowed to access the local web server. Since our page is running over HTTPS, the browser also imposes a restriction that it won't connect to insecure origins, which these chaps circumvent by creating a certificate locally during install (a process that I don't completely understand yet).

So, if someone is to use this, they'll have to install the app based on their platform, then modify config.ini to give access to this URL. That's relatively easy to do. It gives us access to everything that Arduino can do, but from a browser.

Most importantly, it's an official project by the folks at Arduino, so it can be trusted. Also, we don't have to replicate this work, and simply build on top of their APIs.

Thoughts?

@user2684 Thanks for the update. Much appreciated.

The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.

So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.

Sorry for the long silence. it might look like I've been slacking off (and that's mostly true), but I've also been rolling out some updates:

• Code can now either be downloaded for the Arduino IDE or for the PlatformIO IDE. The PlatformIO code generates additional config files to use their dependency manager automatically. This means, you don't have to download and install libraries manually. Just open in PlatformIO and build the project. It's magical.
• Also, you can now download a backup zip of everything. It contains code for the entire network and all it's nodes, and a network.json file. I haven't implemented a 'restore' functionality yet (coming soon), but the network.json file contains everything needed for a restore. The network.json is the internal state data-structure used by the app.
• Couple of minor performance improvements, especially to help with the first-ever-visit scenario. (All subsequent user actions are blazing fast anyway.)

To clarify, while backups are a great idea, you don't need to save anything. Stuff is still automatically saved in the browser's storage anyway during regular use, and is persisted even if you close the browser. Backups are for the assurance of having a neat little zip containing everything on your hard drive, just in case things go horribly wrong.

Very little has changed visually. To find these new features, launch the app, and click the 'hamburger menu' at the top right of the page.

https://rakeshpai.github.io/mysensors-network-manager/

Pretty big update: Added the ability to select the board that the node/gateway is based on. Supported boards are: Pro Mini (5v and 3.3v), Nano and Uno for the atmega328 boards, and NodeMCU and D1 Mini for the esp8266 boards. Please let me know if your favourite board isn't listed here.

The selection of the board effects the list of available pins for sensors. The sense pin dropdown for sensors now shows the appropriate pins (analog/digital/interrupt pins) based on the board selected for the node and the type of sensor being configured.

The selected board also affects the platformio.ini file generated, which makes PlatformIO integration even more enjoyable to use. No need to specify the board during the build.

This was a pretty big update, including changes to the underlying internal datastructure itself, so it's also an acid test for the data migration and versioning system implemented before. If you don't see any errors when you refresh after the update, it means everything worked transparently, and that's great news! (The migration mechanism hasn't been tested so far, so please let me know if you see errors when you get the update.)

Also implemented a reporting interval setting for analog sensors, to bring it in line with the latest changes in NodeManager.

We're getting closer to being release-ready!

EDIT: Also, this project crossed 100 commits in GitHub already! Wow!

Since it is possible to have multiple mysensors gateways communicating with the same controller, how do you identify each gateway uniquely? Do gateways have some sort of unique identifier?

Usage: I'm planning to put together a controller, and want to support multiple gateways for completeness. Since serial port names and IP addresses might all change over time, how do I associate metadata with a gateway in case of disconnection and reconnection?

I'm thinking that there should be some sort of unique id associated with a gateway, which I can query over the serial API. Does such a thing exist?

Thanks, @hek and @Anticimex.

I could add the ATSHA if there's interest. I haven't looked into it yet, and haven't used it, which is why I'm only doing software signing at the moment. If there's interest for it, I'll definitely add it. I'll need to understand how it can be made such that just flashing the SecurityPersonalizer will do everything automatically, so that the user doesn't have to interfere with it.

Which browser are you using? I've only tested this in Firefox and Chrome on Ubuntu (though the OS shouldn't matter). I haven't tested this on IE. I'm certain it doesn't work on any iOS browser, because of browser limitations. (Limitations can be worked around by using a server.) It might work on Chrome Android, but I don't have access to an Android to test.

The library I'm using for creating the zip is here: https://stuk.github.io/jszip/ Can you please tell me if their example works for you in your browser?

Thanks for the catch about the 9-byte device id. I'm also certain that the two other strings I'm using is wrong:
For the HMAC: "A 64 digit hex number used for signing messages"
For the AES: "A 32 digit hex number used for encryption"
If you could help me phrase these correctly, I'll push an update with all three corrections. Thanks.

@bilbolodz I just updated the UI. Please do a hard-refresh (ctrl+r in chrome), delete your existing network, and try again.

In general, you shouldn't trust the output code yet. I haven't even run it on an arduino yet. The idea is just to give you a feel for the goals of the app, and find out if people find this useful/interesting.

Bugs reports are definitely welcome though. A comment on this thread works just fine.

Sorry for the silence. My ISP is having trouble, and I've been mostly offline. I'm sending this message over crappy 3G on a phone.

Update

1. New 'Code' view that shows what the generated code will look like before having to download it. I'm not sure I like this - I would prefer having no mention of code on the UI at all - but it's worth experimenting with. Certainly makes things easy at dev time.
2. A migrations process, so that I don't have to ask you to delete your existing network, as I have been doing. Migrations will now move your local copy of the data to the latest version automatically, without any manual intervention, if there have been any changes in the data format.

That's all I have for now. I'll have more once my connectivity issues are fixed.

I've hit a little bit of a wall, and need help to proceed.

I created a NRF24L01 network in the UI, and tried to flash the default serial gateway code to an Arduino Pro Mini 3.3v/8Mhz. There are no sensors configured on the gateway. I tried to flash the SecurityPersonalizer on the device, and it worked perfectly, and gave a happy sounding serial output. (Yay!) However, when I flash the main gateway sketch, I get a 'not enough memory' error.

Sketch uses 25,792 bytes (79%) of program storage space. Maximum is 32,256 bytes.
Global variables use 2,204 bytes (107%) of dynamic memory, leaving -156 bytes for local variables. Maximum is 2,048 bytes.
processing.app.debug.RunnerException: Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.
	at cc.arduino.Compiler.size(Compiler.java:339)
	at cc.arduino.Compiler.build(Compiler.java:159)
	at processing.app.SketchController.build(SketchController.java:641)
	at processing.app.Editor$BuildHandler.run(Editor.java:1782)
	at java.lang.Thread.run(Thread.java:745)
Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.

For reference, the generated gateway sketch is as follows:

// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES

// load user settings
#include "config.h"
// load MySensors library
#include <MySensors.h>
// load NodeManager library
#include "NodeManager.h"

// create a NodeManager instance
NodeManager nm;

// before
void before() {
  // setup the serial port baud rate
  Serial.begin(9600);

  

  nm.before();
}

// presentation
void presentation() {
  // call NodeManager presentation routine
  nm.presentation();
}

// setup
void setup() {
  // call NodeManager setup routine
  nm.setup();
}

// loop
void loop() {
  // call NodeManager loop routine
  nm.loop();
}

// receive
void receive(const MyMessage &message) {
  // call NodeManager receive routine
  nm.receive(message);
}

// receiveTime
void receiveTime(unsigned long ts) {
  // call NodeManager receiveTime routine
  nm.receiveTime(ts);
}

and the associated config.h is:

#ifndef config_h
#define config_h

/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
//#define MY_REPEATER_FEATURE


/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG


// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24


// Serial gateway settings
#define MY_GATEWAY_SERIAL


/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
//#define DEBUG 1

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 1
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 1

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0

#endif

I tried commenting out the first three lines in the gateway sketch (the #defines to do with security personalisation, and the sketch fits, but only barely.

Sketch uses 19,922 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,710 bytes (83%) of dynamic memory, leaving 338 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

I'm by no means a C/C++ guy (which I why I'm writing this tool ) so I'll need help to fix this. I don't even know where to start looking, since there's the combination of NodeManager and the software signing in the sketch.

Any chance you could look at this, @user2684 and @Anticimex? You could download a copy of the code from the UI if you like. My configuration is an NRF network, and the gateway is the default serial gateway on the UI. I've been using the latest version of the MySensors lib from the development branch on GitHub, and am doing the compilation in the Arduino IDE.

Thanks!

Since I haven't actually done this yet, I thought I'd outline the 'stack' used for this app, in case folks want to dive into the code.

• Based on React, using create-react-app.
• Offline first. Uses service-workers and localStorage for offline support.
• There's no server-side language. It's just a static site hosted on GitHub pages. Node.js is used for local development tooling.
• Uses redux for state management, and react-router for client-side routing.
• CSS-in-JS with glamor.

So, it's a very modern stack. Cutting-edge web stuff.

It's very early in the project, so I don't have any long-term plans outlined. In fact, I'll be honest, I haven't thought that far. But it's a community project, so you are welcome to steer the project as you see right.

Some areas to help:

• I suck at UI beautification work. Any help will be awesome.
• Need to investigate Electron for building native apps. Some things to look for: (a) Reusing the website directly from GitHub pages, so that we have a single deploy for the UI and the site still works fine if people don't want to download an app. Ideally, the electron app will simply be a container for the site. (b) Automatic updates to the container app. (c) Serial communication (node-serialport? avrgirl?) (d) More, as we go deeper.
• Ideas, suggestions? Like I said, I haven't thought too far.

Actually, the absolute best way to help right now is to simply poke around the UI and point out issues, no matter how minor they are. It's the little things and attention to detail in an app that make it nice to use, so it'll help to have an obsessive pair of eyes going over the UI. You can point out bugs, usability issues, things that might be confusing, errors in the generated code, or any other changes you think will improve the app.

You can comment here with suggestions, or use GitHub if you like.

tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.

I've hit a dead-end with Arduino Create Agent.

Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:

However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:

(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent

However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.

If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.

Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.

Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.

Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.

Onwards!

@pjr Noted. Thanks. Are there a lot of people using RS485? (I have no idea, and I'm constantly surprised by how people use MySensors.) My only concern is that I don't want to add too many features that people won't use, since it would make the UI confusing. But if it's a popular feature, it's definitely on the to-do.

@alowhum Sorry, I have abandoned this project. It didn't garner the interest I hoped it would, and I’ve moved on to other microcontrollers/radios/networks since then. This is hosted on GitHub’s free static hosting, so it should continue to work as long as they are in business, and it should work on devices as it was 5-6 years ago. But I won’t be supporting it and won’t continue to build on it. Glad to hear that you like it!