Browser-based firmware generator
-
Not as rosy as I initially thought. I thought I might be able to compile the sketch locally using the app above. However, it looks like compilation is not a responsibility that this app has taken on. It only deals with listing serial devices, connecting to them, and flashing them, but not compiling sketches.
Arduino Web Editor appears to use arduino-builder, which I'm assuming they run in the cloud, since the setup isn't very easy at all. This means that we'll have to do cloud compilation as well. (@Anticimex, you might find arduino-builder interesting. They've specifically documented running arduino-builder in a CI system.)
It's still awesome that we have a solution to talk to the device from the browser. It's just that it'd have been even better if it did compilation as well. Hosting a cloud compilation service is not a huge problem - I run a reasonably high-traffic online service as my main gig, so I have some experience - but it'd have been better if we didn't have to.
I wonder why Arduino chose to compile in the cloud, considering that they ask their users to download a local app anyway to talk to the hardware. They wouldn't have to run a large part of their infrastructure, thus saving lots of associated costs and effort. And, they'd have a local toolchain, removing the need for an internet connection for compilation. They'd have to deal with setting up the toolchain locally, but they've been doing that for years, and have a lot of experience with it. Also, they would've enabled an ecosystem of apps like ours, so that we don't have to host compilers in the cloud. Maybe I'm not seeing some detail, but I think I'd have made the choice of having the compiler run locally too.
EDIT: I've raised an issue in their repo.
@rakeshpai we use the arduino-builder. It is just a shell command bundled with the IDE. You will still need to host a build environment, install all necessary libraries and BSP:s.
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
@rakeshpai omfg that is a massive security cluster f*ck by the Italians. How do they ensure that their key is properly protected? From what you describe, they basically hand over shell access to attackers on a silver platter, signed or not. I do hope that this require you to install some special software on you machine to enable, and it is not something that comes bundled with the arduino IDE, right? Because if it is, it goes out of my machine...
Do you feel secure today? No? Start requiring some signatures and feel better tomorrow ;)
-
@rakeshpai omfg that is a massive security cluster f*ck by the Italians. How do they ensure that their key is properly protected? From what you describe, they basically hand over shell access to attackers on a silver platter, signed or not. I do hope that this require you to install some special software on you machine to enable, and it is not something that comes bundled with the arduino IDE, right? Because if it is, it goes out of my machine...
@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.
You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.
Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.
-
@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.
You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.
Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.
@rakeshpai the way they do it seem very in line with many of the design decisions taken in the IDE and the whole build process. You can just see how we include .cpp files in the library headers due to how their build system (or lack thereof) works...
-
@pjr I guess this is partially dependent on NodeManager as well, @rakeshpai I've added https://github.com/mysensors/NodeManager/issues/169 for this. Thanks
-
@pjr Noted. Thanks. Are there a lot of people using RS485? (I have no idea, and I'm constantly surprised by how people use MySensors.) My only concern is that I don't want to add too many features that people won't use, since it would make the UI confusing. But if it's a popular feature, it's definitely on the to-do.
-
@rakeshpai I just took a quick look at this, and this will be an awesome tool. I didn't read through all 96 comments, so forgive me if my questions have already been asked and answered.
First, are all of the libraries for the included devices in the list going to be included in the zip file. These would be needed at compile time when uploading the firmware to the nodes.
Next, would there be a way to add sensors and their required libraries? If/when adding new devices/sensors, is there going to be something that defines the code structure for the device so when the code is being built it knows how to deal with it?
I am assuming that all of the sensors/devices in a network will be saved in a list for future reference and updating. Is there going to be any fields to add notes about a device and possibly a field where a location can be identified for a device such as "West wall of the great room" for a light switch or something like that.
Last, and this is more of a feature suggestion that would kind of tie into the last question, it would be nice if the user could upload floor plan images and be able to place the devices on the floor plan with an icon or something to be able to better identify where nodes are. A feature like this may be a bit of a stretch, but who would I be if I didn't at least suggest it.
Thanks for the contribution and keep up the good work.
Vera Plus running UI7 with MySensors, Sonoffs and 1-Wire devices
Visit my website for more Bits, Bytes and Ramblings from me: http://dan.bemowski.info/ -
@rakeshpai I just took a quick look at this, and this will be an awesome tool. I didn't read through all 96 comments, so forgive me if my questions have already been asked and answered.
First, are all of the libraries for the included devices in the list going to be included in the zip file. These would be needed at compile time when uploading the firmware to the nodes.
Next, would there be a way to add sensors and their required libraries? If/when adding new devices/sensors, is there going to be something that defines the code structure for the device so when the code is being built it knows how to deal with it?
I am assuming that all of the sensors/devices in a network will be saved in a list for future reference and updating. Is there going to be any fields to add notes about a device and possibly a field where a location can be identified for a device such as "West wall of the great room" for a light switch or something like that.
Last, and this is more of a feature suggestion that would kind of tie into the last question, it would be nice if the user could upload floor plan images and be able to place the devices on the floor plan with an icon or something to be able to better identify where nodes are. A feature like this may be a bit of a stretch, but who would I be if I didn't at least suggest it.
Thanks for the contribution and keep up the good work.
@dbemowsk Thanks! It's very encouraging to know that this is useful.
I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.
I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.
This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.
Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)
Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!
(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)
-
@dbemowsk Thanks! It's very encouraging to know that this is useful.
I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.
I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.
This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.
Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)
Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!
(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)
@rakeshpai You mention about Dropbox, either that or Google Drive would be interesting options.
I know the floor plan thing is a stretch as it is a bit of coding to make that kind of thing work. I just figured I'd throw the hook out there and see who might bite on the worm.
I like the schematic thing too, but that, much like the floor plan thing might be hard to do.
-
@rakeshpai FYI I'm making some good progress with the new version of NodeManager (https://github.com/mysensors/NodeManager/milestones) thanks also to the many users contributing :-) However, the list of things to do is still pretty long and last thing I want to do is to keep your amazing project on hold depending on mine. So whenever you are almost ready, just let me know so that I can push some of the less critical requests I have on the list to another release and hurry up to finalize the current one I'm working on. Whenever you will tell me "I'm ready", I expect at least a couple of weeks for me to finalize the current code and test the release before will be made available. Thanks
-
@rakeshpai FYI I'm making some good progress with the new version of NodeManager (https://github.com/mysensors/NodeManager/milestones) thanks also to the many users contributing :-) However, the list of things to do is still pretty long and last thing I want to do is to keep your amazing project on hold depending on mine. So whenever you are almost ready, just let me know so that I can push some of the less critical requests I have on the list to another release and hurry up to finalize the current one I'm working on. Whenever you will tell me "I'm ready", I expect at least a couple of weeks for me to finalize the current code and test the release before will be made available. Thanks
@user2684 Thanks for the update. Much appreciated.
The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.
So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.
-
@user2684 Thanks for the update. Much appreciated.
The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.
So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.
@rakeshpai fully agree with you. I'll start working in order to push those changes which are not a priority or too complex to implement to a future release in order to finalize v1.6 asap and give you the time to start working with something stable. Mean while I'll let you know once the development code will be good enough to start working on it. I can probably target a final v1.6 by end of August and a stable version in a couple of weeks if it fits your plans. I've also created a specific thread we can use to discuss about the integration (https://forum.mysensors.org/topic/7212/integrating-nodemanager-with-sketch-generator) so to keep this as clean as possible from implementation details. Thanks!
-
Sorry for the long silence. it might look like I've been slacking off (and that's mostly true), but I've also been rolling out some updates:
- Code can now either be downloaded for the Arduino IDE or for the PlatformIO IDE. The PlatformIO code generates additional config files to use their dependency manager automatically. This means, you don't have to download and install libraries manually. Just open in PlatformIO and build the project. It's magical.
- Also, you can now download a backup zip of everything. It contains code for the entire network and all it's nodes, and a
network.jsonfile. I haven't implemented a 'restore' functionality yet (coming soon), but thenetwork.jsonfile contains everything needed for a restore. Thenetwork.jsonis the internal state data-structure used by the app. - Couple of minor performance improvements, especially to help with the first-ever-visit scenario. (All subsequent user actions are blazing fast anyway.)
To clarify, while backups are a great idea, you don't need to save anything. Stuff is still automatically saved in the browser's storage anyway during regular use, and is persisted even if you close the browser. Backups are for the assurance of having a neat little zip containing everything on your hard drive, just in case things go horribly wrong.
Very little has changed visually. To find these new features, launch the app, and click the 'hamburger menu' at the top right of the page.
-
Pretty big update: Added the ability to select the board that the node/gateway is based on. Supported boards are: Pro Mini (5v and 3.3v), Nano and Uno for the atmega328 boards, and NodeMCU and D1 Mini for the esp8266 boards. Please let me know if your favourite board isn't listed here.
The selection of the board effects the list of available pins for sensors. The sense pin dropdown for sensors now shows the appropriate pins (analog/digital/interrupt pins) based on the board selected for the node and the type of sensor being configured.
The selected board also affects the platformio.ini file generated, which makes PlatformIO integration even more enjoyable to use. No need to specify the board during the build.
This was a pretty big update, including changes to the underlying internal datastructure itself, so it's also an acid test for the data migration and versioning system implemented before. If you don't see any errors when you refresh after the update, it means everything worked transparently, and that's great news! (The migration mechanism hasn't been tested so far, so please let me know if you see errors when you get the update.)
Also implemented a reporting interval setting for analog sensors, to bring it in line with the latest changes in NodeManager.
We're getting closer to being release-ready!
EDIT: Also, this project crossed 100 commits in GitHub already! Wow!
-
@rakeshpai , is this project still being developed? The Github page seems to indicate the last development was a year ago. Is there more development in secret? I really like it.
My dream solution for beginners usability would be a web interface where you can manage the network and the Arduino code. Users then connect an Arduino Nano or Micro:bit to the Raspberry Pi, and press the 'upload code' button for a specific device.
That way they don't need to install the Arduino IDE on their own laptops, but it can be incorporated into a smart home disk image instead.
*dreams on
-
@rakeshpai , is this project still being developed? The Github page seems to indicate the last development was a year ago. Is there more development in secret? I really like it.
My dream solution for beginners usability would be a web interface where you can manage the network and the Arduino code. Users then connect an Arduino Nano or Micro:bit to the Raspberry Pi, and press the 'upload code' button for a specific device.
That way they don't need to install the Arduino IDE on their own laptops, but it can be incorporated into a smart home disk image instead.
*dreams on
@alowhum
If you take a look at the gitpage:
Quote: "mysensors-network-manager
A web-based tool to generate firmware for MySensors nodes.
It buggy, incomplete, it doesn't even work. Don't use this for anything important."
