Browser-based firmware generator
-
Awesome @rakeshpai!. Not sure if it is a glitch in my chrome, but I wasn't able to download the code any longer.
@hek Thanks. The download works for me just fine. Can you provide any more details? Do you see any errors in the browser's console? Also, hope you are using a desktop OS.
I haven't made much/any changes to the functionality, so downloads should be working as it was.
-
I just discovered this: https://github.com/arduino/arduino-create-agent
It's a project by the folks at Arduino, so it's as official as it gets. It's an app that you run locally, that runs in the background, puts an icon in the system tray, and starts a WebSocket server, with which a browser can connect to the local USB devices. It looks like it has been built for the Arduino Web Editor, and is in more-or-less active development.
I just spent under 15 mins with it, and I was able to talk to it from a CLI tool over websockets, and get it to list the available serial ports successfully. This is very encouraging.
Does anyone have any opinion about / experience with this? They have binaries for the three big desktop OSs. It's written in Go, so it's very clean in terms of deployment - it's just a single binary with no dependencies.
It looks like they enforce security by having a config.ini file with a list of allowed origins (domains) that are allowed to access the local web server. Since our page is running over HTTPS, the browser also imposes a restriction that it won't connect to insecure origins, which these chaps circumvent by creating a certificate locally during install (a process that I don't completely understand yet).
So, if someone is to use this, they'll have to install the app based on their platform, then modify config.ini to give access to this URL. That's relatively easy to do. It gives us access to everything that Arduino can do, but from a browser.
Most importantly, it's an official project by the folks at Arduino, so it can be trusted. Also, we don't have to replicate this work, and simply build on top of their APIs.
Thoughts?
-
I just discovered this: https://github.com/arduino/arduino-create-agent
It's a project by the folks at Arduino, so it's as official as it gets. It's an app that you run locally, that runs in the background, puts an icon in the system tray, and starts a WebSocket server, with which a browser can connect to the local USB devices. It looks like it has been built for the Arduino Web Editor, and is in more-or-less active development.
I just spent under 15 mins with it, and I was able to talk to it from a CLI tool over websockets, and get it to list the available serial ports successfully. This is very encouraging.
Does anyone have any opinion about / experience with this? They have binaries for the three big desktop OSs. It's written in Go, so it's very clean in terms of deployment - it's just a single binary with no dependencies.
It looks like they enforce security by having a config.ini file with a list of allowed origins (domains) that are allowed to access the local web server. Since our page is running over HTTPS, the browser also imposes a restriction that it won't connect to insecure origins, which these chaps circumvent by creating a certificate locally during install (a process that I don't completely understand yet).
So, if someone is to use this, they'll have to install the app based on their platform, then modify config.ini to give access to this URL. That's relatively easy to do. It gives us access to everything that Arduino can do, but from a browser.
Most importantly, it's an official project by the folks at Arduino, so it can be trusted. Also, we don't have to replicate this work, and simply build on top of their APIs.
Thoughts?
Not as rosy as I initially thought. I thought I might be able to compile the sketch locally using the app above. However, it looks like compilation is not a responsibility that this app has taken on. It only deals with listing serial devices, connecting to them, and flashing them, but not compiling sketches.
Arduino Web Editor appears to use arduino-builder, which I'm assuming they run in the cloud, since the setup isn't very easy at all. This means that we'll have to do cloud compilation as well. (@Anticimex, you might find arduino-builder interesting. They've specifically documented running arduino-builder in a CI system.)
It's still awesome that we have a solution to talk to the device from the browser. It's just that it'd have been even better if it did compilation as well. Hosting a cloud compilation service is not a huge problem - I run a reasonably high-traffic online service as my main gig, so I have some experience - but it'd have been better if we didn't have to.
I wonder why Arduino chose to compile in the cloud, considering that they ask their users to download a local app anyway to talk to the hardware. They wouldn't have to run a large part of their infrastructure, thus saving lots of associated costs and effort. And, they'd have a local toolchain, removing the need for an internet connection for compilation. They'd have to deal with setting up the toolchain locally, but they've been doing that for years, and have a lot of experience with it. Also, they would've enabled an ecosystem of apps like ours, so that we don't have to host compilers in the cloud. Maybe I'm not seeing some detail, but I think I'd have made the choice of having the compiler run locally too.
EDIT: I've raised an issue in their repo.
-
Not as rosy as I initially thought. I thought I might be able to compile the sketch locally using the app above. However, it looks like compilation is not a responsibility that this app has taken on. It only deals with listing serial devices, connecting to them, and flashing them, but not compiling sketches.
Arduino Web Editor appears to use arduino-builder, which I'm assuming they run in the cloud, since the setup isn't very easy at all. This means that we'll have to do cloud compilation as well. (@Anticimex, you might find arduino-builder interesting. They've specifically documented running arduino-builder in a CI system.)
It's still awesome that we have a solution to talk to the device from the browser. It's just that it'd have been even better if it did compilation as well. Hosting a cloud compilation service is not a huge problem - I run a reasonably high-traffic online service as my main gig, so I have some experience - but it'd have been better if we didn't have to.
I wonder why Arduino chose to compile in the cloud, considering that they ask their users to download a local app anyway to talk to the hardware. They wouldn't have to run a large part of their infrastructure, thus saving lots of associated costs and effort. And, they'd have a local toolchain, removing the need for an internet connection for compilation. They'd have to deal with setting up the toolchain locally, but they've been doing that for years, and have a lot of experience with it. Also, they would've enabled an ecosystem of apps like ours, so that we don't have to host compilers in the cloud. Maybe I'm not seeing some detail, but I think I'd have made the choice of having the compiler run locally too.
EDIT: I've raised an issue in their repo.
@rakeshpai we use the arduino-builder. It is just a shell command bundled with the IDE. You will still need to host a build environment, install all necessary libraries and BSP:s.
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
-
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
@rakeshpai omfg that is a massive security cluster f*ck by the Italians. How do they ensure that their key is properly protected? From what you describe, they basically hand over shell access to attackers on a silver platter, signed or not. I do hope that this require you to install some special software on you machine to enable, and it is not something that comes bundled with the arduino IDE, right? Because if it is, it goes out of my machine...
Do you feel secure today? No? Start requiring some signatures and feel better tomorrow ;)
-
@rakeshpai omfg that is a massive security cluster f*ck by the Italians. How do they ensure that their key is properly protected? From what you describe, they basically hand over shell access to attackers on a silver platter, signed or not. I do hope that this require you to install some special software on you machine to enable, and it is not something that comes bundled with the arduino IDE, right? Because if it is, it goes out of my machine...
@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.
You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.
Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.
-
@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.
You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.
Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.
@rakeshpai the way they do it seem very in line with many of the design decisions taken in the IDE and the whole build process. You can just see how we include .cpp files in the library headers due to how their build system (or lack thereof) works...
-
@pjr I guess this is partially dependent on NodeManager as well, @rakeshpai I've added https://github.com/mysensors/NodeManager/issues/169 for this. Thanks
-
@pjr Noted. Thanks. Are there a lot of people using RS485? (I have no idea, and I'm constantly surprised by how people use MySensors.) My only concern is that I don't want to add too many features that people won't use, since it would make the UI confusing. But if it's a popular feature, it's definitely on the to-do.
-
@rakeshpai I just took a quick look at this, and this will be an awesome tool. I didn't read through all 96 comments, so forgive me if my questions have already been asked and answered.
First, are all of the libraries for the included devices in the list going to be included in the zip file. These would be needed at compile time when uploading the firmware to the nodes.
Next, would there be a way to add sensors and their required libraries? If/when adding new devices/sensors, is there going to be something that defines the code structure for the device so when the code is being built it knows how to deal with it?
I am assuming that all of the sensors/devices in a network will be saved in a list for future reference and updating. Is there going to be any fields to add notes about a device and possibly a field where a location can be identified for a device such as "West wall of the great room" for a light switch or something like that.
Last, and this is more of a feature suggestion that would kind of tie into the last question, it would be nice if the user could upload floor plan images and be able to place the devices on the floor plan with an icon or something to be able to better identify where nodes are. A feature like this may be a bit of a stretch, but who would I be if I didn't at least suggest it.
Thanks for the contribution and keep up the good work.
Vera Plus running UI7 with MySensors, Sonoffs and 1-Wire devices
Visit my website for more Bits, Bytes and Ramblings from me: http://dan.bemowski.info/ -
@rakeshpai I just took a quick look at this, and this will be an awesome tool. I didn't read through all 96 comments, so forgive me if my questions have already been asked and answered.
First, are all of the libraries for the included devices in the list going to be included in the zip file. These would be needed at compile time when uploading the firmware to the nodes.
Next, would there be a way to add sensors and their required libraries? If/when adding new devices/sensors, is there going to be something that defines the code structure for the device so when the code is being built it knows how to deal with it?
I am assuming that all of the sensors/devices in a network will be saved in a list for future reference and updating. Is there going to be any fields to add notes about a device and possibly a field where a location can be identified for a device such as "West wall of the great room" for a light switch or something like that.
Last, and this is more of a feature suggestion that would kind of tie into the last question, it would be nice if the user could upload floor plan images and be able to place the devices on the floor plan with an icon or something to be able to better identify where nodes are. A feature like this may be a bit of a stretch, but who would I be if I didn't at least suggest it.
Thanks for the contribution and keep up the good work.
@dbemowsk Thanks! It's very encouraging to know that this is useful.
I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.
I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.
This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.
Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)
Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!
(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)
-
@dbemowsk Thanks! It's very encouraging to know that this is useful.
I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.
I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.
This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.
Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)
Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!
(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)
@rakeshpai You mention about Dropbox, either that or Google Drive would be interesting options.
I know the floor plan thing is a stretch as it is a bit of coding to make that kind of thing work. I just figured I'd throw the hook out there and see who might bite on the worm.
I like the schematic thing too, but that, much like the floor plan thing might be hard to do.
-
@rakeshpai FYI I'm making some good progress with the new version of NodeManager (https://github.com/mysensors/NodeManager/milestones) thanks also to the many users contributing :-) However, the list of things to do is still pretty long and last thing I want to do is to keep your amazing project on hold depending on mine. So whenever you are almost ready, just let me know so that I can push some of the less critical requests I have on the list to another release and hurry up to finalize the current one I'm working on. Whenever you will tell me "I'm ready", I expect at least a couple of weeks for me to finalize the current code and test the release before will be made available. Thanks
-
@rakeshpai FYI I'm making some good progress with the new version of NodeManager (https://github.com/mysensors/NodeManager/milestones) thanks also to the many users contributing :-) However, the list of things to do is still pretty long and last thing I want to do is to keep your amazing project on hold depending on mine. So whenever you are almost ready, just let me know so that I can push some of the less critical requests I have on the list to another release and hurry up to finalize the current one I'm working on. Whenever you will tell me "I'm ready", I expect at least a couple of weeks for me to finalize the current code and test the release before will be made available. Thanks
@user2684 Thanks for the update. Much appreciated.
The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.
So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.
