Browser-based firmware generator
-
@user2684 Woah! Thanks for taking the time to write and share your notes. This is invaluable! Thanks again for this list.
Some quick thoughts:
- I'd rather stick with the 'secure by default, can't be turned off' approach. I understand the usability problems, and want to fix them, so I'm on board with the usability issue. I have a solution in mind, but it's a little ways off. Until then, I'll probably just solve the usability issue with instructions to flash correctly. (It's already at the point where you just have to flash the SecurityPersonalizer.ino once, and then the main sketch. That's it.)
- Running as a client should be possible (reference), but I haven't tried it myself, and the generated code is likely wrong anyway.
- Why set debounce to 0 by default? The debounce option only shows up for door and switch sensors. I'm assuming that both are mechanical switches (even if magnetically actuated as in the case of door switches), so both cases would need debouncing anyway, right?
- I didn't understand your last point about setMode. If you are referring to setMode on the sensor object for switch-type sensors, that doesn't have anything to do with whether the node is battery powered. Clearly, I've misunderstood something.
The rest of what you pointed out either reflect my misunderstanding, or is an incomplete implementation, or are straight-up bugs. Thanks again for your list. It's invaluable. I'll have fixes ready soon.
@rakeshpai no problem, it is just a shame it took so long for me to review it in depth :)
I'd rather stick with the 'secure by default, can't be turned off' approach. I
Having some sort of README file in the zip would probably help in this direction, at least to explain the process (flash the security personalizer first, then the sketch).
Why set debounce to 0 by default? The debounce option only shows up for
I have a few different types of motion sensors around and debounce was never necessary, at least for me. Even worse, I've noticed from the posts here in the forum about NM that can confuse the users leading to weird behaviors. Of course if set very low like in your sketch, it should be fine.
I didn't understand your last point about setMode. If you are referring to
Not your fault at all, it is confusing for everybody I think, this is why I need to do something and change it :) By default all the sensors report their measures at the end of a sleeping/wait cycle regardless if the node is battery powered or not. So if you have e.g. a temperature sensor attached to a node which is NOT battery powered, you will not get anything out of it unless you configure a WAIT or SLEEP cycle. This is why it is important to have this split from the battery powered checkbox. Let me know if not clear I'll share a few examples :)
Thanks -
@rakeshpai no problem, it is just a shame it took so long for me to review it in depth :)
I'd rather stick with the 'secure by default, can't be turned off' approach. I
Having some sort of README file in the zip would probably help in this direction, at least to explain the process (flash the security personalizer first, then the sketch).
Why set debounce to 0 by default? The debounce option only shows up for
I have a few different types of motion sensors around and debounce was never necessary, at least for me. Even worse, I've noticed from the posts here in the forum about NM that can confuse the users leading to weird behaviors. Of course if set very low like in your sketch, it should be fine.
I didn't understand your last point about setMode. If you are referring to
Not your fault at all, it is confusing for everybody I think, this is why I need to do something and change it :) By default all the sensors report their measures at the end of a sleeping/wait cycle regardless if the node is battery powered or not. So if you have e.g. a temperature sensor attached to a node which is NOT battery powered, you will not get anything out of it unless you configure a WAIT or SLEEP cycle. This is why it is important to have this split from the battery powered checkbox. Let me know if not clear I'll share a few examples :)
Thanks -
@user2684 Ah, that makes sense. I was looking for something like a 'reporting interval' in the NodeManager API. So if I understand correctly, I should setMode to WAIT and call setSleep to specify the reporting interval?
@rakeshpai or better directly setSleepMode(WAIT,10,MINUTES). All the sensors report at the end of the sleep/wait cycle. I'm adding new options in the new version for making this more flexible. Also, WAIT would not make sensors waiting for interrupts working correctly since the interrupt is handle by a call to the MySensors sleep() only. I'm fixing this as well. Long story short, I'd recommend to make have SLEEP by default and a fallback to WAIT if the use uncheck the box
-
I've got an update with a bunch of fixes. Thanks, @user2684!
- BATTERY_SENSOR now mirrors BATTERY_MANAGER, and is only applicable for battery powered devices. Doesn't show up on the gateway.
- PERSIST and SERVICE_MESSAGES set to 0.
- Fixes for code generation related to DHCP.
- UI fix - switching to Editor view for gateway doesn't mess up the UI.
switchvariable name changed toinputSwitchto avoid conflict with keywords.- Tests.
@user2684 It looks like I'll wait for your next release before I close some of the other issues:
- setPowerPin() - I'm waiting for a release with this fix.
- If you are fixing how setSleepMode behaves, I'd rather wait for that to come through in a release as well.
No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush. :)
EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!
-
I've got an update with a bunch of fixes. Thanks, @user2684!
- BATTERY_SENSOR now mirrors BATTERY_MANAGER, and is only applicable for battery powered devices. Doesn't show up on the gateway.
- PERSIST and SERVICE_MESSAGES set to 0.
- Fixes for code generation related to DHCP.
- UI fix - switching to Editor view for gateway doesn't mess up the UI.
switchvariable name changed toinputSwitchto avoid conflict with keywords.- Tests.
@user2684 It looks like I'll wait for your next release before I close some of the other issues:
- setPowerPin() - I'm waiting for a release with this fix.
- If you are fixing how setSleepMode behaves, I'd rather wait for that to come through in a release as well.
No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush. :)
EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!
@rakeshpai I really hope my busy life would not impact your project preventing other users to leverage your fantastic tool :-) But honestly could make sense to wait for v1.6 since a few things are changing. But please let me know if and when I'll delaying too much and we'll find a backup plan.
Thanks for your feedback regarding 1.5.1, I'll make it publicly available during the weekend.
-
Need to bounce this off someone. Please share your thoughts.
I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!
The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.
Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.
There's two other alternatives I can think of:
- Create a CLI tool that communicates with a browser. Users have to install and run the CLI app in a terminal, and while its running, they can flash their devices from the browser. This is the most minimal approach that solves the problem, but requires the user to juggle a terminal, which isn't ideal.
- [Preferred] Create a native app, throw away the site, do everything inside the app. The native app can reuse most of the site's code, so the work put in so far is not wasted effort. The app also gets much more privileges, since it is running natively.
The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.
So, in summary, the options are:
- Don't flash the device from the browser, and just offer the generated code for download, as is happening right now. Limits utility and future features, and frankly, isn't exciting.
- Ask the user to install a native app. The app gets lots of capabilities this way, but has the downside of requiring a download and install, which is a massive cause of friction.
If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)
I'm confused about how to proceed. What would you do? What do you think is the best way ahead?
-
Need to bounce this off someone. Please share your thoughts.
I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!
The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.
Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.
There's two other alternatives I can think of:
- Create a CLI tool that communicates with a browser. Users have to install and run the CLI app in a terminal, and while its running, they can flash their devices from the browser. This is the most minimal approach that solves the problem, but requires the user to juggle a terminal, which isn't ideal.
- [Preferred] Create a native app, throw away the site, do everything inside the app. The native app can reuse most of the site's code, so the work put in so far is not wasted effort. The app also gets much more privileges, since it is running natively.
The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.
So, in summary, the options are:
- Don't flash the device from the browser, and just offer the generated code for download, as is happening right now. Limits utility and future features, and frankly, isn't exciting.
- Ask the user to install a native app. The app gets lots of capabilities this way, but has the downside of requiring a download and install, which is a massive cause of friction.
If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)
I'm confused about how to proceed. What would you do? What do you think is the best way ahead?
@rakeshpai if you go for a cloud based programming method, you also need to mind security. Users would like insurance that their nodes that happen to be hooked up to a pc do get exactly the firmware they ask for and only when they ask for it. Also that the firmware goes only in one direction.
Do you feel secure today? No? Start requiring some signatures and feel better tomorrow ;)
-
@rakeshpai if you go for a cloud based programming method, you also need to mind security. Users would like insurance that their nodes that happen to be hooked up to a pc do get exactly the firmware they ask for and only when they ask for it. Also that the firmware goes only in one direction.
@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.
The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.
As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.
So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.
-
I am one of those few users that opened chrome://flags, and set:
Enable WebUSB support: disabled.
I already have installed a tool to program my Arduinos. I guess, it's the first thing most people install after aquiring an Arduino.
I also do not understand why i would ever use a cloud.
Perhaps i am "too oldskool".. using a browser to find what i search for, and (if it is code,) then download it.
When i want code, i do not care how i get it. Important is, that i can download it..
The bare product is what counts.. -
@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.
The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.
As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.
So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.
@rakeshpai just out of curiosity, how do you plan to handle code compilation? If we look beyond the problem of interfacing with a serial device, you also need to compile the sketch, and link it to a hex executable hex file. I'm on saying it can't be done (it has been done before) but I would like to understand how it is practically done. I am curious since I maintain the Jenkins instance for MySensors, and I am open for alternatives to compile things. It uses installed instances of the Arduino IDE on the server side.
-
@Anticimex My current favourite is the PlatformIO Core CLI. While it was built to work in an IDE, it's actually completely independent, and only requires Python to run. It has support for lots of boards too, and has a built-in package manager making dependency management very easy. Since it's a CLI, it can easily be automated with bash scripts, can run headless, and doesn't require a windowing environment. Can't do that with GUIs like Arduino. There are alternatives to PlatformIO, but none that I found were feature-complete.
In fact, while I haven't exposed it, I've already implemented a way to download the code in a folder-structure that platformio prefers. I'll be rolling that out soon.
@core_c I completely understand the security issues of cloud compilation. But I also see the convenience it brings. If I do implement compilation in the cloud, I'm not going to take away the ability to download the code. You won't be forced to use the cloud. That said, I'm not going to implement cloud compilation at this moment, since it's pointless to do so if I can't also flash the device, and I currently can't flash the device from the browser.
-
Hi, I can give a quick look from my (noob) perspective.
I'm no programmer, just tinkering around, new to Mysensors, but already managed to get something working and starting to learn how this stuff works so I can expand my installation.
Im wery interested in this firmware generator because of my lack of knowledge and understand that people with more knowledge probably will not use this tool for their installation as they are more comfortable with programming and Mysensors.Intro aside I do not have any problems with using additional application, I already use Windows GUI/Controller for MySensors made by @tekka and Android application to debug and develop my installation.
Best part would be that code generation and debugging could be in one application, but that's not necessary.Keep up the good work.
-
Hi, I can give a quick look from my (noob) perspective.
I'm no programmer, just tinkering around, new to Mysensors, but already managed to get something working and starting to learn how this stuff works so I can expand my installation.
Im wery interested in this firmware generator because of my lack of knowledge and understand that people with more knowledge probably will not use this tool for their installation as they are more comfortable with programming and Mysensors.Intro aside I do not have any problems with using additional application, I already use Windows GUI/Controller for MySensors made by @tekka and Android application to debug and develop my installation.
Best part would be that code generation and debugging could be in one application, but that's not necessary.Keep up the good work.
@archiijs I'm very much in the same boat as you. While I have some experience writing code, C and C++ is all Greek and Latin to me. I can maybe read a little bit of C/C++ if my life depends on it, but that's about it. That's why I'm creating this tool. :)
Thanks for the feedback about app downloads being ok. I'll keep that in mind. Ideally, there won't be any need for debugging, since the generated code would be perfect right away, but I've been told that we don't live in an perfect world. However, viewing the serial monitor is definitely one of those things a desktop app can do, which a browser can't. I'm definitely excited about the possibility of communicating straight with the device.
It might take me some time to get to it though. However, all of the work I'm doing right now will be reused for the desktop app as well, so effort isn't wasted. At this moment, I guess it's wisest to see if this web app gets traction. Once we know it's being used, it'll be worthwhile making it better.
PS: How does the name 'MySensors Configurator' sound? Not the most creative, I know...
-
@rakeshpai Hi, I'm following this topic closely, found your project on google to see if something similar already exists. I was surprised that it does + it's based on nodejs, couldn't be more happy!
It's really a good intention for people who wants to get started quickly and it would perfectly complete the NodeManager.About the method, i was thinking about something similar that dagoma did ( a 3d printer manufacturer, based on Marlin firmware --> Arduino ). They made a subdomain where you can generate a .hex files, functions of the options you have. Then they made a desktop app which lets you upload the generated file.
That's not the most integrated and straight, but it works fine!
Which looks like your approch you show on your demo (btw the link on your github doesn't work, http://rakeshpai.github.io/~~m~~mysensors-network-manager )That maybe a good start to flash an arduino with an app ? https://github.com/noopkat/avrgirl-arduino
If you need / want help, I'm actually working on a project with nodejs and react, so 'im in the good mood ;)
-
@rakeshpai Hi, I'm following this topic closely, found your project on google to see if something similar already exists. I was surprised that it does + it's based on nodejs, couldn't be more happy!
It's really a good intention for people who wants to get started quickly and it would perfectly complete the NodeManager.About the method, i was thinking about something similar that dagoma did ( a 3d printer manufacturer, based on Marlin firmware --> Arduino ). They made a subdomain where you can generate a .hex files, functions of the options you have. Then they made a desktop app which lets you upload the generated file.
That's not the most integrated and straight, but it works fine!
Which looks like your approch you show on your demo (btw the link on your github doesn't work, http://rakeshpai.github.io/~~m~~mysensors-network-manager )That maybe a good start to flash an arduino with an app ? https://github.com/noopkat/avrgirl-arduino
If you need / want help, I'm actually working on a project with nodejs and react, so 'im in the good mood ;)
@getlarge Thanks! It's always nice to know that this is potentially useful. :)
Dogoma's approach is interesting, but as you said, not the most integrated. If there has to be an app downloaded anyway, why not go the next step and make it all easy to use.
Thanks for pointing out the issue with the readme. Fixed.
avrgirl is awesome! Finally someone decided to modernise a 20 year old piece of software. :)
You are welcome to dive into the code if you like. Node is only used for development. It's hosted on a static server, so there's no server-side language. The only problem with diving into code right now is that things aren't documented well, I haven't outlined what the goals/directions are, and it's kinda early, so I'm making sweeping changes all the time. But if you can get past that, feel free to dive in and familiarise yourself with the code, and even start making changes.
Speaking of which:
Update: No functionality change, but I've applied a fresh coat of paint to the UI. Hopefully it looks a little better now. There are still some cobwebs in the corners - I'll get to them soon. :) -
Since I haven't actually done this yet, I thought I'd outline the 'stack' used for this app, in case folks want to dive into the code.
- Based on React, using create-react-app.
- Offline first. Uses service-workers and localStorage for offline support.
- There's no server-side language. It's just a static site hosted on GitHub pages. Node.js is used for local development tooling.
- Uses redux for state management, and react-router for client-side routing.
- CSS-in-JS with glamor.
So, it's a very modern stack. Cutting-edge web stuff.
It's very early in the project, so I don't have any long-term plans outlined. In fact, I'll be honest, I haven't thought that far. But it's a community project, so you are welcome to steer the project as you see right.
Some areas to help:
- I suck at UI beautification work. Any help will be awesome.
- Need to investigate Electron for building native apps. Some things to look for: (a) Reusing the website directly from GitHub pages, so that we have a single deploy for the UI and the site still works fine if people don't want to download an app. Ideally, the electron app will simply be a container for the site. (b) Automatic updates to the container app. (c) Serial communication (node-serialport? avrgirl?) (d) More, as we go deeper.
- Ideas, suggestions? Like I said, I haven't thought too far. ;)
Actually, the absolute best way to help right now is to simply poke around the UI and point out issues, no matter how minor they are. It's the little things and attention to detail in an app that make it nice to use, so it'll help to have an obsessive pair of eyes going over the UI. You can point out bugs, usability issues, things that might be confusing, errors in the generated code, or any other changes you think will improve the app.
You can comment here with suggestions, or use GitHub if you like.
-
Just a quick update: I've been doing a bunch of work on the visual aspects of the app. Dare I say, it looks a lot better now. :) Check it out. Feedback welcome.
-
Awesome @rakeshpai!. Not sure if it is a glitch in my chrome, but I wasn't able to download the code any longer.
@hek Thanks. The download works for me just fine. Can you provide any more details? Do you see any errors in the browser's console? Also, hope you are using a desktop OS.
I haven't made much/any changes to the functionality, so downloads should be working as it was.
