Browser-based firmware generator

[user2684]

@rakeshpai I assume the radio doesn't matter but for sure I was missing the test with one node communicating with this gateway. I'll try to reproduce exactly the same and report back. Thanks!

[user2684]

Finally I got it! My mistake of course, the message from the NodeManager's configuration child id of the remote node was considered as a message for a local sensor by the gw leading a non existent array index access and so the crash. I've fixed the issue and added a few more safeguards here and there on https://github.com/user2684/NodeManager/tree/release/1.5.1. If you can give it a try would be great. Thanks!

[Alomamabobo]

Thanks for the sharing, good idea, will try your script and let know :)

[user2684]

I finally had the chance to spend some time and review the output and put together some comments. Of course some of those are just personal opinions so take out from here whatever you like the most :)

• I would add the "+" for adding a new node on every screen, when on the gateway page it took me a while to realize the button was on the other page
• I love the way you can see realtime the code generated, easier not only during this phase but also for advanced users to have better understanding of the output
• I would personally disable signing and encryption by default and I'll let the user chose. Especially for newbie this would avoid the user to understand the steps required to set it up
• On the gateway config.h, I'd turn off POWER_MANAGER and BATTERY_SENSOR since do not apply
• On both the gateway and the node I'd turn off PERSIST (requires the user to understand what is persisted so better to leave it for advanced users to manually enable it) and SERVICE_MESSAGES (I'm afraid sending out strings would impact the battery lifetime).
• ESP8266, when DHCP is set, static network configuration is added to config.h anyway
• when I click on code and the back on editor, I see the "Other settings" but I cannot go back to the main settings and I have to click on the node on the left side to restore the full layout
• I'm not sure "As a client, connecting to the controller's server." is applicable. As far as I've understood, it can act either as a server or connecting to a MQTT broker but I may be wrong
• I'd not use setPowerPins() by default since it is not necessarily something a new user would use/understand
• The code generated for setPowerPins() is incorrect, you should request the user the gnd pin AND the vcc pin. I see now 12 is always used as vcc pin.
• For PIR/Door/Switch I'd provide pin 2 or 3 options since those are the only pins accepting interrupts
• When generating the code for a switch sensor, the output includes "int switch". Since switch is a reserved keyword, I guess it would not compile correctly
• I'd keep debounce to 0 by default for the switch sensor
• For the latching relay, I'd add setPulseWidth() since it is a common setting a user would need to customize
• I'd use RF24_PA_HIGH by default, not because is wrong asking the user to confirm regarding the good power supply but I'm afraid a new user would leave it unchecked and would complain about the poor range he will get
• I'd split the battery powered checkbox from the mode the node should operate. When not battery powered, in order to get some data out of your sensors, you still need to use setMode() and use WAIT for having a cycle and sensors reporting at the end of every cycle. I'd probably need to change this in future releases since not very intuitive but this is another story :)

[rakeshpai]

@user2684 Woah! Thanks for taking the time to write and share your notes. This is invaluable! Thanks again for this list.

Some quick thoughts:

• I'd rather stick with the 'secure by default, can't be turned off' approach. I understand the usability problems, and want to fix them, so I'm on board with the usability issue. I have a solution in mind, but it's a little ways off. Until then, I'll probably just solve the usability issue with instructions to flash correctly. (It's already at the point where you just have to flash the SecurityPersonalizer.ino once, and then the main sketch. That's it.)
• Running as a client should be possible (reference), but I haven't tried it myself, and the generated code is likely wrong anyway.
• Why set debounce to 0 by default? The debounce option only shows up for door and switch sensors. I'm assuming that both are mechanical switches (even if magnetically actuated as in the case of door switches), so both cases would need debouncing anyway, right?
• I didn't understand your last point about setMode. If you are referring to setMode on the sensor object for switch-type sensors, that doesn't have anything to do with whether the node is battery powered. Clearly, I've misunderstood something.

The rest of what you pointed out either reflect my misunderstanding, or is an incomplete implementation, or are straight-up bugs. Thanks again for your list. It's invaluable. I'll have fixes ready soon.

[user2684]

@rakeshpai no problem, it is just a shame it took so long for me to review it in depth :)

I'd rather stick with the 'secure by default, can't be turned off' approach. I

Having some sort of README file in the zip would probably help in this direction, at least to explain the process (flash the security personalizer first, then the sketch).

Why set debounce to 0 by default? The debounce option only shows up for

I have a few different types of motion sensors around and debounce was never necessary, at least for me. Even worse, I've noticed from the posts here in the forum about NM that can confuse the users leading to weird behaviors. Of course if set very low like in your sketch, it should be fine.

I didn't understand your last point about setMode. If you are referring to

Not your fault at all, it is confusing for everybody I think, this is why I need to do something and change it :) By default all the sensors report their measures at the end of a sleeping/wait cycle regardless if the node is battery powered or not. So if you have e.g. a temperature sensor attached to a node which is NOT battery powered, you will not get anything out of it unless you configure a WAIT or SLEEP cycle. This is why it is important to have this split from the battery powered checkbox. Let me know if not clear I'll share a few examples :)
Thanks

[rakeshpai]

@user2684 Ah, that makes sense. I was looking for something like a 'reporting interval' in the NodeManager API. So if I understand correctly, I should setMode to WAIT and call setSleep to specify the reporting interval?

[user2684]

@rakeshpai or better directly setSleepMode(WAIT,10,MINUTES). All the sensors report at the end of the sleep/wait cycle. I'm adding new options in the new version for making this more flexible. Also, WAIT would not make sensors waiting for interrupts working correctly since the interrupt is handle by a call to the MySensors sleep() only. I'm fixing this as well. Long story short, I'd recommend to make have SLEEP by default and a fallback to WAIT if the use uncheck the box

[rakeshpai]

This post is deleted!

[rakeshpai]

I've got an update with a bunch of fixes. Thanks, @user2684!

• BATTERY_SENSOR now mirrors BATTERY_MANAGER, and is only applicable for battery powered devices. Doesn't show up on the gateway.
• PERSIST and SERVICE_MESSAGES set to 0.
• Fixes for code generation related to DHCP.
• UI fix - switching to Editor view for gateway doesn't mess up the UI.
• switch variable name changed to inputSwitch to avoid conflict with keywords.
• Tests.

@user2684 It looks like I'll wait for your next release before I close some of the other issues:

• setPowerPin() - I'm waiting for a release with this fix.
• If you are fixing how setSleepMode behaves, I'd rather wait for that to come through in a release as well.

No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush. :)

EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!

[user2684]

@rakeshpai I really hope my busy life would not impact your project preventing other users to leverage your fantastic tool :-) But honestly could make sense to wait for v1.6 since a few things are changing. But please let me know if and when I'll delaying too much and we'll find a backup plan.

Thanks for your feedback regarding 1.5.1, I'll make it publicly available during the weekend.

[rakeshpai]

Need to bounce this off someone. Please share your thoughts.

I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!

The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.

Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.

There's two other alternatives I can think of:

1. Create a CLI tool that communicates with a browser. Users have to install and run the CLI app in a terminal, and while its running, they can flash their devices from the browser. This is the most minimal approach that solves the problem, but requires the user to juggle a terminal, which isn't ideal.
2. [Preferred] Create a native app, throw away the site, do everything inside the app. The native app can reuse most of the site's code, so the work put in so far is not wasted effort. The app also gets much more privileges, since it is running natively.

The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.

So, in summary, the options are:

1. Don't flash the device from the browser, and just offer the generated code for download, as is happening right now. Limits utility and future features, and frankly, isn't exciting.
2. Ask the user to install a native app. The app gets lots of capabilities this way, but has the downside of requiring a download and install, which is a massive cause of friction.

If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)

I'm confused about how to proceed. What would you do? What do you think is the best way ahead?

[Anticimex]

@rakeshpai if you go for a cloud based programming method, you also need to mind security. Users would like insurance that their nodes that happen to be hooked up to a pc do get exactly the firmware they ask for and only when they ask for it. Also that the firmware goes only in one direction.

[rakeshpai]

@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.

The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.

As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.

So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.

[core_c]

I am one of those few users that opened chrome://flags, and set:
Enable WebUSB support: disabled.
I already have installed a tool to program my Arduinos. I guess, it's the first thing most people install after aquiring an Arduino.
I also do not understand why i would ever use a cloud.
Perhaps i am "too oldskool".. using a browser to find what i search for, and (if it is code,) then download it.
When i want code, i do not care how i get it. Important is, that i can download it..
The bare product is what counts..

[Anticimex]

@rakeshpai just out of curiosity, how do you plan to handle code compilation? If we look beyond the problem of interfacing with a serial device, you also need to compile the sketch, and link it to a hex executable hex file. I'm on saying it can't be done (it has been done before) but I would like to understand how it is practically done. I am curious since I maintain the Jenkins instance for MySensors, and I am open for alternatives to compile things. It uses installed instances of the Arduino IDE on the server side.

[rakeshpai]

@Anticimex My current favourite is the PlatformIO Core CLI. While it was built to work in an IDE, it's actually completely independent, and only requires Python to run. It has support for lots of boards too, and has a built-in package manager making dependency management very easy. Since it's a CLI, it can easily be automated with bash scripts, can run headless, and doesn't require a windowing environment. Can't do that with GUIs like Arduino. There are alternatives to PlatformIO, but none that I found were feature-complete.

In fact, while I haven't exposed it, I've already implemented a way to download the code in a folder-structure that platformio prefers. I'll be rolling that out soon.

@core_c I completely understand the security issues of cloud compilation. But I also see the convenience it brings. If I do implement compilation in the cloud, I'm not going to take away the ability to download the code. You won't be forced to use the cloud. That said, I'm not going to implement cloud compilation at this moment, since it's pointless to do so if I can't also flash the device, and I currently can't flash the device from the browser.

[archiijs]

Hi, I can give a quick look from my (noob) perspective.
I'm no programmer, just tinkering around, new to Mysensors, but already managed to get something working and starting to learn how this stuff works so I can expand my installation.
Im wery interested in this firmware generator because of my lack of knowledge and understand that people with more knowledge probably will not use this tool for their installation as they are more comfortable with programming and Mysensors.

Intro aside I do not have any problems with using additional application, I already use Windows GUI/Controller for MySensors made by @tekka and Android application to debug and develop my installation.
Best part would be that code generation and debugging could be in one application, but that's not necessary.

Keep up the good work.

[rakeshpai]

@archiijs I'm very much in the same boat as you. While I have some experience writing code, C and C++ is all Greek and Latin to me. I can maybe read a little bit of C/C++ if my life depends on it, but that's about it. That's why I'm creating this tool. :)

Thanks for the feedback about app downloads being ok. I'll keep that in mind. Ideally, there won't be any need for debugging, since the generated code would be perfect right away, but I've been told that we don't live in an perfect world. However, viewing the serial monitor is definitely one of those things a desktop app can do, which a browser can't. I'm definitely excited about the possibility of communicating straight with the device.

It might take me some time to get to it though. However, all of the work I'm doing right now will be reused for the desktop app as well, so effort isn't wasted. At this moment, I guess it's wisest to see if this web app gets traction. Once we know it's being used, it'll be worthwhile making it better.

PS: How does the name 'MySensors Configurator' sound? Not the most creative, I know...

[getlarge]

@rakeshpai Hi, I'm following this topic closely, found your project on google to see if something similar already exists. I was surprised that it does + it's based on nodejs, couldn't be more happy!
It's really a good intention for people who wants to get started quickly and it would perfectly complete the NodeManager.

About the method, i was thinking about something similar that dagoma did ( a 3d printer manufacturer, based on Marlin firmware --> Arduino ). They made a subdomain where you can generate a .hex files, functions of the options you have. Then they made a desktop app which lets you upload the generated file.
That's not the most integrated and straight, but it works fine!
Which looks like your approch you show on your demo (btw the link on your github doesn't work, http://rakeshpai.github.io/~~m~~mysensors-network-manager )

That maybe a good start to flash an arduino with an app ? https://github.com/noopkat/avrgirl-arduino

If you need / want help, I'm actually working on a project with nodejs and react, so 'im in the good mood ;)