Browser-based firmware generator

[rakeshpai]

@user2684 Sorry for the delay. I tested the whole thing as above with signing and without encryption. During compile, it did appear to reduce memory usage by about 20%. However, it didn't change the behaviour of the gateway. It still hangs after receiving the first message.

Good catch about the RADIO OK message. So, I tried removing signing and encryption, just to eliminate variables, and I still didn't see the RADIO OK message. Also, I was still seeing the gateway crash even without signing and encryption. I could have sworn this used to work fine. Now I don't even know what to believe of my previous tests anymore. Did I botch them up, or mis-remember them? Is this real life? :P

I guess I'll build another Arduino Pro Mini + NRF on a perfboard and try again. I'll update once I have something. Meanwhile, and I know this is a big ask, but if you could please test on hardware you have laying around that you know works well, it'll really help to know if this is somehow just my setup acting up.

EDIT: I flashed the node sketch on the board I was normally using for the gateway, and I still don't see a RADIO OK message. This either means that both my boards are broken (possible but unlikely), or that the RADIO OK message isn't printed in the serial output by the sketch at all. I haven't looked at the code yet, so I'm not sure which it is. Also, worth remembering that even though the node doesn't print a RADIO OK message, the gateway definitely reacts to the node being powered on (by crashing), so there's definitely some radio communication going on.

[rakeshpai]

I'm glad to report that I'm not completely losing my mind. :P

The difference between the recent tests, and my initial tests (where I saw everything working), was that in my recent tests, I was using NodeManager 1.5.1 from your fork.

I reverted to using the NodeManager from the MySensors repo's master branch, disabled signing and encryption, and everything worked. To push the boundaries slightly, I enabled signing, but kept encryption disabled, and flashed this on the gateway and on the node. Compiler messages:

Sketch uses 22,772 bytes (74%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,871 bytes (91%) of dynamic memory, leaving 177 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

So, not enough leg room memory-wise, but I tried running it anyway:

NodeManager v1.5
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY

MY I=0 M=255
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=
255;155;3;0;3;
RECV S=255 I=155 C=3 T=3 P=
255;86;3;0;3;
RECV S=255 I=86 C=3 T=3 P=
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=

I don't have any controller running, so nothing's handling the serial data from the gateway. However, you can see the last few lines in the paste above, where the gateway receives ID requests from the node just fine, and without a crash.

Considering that the only difference between my last post and this one is the version of NodeManager I've been using, I think it's safe to conclude that the crashes I was facing was due to 1.5.1. Not sure what the fix is, but at least we've identified the issue, somewhat at least.

That said, 1.5.1 was definitely the correct direction in terms of memory usage, since I could fit everything in with it. (With 1.5, I have to disable encryption to get it to fit.) So, if it is possible to identify and fix the stability issue with 1.5.1, and still be able to fit within memory limits, that'd be ideal.

EDIT: I still don't see the RADIO_OK message, or anything else really, in the node's serial output, though the gateway behaves like it's receiving messages just fine. I suspect the node just isn't printing that string to the serial output.

[user2684]

@rakeshpai thanks for reporting back, sounds like a serious problem with 1.5.1, I'm glad I've not released it yet :) I will try to reproduce and of course fix the issue and I'll see if I'll get the same with a serial gw as well during the weekend. Sorry for making you wait for so long :-(

[user2684]

@rakeshpai unfortunately I'm struggling in reproducing the issue :-(
I open up 1.5.1 and just changed the radio settings and defined MY_GATEWAY_SERIAL without touching anything else. This is my config.h file:

#ifndef config_h
#define config_h

/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "NodeManager"
#define SKETCH_VERSION "1.0"

/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG
//#define MY_NODE_ID 100

// NRF24 radio settings
//#define MY_RADIO_NRF24
//#define MY_RF24_ENABLE_ENCRYPTION
//#define MY_RF24_CHANNEL 76
//#define MY_RF24_PA_LEVEL RF24_PA_HIGH
//#define MY_DEBUG_VERBOSE_RF24

// RFM69 radio settings
#define MY_RADIO_RFM69
#define MY_RFM69_FREQUENCY RFM69_868MHZ
#define MY_IS_RFM69HW
//#define MY_DEBUG_VERBOSE_RFM69
#define MY_RFM69_NEW_DRIVER
//#define MY_RFM69_ENABLE_ENCRYPTION
#define MY_RFM69_NETWORKID 110
//#define MY_RF69_IRQ_PIN D1
//#define MY_RF69_IRQ_NUM MY_RF69_IRQ_PIN
//#define MY_RF69_SPI_CS D2

/**********************************
 * MySensors gateway configuration
 */
// Common gateway settings
//#define MY_REPEATER_FEATURE

// Serial gateway settings
#define MY_GATEWAY_SERIAL

// Ethernet gateway settings
//#define MY_GATEWAY_W5100

// ESP8266 gateway settings
//#define MY_GATEWAY_ESP8266
//#define MY_ESP8266_SSID ""
//#define MY_ESP8266_PASSWORD ""

// Gateway networking settings
//#define MY_IP_ADDRESS 192,168,178,87
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0
//#define MY_PORT 5003
//#define MY_GATEWAY_MAX_CLIENTS 2
//#define MY_USE_UDP

// Gateway MQTT settings
//#define MY_GATEWAY_MQTT_CLIENT
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68
//#define MY_PORT 1883
//#define MY_MQTT_USER "username"
//#define MY_MQTT_PASSWORD "password"
//#define MY_MQTT_CLIENT_ID "mysensors-1"
//#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
//#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Gateway inclusion mode
//#define MY_INCLUSION_MODE_FEATURE
//#define MY_INCLUSION_BUTTON_FEATURE
//#define MY_INCLUSION_MODE_DURATION 60
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Gateway Leds settings
//#define MY_DEFAULT_ERR_LED_PIN 4
//#define MY_DEFAULT_RX_LED_PIN  5
//#define MY_DEFAULT_TX_LED_PIN  6

/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
#define DEBUG 1

// if enabled, enable the capability to power on sensors with the arduino's pins to save battery while sleeping
#define POWER_MANAGER 1
// if enabled, will load the battery manager library to allow the battery level to be reported automatically or on demand
#define BATTERY_MANAGER 1
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the remote configuration settings on EEPROM
#define PERSIST 0
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 0

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 1
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 1
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 1
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0
#endif

This is what I get on my serial output:

NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;NodeManager
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
PRES I=201, T=30
0;201;0;0;30;
BATT V=3.24 P=91
SEND D=0 I=201 C=0 T=38 S= I=0 F=3.24
0;201;1;0;38;3.24
0;255;3;0;0;91
READY

MY I=0 M=1

Do you see any difference between my settings and yours?
Thanks

[rakeshpai]

@user2684 I can't run your sketch because I don't have an RFM69 handy. However, that's one noticeable difference between your sketch and mine - I am using the NRF24L01+.

Other than that, the generated gateway sketch sets the following flags differently from yours, in the config.h:

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
#define PERSIST 1
#define BATTERY_SENSOR 1

I've assumed that the gateway isn't battery powered, so I've set the BATTERY_MANAGER to 0. POWER_MANAGER is only set depending on choices made for sensor settings, so if there are no sensors POWER_MANAGER is set to 0. I guess the BATTERY_SENSOR value in the generated sketch is wrong, since it should be 0 for the gateway as the gateway is not battery powered.

All that said, I really only experience the crash when the gateway receives a message, so to replicate my scenario, you'd need to set up another node and (if it hasn't got an ID already), it'll send out an ID request to the gateway. It's on receiving that ID request that the gateway dies in my case.

Hope that helps. Let me know if there's any more details I can provide.

[user2684]

@rakeshpai I assume the radio doesn't matter but for sure I was missing the test with one node communicating with this gateway. I'll try to reproduce exactly the same and report back. Thanks!

[user2684]

Finally I got it! My mistake of course, the message from the NodeManager's configuration child id of the remote node was considered as a message for a local sensor by the gw leading a non existent array index access and so the crash. I've fixed the issue and added a few more safeguards here and there on https://github.com/user2684/NodeManager/tree/release/1.5.1. If you can give it a try would be great. Thanks!

[Alomamabobo]

Thanks for the sharing, good idea, will try your script and let know :)

[user2684]

I finally had the chance to spend some time and review the output and put together some comments. Of course some of those are just personal opinions so take out from here whatever you like the most :)

• I would add the "+" for adding a new node on every screen, when on the gateway page it took me a while to realize the button was on the other page
• I love the way you can see realtime the code generated, easier not only during this phase but also for advanced users to have better understanding of the output
• I would personally disable signing and encryption by default and I'll let the user chose. Especially for newbie this would avoid the user to understand the steps required to set it up
• On the gateway config.h, I'd turn off POWER_MANAGER and BATTERY_SENSOR since do not apply
• On both the gateway and the node I'd turn off PERSIST (requires the user to understand what is persisted so better to leave it for advanced users to manually enable it) and SERVICE_MESSAGES (I'm afraid sending out strings would impact the battery lifetime).
• ESP8266, when DHCP is set, static network configuration is added to config.h anyway
• when I click on code and the back on editor, I see the "Other settings" but I cannot go back to the main settings and I have to click on the node on the left side to restore the full layout
• I'm not sure "As a client, connecting to the controller's server." is applicable. As far as I've understood, it can act either as a server or connecting to a MQTT broker but I may be wrong
• I'd not use setPowerPins() by default since it is not necessarily something a new user would use/understand
• The code generated for setPowerPins() is incorrect, you should request the user the gnd pin AND the vcc pin. I see now 12 is always used as vcc pin.
• For PIR/Door/Switch I'd provide pin 2 or 3 options since those are the only pins accepting interrupts
• When generating the code for a switch sensor, the output includes "int switch". Since switch is a reserved keyword, I guess it would not compile correctly
• I'd keep debounce to 0 by default for the switch sensor
• For the latching relay, I'd add setPulseWidth() since it is a common setting a user would need to customize
• I'd use RF24_PA_HIGH by default, not because is wrong asking the user to confirm regarding the good power supply but I'm afraid a new user would leave it unchecked and would complain about the poor range he will get
• I'd split the battery powered checkbox from the mode the node should operate. When not battery powered, in order to get some data out of your sensors, you still need to use setMode() and use WAIT for having a cycle and sensors reporting at the end of every cycle. I'd probably need to change this in future releases since not very intuitive but this is another story :)

[rakeshpai]

@user2684 Woah! Thanks for taking the time to write and share your notes. This is invaluable! Thanks again for this list.

Some quick thoughts:

• I'd rather stick with the 'secure by default, can't be turned off' approach. I understand the usability problems, and want to fix them, so I'm on board with the usability issue. I have a solution in mind, but it's a little ways off. Until then, I'll probably just solve the usability issue with instructions to flash correctly. (It's already at the point where you just have to flash the SecurityPersonalizer.ino once, and then the main sketch. That's it.)
• Running as a client should be possible (reference), but I haven't tried it myself, and the generated code is likely wrong anyway.
• Why set debounce to 0 by default? The debounce option only shows up for door and switch sensors. I'm assuming that both are mechanical switches (even if magnetically actuated as in the case of door switches), so both cases would need debouncing anyway, right?
• I didn't understand your last point about setMode. If you are referring to setMode on the sensor object for switch-type sensors, that doesn't have anything to do with whether the node is battery powered. Clearly, I've misunderstood something.

The rest of what you pointed out either reflect my misunderstanding, or is an incomplete implementation, or are straight-up bugs. Thanks again for your list. It's invaluable. I'll have fixes ready soon.

[user2684]

@rakeshpai no problem, it is just a shame it took so long for me to review it in depth :)

I'd rather stick with the 'secure by default, can't be turned off' approach. I

Having some sort of README file in the zip would probably help in this direction, at least to explain the process (flash the security personalizer first, then the sketch).

Why set debounce to 0 by default? The debounce option only shows up for

I have a few different types of motion sensors around and debounce was never necessary, at least for me. Even worse, I've noticed from the posts here in the forum about NM that can confuse the users leading to weird behaviors. Of course if set very low like in your sketch, it should be fine.

I didn't understand your last point about setMode. If you are referring to

Not your fault at all, it is confusing for everybody I think, this is why I need to do something and change it :) By default all the sensors report their measures at the end of a sleeping/wait cycle regardless if the node is battery powered or not. So if you have e.g. a temperature sensor attached to a node which is NOT battery powered, you will not get anything out of it unless you configure a WAIT or SLEEP cycle. This is why it is important to have this split from the battery powered checkbox. Let me know if not clear I'll share a few examples :)
Thanks

[rakeshpai]

@user2684 Ah, that makes sense. I was looking for something like a 'reporting interval' in the NodeManager API. So if I understand correctly, I should setMode to WAIT and call setSleep to specify the reporting interval?

[user2684]

@rakeshpai or better directly setSleepMode(WAIT,10,MINUTES). All the sensors report at the end of the sleep/wait cycle. I'm adding new options in the new version for making this more flexible. Also, WAIT would not make sensors waiting for interrupts working correctly since the interrupt is handle by a call to the MySensors sleep() only. I'm fixing this as well. Long story short, I'd recommend to make have SLEEP by default and a fallback to WAIT if the use uncheck the box

[rakeshpai]

This post is deleted!

[rakeshpai]

I've got an update with a bunch of fixes. Thanks, @user2684!

• BATTERY_SENSOR now mirrors BATTERY_MANAGER, and is only applicable for battery powered devices. Doesn't show up on the gateway.
• PERSIST and SERVICE_MESSAGES set to 0.
• Fixes for code generation related to DHCP.
• UI fix - switching to Editor view for gateway doesn't mess up the UI.
• switch variable name changed to inputSwitch to avoid conflict with keywords.
• Tests.

@user2684 It looks like I'll wait for your next release before I close some of the other issues:

• setPowerPin() - I'm waiting for a release with this fix.
• If you are fixing how setSleepMode behaves, I'd rather wait for that to come through in a release as well.

No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush. :)

EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!

[user2684]

@rakeshpai I really hope my busy life would not impact your project preventing other users to leverage your fantastic tool :-) But honestly could make sense to wait for v1.6 since a few things are changing. But please let me know if and when I'll delaying too much and we'll find a backup plan.

Thanks for your feedback regarding 1.5.1, I'll make it publicly available during the weekend.

[rakeshpai]

Need to bounce this off someone. Please share your thoughts.

I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!

The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.

Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.

There's two other alternatives I can think of:

1. Create a CLI tool that communicates with a browser. Users have to install and run the CLI app in a terminal, and while its running, they can flash their devices from the browser. This is the most minimal approach that solves the problem, but requires the user to juggle a terminal, which isn't ideal.
2. [Preferred] Create a native app, throw away the site, do everything inside the app. The native app can reuse most of the site's code, so the work put in so far is not wasted effort. The app also gets much more privileges, since it is running natively.

The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.

So, in summary, the options are:

1. Don't flash the device from the browser, and just offer the generated code for download, as is happening right now. Limits utility and future features, and frankly, isn't exciting.
2. Ask the user to install a native app. The app gets lots of capabilities this way, but has the downside of requiring a download and install, which is a massive cause of friction.

If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)

I'm confused about how to proceed. What would you do? What do you think is the best way ahead?

[Anticimex]

@rakeshpai if you go for a cloud based programming method, you also need to mind security. Users would like insurance that their nodes that happen to be hooked up to a pc do get exactly the firmware they ask for and only when they ask for it. Also that the firmware goes only in one direction.

[rakeshpai]

@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.

The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.

As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.

So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.

[core_c]

I am one of those few users that opened chrome://flags, and set:
Enable WebUSB support: disabled.
I already have installed a tool to program my Arduinos. I guess, it's the first thing most people install after aquiring an Arduino.
I also do not understand why i would ever use a cloud.
Perhaps i am "too oldskool".. using a browser to find what i search for, and (if it is code,) then download it.
When i want code, i do not care how i get it. Important is, that i can download it..
The bare product is what counts..