Browser-based firmware generator

[rakeshpai]

I've hit a little bit of a wall, and need help to proceed.

I created a NRF24L01 network in the UI, and tried to flash the default serial gateway code to an Arduino Pro Mini 3.3v/8Mhz. There are no sensors configured on the gateway. I tried to flash the SecurityPersonalizer on the device, and it worked perfectly, and gave a happy sounding serial output. (Yay!) However, when I flash the main gateway sketch, I get a 'not enough memory' error.

Sketch uses 25,792 bytes (79%) of program storage space. Maximum is 32,256 bytes.
Global variables use 2,204 bytes (107%) of dynamic memory, leaving -156 bytes for local variables. Maximum is 2,048 bytes.
processing.app.debug.RunnerException: Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.
	at cc.arduino.Compiler.size(Compiler.java:339)
	at cc.arduino.Compiler.build(Compiler.java:159)
	at processing.app.SketchController.build(SketchController.java:641)
	at processing.app.Editor$BuildHandler.run(Editor.java:1782)
	at java.lang.Thread.run(Thread.java:745)
Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.

For reference, the generated gateway sketch is as follows:

// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES

// load user settings
#include "config.h"
// load MySensors library
#include <MySensors.h>
// load NodeManager library
#include "NodeManager.h"

// create a NodeManager instance
NodeManager nm;

// before
void before() {
  // setup the serial port baud rate
  Serial.begin(9600);

  

  nm.before();
}

// presentation
void presentation() {
  // call NodeManager presentation routine
  nm.presentation();
}

// setup
void setup() {
  // call NodeManager setup routine
  nm.setup();
}

// loop
void loop() {
  // call NodeManager loop routine
  nm.loop();
}

// receive
void receive(const MyMessage &message) {
  // call NodeManager receive routine
  nm.receive(message);
}

// receiveTime
void receiveTime(unsigned long ts) {
  // call NodeManager receiveTime routine
  nm.receiveTime(ts);
}

and the associated config.h is:

#ifndef config_h
#define config_h

/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
//#define MY_REPEATER_FEATURE


/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG


// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24


// Serial gateway settings
#define MY_GATEWAY_SERIAL


/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
//#define DEBUG 1

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 1
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 1

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0

#endif

I tried commenting out the first three lines in the gateway sketch (the #defines to do with security personalisation, and the sketch fits, but only barely.

Sketch uses 19,922 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,710 bytes (83%) of dynamic memory, leaving 338 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

I'm by no means a C/C++ guy (which I why I'm writing this tool ) so I'll need help to fix this. I don't even know where to start looking, since there's the combination of NodeManager and the software signing in the sketch.

Any chance you could look at this, @user2684 and @Anticimex? You could download a copy of the code from the UI if you like. My configuration is an NRF network, and the gateway is the default serial gateway on the UI. I've been using the latest version of the MySensors lib from the development branch on GitHub, and am doing the compilation in the Arduino IDE.

Thanks!

[Anticimex]

@rakeshpai I have never used node manager myself but it appears that it consumes quite some memory. I have to my best effort optimized the signing code to use as little memory as possible. That said, it could very well be possible to optimize it further, but I would then need other eyes on it. I have not looked into node manager, but as I understand it, it relies heavily on c++ (signing code is pretty much just c) and the atmega compiler optimized c++ code horribly bad. So I think that it is in nodemanager the biggest memory savings can be made.

[rakeshpai]

@Anticimex That's the feeling I had as well, considering that even after I disabled signing, while the sketch fits in memory, it still seemed to complain about low memory. @user2684 does this match your experience too?

[Anticimex]

@rakeshpai the warning is because the heap and stack is shared. So as the program claims storage for variables (typically globals statics and classes) the ram available for stack shrinks. And at some point it becomes to small. But the tools cannot determine when that happens as such analysis is quite complex. So they warn when it "feels" to be getting tight. But it ultimately comes down to how the program behaves and how the code handles function calls and parameters. So if you get the warning, it does not mean there is a problem. But you have entered what somebody decided to be the "danger zone" (probably based on typical sketch stack usage).

[user2684]

@rakeshpai sorry for the delay! NodeManager consumes some good memory indeed but I should have optimized this in v1.5 and the problem should have gone unless a lot of sensors are registered but this is not your case. Are you already using this version?
As a side note, I've noticed when MY_RF24_ENABLE_ENCRYPTION is defined a good 20% of memory got consumed on a pro mini but I guess this is not something we can do much about

[user2684]

Correction, this has been solved only in the development version (https://forum.mysensors.org/topic/6183/nodemanager-plugin-for-a-rapid-development-of-battery-powered-sensors/168).
But I'd recommend not to move there because a few things have changed and I'd prefer to provide you with a better recap of the changes to make your life easier. Probably best is that I put together a v1.5.1 version with just that issue fixed for you to use. I'll work on it and let you know between today and tomorrow. Thanks!

[Anticimex]

Generally, the beta branch of the MySensors library has much "better" memory footprint than the current release. This is also true for signing.

[rakeshpai]

@user2684 My version management is pretty crappy at the moment but I'm simply using the latest version of NodeManager.cpp and NodeManager.h from the master branch on GitHub. Users keep a local copy of NodeManager for purposes of offline availability, but they are forced to update their local versions of NodeManager everyday. So, it's safe to assume that within a margin of error of a day, everyone's on the latest version of your checkins on GitHub. (I know this isn't ideal and I should do version management better, but I too like to live dangerously B).)

Also, while MySensors itself isn't shipped as part of the download bundle, I've manually used the latest version from the dev branch in GitHub as well, in my arduino libraries folder.

Maybe I'm just doing something stupid that's causing the memory usage to inflate? I'll wait for whenever you can get a chance to review the output code. Thanks for looking at this!

[user2684]

@rakeshpai great so yes, you are using the latest version and btw, very smart method for keeping it up to date No, you are not doing anything stupid, it was me doing something stupid when releasing the latest version of NodeManager Of course together with encryption enabled this would end up consuming almost all the memory but the original sin was on my side. I'll definitely work on a hotfix for you, otherwise the code would become useless. And of course I'll review the output code but apologize in advance, real life will be very demanding for me in the next few months so my response will be almost always delayed

[rakeshpai]

@user2684 No problem with the delays - I've got enough things to do in the meantime. All those real-life things coming in the way all the time, amirite?

[user2684]

@rakeshpai yes indeed
I wonder if you can give a try to https://github.com/user2684/NodeManager/tree/release/1.5.1 before I'll make publicly available. It is basically the same v1.5 version you are using which just applied the memory optimization patch on top. It should save 25% of the memory so it is a huge fix
Let me know if works better. Something else I recommend to disable when you are approaching the memory limit is #define DEBUG 1 which will prevent storing all the debug messages giving you another boost.
Generally speaking you need to stay pretty distant from the 100% memory utilization with NodeManager: since it is allocating its objects dynamically (which is not a good practice but I couldn't find a better way), you can get weird behavior even if the compiler doesn't not complain.

Thanks!

[rakeshpai]

@user2684 Tried with the 1.5.1 release, and it fits!

Sketch uses 25,786 bytes (83%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,716 bytes (83%) of dynamic memory, leaving 332 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

It's still complaining about low memory, and from what I understand of what you said, this is a problem, but this is definitely an awesome direction. It's also a concern that I still haven't added any sensors yet - this is just a bare serial gateway sketch. However, I'm still excited. :metal:

I've kept debug off by default in the config for now, so debug messages aren't pushing us to the limit here.

EDIT: I must add, the result above is with signing and encryption, and all that good stuff.

[Anticimex]

@rakeshpai looks good. I get that warning myself and I have only seen issues when I push 90% but it also depend on what your sketch does. In any case, the warning is no guarantee for failure so I think it can be considered acceptable in most situations.

[rakeshpai]

Output from the gateway's serial:

NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY

MY I=0 M=1
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED

Output from a dummy node's serial:

REG I=1 P=14 P=16 T=23
PWR G=0 REG I=1 P=14 P=16 T=23
PWR G=0 V=12
NodeManager v1.5.1
INT1 M=255
INT2 M=255

Ok, so the both aren't talking to each other yet, and I'll investigate that. BUT! This is NodeManager 1.5.1 + Signing + Encryption, flashed on to two Arduino Pro Minis 3.3v, one with a serial gateway sketch, and the other with the node for a simple sketch of a LDR sensor, both generated from the UI!

You can't see me here, but I'm doing a little dance!

While I complete my little dance here, if you can spot anything obvious about why they aren't talking to each other, please let me know.

Also, minor UI update: New 'Are you sure?' dialogs when you are performing any deletions on the UI.

[Anticimex]

@rakeshpai The first question that comes to mind is, have you double checked your AES personalization? If you run with encryption enabled and have a mismatch in the PSK then nothing comes through in any direciton.
You can execute the personalizer without flags set and it should dump the EEPROM data on serial.

[rakeshpai]

@Anticimex From what it looks like at the moment, it looks like I'm hitting up against the memory issue. (But I might be wrong.)

I don't know why the two nodes weren't communicating in my previous post. However, I decided to start from scratch, and ran the ClearEEPROM sketch on both my arduinos. Then, I disabled personalisation on both the gateway and the node sketches, flashed them to the boards, and watched ID requests come on the gateway's serial output, from the node. (I don't have a controller running.) So, everything looks good so far.

Then I ran the SecurityPersonalizer on the gateway, and flashed the gateway sketch to it again. This was the output of the personalizer:

+------------------------------------------------------------------------------------+
|                           MySensors security personalizer                          |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                               Configuration settings                               |
+------------------------------------------------------------------------------------+
| * Guided personalization/storage of keys in EEPROM                                 |
| * Software based personalization (no ATSHA204A usage whatsoever)                   |
| * Will not require any UART confirmations                                          |
| * Will store HMAC key to EEPROM                                                    |
| * Will store AES key to EEPROM                                                     |
| * Will store soft serial to EEPROM                                                 |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                           Hardware security peripherals                            |
+--------------+--------------+--------------+------------------------------+--------+
| Device       | Status       | Revision     | Serial number                | Locked |
+--------------+--------------+--------------+------------------------------+--------+
| AVR          | DETECTED     | N/A          | N/A (generation required)    | N/A    |
+--------------+--------------+--------------+------------------------------+--------+

+------------------------------------------------------------------------------------+
|                                    Key storage                                     |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key                                                              |
+--------+--------+------------------------------------------------------------------+
| HMAC   | OK     | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES    | OK     | 1E4FAF1129029D34C4832F6B48044C8B                                 |
| SERIAL | OK     | BED2C17579AF3CDA6D                                               |
+--------+--------+------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                       EEPROM                                       |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key                                                              |
+--------+--------+------------------------------------------------------------------+
| HMAC   | OK     | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES    | OK     | 1E4FAF1129029D34C4832F6B48044C8B                                 |
| SERIAL | OK     | BED2C17579AF3CDA6D                                               |
+--------+--------+------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                      This nodes whitelist entry on other nodes                     |
+------------------------------------------------------------------------------------+
{.nodeId = <ID of this node>,.serial = {0xBE,0xD2,0xC1,0x75,0x79,0xAF,0x3C,0xDA,0x6D}}
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                  WHAT TO DO NEXT?                                  |
+------------------------------------------------------------------------------------+
| This device has now been personalized. Run this sketch with its current settings   |
| on all the devices in your network that have security enabled.                     |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                  Execution result                                  |
+------------------------------------------------------------------------------------+
| SUCCESS                                                                            |
+------------------------------------------------------------------------------------+

So, that's a success. A little confusingly, the output says 'Run this sketch with its current settings on all the devices in your network that have security enabled.' However, won't the device IDs / serial key be different on every device?

I then flashed the SecurityPersonalizer and node sketch on the other arduino. The output of the security personalizer is as follows, notably with a different serial key:

+------------------------------------------------------------------------------------+
|                           MySensors security personalizer                          |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                               Configuration settings                               |
+------------------------------------------------------------------------------------+
| * Guided personalization/storage of keys in EEPROM                                 |
| * Software based personalization (no ATSHA204A usage whatsoever)                   |
| * Will not require any UART confirmations                                          |
| * Will store HMAC key to EEPROM                                                    |
| * Will store AES key to EEPROM                                                     |
| * Will store soft serial to EEPROM                                                 |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                           Hardware security peripherals                            |
+--------------+--------------+--------------+------------------------------+--------+
| Device       | Status       | Revision     | Serial number                | Locked |
+--------------+--------------+--------------+------------------------------+--------+
| AVR          | DETECTED     | N/A          | N/A (generation required)    | N/A    |
+--------------+--------------+--------------+------------------------------+--------+

+------------------------------------------------------------------------------------+
|                                    Key storage                                     |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key                                                              |
+--------+--------+------------------------------------------------------------------+
| HMAC   | OK     | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES    | OK     | 1E4FAF1129029D34C4832F6B48044C8B                                 |
| SERIAL | OK     | 31D46BF4ADBEF72803                                               |
+--------+--------+------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                       EEPROM                                       |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key                                                              |
+--------+--------+------------------------------------------------------------------+
| HMAC   | OK     | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES    | OK     | 1E4FAF1129029D34C4832F6B48044C8B                                 |
| SERIAL | OK     | 31D46BF4ADBEF72803                                               |
+--------+--------+------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                      This nodes whitelist entry on other nodes                     |
+------------------------------------------------------------------------------------+
{.nodeId = <ID of this node>,.serial = {0x31,0xD4,0x6B,0xF4,0xAD,0xBE,0xF7,0x28,0x03}}
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                  WHAT TO DO NEXT?                                  |
+------------------------------------------------------------------------------------+
| This device has now been personalized. Run this sketch with its current settings   |
| on all the devices in your network that have security enabled.                     |
+------------------------------------------------------------------------------------+

+------------------------------------------------------------------------------------+
|                                  Execution result                                  |
+------------------------------------------------------------------------------------+
| SUCCESS                                                                            |
+------------------------------------------------------------------------------------+

Then, I watched the gateway serial output. It looks as follows:

NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY

MY I=0 M=255
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
// stable till here - then I power up my node...
255;118;3;0;3;
RECV S=255 I=118 C=3 T=3 P=
255;x00

...and it just dies. Sometimes it prints non-printable characters to the serial output, sometimes it goes into a reset loop, and sometimes the only way to reset it is to cycle power to it.

However, it does appear to receive the message from the node before it dies, judging by the 255;118;3;0;3; message, which is an ID request. It looks like the gateway dies when trying to handle that incoming message maybe? If I don't power up the node, the gateway seems to appear stable, but once I power up a node, the gateway receives the ID request, and it dies.

Also, I don't suspect this is a hardware issue. I've used the same hardware in past experiments with MySensors (without signing and NodeManager at the time), and it has worked fine. Also, I tried all this without personalisation earlier today, and it has worked fine. I've added 47uF electrolytic caps to both radios, so power shouldn't be an issue.

So, the erratic behaviour of the gateway when receiving a message seems to be suggesting that we're hitting memory limits, maybe?

[Anticimex]

@rakeshpai yes, that is a plausible explanation. The confusing message is probably because you have opted to manually set your serial instead of having it generated. Then you have to adjust your manual setting if you want unique serials on all devices.

[user2684]

@rakeshpai in your first test, I notice the gateway logs do not have the "RADIO OK" message which should show up once the radio is initialized so probably the radio was faulty or not connected correctly I may guess (a #define MY_DEBUG should tell you the truth if it will happen again).
In your second test, I'd definitely say it has something to do with the memory utilization. I wonder if you can keep signing on but turn encryption off so to save a good amount of memory and see if the tests will run just fine.

[rakeshpai]

@user2684 Sorry for the delay. I tested the whole thing as above with signing and without encryption. During compile, it did appear to reduce memory usage by about 20%. However, it didn't change the behaviour of the gateway. It still hangs after receiving the first message.

Good catch about the RADIO OK message. So, I tried removing signing and encryption, just to eliminate variables, and I still didn't see the RADIO OK message. Also, I was still seeing the gateway crash even without signing and encryption. I could have sworn this used to work fine. Now I don't even know what to believe of my previous tests anymore. Did I botch them up, or mis-remember them? Is this real life?

I guess I'll build another Arduino Pro Mini + NRF on a perfboard and try again. I'll update once I have something. Meanwhile, and I know this is a big ask, but if you could please test on hardware you have laying around that you know works well, it'll really help to know if this is somehow just my setup acting up.

EDIT: I flashed the node sketch on the board I was normally using for the gateway, and I still don't see a RADIO OK message. This either means that both my boards are broken (possible but unlikely), or that the RADIO OK message isn't printed in the serial output by the sketch at all. I haven't looked at the code yet, so I'm not sure which it is. Also, worth remembering that even though the node doesn't print a RADIO OK message, the gateway definitely reacts to the node being powered on (by crashing), so there's definitely some radio communication going on.

[rakeshpai]

I'm glad to report that I'm not completely losing my mind.

The difference between the recent tests, and my initial tests (where I saw everything working), was that in my recent tests, I was using NodeManager 1.5.1 from your fork.

I reverted to using the NodeManager from the MySensors repo's master branch, disabled signing and encryption, and everything worked. To push the boundaries slightly, I enabled signing, but kept encryption disabled, and flashed this on the gateway and on the node. Compiler messages:

Sketch uses 22,772 bytes (74%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,871 bytes (91%) of dynamic memory, leaving 177 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

So, not enough leg room memory-wise, but I tried running it anyway:

NodeManager v1.5
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY

MY I=0 M=255
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=
255;155;3;0;3;
RECV S=255 I=155 C=3 T=3 P=
255;86;3;0;3;
RECV S=255 I=86 C=3 T=3 P=
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=

I don't have any controller running, so nothing's handling the serial data from the gateway. However, you can see the last few lines in the paste above, where the gateway receives ID requests from the node just fine, and without a crash.

Considering that the only difference between my last post and this one is the version of NodeManager I've been using, I think it's safe to conclude that the crashes I was facing was due to 1.5.1. Not sure what the fix is, but at least we've identified the issue, somewhat at least.

That said, 1.5.1 was definitely the correct direction in terms of memory usage, since I could fit everything in with it. (With 1.5, I have to disable encryption to get it to fit.) So, if it is possible to identify and fix the stability issue with 1.5.1, and still be able to fit within memory limits, that'd be ideal.

EDIT: I still don't see the RADIO_OK message, or anything else really, in the node's serial output, though the gateway behaves like it's receiving messages just fine. I suspect the node just isn't printing that string to the serial output.

[user2684]

@rakeshpai thanks for reporting back, sounds like a serious problem with 1.5.1, I'm glad I've not released it yet I will try to reproduce and of course fix the issue and I'll see if I'll get the same with a serial gw as well during the weekend. Sorry for making you wait for so long

[user2684]

@rakeshpai unfortunately I'm struggling in reproducing the issue
I open up 1.5.1 and just changed the radio settings and defined MY_GATEWAY_SERIAL without touching anything else. This is my config.h file:

#ifndef config_h
#define config_h

/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "NodeManager"
#define SKETCH_VERSION "1.0"

/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG
//#define MY_NODE_ID 100

// NRF24 radio settings
//#define MY_RADIO_NRF24
//#define MY_RF24_ENABLE_ENCRYPTION
//#define MY_RF24_CHANNEL 76
//#define MY_RF24_PA_LEVEL RF24_PA_HIGH
//#define MY_DEBUG_VERBOSE_RF24

// RFM69 radio settings
#define MY_RADIO_RFM69
#define MY_RFM69_FREQUENCY RFM69_868MHZ
#define MY_IS_RFM69HW
//#define MY_DEBUG_VERBOSE_RFM69
#define MY_RFM69_NEW_DRIVER
//#define MY_RFM69_ENABLE_ENCRYPTION
#define MY_RFM69_NETWORKID 110
//#define MY_RF69_IRQ_PIN D1
//#define MY_RF69_IRQ_NUM MY_RF69_IRQ_PIN
//#define MY_RF69_SPI_CS D2

/**********************************
 * MySensors gateway configuration
 */
// Common gateway settings
//#define MY_REPEATER_FEATURE

// Serial gateway settings
#define MY_GATEWAY_SERIAL

// Ethernet gateway settings
//#define MY_GATEWAY_W5100

// ESP8266 gateway settings
//#define MY_GATEWAY_ESP8266
//#define MY_ESP8266_SSID ""
//#define MY_ESP8266_PASSWORD ""

// Gateway networking settings
//#define MY_IP_ADDRESS 192,168,178,87
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0
//#define MY_PORT 5003
//#define MY_GATEWAY_MAX_CLIENTS 2
//#define MY_USE_UDP

// Gateway MQTT settings
//#define MY_GATEWAY_MQTT_CLIENT
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68
//#define MY_PORT 1883
//#define MY_MQTT_USER "username"
//#define MY_MQTT_PASSWORD "password"
//#define MY_MQTT_CLIENT_ID "mysensors-1"
//#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
//#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Gateway inclusion mode
//#define MY_INCLUSION_MODE_FEATURE
//#define MY_INCLUSION_BUTTON_FEATURE
//#define MY_INCLUSION_MODE_DURATION 60
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Gateway Leds settings
//#define MY_DEFAULT_ERR_LED_PIN 4
//#define MY_DEFAULT_RX_LED_PIN  5
//#define MY_DEFAULT_TX_LED_PIN  6

/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
#define DEBUG 1

// if enabled, enable the capability to power on sensors with the arduino's pins to save battery while sleeping
#define POWER_MANAGER 1
// if enabled, will load the battery manager library to allow the battery level to be reported automatically or on demand
#define BATTERY_MANAGER 1
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the remote configuration settings on EEPROM
#define PERSIST 0
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 0

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 1
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 1
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 1
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0
#endif

This is what I get on my serial output:

NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;NodeManager
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
PRES I=201, T=30
0;201;0;0;30;
BATT V=3.24 P=91
SEND D=0 I=201 C=0 T=38 S= I=0 F=3.24
0;201;1;0;38;3.24
0;255;3;0;0;91
READY

MY I=0 M=1

Do you see any difference between my settings and yours?
Thanks

[rakeshpai]

@user2684 I can't run your sketch because I don't have an RFM69 handy. However, that's one noticeable difference between your sketch and mine - I am using the NRF24L01+.

Other than that, the generated gateway sketch sets the following flags differently from yours, in the config.h:

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
#define PERSIST 1
#define BATTERY_SENSOR 1

I've assumed that the gateway isn't battery powered, so I've set the BATTERY_MANAGER to 0. POWER_MANAGER is only set depending on choices made for sensor settings, so if there are no sensors POWER_MANAGER is set to 0. I guess the BATTERY_SENSOR value in the generated sketch is wrong, since it should be 0 for the gateway as the gateway is not battery powered.

All that said, I really only experience the crash when the gateway receives a message, so to replicate my scenario, you'd need to set up another node and (if it hasn't got an ID already), it'll send out an ID request to the gateway. It's on receiving that ID request that the gateway dies in my case.

Hope that helps. Let me know if there's any more details I can provide.

[user2684]

@rakeshpai I assume the radio doesn't matter but for sure I was missing the test with one node communicating with this gateway. I'll try to reproduce exactly the same and report back. Thanks!

[user2684]

Finally I got it! My mistake of course, the message from the NodeManager's configuration child id of the remote node was considered as a message for a local sensor by the gw leading a non existent array index access and so the crash. I've fixed the issue and added a few more safeguards here and there on https://github.com/user2684/NodeManager/tree/release/1.5.1. If you can give it a try would be great. Thanks!

[Alomamabobo]

Thanks for the sharing, good idea, will try your script and let know

[user2684]

I finally had the chance to spend some time and review the output and put together some comments. Of course some of those are just personal opinions so take out from here whatever you like the most

• I would add the "+" for adding a new node on every screen, when on the gateway page it took me a while to realize the button was on the other page
• I love the way you can see realtime the code generated, easier not only during this phase but also for advanced users to have better understanding of the output
• I would personally disable signing and encryption by default and I'll let the user chose. Especially for newbie this would avoid the user to understand the steps required to set it up
• On the gateway config.h, I'd turn off POWER_MANAGER and BATTERY_SENSOR since do not apply
• On both the gateway and the node I'd turn off PERSIST (requires the user to understand what is persisted so better to leave it for advanced users to manually enable it) and SERVICE_MESSAGES (I'm afraid sending out strings would impact the battery lifetime).
• ESP8266, when DHCP is set, static network configuration is added to config.h anyway
• when I click on code and the back on editor, I see the "Other settings" but I cannot go back to the main settings and I have to click on the node on the left side to restore the full layout
• I'm not sure "As a client, connecting to the controller's server." is applicable. As far as I've understood, it can act either as a server or connecting to a MQTT broker but I may be wrong
• I'd not use setPowerPins() by default since it is not necessarily something a new user would use/understand
• The code generated for setPowerPins() is incorrect, you should request the user the gnd pin AND the vcc pin. I see now 12 is always used as vcc pin.
• For PIR/Door/Switch I'd provide pin 2 or 3 options since those are the only pins accepting interrupts
• When generating the code for a switch sensor, the output includes "int switch". Since switch is a reserved keyword, I guess it would not compile correctly
• I'd keep debounce to 0 by default for the switch sensor
• For the latching relay, I'd add setPulseWidth() since it is a common setting a user would need to customize
• I'd use RF24_PA_HIGH by default, not because is wrong asking the user to confirm regarding the good power supply but I'm afraid a new user would leave it unchecked and would complain about the poor range he will get
• I'd split the battery powered checkbox from the mode the node should operate. When not battery powered, in order to get some data out of your sensors, you still need to use setMode() and use WAIT for having a cycle and sensors reporting at the end of every cycle. I'd probably need to change this in future releases since not very intuitive but this is another story

[rakeshpai]

@user2684 Woah! Thanks for taking the time to write and share your notes. This is invaluable! Thanks again for this list.

Some quick thoughts:

• I'd rather stick with the 'secure by default, can't be turned off' approach. I understand the usability problems, and want to fix them, so I'm on board with the usability issue. I have a solution in mind, but it's a little ways off. Until then, I'll probably just solve the usability issue with instructions to flash correctly. (It's already at the point where you just have to flash the SecurityPersonalizer.ino once, and then the main sketch. That's it.)
• Running as a client should be possible (reference), but I haven't tried it myself, and the generated code is likely wrong anyway.
• Why set debounce to 0 by default? The debounce option only shows up for door and switch sensors. I'm assuming that both are mechanical switches (even if magnetically actuated as in the case of door switches), so both cases would need debouncing anyway, right?
• I didn't understand your last point about setMode. If you are referring to setMode on the sensor object for switch-type sensors, that doesn't have anything to do with whether the node is battery powered. Clearly, I've misunderstood something.

The rest of what you pointed out either reflect my misunderstanding, or is an incomplete implementation, or are straight-up bugs. Thanks again for your list. It's invaluable. I'll have fixes ready soon.

[user2684]

@rakeshpai no problem, it is just a shame it took so long for me to review it in depth

I'd rather stick with the 'secure by default, can't be turned off' approach. I

Having some sort of README file in the zip would probably help in this direction, at least to explain the process (flash the security personalizer first, then the sketch).

Why set debounce to 0 by default? The debounce option only shows up for

I have a few different types of motion sensors around and debounce was never necessary, at least for me. Even worse, I've noticed from the posts here in the forum about NM that can confuse the users leading to weird behaviors. Of course if set very low like in your sketch, it should be fine.

I didn't understand your last point about setMode. If you are referring to

Not your fault at all, it is confusing for everybody I think, this is why I need to do something and change it By default all the sensors report their measures at the end of a sleeping/wait cycle regardless if the node is battery powered or not. So if you have e.g. a temperature sensor attached to a node which is NOT battery powered, you will not get anything out of it unless you configure a WAIT or SLEEP cycle. This is why it is important to have this split from the battery powered checkbox. Let me know if not clear I'll share a few examples
Thanks

[rakeshpai]

@user2684 Ah, that makes sense. I was looking for something like a 'reporting interval' in the NodeManager API. So if I understand correctly, I should setMode to WAIT and call setSleep to specify the reporting interval?

[user2684]

@rakeshpai or better directly setSleepMode(WAIT,10,MINUTES). All the sensors report at the end of the sleep/wait cycle. I'm adding new options in the new version for making this more flexible. Also, WAIT would not make sensors waiting for interrupts working correctly since the interrupt is handle by a call to the MySensors sleep() only. I'm fixing this as well. Long story short, I'd recommend to make have SLEEP by default and a fallback to WAIT if the use uncheck the box

[rakeshpai]

This post is deleted!

[rakeshpai]

I've got an update with a bunch of fixes. Thanks, @user2684!

• BATTERY_SENSOR now mirrors BATTERY_MANAGER, and is only applicable for battery powered devices. Doesn't show up on the gateway.
• PERSIST and SERVICE_MESSAGES set to 0.
• Fixes for code generation related to DHCP.
• UI fix - switching to Editor view for gateway doesn't mess up the UI.
• switch variable name changed to inputSwitch to avoid conflict with keywords.
• Tests.

@user2684 It looks like I'll wait for your next release before I close some of the other issues:

• setPowerPin() - I'm waiting for a release with this fix.
• If you are fixing how setSleepMode behaves, I'd rather wait for that to come through in a release as well.

No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush.

EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!

[user2684]

@rakeshpai I really hope my busy life would not impact your project preventing other users to leverage your fantastic tool But honestly could make sense to wait for v1.6 since a few things are changing. But please let me know if and when I'll delaying too much and we'll find a backup plan.

Thanks for your feedback regarding 1.5.1, I'll make it publicly available during the weekend.

[rakeshpai]

Need to bounce this off someone. Please share your thoughts.

I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!

The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.

Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.

There's two other alternatives I can think of:

1. Create a CLI tool that communicates with a browser. Users have to install and run the CLI app in a terminal, and while its running, they can flash their devices from the browser. This is the most minimal approach that solves the problem, but requires the user to juggle a terminal, which isn't ideal.
2. [Preferred] Create a native app, throw away the site, do everything inside the app. The native app can reuse most of the site's code, so the work put in so far is not wasted effort. The app also gets much more privileges, since it is running natively.

The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.

So, in summary, the options are:

1. Don't flash the device from the browser, and just offer the generated code for download, as is happening right now. Limits utility and future features, and frankly, isn't exciting.
2. Ask the user to install a native app. The app gets lots of capabilities this way, but has the downside of requiring a download and install, which is a massive cause of friction.

If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)

I'm confused about how to proceed. What would you do? What do you think is the best way ahead?

[Anticimex]

@rakeshpai if you go for a cloud based programming method, you also need to mind security. Users would like insurance that their nodes that happen to be hooked up to a pc do get exactly the firmware they ask for and only when they ask for it. Also that the firmware goes only in one direction.

[rakeshpai]

@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.

The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.

As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.

So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.

[core_c]

I am one of those few users that opened chrome://flags, and set:
Enable WebUSB support: disabled.
I already have installed a tool to program my Arduinos. I guess, it's the first thing most people install after aquiring an Arduino.
I also do not understand why i would ever use a cloud.
Perhaps i am "too oldskool".. using a browser to find what i search for, and (if it is code,) then download it.
When i want code, i do not care how i get it. Important is, that i can download it..
The bare product is what counts..

[Anticimex]

@rakeshpai just out of curiosity, how do you plan to handle code compilation? If we look beyond the problem of interfacing with a serial device, you also need to compile the sketch, and link it to a hex executable hex file. I'm on saying it can't be done (it has been done before) but I would like to understand how it is practically done. I am curious since I maintain the Jenkins instance for MySensors, and I am open for alternatives to compile things. It uses installed instances of the Arduino IDE on the server side.

[rakeshpai]

@Anticimex My current favourite is the PlatformIO Core CLI. While it was built to work in an IDE, it's actually completely independent, and only requires Python to run. It has support for lots of boards too, and has a built-in package manager making dependency management very easy. Since it's a CLI, it can easily be automated with bash scripts, can run headless, and doesn't require a windowing environment. Can't do that with GUIs like Arduino. There are alternatives to PlatformIO, but none that I found were feature-complete.

In fact, while I haven't exposed it, I've already implemented a way to download the code in a folder-structure that platformio prefers. I'll be rolling that out soon.

@core_c I completely understand the security issues of cloud compilation. But I also see the convenience it brings. If I do implement compilation in the cloud, I'm not going to take away the ability to download the code. You won't be forced to use the cloud. That said, I'm not going to implement cloud compilation at this moment, since it's pointless to do so if I can't also flash the device, and I currently can't flash the device from the browser.

[archiijs]

Hi, I can give a quick look from my (noob) perspective.
I'm no programmer, just tinkering around, new to Mysensors, but already managed to get something working and starting to learn how this stuff works so I can expand my installation.
Im wery interested in this firmware generator because of my lack of knowledge and understand that people with more knowledge probably will not use this tool for their installation as they are more comfortable with programming and Mysensors.

Intro aside I do not have any problems with using additional application, I already use Windows GUI/Controller for MySensors made by @tekka and Android application to debug and develop my installation.
Best part would be that code generation and debugging could be in one application, but that's not necessary.

Keep up the good work.

[rakeshpai]

@archiijs I'm very much in the same boat as you. While I have some experience writing code, C and C++ is all Greek and Latin to me. I can maybe read a little bit of C/C++ if my life depends on it, but that's about it. That's why I'm creating this tool.

Thanks for the feedback about app downloads being ok. I'll keep that in mind. Ideally, there won't be any need for debugging, since the generated code would be perfect right away, but I've been told that we don't live in an perfect world. However, viewing the serial monitor is definitely one of those things a desktop app can do, which a browser can't. I'm definitely excited about the possibility of communicating straight with the device.

It might take me some time to get to it though. However, all of the work I'm doing right now will be reused for the desktop app as well, so effort isn't wasted. At this moment, I guess it's wisest to see if this web app gets traction. Once we know it's being used, it'll be worthwhile making it better.

PS: How does the name 'MySensors Configurator' sound? Not the most creative, I know...

[getlarge]

@rakeshpai Hi, I'm following this topic closely, found your project on google to see if something similar already exists. I was surprised that it does + it's based on nodejs, couldn't be more happy!
It's really a good intention for people who wants to get started quickly and it would perfectly complete the NodeManager.

About the method, i was thinking about something similar that dagoma did ( a 3d printer manufacturer, based on Marlin firmware --> Arduino ). They made a subdomain where you can generate a .hex files, functions of the options you have. Then they made a desktop app which lets you upload the generated file.
That's not the most integrated and straight, but it works fine!
Which looks like your approch you show on your demo (btw the link on your github doesn't work, http://rakeshpai.github.io/mmysensors-network-manager )

That maybe a good start to flash an arduino with an app ? https://github.com/noopkat/avrgirl-arduino

If you need / want help, I'm actually working on a project with nodejs and react, so 'im in the good mood

[rakeshpai]

@getlarge Thanks! It's always nice to know that this is potentially useful.

Dogoma's approach is interesting, but as you said, not the most integrated. If there has to be an app downloaded anyway, why not go the next step and make it all easy to use.

Thanks for pointing out the issue with the readme. Fixed.

avrgirl is awesome! Finally someone decided to modernise a 20 year old piece of software.

You are welcome to dive into the code if you like. Node is only used for development. It's hosted on a static server, so there's no server-side language. The only problem with diving into code right now is that things aren't documented well, I haven't outlined what the goals/directions are, and it's kinda early, so I'm making sweeping changes all the time. But if you can get past that, feel free to dive in and familiarise yourself with the code, and even start making changes.

Speaking of which:
Update: No functionality change, but I've applied a fresh coat of paint to the UI. Hopefully it looks a little better now. There are still some cobwebs in the corners - I'll get to them soon.

[rakeshpai]

Since I haven't actually done this yet, I thought I'd outline the 'stack' used for this app, in case folks want to dive into the code.

• Based on React, using create-react-app.
• Offline first. Uses service-workers and localStorage for offline support.
• There's no server-side language. It's just a static site hosted on GitHub pages. Node.js is used for local development tooling.
• Uses redux for state management, and react-router for client-side routing.
• CSS-in-JS with glamor.

So, it's a very modern stack. Cutting-edge web stuff.

It's very early in the project, so I don't have any long-term plans outlined. In fact, I'll be honest, I haven't thought that far. But it's a community project, so you are welcome to steer the project as you see right.

Some areas to help:

• I suck at UI beautification work. Any help will be awesome.
• Need to investigate Electron for building native apps. Some things to look for: (a) Reusing the website directly from GitHub pages, so that we have a single deploy for the UI and the site still works fine if people don't want to download an app. Ideally, the electron app will simply be a container for the site. (b) Automatic updates to the container app. (c) Serial communication (node-serialport? avrgirl?) (d) More, as we go deeper.
• Ideas, suggestions? Like I said, I haven't thought too far.

Actually, the absolute best way to help right now is to simply poke around the UI and point out issues, no matter how minor they are. It's the little things and attention to detail in an app that make it nice to use, so it'll help to have an obsessive pair of eyes going over the UI. You can point out bugs, usability issues, things that might be confusing, errors in the generated code, or any other changes you think will improve the app.

You can comment here with suggestions, or use GitHub if you like.

[rakeshpai]

Just a quick update: I've been doing a bunch of work on the visual aspects of the app. Dare I say, it looks a lot better now. Check it out. Feedback welcome.

https://rakeshpai.github.io/mysensors-network-manager/

[hek]

Awesome @rakeshpai!. Not sure if it is a glitch in my chrome, but I wasn't able to download the code any longer.

[rakeshpai]

@hek Thanks. The download works for me just fine. Can you provide any more details? Do you see any errors in the browser's console? Also, hope you are using a desktop OS.

I haven't made much/any changes to the functionality, so downloads should be working as it was.

[hek]

Sorry, I had a old cached version in my browser.

[rakeshpai]

I just discovered this: https://github.com/arduino/arduino-create-agent

It's a project by the folks at Arduino, so it's as official as it gets. It's an app that you run locally, that runs in the background, puts an icon in the system tray, and starts a WebSocket server, with which a browser can connect to the local USB devices. It looks like it has been built for the Arduino Web Editor, and is in more-or-less active development.

I just spent under 15 mins with it, and I was able to talk to it from a CLI tool over websockets, and get it to list the available serial ports successfully. This is very encouraging.

Does anyone have any opinion about / experience with this? They have binaries for the three big desktop OSs. It's written in Go, so it's very clean in terms of deployment - it's just a single binary with no dependencies.

It looks like they enforce security by having a config.ini file with a list of allowed origins (domains) that are allowed to access the local web server. Since our page is running over HTTPS, the browser also imposes a restriction that it won't connect to insecure origins, which these chaps circumvent by creating a certificate locally during install (a process that I don't completely understand yet).

So, if someone is to use this, they'll have to install the app based on their platform, then modify config.ini to give access to this URL. That's relatively easy to do. It gives us access to everything that Arduino can do, but from a browser.

Most importantly, it's an official project by the folks at Arduino, so it can be trusted. Also, we don't have to replicate this work, and simply build on top of their APIs.

Thoughts?

[rakeshpai]

Not as rosy as I initially thought. I thought I might be able to compile the sketch locally using the app above. However, it looks like compilation is not a responsibility that this app has taken on. It only deals with listing serial devices, connecting to them, and flashing them, but not compiling sketches.

Arduino Web Editor appears to use arduino-builder, which I'm assuming they run in the cloud, since the setup isn't very easy at all. This means that we'll have to do cloud compilation as well. (@Anticimex, you might find arduino-builder interesting. They've specifically documented running arduino-builder in a CI system.)

It's still awesome that we have a solution to talk to the device from the browser. It's just that it'd have been even better if it did compilation as well. Hosting a cloud compilation service is not a huge problem - I run a reasonably high-traffic online service as my main gig, so I have some experience - but it'd have been better if we didn't have to.

I wonder why Arduino chose to compile in the cloud, considering that they ask their users to download a local app anyway to talk to the hardware. They wouldn't have to run a large part of their infrastructure, thus saving lots of associated costs and effort. And, they'd have a local toolchain, removing the need for an internet connection for compilation. They'd have to deal with setting up the toolchain locally, but they've been doing that for years, and have a lot of experience with it. Also, they would've enabled an ecosystem of apps like ours, so that we don't have to host compilers in the cloud. Maybe I'm not seeing some detail, but I think I'd have made the choice of having the compiler run locally too.

EDIT: I've raised an issue in their repo.

[Anticimex]

@rakeshpai we use the arduino-builder. It is just a shell command bundled with the IDE. You will still need to host a build environment, install all necessary libraries and BSP:s.

[rakeshpai]

tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.

I've hit a dead-end with Arduino Create Agent.

Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:

However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:

(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent

However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.

If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.

Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.

Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.

Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.

Onwards!

[rakeshpai]

I'll be stripping out the code I've written, so here's a screenshot for posterity. Serial comms in a browser!

[Anticimex]

@rakeshpai omfg that is a massive security cluster f*ck by the Italians. How do they ensure that their key is properly protected? From what you describe, they basically hand over shell access to attackers on a silver platter, signed or not. I do hope that this require you to install some special software on you machine to enable, and it is not something that comes bundled with the arduino IDE, right? Because if it is, it goes out of my machine...

[rakeshpai]

@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.

You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.

Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.

[Anticimex]

@rakeshpai the way they do it seem very in line with many of the design decisions taken in the IDE and the whole build process. You can just see how we include .cpp files in the library headers due to how their build system (or lack thereof) works...

[pjr]

Feature request: support for RS485 network!

[user2684]

@pjr I guess this is partially dependent on NodeManager as well, @rakeshpai I've added https://github.com/mysensors/NodeManager/issues/169 for this. Thanks

[rakeshpai]

@pjr Noted. Thanks. Are there a lot of people using RS485? (I have no idea, and I'm constantly surprised by how people use MySensors.) My only concern is that I don't want to add too many features that people won't use, since it would make the UI confusing. But if it's a popular feature, it's definitely on the to-do.

[pjr]

@user2684 wow, that was fast! Thanks!

@rakeshpai I can understand that. Not sure about the amount of rs485 users but one at least

[dbemowsk]

@rakeshpai I just took a quick look at this, and this will be an awesome tool. I didn't read through all 96 comments, so forgive me if my questions have already been asked and answered.

First, are all of the libraries for the included devices in the list going to be included in the zip file. These would be needed at compile time when uploading the firmware to the nodes.

Next, would there be a way to add sensors and their required libraries? If/when adding new devices/sensors, is there going to be something that defines the code structure for the device so when the code is being built it knows how to deal with it?

I am assuming that all of the sensors/devices in a network will be saved in a list for future reference and updating. Is there going to be any fields to add notes about a device and possibly a field where a location can be identified for a device such as "West wall of the great room" for a light switch or something like that.

Last, and this is more of a feature suggestion that would kind of tie into the last question, it would be nice if the user could upload floor plan images and be able to place the devices on the floor plan with an icon or something to be able to better identify where nodes are. A feature like this may be a bit of a stretch, but who would I be if I didn't at least suggest it.

Thanks for the contribution and keep up the good work.

[rakeshpai]

@dbemowsk Thanks! It's very encouraging to know that this is useful.

I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.

I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.

This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.

Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)

Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!

(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)

[dbemowsk]

@rakeshpai You mention about Dropbox, either that or Google Drive would be interesting options.

I know the floor plan thing is a stretch as it is a bit of coding to make that kind of thing work. I just figured I'd throw the hook out there and see who might bite on the worm.

I like the schematic thing too, but that, much like the floor plan thing might be hard to do.

[user2684]

@rakeshpai FYI I'm making some good progress with the new version of NodeManager (https://github.com/mysensors/NodeManager/milestones) thanks also to the many users contributing However, the list of things to do is still pretty long and last thing I want to do is to keep your amazing project on hold depending on mine. So whenever you are almost ready, just let me know so that I can push some of the less critical requests I have on the list to another release and hurry up to finalize the current one I'm working on. Whenever you will tell me "I'm ready", I expect at least a couple of weeks for me to finalize the current code and test the release before will be made available. Thanks

[rakeshpai]

@user2684 Thanks for the update. Much appreciated.

The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.

So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.

[user2684]

@rakeshpai fully agree with you. I'll start working in order to push those changes which are not a priority or too complex to implement to a future release in order to finalize v1.6 asap and give you the time to start working with something stable. Mean while I'll let you know once the development code will be good enough to start working on it. I can probably target a final v1.6 by end of August and a stable version in a couple of weeks if it fits your plans. I've also created a specific thread we can use to discuss about the integration (https://forum.mysensors.org/topic/7212/integrating-nodemanager-with-sketch-generator) so to keep this as clean as possible from implementation details. Thanks!

[rakeshpai]

Sorry for the long silence. it might look like I've been slacking off (and that's mostly true), but I've also been rolling out some updates:

• Code can now either be downloaded for the Arduino IDE or for the PlatformIO IDE. The PlatformIO code generates additional config files to use their dependency manager automatically. This means, you don't have to download and install libraries manually. Just open in PlatformIO and build the project. It's magical.
• Also, you can now download a backup zip of everything. It contains code for the entire network and all it's nodes, and a network.json file. I haven't implemented a 'restore' functionality yet (coming soon), but the network.json file contains everything needed for a restore. The network.json is the internal state data-structure used by the app.
• Couple of minor performance improvements, especially to help with the first-ever-visit scenario. (All subsequent user actions are blazing fast anyway.)

To clarify, while backups are a great idea, you don't need to save anything. Stuff is still automatically saved in the browser's storage anyway during regular use, and is persisted even if you close the browser. Backups are for the assurance of having a neat little zip containing everything on your hard drive, just in case things go horribly wrong.

Very little has changed visually. To find these new features, launch the app, and click the 'hamburger menu' at the top right of the page.

https://rakeshpai.github.io/mysensors-network-manager/

[rakeshpai]

Pretty big update: Added the ability to select the board that the node/gateway is based on. Supported boards are: Pro Mini (5v and 3.3v), Nano and Uno for the atmega328 boards, and NodeMCU and D1 Mini for the esp8266 boards. Please let me know if your favourite board isn't listed here.

The selection of the board effects the list of available pins for sensors. The sense pin dropdown for sensors now shows the appropriate pins (analog/digital/interrupt pins) based on the board selected for the node and the type of sensor being configured.

The selected board also affects the platformio.ini file generated, which makes PlatformIO integration even more enjoyable to use. No need to specify the board during the build.

This was a pretty big update, including changes to the underlying internal datastructure itself, so it's also an acid test for the data migration and versioning system implemented before. If you don't see any errors when you refresh after the update, it means everything worked transparently, and that's great news! (The migration mechanism hasn't been tested so far, so please let me know if you see errors when you get the update.)

Also implemented a reporting interval setting for analog sensors, to bring it in line with the latest changes in NodeManager.

We're getting closer to being release-ready!

EDIT: Also, this project crossed 100 commits in GitHub already! Wow!

[alowhum]

@rakeshpai , is this project still being developed? The Github page seems to indicate the last development was a year ago. Is there more development in secret? I really like it.

My dream solution for beginners usability would be a web interface where you can manage the network and the Arduino code. Users then connect an Arduino Nano or Micro:bit to the Raspberry Pi, and press the 'upload code' button for a specific device.

That way they don't need to install the Arduino IDE on their own laptops, but it can be incorporated into a smart home disk image instead.

*dreams on

[bjacobse]

@alowhum
If you take a look at the gitpage:
Quote: "mysensors-network-manager
A web-based tool to generate firmware for MySensors nodes.
It buggy, incomplete, it doesn't even work. Don't use this for anything important."

https://github.com/rakeshpai/mysensors-network-manager

[alowhum]

@bjacobse Yes I know, but I was hoping that the latest code hadn't reached Github. And that there could be some explanation of that claim. It seemed to work ok when I tried it, and the interface looks really good.

[rakeshpai]

@alowhum Sorry, I have abandoned this project. It didn't garner the interest I hoped it would, and I’ve moved on to other microcontrollers/radios/networks since then. This is hosted on GitHub’s free static hosting, so it should continue to work as long as they are in business, and it should work on devices as it was 5-6 years ago. But I won’t be supporting it and won’t continue to build on it. Glad to hear that you like it!