Browser-based firmware generator

[Anticimex]

@rakeshpai Great!
Repeaters are completely signing agnostic. If they receive a signed message not addressed to them, they just forward it as is. So they don't need to carry whitelist entries to validate messages that are not directed directly to them.

[Anticimex]

@rakeshpai Looks good with the ATSHA option. One thing to consider is that we have "official" HW that comes predefined (the SenseBender GW for examples). Currently, it seem your tool (which is incomplete, I know) only seem to support AVR boards from looking at the pin options. At some point, you could perhams have a HW device list to pick from to specify what kind of HW the gw or node execute on. And in the case of the SenseBender GW (there could be others) the ATSHA pin is already set by the Arduino environment and should not be overridden by user config. Consider this just FYI right now. The tool looks really promising for people uncomfortable with modifying code.

[rakeshpai]

Noted. Right now, the user will have to specify the ATSHA pin correctly for the SenseBender, so I guess it's not the end of the world, but I understand that it can be made easier.

There's actually a little bit of an understanding of the board already - if you choose an ESP8266 gateway and decide to add sensors to it, you'll see pin numbers matching the ESP8266's pinout. That's pin configuration in a different sense though. Let me think about how I can incorporate this.

Thanks for the kind words. :)

[Anticimex]

@rakeshpai ah, ok. But the list of pins does not really make sense for the SenseBender gw since it is based on SAMD and not AVR.

[rakeshpai]

Ah, didn't know that. Noted. I'll see how I can handle alternate boards.

[user2684]

@rakeshpai I personally think this is just amazing! When I wrote NodeManager I tried to make the world of MySensors just a step closed to a user who is not necessarily a programmer, this is doing the other 99 steps ;-) I'm sure there will be a lot of interest around this project

[rakeshpai]

Thanks, @user2684. @hek tells me that this is not the first attempt at making such a tool, but those attempts didn't end up succeeding. However, NodeManager might be changing that, making such a tool even possible. So, all compliments should be directed to you.
I'll need to bother you soon, because I'll need your help to validate the generated code, since it directly uses NodeManager. Still working on it - I'll keep you posted.

[user2684]

@rakeshpai sure, I'll be more than happy to double check any code generated for NodeManager! And I'll do a better job documenting future changes to the API since may have an impact on your project as well.

[rakeshpai]

Update: Not much has changed functionally, but I've now added a UI notification for when an update is available, so I don't have to tell you to do the double-refresh/hard-refresh ;) and people are always on the latest version.

I've also started writing some tests so that I can start to trust the generated code. I hope these tests are (at least somewhat) human-readable - they might be the easiest way to find out what the generated code is doing.

For eg. this file tests the NodeManager parts of config.h, and this file tests the sensor-specific code for NodeManager. The tests aren't exhaustive yet - still work in progress.

@user2684 Not sure if you'd prefer to read these tests. The alternative would be to change stuff on the UI, download zips, and verify the contents, which can get painful fast.

[user2684]

@rakeshpai I like the idea of the tests but I'm just afraid it will take more for you to write tests for all the possible combinations more than the time it took already to write the entire sketch generator :-) I'm wondering, is it something easy by chance having in console.log() the resulting code or the change performed upon any action for troubleshooting purposes? If it is too much effort, since it would be for myself only probably, let's explore a better alternative :-)

[core_c]

@rakeshpai The idea of creating a generator is really cool. There are many non-programmers out there, just wanting their sensor-network up and running.
I bet you finish this project, and make many people happy. Keep up the good work rakeshpai.
The awesome MySensors generator

[rakeshpai]

@core_c Thanks! That reminds me...

I need a name for this project, and I'm horrible at naming things. If some creative folks can chip in, that'd be awesome.

I've been referring to this as 'this', or app, or tool, or site, or things like that. None of them are good enough. A nice label would be great. Something catchy, maybe?

[hek]

MySensors Sketch Generator,
MySensors Sketchy, ;)
MyGenerator,
NodeManager Generator,
...

[core_c]

the word "Generator" scores. :)
i personally like "MyGenerator" most..

[rakeshpai]

Sorry, I've been offline for the last few days. Just catching up.

'Sketchy' is hilarious. Certainly seems to describe the current reliability of the output code. ;) I'll wait for more suggestions though.

[Anticimex]

MyFactory?

[rakeshpai]

Sorry for the silence. My ISP is having trouble, and I've been mostly offline. I'm sending this message over crappy 3G on a phone.

Update

1. New 'Code' view that shows what the generated code will look like before having to download it. I'm not sure I like this - I would prefer having no mention of code on the UI at all - but it's worth experimenting with. Certainly makes things easy at dev time.
2. A migrations process, so that I don't have to ask you to delete your existing network, as I have been doing. Migrations will now move your local copy of the data to the latest version automatically, without any manual intervention, if there have been any changes in the data format.

That's all I have for now. I'll have more once my connectivity issues are fixed.

[rakeshpai]

I've hit a little bit of a wall, and need help to proceed.

I created a NRF24L01 network in the UI, and tried to flash the default serial gateway code to an Arduino Pro Mini 3.3v/8Mhz. There are no sensors configured on the gateway. I tried to flash the SecurityPersonalizer on the device, and it worked perfectly, and gave a happy sounding serial output. (Yay!) However, when I flash the main gateway sketch, I get a 'not enough memory' error.

Sketch uses 25,792 bytes (79%) of program storage space. Maximum is 32,256 bytes.
Global variables use 2,204 bytes (107%) of dynamic memory, leaving -156 bytes for local variables. Maximum is 2,048 bytes.
processing.app.debug.RunnerException: Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.
	at cc.arduino.Compiler.size(Compiler.java:339)
	at cc.arduino.Compiler.build(Compiler.java:159)
	at processing.app.SketchController.build(SketchController.java:641)
	at processing.app.Editor$BuildHandler.run(Editor.java:1782)
	at java.lang.Thread.run(Thread.java:745)
Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.

For reference, the generated gateway sketch is as follows:

// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES

// load user settings
#include "config.h"
// load MySensors library
#include <MySensors.h>
// load NodeManager library
#include "NodeManager.h"

// create a NodeManager instance
NodeManager nm;

// before
void before() {
  // setup the serial port baud rate
  Serial.begin(9600);

  

  nm.before();
}

// presentation
void presentation() {
  // call NodeManager presentation routine
  nm.presentation();
}

// setup
void setup() {
  // call NodeManager setup routine
  nm.setup();
}

// loop
void loop() {
  // call NodeManager loop routine
  nm.loop();
}

// receive
void receive(const MyMessage &message) {
  // call NodeManager receive routine
  nm.receive(message);
}

// receiveTime
void receiveTime(unsigned long ts) {
  // call NodeManager receiveTime routine
  nm.receiveTime(ts);
}

and the associated config.h is:

#ifndef config_h
#define config_h

/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
//#define MY_REPEATER_FEATURE


/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG


// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24


// Serial gateway settings
#define MY_GATEWAY_SERIAL


/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
//#define DEBUG 1

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 1
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 1

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0

#endif

I tried commenting out the first three lines in the gateway sketch (the #defines to do with security personalisation, and the sketch fits, but only barely.

Sketch uses 19,922 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,710 bytes (83%) of dynamic memory, leaving 338 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.

I'm by no means a C/C++ guy (which I why I'm writing this tool ;) ) so I'll need help to fix this. I don't even know where to start looking, since there's the combination of NodeManager and the software signing in the sketch.

Any chance you could look at this, @user2684 and @Anticimex? You could download a copy of the code from the UI if you like. My configuration is an NRF network, and the gateway is the default serial gateway on the UI. I've been using the latest version of the MySensors lib from the development branch on GitHub, and am doing the compilation in the Arduino IDE.

Thanks!

[Anticimex]

@rakeshpai I have never used node manager myself but it appears that it consumes quite some memory. I have to my best effort optimized the signing code to use as little memory as possible. That said, it could very well be possible to optimize it further, but I would then need other eyes on it. I have not looked into node manager, but as I understand it, it relies heavily on c++ (signing code is pretty much just c) and the atmega compiler optimized c++ code horribly bad. So I think that it is in nodemanager the biggest memory savings can be made.

[rakeshpai]

@Anticimex That's the feeling I had as well, considering that even after I disabled signing, while the sketch fits in memory, it still seemed to complain about low memory. @user2684 does this match your experience too?