Integrating NodeManager with Sketch Generator

[user2684]

This thread is intended to hold any discussion for the integration between Nodemanager and https://forum.mysensors.org/topic/6980/browser-based-firmware-generator

[user2684]

@rakeshpai I think once https://github.com/mysensors/NodeManager/pull/177 will be done, you can start working with it. I'm completely changing the default behavior the board operates and I hope this would be more intuitive. There will be only 2 modes instead of 4 and reporting and sleeping will be completely decoupled since it was not very intuitive before.

[user2684]

Hi @rakeshpai, I've finally merged the big update I mentioned in my previous post and postponed to v1.7 a few things I had on the list so to speed up the development and having v1.6 out asap. There will be no major changes left (apart from adding a few other sensors) so I think you can start from the current development branch https://github.com/mysensors/NodeManager/tree/development. With https://github.com/mysensors/NodeManager/pull/177 I've introduced a completely different way to define reporting intervals and sleeping cycles, more details in the PR and in the updated documentation. Of course feel free to let me know in case of any doubt.

Something else you would need to evaluate is what explained at https://github.com/mysensors/NodeManager/issues/176. This has always been in this way but not sure if you took it into consideration.

Also, regarding your screen for configuring every sensor, I was wondering if you want to somehow apply a logic similar to mine. I mean, for every sensor in NM you can call a member function of the Sensor class or one from the subclass. E.g. for the thermistor you can set the number of samples (like every sensor since setSamples() is a member of Sensor or a thermistor class specific function. This could for example translate into two tabs, one common for all the sensors and another sensor specific; which has no impact in the code since the object to use is still the same but would probably help you in the configuration screen.
Thanks

[rakeshpai]

Sorry for the silence. I didn't realise I wasn't getting notifications about this thread. Thanks for the updates. I'll go over your comments and issues and start making changes accordingly. I'll update here once I have something. Thanks again. Woohoo!

[rakeshpai]

Just want to be sure I understand the changes, so please bear with me.

When a node is awake, each sensor can have its individual reporting times (or a global reporting time), which need not have any relation to anything else. However, if a node is sleeping, its reporting time has to be a multiple of the sleep interval of the node. Is this understanding correct?

If so, I'm thinking of the following UI:

• If a node is not battery powered, the UI can let you set the reporting interval on individual sensors. Even though setting reporting interval on every sensor gets repetitive, it might be the easiest to use. The alternative would be to have a shared global setting influencing the reporting interval, but that gets tricky to convey in the UI.
• If a node is battery powered, the user will have to set a sleep interval. We could then use one of two approaches: (1) The user can't set the reporting interval for sensors at all, in which case the sleep interval is used as the reporting interval by default, or (2) We can let the user set a reporting interval, but only if it's a multiple of the sleep interval. Either one works for me, the second seems most flexible.

Is my understanding correct, and would this kind of a UI implementation do the job?

[user2684]

@rakeshpai correct (almost). There is a global setReportInterval which will set the reporting interval for all the sensors. If setReportInterval of the sensor is called, the global setting of course will not apply. All of this has been done to allow potential per-sensor reporting intervals and to have a consistent behavior, regardless if the node is sleeping or not.
If battery powered, the logic is still the same, of course the condition (is it the time to report?) in this case is evaluated when the node is awake next. So yes, in this case better if the reporting interval is a multiple of the sleep time but not necessarily. The end result of course would be the same.

In terms of UI, I'd probably add a global reporting interval input (and if you prefer with a drop down menu with seconds/minutes/hours so to call the right function) together with global optional sleep interval and an optional per-sensor reporting interval. I'd probably not enforce the "multiple of" thing to keep it simple on your side. Users may guess if the node is sleeping it will not report.

Also, do you think all of this NodeManager's restructuring makes the logic more clear to understand? I felt like reporting by default at every sleep cycle was not intuitive and confusing, hope now looks better.
Thanks!

[rakeshpai]

@user2684 The clear separation between sleeping and reporting definitely makes things easier to understand and use. Thanks for doing this.

I've now published a release with the changes for sleep time and reporting time in place. Could you please give it a spin? https://rakeshpai.github.io/mysensors-network-manager/ You may need to delete your existing sensors first.

• Sleep config is as before. Nothing's changed there.
• New reporting interval fields added. Only applies to analog sensors ATM.
• There's additional messaging for the reporting interval fields to clarify what happens when the node is sleeping.
• The generated code has been updated.

I haven't implemented a global reporting time. I felt it would make the UI complicated. I might be mistaken, but it isn't relevant for all types of sensors, isn't it? For eg. there's not much to sense by polling a motion sensor, or a switch. So, I felt like it's best if we ask for it only where relevant, and not out of context. I'm open to change this though. Please let me know what you think.

[rakeshpai]

I'm trying to compile a serial gateway, no sensors, against the development branch of NodeManager, which works fine. However, when I open the serial monitor, I get the following output:

$ pio device monitor --port /dev/ttyUSB2
--- Miniterm on /dev/ttyUSB2  9600,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
 __  __       ____
|  \/  |_   _/ ___|  ___ _ __  ___  ___  _ __ ___
| |\/| | | | \___ \ / _ \ `_ \/ __|/ _ \| `__/ __|
| |  | | |_| |___| |  __/ | | \__ \  _  | |  \__ \
|_|  |_|\__, |____/ \___|_| |_|___/��

...and it hangs.

Any idea what the issue might be?

PS: As a self-confessed connoisseur of ASCII-art, I tip my hat in your general direction. Well done, good sir.

[rakeshpai]

Sorry for the lack of detail. You can get the latest gateway sketch from the app itself at https://rakeshpai.github.io/mysensors-network-manager/

The sketch is just a barebones sketch, wiring up the before, presentation, etc. functions to node manager. The config.h is as follows, though nothing stands out to me as being out of place.

#ifndef config_h
#define config_h

// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES


/**********************************
 * Sketch configuration
 */

#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
#define MY_REPEATER_FEATURE


/**********************************
 * MySensors node configuration
 */

// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG


// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24


// Serial gateway settings
#define MY_GATEWAY_SERIAL


/***********************************
 * NodeManager configuration
 */

// if enabled, enable debug messages on serial port
//#define DEBUG 1

#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 0
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 0
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 0

// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0

#endif

[user2684]

That's correct, for a motion sensor or a switch, reporting inteval does not make sense (even if technically the loop() function is called every 10 minutes which is the default reporting interval if a custom one is not provided but this does not make any difference). I'd keep per-sensor reporting inteval as you suggest, it seems easier for a new user to me as well.

• Regarding that amazing piece of ascii art, unfortunately it is not coming from me No idea where it comes from! I'll look into this gateway issue (https://github.com/mysensors/NodeManager/issues/187)
• When opening https://rakeshpai.github.io/mysensors-network-manager/ after clicking on the "there is an update please click here" message, I have now a "Just a moment..." message fixed on the screen and I couldn't recover the app even after clearing the cache of my browser. In an incognito window of course works just fine
• I'd probably add a global setting (maybe in advanced) for enabling MySensors and/or NodeManager debug since this may be a common task even of a new user when requested
• Having soft signing defined in config.h is in my backlog as well and will be added soon
• We probably need a way to select which MySensors API version we are using. MY_RFM69_NEW_DRIVER is only available in v2.2-beta and the old RF69_868MHZ will become RFM69_868MHZ which may cause compilation errors if the wrong library version is installed
• Are you sure MY_RF69_IRQ_PIN, MY_RF69_IRQ_NUM and MY_RF69_SPI_CS have to be defined with a pro mini board? I thought they were only to be set when using a ESP8266
• You may want to use the updated config.h which now includes a few more common directives. Even if they are commented out can be useful for a user to have those example available
• When adding a PIR sensor, the pin in the code is always 0
• You can add automatically in config.h a #define MAX_SENSORS 10 where the number is automatically calculated based on the number of sensors configured. This will help in saving storage if defining just a few sensors or allowing multiple sensors to be allocated
• We can probably start adding all the other built-in sensors left out so far, now that the available functions are not subject of big changes. Let me know if you need any help with it

Thanks and once again excellent job!

[user2684]

PS. I think the ascii art is coming from https://github.com/mysensors/MySensors/blob/development/core/MySplashScreen.cpp

[user2684]

@rakeshpai I cannot reproduce the issue, with your config.h looks like it is working fine here

NodeManager v1.6-dev
LIB R=N E=X T=R A=A S=S B=-
INT P=3 M=255
INT P=2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY

MY I=0 M=1

[rakeshpai]

@user2684 Woops, if you only see the loading message, I've dun goofed. Can you please look in your JS console (Ctrl+Shift+I) and tell me what is the error you see?

To fix the problem, you can delete your existing data. You can do that within the devtools itself. Click on the Application tab in the devtools, and in the left pane, under Storage, select Localstorage > https://rakeshpai.github.io, and delete the key called 'app-data'. Please make sure you copy your error messages before you do this though, so that I know what went wrong.

[user2684]

@rakeshpai You're right! there was a js error:

TypeError: Cannot read property 'pinType' of undefined
    at https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238265
    at Array.map (native)
    at https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238221
    at Array.map (native)
    at o (https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238079)
    at https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238546
    at Array.reduce (<anonymous>)
    at i (https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238518)
    at Array.map (native)
    at a (https://rakeshpai.github.io/mysensors-network-manager/static/js/main.1fdf5f27.js:1:238659)

[rakeshpai]

@user2684 Thanks. If you haven't already deleted the app_data key, could you please share the value of that key with me? If you've already deleted it, do you remember which sensors you had configured across your nodes in your settings?

This was an error in the migration process, where the old data is ported over to the new format, if I make changes in the format of the data. I've obviously done something wrong in that porting logic.

[user2684]

@rakeshpai sure here it is:

{"networks":[{"radio":"NRF24L01+","nrfChannel":76,"rfmFrequency":868,"id":"3yu5c1pv","nodes":[{"name":"Gateway","type":"gateway","key":"554766e4d3b69590e5","pa":true,"hw":false,"battery":{"powered":false,"min":2.5,"max":3.3,"measure":"internal","measurePin":"A0","voltsPerBit":0.003363075},"sleepTime":60,"sleepUnit":"minutes","sensors":[],"signing":"software","softSigningPin":"A7","atshaSigningPin":"A3","id":"vv4cn7l3","gatewayType":"ethernet","ethernet":{"dhcp":true,"module":"w5100","ip":"192.168.1.10","gateway":"192.168.1.1","subnet":"255.255.255.0"},"wifi":{"ssid":"adsad","password":"asdsad"},"conn":{"type":"client","serverPort":5003,"serverMaxClients":1,"controllerIp":"1111","controllerPort":5003,"mqttHost":"","mqttPort":1883}},{"id":"anzgjjrk","name":"MyNode","type":"node","key":"22cb1534bdec90d3bf","pa":true,"hw":false,"battery":{"powered":true,"min":2.5,"max":3.3,"measure":"internal","measurePin":"A0","voltsPerBit":0.003363075},"sleepTime":60,"sleepUnit":"minutes","sensors":[{"type":"ldr","pin":"A0","usePowerPin":true,"reportPercentage":true,"reverse":true,"powerPin":"D9","percentageRangeMin":0,"percentageRangeMax":1024},{"type":"rain","pin":"A1","usePowerPin":true,"reportPercentage":true,"reverse":true,"powerPin":"D0","percentageRangeMin":0,"percentageRangeMax":1024},{"type":"relay","pin":"D0","onValue":"high","initialValue":"low","autoTurnOff":true,"turnOffTime":1},{"type":"motion","pin":"D1","interruptMode":"rising"},{"type":"switch","pin":"D2","interruptMode":"change","debounceTime":30,"normalValue":"high"},{"type":"latchingRelay","pin":"D3","onValue":"high","initialValue":"low","autoTurnOff":false,"turnOffTime":1,"pulseOutput":false,"pulseDuration":100}],"signing":"software","softSigningPin":"A7","atshaSigningPin":"A3"}],"hmac":"e7f21811523c4e8e463843367bf5f415c9c8cf776eebf4e1fbfd7ae3135fcbc8","aes":"b07372524a15001e092200550200a15b"},{"radio":"NRF24L01+","nrfChannel":76,"rfmFrequency":868,"id":"fi253u7o","nodes":[{"name":"Gateway","type":"gateway","key":"f98623fd783becc370","pa":false,"hw":false,"battery":{"powered":false,"min":2.5,"max":3.3,"measure":"internal","measurePin":"A0","voltsPerBit":0.003363075},"sleepTime":60,"sleepUnit":"minutes","sensors":[],"signing":"software","softSigningPin":"A7","atshaSigningPin":"A3","id":"015okgk","gatewayType":"serial","ethernet":{"dhcp":true,"module":"w5100","ip":"192.168.1.10","gateway":"192.168.1.1","subnet":"255.255.255.0"},"wifi":{"ssid":"","password":""},"conn":{"type":"server","serverPort":5003,"serverMaxClients":1,"controllerIp":"","controllerPort":5003,"mqttHost":"","mqttPort":1883}},{"id":"v6301ydk","name":"MyNode","type":"node","key":"5785ead033fe668a9e","pa":false,"hw":false,"battery":{"powered":false,"min":2.5,"max":3.3,"measure":"internal","measurePin":"A0","voltsPerBit":0.003363075},"sleepTime":60,"sleepUnit":"minutes","sensors":[{"type":"relay","pin":"D0","onValue":"high","initialValue":"low","autoTurnOff":false,"turnOffTime":1}],"signing":"software","softSigningPin":"A7","atshaSigningPin":"A3"}],"hmac":"1108c29e98d1274aa5a4e0a0c23de71053584fbda39ffe68a3c1477415fd9aab","aes":"3c44673691a361ae3f22c15f50dce5b9"}],"version":1}

[rakeshpai]

@user2684 Thanks. That helped identify the problem. I've changed the key switch to inputSwitch to avoid clashes with C keywords, which is causing the issue. I had done this long ago, when I wasn't caring about data migrations. Looks like your data still has that old bit in it.

Sorry for the trouble. The easiest thing to do is to delete that value, and start afresh. I'll be more diligent about making modifications in the future.

[user2684]

@rakeshpai thanks, not a big issue, I'm just running a few tests so I'm not losing anything significant by deleting that key

[rakeshpai]

@user2684 We will need to decide which version of MySensors to use. I've been using the latest version from the development branch. In fact, the ascii-art splash screen is the latest commit, as of 5 days ago. Since you aren't seeing the ascii-art, you are probably on a slightly older commit.

I'm not sure if we can start supporting from the current stable 2.1.1 onwards? I don't know about the advantages that the new driver for RFM69 brings. If it's any effort at all to get it to work with older versions, it's probably not worth it. I don't mind waiting till 2.2 is released.

I've simply copy-pasted the MY_RFM_* defines from somewhere in the MySensors examples. They are currently hardcoded and probably are just the defaults. I'll clean that up.

Thanks for pointing out the other issues. I'll get on it. Thanks!

[user2684]

@rakeshpai I think you're right, making it compatible with too many versions can become challenging. The only remarkable difference between 2.1.1 and 2.2.0 is the RF69_868MHZ turned into RFM69_868MHZ. MY_RFM69_NEW_DRIVER if set in 2.1.1 has no impact and the radio signal level child id I've just introduced is not enabled unless MY_SIGNAL_REPORT_ENABLED is defined (only in 2.2.0). So the issue apparently is only on RF69_868MHZ which is probably minor since the user can add or delete a single letter. So we can definitely target 2.1.1.
Just a note on the MAX_SENSORS: what I wrote before is wrong, MAX_SENSORS defines the maximum number of child ids, not the maximum number of sensors so if you want to set it up, you need to take into account for each sensor how many child ids are created (which is static but to take into account). Thanks!

[rakeshpai]

MAX_SENSORS defines the maximum number of child ids

How can I know how many child ids exist in total? Is there a way to arrive at it from the sensors? Sorry for my ignorance.

Fixed most of the other bugs you pointed out. Thanks!

[user2684]

@rakeshpai not your fault, it is just not easy to see When you register a sensor this could create one or more child IDs. How many, depends on the sensor. If you have a look at https://github.com/mysensors/NodeManager/blob/development/NodeManager.cpp#L3139 you can see for each sensor_type how many. As a rule of thumb, add 1 every time _getAvailableChildId() is called. If you need a summary table I can make it for you.
Defining MAX_SENSORS is not mandatory but since we may risk to have precious storage wasted, better to address this at the first run I think

[user2684]

Hi @rakeshpai, I'm finalizing v1.6 and I was wondering if you had any other issue with NM along the way so I can fix it timely. Also, are you still experiencing the gateway issue (https://github.com/mysensors/NodeManager/issues/187) which I couldn't reproduce? Thanks!

[rakeshpai]

@user2684 Sorry, I'm not at a computer. IIRC, the issue I've been facing is that when I used the latest version of NodeManager with the latest version of MySensors dev branch, with signing and encryption, the gateway would crash on startup. This has been preventing me from proceeding.

Sorry for the lack of detail - I can't verify things at the moment. Looking forward to 1.6!

[user2684]

@rakeshpai no hurry thanks! The gateway issue sounds like a show stopper to me though but maybe I've been able to reproduce it: with both encryption AND signing enabled I got 94% of storage utilization on a pro mini hence causing the crash. If I eg disable debug so saving some storage I get no issues. Whenever you can this is probably something you can test: if this is the issue we would need to find something else to disable in order to have the gw running fine.

Also I'd recommend to disable repeater mode by default otherwise I guess you can get some issues if every node is acting as a repeater on the network

[ru64]

@rakeshpai I've got the same problem. Occasionally found out that this reproduces if DEBUG is off (#define DEBUG 0). As soon as I turned it on - it works.