💬 NodeManager
-
@mvader for question #1, I've added it to the existing issue https://github.com/mysensors/NodeManager/issues/221. For question #2, for sensors using i2c the pin is not required despite within the code nodemanager will set a random pin (have a look at SensorSHT21 for example)
@user2684 said in 💬 NodeManager:
@mvader for question #1, I've added it to the existing issue https://github.com/mysensors/NodeManager/issues/221. For question #2, for sensors using i2c the pin is not required despite within the code nodemanager will set a random pin (have a look at SensorSHT21 for example)
great.. thank you!
-
After a few months working behind the scene, NodeManager got kind of a full review in terms of architecture and usability. The code should be now simpler to understand for new users, the memory utilization has been optimized and the overall user experience has been improved.
The looooong list of things that have been changed are detailed on https://github.com/mysensors/NodeManager/pull/229.
Please note this is only available in the development branch (https://github.com/mysensors/NodeManager/tree/development) and will be generally available only when NodeManager's v1.7 will be released (it will take I guess still a few months).Since this change was blocking other pending changes, I hope now the development will be faster and feel free to submit your PRs now that the new architecture has been finalized.
As always, any feedback is more than welcome.
Thanks -
@user2684 Thanks a lot for the architecture change, which makes a lot of sense conceptually.
Unfortunately, it appears to use more memory than before. I'm working on an air quality board with eight MQ sensors, an MH-Z19 CO2 sensor and a Plantower PMSA003 particulate matter sensor. In previous development versions (1.7-dev, before today), the eight MQ sensors and the MH-Z19 worked fine, but now after your big merge of #229 the sketch appears to run out of memory after adding the child of the sixth MQ sensor. Do you see any chance to optimize for memory?
Old sketch (using a previous development version of NodeManager, without the PMSA003; 155 bytes left after all nine sensors are fully initialized): https://github.com/open-tools/NodeManager_GasSensor/tree/master/NodeManager_GasSensor
New sketch (using the latest version of NodeManager, plus a PMSA003): https://github.com/kainhofer/NodeManager/tree/GasSensor
That sketch runs out of memory with the sixth MQ sensor (I added debug output with free memory at a lot of spots). Of course, I have commented out the PMSA003 completely to have a correct comparison.FWIW, the sketch is for this board: https://github.com/open-tools/NodeManager_GasSensor/blob/master/images/IMAG2267.jpg
-
@user2684 Thanks a lot for the architecture change, which makes a lot of sense conceptually.
Unfortunately, it appears to use more memory than before. I'm working on an air quality board with eight MQ sensors, an MH-Z19 CO2 sensor and a Plantower PMSA003 particulate matter sensor. In previous development versions (1.7-dev, before today), the eight MQ sensors and the MH-Z19 worked fine, but now after your big merge of #229 the sketch appears to run out of memory after adding the child of the sixth MQ sensor. Do you see any chance to optimize for memory?
Old sketch (using a previous development version of NodeManager, without the PMSA003; 155 bytes left after all nine sensors are fully initialized): https://github.com/open-tools/NodeManager_GasSensor/tree/master/NodeManager_GasSensor
New sketch (using the latest version of NodeManager, plus a PMSA003): https://github.com/kainhofer/NodeManager/tree/GasSensor
That sketch runs out of memory with the sixth MQ sensor (I added debug output with free memory at a lot of spots). Of course, I have commented out the PMSA003 completely to have a correct comparison.FWIW, the sketch is for this board: https://github.com/open-tools/NodeManager_GasSensor/blob/master/images/IMAG2267.jpg
@reinhold if you have high memory requirement, what about switching to a more modern architecture like stm32f103c8t6 on the famous "blue pill" boards ? It has 20k or RAM (10 times what you have on Arduino) and 64k of flash and it's supported by MySensors, in addition to be very cheap (2$/board). You will need voltage converter between 3.3V of the board and 5V of your sensors but that's not a big deal in exchange of the possibility to expand your board with more sensors in the future.
-
@user2684 Thanks a lot for the architecture change, which makes a lot of sense conceptually.
Unfortunately, it appears to use more memory than before. I'm working on an air quality board with eight MQ sensors, an MH-Z19 CO2 sensor and a Plantower PMSA003 particulate matter sensor. In previous development versions (1.7-dev, before today), the eight MQ sensors and the MH-Z19 worked fine, but now after your big merge of #229 the sketch appears to run out of memory after adding the child of the sixth MQ sensor. Do you see any chance to optimize for memory?
Old sketch (using a previous development version of NodeManager, without the PMSA003; 155 bytes left after all nine sensors are fully initialized): https://github.com/open-tools/NodeManager_GasSensor/tree/master/NodeManager_GasSensor
New sketch (using the latest version of NodeManager, plus a PMSA003): https://github.com/kainhofer/NodeManager/tree/GasSensor
That sketch runs out of memory with the sixth MQ sensor (I added debug output with free memory at a lot of spots). Of course, I have commented out the PMSA003 completely to have a correct comparison.FWIW, the sketch is for this board: https://github.com/open-tools/NodeManager_GasSensor/blob/master/images/IMAG2267.jpg
-
@reinhold if you have high memory requirement, what about switching to a more modern architecture like stm32f103c8t6 on the famous "blue pill" boards ? It has 20k or RAM (10 times what you have on Arduino) and 64k of flash and it's supported by MySensors, in addition to be very cheap (2$/board). You will need voltage converter between 3.3V of the board and 5V of your sensors but that's not a big deal in exchange of the possibility to expand your board with more sensors in the future.
@nca78 Thanks for the suggestion to use the blue pill STM32 boards. I haven't thought of it (I only looked at the ESP8266, but that has only one ADC). I have a few lying around here, but never got around to trying them, as I first need to burn the bootloader. It looks quite promising, but the boards are huge, so I'll need to enlarge my PCB...
The need for a 5V->3.3V regulator is no issue, as the NRF24 already needs 3.3V. The 5V analog level of the MQ sensors will need some converter (the analog ADC inputs are marked as non-5V-tolerant...), but then in turn I can get rid of the voltage dividers for the PMSA003. The larger issue seems to be the connection of the nrf24, which takes away 4 ADC pins, so there are only 6 ADC pins left, while I have 8 analog sensors :(Still, for now I have the PCBs with the ProMini, so I'll try to strip down the Sensor or SensorMQ classes for my own use.
-
@nca78 Two more things popped up with the latest development version of NodeManager:
- In NodeManager.ino: function void before(): The sample code to set the child id of a child is sht.children.get(1)->child_id = 5, but that does not work, because at that point the children are not yet even generated. They are created by the node.before(); call at the very end of the function... So, currently this will not assign any new child ID and the default IDs will be used. I have not checked in detail whether calling node.before() at the beginning of the before() function and then assigning child_id to the children will solve the issue. From a quick look at the code, it might work...
- I'm trying to implement an OLED display, where the node would wake up every 5 seconds, take a measurement and display it on the OLED. But the report interval to the MySensors gateway should only be 10 minutes. I thought that
node.setSleepSeconds(5); node.setReportIntervalMinutes(10);would be the correct way to setup NodeManager to take measurements every 5 seconds, but only report it every 10 minutes to the network. Unfortunately, Sensor::loop(MyMessage* message) seems to immediately return without measurement if the reporting timer has not elapsed. Somehow I don't understand that logic. Why have different sleep and reporting intervals when measurements will only be taken at reporting intervals anyway? I would expect measurements to be taken with the sleep intervals (e.g. to calculate an average over the whole reporting interval) and report back to the network using report intervals.
-
@user2684 Thanks a lot for the architecture change, which makes a lot of sense conceptually.
Unfortunately, it appears to use more memory than before. I'm working on an air quality board with eight MQ sensors, an MH-Z19 CO2 sensor and a Plantower PMSA003 particulate matter sensor. In previous development versions (1.7-dev, before today), the eight MQ sensors and the MH-Z19 worked fine, but now after your big merge of #229 the sketch appears to run out of memory after adding the child of the sixth MQ sensor. Do you see any chance to optimize for memory?
Old sketch (using a previous development version of NodeManager, without the PMSA003; 155 bytes left after all nine sensors are fully initialized): https://github.com/open-tools/NodeManager_GasSensor/tree/master/NodeManager_GasSensor
New sketch (using the latest version of NodeManager, plus a PMSA003): https://github.com/kainhofer/NodeManager/tree/GasSensor
That sketch runs out of memory with the sixth MQ sensor (I added debug output with free memory at a lot of spots). Of course, I have commented out the PMSA003 completely to have a correct comparison.FWIW, the sketch is for this board: https://github.com/open-tools/NodeManager_GasSensor/blob/master/images/IMAG2267.jpg
@reinhold thanks for reporting this issue. One of the main objectives of the new architecture was to reduce the memory footprint so if this is not the case, there is definitely something to fix. I've noticed in your sketch you are using the remote configuration sensor, that one is huge in terms of memory, if not strictly required I'd comment it out (BTW very nice project!). In PR #229 I've also uploaded a spreadsheet with all tests I run for realizing what was using the memory the most. Anyway, there is still a lot of room for improvement looks like :)
Thanks @mfalkvidd , I'll give the directives you pointed out a try! -
@nca78 Two more things popped up with the latest development version of NodeManager:
- In NodeManager.ino: function void before(): The sample code to set the child id of a child is sht.children.get(1)->child_id = 5, but that does not work, because at that point the children are not yet even generated. They are created by the node.before(); call at the very end of the function... So, currently this will not assign any new child ID and the default IDs will be used. I have not checked in detail whether calling node.before() at the beginning of the before() function and then assigning child_id to the children will solve the issue. From a quick look at the code, it might work...
- I'm trying to implement an OLED display, where the node would wake up every 5 seconds, take a measurement and display it on the OLED. But the report interval to the MySensors gateway should only be 10 minutes. I thought that
node.setSleepSeconds(5); node.setReportIntervalMinutes(10);would be the correct way to setup NodeManager to take measurements every 5 seconds, but only report it every 10 minutes to the network. Unfortunately, Sensor::loop(MyMessage* message) seems to immediately return without measurement if the reporting timer has not elapsed. Somehow I don't understand that logic. Why have different sleep and reporting intervals when measurements will only be taken at reporting intervals anyway? I would expect measurements to be taken with the sleep intervals (e.g. to calculate an average over the whole reporting interval) and report back to the network using report intervals.
@reinhold you're right, assigning the child_id before calling node.before() does not work. The way to go would be to have all the methods being called before node.before() and the child_id assigned just after. Not very intuitive, I probably need to find a completely different and a better way to do so, probably by creating the Child in the constructor instead (reopening https://github.com/mysensors/NodeManager/issues/198 for this).
Regarding setSleep and setReportInterval, measurement are driven only by setReportInterval. Those are different because there are situations (e.g. relays, motion sensors) when you have nothing to report but you want the capture the heartbeat from sleep(). What you are looking for is something I never considered which is actually a pretty nice idea (taking measurements over a given time period and than reporting at the end). I've created https://github.com/mysensors/NodeManager/issues/243 for this. With the new architecture, since keeping already track of the average within Child, should not be so difficult even if would probably require an additional Timer. Thanks!
-
@user2684 Thanks for the spreadsheet with your memory footprints. Are these the sketch size and the free memory that the compiler reports? Or do you measure them when running the sketch (i.e. including all dynamically allocated variables in SRAM)?
Also, one problem you don't seem to have covered is heap fragmentation. Particularly your new List class seems to be prone to heap fragmentation with a large number of sensors: each time a sensor is added, the list needs to re-allocate the memory for its pointers. The old memory will be released, but it will remain as a hole in the memory, where new variables might be allocated. However, the next time you add an element to the list, the new pointer array can probably not re-use some of those freed holes.
I'll provide a PR (https://github.com/mysensors/NodeManager/pull/247) that adds an allocateBlocks(count) method to the List, which can be used in the sketch and in the sensor's onSetup methods to pre-allocate the required sized. In turn, the _preAllocBlocks member does currently not serve any purpose, and I don't think it's really required functionality. In my eyes, it's more important to ensure that at a certain moment the required memory is allocated. It is not neccessary to keep a minimum nuber of blocks allocated at all times.
For a node with 10 physical sensors (and maybe one or more built-in sensors), this will save at least 22 bytes (1,1% of total SRAM!) and probably reduce heap fragmentation by quite a bit.Another are where >22 bytes can be saved is the power manager. None of my nodes makes use of the power manager, yet each sensor needs to keep a (null) pointer to a power manager. Providing a compiler #define NO_MODULE_POWER_MANAGER can exclude everything PowerManager-related from compilation and save one pointer per sensor and node, i.e. with 10 sensors you'll save another 22 bytes (1,1% of total SRAM!) I'll provide a PR for this, too: https://github.com/mysensors/NodeManager/pull/248
Apart from that, I simply stripped down the SensorMQ class and removed all configurability for my own use (don't worry, I won't submit this as a pull request... it is simply meant to squeeze the sketch onto the pro mini with the boards that I have already built).
BTW, the SensorConfiguration is no problem as far as SRAM is concerned (it just adds the normal SRAM footprint of a Sensor instance): It has no class members of its own. Only the flash size is considerable, but for me this is not the issue, I'm running out of SRAM. -
@user2684 Thanks for the spreadsheet with your memory footprints. Are these the sketch size and the free memory that the compiler reports? Or do you measure them when running the sketch (i.e. including all dynamically allocated variables in SRAM)?
Also, one problem you don't seem to have covered is heap fragmentation. Particularly your new List class seems to be prone to heap fragmentation with a large number of sensors: each time a sensor is added, the list needs to re-allocate the memory for its pointers. The old memory will be released, but it will remain as a hole in the memory, where new variables might be allocated. However, the next time you add an element to the list, the new pointer array can probably not re-use some of those freed holes.
I'll provide a PR (https://github.com/mysensors/NodeManager/pull/247) that adds an allocateBlocks(count) method to the List, which can be used in the sketch and in the sensor's onSetup methods to pre-allocate the required sized. In turn, the _preAllocBlocks member does currently not serve any purpose, and I don't think it's really required functionality. In my eyes, it's more important to ensure that at a certain moment the required memory is allocated. It is not neccessary to keep a minimum nuber of blocks allocated at all times.
For a node with 10 physical sensors (and maybe one or more built-in sensors), this will save at least 22 bytes (1,1% of total SRAM!) and probably reduce heap fragmentation by quite a bit.Another are where >22 bytes can be saved is the power manager. None of my nodes makes use of the power manager, yet each sensor needs to keep a (null) pointer to a power manager. Providing a compiler #define NO_MODULE_POWER_MANAGER can exclude everything PowerManager-related from compilation and save one pointer per sensor and node, i.e. with 10 sensors you'll save another 22 bytes (1,1% of total SRAM!) I'll provide a PR for this, too: https://github.com/mysensors/NodeManager/pull/248
Apart from that, I simply stripped down the SensorMQ class and removed all configurability for my own use (don't worry, I won't submit this as a pull request... it is simply meant to squeeze the sketch onto the pro mini with the boards that I have already built).
BTW, the SensorConfiguration is no problem as far as SRAM is concerned (it just adds the normal SRAM footprint of a Sensor instance): It has no class members of its own. Only the flash size is considerable, but for me this is not the issue, I'm running out of SRAM.@reinhold I measured only what the compiler reported. Main focus was to reduce the flash size, I honestly ignored any potential SRAM issues which is bad :) Thanks for the hit and for the great contributions and PRs so far. You're right about the List class and especially when the number of items is known, there is not reason why not allocating the blocks beforehand. I will merge your changes as they are and then take it back and adjust something (e.g. make the number of block you are passing to "node" optional, moving child creation into the constructor, always allocating the blocks for children in each sensor, generalizing a bit more the NO_MODULE thing etc.). Thanks again!
-
Hi, for those looking for making NodeManager time aware (with and without an attached RTC), this capability has been added to the development branch through https://github.com/mysensors/NodeManager/pull/259 and it is of course transparent to the end user. When this feature is enabled, also, the node resumes the remainder sleep time when woken up by an interrupt and allows SensorPulseMeter and subclasses to support sleep mode. Thanks
-
Hi, for those looking for making NodeManager time aware (with and without an attached RTC), this capability has been added to the development branch through https://github.com/mysensors/NodeManager/pull/259 and it is of course transparent to the end user. When this feature is enabled, also, the node resumes the remainder sleep time when woken up by an interrupt and allows SensorPulseMeter and subclasses to support sleep mode. Thanks
-
The SensorSwitch/Door/Motion all work on interrupt pins only ? Sorry if I've misunderstood the code, thats possible.
Anyway I've made a few changes so that busywait, if thats what they are called buttons can be also used. I'll see if I can use git somehow to share the changes, or show them at least
Great library, looking to convert other things to use this.
-
Great code, another question.
For the SensorDimmer, each sensor has only one type set so a dimmer can't receive v_status & v_percentage messages.
I've worked around this by registering two sensors at the same Analog out pin. But they don't then have access in a nice way to the status of each other as values are changed. So if I turn the led on/off via the v_status message I can't preserve the previous percentage, set by the other v_percentage sensor.
Is this normal ? I'm new to all this but at least mycontroller lets you specify for a single sensor both v_status,v_percentage so both types of messages can be sent and processed in onRecieve()...
Perhaps some enhancement on the SensorDimmer class could do this ? in ModeManager.cpp (if (message.type != _type) return;) inside Sensor::recieve.... isn't that too restrictive ?
-
The SensorSwitch/Door/Motion all work on interrupt pins only ? Sorry if I've misunderstood the code, thats possible.
Anyway I've made a few changes so that busywait, if thats what they are called buttons can be also used. I'll see if I can use git somehow to share the changes, or show them at least
Great library, looking to convert other things to use this.
@rfm69 yes, they are all interrupt-based. Reason is they would need to be functional either when sleeping or not so I went for this approach. If you need an example of such a sensor not interrupt-based, have a look at the SensorSonoff, it is using a different approach for handling the button. Thanks
-
Great code, another question.
For the SensorDimmer, each sensor has only one type set so a dimmer can't receive v_status & v_percentage messages.
I've worked around this by registering two sensors at the same Analog out pin. But they don't then have access in a nice way to the status of each other as values are changed. So if I turn the led on/off via the v_status message I can't preserve the previous percentage, set by the other v_percentage sensor.
Is this normal ? I'm new to all this but at least mycontroller lets you specify for a single sensor both v_status,v_percentage so both types of messages can be sent and processed in onRecieve()...
Perhaps some enhancement on the SensorDimmer class could do this ? in ModeManager.cpp (if (message.type != _type) return;) inside Sensor::recieve.... isn't that too restrictive ?
@rfm69 you're right, this also came out a while ago. I was sure we had an associated issue on github for this but it doesn't look like. Here's the new issue for your reference https://github.com/mysensors/NodeManager/issues/296. Implementing this in v1.6 is really challenging but with the new v1.7 architecture of the development branch should be way easier since all the children share the same variables. Thanks!
-
@rfm69 yes, they are all interrupt-based. Reason is they would need to be functional either when sleeping or not so I went for this approach. If you need an example of such a sensor not interrupt-based, have a look at the SensorSonoff, it is using a different approach for handling the button. Thanks
@user2684 Thanks, I've implemented something myself with changes to the library, just playing around to understand it better. But will look at the SensorSonoff you mention.
Thanks again, its a great library.
Had a quick look at 1.7 dev branch :) is this ready for using already, testing or trying, or have you got more to do before its refactored etc ? I was interested in looking what the changes in architecture were.
-
@user2684 Thanks, I've implemented something myself with changes to the library, just playing around to understand it better. But will look at the SensorSonoff you mention.
Thanks again, its a great library.
Had a quick look at 1.7 dev branch :) is this ready for using already, testing or trying, or have you got more to do before its refactored etc ? I was interested in looking what the changes in architecture were.
@rfm69 v1.7 is still not completed but theoretically fully functional and with most of the architectural changes already done (https://github.com/mysensors/NodeManager/milestone/9 for the full list of things still to be completed and those already implemented). For sure safe to be tested out! Thanks
-
Great code, another question.
For the SensorDimmer, each sensor has only one type set so a dimmer can't receive v_status & v_percentage messages.
I've worked around this by registering two sensors at the same Analog out pin. But they don't then have access in a nice way to the status of each other as values are changed. So if I turn the led on/off via the v_status message I can't preserve the previous percentage, set by the other v_percentage sensor.
Is this normal ? I'm new to all this but at least mycontroller lets you specify for a single sensor both v_status,v_percentage so both types of messages can be sent and processed in onRecieve()...
Perhaps some enhancement on the SensorDimmer class could do this ? in ModeManager.cpp (if (message.type != _type) return;) inside Sensor::recieve.... isn't that too restrictive ?
@rfm69 for your SensorDimmer request, feel free to give v1.7-dev a try from here https://github.com/mysensors/NodeManager/tree/development. Details on the implementation of your request can be found at https://github.com/mysensors/NodeManager/pull/302. Thanks
