@alowhum Sorry, I have abandoned this project. It didn't garner the interest I hoped it would, and Iβve moved on to other microcontrollers/radios/networks since then. This is hosted on GitHubβs free static hosting, so it should continue to work as long as they are in business, and it should work on devices as it was 5-6 years ago. But I wonβt be supporting it and wonβt continue to build on it. Glad to hear that you like it!
@user2684 Sorry, I'm not at a computer. IIRC, the issue I've been facing is that when I used the latest version of NodeManager with the latest version of MySensors dev branch, with signing and encryption, the gateway would crash on startup. This has been preventing me from proceeding.
Sorry for the lack of detail - I can't verify things at the moment. Looking forward to 1.6!
MAX_SENSORS defines the maximum number of child ids
How can I know how many child ids exist in total? Is there a way to arrive at it from the sensors? Sorry for my ignorance.
Fixed most of the other bugs you pointed out. Thanks!
@user2684 We will need to decide which version of MySensors to use. I've been using the latest version from the development branch. In fact, the ascii-art splash screen is the latest commit, as of 5 days ago. Since you aren't seeing the ascii-art, you are probably on a slightly older commit.
I'm not sure if we can start supporting from the current stable 2.1.1 onwards? I don't know about the advantages that the new driver for RFM69 brings. If it's any effort at all to get it to work with older versions, it's probably not worth it. I don't mind waiting till 2.2 is released.
I've simply copy-pasted the MY_RFM_* defines from somewhere in the MySensors examples. They are currently hardcoded 
and probably are just the defaults. I'll clean that up.
Thanks for pointing out the other issues. I'll get on it. Thanks!
@user2684 Thanks. That helped identify the problem. I've changed the key switch to inputSwitch to avoid clashes with C keywords, which is causing the issue. I had done this long ago, when I wasn't caring about data migrations. Looks like your data still has that old bit in it.
Sorry for the trouble. The easiest thing to do is to delete that value, and start afresh. I'll be more diligent about making modifications in the future.
@user2684 Thanks. If you haven't already deleted the app_data key, could you please share the value of that key with me? If you've already deleted it, do you remember which sensors you had configured across your nodes in your settings?
This was an error in the migration process, where the old data is ported over to the new format, if I make changes in the format of the data. I've obviously done something wrong in that porting logic.
@user2684 Woops, if you only see the loading message, I've dun goofed. Can you please look in your JS console (Ctrl+Shift+I) and tell me what is the error you see?
To fix the problem, you can delete your existing data. You can do that within the devtools itself. Click on the Application tab in the devtools, and in the left pane, under Storage, select Localstorage > https://rakeshpai.github.io, and delete the key called 'app-data'. Please make sure you copy your error messages before you do this though, so that I know what went wrong.
Sorry for the lack of detail. You can get the latest gateway sketch from the app itself at https://rakeshpai.github.io/mysensors-network-manager/
The sketch is just a barebones sketch, wiring up the before, presentation, etc. functions to node manager. The config.h is as follows, though nothing stands out to me as being out of place.
#ifndef config_h
#define config_h
// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES
/**********************************
* Sketch configuration
*/
#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
#define MY_REPEATER_FEATURE
/**********************************
* MySensors node configuration
*/
// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG
// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24
// Serial gateway settings
#define MY_GATEWAY_SERIAL
/***********************************
* NodeManager configuration
*/
// if enabled, enable debug messages on serial port
//#define DEBUG 1
#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 0
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 0
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 0
// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0
#endif
I'm trying to compile a serial gateway, no sensors, against the development branch of NodeManager, which works fine. However, when I open the serial monitor, I get the following output:
$ pio device monitor --port /dev/ttyUSB2
--- Miniterm on /dev/ttyUSB2 9600,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
__ __ ____
| \/ |_ _/ ___| ___ _ __ ___ ___ _ __ ___
| |\/| | | | \___ \ / _ \ `_ \/ __|/ _ \| `__/ __|
| | | | |_| |___| | __/ | | \__ \ _ | | \__ \
|_| |_|\__, |____/ \___|_| |_|___/οΏ½οΏ½
...and it hangs.
Any idea what the issue might be?
PS: As a self-confessed connoisseur of ASCII-art, I tip my hat in your general direction. Well done, good sir.
Pretty big update: Added the ability to select the board that the node/gateway is based on. Supported boards are: Pro Mini (5v and 3.3v), Nano and Uno for the atmega328 boards, and NodeMCU and D1 Mini for the esp8266 boards. Please let me know if your favourite board isn't listed here.
The selection of the board effects the list of available pins for sensors. The sense pin dropdown for sensors now shows the appropriate pins (analog/digital/interrupt pins) based on the board selected for the node and the type of sensor being configured.
The selected board also affects the platformio.ini file generated, which makes PlatformIO integration even more enjoyable to use. No need to specify the board during the build.
This was a pretty big update, including changes to the underlying internal datastructure itself, so it's also an acid test for the data migration and versioning system implemented before. If you don't see any errors when you refresh after the update, it means everything worked transparently, and that's great news! (The migration mechanism hasn't been tested so far, so please let me know if you see errors when you get the update.)
Also implemented a reporting interval setting for analog sensors, to bring it in line with the latest changes in NodeManager.
We're getting closer to being release-ready!
EDIT: Also, this project crossed 100 commits in GitHub already! Wow!
@user2684 The clear separation between sleeping and reporting definitely makes things easier to understand and use. Thanks for doing this.
I've now published a release with the changes for sleep time and reporting time in place. Could you please give it a spin? https://rakeshpai.github.io/mysensors-network-manager/ You may need to delete your existing sensors first.
I haven't implemented a global reporting time. I felt it would make the UI complicated. I might be mistaken, but it isn't relevant for all types of sensors, isn't it? For eg. there's not much to sense by polling a motion sensor, or a switch. So, I felt like it's best if we ask for it only where relevant, and not out of context. I'm open to change this though. Please let me know what you think.
Just want to be sure I understand the changes, so please bear with me.
When a node is awake, each sensor can have its individual reporting times (or a global reporting time), which need not have any relation to anything else. However, if a node is sleeping, its reporting time has to be a multiple of the sleep interval of the node. Is this understanding correct?
If so, I'm thinking of the following UI:
Is my understanding correct, and would this kind of a UI implementation do the job?
Sorry for the silence. I didn't realise I wasn't getting notifications about this thread. Thanks for the updates. I'll go over your comments and issues and start making changes accordingly. I'll update here once I have something. Thanks again. Woohoo!
Sorry for the long silence. it might look like I've been slacking off (and that's mostly true), but I've also been rolling out some updates:
network.json file. I haven't implemented a 'restore' functionality yet (coming soon), but the network.json file contains everything needed for a restore. The network.json is the internal state data-structure used by the app.To clarify, while backups are a great idea, you don't need to save anything. Stuff is still automatically saved in the browser's storage anyway during regular use, and is persisted even if you close the browser. Backups are for the assurance of having a neat little zip containing everything on your hard drive, just in case things go horribly wrong.
Very little has changed visually. To find these new features, launch the app, and click the 'hamburger menu' at the top right of the page.
@user2684 Thanks for the update. Much appreciated.
The thing is, I think it'll be really great if this project can come to a point where I can confidently tell people that they can flash the output code to their devices, at least in a test setup, so that I can start to get some usage-based feedback. So far, I've very vocally said that the output isn't ready, and that's definitely limiting usage.
So, I wouldn't say it's urgent, but user-feedback is important, so I'd like to start making releases that I can ask people to try.
@dbemowsk Thanks! It's very encouraging to know that this is useful.
I've been thinking about the libraries problem. The easiest-to-use solution is to put it in the zip, so I'll probably just ultimately choose that. Problem is the way Arduino looks up dependencies. It has to be globally installed, or put in a flat directory structure in the local folder. Both of these are unappealing solutions.
I'll be adding the ability to compile with PlatformIO soon, which has a built in dependency manager, but that's really just an alternative, not a solution.
This tool is built on top of NodeManager, so if you need a new sensor, NodeManager is the right place to contribute. There will be a lag between when NodeManager implements something and I make it available here, but I could prioritise based on demand. I haven't considered the ability to add custom sensors by yourself directly into the UI. I'll think about it. I suspect it might make things too complicated though.
Fields for notes and stuff is a great idea! Thanks for suggesting. I'll add this to the UI soon. Adding an ability to import and export all your data is on the cards. I completely agree that having an archivable dump of everything is very important for backups. (Would things like Dropbox integration be interesting as well?)
Noted about the floor plan thing. I'll wait for more votes for it. I've always thought it would be cool too, but I've felt that it's a lot of effort to implement, then a lot of effort to configure correctly, and finally too little to gain from it. But then, if there's interest, why not!
(PS: Speaking of diagramming, I've given some superficial thought to having a feature where it generates a schematic of the circuit, based on your configuration. What do you think?)
@pjr Noted. Thanks. Are there a lot of people using RS485? (I have no idea, and I'm constantly surprised by how people use MySensors.) My only concern is that I don't want to add too many features that people won't use, since it would make the UI confusing. But if it's a popular feature, it's definitely on the to-do.
@Anticimex Haha. No, it doesn't come with the IDE, and you have to install it separately. It's required if you want to use their Cloud IDE. Besides, it only validates and executes commands from their server, and not from anywhere else.
You still need to have a lot of trust in their server when using their cloud IDE - they appear to indirectly effectively have local shell access with your user privileges when using their cloud IDE, and their installation process sets up additional udev rules for serial access.
Also, I'm no security expert, so don't take my words at face-value - I don't want to blame them incorrectly. I might very well be wrong in my assessment.
I'll be stripping out the code I've written, so here's a screenshot for posterity. Serial comms in a browser!
tl;dr: Arduino Create Agent is only for the Arduino Create site. Everyone else is locked out. We're back to square one, and we'll need to implement a native app on the desktop.
I've hit a dead-end with Arduino Create Agent.
Firstly, I got bad vibes when I was using their API. It felt very kludgey, and it was very hard to do very simple things. API troubles aren't that big a deal though, and soon enough, I had a decent reliable wrapper on top of their API. I was able to successfully list serial ports, connect to one, and talk over serial to it. I also found a lot of undocumented reliability issues along the way, which I had to fix by trial and error, but in the end it was pretty solid. It felt great to see MySensors' Serial.print messages in the browser! :metal:
However, I hit a complete road-block when trying to upload firmware. They require that every upload be signed, and they verify signatures. About signatures, their readme says that the signature should be:
(...) signed with the private key that matches the public key contained in the config.ini of the arduino-create-agent
However, the config.ini doesn't have any field allowing for public keys to be put in there. Also, their code for verification uses a hard-coded public key anyway. Presumably the private key lies on their servers.
If I understand correctly, this basically locks out everyone but Arduino from using Arduino Create Agent to upload firmware.
Aside: It's the upload 'commandline' that requires signatures, not the firmware itself, which is very strange. Also, the upload command-line is literally the shell command used to invoke avrdude. It doesn't sound like a good idea at all to take a shell command as input from a web server, via a browser, and execute it locally, signed or not. See what I mean when I say their API is kludgy? This isn't the most massive security loophole though, since they cryptographically verify the command's signature, and this is all over SSL anyway so MITM isn't possible, but I don't think I would have implemented it like this.
Can't fault Arduino for the agent not working with our site. There's no reason they would have designed for third-parties using their app to flash Arduinos. But it does mean that we are locked out.
Now, I can file bugs, and if they are interested they could address them, but even if they agree that this is a good idea, it's unlikely that this would be a priority for them. We can't wait for them, but if/when they are ready, we could move over to using their plugin. Until then, we are on our own.
Onwards!
Not as rosy as I initially thought. I thought I might be able to compile the sketch locally using the app above. However, it looks like compilation is not a responsibility that this app has taken on. It only deals with listing serial devices, connecting to them, and flashing them, but not compiling sketches.
Arduino Web Editor appears to use arduino-builder, which I'm assuming they run in the cloud, since the setup isn't very easy at all. This means that we'll have to do cloud compilation as well. (@Anticimex, you might find arduino-builder interesting. They've specifically documented running arduino-builder in a CI system.)
It's still awesome that we have a solution to talk to the device from the browser. It's just that it'd have been even better if it did compilation as well. Hosting a cloud compilation service is not a huge problem - I run a reasonably high-traffic online service as my main gig, so I have some experience - but it'd have been better if we didn't have to.
I wonder why Arduino chose to compile in the cloud, considering that they ask their users to download a local app anyway to talk to the hardware. They wouldn't have to run a large part of their infrastructure, thus saving lots of associated costs and effort. And, they'd have a local toolchain, removing the need for an internet connection for compilation. They'd have to deal with setting up the toolchain locally, but they've been doing that for years, and have a lot of experience with it. Also, they would've enabled an ecosystem of apps like ours, so that we don't have to host compilers in the cloud. Maybe I'm not seeing some detail, but I think I'd have made the choice of having the compiler run locally too.
EDIT: I've raised an issue in their repo.
I just discovered this: https://github.com/arduino/arduino-create-agent
It's a project by the folks at Arduino, so it's as official as it gets. It's an app that you run locally, that runs in the background, puts an icon in the system tray, and starts a WebSocket server, with which a browser can connect to the local USB devices. It looks like it has been built for the Arduino Web Editor, and is in more-or-less active development.
I just spent under 15 mins with it, and I was able to talk to it from a CLI tool over websockets, and get it to list the available serial ports successfully. This is very encouraging.
Does anyone have any opinion about / experience with this? They have binaries for the three big desktop OSs. It's written in Go, so it's very clean in terms of deployment - it's just a single binary with no dependencies.
It looks like they enforce security by having a config.ini file with a list of allowed origins (domains) that are allowed to access the local web server. Since our page is running over HTTPS, the browser also imposes a restriction that it won't connect to insecure origins, which these chaps circumvent by creating a certificate locally during install (a process that I don't completely understand yet).
So, if someone is to use this, they'll have to install the app based on their platform, then modify config.ini to give access to this URL. That's relatively easy to do. It gives us access to everything that Arduino can do, but from a browser.
Most importantly, it's an official project by the folks at Arduino, so it can be trusted. Also, we don't have to replicate this work, and simply build on top of their APIs.
Thoughts?
@hek Thanks. The download works for me just fine. Can you provide any more details? Do you see any errors in the browser's console? Also, hope you are using a desktop OS.
I haven't made much/any changes to the functionality, so downloads should be working as it was.
Just a quick update: I've been doing a bunch of work on the visual aspects of the app. Dare I say, it looks a lot better now. 
Check it out. Feedback welcome.
Since I haven't actually done this yet, I thought I'd outline the 'stack' used for this app, in case folks want to dive into the code.
So, it's a very modern stack. Cutting-edge web stuff.
It's very early in the project, so I don't have any long-term plans outlined. In fact, I'll be honest, I haven't thought that far. But it's a community project, so you are welcome to steer the project as you see right.
Some areas to help:
Actually, the absolute best way to help right now is to simply poke around the UI and point out issues, no matter how minor they are. It's the little things and attention to detail in an app that make it nice to use, so it'll help to have an obsessive pair of eyes going over the UI. You can point out bugs, usability issues, things that might be confusing, errors in the generated code, or any other changes you think will improve the app.
You can comment here with suggestions, or use GitHub if you like.
@getlarge Thanks! It's always nice to know that this is potentially useful.
Dogoma's approach is interesting, but as you said, not the most integrated. If there has to be an app downloaded anyway, why not go the next step and make it all easy to use.
Thanks for pointing out the issue with the readme. Fixed.
avrgirl is awesome! Finally someone decided to modernise a 20 year old piece of software.
You are welcome to dive into the code if you like. Node is only used for development. It's hosted on a static server, so there's no server-side language. The only problem with diving into code right now is that things aren't documented well, I haven't outlined what the goals/directions are, and it's kinda early, so I'm making sweeping changes all the time. But if you can get past that, feel free to dive in and familiarise yourself with the code, and even start making changes.
Speaking of which:
Update: No functionality change, but I've applied a fresh coat of paint to the UI. Hopefully it looks a little better now. There are still some cobwebs in the corners - I'll get to them soon.
@archiijs I'm very much in the same boat as you. While I have some experience writing code, C and C++ is all Greek and Latin to me. I can maybe read a little bit of C/C++ if my life depends on it, but that's about it. That's why I'm creating this tool.
Thanks for the feedback about app downloads being ok. I'll keep that in mind. Ideally, there won't be any need for debugging, since the generated code would be perfect right away, but I've been told that we don't live in an perfect world. However, viewing the serial monitor is definitely one of those things a desktop app can do, which a browser can't. I'm definitely excited about the possibility of communicating straight with the device.
It might take me some time to get to it though. However, all of the work I'm doing right now will be reused for the desktop app as well, so effort isn't wasted. At this moment, I guess it's wisest to see if this web app gets traction. Once we know it's being used, it'll be worthwhile making it better.
PS: How does the name 'MySensors Configurator' sound? Not the most creative, I know...
@Anticimex My current favourite is the PlatformIO Core CLI. While it was built to work in an IDE, it's actually completely independent, and only requires Python to run. It has support for lots of boards too, and has a built-in package manager making dependency management very easy. Since it's a CLI, it can easily be automated with bash scripts, can run headless, and doesn't require a windowing environment. Can't do that with GUIs like Arduino. There are alternatives to PlatformIO, but none that I found were feature-complete.
In fact, while I haven't exposed it, I've already implemented a way to download the code in a folder-structure that platformio prefers. I'll be rolling that out soon.
@core_c I completely understand the security issues of cloud compilation. But I also see the convenience it brings. If I do implement compilation in the cloud, I'm not going to take away the ability to download the code. You won't be forced to use the cloud. That said, I'm not going to implement cloud compilation at this moment, since it's pointless to do so if I can't also flash the device, and I currently can't flash the device from the browser.
@Anticimex Agreed, and I'm very sensitive to the security issue of cloud compilation. However, the primary issue at the moment is that we can't flash from the browser. The choice is either (a) forget about flashing altogether, or (b) build a native app. Regardless of the option we pick, the cloud doesn't fit in.
The cloud would've been in the picture if we were to flash from the browser, and I've thought extensively about the security ramifications of doing so, but that's a little irrelevant now, since we can't flash from the browser. A native app could simply do the compilation locally.
As a side note, I've been keeping an eye on web standards like WebSerial and WebUSB, which are designed to put the user in control far more than any native app does. If these standards make it through, we'll have a much more enjoyable way to work with hardware. I'm certainly looking forward to it.
So the question is: (a) Go native to offer more features at the cost of a largish download, or (b) Stay in the browser but with lesser features and conveniences.
Need to bounce this off someone. Please share your thoughts.
I've always wanted to make it possible to directly flash the hardware from the browser. Of course, browsers don't allow communication with serial devices. So, to work around this, I thought I'd create a browser extension. This is already a bad idea, since I'd have to develop extensions for every browser out there. Regardless, I decided to start with Chrome since it has the largest market-share for browsers. However, after a day's work, I discovered that Chrome extensions don't allow serial communication either!
The suggested alternative is to use Chrome Apps. However, Google has end-of-life-'d Chrome Apps, and starting a new project based on Chrome Apps would be a bad idea.
Even if I do somehow get this to work in Chrome, I'd still have the other browsers to care about.
There's two other alternatives I can think of:
The big problem with both these approaches, of course, is that it requires the user to download and install an app. That's a lot to ask for, when compared to hitting a link in the browser, which means that many (most?) people are unlikely to use the app.
So, in summary, the options are:
If I'm building a native app, I'll be using Electron, which is the JS way of building desktop apps. It's awesome, since it is effectively OS agnostic - the same code-base can work across Windows, Mac and Linux. However, Electron is notorious for creating large binaries, since it packages up both Chrome and Node.js within it. Expect download sizes in the 40-50 Mb range, maybe higher. (Popular apps that use Electron include Slack, GitHub desktop, and Atom.)
I'm confused about how to proceed. What would you do? What do you think is the best way ahead?
I've got an update with a bunch of fixes. Thanks, @user2684!
switch variable name changed to inputSwitch to avoid conflict with keywords.@user2684 It looks like I'll wait for your next release before I close some of the other issues:
No pressure though. I've got enough things to iron out before I can call this even ready for beta-testing, and I've been getting caught up with IRL things too. Whenever you are ready - no rush.
EDIT: Also, just tested the latest 1.5.1, and the 'crash-on-message-received' problem is gone! Thanks!
@user2684 Ah, that makes sense. I was looking for something like a 'reporting interval' in the NodeManager API. So if I understand correctly, I should setMode to WAIT and call setSleep to specify the reporting interval?
@user2684 Woah! Thanks for taking the time to write and share your notes. This is invaluable! Thanks again for this list.
Some quick thoughts:
The rest of what you pointed out either reflect my misunderstanding, or is an incomplete implementation, or are straight-up bugs. Thanks again for your list. It's invaluable. I'll have fixes ready soon.
@user2684 I can't run your sketch because I don't have an RFM69 handy. However, that's one noticeable difference between your sketch and mine - I am using the NRF24L01+.
Other than that, the generated gateway sketch sets the following flags differently from yours, in the config.h:
#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
#define PERSIST 1
#define BATTERY_SENSOR 1
I've assumed that the gateway isn't battery powered, so I've set the BATTERY_MANAGER to 0. POWER_MANAGER is only set depending on choices made for sensor settings, so if there are no sensors POWER_MANAGER is set to 0. I guess the BATTERY_SENSOR value in the generated sketch is wrong, since it should be 0 for the gateway as the gateway is not battery powered.
All that said, I really only experience the crash when the gateway receives a message, so to replicate my scenario, you'd need to set up another node and (if it hasn't got an ID already), it'll send out an ID request to the gateway. It's on receiving that ID request that the gateway dies in my case.
Hope that helps. Let me know if there's any more details I can provide.
I'm glad to report that I'm not completely losing my mind. 
The difference between the recent tests, and my initial tests (where I saw everything working), was that in my recent tests, I was using NodeManager 1.5.1 from your fork.
I reverted to using the NodeManager from the MySensors repo's master branch, disabled signing and encryption, and everything worked. To push the boundaries slightly, I enabled signing, but kept encryption disabled, and flashed this on the gateway and on the node. Compiler messages:
Sketch uses 22,772 bytes (74%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,871 bytes (91%) of dynamic memory, leaving 177 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.
So, not enough leg room memory-wise, but I tried running it anyway:
NodeManager v1.5
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY
MY I=0 M=255
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=
255;155;3;0;3;
RECV S=255 I=155 C=3 T=3 P=
255;86;3;0;3;
RECV S=255 I=86 C=3 T=3 P=
255;40;3;0;3;
RECV S=255 I=40 C=3 T=3 P=
255;249;3;0;3;
RECV S=255 I=249 C=3 T=3 P=
255;202;3;0;3;
RECV S=255 I=202 C=3 T=3 P=
I don't have any controller running, so nothing's handling the serial data from the gateway. However, you can see the last few lines in the paste above, where the gateway receives ID requests from the node just fine, and without a crash.
Considering that the only difference between my last post and this one is the version of NodeManager I've been using, I think it's safe to conclude that the crashes I was facing was due to 1.5.1. Not sure what the fix is, but at least we've identified the issue, somewhat at least.
That said, 1.5.1 was definitely the correct direction in terms of memory usage, since I could fit everything in with it. (With 1.5, I have to disable encryption to get it to fit.) So, if it is possible to identify and fix the stability issue with 1.5.1, and still be able to fit within memory limits, that'd be ideal.
EDIT: I still don't see the RADIO_OK message, or anything else really, in the node's serial output, though the gateway behaves like it's receiving messages just fine. I suspect the node just isn't printing that string to the serial output.
@user2684 Sorry for the delay. I tested the whole thing as above with signing and without encryption. During compile, it did appear to reduce memory usage by about 20%. However, it didn't change the behaviour of the gateway. It still hangs after receiving the first message.
Good catch about the RADIO OK message. So, I tried removing signing and encryption, just to eliminate variables, and I still didn't see the RADIO OK message. Also, I was still seeing the gateway crash even without signing and encryption. I could have sworn this used to work fine. Now I don't even know what to believe of my previous tests anymore. Did I botch them up, or mis-remember them? Is this real life?
I guess I'll build another Arduino Pro Mini + NRF on a perfboard and try again. I'll update once I have something. Meanwhile, and I know this is a big ask, but if you could please test on hardware you have laying around that you know works well, it'll really help to know if this is somehow just my setup acting up.
EDIT: I flashed the node sketch on the board I was normally using for the gateway, and I still don't see a RADIO OK message. This either means that both my boards are broken (possible but unlikely), or that the RADIO OK message isn't printed in the serial output by the sketch at all. I haven't looked at the code yet, so I'm not sure which it is. Also, worth remembering that even though the node doesn't print a RADIO OK message, the gateway definitely reacts to the node being powered on (by crashing), so there's definitely some radio communication going on.
@Anticimex From what it looks like at the moment, it looks like I'm hitting up against the memory issue. (But I might be wrong.)
I don't know why the two nodes weren't communicating in my previous post. However, I decided to start from scratch, and ran the ClearEEPROM sketch on both my arduinos. Then, I disabled personalisation on both the gateway and the node sketches, flashed them to the boards, and watched ID requests come on the gateway's serial output, from the node. (I don't have a controller running.) So, everything looks good so far.
Then I ran the SecurityPersonalizer on the gateway, and flashed the gateway sketch to it again. This was the output of the personalizer:
+------------------------------------------------------------------------------------+
| MySensors security personalizer |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Configuration settings |
+------------------------------------------------------------------------------------+
| * Guided personalization/storage of keys in EEPROM |
| * Software based personalization (no ATSHA204A usage whatsoever) |
| * Will not require any UART confirmations |
| * Will store HMAC key to EEPROM |
| * Will store AES key to EEPROM |
| * Will store soft serial to EEPROM |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Hardware security peripherals |
+--------------+--------------+--------------+------------------------------+--------+
| Device | Status | Revision | Serial number | Locked |
+--------------+--------------+--------------+------------------------------+--------+
| AVR | DETECTED | N/A | N/A (generation required) | N/A |
+--------------+--------------+--------------+------------------------------+--------+
+------------------------------------------------------------------------------------+
| Key storage |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key |
+--------+--------+------------------------------------------------------------------+
| HMAC | OK | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES | OK | 1E4FAF1129029D34C4832F6B48044C8B |
| SERIAL | OK | BED2C17579AF3CDA6D |
+--------+--------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| EEPROM |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key |
+--------+--------+------------------------------------------------------------------+
| HMAC | OK | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES | OK | 1E4FAF1129029D34C4832F6B48044C8B |
| SERIAL | OK | BED2C17579AF3CDA6D |
+--------+--------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| This nodes whitelist entry on other nodes |
+------------------------------------------------------------------------------------+
{.nodeId = <ID of this node>,.serial = {0xBE,0xD2,0xC1,0x75,0x79,0xAF,0x3C,0xDA,0x6D}}
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| WHAT TO DO NEXT? |
+------------------------------------------------------------------------------------+
| This device has now been personalized. Run this sketch with its current settings |
| on all the devices in your network that have security enabled. |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Execution result |
+------------------------------------------------------------------------------------+
| SUCCESS |
+------------------------------------------------------------------------------------+
So, that's a success. A little confusingly, the output says 'Run this sketch with its current settings on all the devices in your network that have security enabled.' However, won't the device IDs / serial key be different on every device?
I then flashed the SecurityPersonalizer and node sketch on the other arduino. The output of the security personalizer is as follows, notably with a different serial key:
+------------------------------------------------------------------------------------+
| MySensors security personalizer |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Configuration settings |
+------------------------------------------------------------------------------------+
| * Guided personalization/storage of keys in EEPROM |
| * Software based personalization (no ATSHA204A usage whatsoever) |
| * Will not require any UART confirmations |
| * Will store HMAC key to EEPROM |
| * Will store AES key to EEPROM |
| * Will store soft serial to EEPROM |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Hardware security peripherals |
+--------------+--------------+--------------+------------------------------+--------+
| Device | Status | Revision | Serial number | Locked |
+--------------+--------------+--------------+------------------------------+--------+
| AVR | DETECTED | N/A | N/A (generation required) | N/A |
+--------------+--------------+--------------+------------------------------+--------+
+------------------------------------------------------------------------------------+
| Key storage |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key |
+--------+--------+------------------------------------------------------------------+
| HMAC | OK | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES | OK | 1E4FAF1129029D34C4832F6B48044C8B |
| SERIAL | OK | 31D46BF4ADBEF72803 |
+--------+--------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| EEPROM |
+--------+--------+------------------------------------------------------------------+
| Key ID | Status | Key |
+--------+--------+------------------------------------------------------------------+
| HMAC | OK | 26E779A2876F7FC756C96B60085927EA24F7D8C57E117107BAE92C57080429E1 |
| AES | OK | 1E4FAF1129029D34C4832F6B48044C8B |
| SERIAL | OK | 31D46BF4ADBEF72803 |
+--------+--------+------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| This nodes whitelist entry on other nodes |
+------------------------------------------------------------------------------------+
{.nodeId = <ID of this node>,.serial = {0x31,0xD4,0x6B,0xF4,0xAD,0xBE,0xF7,0x28,0x03}}
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| WHAT TO DO NEXT? |
+------------------------------------------------------------------------------------+
| This device has now been personalized. Run this sketch with its current settings |
| on all the devices in your network that have security enabled. |
+------------------------------------------------------------------------------------+
+------------------------------------------------------------------------------------+
| Execution result |
+------------------------------------------------------------------------------------+
| SUCCESS |
+------------------------------------------------------------------------------------+
Then, I watched the gateway serial output. It looks as follows:
NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY
MY I=0 M=255
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
// stable till here - then I power up my node...
255;118;3;0;3;
RECV S=255 I=118 C=3 T=3 P=
255;x00
...and it just dies. Sometimes it prints non-printable characters to the serial output, sometimes it goes into a reset loop, and sometimes the only way to reset it is to cycle power to it.
However, it does appear to receive the message from the node before it dies, judging by the 255;118;3;0;3; message, which is an ID request. It looks like the gateway dies when trying to handle that incoming message maybe? If I don't power up the node, the gateway seems to appear stable, but once I power up a node, the gateway receives the ID request, and it dies.
Also, I don't suspect this is a hardware issue. I've used the same hardware in past experiments with MySensors (without signing and NodeManager at the time), and it has worked fine. Also, I tried all this without personalisation earlier today, and it has worked fine. I've added 47uF electrolytic caps to both radios, so power shouldn't be an issue.
So, the erratic behaviour of the gateway when receiving a message seems to be suggesting that we're hitting memory limits, maybe?
Output from the gateway's serial:
NodeManager v1.5.1
INT1 M=255
INT2 M=255
0;255;3;0;14;Gateway startup complete.
0;255;0;0;18;2.2.0-beta
RADIO OK
0;255;3;0;11;Gateway
0;255;3;0;12;1.0
PRES I=200, T=23
0;200;0;0;23;
READY
MY I=0 M=1
SEND D=0 I=200 C=0 T=48 S=STARTED I=0 F=0.00
0;200;1;0;48;STARTED
Output from a dummy node's serial:
REG I=1 P=14 P=16 T=23
PWR G=0 REG I=1 P=14 P=16 T=23
PWR G=0 V=12
NodeManager v1.5.1
INT1 M=255
INT2 M=255
Ok, so the both aren't talking to each other yet, and I'll investigate that. BUT! This is NodeManager 1.5.1 + Signing + Encryption, flashed on to two Arduino Pro Minis 3.3v, one with a serial gateway sketch, and the other with the node for a simple sketch of a LDR sensor, both generated from the UI!
You can't see me here, but I'm doing a little dance!
While I complete my little dance here, if you can spot anything obvious about why they aren't talking to each other, please let me know.
Also, minor UI update: New 'Are you sure?' dialogs when you are performing any deletions on the UI.
@user2684 Tried with the 1.5.1 release, and it fits!
Sketch uses 25,786 bytes (83%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,716 bytes (83%) of dynamic memory, leaving 332 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.
It's still complaining about low memory, and from what I understand of what you said, this is a problem, but this is definitely an awesome direction. It's also a concern that I still haven't added any sensors yet - this is just a bare serial gateway sketch. However, I'm still excited. :metal:
I've kept debug off by default in the config for now, so debug messages aren't pushing us to the limit here.
EDIT: I must add, the result above is with signing and encryption, and all that good stuff.
@user2684 No problem with the delays - I've got enough things to do in the meantime. All those real-life things coming in the way all the time, amirite?
@user2684 My version management is pretty crappy at the moment but I'm simply using the latest version of NodeManager.cpp and NodeManager.h from the master branch on GitHub. Users keep a local copy of NodeManager for purposes of offline availability, but they are forced to update their local versions of NodeManager everyday. So, it's safe to assume that within a margin of error of a day, everyone's on the latest version of your checkins on GitHub. (I know this isn't ideal and I should do version management better, but I too like to live dangerously B).)
Also, while MySensors itself isn't shipped as part of the download bundle, I've manually used the latest version from the dev branch in GitHub as well, in my arduino libraries folder.
Maybe I'm just doing something stupid that's causing the memory usage to inflate? I'll wait for whenever you can get a chance to review the output code. Thanks for looking at this!
@Anticimex That's the feeling I had as well, considering that even after I disabled signing, while the sketch fits in memory, it still seemed to complain about low memory. @user2684 does this match your experience too?
I've hit a little bit of a wall, and need help to proceed.
I created a NRF24L01 network in the UI, and tried to flash the default serial gateway code to an Arduino Pro Mini 3.3v/8Mhz. There are no sensors configured on the gateway. I tried to flash the SecurityPersonalizer on the device, and it worked perfectly, and gave a happy sounding serial output. (Yay!) However, when I flash the main gateway sketch, I get a 'not enough memory' error.
Sketch uses 25,792 bytes (79%) of program storage space. Maximum is 32,256 bytes.
Global variables use 2,204 bytes (107%) of dynamic memory, leaving -156 bytes for local variables. Maximum is 2,048 bytes.
processing.app.debug.RunnerException: Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.
at cc.arduino.Compiler.size(Compiler.java:339)
at cc.arduino.Compiler.build(Compiler.java:159)
at processing.app.SketchController.build(SketchController.java:641)
at processing.app.Editor$BuildHandler.run(Editor.java:1782)
at java.lang.Thread.run(Thread.java:745)
Not enough memory; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing your footprint.
For reference, the generated gateway sketch is as follows:
// Message signing
#define MY_SIGNING_SOFT
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7
#define MY_SIGNING_REQUEST_SIGNATURES
// load user settings
#include "config.h"
// load MySensors library
#include <MySensors.h>
// load NodeManager library
#include "NodeManager.h"
// create a NodeManager instance
NodeManager nm;
// before
void before() {
// setup the serial port baud rate
Serial.begin(9600);
nm.before();
}
// presentation
void presentation() {
// call NodeManager presentation routine
nm.presentation();
}
// setup
void setup() {
// call NodeManager setup routine
nm.setup();
}
// loop
void loop() {
// call NodeManager loop routine
nm.loop();
}
// receive
void receive(const MyMessage &message) {
// call NodeManager receive routine
nm.receive(message);
}
// receiveTime
void receiveTime(unsigned long ts) {
// call NodeManager receiveTime routine
nm.receiveTime(ts);
}
and the associated config.h is:
#ifndef config_h
#define config_h
/**********************************
* Sketch configuration
*/
#define SKETCH_NAME "Gateway"
#define SKETCH_VERSION "1.0"
//#define MY_REPEATER_FEATURE
/**********************************
* MySensors node configuration
*/
// General settings
#define MY_BAUD_RATE 9600
//#define MY_DEBUG
// NRF24 radio settings
#define MY_RADIO_NRF24
#define MY_RF24_ENABLE_ENCRYPTION
#define MY_RF24_CHANNEL 76
#define MY_RF24_PA_LEVEL RF24_PA_LOW
//#define MY_DEBUG_VERBOSE_RF24
// Serial gateway settings
#define MY_GATEWAY_SERIAL
/***********************************
* NodeManager configuration
*/
// if enabled, enable debug messages on serial port
//#define DEBUG 1
#define POWER_MANAGER 0
#define BATTERY_MANAGER 0
// if enabled, allow modifying the configuration remotely by interacting with the configuration child id
#define REMOTE_CONFIGURATION 1
// if enabled, persist the configuration settings on EEPROM
#define PERSIST 1
// if enabled, a battery sensor will be created at BATTERY_CHILD_ID and will report vcc voltage together with the battery level percentage
#define BATTERY_SENSOR 1
// if enabled, send a SLEEPING and AWAKE service messages just before entering and just after leaving a sleep cycle and STARTED when starting/rebooting
#define SERVICE_MESSAGES 1
// Enable this module to use one of the following sensors: SENSOR_ANALOG_INPUT, SENSOR_LDR, SENSOR_THERMISTOR, SENSOR_MQ, SENSOR_ML8511, SENSOR_ACS712, SENSOR_RAIN_GAUGE
#define MODULE_ANALOG_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_INPUT
#define MODULE_DIGITAL_INPUT 0
// Enable this module to use one of the following sensors: SENSOR_DIGITAL_OUTPUT, SENSOR_RELAY, SENSOR_LATCHING_RELAY
#define MODULE_DIGITAL_OUTPUT 0
// Enable this module to use one of the following sensors: SENSOR_DHT11, SENSOR_DHT22
#define MODULE_DHT 0
// Enable this module to use one of the following sensors: SENSOR_SHT21
#define MODULE_SHT21 0
// Enable this module to use one of the following sensors: SENSOR_SWITCH, SENSOR_DOOR, SENSOR_MOTION
#define MODULE_SWITCH 0
// Enable this module to use one of the following sensors: SENSOR_DS18B20
#define MODULE_DS18B20 0
// Enable this module to use one of the following sensors: SENSOR_BH1750
#define MODULE_BH1750 0
// Enable this module to use one of the following sensors: SENSOR_MLX90614
#define MODULE_MLX90614 0
// Enable this module to use one of the following sensors: SENSOR_BME280
#define MODULE_BME280 0
// Enable this module to use one of the following sensors: SENSOR_SONOFF
#define MODULE_SONOFF 0
// Enable this module to use one of the following sensors: SENSOR_BMP085
#define MODULE_BMP085 0
// Enable this module to use one of the following sensors: SENSOR_HCSR04
#define MODULE_HCSR04 0
// Enable this module to use one of the following sensors: SENSOR_MCP9808
#define MODULE_MCP9808 0
#endif
I tried commenting out the first three lines in the gateway sketch (the #defines to do with security personalisation, and the sketch fits, but only barely.
Sketch uses 19,922 bytes (61%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,710 bytes (83%) of dynamic memory, leaving 338 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.
I'm by no means a C/C++ guy (which I why I'm writing this tool ) so I'll need help to fix this. I don't even know where to start looking, since there's the combination of NodeManager and the software signing in the sketch.
Any chance you could look at this, @user2684 and @Anticimex? You could download a copy of the code from the UI if you like. My configuration is an NRF network, and the gateway is the default serial gateway on the UI. I've been using the latest version of the MySensors lib from the development branch on GitHub, and am doing the compilation in the Arduino IDE.
Thanks!
Sorry for the silence. My ISP is having trouble, and I've been mostly offline. I'm sending this message over crappy 3G on a phone.
Update
That's all I have for now. I'll have more once my connectivity issues are fixed.
Sorry, I've been offline for the last few days. Just catching up.
'Sketchy' is hilarious. Certainly seems to describe the current reliability of the output code. I'll wait for more suggestions though.
@core_c Thanks! That reminds me...
I need a name for this project, and I'm horrible at naming things. If some creative folks can chip in, that'd be awesome.
I've been referring to this as 'this', or app, or tool, or site, or things like that. None of them are good enough. A nice label would be great. Something catchy, maybe?
Update: Not much has changed functionally, but I've now added a UI notification for when an update is available, so I don't have to tell you to do the double-refresh/hard-refresh and people are always on the latest version.
I've also started writing some tests so that I can start to trust the generated code. I hope these tests are (at least somewhat) human-readable - they might be the easiest way to find out what the generated code is doing.
For eg. this file tests the NodeManager parts of config.h, and this file tests the sensor-specific code for NodeManager. The tests aren't exhaustive yet - still work in progress.
@user2684 Not sure if you'd prefer to read these tests. The alternative would be to change stuff on the UI, download zips, and verify the contents, which can get painful fast.
Thanks, @user2684. @hek tells me that this is not the first attempt at making such a tool, but those attempts didn't end up succeeding. However, NodeManager might be changing that, making such a tool even possible. So, all compliments should be directed to you.
I'll need to bother you soon, because I'll need your help to validate the generated code, since it directly uses NodeManager. Still working on it - I'll keep you posted.
Ah, didn't know that. Noted. I'll see how I can handle alternate boards.
Noted. Right now, the user will have to specify the ATSHA pin correctly for the SenseBender, so I guess it's not the end of the world, but I understand that it can be made easier.
There's actually a little bit of an understanding of the board already - if you choose an ESP8266 gateway and decide to add sensors to it, you'll see pin numbers matching the ESP8266's pinout. That's pin configuration in a different sense though. Let me think about how I can incorporate this.
Thanks for the kind words.
@bilbolodz I just updated the UI. Please do a hard-refresh (ctrl+r in chrome), delete your existing network, and try again.
In general, you shouldn't trust the output code yet. I haven't even run it on an arduino yet. The idea is just to give you a feel for the goals of the app, and find out if people find this useful/interesting.
Bugs reports are definitely welcome though. A comment on this thread works just fine.
Thanks for the comments, @Anticimex. I've now deployed a new build with the signing changes we discussed about, without the whitelisting. You can choose ATSHA or SoftSigning, and can specify the signing pin. You'll need to refresh and delete your existing network for the changes to take affect.
Sorry about the trouble with refreshing and deleting. I'll smooth out this experience once this has more users.
Regarding whitelists, I must add, if there's interest, I would like to develop a browser extension to flash the device directly from the site. If we have a browser extension that could talk to the device, we could read the key off serial. If we combine that with instructions to flash the gateway last, we might have simplified whitelist creation.
Also, I was hoping that if a user specifies that the node isn't battery operated, it would be configured to run as a repeater. I'm not sure how whitelists would work in such a scenario.
As an aside: The flash-directly-from-the-browser thing would require a server-side thing to do compilation - can't run avr-gcc in the browser yet. (Someone should port avr-gcc to llvm, so I can run it as WebAssembly in the browser! Very soon, I'm sure. :))
Yes, the browser downloads the SecurityPersonaliser from the development branch of GitHub. The choice of the development branch was accidental but if that's the way things are headed, I'd rather stick with it.
There's very few modifications made to the personalizer. The changes are made here. Basically, the three keys are set, and PERSONALIZE_SOFT is defined. I say this just so that there are multiple eye-balls looking at the code.
I haven't worked with ATSHA, and I don't forsee myself doing so soon, so I'll need to understand the process before I make changes. If I understand correctly, I won't need to generate a device ID at all. The user will have to download and flash the personalizer on the arduino, and optionally (if they want whitelisting) read the generated key from the serial output. Is that correct?
I realise now that I haven't thought about adding whitelisting features to this yet. If you have any suggestions about how this can be made easy to use, preferably to the point of being an entirely transparent process, that would be awesome. Meanwhile I'll keep staring at your article on signing, hoping to soak more of it in.
Yes, I also haven't added any of the signing defines. That's why I have the huge disclaimers. Here's what I glean at the moment, please correct me if I'm wrong:
#define MY_SIGNING_SOFT // or MY_SIGNING_ATSHA204
#define MY_SIGNING_SOFT_RANDOMSEED_PIN 7 // or MY_SIGNING_ATSHA204_PIN 4
#define MY_SIGNING_REQUEST_SIGNATURES
// optional whitelisting
#include <MySensors.h>
Is this correct? Thanks for looking at this.
@Anticimex Ah, that makes sense. Based on this, I've rolled out an update. I don't have a Windows box to test, so I'd be awesome if you could have a look once again. Don't forget to do the double-refresh or the hard-refresh I mentioned above.
@pansen Thanks! It only appears to work. A few examples I tried also compiled successfully. However, I haven't checked every scenario yet. Besides, I haven't even actually run any of the code on an Arduino yet. I'm almost certain I've done something wrong. You know how it is - it's never right the first time. Hence the 'It doesn't work'
@Anticimex Thanks for the text fix. I've pushed an update with the device ID change.
General note about getting the latest version: You'll need to refresh once, wait for a bit and refresh a second time to see the latest changes. Alternatively, you can force a hard-refresh, usually by pressing Ctrl+F5 or Ctrl+R. I'll make this experience smoother in some time, but please bear with it for now. This extra refresh is because of the offline support.
I'm on Chrome 58 too. Very strange that you see empty zips. Did the example here https://stuk.github.io/jszip/ work for you?
Thanks, @hek and @Anticimex.
I could add the ATSHA if there's interest. I haven't looked into it yet, and haven't used it, which is why I'm only doing software signing at the moment. If there's interest for it, I'll definitely add it. I'll need to understand how it can be made such that just flashing the SecurityPersonalizer will do everything automatically, so that the user doesn't have to interfere with it.
Which browser are you using? I've only tested this in Firefox and Chrome on Ubuntu (though the OS shouldn't matter). I haven't tested this on IE. I'm certain it doesn't work on any iOS browser, because of browser limitations. (Limitations can be worked around by using a server.) It might work on Chrome Android, but I don't have access to an Android to test.
The library I'm using for creating the zip is here: https://stuk.github.io/jszip/ Can you please tell me if their example works for you in your browser?
Thanks for the catch about the 9-byte device id. I'm also certain that the two other strings I'm using is wrong:
For the HMAC: "A 64 digit hex number used for signing messages"
For the AES: "A 32 digit hex number used for encryption"
If you could help me phrase these correctly, I'll push an update with all three corrections. Thanks.
Disclaimer: This is a proof-of-concept. It doesn't work. Things will change, and will break. In fact, I'm sure many things are broken right now. It would be pretty stupid to use this for anything important. I'm just putting this in the public to gauge interest. You have been warned.
Link to the firmware generator.
If you are like me, you want to put approximately half a million MySensors nodes around the house. You've probably already built the first few nodes, so it feels like you are most of the way there. You love the soldering and the building. However, the coding isn't exactly fun. It requires reading through docs, then trying and failing, and rinse and repeat until you get it right.
This tool doesn't solve that problem yet, but it aims to. It aims to provide a simple browser-based UI for configuring your network, and then generating the firmware for the node automatically, using whatever's the best-practice for writing the firmware. This tool writes your code for you!
The good:
The bad:
The ugly:
With that summary out of the way, I'd love it if you could poke around and let me know what you think. You don't need to download or install anything - it's just a web page - so just click the link above. I'm trying to gauge interest in the community for this, to decide if it's worth pursuing.
Would you like me to spend time on making this better?
The icon to download firmware is on the top-right of the page, and you can see it on the pages for either the nodes you create, or on the page for the gateway. You can create nodes by selecting your network in the left hand side navigation, and clicking on the + icon at the top right of the page.
The firmware is downloaded as a zip, which contains a SecurityPersonalizer with your security settings, and the main sketch which runs on your node. Currently, the firmware might not even compile.
Hope this gives an idea of what I have in mind, even if it doesn't actually work.
All manner of suggestions and feedback is welcome.
Why does the power manager use two pins to power the sensor? Could we just use one pin for Vcc, and assume that the GND is permanently connected?
I can't seem to find any way to request the battery level of a remote node from the controller using the serial API. The I_BATTERY_LEVEL internal message seems to be ignored by the remote node, so it looks like it's a one-way message (node to gateway).
The remote node does seem to publish its battery level every couple of minutes. I just can't request it from the controller at will. I'm using the MockMySensors sketch while testing, if it helps.
Is there a way to get the battery level of a remote node from the controller? If not, should I file a bug to put this into the todo? It looks like this shouldn't require any API change to implement.
Since it is possible to have multiple mysensors gateways communicating with the same controller, how do you identify each gateway uniquely? Do gateways have some sort of unique identifier?
Usage: I'm planning to put together a controller, and want to support multiple gateways for completeness. Since serial port names and IP addresses might all change over time, how do I associate metadata with a gateway in case of disconnection and reconnection?
I'm thinking that there should be some sort of unique id associated with a gateway, which I can query over the serial API. Does such a thing exist?
@Adam-McCartney That certainly works. I'm looking to avoid the overhead of the configuration and setup required at home, and want to keep it simple for everyone to use ("just go to <url>"). There's also the interesting possibility of the hosted controller being used by more than one household, but that's something for the future.
Cool. Doesn't quite do the job, but I'll figure my way around it. Thanks again, @hek.
Client mode is what I was looking for, thanks, @hek.
Is it possible to define the controller IP address dynamically at runtime, rather than having it compiled into the sketch? That way, it could be possible to have some sort of API/UI to configure the gateway after deployment, without having to reflash the Arduino.
(I'm working off the development branch, if that's any help.)
Thanks, @mfalkvidd. That seems to only work for the MQTT gateway, though. I was wondering if there's something similar for the socket API, using just the Ethernet/ESP8266 gateways (without MQTT).
Currently, it looks like the Ethernet/ESP8266 gateways open a socket and act as a server to accept incoming network connections. However, I'd like it to act as a client, and connect to a public host/port. Is this possible?
This is my use-case: I need to be able to control devices at home with a phone/web-based app, from anywhere in the world. To make this possible, I'll need to host the controller software on a public server on the Internet. If this controller has to access the gateway inside my home network, I'll have to do some jiggery-pokery at my router, and maybe configure a VPN. I could avoid the trouble if the gateway would act as a client and connect to the controller instead.
Even better if this was done using standard/expected open ports, rather than ports that might be blocked in different networks. So if the gateway could initiate a HTTP-based WebSocket connection to a controller based on the controller's public address, it would be perfect.
It appears that this is how the gateway connects to an MQTT broker, but I couldn't find any way to do this for the socket API. Is this possible?
@hek Thanks! That's neat! I'm not sure about this - is it possible to ask a node for its presentation, just when I receive data from it? Will it be awake?
For example, if a sensor says that its state is 'triggered', and is expected to sleep right afterwards, do I still have time to ask it for its presentation data? Is the sensor going to wait for an INTERNAL request? If not, what is the best time to ask for data?
@hek, @Fabien Thanks! When writing this module, I faced an issue that is probably worth bringing up. Details follow. Let me know if I'm doing it wrong.
It appears that a sensor node's presentation data is only published when it is power-cycled. So, if my node.js module (or the app that includes it) has to restart for any reason, it can't figure out what the capabilities of nodes are, unless all nodes in the network are restarted too. To work around this, I've had to store the sensor nodes' presentation information to disk as a file in case the controller restarts. This works for now - it isn't ideal, since all nodes still need to be restarted at least once after the controller is executed the first time. Instead, I'd rather just query the nodes to get their presentation data whenever necessary. That way, not only do I avoid persistence, I also only need to rely on one source for the data - the sensor node itself - eliminating a large class of bugs due to data going out of sync with respect to the persisted config file.
Some more details: Right now, my module has to do some trickery to persist data to disk, which involves an in-memory cache, and a timer to flush to disk. (This is necessary since presentation data can involve multiple packets, and issuing a FS write on every packet seems excessive, so some buffering seems appropriate.) To hide this behind the module's abstraction, the module does all the FS operations at a path that it deems appropriate, which is an uncomfortable choice. The module could ask the developer to specify a path for persistence, but that's yet another thing that developers have to do when using this module, with only a hard-to-communicate benefit, and that increases friction. Because we are dealing with the file system, the module also has to handle differences between Windows and Linux paths for cross-platform compatibility. Also, it isn't possible for two different apps to use the same module for the same network since data can get out of sync (the config is local to the app), unless the config files are stored in a shared location, which opens another can of worms (FS watchers or some form of IPC would be required, for example.). I could have avoided this FS jiggery-pokery by taking a dependency on a DB - the official NodeJsController, for example, requires mongodb - but that's an additional thing in the stack that needs to be up-and-running for the module to work, and requires additional configuration in the code to communicate with the DB anyway. The FS operations seem like a milder issue in comparison, in the usually-works-in-the-common-cases sense, but it would be great if I could avoid it altogether. After all, the sensor node knows its presentation, so it's much more authoritative to simply ask the node itself.
I hope all that made sense, and I haven't overstated the problem. A solution in my mind would be to REQ a node for its presentation data when needed, and the node could RES with its presentation. The docs don't say that this is possible, but my reading of the source IIRC tells me that might be coming in v2.0. Is that right? If not, is this something we can consider for future releases? (I would have contributed, but my C/C++ is very weak.)
@danylevskyi. Thanks. Not sure i can deal with the pressure, though.
Long time lurker, first time poster.
I've implemented a node.js module which you can include in your node projects, that does the dirty work of communicating with the serial gateway, and exposes a (hopefully) simple API to interact with your mysensors network.
Find the project on Github or NPM.
I've tried this with a Serial gateway, and it works fine. I haven't been able to test it with a Ethernet / WiFi gateway, but it should work fine with them as well.
Please feel free to use this in your code, if you like, and let me know if you have any suggestions/feedback (or post an issue on Github).