Posts made by ToSa

I had some time today to look at the bootloader code for RFM69 modules. Got the bootloader code to a point where it appears to be working fine - but the instability of the full MySensors RFM69 code is back - the serial gateway stops receiving packets after some time and it appears that the attempt to download a firmware by the bootloader just makes this problem more obvious due to the high volume of packets...
My prime suspect at the moment is the interrupt driven RFM69 library using a single packet buffer. As long is it's used sequentially everything should be fine but with e.g. submission or retrieval of ack messages it opens up the transceiver while the last packet was not fully processed yet potentially causing this... I'll go ahead and replace the routines in the library with the non-interrupt-driven version I coded for the bootloader just to see if this is the right place to dig deeper...

RFM69 is stable now for >48h.
The OTA bootloader version for RFM69 is in progress. I'm stuck right now where the node debug messages show that it's sending a packet but the gateway doesn't receive anything... I moved the full initialization code back in that I thought would be ok to skip but still the same behavior. Only difference to the full blown driver at this point is that I don't attach the interrupt but that should not be relevant for tx as I'm polling the interrupt register for status (ready to send / send complete etc.).

I have the RFM69 setup working now with a couple of additional tweaks mainly for code optimization. Working fine for some time but stops after a few hours - I'll have a closer look the next couple of days.
Felix added three bytes to the payload (source address / destination address / control byte) to manage the acknowledge etc. in the lowpowerlab library. For our specific needs this is redundant as the current message format in development already includes this data. The three additional bytes are not that bad though and I'd propose to leave it like that for now and reconsider when we get to the revised message format :).

@tekka
That sounds great! Can you post your code (or share a pull request)? This would either allow us to free up the remaining space or to add encryption
The only neck-breaker would be if any of the size reduction increases the risk for a bricked node that needs manual intervention (e.g. reset / power cycle etc.).

@lunarok said:

And last point, there is actually a internal message type Reboot. This one is indicated only working for OTA bootloader. Is that true ?

Reset and OTA bootloader are independent - but you need to set the fuse bits of the MCU to enable the watchdog for the software reset to work. This is done automatically when flashing the OTA bootloader but you can do that as well in a separate step keeping the existing bootloader (e.g. optiboot).

@kolaf
I've been busy with other stuff for some time but looking at the driver separation now once again. With nRF24 working (even bootloader adjusted to pick the right header files etc.) and two Moteinos at hand (Moteino RFM69W 868/915), I compiled and flashed a serial gateway and a simple temperature sensor node. Both appear to work fine looking at the debug messages except the fact that they don't listen to eachother. Both are sending but none is receiving anything (2m distance).
I did not touch or even closely look at the RFM69 driver code yet but will try to do so over the weekend. Do you have any hint where to check first?

@gregl the fuses determine how much space is reserved for the bootloader. I don't have the datasheet at hand but I think it varies from .5k to 4k. Optiboot is one of the smallest out there and the OTA bootloader consumes the full space because it needt to includs the (shrinked) wireless driver.

@marceltrapman I'm running almost all of my nodes (+gateway) using custom ATmega328 setup vs. using an Arduino (quite a few of them just on a breadboard for now). I did not try without a crystal. At the end you need to look at which operation is most time critical. Usually these are communication but SPI is not that time critical with the clock signal being generated by the MCU itself. I don't know for sure about e.g. one-wire if you want to use a DS18B20 or so. If you want a time aware sensor you probably should go with a crystal to get at least a little accuracy
For power consumption you should carefully check. I guess the ATmega based on the fuses turns on or off the internal clock - you might lose some of the lower power consumption again because now the ATmega itself consumes more...

@kolaf said:

maybe because I already have a fork of the main mysensors repository?

yes - Git doesn't allow two forks with the same base name (Arduino) for one user. We are about to move the changes into the mysensors/development branch. That way you can pick it up from there...

or a WLAN Gateway

@hek @John
using the "time.h" library on the node the time will continue to run. It will not be very accurate (as accurate as the "millis" in arduino) which should be sufficient to turn off the water after 15 minutes (no matter if that's a few seconds off) and good enough to let the clock tick further unless the next successful submission is received from the controller.

@larky said:

connection error - trying to reconnect, I think that is a good error!

Yes!!! Glad it was that easy

All - help me understand the use-case for a RTC on a sensor please.

From the discussion above I get that we might have two different needs:

• really high accuracy
• a second (or few seconds) deviation are acceptable

We lose accuracy in RF transmission already, therefore fetching the time on the controller or gateway and sending it to the nodes is not appropriate for the high accuracy need. A RTC would not help in this case because you need to tell the RTC the current time to start with and if that's already inaccurate, you won't get the accuracy needed. The only way to get real accuracy would be to fetch it on the node directly e.g. with a DCF77 receiver.
For the "less accuracy needed" scenario why would you add hardware to the node if receiving a time broadcast from the controller every x minutes is good enough to get the required (lower) accuracy "in software" without the need for hardware?

@larky
The package downloaded in this step:

wget http://nodejs.org/dist/v0.10.28/node-v0.10.28-linux-arm-pi.tar.gz

includes not just the "node" executable but the "npm" executable as well. Putting the uncompressed content in the right spot and setting the path appropriately afterwards lets you execute npm without the need for any additional installation step (same as for node). As far as I know the npm version that comes with the download in the installation instructions is v1.4.9.

@larky
I'm in a rush and can't look at the details but npm should not be missing. The installation of node the way it's described in the instructions wouold install npm as well. If you followed the instructions to add the path and the "node -v" command shows correct results, the same should be true for npm (I think "npm -v" would be an appropriate check for that). Once you changed /etc/profile to include the path you need to at least logout and re-login to ensure the changes are "active". When you do "sudo apt-get npm" you will get a very outdated version of npm (which is why I fetched node and npm from the nodejs.org homepage directly in the instructions).

From the error message above:

npm ERR! node -v v0.6.19
npm ERR! npm -v 1.1.4

The instructions are for node 0.10.28 and nodejs modules are version specific in a lot of cases (e.g. only valid from 0.8 onwards). When npm tries to download a module that doesn't exist for the node version you are using that would probably cause an error like this...

Happy to help you resolve this further over the weekend - maybe the hints above will already help you to fix it

@sundberg84
Based on what you describe I would propose that you make use of the MyGateway code. MyGateway essentially offers a way to communicate between the sensor network and the "outside world". With the three gateways available in the examples section that's either via serial (RS232) or ethernet (simple socket communication) or via MQTT (again ethernet but MQTT standard communication). You probably will end up combining the MyGateway class (or one of the gateway examples) with your code talking to nimbits and create the MySensors "NimbitsGateway"

@kolaf click the "here" in my post above - or use this:

https://github.com/ToSa27/Arduino/commit/328757d5bdb2681257326761fe1a0d90cdc3eba3

@kolaf it's only splitting the MySensor application layer form the underlying network layer. Still using the MySensor tree network topology and the RF24 driver. It's adding the framework to allow others to easily integrate other topologies (e.g. mesh) and other radio modules (e.g. RF69). The interface expects a reliable submission - no matter if that's achieved by software (like radiohead) or hardware (nRF24).
The MyDriver header file includes a short description of the routines and it should be fairly easy to attach it to radiohead. The main benefit is that the original MySensor tree setup still works as before if wished.

@ToSa said:

I think we should do one thing first: decouple the "network layer" from the "application layer" in MySensors:

• not having MySensors derived from RF24
• have a "generic" interface between MySensors and the radio

Initial version - tested with a simple DallasTemperatureSensor setup successfully here
Note that this fork/branch does not include the "1.4 update 1" changes for nodeID / CRLF.

Overhead is limited (~150b flash / ~20b ram):
Sketch uses 20,426 bytes (66%) of program storage space. Maximum is 30,720 bytes.
Global variables use 575 bytes (28%) of dynamic memory, leaving 1,473 bytes for local variables. Maximum is 2,048 bytes.

@Yveaux said:

So, anyone still developing on RadioHead integration, or did all of you just give up?

no progress on my end - but that's rather related to the fact that this was my first week back in the office after vacation...

I think we should do one thing first: decouple the "network layer" from the "application layer" in MySensors:

• not having MySensors derived from RF24
• have a "generic" interface between MySensors and the radio

That way it would be rather easy to make use of other radios and other network topologies - could be RH or something else. I've taken a shot at that (a lot of this based on what Kolaf and Yveaux did adjusting MySensors to RH) and will provide a link once it compiles and is tested and I hope I can keep the overhead small enough to be acceptable for the gained flexibility.

@Damme said:

I had to put my project in the trash bin.. There is not enough RAM in the atmega328 to fit mysensors and SD-lib Tried 3 different versions..Too bad..! I could only transmit one package before SRAM got overrunned.

Did you try not using MySensors but the reduced nRF24 driver version I used for the bootloader itself? If the node does not need to do routing and is expected to only respond to one or two types of messages, that might be an option and is definitely smaller...

@Damme said:

@ToSa I'll update mqtt later with option to not translate V_TYPE into name string but to keep it as a byte, that would save a lot of memory and might solve problem you have. I'll keep you posted

I could not follow the discussions the last few days - will definitely have a look over the weekend and let you know if that worked. Thanks!

@kolaf said:

Is it not sufficient that we define the radio at compile time in the sensors and the gateway as we have done now?

I'm talking about changing how the library is compiled and essentially having a separate ReliableDiagram class that bypasses manual handling of ACKs etc. and instead leaves reliable delivery to the radio itself. Mesh and Router are derived from the ReliableDatagram class - therefore we would need to define which radio to use prior to library compilation - not just prior to sketch compilation.

Didn't want to sound too negative yesterday...

• I'm still looking at the RH library but changes will be major and will not necessarily be in line with the RH purpose so Mike might not want to merge them into the main codebase: mainly defining what driver is used at compile time of the library (similar to MyConfig) which would then allow to e.g. "bypass" the majority of ReliableDatagram etc.
• what we should definitely do is splitting application layer and network layer: no longer inherit MySensor from RF24 but create an RF24 instance at runtime / leave application layer message handling in MySensors / move network layer message handling to a separate class (the "driver" for RF24). This will make it way easier to switch radios in the future.

@Zeph said:

I do not think it makes sense to move type (V_code) or command into the RH FLAGs. Typically when such space is reserved for the protocol in the header but not yet fully used, it's subject to change as the protocal evolves. If part of the goal is to take advantage of the maintenance and ongoing development of RH, we don't want to set up a conflict - because other users are not going to use those FLAGs our way anyway.

From the RH documentation / the source code:

• for Datagram FLAGS:
  A bitmask of flags. The most significant 4 bits are reserved for use by RadioHead. The least significant 4 bits are reserved for applications.
• for Router/Mesh FLAGS:
  Optional flags for use by subclasses or application layer

Not setting up a conflict with the RH code will be the biggest issue: the two benefits of RH are the multi radio support and the more advanced Mesh topology - which would be worth a couple of additional bytes in flash and ram - but the other reason for the big overhead is that several advanced features of the nRF24 chip are not used because they are not available in all supported chips.

Two examples:

• the multi-pipe capability of nRF24 would allow to filter traffic to only current address and broadcast and avoid that any other traffic ever reaches the MCU (only slightly less code but wondering if that helps with battery powered nodes).
• the ReliableDiagram processing in code essentially duplicates capabilities that are build into the chipset as well (auto-acknowledge / autoresubmit)

I know you are interested in getting RF69 supported but the question is how much negative impact for all the other users (with current nRF24 setup) is acceptable. That's like using the RF69 and implementing an AES encryption in code rather than using the build-in capabilities of the chip...

@Damme
interesting setup
to make it work with non-static addressed nodes you should probably keep the destination set to GATEWAY_ADDRESS for the REQUEST_ID call and only change afterwards.

Never mind - looking at the line number you mentioned that's probably what you did

@Damme
I need to better understand the setup to think about what's going on. My take from the above:

You have three nodes:

• Gateway (address 0)
• SD OTA Loader Node ?!? (address 254)
• Sensor Node (address 34)

Is that right?

@Yveaux said:

What radio/Arduino IDE did you compile for?

nRF24 / 1.5.6-r2
DEBUG turned on - your values appear to be with DEBUG turned off

I did a few changes starting to reduce the header so I can't rerun with DEBUG off and compare right now...

Test compilation comparing code size with current MySensors 1.4 radio library and with radiohead. The radiohead integration to MySensors is not optimized (pushing full MySensors header through even if the radiohead header has the same content etc.) and the radiohead library provides some benefits (not just a 1:1 replacament) - so it's expected to be larger. Based on the two examples below it's right now about 4-5k flash and ~800 ram for global variables.

=> let's see how far we can get that down reducing the duplication of header data etc.

**SerialGateway with MySensors: **
Sketch uses 17,566 bytes (57%) of program storage space. Maximum is 30,720 bytes.
Global variables use 688 bytes (33%) of dynamic memory, leaving 1,360 bytes for local variables. Maximum is 2,048 bytes.

SerialGateway with RadioHead:
Sketch uses 22,242 bytes (72%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,455 bytes (71%) of dynamic memory, leaving 593 bytes for local variables. Maximum is 2,048 bytes.

**DallasTemp with MySensors: **
Sketch uses 20,288 bytes (66%) of program storage space. Maximum is 30,720 bytes.
Global variables use 551 bytes (26%) of dynamic memory, leaving 1,497 bytes for local variables. Maximum is 2,048 bytes.

DallasTemp with RadioHead:
Sketch uses 24,592 bytes (80%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,319 bytes (64%) of dynamic memory, leaving 729 bytes for local variables. Maximum is 2,048 bytes.

@hek said:

Reply from Mike:

these are end-to-end headers and are not necessarily the same as the hop-to-
hop headers. In the general case they will be different values.

yep

In my view all these header are all necessary.

Well, debatable at least for the FLAGS (or the size of the FLAGS).

Three questions (@Mike, if you are reading here, just reply directly, otherwise I'll contact you via the RH channels...):

• is the RH library itself using the end-to-end FLAGS? I couldn't find any reference other than setting it to 0 (when using RHMesh).
• is the RH library itself using the hop FLAGS for anything other than identifying ACK packets (RH_FLAGS_ACK / RH_FLAGS_NONE)?
• for the hop ID and TO - if the specific RF chip supports this internally (as part of the preamble/header the chip adds to the on-air packet), why duplicating this wasting valuable payload size - the driver for the specific RF chip could handle it internally and make it look the same for the outside world using the RHGenericDriver interface without changes... I'll give it a try and let you know.

Thanks @kolaf @Yveaux.

makes sense - I thought "just" using the drivers at least as an interim step could give us a head-start but it appears that the majority of the "issues" with RH need to be fixed on the driver level anyways so we would still need to work these first with the RH team...

@Yveaux said:

Ok, read a bit through the docs. Apparently this is what is sent:

RHDatagram:  ff:7e:01:00     (TO:FROM:ID:FLAGS)
RHRouter:    ff:7e:00:52:00  (DEST:SOURCE:HOPS:ID:FLAGS)
RHMesh:      01:01:00        (ROUTE_DISCOVERY_REQUEST:<RESERVED>:DEST)

Therefore the destination and source address are sent TWICE (destination even 3 times with nRF24, as the destination address is part of the nRF24 header). An ID byte (incremented with each message sent) is also present in RHDatagram & RHRouter.
After the route is known, sending through RHMesh still requires 1 byte to indicate application payload is transmitted.
For short, Every message will require 10 bytes header, at least, to which the MySensors payload is added (including its own header of currently 4 bytes) giving at least 14 bytes overhead of 32 bytes total.
To me this feels like too much....

I think the addressing is not that bad: the TO/FROM is actually the next/last used in MySensors - only for a single-hop communication these are the same as SOURCE/DEST. With that FROM/SOURCE/DEST in RH == last/sender/destination in MySensors. The TO (next) is the one that is unnecessary because it's part of the nRF24 header - as this is not the case for all radios, this should be adjusted in the NRF24 specific code in RH and shouldn't be that hard to do.

The ID fields are used to determine if a packet is a duplicate (once for the hop and once end-to-end). I'm wondering how this is done in MySensors today. For the single hop that's probably part of the nRF24 internals (auto-acknowledge / auto-resend) and maybe it could be avoided on the driver level in RH but probably needs some tweaking of ReliableDatagram as well. For e2e I don't think there is an equivalent in MySensors today and it might actually be a valuable add rather than a waste of a byte what could happen without it: a node receives a packet and sends an ack - the ack does not make it to the sender and therefore the sender submits the same packet again - the node receives the package once again and thinks it's a new packet... think about a command that toggles a light - woulc toggle twice and go off->on->off instead of off->on.

The FLAGS bytes appear to be a total waste of space.

• Looking at the first FLAGS byte it's actually defined on the driver level already but not used in Driver or Datagram but only in ReliableDatagram and the only flag set is RH_FLAGS_ACK - a full byte for a single bit worth of information... The two options would be to either remove that byte and mark ACK packets differently or to just make use of the remaining 7 bits for other purposes. I would prefer using the 7 bits - protocol version (3 bits) is definitely a good fit - command (3 bits) might be the other. Using the lower four bits of FLAGS for application layer purposes is even foreseen in RH (and could be easily changed to use 6 bits for app layer in RHGenericDriver.h)
• The benefit of the second FLAGS byte (from the Router) is totally unclear to me. It seems like the neither RHMesh nor RHRouter code uses it at all even if it's defined on that level so nothing else should use it either... worth a discussion with the RH team - and either remove it completely or use it again for app layer specific stuff (MySensors type?). I'm surprised that they don't use these FLAGS instead of the first byte of RHMesh messages to determine route discovery vs. app date vs. ...

Assuming that we can get rid of the "TO" and the "driver level ID" and make use of the FLAGS fields as mentioned above the header would be:

• RHDatagram: FROM:FLAGS (covering MyMessage last / ack / version / command)
• RHRouter: DEST:SOURCE:HOPS:ID:FLAGS (covering MyMessage destination / sender / type)
• RHMesh: one byte to determine "application data" - tbc if this can be moved to the FLAGS instead consuming less than a full byte. at least the MyMessage "payload type" can be included in that byte
• MyMessage header: none (would need to talk to @hek but I think MyMessage::sensor should actually not be part of the header anymore but part of specific message types only).

This would leave us with 2+5+1=8 bytes instead of the current 7 bytes for the header (to be fair 8 instead of 6 due to the removed sensor field). That's not as bad as 14!

@hek said:

I wondoer if it would it be possible to implement a mySensors class similar to RHMesh which actually works like MySensors do today using a star-topology and direct-addressed messages?

@Kolaf said:

@hek Maybe we should just use reliable datagram. The ms header could contain just "source, destination, and last", and rhreliabledatagram deals with hop by hop ack and addressing.

@Yveaux said:

I would prefer to use the RadioHead library also for routing, but maybe the current implementation can be made more efficient, in terms of header length and also code space.
Currently I can barely fit in a simple sensor example and Ethernet gateway also gives warning about ram usage...

Just a thought (I did NOT review the entire RH code) - from the RH documentation it sounds like we could use the RH Drivers without using any of the RH Managers - keeping routing etc. within MySensors responsibility.
Just checking as it appears that @kolaf and @Yveaux have spent significant time reviewing the code and I would want to understand if that's an option or a totally stupid idea

@hek said:

RadioHead library does not store any routing info in eeprom so everytime a node is powered up it need to rebuild this.

That's beneficial in terms of available EEPROM space for other purposes - but would be interesting to see the shit-storm after a power-outage when all nodes power-up at the same time and need to rebuild the entire network tree from scratch?

@Damme said:

@ToSa wireshark dump https://www.wireshark.org/ or tcpdump http://www.tcpdump.org/

Thought you were referring to the "#define TCPDUMP" in MyMQTT.h

I did "sudo tcpdump host 10.0.1.99 -i eth0 -w dump.pcap" on the RPi that is running the OpenHAB runtime (10.0.1.6 is the IP of the RPi with OpenHAB / 10.0.1.99 is the IP of the MQTT gateway) and then did the following:

• power-on MQTT gateway
• start OpenHAB and wait until startup finished
• power-on node with DallasTemperature

The pcap file is attached (not knowing a lot about the protocol it looks like a series of keep-alive messages only and suddenly a connection reset): dump2.pcap

I do not see any messages arriving at OpenHAB... looking at the debug output from the node it appears that the "find parent" message is correctly handled by the gateway but the "node ID request" is not.

** OpenHAB output: **

pi@openhab ~/runtime $ ./start.sh
Launching the openHAB runtime...
osgi> 21:55:31.305 INFO o.o.c.internal.CoreActivator[:61] - openHAB runtime has been started (v1.6.0).
21:55:50.955 INFO o.o.i.t.mqtt.MqttService[:106] - MQTT Service initialization completed.
21:55:50.974 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'
21:55:57.178 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'logging.persist'
21:55:59.989 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'exec.persist'
21:56:00.137 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'rrd4j.persist'
21:56:00.303 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'db4o.persist'
21:56:01.162 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.items'
21:56:13.732 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.sitemap'
21:56:16.231 INFO o.o.i.s.i.DiscoveryServiceImpl[:72] - mDNS service has been started
21:56:31.603 INFO o.o.io.rest.RESTApplication[:143] - Started REST API at /rest
21:56:37.387 INFO o.o.u.w.i.s.WebAppServlet[:79] - Started Classic UI at /openhab.app
21:59:50.613 ERROR o.o.i.t.m.i.MqttBrokerConnection[:532] - MQTT connection to broker was lost
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
21:59:50.621 ERROR o.o.i.t.m.i.MqttBrokerConnection[:536] - MQTT connection to 'mysensor' was lost: Connection lost : ReasonCode 32109 : Cause : Connection reset
21:59:50.625 INFO o.o.i.t.m.i.MqttBrokerConnection[:543] - Starting connection helper to periodically try restore connection to broker 'mysensor'
22:00:00.633 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'

** The serial debug output of the node shows (more than that but always repeating as I did several resets of the node): **

sensor started, id 255
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
send: 255-255-255-255 s=255,c=3,t=7,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:
req node id
send: 255-255-0-0 s=255,c=3,t=3,pt=0,l=0,st=fail:

@Damme said:

@ToSa if possible you could make a tcpdump over the mqqt port and I'll look at it. Also a serial dump from the node would be great

After changing the include from Ethernet.h to UIPEthernet.h the sketch fits into the m328 only with DEBUG (and TCPDUMP) turned off:

DEBUG / TCPDUMP turned off:
Sketch uses 30,626 bytes (99%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,330 bytes (64%) of dynamic memory, leaving 718 bytes for local variables. Maximum is 2,048 bytes.

DEBUG / TCPDUMP turned on:
Sketch uses 32,354 bytes (105%) of program storage space. Maximum is 30,720 bytes.
Global variables use 1,393 bytes (68%) of dynamic memory, leaving 655 bytes for local variables. Maximum is 2,048 bytes.
processing.app.debug.RunnerException: Sketch too big;

The only thing I have right now to look at is the OpenHAB output (appears that it's not happening when I reset a node but rather random):

pi@openhab ~/runtime $ ./start.sh
Launching the openHAB runtime...
osgi> 19:54:19.384 INFO o.o.c.internal.CoreActivator[:61] - openHAB runtime has been started (v1.6.0).
19:54:39.316 INFO o.o.i.t.mqtt.MqttService[:106] - MQTT Service initialization completed.
19:54:39.322 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'
19:54:42.838 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'logging.persist'
19:54:46.082 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'exec.persist'
19:54:46.261 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'rrd4j.persist'
19:54:46.494 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'db4o.persist'
19:54:47.009 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.items'
19:55:01.187 INFO o.o.i.s.i.DiscoveryServiceImpl[:72] - mDNS service has been started
19:55:10.219 INFO o.o.m.c.i.ModelRepositoryImpl[:79] - Loading model 'test.sitemap'
19:55:19.042 INFO o.o.io.rest.RESTApplication[:143] - Started REST API at /rest
19:55:24.936 INFO o.o.u.w.i.s.WebAppServlet[:79] - Started Classic UI at /openhab.app
19:55:35.271 ERROR o.o.i.t.m.i.MqttBrokerConnection[:478] - Error starting consumer
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
19:55:35.298 ERROR o.o.i.t.m.i.MqttBrokerConnection[:532] - MQTT connection to broker was lost
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
19:55:35.305 ERROR o.o.i.t.m.i.MqttBrokerConnection[:536] - MQTT connection to 'mysensor' was lost: Connection lost : ReasonCode 32109 : Cause : Connection reset
19:55:35.310 INFO o.o.i.t.m.i.MqttBrokerConnection[:543] - Starting connection helper to periodically try restore connection to broker 'mysensor'
19:55:45.317 INFO o.o.i.t.m.i.MqttBrokerConnection[:112] - Starting MQTT broker connection 'mysensor'

@ToSa said:

I can confirm it's working with UIPEthernet.h and the ENC28J60 module as well. You need to turn off debugging in MyConfig.h to make the code fit.

Maybe that was a little too fast - it starts up and starting OpenHAB finds the gateway and connects. When I reset a node it appears that the gateway crashes (OpenHAB reports disconnect) and only a reset of the gateway helps to get it to work again. Problem is it's hard to debug without debugging enabled due to code size

I can confirm it's working with UIPEthernet.h and the ENC28J60 module as well. You need to turn off debugging in MyConfig.h to make the code fit.

@Damme said:

@Zeph I've been working on a read / write eeprom address thing in MQTT to be able to reset a node and stuff. But it seams there are more usage for it then. This might be coded into mysensors instead. (utilizing c_internal or somthing as the protocol is today)

Good idea - that would allow to check for current value in normal operation - not just during bootloading.

@Zeph
If you urgently want to have the CRC of the current firmware submitted during bootloading, we can add this as a third parameter to the FirmwareConfigRequest message. Actually I was thinking about getting rid of request/response and use the same format for both which would mean crc would be included anyways.

@Zeph
from the MyOtaBootloader.c:

if (firmwareConfigResponse->version == fc.version)
	if (firmwareConfigResponse->blocks == fc.blocks)
		if (firmwareConfigResponse->crc == fc.crc)

so as long as you send the same version / blocks / crc back to the node as what iscurrently installed, no update is started. As soon as one of the three elements differs an update is loaded. It's completely in control of the server if (and which) firmware is bootloaded.

@Zeph
yes, that's what I meant - you might not even need any fork of the bootloader itself and just a slight adjustment on the controller end - because the nodeID is contained in the packet (not in the payload but in the header as sender address) so you have all you need for your setup

@Zeph
What I don't like about Flash is the need to erase before writing / the ability to erase only full sectors (4k if I get it right from the WB datasheet). Not a big deal for bootloading or logging where you can erase all upfront but limits the use-cases.
What I really like is that a lot of the SPI Flash chips and EEPROM chips have the same pinout - which means depending on the specific need you can use a flash or an EEPROM using the same PCB layout (example EEPROM that allows single byte writes without erase: 25LC256 - obviously less memory and higher price but serves different purpose - 256kbit for <1€)

So I really like the idea of having this added to the PCB layout - especially because it can serve different needs depending on what chip is selected

@hek congratulations indeed
nice surprise looking at the forum after a 10 hour drive back home...

@Zeph said:

Each combination of sensors and pin assignments has a unique "node type" (within a given wireless network).

Actually that's part of the question - as @hek mentioned there is a desire to sell MySensors hardware - at some point there might be not just generic pinhead PCBs but real fit-for-use devices. Ideally these would have a unique node type assigned not just within a given network. New firmware could be published on mysensors.org (or via codebender or...) and based on the unique (but common across networks) node type less tech-savvy people could be secured from sending a firmware that doesn't fit the hardware... I know - a LOT of "IF"s...

@Damme
you are right - probably not the full 900 bytes but additional space could be used for encryption etc. so every reduced byte is beneficial at this point. I'll check later how much can be saved by using CRC8 instead of CRC16.
I'm already using a mini version of mysensors / mymessage: not using the cpp code files at all but just the headers and if you have a look at the "#ifdef __cplusplus" statements just added for that purpose, there is almost nothing left (the MyMessage class is stripped down to a struct and the MySensors class removed completely / enums and #defines should not consume space after compilation)

@Zeph said:

Have a look at this little fella: http://www.digikey.com/product-search/en?mpart=W25X40CLSNIG&vendor=256

Any proposals for DIP-8 flash to use for breadboard testing that are compatible to avoid surprises when switching to the SOT-8 ones later-on? Do all of these SPIflash chips use the same "protocol"?

@hek any plans to setup a "dev_1.5" branch or something similar? If we continue adding major functionality changes to 1.4 it will never get "ready"

@Yveaux

This is a request for ideas before I start coding any of this!!!

time awareness was straight forward based on existing functionality (even if I would still argue that it should be an integral part of the sensor code and not require separate code in the custom sketch).The rules engine requires much more prep work and smart ideas.

In an earlier project I ended up with the following (not saying that this is the best approach - just to start the discussion):

• simple memory manager assigning memory addresses so that you can remotely request or set a variable in memory the same way you do today with sensors (think about a memory address having its own sensor-id on the node)
• rules to set variables on the same node or on remote nodes (or on the gateway/controller)
• rules triggered by schedule or triggered by events (button clicked / temperature changed / ...)
• no need to trigger rules remotely - you can get that by setting a variable remotely and have a rule listening on changes to that variable (node A with the light switch telling node B with the relay to turn on the light)
• gateway is always made aware of what's going on but usually not required for the execution

The problem is that the "setup" of the network and the specific nodes gets way more complex - easy to do for software developers but hard to do for a "normal" user.

@JeJ one potential reason is that the port is already in use. I mentioned somewhere that the startup script doesn't yet stop the NodeJsController correctly. Maybe you already have a NodeJsController process running? Try "sudo killall node" and then try starting it again. To check if the port itself is working you can try to open a simple terminal (minicom etc.) and reset the gateway.

@Yveaux Right - I mentioned the garden watering in another post - I want the sensor to be aware of the schedule to reduce the single points of failure to a minimum - and avoid bad surprises getting back home after vacation That means the sensor is able to execute even if the controller is broken or the gateway is broken or the network connectivity is broken due to a relaying node in the middle etc.

@epierre You lose accuracy during RF transmission to the nodes, therefore I didn't go for DCF77 but told the controller (RPi) to fetch the time via NTF from the internet. If you really need DCF77 then have a look here - should be pretty easy to implement on the gateway and send to sensors from there or even on specific sensor nodes that need very accurate time.

@mikeones said:

dev/ttyUSB0

Then /dev/ttyUSB0 is correct. /dev/ttyAMA0 would only be valid for the on-board serial port on the GPIO pinheads.
Are you running into any issues once you set the port in NodeJsGateway accordingly?

@Damme said:

@ToSa I've been looking through the ota bootloader and noticed there are alot of uint16_t wich can be replaced with uint8_t.. saves 128bytes of code. Still needs ~900bytes less until 1024 words bootloader though but is makes more space for other stuff

I'll have a look. I've taken the code from an earlier project and adjusted to MySensors - didn't review the variable types that much. I'm using CRC16 as well where CRC8 might be sufficient...

EDIT: got it down a little from 0x0E18 to 0x0DD0 (72 bytes) changing a few loop counters from uint16 to uint8. I don't want to change type to 8bit looking at the large amount of sensors people are asking for / working on. FOr version I'm planning to keep some of these running for a long time with as little maintenance as possible. With some software improvements over time and minor version changes during development 16bit for version seems to be the better fit as well.

As started here.

Looking at the TimeAwareSensor example essentially everything is there for time synchronization. Once the callback is set with the initial call to requestTime, any subsequent time packets received from the gateway would be used to update/adjust the internal time accordingly.
The only adjustments needed to get to a real time synchronization would be to let the gateway (or rather the controller) broadcast the time every x minutes and ensure that the sensor listens to the time packets even if they are not addressed to the nodeID but broadcasted (today only TIME packets addressed to the specific node are handled).

I've done these changes here and will do some testing.

To schedule events the question is how to store the schedule with a minimum footprint in EEPROM. Keeping hour:minute should be sufficient. Storing these in a byte each is a waste of space (only 5 bits for 0-23h and 6 bits for 0-59m needed or 11 bits for 0-1439m). Looking at usual scenarios you need a bitmap for the weekdays (schedule for Mo/We/Fr only) which takes another 7 bit. With that three bytes are minimum and leave 6 bist for additional purposes.
For home automation you would usually want to have a way to tell if the event should be executed on a public holiday (1 bit) and you want to have a way to tell if it should be executed when you are on vacation (1 bit) - I'll explain in a later post how public holiday and vacation could be determined.

Each event should probably include some kind of payload (what to do). To keep this simple I'd propose a single byte payload and let the callback function in the main sketch decide what to do with it.

This is a request for ideas before I start coding any of this!!!

@Damme said:

@ToSa I've been working on getting OTA to work with MQTTgateway with some success.

great!

[bootloader] 0000 has CHK FF(just filler in first package) REQ 0000 type 01 version 01
[server] load 0000 from hex, send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK FF, REQ 0010 type 01 version 01
[server] (checksum mismatch) send addr 0000 0C9428030C9447240C9474240C947605 C7
[bootloader] 0000 has CHK C7, REQ 0010 type 01 version 01
[server] load 0010 from hex send addr 0010 0C94A3050C94D0050C9480100C945003 00
And so on..

what do you think about this? the total package is 32bytes, mysensors header is 7bytes. and this layout would need 19 bytes from server to bootloader..

If I understand correctly you would send the CRC of the previous bloak back to the server together with the request for the next package - the server would then send the next block if the CRC is correct or resend the previous block if the CRC is not ok...
I'm wondering how the bootloader would ever run into that situation. The package itself is checksum'ed already and wouldn't be treated as correctly received package if the checksum is incorrect. Only if the previous block was received correctly the next block is requested. If that doesn't happen within a given amount of time the same block is requested again.
Can you explain a bit further what issues you are running into?

@Damme said:

I Have also seen some intel hex that is not in order 0010 0020 0030 etc but it could jump address. I do not think arduino ide does this but you never know..

Yes, I've seen that as well - actually Codebender does that sometimes and the "HEX file loader" function on the NodeJsController would need to be adjusted to address that (I'm not reading from the .hex file when a package arrives but from a byte array in Mongo)
Probably the best approach for this one would be to add the next address the bootloader should request to the previous package (e.g. "here is the data for 0x0000 - next request should be for 0x0010) and allow for variable block length because that's another one where Codebender sometimes uses <16 byte rows in the hex (not just at the end). The code on the bootloader side would get a little more complex as the flash is still written in pages and these might not line-up anymore with the 16byte blocks.

@Damme said:

I havn't read this one yet but I guess there is alot of good stuff in it
http://www.nordicsemi.com/eng/nordic/download_resource/10878/2/94069421

That chip is a totally different animal - same RF but 8051 MCU. I'll have a look at the AppNote tomorrow.

@JeJ @mikeones : how is your serial gateway connected? using a USB-Rs232 cable or via the GPIO pins on the RPi? Did you check the Readme.md in the NodeJsController directory?

Topic can be closed - while I still don't understand why this 16x repetition was reproducible consistently between these nodes and always only affected the third packet, it's now no longer reproducible and for now I'll just assume it has been a connectivity issue (even if nodes were sitting with <1m distance and nothing changed)

@Yveaux Yeah - I'm not really good at reading manuals
I've submitted a pull request in GIT for your MySensors2.dll code - nothing major but just adding the latest additions to the enums and adding the stream types so that they show up as plain text in Wireshark as well... I could not test it because I ran into some issues during compilation - probably not the right toolchain installed.

@Yveaux said:

@ToSa @hek I think that all your questions are answered in my blog. I'll get back to this later.

Got it
When fixed payload size of the sniffer is set to a value (much) larger than the actual payload size in the packet, then subsequent packets might be missed. When the nRF24 is still capturing the fixed size payload and a new packet arrives, this will not be detected. A good example for this are the auto-acknowledge packets which can be enabled in Enhanced Shockburst mode. After the target node receives the packet (and it will know the real size of the payload a lot faster than the sniffer will) it switches its radio from receiving to transmitting mode in only 130us. Then it sends out the acknowledge packet. Reception of 32 bytes at 1Mb/s takes 32*8us = 256us, so acknowledge of a short message will definately be missed. For acknowledge packets this isn't much of a problem as the transmitter will retry to send the identical message when no acknowledge was received and we will know it missed the acknowldge.

@ToSa said:

as it's sniffing "on air" in theory it should see all messages including the ACKs...

I take that back - probably the nRF module connected to the sniffer will assume it's an "internal" message and not publish it to the MCU.

@hek it's the full low level packet including the address etc. For the first packet in the screenshot above it looks like this:

setRetries(5,15) would be a fit in terms of # (1 original + 15 retries) but based on the RF24 library documentation the first parameter of 5 would mean (5+1)*250us -> 1.5ms delay between retries

The PIDs don't increase. For the screenshot above messages 1 and 2 have PID=0 and messages 3-19 have PID=2, message 20 has PID=3.
Message 21 starts with PID=0 again which makes sense because the nRF is re-configured between bootloader execution (messages 1-20) and firmware execution (message 21+).
All messages show
.... .... 0... .... = No ack: ACK (0)

@Yveaux do you suppress the ack messages in the sniffer code? as it's sniffing "on air" in theory it should see all messages including the ACKs... Is there a parameter I can set on startup to see the ACKs as well? Maybe I've set the AutoAck incorrectly in the initialization code in the bootloader but then I would be surprised why it works for packet #2 which is just submitted once.

For a different purpose I just looked into the sniffer @Yveaux developed (btw - I'm deeply impressed )
While the sniffer works like a charm the very first sniffs showed that a message arriving at the serial gateway with ReqAck:1 leads to 16 IsAck:1 responses from the gateway to the original sender. This is using the most current development branch. Need to leave now - if somebody else wants to debug then please go ahead - otherwise I'll look into it later today or tomorrow.

@Zeph said:

@ToSa

Did you by any chance find out whether it's really limited to 83 channels vs the 127 of the nRF24L01+?

I don't remember that part, sorry. I played with these radios about two years ago

If anybody is interested in the code I've re-uploaded it to GIT. It's pretty much un-documented and I would need to look at it to answer detailed questions but it's including some of the features we just started talking about (time/date awareness / pin-by-pin configuration vs. node type / ...). The big difference to MySensors is that it was meant to run one and the same firmware on all nodes no matter what the hardware configuration looks like and the firmware is highly configurable... I still have some documentation about that part if anybody is interested... I stopped the development for two reasons: not enough time and the limitations of the hardware. For a setup like this you probably need to move to something like the ATmega256RFR2.

https://github.com/ToSa27/RfMesh

Nice! I'll have a look at the details later - thanks for the link!

From a brief first look it appears that they can't fit the code into the bootloader which is why the main firmware downloads the new firmware to an external flash first and then the bootloader picks it up from the external flash and loads it into the internal flash.
I personally don't like that approach for two reasons:

1. need for additional hardware on every node just for OTA bootloading (external flash)
2. less fail-save: with the bootloader executing the download you can always refresh a node even if the actual firmware has a bug that prevents it from working properly (worst case with a power-cycle to activate the bootloader). With the main firmware executing the download you would need to connect an ISP and locally program either the external flash memory or the AVR directly.

@Zeph said:

But suppose we DO want to go further and determine accumulation time for all nodes, even the above.
...
System Time approach 1: Master Clock Broadcast to nodes
...
System Time approach 2: Track accumulation times in the hub

I'm in favor of approach 1 as well - and an optional addition to the MySensors class to keep track of time. Only specific sensor types would use this feature - interesting mainly for actuators rather than sensors:

Let's assume you use a relay to turn on/off your garden watering while you are on vacation. The network breaks (controller or gateway or whatever node needed to relay messages).

• If the sensor node itself knows about the time and the schedule for watering, it can turn on and off the watering itself. If there is an issue with the communication, the time would not be 100% accurate over time but still good enough.
• If the sensor node relies completely on the controller for time and (worst case) the communication breaks just between the start watering and stop watering command you can imagine what happens...

Knowing about the time heavily reduces the single points of failure. The protocol already covers the "request time" communication so this should be relatively easy to implement...

• controller sends a time broadcast every e.g. 15 minutes
• only the nodes interested in time care about it and update their internal time
• when a node needs the current time it calculates it based on the last timestamp received, the millis() when it was received and the current millis...

Doesn't solve the sleep issue (as long as you don't want to submit a separate "get time request" every time you wake up) but at least for sensors that are not battery powered (relay nodes in most cases are not) it's a feasible approach.

This is probably worth a separate thread

@hek said:

This would only leave 3 COMMAND types left. REQ, SET and STREAM. Where STREAM-data probably could be handled by S_NODE (internal). This means just 1 bit is needed for a SET/REQ flag. But @ToSa would have to feedback on this.

I switched the bootloader code between using stream and using simple internal messages - at the end doesn't matter as long as it's simple enough to dis-samble and assemble a packet without a lot of additional code overhead.

@Yveaux The main purpose of the OTA bootloader is to avoid the need to grab the node and connect it to a PC for an update but instead do the update "in-place". Using the existing "infrastructure" to relay messages is a must as we don't want to implement separate code in each relaying node to handle the bootloader messages separately.

@Zeph not sure what you are asking for as I mentioned above that you can use the bootloader as it is today to just update specific nodes (by nodeID, update one and not update another even if they have the same node type). The implementation is not a "pull by node type" but it's a "pull by node ID, node type, node version" - which information the controller uses to decide if an update should be executed is is up to you!!!

Terminology: the "node type" I'm referring to means the specific setup of the hardware - only if that's the same then the node type would be the same (combination of sensors / pin connections / same sketch to be used). The back-end cares about a node type because it needs to know which sketch to use/send.
The user interface ideally never cares about a node type but really cares about the specific sensor type(s). This "translation" needs to happen in the background no matter if you use an OTA bootloader or not.

Examples:

1. Let's assume you have two nodes in the living room - the user interface should just show "living room temperature" no matter if the temperature sensor is connected to node 1 together with the light switch or connected to node 2 together with the blinds. This "translation" needs to happen anyways - ideally in the controller.
2. Let's assume you have two temperature sensors connected to one node - one measures the room temperature at 1.5m height and one is a floor temperature (not unusual for floor heating). Just knowing that there are two temperature sensors but not knowing which one is which will not be sufficient for the heating controller to make the correct adjustments. Again that translation from "node 23 with one XYZ temp sensor on pin 5 and one XYZ temp sensor on pin 7" to "node 23 temp sensor at pin 5 is the floor temperature" needs to happen anyways.

@Zeph I would need to re-write (or at least adjust) that part of the bootloader. Would need to see what radiohead does to the header before I can judge if it's still possible to fit it into the limited bootloader space but at this point I'm pretty positive... current bootloader size is 3608 of 4096 bytes and maybe there is still potential to reduce it further. I was planning to use the remaining size for encryption though which might or might not be feasible with 400 bytes only...

The bootloader would never use the radiohead library due to its size but would mimic the header format. If you are looking for multiple radio modules, the effort to reduce the "driver" to fit into the bootloader section would be unique for each radio module type

@Damme look at NodeJsController/Readme.html - actually for now better look at this version which has a couple of updates (will send another pull request tomorrow for the documentation as well as some minor changes).

If you are looking for tech documentation (protocol etc.) that's not yet included but the communication is fairly easy (complexity is mainly to make it robust - not kill a node if something goes wrong etc.):

• the bootloader is using the same procedure to find its parent / request a nodeID etc. as a normal MySensors sketch would do
• then a config request / config response is exchanged between node and controller
• assuming an update is needed a series of code block requests / responses is executed until the full firmware is submitted

Data is submitted as binary - you can see the message payload details in MyOtaBootloader.h:

typedef struct
{
uint16_t type;
uint16_t version;
} FirmwareConfigRequest;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t blocks;
uint16_t crc;
} FirmwareConfigResponse;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t block;
} FirmwareRequest;

typedef struct
{
uint16_t type;
uint16_t version;
uint16_t block;
uint8_t data[FIRMWARE_BLOCK_SIZE];
} FirmwareResponse;

@Zeph: yep - that means in your case you don't really care about the node type. In other scenarios where you have 60 nodes installed and 20 of them are relay nodes, another 20 are switch detectors and another 20 are temperature sensors (all of them having the same hardware setup) the node type is pretty useful. For your specific need you probably should not care about the node type at all - maybe set node type == nodeID and that's it. The additional 2byte payload should not matter too much.

The ideal setup from my perspective would look like this (dreaming): based on the information shared back (combination of sensors and pin connections) the controller would reassemble the source code and build a new sketch for the given configuration, compile it and send it (I'm not kidding - I worked on a very similar approach a few years back).

Reality is: this is meant to be a bootloader for MySensors. The way MySensors currently works is that the combination you mentioned (18B20 temp sensor on pin 7 and power blind relays on pins 5 and 6 and an IR detector on pin 12) requires a specific sketch to be loaded that has these pin assignments etc. hard-coded.

This is the piece of code you would want to adjust - at this point it pulls all available firmware records for the given type and sorts descending by version - which delivers the highest available version back as the first record:

db.collection('firmware', function(err, c) {
	c.findOne({
		$query: {
			'type': fwtype
		},
		$orderby: {
			'version': -1
		}
	}, function(err, result) {

Instead the "expected firmware" type and version could be an attribute for the given node in the "node" collection which is manually maintained:

db.collection('node', function(err, c) {
	c.findOne({
		'id': destination
	}, function(err, noderesult) {
		db.collection('firmware', function(err, c) {
			c.findOne({
				'type': noderesult.expected_firmware_type,
				'version': noderesult.expected_firmware_version
			}, function(err, result) {

@Zeph It actually depends on how that "somewhere" I mentioned in the initial description is coded.

The bootloader sends a message to the controller like "I'm node 23. I'm a temperature node and I'm currently running version 5 of the temperature node sketch which has a CRC of 0xABCD"

The controller sends something back like "You should be running version 6 of the temperature node sketch with CRC 0xFEDC"

So it's truly the controller (the central authority) that decides. At this point what I've done in the NodeJsController (which really is pretty dump and only meant for testing) is that I did not care about the nodeID bud only submitted a response based on the latest version available in the database for the given node type. You could obviously maintain a list of "expected sketches / sketch versions" for each nodeID and drive the decision on what the controller sends back based on that list instead of the node type only.

It really does exactly what you want it to do - the "pull" truly is a "pull for information if the central authority wants me to update". The big benefit of this "pull" setup is that the controller is stateless and just answers each request coming from the node making the code way cleaner and the overall setup way more reliable.

I used them a long time ago and they actually never left my desk where I did initial coding and debugging so I don't have real world range experience as far as I remember from the specs the range should be slightly better than the nRF24.

@Damme that's exactly how it works - and yes, the soft reset (using the watchdog) executes the bootloader and asks the server if a new version is available.

Nobody ever mentioned it would be fast
Takes a couple of minutes to load the DallasTemperatureSensor sketch that I user for testing - but depending on where your sensor sits that's still less time than going two stairs up, moving that big cabinet to the side, getting the sensor out going two stairs down again, dissembling the enclosure, connecting it to the PC/Mac, flashing the new firmware and then all of this in the opposite sequence to get it back where it belongs...

The OTA bootloader was merged into the development branch some time ago. It consists of two components at this point: the OTA bootloader itself and a quick&dirty NodeJSController that connects through a standard SerialGateway or EthernetGateway and is used as repository and sketch distributor for the sensors.
I've created another pull request just now to include a couple of additional tweaks/fixes and an installation guide to get you started (NodeJsController/Readme.html).

I just tested using the DallasTemperature sketch. It picked up the MySensors libraries that @hek uploaded to codebender a few days ago and compiled successfully!!!
Are all other libraries that are used by MySensors unchanged and up-to-date or would we run into issues with specific changes done in mysensors/Arduino/libraries/... compared to the original libraries that we would find in codebender? If so, uploading the changed versions as private libraries might be an option but I'm not a codebender expert

I've used these in the past - mainly due to their size (half of the nRF24 or even less). They are "almost" compatible but you need to tweak the code in some areas to make it work - mainly the initialization differs quite a bit.

@donnib - I didn't continue this way but instead started using node-js based controller code that connects to a standard Ethernet gateway. @hek merged the code about two weeks ago. I will provide a tutorial on how to set it up from scratch shortly...

Quick update: I have the low level hardware access code ready (ability to communicate with the nRF24 without the library as the library is too big for the bootloader) and most of the other arduino side boodloader code as well. The raspberry pi side of the story is behind as binary data submission and a database layer are a prereq. I started based on the initial mongodb setup in the 1.4 dev branch but not sure if that's the strategy longer term.
I had some initial success testing the bootloader with some dirty hacks on the raspberry side (removing all debugging that would fail on binary data / removing the handling of trailing 0 etc.) when my hardware started to fail. I replaced the arduino / the nRF24 on both ends and even the raspberry Pi - without success... loaded old code that I knew was working on both ends and it still doesn't work... Both Arduino and RPi seem to work fine but once the first packet arrives from the Arduino to the RPi it reports retrieval and then is stuck unless I reboot the RPi... I'll retry once I'm back from China in two weeks - don't expect to hear anything from my end in the meantime as I won't able to take any hardware with me.

@axillent : the universal bootloader is great but would not be able to utilize the infrastructure (routing / packet format) to communicate and hence would not allow to update sensors that are out of reach for direct communication to the central node providing the updates (gateway or separate).

Independent from z-wave I like the idea to abstract the RF hardware layer - similar to a device driver that is picked at build time. The interface should be relatively simple (init / send / check if received / receive).

Yes, that's exactly how you would do it - the defaults only apply if the function is called without parameters.

From my pov that's one of the biggest benefits of ota updates : you can do updates "in place" without the need to move the sensor towards the gateway or the other way around.
If we use the same message structure, the additional complexity is limited: gateway and relay nodes know how to deal with it and the only additional step for the sensor is to find the correct relay address. Error handling (switching relay during ota update etc.) would be limited or not existent keeping the bootloader as small as possible - if something unexpected happens like a disappearing relay during the update, the entire update would fail, the sensor reboots and tries again.

I need to transfer binary data for the ota updates, to keep the packet rate at a minimum - but the code is currently tailored for text. Especially using \0 for message termination and printing message directly in e.g. debug statements won't work. As this is pretty deep in the Sensor class reused all over the place I'm a bit hesitant to change (e.g. Sensor::readMessage()). This wouldn't be an issue for the bootloader as I won't use the Sensor class anyways (too big) but for the RPi the entire chain would be used: RadioGateway <- Gateway <- Relay <- Sensor ...

Now that the Arduino and the RPi talk to eachother (http://forum.mysensors.org/topic/8/code-for-beta-testing#27) I'm looking into ota updates.

The bootloader needs to know what firmware to request / what firmware the sensor is running - finally it needs to know what hardware is build into the sensor to load the correct firmware update. One option would be to have a separate bootloader for each sensor hardware - but that would be a nightmare to maintain over time.
Looking at the code the sketch info provides this kind of detail - but the sketch info is part of the firmware which ends up in a catch 22. We need to store the sketch info in EEPROM at an address known to the bootloader as well as to the firmware later on. Instead of storing the sketch name I would recommend to store some kind of a sensor type ID to reduce EEPROM consumption - sensors with the same hardware would get the same sensor type ID. The ota bootloader could then check with the firmware provider if there is a new version firmware for this kind of hardware...

I would propose the following: instead of storing separate bytes for address / relay / ... in EEPROM, create a struct and store that struct in EEPROM - load the struct from EEPROM to RAM on startup and check if the crc is valid... The most sophisticated approach would probably be to add a kind of unique MAC address to each sensor and to separate hardware specific and software specific configuration - something like:

struct HardwareConfig {
uint16_t hwType;
uint8_t hwVersion;
uint8_t macAddress[6];
uint16_t crc;
};

struct SensorConfig {
uint8_t address;
uint8_t relayAddress;
uint16_t fwVersion;
uint16_t crc;
};

Rough bootloader code could look like this;

--if hardware config is not valid (crc)
----// this is the probably the very first startup of the sensor after flashing of the bootloader
----send "who am i" request to firmware provider
----wait for "who am i" response from firmware provider
----store response in hardware config -> EEPROM
--if sensor config is not valid (CRC)
----// this is equivalent to the current implementation of "no radioID in EEPROM"
----request radio ID (same way as implemented already)
----store response in sensor config -> EEPROM
--send "what is the latest firmware for hardware type x" request to firmware provider
--wait for "what is the latest firmware for hardware type x" response from firmware provider
--if newer firmware available
----invalidate firmware version/crc in EEPROM
----for each block (small chunks due to max packet size)
------send firmware request to firmware provider
------wait for firmware response from firmware provider
------write to progmem
----write new firmware version/crc to EEPROM
--boot into sensor firmware

Any comments/suggestions?
If none then I'll share some code in the next few days to get the ota going - will start with the most sophisticated and then reduce / adjust if needed e.g. to get the code small enough to fit into the bootloader section

RPi and Arduino now talk to each other, the Arduino receives the radioId from the Rpi and stores it in the EEPROM and the Arduino presents itself to the network. I still see some timing issues (ACKs too late / replies too early while the other end is still waiting for an ACK / ...) but in general it works.
To get there I forked both the Arduino and the RPi project and tried to get to the latest and greatest code by merging some of the branches (mainly master and 1.4dev)... When I ran into additional issues due to the RPi code not getting the latest updates and bug fixes available in the Arduino project, I decided to merge across the two and ended up with a common set of RF files: a common codebase for RF24 as well as Sensor/Relay/Gateway. If any of this is of interest for themain development then please let me know and I'll create a pull request.

What I did in detail:

Arduino:

• started new branch tosa-dev based on latest master branch
• adjusted the message header to use full bytes as mentioned earlier in this thread
• no need to merge the 1.4dev branch as the message header was essentially the only remaining relevant change in there
• no need to merge other branches as the majority of these were already merged back into master

Raspberry:

• started new branch tosa-dev based on latest master branch
• merged the 1.4dev branch into tosa-dev
• a couple of minor tweaks (mongoDb URL / start and stop scripts / data rate / ... majority listed earlier in this thread)

When I found the missing Relay::sendData() routine I decided to stop tweaking bit by bit and instead looked at what it would take to use the more advanced Arduino code for RF24 and Sensor/Relay/Gateway for the RPi as well. Based on comments in the forum this seems to work for several people using a SerialGateway or EthernetGateway so my expectation was to find well maintained code there avoiding some of the low level communication issues I ran into so far with the RPi code. This code is available in a separate branch "tosa-dev-commonrf" for both Arduino and Raspberry.

The code is not perfect yet but it works. I'm not a GIT expert but as far as I can tell the fork is public and everybody should be able to see it:
https://github.com/ToSa27/Raspberry/tree/tosa-dev-commonrf
https://github.com/ToSa27/Arduino/tree/tosa-dev-commonrf

Again: happy to create pull requests if you want to use this setup as part of the master branch at some point. The main reason for me to do this is to get a solid foundation to start working on the OTA update code...

@BMLSX - probably rather saved your stomach - eating the raspberry sounds noxious

I got one step further (maybe) - the Arduino now receives the radioId... looking at the log from the RPi above I stumbled over the RPi sending the data to the gateway ??? the RPi is the gateway !!! Comparing the code to the SerialGateway which seems to work for a lot of folks I found that the sendData() function is missing in the Relay.cpp code used for the RPi. Once that procedure is in place the RPi actually submits the new radioId to the sensor:

RPi : now "sent via 255" instead of "sent via 0"
Dynamic payload size=9
Received: from=255, to=0, childId=255, mtype=4, type=5, crc=65, ''
Message crc ok.
header.type=5, header.to=0, radioId=0
Got message
Message type: 4
Writing back message
msgTypeId=4 int=4
Sending data 255;255;4;5;1
We have input
Nread=14
INP:255;255;4;5;1
.
INGOT radioId=255, childId=255, messageType=4, type=5, value=1
Sending data to GATEWAY
No route... try sending direct.
Sending: from=0, to=255, childId=255, mtype=4, type=5, crc=11, '1
', sent via 255
radioId: Inserted id=1
Send failed. No ack received.
Sent to JS

Arduino
Started sensor.
Open ping reading pipe: 255
Tx: fr=255,to=255,la=255,ne=255,ci=255,mt=4,ty=9,cr=235:
Message available on pipe 1
Rx: fr=0,to=255,la=0,ci=255,mt=4,t=10,cr=106(ok): 0
Message addressed for this node.
Using relay 0. Distance is 0
Relay=0, distance=1
No radio id found in EEPROM fetching one from sensor net gateway
TH: get: n=0,c=255,st=4,rt=4,vt=5
Relaying message back to gateway.
Tx: fr=255,to=0,la=255,ne=0,ci=255,mt=4,ty=5,cr=65:
Ack: receive timeout
Message available on pipe 1
Sent ack msg to 0
Rx: fr=0,to=255,la=0,ci=255,mt=4,t=5,cr=11(ok): 1

Message addressed for this node.
Radio id received: 1
Radio id stored in EEPROM was: 1
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=255,mt=0,ty=17,cr=17: 1.3b3 (7afb55c)
Ack: receive timeout
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=255,mt=4,ty=7,cr=176: 0
Ack: receive timeout
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=255,mt=4,ty=14,cr=21: Temperature Sensor
Ack: receive timeout
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=255,mt=4,ty=15,cr=13: 1.0
Ack: receive timeout
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=0,mt=0,ty=6,cr=99: 1.3b3 (7afb55c)
Ack: receive timeout
TH: get: n=0,c=255,st=4,rt=4,vt=13
Relaying message back to gateway.
Tx: fr=1,to=0,la=1,ne=0,ci=255,mt=4,ty=13,cr=200:
Ack: receive timeout

I'm still not sure if the missing ACK for the majority of the messages is by purpose or by accident... would be relatively easy to fix by either not waiting for an ACK for all messages or by always sending an ACK - but I don't want to mess with this code without knowing why it was implemented this way.

Changing the mongo database URL in the dbDetails,js file
from:
mongodb://mysensors:27017/pi
to:
mongodb://localhost:27017/mysensors

fixed the "Error looking up radioIdo" issue on the RPi. Now the Rpi generates a new radioId but the attempt to send this to the Arduino fails:

RPi output:
Received: from=255, to=0, childId=255, mtype=4, type=5, crc=139, ''
139, 2, 0, 255, 0, 255, 255, 4, 5
Message crc ok.
header.type=5, header.to=0, radioId=0
Got message
Message type: 4
msgTypeId=4 int=4
Sending data 255;255;4;5;1
Writing back messageWe have input
Nread=14
INP:255;255;4;5;1
.
INGOT radioId=255, childId=255, messageType=4, type=5, value=1
Sending data to GATEWAYRelaying message back to gateway.
Sending: from=0, to=255, childId=255, mtype=4, type=5, crc=193, '1
', sent via 0
193, 2, 0, 0, 255, 0, 255, 4, 5
radioId: Inserted id=1
Send failed. No ack received.
Sent to JS
Dynamic payload size=9

The "Message crc error." is a bit misleading. The code in the validate() routine actually checks the crc as well as the protocol version. Looking at the Arduino and the RPi 1.4 branch the versions don't match. I've adjusted the library and protocol version in the Sensor.h file on the Rpi to:

LIBRARY_VERSION "1.4"
PROTOCOL_VERSION 2

and the crc error disappeared.

While changing the data rate from 2M to 1M in line 30 of the Arduino Sensor.cpp, I had to fix a syntax error to recompile successfully in line 361: a missing "}" to close the do/while. The full line should be "} while (--retry);"

With these two fixed the RPi and the Arduino now communicate - but I get a lot of "Send failed" messages on the Arduino and a lot of "Error looking up radioIds" on the RPi now:

RPi output:
Received: from=255, to=0, childId=255, mtype=4, type=5, crc=139, ''
139, 2, 0, 255, 0, 255, 255, 4, 5
Message crc ok.
header.type=5, header.to=0, radioId=0
Got message
Message type: 4
msgTypeId=4 int=4
Error looking up radioIds
...and so on...

Android in DEBUG mode:
Started sensor.
Relay=0, distance=1
No radio id found in EEPROM fetching one from sensor net gateway
Relaying message back to gateway.
Tx: fr=255,to=0,la=255,ne=0,ci=255,mt=4,ty=5,cr=139:
Send failed.
Relaying message back to gateway.
Tx: fr=255,to=0,la=255,ne=0,ci=255,mt=4,ty=5,cr=139:
Send failed.
...and so on...

I couldn't get the connection working by just downloading and installing the Arduino and Raspberry components from GIT. It appears that even the defaults for the connection settings are different (e.g. 1M vs 2M transfer rate). I did not dig deeper but I would highly recommend to merge the projects or at least reference on a common set of header files for e.g. protocol specific details (e.g. the enums). That way the risk to run out of sync is way smaller. That said I don't have a running environment yet to test but started reviewing the protocol...

• the max bootloader size is 2048 words -> 4096 bytes (still not a lot)
• there are two ways to initiate the bootloader: either on power on of the sensor node or by submitting a specific package to the node that causes a reboot
• during bootloading the node does not route any packets to other nodes - it is a pure leaf node for that short period of time even if it's a router under normal condition
• the bootloader first of all submits its identity (unique address / type of node / current firmware version) to the master and waits for some time for a response. If the identity is not known (first start / no address known), a new address is requested similar to DHCP. If no response, it does a few retries and then finally starts the existing program. Fir this step to work the details about the node can't be stored "somewhere" in the EEPROM but need to be ad a well defined address so that both the bootloader and the program itself can access the same data.
• the master replies with details about the latest program version for the sensor node type
• if the response from the master lists the same version as the one installed, the node boots into the existing program
• if the master has a new program version then the node starts fetching the new program in small chunks and writes to the 328p program mem - once done it reboots running through the full process again assuming that this time the program is the latest available

In my previous setup I used the master node as software distributor - but now realized that some folks might want to keep the two separate, hence I'll add the option to use a different node for software distribution. This should allow e.g. users with a Vera to keep the Vera as master but use e.g. a Linux PC with an RF module connected (e.g. USB<->ATmega<->nRF24L) for software updates...

Announcing / requesting a new address etc is already covered by the protocol (auto assignment of radioID). A couple of packet types need to be added for the submission of the program etc. ll of this looks doable at this point.

I'll post about progress in this thread.

Agree it's not an easy exercise to shrink the RF code into the bootloader section.

I did an OTA bootloader within the limited space available in the 328p using a different RF module (which shouldn't be a major issue as the specs were pretty similar) and a different protocol (which might be an issue). I had to rewrite the init/rx/tx components of the RF module driver to be rather static and honestly no longer stick to the OSI model but go right through from app layer to hardware layer to make it fit and I had to reduced the use-cases in a way that the node in bootloader mode essentially only sends very specific package types and reacts to very specific package types being received.

I did not add any specific hardware (getting back to the initial comment about more complex PCB). The bootloader in inside the 328p and software reboot was handled by setting the watchdog to a short period and start an endless loop (instead of adding some additional hardware components to handle that).

I'll share more details when I had a chance to look into the protocol a bit further probably over the weekend. If I can't find the time, I'll share the code as it is...

Really really nice project !!!

I worked on a similar setup about two years ago with different RF modules but didn't finish. Now I was about to restart and realized that the nRF24 modules are waaaaay less expensive. I just started adjusting the old code for the nRF24 modules I ordered when I found this great project. The raspberry PI for me is the way to go as I own two sitting almost idle and don't own a Vera.

The one feature I'm missing after reading through the majority of the available documentation is over the air updates of the sensor node software. As this is one of the features I completed for my old design, I'll go ahead and try to port the bootloader and the RPi based state-less firmware server to work with the protocol and routing implemented here... if successful I'll post the results.

If you worked already on over the air updates or you know somebody who did, please let me know and I'll focus my efforts on something else

Tobias