1.4 Beta

[marceltrapman]

@hek Do you know if the ota is something initiated by the controller or do we update the gateway with a new version and will the gateway be responsible?

[hek]

@marceltrapman

The controller will be responsible for initiating and serving the new sketch version. There is basically no changes in gateway except being able to handle binary data.

[hek]

@Damme

mSetRequestAck(msg,true);

You should probably not request ack back on a id-response. But I doubt that is the cause for transmission error. What is happening in the sensor node?

[Damme]

@hek Tried with no ack also. And the node doesn't recieve anything. I'm out of components to build myself a sniffer right now.. Might have to kill something.. I can recieve From the node though.. but the node does not read from gateway..

[hek]

@Damme

You only live 190 km from me. Come and get some

What kind of antenna on your gateway? Have you tried lowering transmit power of gateway?

[Damme]

@hek I fixed the problem , it was the RF24_PA_LEVEL_GW .. needed to be low.

You are correct with the ack on id-response, it gets all fucked up. Stuck in loop requesting ID 10+times/second. And doesnt register the id.. Too bad. would be really good for me to know if the sensor actually got the id_response or not... (and if it's sent as node 255 or node newid doesnt really matter..)

edit:
I changed my aproach to checking if sendRoute(msg) is true. But I still have a problem that the node actually recieves the packagt but just ignored is. sometimes I need to reset the node for it to get accept the new ID.. example;

0;0;3;9;read: 255-255-0 s=255,c=3,t=3,pt=0,l=0:
0;0;3;9;send: 0-0-255-255 s=255,c=3,t=4,pt=1,l=1,st=ok:20
0;0;3;9;read: 255-255-0 s=255,c=3,t=3,pt=0,l=0:
0;0;3;9;send: 0-0-255-255 s=255,c=3,t=4,pt=1,l=1,st=ok:21
0;0;3;9;read: 255-255-0 s=255,c=3,t=3,pt=0,l=0:
0;0;3;9;send: 0-0-255-255 s=255,c=3,t=4,pt=1,l=1,st=ok:22
0;0;3;9;read: 255-255-0 s=255,c=3,t=3,pt=0,l=0:
0;0;3;9;send: 0-0-255-255 s=255,c=3,t=4,pt=1,l=1,st=ok:23

I'll try to look at the debug output on the node tomorrow..

btw: \n missing line 278 mysensor.cpp debug(PSTR("version mismatch"));

[marceltrapman]

@hek said:

The controller will be responsible for initiating and serving the new sketch version. There is basically no changes in gateway except being able to handle binary data.

How do I get this started Henrik?

[hek]

@marceltrapman

Until @ToSa is back, I suggest you have a look at how the NodeJSController serves hex-files.
https://github.com/mysensors/Arduino/blob/development/NodeJsController/NodeJsController.js

[Yveaux]

@hek I wanted to update my MQTT gateway to 1.4beta state but run into a problem with the serial message format.
If I interpret the code right, the messages sent (in serial message format) to the ethernet gateway (over ethernet, so from my linux system running the MQTT gateway) contain 6 fields, of which the 4th is the ack flag.
Messages received from the ethernet gateway contain 5 fields, with no ack indication.
Shouldn't the message format be identical in both directions, as it is for the 1.3 version?

[hek]

@Yveaux

Yes, It might be interesting (for the controller side) to use the 4th field to flag if the received message is an ack or regular message.

...so for outgoing you enable receiver to send ack back. And incoming the flag indicates if this as an ack or regular message.

Make sense?

[Yveaux]

@hek said:

Make sense?

Makes sense

[John]

Short question. I'm browsing the 1.4 code and is there a replacement for waitForMessage? I've looked into the code online but can't seem to find it. And if not, is it possible to attach the radio to an interrupt to call an interrupt routine on incoming message?

[Yveaux]

@John said:

is it possible to attach the radio to an interrupt to call an interrupt routine on incoming message

The last question is a definite yes, see for example https://github.com/Yveaux/NRF24_Sniffer/blob/master/Arduino/NRF24_sniff/NRF24_sniff.ino function handleNrfIrq. Connect the NRF24 IRQ to pin 2 and call attachInterrupt to register the interrup handler.

Be very careful what you do in this interrupt handler, though. Don't try to receive any message for example, as the SPI access to the NRF24 is not interrupt-proof, as is the MySensors library!

[hek]

@John

Yep, as @Yveaux says, be careful what you do inside interrupt handler.

Not sure what you try to archive here.. but in 1.4 you should usually just call process() in your loop(). The callback you registered a in begin(callback) will be called on incoming messages.

See RelayActuator for an example.

[John]

@Yveaux

Be very careful what you do in this interrupt handler, though. Don't try to receive any message for example, as the SPI access to the NRF24 is not interrupt-proof, as is the MySensors library!

Darn, i hoped it could.

@hek

Not sure what you try to archive here.. but in 1.4 you should usually just call process() in your loop(). The callback you registered a in begin(callback) will be called on incoming messages.

I'm averaging sensor results, so i read sensors every second an send them every minute, also this setup has an 16x2 display attached to show the values. But i would like it to be bi-directional with the server so it can display the server send data. Thus if i would use waitForMessage i can let it wait about a second for a message read sensor values, and let it wait for a second again. (because i understood it blocks?)

[Yveaux]

@John You can obtain the same result as the old waitForMessage by registering a callback. Have this callback set a flag and call process() in the loop, as hek suggests.
You could also turn things around and read and average your sensors from a (timer)interrupt handler.
But when you're not completely confident with using interrupts then its better to avoid them and just process everything from your loop(). Interrupts can introduce lots of nasty problems if you don't know exactly what you're doing!

[John]

@Yveaux

You can obtain the same result as the old waitForMessage by registering a callback. Have this callback set a flag and call process() in the loop, as hek suggests.

Well, that was my first try and hoped it was interrupt based because of one wire connected on arduino pin 2 to IRQ on the NRF. The reason i asked it because i saw an example from an other library: http://maniacbug.github.io/RF24/pingpair_irq_8pde-example.html .

You could also turn things around and read and average your sensors from a (timer)interrupt handler.

I will have to investigate if any of the components is using which timer and what will be influenced by it. I'm using an HD44780 alike as display so i'm not sure because never dived deep into the used library.

But when you're not completely confident

about 90%?

@hek
It would be nice if a waitForMessage like functionality would be included in 1.4.

[clippermiami]

@hek Any idea when 1.4 will become the production version?

[hek]

@clippermiami

We're still making small changes in the protocol and serial API which we need to stabilize before release.
I also need time to update the main site to reflect the new version before its "released" .

[hek]

Added few small changes:

• Add dimmable LED actuator example (converted @blacey 1.3 example)
• Rename PING to the more suitable FIND_PARENT
• Float values sent in binary format over the air
• Move base-radio-id to MyConfig.h

https://github.com/mysensors/Arduino/commits/development

[Damme]

@hek another suggestion for addition

RelayWithButtonActuator
incomingMessage should check if there is any payload or not. (could be bad package)
I use empty payload as a request for latest state (Yes I could do this some other way but I think it is logical..)

  if (msg.type==V_LIGHT && strlen(msg.getString())!=0) {

[John]

Because of the use of custom variables and the possibility of receiving and sending and of many sensor type possibilities is it possible to add a "sensor" type of S_OTHER (or alike)? I think this then would be in line of having var types V_VAR1, etc.. of which then would be something else then defined.

This will add the possibility to send other then sensor related data. This is of course possible but could semantic wise be more applicable.

[hek]

@John

Yes

[hek]

A small serial protocol breaking change was just checked in. @John and @marceltrapman you might want to have a look at this.

Serial protocol changed to allowing controller to know if an incoming message is an ack. Outgoing and incoming serial messages now also have the same parameter count.

radioId;childId;messageType;ack;subType;payload\n

Where ack parameter means the following:
outgoing: 0 = unacknowledged message, 1 = request ack from destination node
Incoming: 0 = normal message, 1 = this is an ack message

(previously only outging messages had the ack-flag)

[John]

@hek

Thanks for the heads up. This will give a good way of tracking messages send controller wise. I can take a look at it tomorrow evening if you would like a third party ack this works ok.

[Yveaux]

@hek Thanks for the quick response!

[Damme]

@Yveaux take a look at (was thinking about your sniffer) https://github.com/mysensors/Arduino/commit/c910bfdb9e54a9a41e991734b60c757968bd8210

[John]

@hek I have a couple of questions:

Have not tested it yet, but that now ack's are possible i would like to implement a retry queue next to a normal send queue. The questions then would be:

• How long does it take for the longest possible message to be send?
• Is there a delay needed between each send?
• Is there an ack timeout?

John.

[Yveaux]

@Damme Yeah, so header changed again... Madness
I'll have to update the dissector then...

[hek]

@John

The send time depends on how many hops the message need to make. But on the controller side you could perhaps wait for a couple of second before trying again.

There is no ack timeout but each hop on the way to the destination will only be tried once (using the build in retry functionality of the nrf-chip).

[lininger]

@hek Any tips on upgrading the June 1.4b1 to the August 1.4b1. I just attempted it with disastrous results.

I started off.

1. Uploading the Vera files
2. Re-compiled the new gateway (after changing the my IP Address and such)
3. Re-complied the Repeaters

Just to get going. Both repeaters failed at startup with the standard FAIL in the debug logs. Wiping the Eeprom and trying again did not work. Dropping the repeaters for vera and re-including did not work? Finally went back to the June build and restored the vera from last night and then two other units failed to come back online. Had to replace the antennas on those two units to get them back on-line – BIZARRE.

1. Do all the sensors have to be updated at once?
2. Will existing sensors under the June 1.4b1 build co-exist with the August 1.4b1 Gateway build?
3. Can we just recompile our existing June sketches under the new mySensors library as long as no compile errors show?

Thanks

[hek]

@lininger

Yes, you'll have to recompile/update them all as message format has changed. No need for wiping eeprom or re-include sensors. Also update Vera plugin.

Replacing antenna should not be necessary and I don't understand why that would change anything..

[Yveaux]

@hek I had a look at the latest state of the 1.4 header format and saw the sender/last/destination have moved to the front position.
This means the protocol version is, once again, located at a different position w.r.t. 1.3 and earlier 1.4 version.
For software trying to interpret any MySensors message (e.g. my sniffer or a (future) MySensors version supporting different protocol versions) or software that checks the library version to match with its own version it becomes nearly impossible to detect which library version a message was sent with.
I propose to start any message, now and in the future, with a protocol version (use a full byte or use some remaining bits for protocol independent data) and stick to that. This makes life for sugested applications a lot easier.
Adding a version which has a different offset every time makes no sense...

[hek]

@Yveaux

Sorry.. the sniffer slipped my mind. The purpose was to prepare for future encryption (keeping the changing "last"-field first) and potential split of the routing information and payload. Which would be necessary for supporting other transportation layers (like RadioHead). They might not be interested in any MySensors version-specifics first in the radio message.

Do you have any suggestion?

[Yveaux]

@hek said:

Do you have any suggestion?

It's in my post hek

[Yveaux]

@hek I have another question: when sending from a sensor to a gateway the data can have all kinds of formats (text, byte, float etc) which comes out the gateway through the serial protocol. When sending the other way, I guess the serial data is always sent as text by the gateway to the sensor, right?
Too bad we can't also use different formats when sending through the serial api... Did you think about this?

[hek]

@Yveaux said:

It's in my post hek

Yes, you suggest to always have but that might not be possible if we choose some other transportation layer (which most probably will use the first bytes for it's own routing).

When sending the other way, I guess the serial data is always sent as text by the gateway to the sensor, right?
Too bad we can't also use different formats when sending through the serial api... Did you think about this?

Yes, I've thought about it. We have a couple of options here. Start type:ing variables through the serial protocol (controller must send datatype) or have a hard coded variable->type map on arduino side. Both of these are valid options.
A binary serial protocol is also an option (could be an option when building gateway).. But that makes it harder to debug e.g. Arduino Serial Monitor.

[Yveaux]

@hek said:

but that might not be possible if we choose some other transportation layer

Then the transportation layer data is not part of mysensors protocol. It's that simple!
IMHO This again boils down to nested protocol headers which we talked about before.
The routing info is part of transportation layer (it could even have its own version info) followed by the mysensors header ( starting with a version number) which tells us what's in the message. Then come the actual data, be it a value, presentation info or whatever. I can make a sketch of this structure if that helps in the discussion.

[hek]

@Yveaux

I think we agree. Only problem is that I haven't split up the structure in mysensors-transportation/mysensors-header/mysensors-payload yet but if you look from the sniffer side this would not matter.

Please post you suggested structure (but in a new thread).

[Damme]

I got this question from a friend and since I dont know the answer either;

I'm trying to figure out the sleep mode and if the radio module wakes up or not (via INT pin 2) .. but for me it does not seam to do this. Which sleep-modes does 1.4b have and how do I see if the message is for me, and if not continue sleep?

[hek]

@Damme

The radio interrupts is not used at the moment.

It uses the following sleep modes

LowPower.powerDown(SLEEP_FOREVER, ADC_OFF, BOD_OFF);

or

LowPower.powerDown(SLEEP_XXXXS, ADC_OFF, BOD_OFF); 

Depending on if you want timer to wake up or not. Radio is always put to sleep.
For more details, check out MySensor.cpp.

[warawara]

I'm trying to create my own controller in Node-Red.

My setup is as followed:
Motion Sensor arduino with standard example sketch <--> SerialGateway arduino with standard example sketch <--> USB <--> Node-Red (installed on Raspberry PI)

My SerialGateway receives the request for a node ID (255;255;3;0;3;)
I then return a serial command: 255;255;3;0;4;1

You can see my Node-Red output below:

For some reason my SerialGateway is not broadcasting my command. It's only repeating the above commands sevral times;

Am i using the correct protocol values or is there something else I should do?

[hek]

@warawara

Did you remember sending a newline?

[warawara]

@hek
Yes I did.

My Node-Red is configured to send a newline after every command.

But just to make sure I also tried adding it to the string.

Same result.

[warawara]

@hek

I assume sending de command in ascii is good or do I need to send a binary buffer?

[hek]

@warawara

If it's working from your gateway Serial Monitor in IDE you must be doing something wrong in your program.

[warawara]

@hek
Not working in IDE.

[warawara]

@hek

Got it working, but my solution is not very satisfying.

I swapt the sketches on my 2 arduino's.

Original arduino setup was:
SerialGatway - Arduino pro mini 3.3v (including Decoupling-Capacitor)
Motion Sensor - Arduino Uno (including Decoupling-Capacitor)

Now the sketches are switched and it works.

Eventually I will be working with only Arduino pro mini's but still the original setup should have worked.

[Damme]

@warawara did you try clearing using clear eeprom? That helped me some times. Dont really know why some nodes just wont want to recieve packages..

[warawara]

@Damme
Thanks for the tip.

I will try that in the future. But I don't think it was a problem of not receiving on the sensor part. I think it was a problem with not sending on the Serial Gateway part.

But I got it working now!

[RJ_Make]

Ok, So I took the plunge and upgraded to 1.4.b1, and everything went "OK" with one glaring exception...... All temperatures display in Celsius. No matter what I do, I can't seem to get it to display in Fahrenheit..

Any Idea's?

[Bandra]

@hek
I'd like to ask for a small change to be made to the sleep function.

When we call gw.sleep(x) or gw.sleep(x,y,z) then the arduino disables the interrupt timer (to save power) that triggers the counter that drives millis(). So the value returned from millis() stalls when we sleep.

This is ok on our battery powered devices, since it's saving power, but makes it hard to determine how long it's been since we sent values to the gateway.

For example, if I use this code:

if ((unsigned long)(millis() - timeOfLastSensorSend) >= MIN_SAMPLING_INTERVAL) {
    // read sensors and send to gateway
    timeOfLastSensorSend = millis();
}
gw.sleep(SLEEP_TIME);

Then it doesn't work because millis() does not increment while the the sketch is asleep.

I have a thought on how we can solve this but I have a to get to a work meeting.

[Bandra]

Ok, back from meeting.

So, for sketches that don't need to sleep on interrupt, then things are ok because I can code around it like this:

if ((unsigned long)(millis() - timeOfLastSensorSend) >= MIN_SAMPLING_INTERVAL) {
    // read sensors and send to gateway
    timeOfLastSensorSend = millis();
}
gw.sleep(SLEEP_TIME);
timeOfLastSensorSend -= SLEEP_TIME;

My rationale here is that if millis doesn't increment then I should make my variable decrement by the same amount. Same effect.

However, the catch is when sleeping with interrupt. My thinking was to modify MySensors.cpp to change the return value of sleep. Something like this:

unsigned long MySensor::internalSleep(unsigned long ms) {
    unsigned long origMs = ms;
    while (continueTimer && ms >= 8000) { LowPower.powerDown(SLEEP_8S, ADC_OFF, BOD_OFF); ms -= 8000; }
    if (continueTimer && ms >= 4000)    { LowPower.powerDown(SLEEP_4S, ADC_OFF, BOD_OFF); ms -= 4000; }
    ...
    if (continueTimer && ms >= 16)      { LowPower.powerDown(SLEEP_15Ms, ADC_OFF, BOD_OFF); ms -= 15; }
    return (origMs - ms);
}

unsigned long MySensor::sleep(unsigned long ms) {
    // Let serial prints finish (debug, log etc)
    Serial.flush();
    RF24::powerDown();
    continueTimer = true;
    return internalSleep(ms);
}

unsigned long MySensor::sleep(int interrupt, int mode, unsigned long  ms) {
    // Let serial prints finish (debug, log etc)
    unsigned long sleptFor = 0;
    Serial.flush();
    RF24::powerDown();
    attachInterrupt(interrupt, wakeUp, mode); //Interrupt on pin 3 for any change in solar power
    if (ms>0) {
            continueTimer = true;
            sleptFor = sleep(ms);
    } else {
            Serial.flush();
            LowPower.powerDown(SLEEP_FOREVER, ADC_OFF, BOD_OFF);
    }
    detachInterrupt(interrupt);
    return sleptFor;
}

So rather than return a boolean on sleep(x,y,z), return the amount of time it slept for before it was interrupted. The check of the boolean is replaced with a check if return code > 0.

Hopefully my ramblings make some kind of sense.

[hek]

@ServiceXp

The sensor fetches the unit setting from Vera at startup. Something probably fails during this data exchange. You'll have to look at the debug logs to see what happens.

This behavior might change in 1.4 with sensors just reporting SI-units and controller makes the necessary conversion/scaling.

[RJ_Make]

@hek
This is the first time I've ssh into vera, so I have no idea what I'm looking for.. I have 2 Everspring ST-814's that display correctly, and of course 2 MySensor sensor which did in 1.3, but not in 1.4b1.

I found this, but not sure if this is what you need.

02 08/22/14 7:47:04.914 luup_log:87: Arduino: Incoming internal command '0;0;3;9;read: 2-2-0 s=1,c=1,t=0,pt=7,l=5:22.6' discarded for child: nil <0x2f8b3680>
50 08/22/14 7:47:04.916 luup_log:87: Arduino: Set variable: 2;1;1;0;0;22.6 <0x2f8b3680>
50 08/22/14 7:47:04.916 luup_log:87: Arduino: Setting variable 'CurrentTemperature' to value '22.6' <0x2f8b3680>
50 08/22/14 7:47:04.916 luup_log:87: Arduino: urn:upnp-org:serviceId:TemperatureSensor1,CurrentTemperature, 22.6, 92 <0x2f8b3680>
06 08/22/14 7:47:04.917 Device_Variable::m_szValue_set device: 92 service: urn:upnp-org:serviceId:TemperatureSensor1 variable: CurrentTemperature was: 22.5 now: 22.6 #hooks: 3 upnp: 0 v:0xda4620/NONE duplicate:0 <0x2f8b3680>
01 08/22/14 7:47:04.918 LuaInterface::CallFunction_Variable func: w_switch Device_Variable 92 urn:upnp-org:serviceId:TemperatureSensor1:CurrentTemperature failed [string "..."]:14: bad argument #1 to 'sub' (string expected, got nil) <0x2f8b3680>
01 08/22/14 7:47:04.919 LuaInterface::CallFunction_Variable func: w_switch Device_Variable 92 urn:upnp-org:serviceId:TemperatureSensor1:CurrentTemperature failed [string "..."]:14: bad argument #1 to 'sub' (string expected, got nil) <0x2f8b3680>

[hek]

@ServiceXp

The node will be transferring celsius data until it manage to receive settings from controller (this is done in the background) by setup().
Attach your failing sensor to the computer. Upload sketch with debug enabled. And look at the Serial monitor. Restart a sensor a few times .

[RJ_Make]

@hek

Thanks, I'll do that when I get home later. Just so I'm clear..

To enable debugging: I Remove the "//" before the #define DEBUG ** in and ONLY in** the /libraries/MySensors/Config.h file ? Then upload the sketch again to the sensors? (I don't have to upload the gateway sketch also do I?)

[hek]

@ServiceXp

Yes correct!

[RJ_Make]

@hek

I can only find MyConfig.h file, and DEBUG was already without the "//".

So I noticed something VERY strange. It seems when I have the sensors connected to the computer, they start reading in Fahrenheit, and once I pull the USB cable it reverts back to Celsius with-in 30-60 seconds..

[RJ_Make]

Yep, I have no idea how to debug this. The debug window reads in Fahrenheit, (and in Vera), but 30-60 seconds after I disconnect the usb cable from my computer (at which point I can't use the serial com window) it reverts back to Celsius.

I've deleted one of the sensor nodes and child from Vera, cleared the eEPROM and re-install sketch and re-incuded back into Vera...

I just don't understand why it works while connected to my computer...

[hek]

@ServiceXp

No, this seems strange. The unit settings should get stored in eeprom.

[RJ_Make]

@hek

Do you think I can rule out my sensor hardware as it works with 1.3? Is there any way I can determine if it's being written to eEPROM while connected to the computer and then after it's disconnected (to see if it's getting overwritten)?

[hek]

@ServiceXp

Do you call process() in loop()?

But, really.. Don't put too much effort into this. Just hardcode sensor to send fahrenheit until conversion is done by controller plugin.

[RJ_Make]

@hek
I'm using the boiler plate sketch provided in the 1.4b1 files. The only modifications I've done is add battery code. (Copy and Paste).

How to I hard code sensor to send in Fahrenheit?

Sorry you have to hold my hand through this, I'm very new at this.....

[hek]

@ServiceXp

Change

float temperature = static_cast<float>(static_cast<int>((gw.getConfig().isMetric?sensors.getTempCByIndex(i):sensors.getTempFByIndex(i)) * 10.)) / 10.;

To

float temperature = static_cast<float>(static_cast<int>((sensors.getTempFByIndex(i)) * 10.)) / 10.;

But I still wonder what goes wrong...

[RJ_Make]

@hek

Worked perfectly for the temperature sensor, but it appears the Humidity and Temp. Sketch handles this differently. Not sure what to replace and where.

[RJ_Make]

Not very nice but I think this will work..

  //float temperature = dht.getTemperature();
 float temperature = dht.getTemperature()*9/5 + 32;
  
 if (isnan(temperature)) {
    Serial.println("Failed reading temperature from DHT");
 } else if (temperature != lastTemp) {
 lastTemp = temperature;
   if (!metric) {
  temperature = dht.getTemperature()*9/5 + 32;   //dht.toFahrenheit(temperature);
 }