2.0 Discussion: Units, sensor types and protocol

[ntruchsess]

@hek said:

Request for metadata is a good idea. Would require sensor to keep this available at all time in memory.

This is allready in memory, just requires a move of send-presentation-messages code into own method that is called from begin and in response to 'request-metadata'.

[ntruchsess]

@hek said:

If thin things like the Arduino based MQTT gateway would manage without it we could probably remove it from the normal payload.

As MQTT-protocoll is not stateless (it does Quality-of-service with message-storage and redelivery - though the existing gateway code doesn't support this yet) the MQTT-gateway cannot be a thin thing anyway. I wouldn't mind if it doesn't run on Uno due to memory-constraints, there's the Mega2560 or even the DUE which seems to be affordable as you only need a single gateway per install.

[hek]

First post updated.

[tbowmo]

what about a "RS232" sensor?

The TV in my living room has a serial port to control it (Can do everything I can with the remote) and also request status from the TV (eg. powerstate, channel being watched etc). Could be cool to include that into some kind of domotica solution

/ Thomas

[hek]

This would probably qualify for a special command-type as it's not a "normal" sensor.

Today I've squeezed in scene controller and ir-commands among the "sensors" but they should probably also have their own commands. Thanks for the input!

[Anticimex]

Perhaps rs232 and ir can be fused together as one type? Both are by definition serial protocols. Each with a protocol and symbol rate specific to the appliance at hand.

[hek]

I know too little about rs332 and IR to make a suitable command structure at the moment.

[Anticimex]

Think "uart"

[tbowmo]

Think that, for UART it should be enough to send characters (or strings) to the sensor, and send back replies to the master (If anything comes in at the serial port)

I asume that if one needs to use serial to communicate with some remote device, one will setup baudrate etc. for that particular device in the arduino sketch. So no need to use initialisation commands for the serial stuff.

For IR it's a bit more complicated, because there is no single protocol definition like serial. for IR it could be RC5, RC6, something else, it can be different carrier frequences etc. for each brand. And it can be send to multiple devices, using the same IR "blaster"

/ Thomas

[Anticimex]

@Thomas-Bowman-Mørch well, yes. But I am more thinking a "protocol". No matter what serial interface the sensor has externally, any serial interface implement some form of protocol for managing handshaking, flow control, etc. And considering the limitation in the data packets in RF24, we cannot expect to be able to implement a "stream".
By having a protocol, the sensor node can ensure it collects enough data to be able to satisfy it's external device, i.e. caching data packets for making a larger continuous serial transmission. Or reading a larger continuous transmission and caching it from the device, before starting to transmitting it to the controller. If transmission starts too early, data sent from the device, could be lost (depending on HW/SW solution) as serial protocols do not have to implement flow control (and rarely does).
Both IR and UART share the same caveats so in that respect, I think it makes good sense to figure out a command type that can satisfy both types as the needs are similar. And then it potentially can also be applied to other serial protocols (CAN for instance).

[lunarok]

Hi,

Not very usefull post for you here, but I want to say I'm very pleased to see this discussion and the news I can read here or what appears in dev on the github.

About the presentation of sub-type accepted. After getting an almost complete controller for Jeedom, we have difficulties to "predict" things to automate and make it very simple for user to use the plugin. Like for exemple, create commands. For relays and dimmer it was easy, but after when we look for servo or some extra type it's impossible to know for now what we can create.
So reading, the sensor will tell it in presentation it's just perfect.

Same for the presentation during node of lib used, that's just better than during the sensors, make sense.

I read also on the protocol page, that there is a way for a node to request data from gateway/controller. Can this point be clarified, maybe it needs also the sensor/node to say what it can request ? That's also for automation of autolink. For us (Jeedom plugin) and I think for other, if we know the sensor is able to request some data, we can automaticly create the information item and the user will have only to map it to the equipment it wants to point. I don't know if this is still included/planned as I'm aware many of the "smart" part must be on the controller side, but with beautiful things like Scene Controller and LCD display, I can imagine there is a need ?

I read also the OTA process will change a lot. So, is it better waiting 2.0 to include it on our side ? I mean, for the controller part, will this change a lot of things ?

And last point, there is actually a internal message type Reboot. This one is indicated only working for OTA bootloader. Is that true ?
If so, is it possible in 2.0 to include a way to request a complete presentation of an already started sensor. (that's why I ask for the Reboot message, simpler to request a reboot so the node will present itself at the same type)
Sometimes it's easier when migrating for exemple or reinstall, and we know the ID of sensor to ask it for presentating itself than going there to unplug it.

That's my thoughs after getting a little more in deep of protocol for the controler/plugin

Cédric

[aliasdoc]

Hi, I think we could integrate an uuid system in protocol, like this one https://code.google.com/p/tinkerit/wiki/TrueRandom .

[hek]

@lunarok said:

If so, is it possible in 2.0 to include a way to request a complete presentation of an already started sensor. (that's why I ask for the Reboot message, simpler to request a reboot so the node will present itself at the same type)
Sometimes it's easier when migrating for exemple or reinstall, and we know the ID of sensor to ask it for presentating itself than going there to unplug it.

True, reboot would send presentation again. But another option is to save the presentations in the api. Still a bit unclear which option I will choose.

@aliasdoc
What do you mean the random number should be used for?

[nneeoo]

Hi. I would like my nodes measuring various values (temperature, humidity, ...) to send those values at specific times. I don't want to include the RTC on the node, so would like the node to request a sleep period from the controller. I can see there is no such possibility with the version 1.4 of the protocol. What do you think about including such thing?

[hek]

You can use VAR1-5 for this in 1.4.

2.0 will have both VAR (used mostly for pushing custom data) and CONFIG (for custom node configuration).

[BulldogLowell]

Right now you can use the Time.h library and you don't need an RTC.

Just sync the time with MySensors call...

gw.requestTime(receiveTime);

with the function:

void receiveTime(unsigned long controllerTime) 
{
  Serial.print("Time value received: ");
  Serial.println(controllerTime);
  RTC.set(controllerTime);
}

[nneeoo]

@hek said:

You can use VAR1-5 for this in 1.4.

2.0 will have both VAR (used mostly for pushing custom data) and CONFIG (for custom node configuration).

Yeah, good point. I somehow missed these general purpose variables. It might be good to add a dedicated type for it if you'll find it useful. If not I can live with the VAR for sure ;). Thanks.

[nneeoo]

@BulldogLowell said:

Right now you can use the Time.h library and you don't need an RTC.

Just sync the time with MySensors call...

gw.requestTime(receiveTime);

with the function:

void receiveTime(unsigned long controllerTime) 
{
  Serial.print("Time value received: ");
  Serial.println(controllerTime);
  RTC.set(controllerTime);
}

Thanks for pointing out this library. However I don't want to track the time on the node, but would like just request the time for which the node should sleep from the controller. I think it makes more sense to have the logic in controller.

[BulldogLowell]

@nneeoo

Oh, you are requesting the duration of the sleep... not the moment at which you wish it to sleep.

I thought you were wanting it to transmit data at certain times...

[nneeoo]

@BulldogLowell said:

@nneeoo

Oh, you are requesting the duration of the sleep... not the moment at which you wish it to sleep.

I thought you were wanting it to transmit data at certain times...

Well, yes. I want the node to transmit at certain time. Since the controller should know when that time is, it has to be able to calculate for how long should the node sleep, when the node asks the controller. I hope you get my point.

[BulldogLowell]

@nneeoo

yes, so you can use Henrik's advice and store a variable which is the time interval in seconds, minutes or even milliseconds.

pass that variable into a function that uses that time as the interval to transmit your data from the sensor.

something like this... pseudo code

int intervalFromController; //in minutes

void setup()
{

}

void loop()
{
  gw.begin(getVariables, RADIO_ID, false);
  //your other stuff
  updateSensorData(intervalFromController);
}

void getVariables(const MyMessage &message) //
{
  if (message.sensor == CHILD_ID)
  {
    if (message.type == V_VAR1)
    {
      // get your V_VAR1
    }
  }
}
updateSensorData(int interval)
{
  static unsigned long lastTransmitTime;
  if (millis() - lastTransmitTime >= interval *60000UL)
  {
    //collect and transmit sensor data
   //request sensor interval
  lastTransmitTime += interval * 60000UL;
  }  
}

[ToSa]

@lunarok said:

And last point, there is actually a internal message type Reboot. This one is indicated only working for OTA bootloader. Is that true ?

Reset and OTA bootloader are independent - but you need to set the fuse bits of the MCU to enable the watchdog for the software reset to work. This is done automatically when flashing the OTA bootloader but you can do that as well in a separate step keeping the existing bootloader (e.g. optiboot).

[aliasdoc]

@hek said:

What do you mean the random number should be used for?

I am currently trying to develop a bridge between mysensors and home kit managed with RPi and for this purpose I need an uuid different of the id used in the my sensors library and I thought the above library is a good start for the generation of a uuid (almost unique).

[Zeph]

If there is interest in using somewhat of a standard rather than rolling our own (and I think a case could be made either way), it would be worth reviewing the IoTDB effort which is also trying to describe the semantics of sensor/actuator control. For example:

https://iotdb.org/pub/iot-attribute.html

[hek]

@Zeph said:

https://iotdb.org/pub/iot-attribute.html

Thanks interesting reading.

[daulagari]

Yes, indeed interesting reading also in the light of controllers.

[lunarok]

Hi,

Please to see this post. As our plugin for Jeedom is complet now, I have some feedback to share.
First, is there a date and status of version 2 for now ?

Here my thoughs :

• about units, I agree with the fact that it's lightier to have the Arduino not taking care of it, but instead use some conversion on the controller side

• about value types, it will be better to have really type and not a notion of unit sometime here too like for power get only consuption and not kwh, really going all the way and give the unit control to the controller

• about value types and sensor types, after writing the plugin, still weird for me to have a sensor presentation giving some clue on what type it is, but need to wait for some value to be sure. I mean power sensor for exemple that can get some different type of values, this is not really clean I think. Or the weather sensor, in one case it will send a value type that is numeric, the other one is text (forecast)

• will it not be better to have some mechanism when it's an actuator to deliver the type of message it is accepting ? for exemple a heater, there is the off/on but also a mode type. How is it possible to know that the sensor is accepting all or only on/off ?

So, for me, something like after a presentation being able on the controller to say what type of data and what commands we can say, will be the best. And important too, what data it can be requesting. I mean, now with the scene_controller to extend functions, we need to get data from the automation software too. Imagine to display some informations from outside the mySensors network.
Maybe by completing the sensor_type and merging with value_types, removing the unit notion inside the value. And so during presentation we know all about the data it can send. And in second step during presentation, declare the actuator commands.

One other point important and asked by some users, is it possible to integrate some security in the protocol ? As soon you talk about actuator accepting command from anyone, that's unsecure. Key given during inclusion by gateway and use for encode the data ?

Thanks for reading

Cédric

[Anticimex]

About security, I'm looking into that. I am working on a concept involving key exchange and signing using an external circuit.

[lunarok]

Yes for security can be also a first presentation requiring a signing. Like what Puppet is doing, agents first present themself to the master, master is waiting for someone to accept a request, and after it gives the certificate to the client and then the client tell him verything about itself (facts in this view can be compared to the sketch and sensors presentation)

But I don't know if it's easy with arduino this type of exchange.
Other way can be a simple key we put in sketchs, and using the same on every of our home, and use it in the lib to encode the data.

[Anticimex]

I am working on a security protocol, and have posted my take on things in the security thread. I will weigh in on this thread once I have verified my design, but so far, only two new message types should be needed, one to request security capabilities and once for capabilities and a nonce. I do not think software based security is a suitable solution due to memory constraints so I am going for a hardware based solution with a pre-shared key. I will publish more concrete examples once I have verified them.

[lunarok]

Hi,

Is there any move on the V2 ? Can we see the actual status and what it is going to look like ?
Going to a V2 for the Jeedom controller, will like to see it coming with mySensors v2 changes.
Precisly will like to have a status about :

• reboot of node without needing a special bootloader (inside the lib will be the best)
• sending libversion from presentation including for gateway (this is include for nodes, I don't know for gateway)
• possible of sending V_type used during presentation, no need to wait to send data
• possible of sending a short desc name for each sensors created (can be helpfull when you create many sensors with same type and the difference is not only the order)
• possible to send the power source of node, I don't know inside battery or else. This will be very helpful with battery/plug or battery/solar source for exemple. So we can know how much the battery is full but also if the sensor is actually on battery.
• getConfig to be used for any paramteres instead of units, like this the node can request parameters from the conrtoler (think about a global sketch for a switch that can talk to a relay node, by the controller side you set which node it's controling)

Can we have a wiki page maybe with an actual status of where is the V2 ? And is there a dev version we can try ?

[hek]

@lunarok said:

Hi,

Is there any move on the V2 ? Can we see the actual status and what it is going to look like ?

There has been some delays due to workload. But i hope to finish it eventually.

• reboot of node without needing a special bootloader (inside the lib will be the best)

Not sure it is possible to reboot an arduino without watchdog enabled (anyone knows any tricks?).

• sending libversion from presentation including for gateway (this is include for nodes, I don't know for gateway)

Yep, possible to get version from gateway today.

• possible of sending V_type used during presentation, no need to wait to send data

There will be a different setup.

• possible of sending a short desc name for each sensors created (can be helpfull when you create many sensors with same type and the difference is not only the order)

Good idea! I'll take that into consideration.

• possible to send the power source of node, I don't know inside battery or else. This will be very helpful with battery/plug or battery/solar source for exemple. So we can know how much the battery is full but also if the sensor is actually on battery.

Thats also a good idea. I'll have to think about how it could be incorporated.

• getConfig to be used for any paramteres instead of units, like this the node can request parameters from the conrtoler (think about a global sketch for a switch that can talk to a relay node, by the controller side you set which node it's controling)

Config/settings will be a bit different as well.

Can we have a wiki page maybe with an actual status of where is the V2 ? And is there a dev version we can try ?

Hmm.. my github account (henrikekblad) will contain the source until I feel it's worth trying out by the community. But there is a part of it I'm waiting for a c++ guru at work helping out with (advanced c++ templating which is a bit over my level of c++ knowledge). Those will hopefully make callbacks (incoming messages) usage awesome.

The plan is the gateway will be able to accept/push json objects instead of the semicolon separated parameters, Much depends on how memory demanding it becomes.

[axillent]

can we avoid float?
instead of float a multiplied integer can be used
for example 34.6 C temperature can be presented as 346 (no dot)

a single float operation increase HEX size significantly

@hek what about idea we discussed to have a network with multiple gateways?
what about adding support for the 2.0 for the nodes be able to have a different BASE address per each gateway?
this way the future network will allows to have 255 sensors per EACH gateway, not per NETWORK

[hek]

@axillent

Yes, configurable base address would be handy if we choose to create an ESP-gateway. It is already supported today but you have to hard code it into MyConfig.h. Would be neat to have this configurable over WiFi.

Using OTA transmitted floats is optional. I haven't looked so deeply on how/if this will be supported yet in the upcoming version.

[tekka]

Any plans for encryption?

[Anticimex]

This post is deleted!

[daulagari]

what benefit is there from encryption vs signing in the MySensors case?

If you have a lock or door sensor do you want others to know when it is opened or closed?

Signing is quite expensive if you look at additional payload size, you need a big counter to prevent replay and a big MAC to prevent attacks. I think that when encrypting things you can do it with the same additional payload and maybe even a less.

[Anticimex]

This post is deleted!

[daulagari]

Yes, signing, when properly done, is for sure a good begin.

How many bytes are you using for the truncated MAC and nonce?

The AES block size is 128 bytes, so 16 bytes so I do not see why that would not fit.

Encryption is already some kind of authentication is a sense that if you can successfully decode the message you can be sure that the other side knows the shared secret, just like in the cause of your SHA25-HMAC.

[daulagari]

Would be good to move the last five message into a separate thread...

[Anticimex]

This post is deleted!

[Anticimex]

This post is deleted!

[hek]

I'm afraid I can only fork a topic int two threads. Haven't found a way to move posts to another thread/topic.

[Anticimex]

This post is deleted!

[Anticimex]

@tekka Please see the discussion on security.

[lunarok]

Hi,

A new wish. Possible to have the network gateway listening for nodes also on the ethernet network. Looking for ethernet nodes inside the electric panel. No RF, no security break

[axillent]

A support for multi-gateway setup is needed. Currently it is required to hardcode radio-channel or base-address for different segments-different gateways

[marceloaqno]

There is a solution for multi-gateway that I'm working to port it to the development branch which could use the same radio channel and also to have sensors at gateway node.

[axillent]

@celonunes what kind of a solution it will be?
will it be possible to route messages between nodes managed by different gateways?

[marceloaqno]

Gateway become normal node, with an id(1-254), like a repeater. Address 0 become like a virtual address which is used when a node wants to send a message to the controller.
This solution will give more redundancy to the network but the same limitations apply, like the max of 254 per network.

[marceloaqno]

In the case raised by @lunarok, I'm not sure it will fit.

[axillent]

@celonunes said:

Gateway become normal node, with an id(1-254), like a repeater. Address 0 become like a virtual address which is used when a node wants to send a message to the controller.
This solution will give more redundancy to the network but the same limitations apply, like the max of 254 per network.

OK. But how this will prevent conflicts on the air between nodes connected to different gateways and having same radio channel and BASE_ADDRESS?

[marceloaqno]

Each node will continue to have a unique id in the network.
The controller would have additional work to map nodes to their proper gateway, and this could make the whole idea into something impracticable depending on the controller software.

[axillent]

@celonunes will you keep a limit 8 bit for ID?

currently with multi-gateway setup I have N * 255 address space, where N is a number of gateways

[marceloaqno]

@axillent said:

@celonunes will you keep a limit 8 bit for ID?

currently with multi-gateway setup I have N * 255 address space, where N is a number of gateways

Yes.
Your solution seems interesting, do you have some published code of it?

[axillent]

@celonunes it is not my solution) it is a way of using 1.4.1 with multiple gateways

[marceloaqno]

@axillent said:

@celonunes it is not my solution) it is a way of using 1.4.1 with multiple gateways

I'm aware of this.

@lunarok your wish may become possible in future developments using MQTT