Gateway is sending command=1, type =1 (c=1,t=1) message to sensor

[tomkxy]

I am playing around with a test setup of one node with a temp/humid sensor and a MQTT Client Gateway. I setup both with soft signing which what I saw is probably not really relevant.
The gateway receives the sensor data properly but obviously tries to send a message back to the sensor for which it requests a nonce from the sensor. I put a debug statement printing the message for which the nonce should be sent.

0;0;3;0;9;sign fail
0;0;3;0;9;send: 0-0-21-21 s=2,c=1,t=1,pt=0,l=4,sg=0,st=-:63.8 <<<<<<< gateway sends to sensor???
0;0;3;0;9;send: 0-0-21-21 s=2,c=3,t=16,pt=0,l=0,sg=0,st=fail: <<<<<< nonce request to sensor
0;0;3;0;9;nonce tr err

If I am not mistaken command=1 is a set command and type 1 corresponds to humidity. I have no glue why the the gateway node sends this to the sensor. I am pretty sure that my sketch does not trigger that directy.

Below you find the code extract with the debug statement I put just before the nonce is requested.

#ifdef MY_SIGNING_FEATURE
	// If destination is known to require signed messages and we are the sender, sign this message unless it is an ACK or a handshake message
	if (DO_SIGN(message.destination) && message.sender == nc.nodeId && !mGetAck(message) && mGetLength(message) &&
		(mGetCommand(message) != C_INTERNAL ||
		 (message.type != I_GET_NONCE && message.type != I_GET_NONCE_RESPONSE && message.type != I_REQUEST_SIGNING &&
		  message.type != I_ID_REQUEST && message.type != I_ID_RESPONSE &&
		  message.type != I_FIND_PARENT && message.type != I_FIND_PARENT_RESPONSE))) {
		bool signOk = false;
	debug(PSTR("send: %d-%d-%d-%d s=%d,c=%d,t=%d,pt=%d,l=%d,sg=%d,st=%s:%s\n"),
			message.sender,message.last, message.destination, message.destination, message.sensor, mGetCommand(message), message.type,
			mGetPayloadType(message), mGetLength(message), mGetSigned(message), "-", message.getString(convBuf));
		// Send nonce-request
		if (!sendRoute(build(tmpMsg, nc.nodeId, message.destination, message.sensor, C_INTERNAL, I_GET_NONCE, false).set(""))) {
			debug(PSTR("nonce tr err\n"));
			return false;
		}
		// We have to wait for the nonce to arrive before we can sign our original message
		// Other messages could come in-between. We trust process() takes care of them
		unsigned long enter = hw_millis();
		msgSign = message; // Copy the message to sign since message buffer might be touched in process()

[tomkxy]

I figured the issue out myself. I have indeed an issue in the MQTTClient Gateway. The gateway subscribes to all messages. Hence, once it receives a reading from a sensor which it publishes it, it receives the same message which it sends down to the sensor as set message.

[TD22057]

@tomkxy said:

I figured the issue out myself. I have indeed an issue in the MQTTClient Gateway. The gateway subscribes to all messages. Hence, once it receives a reading from a sensor which it publishes it, it receives the same message which it sends down to the sensor as set message.

I was wondering about that. It's too bad that the MQTT controllers don't use a different MQTT prefix for "commands to sensors" vs "data from sensors". Then the client could just subscribe to the command channel and ignore the data channel. Seems like the client gateway is going to have to filter out those messages by data type. There is some code that might be doing that in MyMQTT.cpp now but I haven't got a test set up yet so I can't be certain.

[tomkxy]

I don't think myMQTT.cpp is actually doing that. It splits the topics apart to identify receiver etc.
Either I have to check whether I can subscribe to two different topic string. Then I would change the subscription so that it is restricted to set and request "commands".

[TD22057]

@tomkxy said:

I don't think myMQTT.cpp is actually doing that. It splits the topics apart to identify receiver etc.
Either I have to check whether I can subscribe to two different topic string. Then I would change the subscription so that it is restricted to set and request "commands".

Can you easily tell that? Looking at the serial api description, it seems like "set" is used for just about everything and whether it's sensor data or a command is dependent on the context or sender of the message. http://www.mysensors.org/download/serial_api_15 shows an example where a temperature sensor data report and a turn this light off command are both "set" commands. If I light reported itself as on or off, it seems like the message has the same format as the one requesting the light to be on or off. Or maybe I'm missing something somewhere...

It seems like the other gateways know the direction of communication (either gateway->controller or controller->gateway) but the MQTT client doesn't. Since anyone can publish a message, it can't tell whether the message is from or to the controller.

[TD22057]

@tomkxy said:

I don't think myMQTT.cpp is actually doing that. It splits the topics apart to identify receiver etc.
Either I have to check whether I can subscribe to two different topic string. Then I would change the subscription so that it is restricted to set and request "commands".

Looking at the MQTT broker gateway, it can tell the difference between sensors and commands because it's the broker (radio == sensor, MQTT == command). It seems like that's not possible with a client since it just subscribes to a prefix. I still think the client design is a much better system than the fake broker design but it might require using two MQTT prefixes - one for data and one for commands. I think that's a very nice design but it would require changing the MQTT plugins that controllers use to understand that it they want to send commands back to the sensor network.

[tomkxy]

I was thinking along the line to require the controller who wants to send a command or request to a sensor or actuator has to append the command to the topic. So that the topic name space would like that:

MyMQTT/<node id>/<child id>/<sensor type>/<cmd>

cmd could be set, request

The gateway either makes an appropriate subscription or parses the topic correspondingly and if the command is present forwards the message to the addressed sensor.
The question of course is how the MQTT controller bindings work. For OpenHab (v1.x) which I am using - and which is the only one I know at all - this should not be a problem since the binding requires a mapping of the sensors to topic, differentiating between inbound and outbound.

@TD22057: What controller are you using? What control over the MQTT topic name space does the controller offer?

Btw, which controllers do support MQTT at all?

[TD22057]

According to the current docs, these controllers support MQTT: MajorDoMo, OpenHAB, and PiDome. Of course they support it via the current MQTT broker gateway, not the more general system we're building now. I'm using OpenHAB. I haven't done a complete set up with it yet - I'm hoping that OpenHAB 2.0 is ready soon since it seems like that's a big improvement.

It seems like there are three possible types of messages that could go over the MQTT interface:

1. presentations. New nodes/sensor notifications
2. sensor data. Outputs from the sensors
3. commands. Commands sent to the sensors.

In my experience with MQTT, it would be good to make each of these have a different topic structure so it's easy to subscribe to just the messages you can process. For example, the above 3 types could have topics like this:

1. [PREFIX] / new / <node id> / <child id> / <sensor type> with a payload containing the presentation string.
2. [PREFIX] / data / <node id> / <child id> / <sensor type> with a payload containing the sensor data payload (same as current MQTT setup).
3. [PREFIX] / cmd / <node id> / <child id> / <sensor type> / <cmd> with a payload containing the command data and <cmd> is set or req.

The controller can subscribe to types 1 and 2 (to get data from the nodes) and publish type 3 (to send commands to the nodes). The gateway can subscribe to type 3 (to get commands from the controller) and publish types 1 and 2 (by converting radio messages received from the nodes).

I think having the message group (new/data/cmd) as the second topic level makes it easy to do MQTT subscribes like "[PREFIX]/data/#" to get all the sensor data and ignore everything else.

Thoughts?

[tomkxy]

I understand your arguments and it seems to be tempting to follow that approach. However, I am not in favor of that. If we put a functional discriminator in front it separates the name/topic space by functions. If you think about each sensor or actuator as a thing (Internet of thing) which can be addressed - which it is basically - the naming prefix/node/child/sensor specify the address. The prefix is your namespace and defines your sensor network.

Command appended to that will specify particular functions to be applied to that sensor or actuator.

Btw, I had a look to the MQTT (Mosquito) doc. It is possible to subscribe selectively based on that pattern.
For instance to only receive messages with any command, you can subscribe to the topic: 'MyMQTT/+/+/+/+' To only see "set" commands, subscribe to 'MyMQTT/+/+/+/set'

I will implement a change based on that proposition and will make it available on my github repository.

[TD22057]

Sounds good to me. By the way - when I compiled your branch earlier, the pubsubclient was giving tons of compiler warnings. I looked at the original owner and it seems to abandoned (tons of pull requests still pending). Looking through the forks, this repository seems to be the most current version of the pubsubclient library and fixes the issues I saw. Might be worth upgrading that as well.

[tomkxy]

Are you using the ESP8266? The library you referenced seems to be an adaption for ESP8266.

I am using an Ethernet Shield and getting no warning at all?!??!

[TD22057]

Sorry - you're right - wrong library. You have to turn on all compiler warnings in the IDE build options which the main MySensors repository is now doing by default for tests. I'm think that any new pull requests to MySensors need to build without any warnings (at least my last pull to them did) That requires some changes to the pubsubclient - mostly char* -> const char* in the prototypes. I went to do that but pubsub already had a pull request (and 100 more) waiting so I tried to find a fork that looked like it was still under active development to see if the other bugs in the pubsub repository had been fixed. Feel free to ignore me (probably the best approach with my suggestions most of the time anyway)...

[tomkxy]

Feel free to ignore me (probably the best approach with my suggestions most of the >time anyway)...

No problem. I am rather new to the whole Arduino staff. And this sort of things happen to me all the time.

[tomkxy]

Btw, I pushed the changes into my fork. The MQTTClientGateway will now listen to the topic MyMQTT/+/+/+/+ defined in the header file (MQTT_TOPIC_MASK)

If it receives a message on that topic it will parse it and map the command get to a C_REQ message and set to a C_SET message and set it along with the payload to the sensor/actuator.