MQTT Setup - Client or Broker & Testing

[diggs]

Fairly new to all of this and still trying to get my head around it.

Setting up for MQTT as my gateway and trying to understand it.

With MQTT running as the gateway, is it acting as a client or a broker? I have read a few conflicting posts.

I have mosquito also running on my system, so when I setup the MQTT sketch for the gateway, do I put in the mosquitto IP and port (the broker) or am I assigning the IP and port to the actual arduino gateway to set it up as a broker ?

I have tried setting up the sketch with an IP and port number as a broker, but cannot seem to connect to it using the testing clients like MQTT Spy and MQTT FX. Should I be able to use these software packages to connect?

Any guidance appreciated.

[tomkxy]

I have running an Arduino with an Ethernet Shield acting as an MQTT gateway. It uses the sketch you can find here in the MQTT Client branch
https://github.com/FotoFieber/MySensors.git

There is a discussion here:
http://forum.mysensors.org/topic/524/mqtt-client-gateway

In general, the MQTT Client Gateway acts as a broker client. So you have to have a Mosquitto broker running somewhere. You need to configure the IP and port from the broker and the MAC address from your ethernet shield. That's basically all.

Please note, that the above mentioned sketch does not fit into the memory of an Uno. So, I bought a Mega.

[TD22057]

Here's my understanding: It's a broker, but not one that implements the full feature set so some clients may not work. I don't believe you need another broker running but I'm not positive about that. I believe that openhab can connect directly to the gateway without another broker running. If you want to run another broker like mosquitto, connect it to the gateway using the "bridge" protocol (google mosquito bridge or search in this forum (see this thread: http://forum.mysensors.org/topic/1034/mqtt-gateway-openhab-mosquitto/2) then connect your clients to mosquitto.

FYI - the v1.5 MQTT gateway fit find on my uno with an ethernet shield.

[diggs]

thanks @TD22057 & @tomkxy

So @tomkxy when you say I need to configure the ip and port from the broker, that would be the mosquito broker you are referring to not the arduino MQTT client gateway?

Is that correct?

Is there still a requirement to set the IP & port on the arduino gateway running the MQTT "client"

[tomkxy]

A general remark: When I was looking at MQTT support I was confused as well. When I looked at the MQTTGateway included in MySensors I was expecting a kind of MQTT broker client. The MQTTGateway - which I never tried - is kind of a broker.
The GIT repositories I was referring to contain a MQTTClientGateway which is based on the MQTTGateway but modified so that they really act as a broker client. They are using a library PubSub which also part of those repositories.

Now tow you question: In the MQTTClientGateway sketch you need to insert the IP from the Mosquito broker. See code snippet below. My Mosquito broker is running on Raspberry PI on 192.168.74 with a listener on the default port.
My Ethernet Shield on the Arduino has the 192.168.178.1, configured in the variable local_ip.

...
//replace with ip of server you want to connect to, comment out if using 'remote_host'
uint8_t remote_ip[] =  { 192, 168, 178, 74 };  // Mosquitto broker

//replace with hostname of server you want to connect to, comment out if using 'remote_ip'
//char* remote_ip = "server.local";
//replace with the port that your server is listening on
#define remote_port 1883
//replace with arduinos ip-address. Comment out if Ethernet-startup should use dhcp. Is ignored on Yun
uint8_t local_ip[] = {192, 168, 178, 11};
//replace with ethernet shield mac. It's mandatory every device is assigned a unique mac. Is ignored on Yun
uint8_t mac[] = { 0xA2, 0xAE, 0xAD, 0xA0, 0xA0, 0xA2 };

//////////////////////////////////////////////////////////////////

[diggs]

@tomkxy thanks for the detailed reply. Will give it a go when I am back home. When testing the MQTTClientGateway, which you say is a Broker Client, then programs like MQTT-Spy would not be able to connect to it. You would really just be looking for the pub's on the Mosquitto to check functionality?? Is that the correct understanding?

With the original MQTTGateway, would be good at some point to find some way of connecting to and testing (mqtt-spy or mqtt.fx etc)

I will eventually work this stuff out!

Cheers

[tomkxy]

I never used MQTT-Spy and yes it is really just a broker client. You can check whether it works subscribing from the command line to the broker.

[Yveaux]

A single MQTT network can contain multiple brokers, which can connect to each other to distribute messages.
From a user perspective it doesn't make a real difference whether the gateway is a broker or a client.
Main advantage of it being a broker is you don't need another broker in your network.
Disadvantage using it as the only broker is that persistence and quality of service are not implemented completely.

[diggs]

Thanks guys.

It is all a little bit clearer now. Will do some playing around.

The problem I currently have is I don't seem to be able to get OpenHAB to publish to my Mosquito broker, but I think it is an OpenHAB issue.

Broker is running and I have the MQTT-spy subscribed to all topics "#"

My broker shows messages sent from the arduino (running MQTT_Basic sketch) as well as messages sent from the command line, but nothing from OpenHab. I have the installed the MQTT binding in OpenHab and enabled in the configuration file. Also setup the switch the way I believe to be correct, but noting sent.

Will quiz them on the openhab forum

[diggs]

Thanks for all help everyone. Finally getting everything up and running generally. I got the system running and talking to OpenHAB initially with the MQTTGateway (1.5) and the instructions here for setting up items etc in openhab. So I can now see Temp and Humidity readings.

http://forum.mysensors.org/topic/303/mqtt-broker-gateway

It ended up pretty straight forward. With that MQTTGateway broker running, I could actually subscribe to it with MQTTSpy and see the data. Only issue was that the MQTTGateway only seems to allow one client connection at a time, so once I had OpenHAB connected I could no longer connect with MQTTSpy, but at least it all worked and I can get reading in OpenHAB.

What I then tried was to bridge my Mosquitto Broker to the MQTTGateway broker to make broker access a little more flexible.

OpenHAB <> Mosquitto <> MQTTGateway

I got them talking/bridged and the data feed made it through to OpenHAB, so all good.

Only question I now have is about subscribing with the Mosquitto Broker in bridge mode.

If I connect MQTTSpy to the Mosquitto Broker and create a catch-all subscribed topic as "#" I see all the readings coming from the MQTTGateway to Mosquitto/OpenHAB, all good. However if I use MQTTSpy and subscribe to a specific topic such as "MyMQTT/20/0/V_HUM" nothing shows in the feed, though that topic is shown under the "#" topic and OpenHAB is picking up the feed data just fine.

This subscription matches how the item is setup and also what I see in the "#" feed.

{mqtt="<[mysensor:MyMQTT/20/0/V_HUM:state:default]"}

Anyone have any suggestions why the direct subscription does not pickup the feed?

Cheers

This is my bridge setting in Mosquitto config

connection MMQTTtoSensor
address 192.168.0.175:1883
clientid MyMQTT
cleansession true
notifications true
topic #

[John Connolly]

I've just built the MQTT gateway as I'm planning to replace the serial one I have in place currently, since I understand MQTT/IP comms a lot better than I do serial. I've built it but I can't connect to it with an MQTT client like MQTT Spy or the command line MQTT tools. I can ping it, but it won't let me connect in other ways.

I've built the gateway with a W5100 shield (which I just don't trust) and messed around with the pin assignments/SOFTSPI etc - I can see the gateway receiving the sensor updates if I open serial monitor. I had a similar issue with some basic webserver tests with that W5100 shield. I should throw it away.

I'm waiting for some new ethernet modules (ENC28J60-based) and will give it another go then. The next step will be to get it bridged to Mosquitto on my Pi. I'll be back...

[TD22057]

@John-Connolly You may want to the try the client gateway from this thread: http://forum.mysensors.org/topic/524/mqtt-client-gateway
It's a real MQTT client (point it at your Mosquitto broker) which I find easier to understand than the pseudo-broker that the MQTT gateway uses.

[tomkxy]

@John: You find the MQTTClientGateway in the MySensors Development branch. It has been merged recently.

[John]

I think this was meant for @John-Connolly ?

[jkandasa]

I guess if create topic like mentioned below(Proposed Topic), we can achieve all the actions that we are doing with serial gateway or Ethernet gateway.

Proposed Topic: MyMQTT/node-id/child-sensor-id/message-type/sub-type
*Current Topic: * MyMQTT/node-id/child-sensor-id/sub-type

Is there any specific reason to skip 'message-type' from topic?

[tomkxy]

@jkandasa: I am not familiar with the Serial Gateway or the Ethernet gateway. What are the particular actions which cannot be done MQTTClientGateway?

With respect to your question whether there is any specific reason to not include the message-type, I have to admit that I am not aware right now of any particular reasons. However, I have to admit that I more or less just adapted the code created by other forum members to run with the new API version.

So in order to get it right you suggest to put the message type defined in the enum of MyMessage.h between the sensor-id and the sub-type?
For example: MyMQTT/node-id/child-sensor-id/0/sub-type -> for the presentation message

[jkandasa]

@tomkxy I guess you are also pointing this MQTT gateway: http://www.mysensors.org/build/mqtt_gateway

If we do not include 'message-type' we could not understand Set,Req,Presentation,Internal and Stream types. Right now we have added MQTT specific type which is not require if we use message-type

In the current design, we cannot do reboot, OTA request/response, get Sensor Name, req from sensor, etc.,

If we follow like we are receiving from Ethernet Gateway or Serial Gateway (node-id;child-sensor-id;message-type;ack;sub-type;payload), no need to maintain types separately in MQTT Gateway code same time we can do all the operations that is supported by MySensors. In MQTT topic will be MyMQTT/node-id/child-sensor-id/message-type/sub-type, we can add payload in mqtt payload and ignore ack bit(if it's 1) in MQTT Gateway itself.

[tomkxy]

@jkandasa No sorry I was referring to the MQTTClientGateway which as acting as a client to a MQTT broker. However, I see your point. The issue to be dealt with in the MQTTClientGateway is that it subscribes to the broker in order to receive messages from the controller and it needs some way to distinguish between inbound and outbound messages otherwise it would try to act on a message it just published on the broker.

One idea was to introduce two disjunct topic hierarchies which I would not like.
If the message types from send from the gateway to the controller and the message types send from the controller to the gateway would be disjoint I might be able to handle that in the MQTTClientGateway. Do you know whether the message types are disjoint?

[John Connolly]

@TD22057 said:

@John-Connolly You may want to the try the client gateway from this thread: http://forum.mysensors.org/topic/524/mqtt-client-gateway
It's a real MQTT client (point it at your Mosquitto broker) which I find easier to understand than the pseudo-broker that the MQTT gateway uses.

That's EXACTLY what I need/want! Thanks! I'll give it a shot the second the new kit arrives from AliExpress...which is taking its time.