Is MQTT Necessary? or, Use Case for MQTT?

[TRS-80]

Hello Everyone!

I have been reading and reading, and I think I have now achieved the level of "knows just enough to be dangerous."

There is one thing however that, as a beginner, I haven't been able to get my head around yet. I figured other beginners may also stumble upon the same issue, so I thought I would make this post.

I see a lot of forum posts and information about MQTT, but it seems to me that the better way to connect your gateway to your controller would be just straight serial via USB.

My understanding is that MQTT is a TCP/IP based protocol, which in my mind adds another entirely unnecessary layer to the equation, especially when we are talking about what might (in some cases) be a slightly underpowered gateway device (Arduino Nano) to start with?

So, I suppose these are the questions I have:

1.Does MQTT add some sort of better acknowledgement or traffic control than the MySensors library? I thought that traffic control was already included in the MySensors library (perhaps I am conflating the two)?

2.Is MQTT just "easier" to implement with certain controllers (OpenHab for instance) than straight serial protocol / MySensors library? And/or, does it work better/easier with perhaps more/other devices/controllers on the greater home / automation network? Does MQTT perhaps act as a sort of "glue" to hold everything together across protocols?

3.Or is MQTT only used to make the bridge between controller and gateway (and particularly in cases where they might be physically separated)?

If this is the use case, then maybe this is starting to make sense to me.

I remember reading somewhere that as you move down (or up?) the (IP?) protocol stack, that each layer "encapsulates" the previous layer, and perhaps that is what is happening here? In other words, the MySensors library/protocol becomes "encapsulated" in TCP/IP between the controller and gateway, in order to send the information over a physically greater distance (for example, large 2 or 3 story house with basement, controller in basement and gateway in middle of second floor for better signal on the sensor network)?

Is that the use case then, or is there something else I am overlooking here? Or have I completely misunderstood the issues at hand?

[mfalkvidd]

I'd say that mqtt is the closes we have to an industry standard protocol for IoT. Want to connect to multiple controllers in your home? Mqtt seems to be the easuest solution. Want to send data to Amazon's IoT platform? Mqtt is the solution. Microsoft Azure IoT hub, same thing.

[TD22057]

The biggest thing MQTT adds for me is abstraction. The RF messages MS uses get converted to MQTT which the controller (and other things) understands. This means the controller only needs to understand MQTT, not MS. And I can merge lots of different sensor sources into a single system by converting them all to MQTT.

As a trivial example, I have three sources for temperature data in my house. 1) MySensors nodes, 2) two WIFI thermostats, 3) several Acurite wireless weather sensors. All of those get converted to a single, standard format MQTT temperature messages. So it doesn't matter what the hardware is, if I want to graph/see/store temperature data, I just need to know the MQTT message format.

[Dwalt]

@TD22057 said:

..., 3) several Acurite wireless weather sensors. All of those get converted to a single, standard format MQTT temperature messages.

How are you converting the Acurite sensors to MQTT?

[TD22057]

@Dwalt said:

How are you converting the Acurite sensors to MQTT?

The Acurite Bridge is a 433MHz receiver that reads any Acurite sensors and posts them to the Acurite web site using a simple HTTP call. I have it connected through a USB ethernet dongle to a Raspberry Pi with the dongle bridged to the main network. I have a script which runs tcpflow to intercept that traffic and send it to some simple Python scripts I wrote which convert the data to MQTT. There are instructions here - they're out of date but give you the basic idea. So it's basically a man-in-the-middle attach against the bridge traffic. Let me know if you want more information - I have the code in git but haven't pushed it up to github lately.

[tomkxy]

I am using MQTT mostly because of the reasons @mfalkvidd has given. It is a industry standard protocol and it "opens up" my home automation project. For instance, I am using MqttWarn to publish messages to Instapush independently from the controller I am using. Furthermore, I am going to experiment with OwnTracks or other 3rd party components.

[TRS-80]

OK, reading back over my own post, I realize I have omitted an assumption. I probably should have stated that I am leaning heavily toward using OpenHab as hub software, running on the new Raspberry Pi.

So perhaps I should have phrased my question as "Is MQTT necessary if you are using OpenHab (and especially, with MySensors)?"

Because it seems to me that they are doing largely the same thing (well, insofar as functionally being "the glue" that connects everything together, although achieving it through vastly different means; also, obviously OpenHab is doing much much more, giving you a UI, controller logic/scripts, etc...)?

Or, in other words, does the functionality of OpenHab supersede (and obviously, also greatly extend) that of MQTT?

So, assuming that I was able to implement everything I wanted to do via MySensors + OpenHab, then potentially I could do that just via straight serial connection from RPi/OpenHab to MySensors gateway device (Arduino)? Have I got that about right?

What else would I be missing out on by not using MQTT? Potentially not being able to interface with some other device down the road? It seems as OpenHab has (or will soon have) bindings and integrations for just about everything. And it seems that the MySensors bindings for OpenHab are getting nowadays to a pretty good usability/stability as well.

In the event of any incompatibility or other issue, I would suspect it would be only a matter of time until someone from one of those 2 communities (perhaps even myself ) might contribute some code to remedy the issue, extend functionally, add a feature, etc...

Have I got this about right then? Or, what else am I not seeing?

EDIT: Does MQTT provide some sort of better message queuing / acknowledgement / buffering / persistence than MySensors library (my understanding is that is all done by the MySensors library) for messages on the RF network?

[hek]

You should be fine with a serial connection.

More ambassadors over at the OpenHab community is always welcome!

[TRS-80]

Thanks for the clarification, @hek!

I think I have done just about as much research as is needed to select hub hardware, software, and sort of overall topology and protocols, etc... And I've already got several ideas for various nodes, of course!

Which means, time to order parts soon and start tinkering! I'm excited!

[UglyKidJoe]

@TRS-80, this topic is very interesting (so I though I would revive it )

I'm building my system based on Openhhab2 which it a pretty awesome framework to say the least. Immensely powerful.

Your post here touches on one area that I just can't get my head around - the link between OH2 on the Pi and Arduino RF boards. I'm probably missing something obvious but it's never been clear to me.

Admittedly I haven't set up any Arduino-based sensors yet but I'm hoping to start shortly (my dream is to build motion sensors as they are horribly expensive to buy manufactured (i.e. Aeotec etc)).

Have you bypassed MQTT or has it become integral in your workflow?

[gohan]

With OH2 you can use either serial gateway or mqtt. Mqtt allows you to have much more flexibility in terms of monitoring data or send data to your controller: for example if you want to tinker with sensor values for testing purposes, you could just power off sensor and send data you want using a simple mqtt client, or if you want to play around with multiple controllers you only need to configure them to subscribe to mqtt and they can run simultaneously. I've even seen a post about using zabbix to plot data from mqtt

[hek]

Sidenote: MySensors has a Node RED plugin available as well. It's also a good alternative for tinkering/route messages from a MySensors network. Even more flexible then a raw MQTT broker.

[UglyKidJoe]

@gohan

Thanks for the quick response.

It will be my first implementation of MQTT so I'm pretty much a blank canvas. I'll go with setting up the MQTT broker (server it's called now I believe) on my Raspberry Pi 3.

Installing OpenHAB2 on the Pi has been made a breeze by an unattended installer called OpenHABian. Bundled with the installer is an MQTT option which should set it up (it's accessed via a menu).

I'll be happy to share my lessons learned here, no doubt there will be a few

I've still got to order a few sensors and boards etc to get started so it will take a little while. That;s why I read this thread to get an idea of how the components all talk to each other. That would at least help me determine what I need to buy upfront so I can get busy with it. Any advice would be appreciated (just to get a motion sensor up and running)

I imagine it can become addictive!

[TimO]

For the OpenHAB users: take a look at the MQTT eventbus. This is really cool!
I'm using OpenHAB 2 with the MySensors binding and currently three serial gateways. From OpenHAB 2 the messages are send to the MQTT broker to Node Red.

So:
Serial gateway <-> OpenHAB 2 (with binding) <-> MQTT (Mosquitto) <-> Node Red

[gohan]

@UglyKidJoe
I started with openhabian too, but I figured it's too complicated to start with. So now I moved to domoticz that's much simpler for beginners: at the moment I'm only using one ethernet gateway for sensors but I am planning to move to mqtt later on, after I learned better how to make sensors nodes.

[UglyKidJoe]

@hek

This got me interested so I looked at Node Red. Wow! I started looking at some youtube intros and it looks very good. So let the learning curve begin

I suppose the first challenge will be to figure out how the architectures relate to one another.

Would I be able to use Node Red with the OH2 Z-Wave binding as well as mysensors for instance? Similarly, OH2 controls my WEMO lights, so they will also play with Node Red via OH2?

I know these are real noob questions but it would be a huge help to get set off in the right direction

[UglyKidJoe]

@TimO said:

So:
Serial gateway <-> OpenHAB 2 (with binding) <-> MQTT (Mosquitto) <-> Node Red

Interesting infrastructure. I would imagine there are endless possibilities with it. Thanks for the pointer

I got some homework to do on this, but looks like a helpful introduction video is here: here

[bgunnarb]

@UglyKidJoe
I have been using OpenHAB and MQTT for a number of years now.

The reason for using MQTT and not just a serial connection from the MySensors GW is partly the flexibility as many have stated previously here. Easy to test, using e.g. the mqtt.fx tool. If you want to know what is going on you simply connect to the MQTT stream and "snoop" on the messages.

Another reason is that I am running three geographical sites on one OH installation. OH is running on a NUC at my home. There is also an MQTT GW here. Two other My Sensors MQTT-GW are running remotely in two summer houses. The three GW connect over internet to a cloud-based MQTT broker and OH connects to the same broker. In that way I do not have to open any ports into my home network for access should I have had a local MQTT-broker at home.