Using RaspPi as Gateway + Controller AND MQTT

[robertjuric]

Hey all, home automation new guy here. I have a basic question about the high level components.

I'm planning out a small sensor build, and my thought was to use a RaspPi as the gateway for the sensor network. I understand I could also install the Controller such as OpenHAB. My ultimate goal is to have an Automation Controller reading the sensors, but also to get the data into another system (InfluxDB + Grafana) for historical logging.

I'm a little confused on the use of MQTT in the gateway. Does an MQTT gateway act as a server? So that both the controller and my InfluxDB system could read the MQTT data? I also don't understand the role of the MQTT broker.

Would there be any benefit to building a Arduino based MQTT gateway and dedicating the RaspPi for the controller?

[electrik]

@robertjuric you can see the broker as 'man in the middle' between the gateway and the controller. You will need it when using MQTT. it is lightweight and is easy to install, and personally I like this solution because of its compatibility and flexibility.
I would recommend the ESP or Arduino based gateway instead of a gateway on a raspberry pi directly, as it is easier to modify and replace.

[robertjuric]

@electrik Thanks for the response. Where is the broker normally installed, on the gateway or a separate device?

As the middle man, does the broker act as a server answer requests from multiple clients? Or does the broker broadcast the data to configured controllers?

Basically I want to ensure that the broker can support an OpenHAB controller and getting the data to Telegraf/InfluxDB.

[electrik]

It is normally installed on e.g. a raspberry pi, and can talk to the controller also. Influxdb should be filled through the controller AFAIK. I use this with node red and that works well

[robertjuric]

Ahh, I've done some more research and have a better grasp of that MQTT broker now.

So the sensors publish their readings through the MQTT gateway to the MQTT broker. At the broker multiple devices/services can subscribe to the sensor data which is then published via the broker?

So I would have:
Various sensors
MQTT gateway (ESP or Arduino based)
Raspberry Pi running:

• MQTT broker (Mosquitto)
• Telegraf MQTT Consumer (sends to InfluxDB on another box)
• Controller as MQTT Client (OpenHAB, HomeAssistant, NodeRed)

Does that sound right, or am I missing anything?

[electrik]

Yes that's all you need!

[momo90]

You dont need a esp or arduino as gateway. You can use the RPi as gateway
Look at this tutorials: https://www.smarthomeblog.net/diy-home-automation-sensors/

[robertjuric]

Ahh man, that site is a great resource for putting all the pieces together.

I was debating putting the gateway on the RPi. I see the flexibility in keeping it separate and Arduino based, but consolidating into the RPi would keep things a little cleaner in my control room (closet).

It also seemed like the Ethernet shield for Arduino was discontinued and I want to keep as much ethernet cabled as possible.

[wassfila]

Hi @robertjuric ,
as you're particularly interested in MQTT for logging through influx and grafana, that is exactly what I've been doing so this might be of help.
https://hackaday.io/project/20388-home-smart-mesh
My project did not exactly use MySensors, but I'm interested to get MySensors to work on the rest of my raspberry py, database and dashboard environment.
The raspberry pi has a python script running an influx client here : https://github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx
That client collects MQTT topics and forward them to the database.
some example dashboards are shared here that show how influx queries are written in grafana, also to make hour averages: https://github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/grafana

I do recommend keeping as many things running on the raspberry pi as possible, e.g. MQTT broquer, scripts, services, as linux is an advanced operating system that makes logging and debugging of errors much easier than running on a microcontroller.

I'm not sure how much what I shared is relevant to you, but in case you're interested or have questions, just let me know.

[robertjuric]

Ahh thanks for this additional info. I see that your script is taking the data and writing it directly to the Influx DB. I've been weighing doing it that way, versus using the Telegraf agent with a MQTT plugin. I'll have other Telegraf based metrics coming into Influx for bandwidth, system utilization, etc, but the custom script gives you more flexibility.

I really like your project, I'm going to spend some time looking through everything, since the Influx+Grafana is my ultimate goal from a sensor/data standpoint. Ideally I'd like to be able to see the data behind efficiency improvements through automation or home infrastructure upgrades (new HVAC, water heater, etc). So some of my first goals are to monitor environmentals and power consumption.

I actually have some x86 hardware I use for virtual machines so my concern has been more of what do I outsource to the RaspPi instead of putting on the power powerful hardware. Right now Influx and Grafana will be virtual machines. I'm planning on using the RaspPi for MQTT gateway and Zwave or Zigbee controller. I'm not sure if the HomeAssistant/OpenHAB should go on the Pi as well.

[bgunnarb]

@robertjuric Hi!
For a number of years I was running OpenHAB with influx and grafana on an RPi and it worked very well but eventually I gave that up. A few months back I got a NUC, Intel Celeron based with SSD, $200 from China.

The reason for switching HW was that sooner or later, after a year or so, the SD-card would get corrupt. I would get all sorts of mystical errors. The response time increased and the system got harder and harder to maintain. I suspect that influx data storage to the card was the main culprit. Of course you can connect an SSD and move the system to that but it all got too complex and at the end I felt that the RPi in the long run would not provide the power needed.

Now I run OH 2.3 + influx + grafana under Ubuntu 18.04 and it runs very cool. CPU-load is almost never above 1%.
In my set-up I have two geographical sites with a number of MySensors nodes and some 433 MHz switches. They connect to an MQTT GW on each site. One is a 1:st gen RPi, the other is an MCU.

Since I then have two MQTT streams I use an external, commercial MQTT broker and OH connects as a client to that broker and subscribes to the various topics. I prefer this rather than setting up a VPN from the remote site, which would be an alternative to using the commercial MQTT broker.

If you only have a local set-up of course you could run your mosquitto broker on the same node as OH, influx and grafana. Personally I would still use a separate GW. I feel that this would be easier to maintain but I recognise that this brings one more node to the "closet".

[robertjuric]

Thanks for that info @bgunnarb !