Instructions how-to use Zabbix as your Controller with MQTT, part 1



  • Note: edited 2017-01-22. Key format on Zabbix Items has been modified to fit Auto-Discovery and automatic creation of sensors and nodes. Item-key = child-id. See part 2.

    I wanted to evaluate Zabbix as a Controller, installing it on my Raspberry Pi and get it to work together with MySensors. Here's my experience.

    1. Introduction

    Zabbix is a very powerful Open Source monitoring system used to monitor e.g. IT or Telecom Data Centers and Data Networks. It comes with a variety of functions, it's very flexible, has a strong Forum and a lot of documentation. It can be integrated via zabbix.api, zabbix-sender or it’s Low Level Discovery protocol zabbix LLD, among other stuff. There are several software libraries developed for a variety of program languages for integrations with external systems and software. Take a look at the web pages www.zabbix.org and www.zabbix.com.

    So far Zabbix has been running smoothly on my Raspberry Pi 2 together with my MySensors integration.

    I have it running with an MQTT broker, MQTT Gateway and some sensors. In fact, it’s running in parallel with a Domoticz Controller, both registering the same data from my sensors, installed on the same Raspberry.

    • Here are a couple of screens from my setup, showing temperature and energy sensors in operation.

    0_1484536092844_zabbix power_2.png
    0_1484528051690_zabbix temperature.png
    0_1484528092916_zabbix dht22.png

    • Zabbix comes with a Plug and Play Linux Servers monitoring setup via preinstalled templates, like the Raspberry Pi. It sends alerts automatically if there’s some issue. Here's a screen example from my setup.

    0_1484538364986_zabbix Linux_2.png

    1. Setup procedure. MQTT Broker

    First the Raspberry needs to have an MQTT broker running to capture and distribute MQTT messages, from a MySensors MQTT gateway client in this case. Mosquitto I guess is the best choice.
    Follow the instructions on this page to install the MQTT Mosquitto Broker:
    http://www.instructables.com/id/Installing-MQTT-BrokerMosquitto-on-Raspberry-Pi/
    Or run directly with the following commands on your Raspberry Pi:

    sudo apt-get update
    sudo apt-get install mosquitto
    sudo apt-get install mosquitto-clients
    
    1. Instructions on how to install zabbix on the Raspberry Pi, using MySQL as database

    Zabbix needs a database to run. MySQL is selected as database in this setup.
    There are two packages needed from Zabbix to get it working. zabbix-server-mysql and zabbix-frontend-php. As well it’s recommended you install zabbix-agent on the Raspberry. Zabbix can then monitor the Raspberry Pi performance automatically through the zabbix-agent tpgether with pre-installed templates.
    If you install Zabbix directly via "apt-get install" the version you get is rather old (zabbix 2.x), and I got into some trouble trying to get it to work. The web page further down did work straight forward in my case. It installs zabbix v3.0. Have a look at the zabbix 3.0 manual: https://www.zabbix.com/documentation/3.0/manual

    If you don't have mysql installed. Do this first on the Raspberry:

    sudo apt-get install mysql-server mysql-client
    
    1. Installing zabbix 3.0

    Follow the page, step by step, to install Zabbix on your Raspberry Pi. It installs the version zabbix 3.0:

    http://devopsish.blogspot.com/2016/05/installing-zabbix-3-on-raspberry-pi.html

    You should now have got installed

    • zabbix-server-mysql

    • zabbix-frontend-php

    • zabbix-agent

    on your Raspberry Pi. You can now access Zabbix from a web browser pointing at the Raspberry IP address. Eg. http://192.168.1.70/zabbix

    • You should get to the login screen.

    0_1484528956759_zabbix Login.png
    Enter with User = Admin, Password = zabbix. Later on you can add and modify other users for access control.

    1. Setup zabbix-sender

    Additionally we’ll need to install the package zabbix-sender to be able to send the MQTT messages from a script to the Zabbix Server.
    On your Raspberry Pi enter:

    cd ~
    cd zabbix3-rpi-master
    sudo dpkg –i zabbix-sender_3.0.*-1+jessie_armhf.deb
    
    1. Zabbix setup concepts

    To pass MQTT messages from the MQTT broker to Zabbix I wrote a small Python script. Please note, the MQTT format with MySensors is "topic/a/b/c/d/e", but Zabbix doesn't allow the letter "/" in an “item key”, it's therefore changed in the Python script to the format "topic.a.b.c.d.e".
    In Zabbix I have setup a hostgroup named “MySensors”. Added to the hostgroup are the sensor nodes (hosts). To the hosts are added the sensor children (items) following the MySensors MQTT format:

    MY_MQTT_TOPIC_PREFIX/SENSOR-NODE-ID/SENSOR-CHILD-ID/CMD-TYPE/ACK-FLAG/SUB-TYPE

    • hostgroup = "MySensors”

    • host = MY_MQTT_TOPIC_PREFIX + "." + MY_NODE_ID
      eg. mygateway1-out.5

    • item = CHILD_ID + "." + CMD-TYPE + "." + "ACK-FLAG" + "." + SUB-TYPE
      eg. 1.1.0.0 (Child Id = 1, Cmdtype = set, Ack = 0, Subtype = V_TEMP)

    The Zabbix items should be set to the type “zabbix trapper”. You find the items at -configuration-, -host-, -item-. Zabbix trapper listens to messages sent from “zabbix-sender”, which we’re going to use here.
    Be aware. No automatic node assignment to the Sensor Nodes from the Controller would work with Zabbix in this setup.
    So we need to preset the Node_ID in the MySensor arduino scripts e.g.:

    #define MY_NODE_ID   1
    
    1. Zabbix setup
    • Define a hostgroup and give it a name.

    0_1484529453507_zabbix hostgroup.png

    • Define some hosts (sensor nodes). The host name must have an exact match with your mqtt gateway topic name and node id separated with a dot (MY_MQTT_TOPIC_PREFIX + "." + MY_NODE_ID)
      In this example I'm using “domoticz.in.MyMQTT.11” with the node = 11.

    0_1484529557434_zabbix host.png

    • Define some items (children) to your sensor nodes
      Give them a name of your choice. Select Type = Zabbix trapper. The key for each item is the Child-Id taken from the MySensors MQTT message. In this case Child-ID = 1.

    Note: edited 2017-01.22. Key for items has been modified to only use Child-id, to fit integration of Auto-Discovery. See part 2.

    0_1485145829823_zabbix item.png

    1. Zabbix-sender

    There is an executable that’s now available “zabbix_sender” that you can use to test directly from the command line in the Raspberry. Try sending to an item that’s been previously setup in Zabbix by executing the command:

    zabbix_sender -v -z localhost -s domoticz.in.MyMQTT.11 -k 1 -o 10.5
    

    A response should look something like:

    info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000176"
    sent: 1; skipped: 0; total: 1
    

    If failed = 0, sent = 1 everything should be ok in the setup.

    You should now see the message show up in Zabbix. Look it up in Zabbix, in “Monitoring” and select “Latest Data”. You should see an event with temperature 10.5 °C registered in the node = 11, Child = 1. That is, hostname = “domoticz.in.MyMQTT.11” with key item = “1” and with the payload = 10.5 received.

    Note: edited 2017-01-22. Key format modified to key=child-id to fit Auto-Discovery. Check on part 2.

    1. Python script

    The Python script uses two libraries that we have to download first:
    Mosquito client library. See: www.mosquitto.org and https://pypi.python.org/pypi/paho-mqtt

    pip install paho-mqtt
    

    Library that uses zabbix-sender / zabbix-api See: https://www.zabbix.org/wiki/Docs/api/libraries
    And: https://github.com/blacked/py-zabbix

    pip install py-zabbix
    

    Prepare a script named e.g. “mqtt_zabbix.py”, with your favorite editor, like vi or nano.

    nano mqtt_zabbix.py
    

    Paste the Python script below and save the file. Run for example the script in the background, as a daemon, from the command line. The script stores the stdout and stderr printouts in the text file “zabbix.log”:

    python -u mqtt_zabbix.py > zabbix.log 2>&1 &
    

    The file zabbix.log stores the sent MQTT events, with a time stamp and the result received from zabbix-sender. Useful for debugging.

    Python script:

    import paho.mqtt.client as mqtt
    import time
    from  pyzabbix import ZabbixMetric, ZabbixSender
    
    topic_sub = "domoticz/in/MyMQTT/"
    
    # The callback for when the client receives a CONNACK response from the server.
    def on_connect(client, userdata, flags, rc):
        print("Connected with result code "+str(rc))
    
        # Subscribing in on_connect() means that if we lose the connection and
        # reconnect then subscriptions will be renewed.
        client.subscribe(topic_sub+"+/+/+/+/+")
    
    # The callback for when a PUBLISH message is received from the server.
    def on_message(client, userdata, msg):
        msg.topic = msg.topic.replace("/", ".")
    
        myNode = msg.topic[:msg.topic.find(".", len(topic_sub)+1)]
        myItem = msg.topic[len(myNode)+1:]
        mySplit = myItem.split('.')  # split topic to indexed list
    
        if mySplit[1] == '1':        # type = set
            myItem = mySplit[0]      # change item to child-id
    
        metrics = [ZabbixMetric(myNode, myItem, msg.payload)]
        result = ZabbixSender(use_config=True).send(metrics)
    
        print(time.strftime("%c")+" "+str(result)+" : "+myNode+" "+myItem+" "+str(msg.payload))
    
    
    client = mqtt.Client()
    client.on_connect = on_connect
    client.on_message = on_message
    
    client.connect("localhost", 1883, 60)
    
    

    In the script, to change the MQTT format from topic/a/b/c/d/e to topic.a.b.c.d.e, required by zabbix, this Python line fixes it:

        msg.topic = msg.topic.replace("/", ".")
    

    These two lines make the deal to send the MQTT events to Zabbix:

        metrics = [ZabbixMetric(myNode, myItem, msg.payload)]
        result = ZabbixSender(use_config=True).send(metrics)
    

    It’s possible to add triggers to the items in Zabbix, to generate alerts from MQTT sensor events received. Alerts can as well be sent by e.g. SMS, Email or script (outside this scope).

    My next steps will be to create sensor nodes automatically (hosts) and it’s children (items), by preparing some Templates in Zabbix together with zabbix.api or zabbix LLD. And as well have a look into how to send messages from Zabbix to MySensor relays and actuators, and for autonomous flow actions.

    Enjoy and have fun!


  • Admin

    Creative use of Zabbix @jpaulin!


  • Mod

    Nice tutorial. My question is: what do you get more from zabbix other than what you already have in domoticz?



  • @gohan

    integration to existing monitoring infrastructure


  • Mod

    @b0rmann ok, that makes sense, because I wouldn't go through the hassle of installing zabbix at home 😄



  • @gohan I would add, more flexibility and control over your Controller set-up, nicer look and easier for integration with different MySensors networks. Or other IOT solutions on the same monitoring platform. The graphs are more powerful and nicer to look at. A number of features are available and it's possible to integrate with most anything you'd like through the zabbix-API or zabbix-sender. I was using Domoticz, but somehow I got bored with Domoticz when I got Zabbix installed on my RPI. For the moment I have both running in parallel listening to the same topics from the mosquitto broker.



  • @jpaulin
    I've prepared another thread today showing how it's possible to add Auto-Discovery and automatic creation o MySensor nodes and sensors to Zabbix. Please have a look at this thread.


  • Mod

    At the moment I'm still learning the hw part of home automation, so I haven't looked much at the data logging and graphs yet and forgive me for any silly questions, but when you said you got tired of domoticz after you set up zabbix, are you referring to the graph viewing? Isn't also node red used to display graphs?



  • @gohan haven't tried node red. But I have heard some positive feedback about it. Maybe I'll look into it later and give it a try. Yes mainly to the static graphs.


  • Mod

    Of course node red can do much more than showing graphs, but I'll tinker with it later on



  • @jpaulin

    very nice ... same idea, and found your tutorial

    Is it working well? What are your latencies from sensor-event to zabbix-action/outgoing?
    Did you prepared the Templates in Zabbix?

    thanks ... Manfred



  • @mhe
    Zabbix works very well with a Raspberry Pi in my opinion. I have a Raspberry-2B and the interactions with graphs, maps and screens are instantaneous. Click a graph or screen and the response is immediate. It's accessible over the wifi to any device with a web-browser. Make a zoom of the graphical data and the response is instantaneous. Even with more than one year of sensor data stored in Zabbix it's still equally smooth and fast. The CPU load in the RPI is low. See the example graph above (CPU below 5%). Have tested some other controllers, but Zabbix has been my best experience so far. The data collected from the MQTT broker from the sensors is stored as well without any delay in Zabbix.
    One thing I could notice. There's a higher rate of Read/Write to the SD-micro card. I guess it probably wears out faster. I did have an SD-card that crashed with my RPI where Zabbix runs.

    Templates is a good solution when there are many sensors of the same type. I didn't use templates in this project, no need in my case. If you look at part 2, there's an Auto-Discovery solution with Zabbix I made to create the hosts with sensor data automatically.
    If you refer to the examples above of Screens I prepared. They are easily created inside Zabbix to organize and display the sensor data at you preference.



  • @jpaulin
    thank you very much for feedback. Sorry for the late reply ... I missed the forum-notification.

    The wear-out of micro-sd-cards is a well known problem. I would use industrial-grade cards or ssd-drive (if power and size not matter). Industrial grade micro-sd are expensive. The "pSLC" type cards would be a good fit "better power-fail protection, very durable ... not very expensive (from 30 EUR...)".


 

421
Online

7.9k
Users

8.8k
Topics

93.8k
Posts