Using mqttGateway2.pl for Serial to MQTT on RaspberryPI

[aaron832]

Re: Raspberry PI + MQTT (Mosquitto) + Serial Gateway

Was wondering if anyone else had luck with the mqttGateway2.pl script Yveaux created. I believe I have everything setup correctly. It appears to be initializing correctly. The code is getting all the way down into registration of the on_error and on_read. I know I am definitely getting activity over the serial because I can see it just tailing it. Anyone have any suggestions? Maybe a common pitfall?

```# -- Switch to choose between Ethernet gateway or serial gateway
my $useSerial = 1;

# -- Settings when using Serial gateway
my $serialPort   = "/dev/ttyMySensorsGateway";
my $serialBaud   = 115200;
my $serialBits   = 8;
my $serialParity = "none";
my $serialStop   = 1;

pi@raspi-hab:~ $ sudo tail -f /dev/ttyMySensorsGateway



105;255;3;0;15;
105;255;0;0;17;2.0.0-beta
105;255;3;0;6;0
105;255;3;0;11;Garage Sensor
105;255;3;0;12;1.0
105;0;0;0;7;
105;1;0;0;6;
105;0;1;0;1;36.0

[Yveaux]

@aaron832 serial support is untested in the github version. I have an updated version that runs fine with a serial gateway. I'll upload it to github and let you know.

[Yveaux]

@aaron832 You can now find the latest version at https://github.com/Yveaux/MySensors_MQTTGateway
I did a major rework of the script and you can now configure it from the commandline.
Run the script with --help to get usage info.

E.g. usage:

• Ethernet gateway: ./mqttGateway2.pl --gwhost 192.168.1.100
• Serial gateway: ./mqttGateway2.pl --serial /dev/ttyMySensorsGateway

[aaron832]

Awesome. Thanks Yveaux. Ill test it out today. I think your script is the best solution for a Raspberry PI acting as an OpenSensor gateway and OpenHab server at the moment, which is why I am really glad you made it. A straight serial interface to OpenHab just doesn't work.

[aaron832]

Thanks it works great! On my forked version I just updated the script to contain the latest protocol definitions.

[Yveaux]

@aaron832 good to hear!
I had some issues today after a reboot of my server. Seemed like nothing was coming in at first through the serial gateway. Restarted a few times and looked at the serial output and then it started working suddenly... Fishy...
Anyway, if you create a pull request I can merge your changes, if you like.

[aaron832]

Done. I am a little confused about getting subscriptions working with your script. If a node should be receiving a message from the gateway... should it perform a one time request() or should the subscription be manually added to the /var/run/mqttMySensors/.... subscription file? I guess the confusion being a node that although it might receive a message from a gateway... it may not necessarily ask for it specifically.

[Yveaux]

@aaron832 The idea is that a node subscribes to events automatically once it has requested a value once.
The MySensors' MQTT implementation (for the ethernet gateway) will just subscribe to all events for a sensor once the sensor presents itself to the gateway. This is probably a more convenient implementation, but I just implemented it differently originally.

[aaron832]

@Yveaux I see. Well I added the request() call to my sensor and the perl script is picking it up. However, it is not able to write to the serial device I guess?

20160201-12:14:57 RX: [105;4;2;0;2; ] fr=105 ci=004 c=C_REQ st=V_LIGHT noack -> Subscribe to '/mySensors/105/4/V_LIGHT'
20160201-12:14:57 [Subscribe '/mySensors/105/4/V_LIGHT']
20160201-12:14:57 TX: [105;4;1;0;2;0 ] fr=105 ci=004 c=C_SET st=V_LIGHT noack '0' -> Publish to gateway '/mySensors/105/4/V_LIGHT':'0'
EV: error in callback (ignoring): Filehandle GEN0 opened only for input at /usr/local/lib/arm-linux-gnueabihf/perl/5.20.2/AnyEvent/Handle.pm line 966.
20160201-12:14:58 [105;255;3;0;15;]

pi@raspi-hab:~ $ ls -la /dev/ttyMySensorsGateway
lrwxrwxrwx 1 root root 10 Feb 1 12:14 /dev/ttyMySensorsGateway -> /dev/pts/2
pi@raspi-hab:~ $ ls -la /dev/pts/2
crw-rw---- 1 root tty 136, 2 Feb 1 12:14 /dev/pts/2
pi@raspi-hab:~ $ ps -ealf | grep mq
4 S root 17624 7825 0 80 0 - 1650 poll_s 12:14 pts/0 00:00:00 sudo ./mqttGateway2.pl --serial /dev/ttyMySensorsGateway
4 S root 17631 17624 0 80 0 - 3815 SyS_ep 12:14 pts/0 00:00:01 /usr/bin/perl -w? ./mqttGateway2.pl --serial /dev/ttyMySensorsGateway
0 S pi 17941 13671 0 80 0 - 1065 pipe_w 12:20 pts/4 00:00:00 grep --color=auto mq

[aaron832]

I figure it has something to do with this line from http://cpansearch.perl.org/src/COOK/Device-SerialPort-1.04/SerialPort.pm

# a handle object for ioctls: read-only ok
$self->{HANDLE} = new_from_fd IO::Handle ($self->{FD}, "r");

[hoegaarden_bier]

I am having the same problem when sending the response back to the node over MQTT.
My mqttGateway2.pl crashes and gets in a infinite loop:
Anyone got a sollution for this problem?

20160216-21:38:05 TX: [10;10;1;0;24;1 ] fr=010 ci=010 c=C_SET st=V_VAR1 noack '1' -> Publish to gateway '/mySensors/10/10/V_VAR1':'1'
20160216-21:38:05 Exception: Filehandle GEN0 opened only for input at /usr/local/lib/arm-linux-gnueabihf/perl/5.20.2/AnyEvent/Handle.pm line 966.

20160216-21:38:05 Restarting...
20160216-21:38:05 .
20160216-21:38:05 .
20160216-21:38:05 trying to restore subscriptions
20160216-21:38:05 Restoring previous subscriptions:
20160216-21:38:05 Exception: Filehandle GEN0 opened only for input at /usr/local/lib/arm-linux-gnueabihf/perl/5.20.2/AnyEvent/Handle.pm line 2308.

[Yveaux]

@hoegaarden_bier said:

Exception: Filehandle GEN0 opened only for input at /usr/local/lib/arm-linux-gnueabihf/perl/5.20.2/AnyEvent/Handle.pm line 966.

AnyEvent seems to crash. I run the script using Perl 5.14.2 on 64-bit Ubuntu; you seem to run at an ARM platform...
Probably there's the catch.
Can you try on an Intel Linux system?

[hoegaarden_bier]

thanks for the response, unfortunately I am running the Openhab on my raspberry so switching to linux is not an option....

[aaron832]

@hoegaarden_bier I have the same setup (raspberry pi running openhab with a radio). Two of the different methods for getting Raspberry Pi Gateway sensor serial data to Openhab have failed me. One being the perl script mentioned here. Still haven't figured out why it won't work (fully) under a raspberry pi environment. The other option is having openhab read from serial rather than MQTT as mentioned here: http://forum.mysensors.org/topic/655/serial-gateway-connection-to-openhab/32 However, I have noticed that openhab will run the "Item Arduino received update" rule in parrallel when receiving serial messages in short succession. This causes the parrallel rule to overwrite the currently running rules' variable and it becomes a problem. Its also pretty messy and would require you to update your openhab rules on any mysensor updates.

Not to mention, the Serial Gateway you are likely running on your raspberry pi is for opensensor 1.4 (i think) and it going to be completely left in the dust as soon as 2.0 comes around I think.

If you dare, you can follow my project here: https://github.com/aaron832/Arduino/tree/raspberry-gateway-MQTT (Edit: See Newer Version Below)
Basically I took an updated Raspberry PI serial gateway and updated it to include MQTT.
Just run 'make' under libraries/MySensors and run libraries/MySensors/examples_RPi/PiGatewayMQTT

I am currently looking at getting this stuff onto the develop branch so that it will be around with 2.0

[Artur Maj]

Thanks aaron832 for the gateway.
While running make all from under libraries/MySensors I am getting:
examples_RPi/PiGatewayMQTT.cpp:42:23: fatal error: mosquitto.h: No such file or directory
compilation terminated.
Makefile:102: recipe for target 'examples_RPi/PiGatewayMQTT.o' failed
make: *** [examples_RPi/PiGatewayMQTT.o] Error 1
Can you help?

[Artur Maj]

OK, I had to install mosquitto libraries: sudo apt-get install libmosquitto-dev . All compiled fine. It is always good to ask.

[aaron832]

@Artur-Maj You also have the option of going towards a prototype 2.0 branch that is being worked on. Doing this, you will be able to get support and updates in the future.

My branch is here: https://github.com/aaron832/Arduino/tree/feature/MQTTSupport
Perform "make mqtt" under Arduino/libraries/MySensors
You can change any mqtt settings by uncommenting lines in Arduino/libraries/MySensors/examples_RPi/PiGateway.cpp
The format MQTT messages are as follows:
From MySensors to Openhab: mygateway1-out/node/sensor/command/ack/dataType
From OpenHab to MySensors: mygateway1-in/node/sensor/command/ack/dataType

There is still more work to be done, but it works at least as well as the previous version.

[Artur Maj]

@aaron832 Thanks I will do that. In fact I am connecting MySensors to eventghost not OpenHab but the ide MQTT is the same.

[Artur Maj]

@aaron832 While running make all from libraries/MySensors I am getting errors like that:

examples_RPi/PiGateway.o: In function transportSendWrite(unsigned char, MyMessage&)': /home/artur/Arduino-feature-MQTTSupport/libraries/MySensors/./core/MyTransportNRF24.cpp:102: undefined reference to RF24::powerUp()'
/home/artur/Arduino-feature-MQTTSupport/libraries/MySensors/./core/MyTransportNRF24.cpp:103: undefined reference to RF24::stopListening()' /home/artur/Arduino-feature-MQTTSupport/libraries/MySensors/./core/MyTransportNRF24.cpp:104: undefined reference to RF24::openWritingPipe
and continues....
I have installed : https://github.com/TMRh20/RF24.
What I am doing wrong?

[aaron832]

Try typing

"uname -m"

into your console (that hosts the gateway). If it does not say
"armv6l" or "armv7l" (edit: correction)
then the Makefile will not include -lrf24-bcm in the LDFLAGS (libraries include). If this is the problem then what hardware are you making the gateway on?

[Artur Maj]

@aaron832 It says armv7l. Please, note it is not armv71 the last character is small "L" not 1 (one). This is raspberry pi 2 with ubuntu mate 15.10. I am going to edit Makefile replacing armv71 with armv7l and see what's happening. Thanks again.

[cingolanifede]

Hi, i tried to make the script work but i couldnt. did anybody make it work? I used the serial gateway option, installed all the libraries.

. thankspi@raspberrypi:/ $ sudo ./mqttGateway2.pl --serial /dev/ttyUSB0
Subroutine main::inet_aton redefined at /usr/share/perl/5.20/Exporter.pm line 66.
at ./mqttGateway2.pl line 42.
given is experimental at ./mqttGateway2.pl line 267.
when is experimental at ./mqttGateway2.pl line 269.
when is experimental at ./mqttGateway2.pl line 270.
when is experimental at ./mqttGateway2.pl line 271.
when is experimental at ./mqttGateway2.pl line 272.
Smartmatch is experimental at ./mqttGateway2.pl line 313.
Smartmatch is experimental at ./mqttGateway2.pl line 332.
given is experimental at ./mqttGateway2.pl line 400.
when is experimental at ./mqttGateway2.pl line 402.
when is experimental at ./mqttGateway2.pl line 411.
Name "main::fh" used only once: possible typo at ./mqttGateway2.pl line 207.
can't create /var/run/mqttMySensors/Gateway__dev_ttyUSB0: No existe el fichero o el directorio at ./mqttGateway2.pl line 452.
What does it means?
Thanks

[cingolanifede]

@Yveaux Keep on getting this error

[Yveaux]

@cingolanifede said:

/var/run/mqttMySensors/Gateway__dev_ttyUSB0: No existe el fichero o el directorio at

This path doesn't exist. Create it, or change the path.
All others are warnings, due to an incompatible perl version you're using.

[cingolanifede]

@Yveaux ok. Thanks. I did it. But i am having a different problem now. I have serial gateway running fine. I installed mosquitto and try to follow different steps from the forum and I dont know how to make this work.K cant make the connection beetwen openhab--mosquitto-script--serial gateway. Can you help me ? Is there any post where this is done? Thanks in advance.

[Yveaux]

@cingolanifede I don't have a step by step description for your solution. Searching (and reading) the forum sounds like a good start. If you give us a clear description of what seems to be working and what not, or where you get stuck, it's easier to help.

[cingolanifede]

@Yveaux 1) The script is running. It reads from serial port, this is what I get, then it crashes, and says broken pipe.

20160509-10:48:33 RX: [0;0;3;0;9;read: 30-30-0 s=1,c=1,t=16,pt=2,l=2,sg=1:1] fr=000 ci=000 c=C_INTERNAL st=I_LOG_MESSAGE noack 'read: 30-30-0 s=1,c=1,t=16,pt=2,l=2,sg=1:1' -> Publish to broker 'MyMQTT/0/0/I_LOG_MESSAGE':'read: 30-30-0 s=1,c=1,t=16,pt=2,l=2,sg=1:1'
20160509-10:48:33 RX: [30;1;1;0;16;1 ] fr=030 ci=001 c=C_SET st=V_TRIPPED noack '1' -> Publish to broker 'MyMQTT/30/1/V_TRIPPED':'1'
20160509-10:48:43 Exception: Error: Tubería rota

20160509-10:48:43 Restarting...
20160509-10:48:43 trying to restore subscriptions

====2)On openhab log I get this========

2016-05-09 10:48:33.189 [ERROR] [o.i.t.m.i.MqttBrokerConnection] - MQTT connection to broker was lost
org.eclipse.paho.client.mqttv3.MqttException: Se ha perdido la conexión
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138) [mqtt-client-0.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.8.0]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[na:1.8.0]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:56) ~[na:na]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:100) [mqtt-client-0.4.0.jar:na]
... 1 common frames omitted
2016-05-09 10:48:33.191 [ERROR] [o.i.t.m.i.MqttBrokerConnection] - MQTT connection to 'mysensor' was lost: Se ha perdido la conexión : ReasonCode 32109 : Cause : null
2016-05-09 10:48:33.193 [INFO ] [o.i.t.m.i.MqttBrokerConnection] - Starting connection helper to periodically try restore connection to broker 'mysensor'

On openhab.cfg i put this and i have the addons installed

mqtt:mysensor.url=tcp://192.168.1.109:1883
mqtt:mysensor.clientId=mys

A simple switch setting
Switch node2_sw1 "sw1" (node2,all) {mqtt=">[mysensor:MyMQTT/30/10/V_LIGHT:command:ON:1],>[mysensor:MyMQTT/30/10/V_LIGHT:command:OFF:0]"}

I think the problem is that i am not setting mosquitto well beetwen openhab and your script. I tested an example publishing and i get the message. Do I have to change conf.d file? or setting?

Thanks.

[cingolanifede]

@Yveaux I made improvement:
I am using mosquitto_sub to read what the script posted. Thats ok.

pi@domotic /opt/domotic/runtime $ mosquitto_sub -d -t MyMQTT/30/1/V_TRIPPED
Client mosqsub/3629-domotic sending CONNECT
Client mosqsub/3629-domotic received CONNACK
Client mosqsub/3629-domotic sending SUBSCRIBE (Mid: 1, Topic: MyMQTT/30/1/V_TRIPPED, QoS: 0)
Client mosqsub/3629-domotic received SUBACK
Subscribed (mid: 1): 0
Client mosqsub/3629-domotic received PUBLISH (d0, q0, r1, m0, 'MyMQTT/30/1/V_TRIPPED', ... (1 bytes))
1
Client mosqsub/3629-domotic received PUBLISH (d0, q0, r0, m0, 'MyMQTT/30/1/V_TRIPPED', ... (1 bytes))
0
Client mosqsub/3629-domotic sending PINGREQ
Client mosqsub/3629-domotic received PINGRESP
Client mosqsub/3629-domotic sending PINGREQ
Client mosqsub/3629-domotic received PINGRESP
Client mosqsub/3629-domotic sending PINGREQ
Client mosqsub/3629-domotic received PINGRESP

The thing is when I start openhab this happens, it crashes and i get this error. What could it be?

2016-05-09 11:56:52.216 [ERROR] [o.i.t.m.i.MqttBrokerConnection] - MQTT connection to broker was lost
org.eclipse.paho.client.mqttv3.MqttException: Se ha perdido la conexión
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138) [mqtt-client-0.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.8.0]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(DataInputStream.java:267) ~[na:1.8.0]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:56) ~[na:na]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:100) [mqtt-client-0.4.0.jar:na]
... 1 common frames omitted
2016-05-09 11:56:52.219 [ERROR] [o.i.t.m.i.MqttBrokerConnection] - MQTT connection to 'mysensor' was lost: Se ha perdido la conexión : ReasonCode 32109 : Cause : null
2016-05-09 11:56:52.220 [INFO ] [o.i.t.m.i.MqttBrokerConnection] - Starting connection helper to periodically try restore connection to broker 'mysensor'

[Yveaux]

The script is running for months without issues on my place; both for serial and Ethernet gateway. I am not using the script in combination with a raspberry pi. Maybe there is an issue with the version of mosquitto and/or AnyEvent::MQTT.
I remember other people having issues in combination with raspberry pi - search the forum.

[cingolanifede]

@Yveaux Thanks again. I used a phyton script. Now is working.

[robosensor]

Not sure I'm doing this right way, but changing this line:

        fh => $serialDevice->{'HANDLE'},

to this line

        fh => new_from_fd IO::Handle ($serialDevice->{FD}, "w"),

Fixed Filehandle GEN0 opened only for input bug for me on RPi 2. Fix works because of $serialDevice->{'HANDLE'} is opened as read only in SerialDevice and should not be used for write:

    # a handle object for ioctls: read-only ok
    $self->{HANDLE} = new_from_fd IO::Handle ($self->{FD}, "r");