MQTT Client gateway

[ntruchsess]

I just took the time this afternoon and implemented a mqtt-gateway that works as mqtt-client:

https://github.com/ntruchsess/MySensors/tree/mqttclient/libraries/MySensors/examples/MQTTClientGateway

it is based on the work of @Damme (http://forum.mysensors.org/topic/260/mysensors-mqtt-gateway) and makes use of nick o'learys PubSubClient-library (http://knolleary.net/arduino-client-for-mqtt/)

The main difference to the existing MyMQTT-gateway is that it will connect to an existing mqtt-broker (e.g. mosquitto). This is beneficial if you want to integrate with other mqtt-client-devices sharing a single broker for your automation solution. It also ensures your mqtt-client that is going to pick up the messages from the gateway will talk to a 'real' broker understanding the whole mqtt protocolls semantics

It works both ways:
on startup or reconnect it subscribes to "MyMQTT/#" (configurable as MQTT_PREFIX). Any publish-messages payload that is delivered by the broker to a topic that complies with the sheme <prefix>/<nodeId>/<childId>/<variable_type> is forwarded as a C_SET-msg to the MySensors radio.
Any C_SET-message received from a node is published to the mqtt-broker using the same sheme.

• Norbert

[Yveaux]

@ntruchsess Hey man, thanks for the effort!
I tried the same in the past, but more or less stranded on the available amount of flash on an ATMega328.
When all MySensors related constants have to be translated to strings flash tends to run out very fast...
It made me realize I needed more powerful hardware anyway to run e.g. the MQTT broker and controller on, so I chose the path to implement & run the MQTT translation outside the gateway (see https://github.com/Yveaux/MySensors_MQTTGateway)

How much flash does your MQTT gateway consume?

[ntruchsess]

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/00_MYSENSORS.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/10_MYSENSORS_DEVICE.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Constants.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Message.pm

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

[Johnny B Good]

@ntruchsess said:

https://github.com/ntruchsess/fhem-mirror/blob/mysensors_unified

Hi ntruchsess,

can you fix your link to --> https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM
Your links don't work..

Greetings ...
;)

[ntruchsess]

@Johnny-B-Good said:

Your links don't work..
@ntruchsess said:

https://github.com/ntruchsess/fhem-mirror/blob/mysensors_unified

Thanks for giving notice, I've just merged into master and deleted branch 'mysensors_unified', links are fixed now.

[Yveaux]

@ntruchsess said:

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

So not much space left on an ATMega328 with default bootloader.

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

No probs! Go ahead and hopefully it'll lead to something more userfriendly/maintainable than my implementation.

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

I'm not a fan of both Perl MQTT implementations but wanted to make a headstart...
AnyEvent can be very powerful but a bit overkill for just a MQTT gateway and badly documented.

[Yveaux]

@ntruchsess said:

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

So not much space left on an ATMega328 with default bootloader.

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

No probs! Go ahead and hopefully it'll lead to something more userfriendly/maintainable than my implementation.

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

I'm not a fan of both Perl MQTT implementations but wanted to make a headstart...
AnyEvent can be very powerful but a bit overkill for just a MQTT gateway and badly documented.

[novicit]

Forgive what may be a simple question, I am a novice in code but trying
to learn. I am trying to compile the MQTTClientGateway listed in the
first message of this thread.I have downloaded the MQTTClientGateway
code as well as PubSubClient software. I have made absolutely no
changes to any of the code. But when I compile I get the following error:

In file included from /MyMQTTClient.h:17,
from MQTTClientGateway.ino:63:
C:\Arduino\libraries\PubSubClient/PubSubClient.h:68: error: section attribute not allowed for '<anonymous>'

If I can get this to compile clean, I can see if it will work with a
Domoticz setup I already have working on a RPi, with Mosquitto.

I believe the offending line is:
void disconnect();

Since I made no changes, & clearly this compiled by the original
authors, why do I get an error, and how do I fix it?

Any help will be appreciated.

Regards,

PS: below is the full code section

PubSubClient();
PubSubClient(uint8_t , uint16_t, void()(char*,uint8_t*,unsigned int),Client& client);
PubSubClient(uint8_t , uint16_t, void()(char*,uint8_t*,unsigned int),Client& client, Stream&);
PubSubClient(char*, uint16_t, void()(char,uint8_t*,unsigned int),Client& client);
PubSubClient(char*, uint16_t, void()(char,uint8_t*,unsigned int),Client& client, Stream&);
boolean connect(char *);
boolean connect(char *, char *, char *);
boolean connect(char *, char *, uint8_t, uint8_t, char *);
boolean connect(char *, char *, char *, char , uint8_t, uint8_t, char);
void disconnect();
boolean publish(char *, char *);
boolean publish(char *, uint8_t *, unsigned int);

[ntruchsess]

Ok, I can reproduce this with Arduino IDE 1.0.6.
It did compile fine on IDE version 1.0.5 though.

EDIT:
go to PubSubClient.h and remove the 'PROGMEM' in line 72:

change:
boolean publish_P(char *, uint8_t PROGMEM *, unsigned int, boolean);
to:
boolean publish_P(char *, uint8_t *, unsigned int, boolean);

see issue on github and pullrequest with fix:
https://github.com/knolleary/pubsubclient/issues/46
https://github.com/knolleary/pubsubclient/issues/47

[novicit]

@ntruchsess , thank you very much for taking a look, much appreciated.

I deleted PROGMEM as you suggested, resulting in the following:

C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\MQTTclientGateway\MyMQTTClient.cpp: In constructor 'MyMQTTClient::MyMQTTClient(PubSubClient, uint8_t, uint8_t)':
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\MQTTclientGateway\MyMQTTClient.cpp:116: error: no matching function for call to 'PubSubClient::PubSubClient()'
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:59: note: candidates are: PubSubClient::PubSubClient(char*, uint16_t, void ()(char, uint8_t*, unsigned int), Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:58: note: PubSubClient::PubSubClient(uint8_t*, uint16_t, void ()(char, uint8_t*, unsigned int), Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:57: note: PubSubClient::PubSubClient(Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:40: note: PubSubClient::PubSubClient(const PubSubClient&)

Looks to me like the compiler is finding the code ambiguous. For novices like me this is more challenging as the code is fine, just seems to confuse the compiler. I am using Arduino IDE 1.0.6 as you suspected. I am able to compile fine the examples included with pubsubclient, Ethernet, etc. Originally I was using a MAC, so I then tried a Windows pc in case something was there, but same results. Since you are able to compile after removing PROGMEM, I assumed it was part of my setup.

Trial & error is frustrating, but perhaps a way for novices to learn.

I am sure you are busy, so thanks for taking a look!

[Victor Klijmeij]

Hi novicit, Look at https://github.com/knolleary/pubsubclient/pull/47/files
ntruchsess proposes changes to pubsubclient header and cpp files

[binladin]

Hello! Thanks for your great work.
I have some question. After starting MajorDoMo controller and mosquitto on my PC, i made MQTTClientGateway and DHT MySensors node, and in mosquitto log i can see:

1415812670: Received PUBLISH from MySensor (d0, q0, r0, m0, 'MyMQTT/20/1/V_TEMP'
, ... (4 bytes))
1415812670: Sending PUBLISH to MySensor (d0, q0, r0, m0, 'MyMQTT/20/1/V_TEMP', .
.. (4 bytes))
1415812670: Sending PUBLISH to MajorDoMo MQTT Client (d0, q0, r0, m0, 'MyMQTT/20
/1/V_TEMP', ... (4 bytes))

Can you explain why mqttclientgateway subscribing to its own data in next part of sketch:

if (!client.connected())
{
client.connect("MySensor");
client.subscribe("MyMQTT/#");

I apologize if my question will seem ignorant, and sorry for my poor english.

[ntruchsess]

intention is you may controll MySensor-actuators via mqtt-client-gateway by publishing messages to topic MyMQTT/<radioId>/<childId>/<variableType>

[novicit]

SUCCESS! Found the reason I could not compile the MQTT Gtwy client.

When I first looked for the PubSubClient library, used google and went to http://knolleary.net/arduino-client-for-mqtt/ which has a link to GitHub for the library. That is the one I used. When @Victor-Klijmeij suggested I look at @ntruchsess suggested changes again - I noticed the line numbers were not the same as mine, but the code was. Bottom line .... I had an old version of PubSubClient. When I used the version @ntruchsess made changes to, it compiled immediately after his suggested changes. The old PubSubClient .cpp file was ~70 lines of code shorter than the current version.

Thank you both again for your help, much appreciated! @ntruchsess, great job, thanks for sharing the code.

[b0rmann]

PubSubClient library cannot currently be used with hardware based on the ENC28J60 :(

[ntruchsess]

@b0rmann
PubSubClient and ENC28J60 require more memory than an Uno provides. It should run on a Mega256 though.

[b0rmann]

DEBUG:

0;0;3;0;9;read: 16-16-0 s=61,c=1,t=0,pt=7,l=5:38.0
publish: MyMQTT/16/61/V_TEMP 38.0
0;0;3;0;9;send: 0-0-16-16 s=61,c=1,t=0,pt=0,l=5,st=ok:38.00 
0;0;3;0;9;send: 0-0-18-18 s=1,c=1,t=2,pt=0,l=5,st=ok:18.00

mqtt payload for node 18 contain only string '1' (it's relay), '8.00'- trash from previous message :(
yes, '18.00' may by interprait as boolean TRUE, but '01.00' (next par of log is not FALSE :(

0;0;3;0;9;read: 16-16-0 s=91,c=1,t=0,pt=7,l=5:61.0
publish: MyMQTT/16/91/V_TEMP 61.0
0;0;3;0;9;send: 0-0-16-16 s=91,c=1,t=0,pt=0,l=5,st=ok:61.00
0;0;3;0;9;send: 0-0-18-18 s=4,c=1,t=2,pt=0,l=5,st=ok:01.00

dirty hack - i use '000000000000' instead '0' to rewrite old data

[DirkB19]

Hi Norbert, Bormann or Novicit,

I'm an Arduino newbie and so far haven't found a way to get the MqttClientGateway to compile.
Obviously I'm using the wrong include files (libraries), so could someone give me some precise links as to where to download the right PubSubClient stuff, or other libraries, if any needed to compile. (I'm not really finding my way in Github).

My setup is Arduino UNO+W5100, Mosquitto on RPi (working :-) and Openhab (working :-). On the Raspberry I already have the mqtt-gpio-monitor working with Mosquitto and Openhab, now I want to add Mysensors;

Thanks in advance,
DirkB

UPDATE :+1:
Ok, I found it meanwhile. Everything works now. Was complicated for a newbie, had some issues with libraries not being in the right folders as well.
Thanks.

[SagittaTheArrow]

Has anybody tried to get this compiled with the development branch (1.4.1) with the RFM69 radios? I have been trying to get a successful compile, but no luck thus far. I do have it working with the stable branch, but I am not happy with the radios and would love to get this client working with the RFM69 radios. I have looked over the code and started to merge the two, but I am not a C++ programmer and have had some difficulty with the following error:
yMQTTClient.cpp:16:14: error: variable 'VAR_0' must be const in order to be put into read-only section by means of 'attribute((progmem))'
char VAR_0[] PROGMEM = "TEMP"; //V_TEMP
I tried some of the workarounds that I found (const char VAR_0) but then the GetType functions break and I do not understand enough of the code to correct that issue. Anyway, I thought I would check here to see if anybody else has made some headway.

Thanks in advance for any information / help.

[hek]

what happens if you add const as the compiler suggest?

const char VAR_0[] PROGMEM = "TEMP";