MQTT Client gateway

[ntruchsess]

I just took the time this afternoon and implemented a mqtt-gateway that works as mqtt-client:

https://github.com/ntruchsess/MySensors/tree/mqttclient/libraries/MySensors/examples/MQTTClientGateway

it is based on the work of @Damme (http://forum.mysensors.org/topic/260/mysensors-mqtt-gateway) and makes use of nick o'learys PubSubClient-library (http://knolleary.net/arduino-client-for-mqtt/)

The main difference to the existing MyMQTT-gateway is that it will connect to an existing mqtt-broker (e.g. mosquitto). This is beneficial if you want to integrate with other mqtt-client-devices sharing a single broker for your automation solution. It also ensures your mqtt-client that is going to pick up the messages from the gateway will talk to a 'real' broker understanding the whole mqtt protocolls semantics

It works both ways:
on startup or reconnect it subscribes to "MyMQTT/#" (configurable as MQTT_PREFIX). Any publish-messages payload that is delivered by the broker to a topic that complies with the sheme <prefix>/<nodeId>/<childId>/<variable_type> is forwarded as a C_SET-msg to the MySensors radio.
Any C_SET-message received from a node is published to the mqtt-broker using the same sheme.

• Norbert

[Yveaux]

@ntruchsess Hey man, thanks for the effort!
I tried the same in the past, but more or less stranded on the available amount of flash on an ATMega328.
When all MySensors related constants have to be translated to strings flash tends to run out very fast...
It made me realize I needed more powerful hardware anyway to run e.g. the MQTT broker and controller on, so I chose the path to implement & run the MQTT translation outside the gateway (see https://github.com/Yveaux/MySensors_MQTTGateway)

How much flash does your MQTT gateway consume?

[ntruchsess]

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/00_MYSENSORS.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/10_MYSENSORS_DEVICE.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Constants.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Message.pm

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

[Johnny B Good]

@ntruchsess said:

https://github.com/ntruchsess/fhem-mirror/blob/mysensors_unified

Hi ntruchsess,

can you fix your link to --> https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM
Your links don't work..

Greetings ...

[ntruchsess]

@Johnny-B-Good said:

Your links don't work..
@ntruchsess said:

https://github.com/ntruchsess/fhem-mirror/blob/mysensors_unified

Thanks for giving notice, I've just merged into master and deleted branch 'mysensors_unified', links are fixed now.

[Yveaux]

@ntruchsess said:

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

So not much space left on an ATMega328 with default bootloader.

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

No probs! Go ahead and hopefully it'll lead to something more userfriendly/maintainable than my implementation.

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

I'm not a fan of both Perl MQTT implementations but wanted to make a headstart...
AnyEvent can be very powerful but a bit overkill for just a MQTT gateway and badly documented.

[Yveaux]

@ntruchsess said:

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

So not much space left on an ATMega328 with default bootloader.

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

No probs! Go ahead and hopefully it'll lead to something more userfriendly/maintainable than my implementation.

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

I'm not a fan of both Perl MQTT implementations but wanted to make a headstart...
AnyEvent can be very powerful but a bit overkill for just a MQTT gateway and badly documented.

[novicit]

Forgive what may be a simple question, I am a novice in code but trying
to learn. I am trying to compile the MQTTClientGateway listed in the
first message of this thread.I have downloaded the MQTTClientGateway
code as well as PubSubClient software. I have made absolutely no
changes to any of the code. But when I compile I get the following error:

In file included from /MyMQTTClient.h:17,
from MQTTClientGateway.ino:63:
C:\Arduino\libraries\PubSubClient/PubSubClient.h:68: error: section attribute not allowed for '<anonymous>'

If I can get this to compile clean, I can see if it will work with a
Domoticz setup I already have working on a RPi, with Mosquitto.

I believe the offending line is:
void disconnect();

Since I made no changes, & clearly this compiled by the original
authors, why do I get an error, and how do I fix it?

Any help will be appreciated.

Regards,

PS: below is the full code section

PubSubClient();
PubSubClient(uint8_t , uint16_t, void()(char*,uint8_t*,unsigned int),Client& client);
PubSubClient(uint8_t , uint16_t, void()(char*,uint8_t*,unsigned int),Client& client, Stream&);
PubSubClient(char*, uint16_t, void()(char,uint8_t*,unsigned int),Client& client);
PubSubClient(char*, uint16_t, void()(char,uint8_t*,unsigned int),Client& client, Stream&);
boolean connect(char *);
boolean connect(char *, char *, char *);
boolean connect(char *, char *, uint8_t, uint8_t, char *);
boolean connect(char *, char *, char *, char , uint8_t, uint8_t, char);
void disconnect();
boolean publish(char *, char *);
boolean publish(char *, uint8_t *, unsigned int);

[ntruchsess]

Ok, I can reproduce this with Arduino IDE 1.0.6.
It did compile fine on IDE version 1.0.5 though.

EDIT:
go to PubSubClient.h and remove the 'PROGMEM' in line 72:

change:
boolean publish_P(char *, uint8_t PROGMEM *, unsigned int, boolean);
to:
boolean publish_P(char *, uint8_t *, unsigned int, boolean);

see issue on github and pullrequest with fix:
https://github.com/knolleary/pubsubclient/issues/46
https://github.com/knolleary/pubsubclient/issues/47

[novicit]

@ntruchsess , thank you very much for taking a look, much appreciated.

I deleted PROGMEM as you suggested, resulting in the following:

C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\MQTTclientGateway\MyMQTTClient.cpp: In constructor 'MyMQTTClient::MyMQTTClient(PubSubClient, uint8_t, uint8_t)':
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\MQTTclientGateway\MyMQTTClient.cpp:116: error: no matching function for call to 'PubSubClient::PubSubClient()'
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:59: note: candidates are: PubSubClient::PubSubClient(char*, uint16_t, void ()(char, uint8_t*, unsigned int), Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:58: note: PubSubClient::PubSubClient(uint8_t*, uint16_t, void ()(char, uint8_t*, unsigned int), Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:57: note: PubSubClient::PubSubClient(Client&)
C:\Active Sync Folder\Home Automation\MySensors active sketch folder\libraries\PubSubClient/PubSubClient.h:40: note: PubSubClient::PubSubClient(const PubSubClient&)

Looks to me like the compiler is finding the code ambiguous. For novices like me this is more challenging as the code is fine, just seems to confuse the compiler. I am using Arduino IDE 1.0.6 as you suspected. I am able to compile fine the examples included with pubsubclient, Ethernet, etc. Originally I was using a MAC, so I then tried a Windows pc in case something was there, but same results. Since you are able to compile after removing PROGMEM, I assumed it was part of my setup.

Trial & error is frustrating, but perhaps a way for novices to learn.

I am sure you are busy, so thanks for taking a look!

[Victor Klijmeij]

Hi novicit, Look at https://github.com/knolleary/pubsubclient/pull/47/files
ntruchsess proposes changes to pubsubclient header and cpp files

[binladin]

Hello! Thanks for your great work.
I have some question. After starting MajorDoMo controller and mosquitto on my PC, i made MQTTClientGateway and DHT MySensors node, and in mosquitto log i can see:

1415812670: Received PUBLISH from MySensor (d0, q0, r0, m0, 'MyMQTT/20/1/V_TEMP'
, ... (4 bytes))
1415812670: Sending PUBLISH to MySensor (d0, q0, r0, m0, 'MyMQTT/20/1/V_TEMP', .
.. (4 bytes))
1415812670: Sending PUBLISH to MajorDoMo MQTT Client (d0, q0, r0, m0, 'MyMQTT/20
/1/V_TEMP', ... (4 bytes))

Can you explain why mqttclientgateway subscribing to its own data in next part of sketch:

if (!client.connected())
{
client.connect("MySensor");
client.subscribe("MyMQTT/#");

I apologize if my question will seem ignorant, and sorry for my poor english.

[ntruchsess]

intention is you may controll MySensor-actuators via mqtt-client-gateway by publishing messages to topic MyMQTT/<radioId>/<childId>/<variableType>

[novicit]

SUCCESS! Found the reason I could not compile the MQTT Gtwy client.

When I first looked for the PubSubClient library, used google and went to http://knolleary.net/arduino-client-for-mqtt/ which has a link to GitHub for the library. That is the one I used. When @Victor-Klijmeij suggested I look at @ntruchsess suggested changes again - I noticed the line numbers were not the same as mine, but the code was. Bottom line .... I had an old version of PubSubClient. When I used the version @ntruchsess made changes to, it compiled immediately after his suggested changes. The old PubSubClient .cpp file was ~70 lines of code shorter than the current version.

Thank you both again for your help, much appreciated! @ntruchsess, great job, thanks for sharing the code.

[b0rmann]

PubSubClient library cannot currently be used with hardware based on the ENC28J60

[ntruchsess]

@b0rmann
PubSubClient and ENC28J60 require more memory than an Uno provides. It should run on a Mega256 though.

[b0rmann]

DEBUG:

0;0;3;0;9;read: 16-16-0 s=61,c=1,t=0,pt=7,l=5:38.0
publish: MyMQTT/16/61/V_TEMP 38.0
0;0;3;0;9;send: 0-0-16-16 s=61,c=1,t=0,pt=0,l=5,st=ok:38.00 
0;0;3;0;9;send: 0-0-18-18 s=1,c=1,t=2,pt=0,l=5,st=ok:18.00

mqtt payload for node 18 contain only string '1' (it's relay), '8.00'- trash from previous message
yes, '18.00' may by interprait as boolean TRUE, but '01.00' (next par of log is not FALSE

0;0;3;0;9;read: 16-16-0 s=91,c=1,t=0,pt=7,l=5:61.0
publish: MyMQTT/16/91/V_TEMP 61.0
0;0;3;0;9;send: 0-0-16-16 s=91,c=1,t=0,pt=0,l=5,st=ok:61.00
0;0;3;0;9;send: 0-0-18-18 s=4,c=1,t=2,pt=0,l=5,st=ok:01.00

dirty hack - i use '000000000000' instead '0' to rewrite old data

[DirkB19]

Hi Norbert, Bormann or Novicit,

I'm an Arduino newbie and so far haven't found a way to get the MqttClientGateway to compile.
Obviously I'm using the wrong include files (libraries), so could someone give me some precise links as to where to download the right PubSubClient stuff, or other libraries, if any needed to compile. (I'm not really finding my way in Github).

My setup is Arduino UNO+W5100, Mosquitto on RPi (working and Openhab (working :-). On the Raspberry I already have the mqtt-gpio-monitor working with Mosquitto and Openhab, now I want to add Mysensors;

Thanks in advance,
DirkB

UPDATE
Ok, I found it meanwhile. Everything works now. Was complicated for a newbie, had some issues with libraries not being in the right folders as well.
Thanks.

[SagittaTheArrow]

Has anybody tried to get this compiled with the development branch (1.4.1) with the RFM69 radios? I have been trying to get a successful compile, but no luck thus far. I do have it working with the stable branch, but I am not happy with the radios and would love to get this client working with the RFM69 radios. I have looked over the code and started to merge the two, but I am not a C++ programmer and have had some difficulty with the following error:
yMQTTClient.cpp:16:14: error: variable 'VAR_0' must be const in order to be put into read-only section by means of 'attribute((progmem))'
char VAR_0[] PROGMEM = "TEMP"; //V_TEMP
I tried some of the workarounds that I found (const char VAR_0) but then the GetType functions break and I do not understand enough of the code to correct that issue. Anyway, I thought I would check here to see if anybody else has made some headway.

Thanks in advance for any information / help.

[hek]

what happens if you add const as the compiler suggest?

const char VAR_0[] PROGMEM = "TEMP";

[SagittaTheArrow]

If I change the lines to this:
const char VAR_63[] PROGMEM = "UNKNOWN"; //Custom for MQTTGateway

And leave this line alone:
PROGMEM const char *VAR_Type[] = [<removed to keep short>]

I get this error:
MyMQTTClient.cpp:82:30: error: variable 'VAR_Type' must be const in order to be put into read-only section by means of 'attribute((progmem))'
PROGMEM const char *VAR_Type[] =

I tried const char VAR_Type[] PROGMEM = [ ]; and char const VAR_Type[] PROGMEM = ; but I get the same error. I"m going to do some reading on PROGMEM to see if I can find out what it does.

[SagittaTheArrow]

and then changing the getType function to: char * MyMQTTClient::getType(char *b, const char *const *index)

Now onto trying to merge the the SerialGateway and MQTTClientGateway. Need to see if it would be easier to merge the MQTTClientGateway into the new SerialGateway or the other way around. Time to study some code.

[FotoFieber]

Thanks Norbert for the great work! Works very well with mosquitto.

I added support for gw.requestTime(...); by modifying MyMQTTClient.cpp after the lines handling I_CONFIG)

  else if (msg.type == I_TIME)
  {
    txBlink(1);
    if (!sendRoute(
          build(msg, GATEWAY_ADDRESS, msg.sender, 255, C_INTERNAL,
                I_TIME, 0).set(now())))
      errBlink(1);
  }

Now I have a timebase for low power (deep standby) sensors.

Next I will try to set the time on the gateway with a MQTT Message or NTP....

[mainali]

@ntruchsess said:

<prefix>/<nodeId>/<childId>/<variable_type> is forwarded as a C_SET-msg to the MySensors radio.

Hi, I followed the steps, gateway started but not able to use the existing mysensors sketch. I tried with relay and motion sensor but I get nothing in the serial monitor of MQTT client gateway.

Am I missing something here ?

[Francois]

@ntruschsess

I am getting this compile error when I compile the MQTT Client gateway on lib1.4.1.

In file included from MQTTClientGateway.ino:63:0:
MyMQTTClient.h:43:35: error: 'RF24_PA_LEVEL_GW' was not declared in this scope
void begin(rf24_pa_dbm_e paLevel=RF24_PA_LEVEL_GW, uint8_t channel=RF24_CHANNEL, rf24_datarate_e dataRate=RF24_DATARATE, uint8_t _rx=6, uint8_t _tx=5, uint8_t _er=4 );
^
MQTTClientGateway.ino: In function 'void setup()':
MQTTClientGateway.ino:136:12: error: 'RF24_PA_LEVEL_GW' was not declared in this scope
Error compiling.

[FotoFieber]

In MyMQTTClient.cpp the function
MyMQTTClient::processMQTTMessage(char* topic, byte* payload, unsigned int length)
does ignore length. This can lead to garbage transferred to the clients in the payload

char* ca;
ca = (char *)payload;
ca += length;
*ca = '\0';

before
msg.set((const char*)payload); //Payload

(I am sure, this can be written in a more elegant way. :))

[FotoFieber]

Added support for:

• sketchinfo
• sketchversion
• time
• batterylevel
• dallas RTC and message to set time from outside

https://github.com/FotoFieber/MySensors/tree/mqttclient/libraries/MySensors/examples/MQTTClientGateway

[Gambituk]

@FotoFieber @ntruchsess i have a parallel post regarding this version of the controller, i did not know it was the same as this one when i opened the thread but i am having some garbage issues, do you think this is the same problem as you have described 2 posts up? http://forum.mysensors.org/topic/921/strange-value-being-sent-from-controller-using-mixed-temp-relay-node thanks for any comments or pointers

[FotoFieber]

@Gambituk

This seems to be the problem I fixed with adding \0 at the end of the payload. Try the patch from above.

[TommySharp]

Hi guys! Took me a while but I've finally got this compiling and uploading to the Arduino Uno.

I can see the Uno on my ethernet network and it's visible as the IP address I set in the sketch.

But my mosquitto MQTT broker never receives anything from the Uno, I was expecting even just a New Client Connection notification or something... But nothing.... I have OpenHAB talking to Mosquitto and the Mosquitto screen shows it receiving ping requests from OpenHAB so I know the broker side of things is working well.

Anyone got any ideas? I'm confident that the IP address and port for my broker have been properly configured in the sketch.

I just configured 4 things in the scketch..

1. Broker IP Address
2. Broker Port
3. Arduino IP address
4. Arduino MAC Address

[FotoFieber]

Hi TommySharp

Question 1:
Do you see the connection on mosquitto:

mosquitto_sub -h mosquittoip -v -t '$SYS/broker/clients/active'

Question 2:
Do your mysensor-nodes get connected to the gateway? (You can have only one gateway.)

Tip 1:
Program your arduino with an isp (or arduinoisp): you then have more room as you don't need the bootloader and can enable debug.

[TommySharp]

@FotoFieber
Thanks for the reply....

I'm running mosquitto on windows so not sure how to "run" the command you suggest? I can see the mosquitto DOS looking dialogue box onscreen and it shows OpenHAB pinging mosquitto every few minutes, not seeing any pings or subscriptions coming from the MySensors MQTTClient Gateway though?

This is the only gateway I am trying to setup and I have one wireless temperature/humidity sensor running and it's located about 1 meter from the gateway. I expected that even without any sensor nodes setup that at least the Gateway should connect to mosquitto in some way? I did mess about with trying the other MQTT gateway where it is also the broker and I did get a few temperature readings all the way into OpenHAB that way but OpenHAB randomly lost the connection to the broker and then I'd have to reboot everything. Just mentioning that because that tells me that my one sensor node is working properly I guess.

Tip 1 : I'll have to do some googling about an "isp" to see what that's all about....

[dakky]

will this feature be ported back to the official repo?

[dakky]

another question: i'm using the uno with the ethernet shield. Is the wiring the same as documented here?

I double checked the wiring but keep on receiving the "check wires" message in the serial monitor.

What did i do?

• wiring the nrf
• enabled DEBUG in myConfig.h
• threw the PubSubClient and DS1307RTC libraries in my lib folder
• "fixed" PubSub code as told by @ntruchsess somewhere in this thread
• configured mqtt server and port, arduino ip an mac (btw, commenting out the arduino IP, which should result in using DHCP, does not work, because local_ip is not configured anymore?)

the code itself compiled fine, but i keep on getting this wiring error
what confuses me too: the pins noted in the sketch: are they handled through the ethernetshield?

greetings
dakky

EDIT: Dammit .. i forgot to reenable SOFTSPI. Could this cause the wiring problem? and now the Sketch is too large (with debug enabled)

[dakky]

nobody ever tried tried to upload the sketch with SOFTSPI and debug enabled to an arduino uno? Even with optiboot the sketch seems to be too large

[hek]

Yep, it uses up all available ram. Debug prints have to be disabled. You could probably ad one or two Serial.println() yourself if you need to debug things.

[FotoFieber]

You can save program space by uploading it without a bootloader (ISP).

[GizMoCuz]

Questions (Sorry)

I have rebuild the whole original MQTT sketch so it supports the standard mysensors protocol (the serial/ethernet gateway).
With this, we have all functionality that the original mysensor serial gateway supports.

What would be the downside with this implementation ?

The data being transferred is like

http://www.mysensors.org/download/serial_api_14

It is now also very easy to pass the payload an internal serial/ethernet gateway handler, but the payload is not easily interpreted as human text of course.

Would this be something to submit as a replacement of the original MQTT library?

Devs just need to push the payload to their original serial/ethernet handler, and sending messages is also in the same format

[kunall]

I'm getting following error while compiling, any solution?

[DNKROZ]

Remove PROGMEM from line 72 at the file PubSubClient.h

Works for me, but I really don't know why, I remember to read it somewhere.... not sure if it is "safe" to do, but at least I can compile it and test it, and BTW, works fine with my mosquitto instance on my NAS, still have to do more tests.

Regards.

EDIT: The solution is on this thread XD

[AWI]

@GizMoCuz Can you elaborate a little on your rebuild? ie. what was the objective?

[TommySharp]

Could someone please post the latest working version of the MQTT Client gateway sketch?

[TD22057]

Does anyone have a version of this reworked for the new 1.5 client? Looking through the MySensor class, it looks like a lot of the code in MyMQTTClient.cpp can be removed as it's now in the sensor class. I'd like to use this for my gateway as it's a much better system than the broker gateway that's in the standard library but it needs to support message signing and the newer 1.5 code base. If someone else finished this already I'll just use that, otherwise I'll start working on it. I have family coming in town so I don't expect to get much done in the next week but assuming that the MQTT code is working fine (I have no experience with that writing an MQTT client), it shouldn't take much work to refactor the client to use the new MySensor base.

[tomkxy]

I just did an adaptation. I removed the MyMQTTClient.cpp completely. It looks codewise more than the MQTTGateway. I am in the process of testing it.

You can find it here https://github.com/tomkxy/Arduino.git in the branch MQTTClient.

[Gambituk]

@tomkxy Any updates to this? my setup has been running well for a long time, but i am itchy to see if it will survive with 1.5.. i know i should just leave it alone but i cannot help but tinker with it.

[tomkxy]

Well it seems to work. Only point I saw and don't understand is that the gateway tries to send a message back to the sensor which does not seem to be an ACK. See my post here http://forum.mysensors.org/topic/1782/gateway-is-sending-command-1-type-1-c-1-t-1-message-to-sensor but so far no feedback on that. Functional wise it works, at least with my test temp and humidity sensor.

[ruslan]

Please help! This MQTT client gateway fails to compile. I use 1.6.5. IDE, 1.5 library. Applied patches as suggested by ntruchsess and still get this error:

MyMQTTClient.cpp: In constructor 'MyMQTTClient::MyMQTTClient(PubSubClient, uint8_t, uint8_t)':
MyMQTTClient.cpp:108: error: no matching function for call to 'MySensor::MySensor(uint8_t&, uint8_t&)'
     MySensor(_cepin, _cspin), buffsize(0)
                                         ^
MyMQTTClient.cpp:108:41: note: candidates are:
In file included from MyMQTTClient.h:16:0,
                 from MyMQTTClient.cpp:14:
R:\Arduino\libraries\MySensors/MySensor.h:153:2: note: MySensor::MySensor(MyTransport&, MyHw&)
  MySensor(MyTransport &radio =*new MyTransportNRF24(), MyHw &hw=*new MyHwDriver()
  ^
R:\Arduino\libraries\MySensors/MySensor.h:153:2: note:   no known conversion for argument 1 from 'uint8_t {aka unsigned char}' to 'MyTransport&'
R:\Arduino\libraries\MySensors/MySensor.h:144:7: note: MySensor::MySensor(const MySensor&)
 class MySensor
       ^
R:\Arduino\libraries\MySensors/MySensor.h:144:7: note:   candidate expects 1 argument, 2 provided
MyMQTTClient.cpp: In member function 'void MyMQTTClient::begin(rf24_pa_dbm_e, uint8_t, rf24_datarate_e, uint8_t, uint8_t, uint8_t)':
MyMQTTClient.cpp:148: error: 'setupRepeaterMode' was not declared in this scope
   setupRepeaterMode();
                     ^
MyMQTTClient.cpp:154: error: 'setupRadio' was not declared in this scope
   setupRadio(paLevel, channel, dataRate);
                                        ^
MyMQTTClient.cpp:155: error: 'BASE_RADIO_ID' was not declared in this scope
   RF24::openReadingPipe(WRITE_PIPE, BASE_RADIO_ID);
                                     ^
MyMQTTClient.cpp:157: error: cannot call member function 'void RF24::startListening()' without object
   RF24::startListening();
                        ^
no matching function for call to 'MySensor::MySensor(uint8_t&, uint8_t&)'

[tomkxy]

@tomkxy said:

here https://github.com/tomkxy/Arduino.git in the branch MQTTClient.

Did you have a look at my repository above? Functional wise it is the same adapted for version 1.5. I have it in use for quite a while now.

[ruslan]

to tomkhy: your gateway sketch compiles, but Arduino runs out of memory (I use Nano with W5100 ethernet module, SOFTSPI enabled). It does fit with MY_SIGNING_FEATURE and DEBUG disabled, but I need debug - somehow it is not registering with MQTT server. Please tell me how to reduce memory footprint.

[tomkxy]

I am using this sketch with a Mega exactly because of that problem. If you post your MQTTClientGateway sketch I can have a look. If you use Mosquito as MQTT broker you may also post the Mosquito config file.

[ruslan]

Many thanks to @tomkhy!
I built my MQTT client with his code. No modifications were needed.

BTW, Will this or any other code get included into the official MySensors library?

[tomkxy]

It is now in the Development branch.

[ahhk]

Hi,

can this run on ESP8266 (as wifi-mqttclient-gateway), too?
Greetings

Andreas

[tomkxy]

I don't know, you have to try it. What I can say that with signing support enabled it doesn't fit into the memory of a ProMini. I run it in a Mega.

[gregl]

@ruslan said:

to tomkhy: your gateway sketch compiles, but Arduino runs out of memory (I use Nano with W5100 ethernet module, SOFTSPI enabled). It does fit with MY_SIGNING_FEATURE and DEBUG disabled, but I need debug - somehow it is not registering with MQTT server. Please tell me how to reduce memory footprint.

Looks like you have it working now, but just incase, Previously when I have had memory issues , I have upgraded my bootloader to use Optiboot which gives you a little bit more.

HTH

[mlava]

@ahhk
Hello all, first post here and a bit new to the mysensors field.
I also want to know the answer as to whether this will run on the ESP8266 as hope to run a wifi gateway.
Thanks in advance!

[chris021]

Also keen to run this on an ESP8266 would be a great for bridging.

[John Connolly]

OK. I've got the gateway to compile and get the following:

Started!
0;0;3;0;9;read: 110-110-0 s=1,c=1,t=0,pt=7,l=5:13.2
0;0;3;0;9;read: 105-105-0 s=0,c=1,t=1,pt=7,l=5:70.0
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.3
0;0;3;0;9;read: 105-105-0 s=0,c=1,t=1,pt=7,l=5:69.9
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.2
0;0;3;0;9;read: 105-105-0 s=0,c=1,t=1,pt=7,l=5:70.0
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.3
0;0;3;0;9;read: 105-105-0 s=0,c=1,t=1,pt=7,l=5:69.9
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.2
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.3
0;0;3;0;9;read: 105-105-0 s=0,c=1,t=1,pt=7,l=5:69.8
0;0;3;0;9;read: 110-110-0 s=1,c=1,t=0,pt=7,l=5:13.3
0;0;3;0;9;read: 110-110-0 s=0,c=1,t=1,pt=7,l=5:58.5

I can ping the MQTTClinetGateway from another machine and I've confirmed Mosquitto is running OK with a series of pub/subs from other machines/devices and that side of things is working perfectly fine.

I'm not seeing any of the above gateway readings publish to my broker. Thoughts?

Cheers.

[tomkxy]

Following points to verify and check:

• is the gateway connected to the broker. As far as I remember you can check that by subscribiing to Mosquitto status messages

mosquitto_sub -v -t \$SYS/#

If not check what the authentication method is mosquitto is. If it allows anonymous access make sure that "MQTT_AUTH_REQUIRED" is not defined in the MQTTClientGateway sketch. If username and password are set correctly.

It seems that you are not using signing. So ensure that your MQTTClientGateway is compiled with the signing feature disabled. That are the two points I can imagine now provided that you are sure that you have of course a working ethernet connection.

[John Connolly]

I downloaded and installed the development branch and got the MQTTClientGateway running on an UNO with a W5100 shield (that I'm suspicious of, in terms of its reliability). The sketch runs great for a while publishing data to Mosquitto on my Mac Mini, but then stops responding after a while. I'm pretty sure the cause of the problem is the ethernet shield as it bombs out when running other sketches - the webserver one for example.

I have a few of these ENC28J60 boards and I'd like to use one of those instead, but have had some problems.

http://www.aliexpress.com/item/ENC28J60-LAN-Ethernet-Network-Board-Module-for-arduino-25MHZ-Crystal-AVR-51-LPC-STM32-3-3V/1956963721.html

I notice in the MQTTClientGateway sketch that I should just need to comment out the one W5100 Ethernet.h library and substitute with the UIPEthernet.h one, but I tried that and had no joy. Does anyone have a working sketch based on the ENC28J60 that they could post, along with the wiring diagram to connect that board in along with any additional tweaks necessary to get it running on an Uno? Thanks in advance. Cheers.

[siod]

Hi Guys,

I get this:

Started!
0;0;3;0;9;read: 0-0-0 s=0,c=0,t=0,pt=0,l=0:
0;0;3;0;9;version mismatch
0;0;3;0;9;read: 0-0-0 s=0,c=0,t=0,pt=0,l=0:
0;0;3;0;9;version mismatch

Also the Gateway does not answer to pings. Any suggestions?

edit: after a rebbot I get valid readings in the Serial monitor of the Arduino IDE now:

0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:0
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:0
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:0
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:0
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:0
0;0;3;0;9;read: 20-20-0 s=3,c=1,t=16,pt=2,l=2:1

But still, can´t ping the gw. Double checked IP and Port, seems to be fine.

[John Connolly]

I've ordered a new W5100 module - this time the red mini W5100 board missing the SD Card slot to see if I have better luck. Still keen to see code for an ENC based MQTTClientGateway though, as this will take a while to arrive...

[ruslan]

To developers of the MQTT client:
I just tried the GatewayW5100MQTTClient in the latest development branch - it gives me "0;0;3;0;9;Radio init fail" error, while GatewayW5100 sketch works fine. What could be the reason? I noticed that the declarations parts in these sketches are rather different.
My setup: ARduino nano clone + W5100 module + NRF24L01 radio. All wired as in the MySensors ethernet gateway guide ([http://www.mysensors.org/build/ethernet_gateway])

[hek]

For the W5100 SOFTSPI has to be enabled. I see that this is missing from the sketch..

Try adding (coming from the normal W5100 sketch)

// W5100 Ethernet module SPI enable (optional if using a shield/module that manages SPI_EN signal)
//#define MY_W5100_SPI_EN 4  

// Enable Soft SPI for NRF radio (note different radio wiring is required)
// The W5100 ethernet module seems to have a hard time co-operate with 
// radio on the same spi bus.
#if !defined(MY_W5100_SPI_EN)
  #define MY_SOFTSPI
  #define MY_SOFT_SPI_SCK_PIN 14
  #define MY_SOFT_SPI_MISO_PIN 16
  #define MY_SOFT_SPI_MOSI_PIN 15
#endif  

I'll update the repository.

[ruslan]

Hi, Hek!
You are correct. except I also had to add these 2 lines and then radio worked:

#define MY_RF24_CE_PIN 5
#define MY_RF24_CS_PIN 6

Pls fix the repo. Looking forward to seeing mqtt client in the master branch.

[hek]

Pushed to build server. Soon merged.
https://github.com/mysensors/Arduino/pull/241

[ruslan]

What is the MQTT message to switch MQTT client gateway into inclusion mode?

[hek]

Something like this should work...

Topic: mygateway/0/255/3/0/6
Payload: 1

http://www.mysensors.org/download/serial_api_15

[dakky]

Is this MQTTClientGateway in the dev branch only? I didn't see something like this in the 1.5 release/master.
I would prefer to use my gateway as client, but I am not skilled enough to use the dev branch and work around resulting problems

greetings
Dakky

edit:
@tomkxy said:

@tomkxy said:

here https://github.com/tomkxy/Arduino.git in the branch MQTTClient.

Did you have a look at my repository above? Functional wise it is the same adapted for version 1.5. I have it in use for quite a while now.

hmm is this repo still functional? with the signing_feature enabled i'm getting compile errors:

mqttClientGateway:175: error: 'signer' was not declared in this scope
             , signer
               ^

[Samuel235]

@dakky It is currently only in the Dev branch, yes.

[maxtox]

I Think HERE is the answer of our problem!!!!

http://electronics.stackexchange.com/questions/67212/how-to-avoid-sd-card-and-w1500-spi-mixup-on-the-ethernet-shield

[cowen71]

Good afternoon gents!

So I have been playing with openHab and an arduino as a mqtt gateway and this has been successful. The only thing I don't like is that the arduino does not seem to like multiple connections to the MQTT broker.

My next step then is to use an MQTT broker (mosquitto) on the pi and then subscribe to this from the arduino. If I understand correctly, this is exactly what this thread is discussing.

So - I have downloaded the codes and without changing anything, tried to compile. This fails with numerous errors. I have te latest libraries installed and to be honest, I am a little lost as to what is wrong. Before I post lots of error messages, are there any known 'tricks' to get the latest version working?

Hopefully someone can help - Thanks guys

[Samuel235]

@cowen71 - Could you please give us a little more information to help troubleshoot your issues.

What ethernet module are you using? W5100? If so you need to makesure that you have enabled soft SPI for NRF radio as it doesn't like it being on the same bus. We also have to move the CE/CSN pins for the radio too. All this is explained inside of the sketch itself.

[cowen71]

Hi Samuel,

OK, I was hoping there was a known 'trick' to get things working, but maybe I was just dreaming. Of course, if not - much more information is needed!

At the moment I am testing with a Mega2560 with an arduino ethernet shield attached. This has (and still does) work well with the mqtt gateway downloaded from mysensors after a few minor pin assignment changes.

The problem I am having is that I cannot compile the mqttClientGateway code. I get errors. I have to be honest that whilst I had experience 20 years ago with PIC's and other uC's, I am incredibly rusty with C and completely new to Arduinos. So it is a steep learning curve currently. In this case, I would not be surprised if I have made a real schoolboy error!

So what did I do?....
At the moment I installed the PubSubLient library into the Arduino libraries file.
I copied the code from here: https://github.com/ntruchsess/MySensors/tree/mqttclient/libraries/MySensors/examples/MQTTClientGateway
creating the three files in my mqttClientGateway development folder.

I changed nothing (accept target and com port and hit the compile button! This fails with the following error initially:

sketch\MyMQTTClient.cpp:14:30: fatal error: utility/MsTimer2.h: No such file or directory
#include "utility/MsTimer2.h"

I modified the include text to: #include "MsTimer2.h"
and tried to compile again again. The result is:

sketch\MyMQTTClient.cpp: In constructor 'MyMQTTClient::MyMQTTClient(PubSubClient, uint8_t, uint8_t)':

MyMQTTClient.cpp:106: error: no matching function for call to 'MySensor::MySensor(uint8_t&, uint8_t&)'

 MySensor(_cepin, _cspin), buffsize(0)

                                     ^

sketch\MyMQTTClient.cpp:106:41: note: candidates are:

In file included from sketch\MyMQTTClient.h:15:0,

             from sketch\MyMQTTClient.cpp:13:

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:158:2: note: MySensor::MySensor(MyTransport&, MyHw&)

MySensor(MyTransport &radio =*new MyTransportNRF24(), MyHw &hw=*new MyHwDriver()

^

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:158:2: note: no known conversion for argument 1 from 'uint8_t {aka unsigned char}' to 'MyTransport&'

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:149:7: note: constexpr MySensor::MySensor(const MySensor&)

class MySensor

   ^

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:149:7: note: candidate expects 1 argument, 2 provided

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:149:7: note: constexpr MySensor::MySensor(MySensor&&)

C:\Program Files (x86)\Arduino\libraries\MySensors/MySensor.h:149:7: note: candidate expects 1 argument, 2 provided

sketch\MyMQTTClient.cpp: In member function 'void MyMQTTClient::begin(rf24_pa_dbm_e, uint8_t, rf24_datarate_e, uint8_t, uint8_t, uint8_t)':

MyMQTTClient.cpp:146: error: 'setupRepeaterMode' was not declared in this scope

setupRepeaterMode();

                 ^

MyMQTTClient.cpp:152: error: 'setupRadio' was not declared in this scope

setupRadio(paLevel, channel, dataRate);

                                    ^

MyMQTTClient.cpp:153: error: 'BASE_RADIO_ID' was not declared in this scope

RF24::openReadingPipe(WRITE_PIPE, BASE_RADIO_ID);

                                 ^

MyMQTTClient.cpp:155: error: cannot call member function 'void RF24::startListening()' without object

RF24::startListening();

My suspicion is I have a compatibility problem between library versions and/or the sketch I have downloaded. The radio stuff I can probably solve (hopefully) its the issues before that I am struggling with.

Any advice would be appreciated.

Cheers

[Samuel235]

@cowen71, from a glancing look over that post, i would expect the issue to lie within your libraries. Whether it be a version merging issue or just a general error in placements.

I will put money on that you have your Arduino folder setup incorrectly when it comes to the libraries. I'll be honest and admit that i'm not too sure why you're getting your client sketch from ntruchsess rather than directly from the MySensors github/download page.

Lets just run through it from the start on where to get your libraries placed, in theory they could be located anywhere as long as the files inside of the mysensors folder are in the correct structure. You firstly need to get the development version, I prefer to just go for the direct download off of the download Page. Once you have that downloaded you should have a .zip folder called 'Arduino-development.zip'. Once you have extracted that you need to place all of its contents (3 folders and 5 files) into your Documents\Arduino folder.

I would do all of the above with a clean folder structure for the arduino folder, save any library confusions for the software. Then we need to go back into your gateway sketch and change your pin configuration like you need to.

Come back with your findings

[cowen71]

Cheers Samuel,
I will let you know. I am happy (actually perhaps not happy!) to admit the failures are purely down to me trying to get to grips with everything here.

I also have just noticed that ruslan reported the same error somewhere above and so I will also look at the solutions that worked for him.

I let you know what happens

[cowen71]

OK so as you suggested - everything redownloaded and a new folder structure created. And actually, I get exactly the same problem. Incidentally, my other sketches compile no problem. Some written by me, some downloaded (e.g. Serial_Gateway, mqtt_Gateway). The only one I have a problem with is the mqttClientGateway.

I see above that ruslan had exactly the same issue 7 months ago and then got it working. The answer that helped him was....

@tomkxy said:
here https://github.com/tomkxy/Arduino.git in the branch MQTTClient.
Did you have a look at my repository above? Functional wise it is the same adapted for version 1.5. I have it in use for quite a while now.

but this link seems dead now.

I better stop for the evening before my wife files for divorce! I get back to this tomorrow!
Thanks for the support so far.

[Samuel235]

@cowen71 Could you post your sketch for either me or a software contributor to see what is going on with it please. If you're certain that the error codes are the same as before then we can use the previous post as reference, if they're different in any way, please post the error codes again with the sketch

[tomkxy]

The link mentioned by @cowen71 was put into a pull request which went into the development branch. However, the MQTTClientGateway has been reworked significantly in the development branch. The message structure is now similar to the serial gateway. I just completed a "migration" from that old version to the latest development branch version which worked perfectly, apart from RF issues I had.

Thus, I suggest to use the latest version of the development branch.

[Samuel235]

@tomkxy, if he followed my instructions like he said he has, he should all being well be running the latest version of the development branch. I may be incorrect but i still feel there may be some sort of folder structure error going on here.

[cowen71]

OK - I couldn't sleep and so I am back for a short time.

So first I have to confess, I did not read the instructions from Samuel correctly and downloaded the Stable libraries rather than the development versions. I have corrected this. With the development libraries several of my other sketches will no longer compile - but lets not worry about that now!!

Next - I could not find within the mqttClientGateway sketch within the downloaded development folder. In the case I am just having a blonde moment, could you tell me exactly where this is located please.

Final question regarding the folder structures. I have the Arduino installation in the program files where it installs automatically, and then my Sketchbook is set up as follows..

\Arduino\Hardware
\Arduino\Documentation
\Arduino\Sketch\libraries
\Arduino\Sketch\mqttGateway
\Arduino\Sketch\pondController (etc... - folders for other sketches)

[Samuel235]

@cowen71, so lets get to know where your sketch book is located. I may have wrongly assumed this in my last post. Could you go to your arduino IDE > file > Preferences. On that panel you should see the location of your sketchbook, the default will be in your Documents folder, within another folder called Arduino. This is now where all of the sketches should be located. So if we go there you should have a folder structure like I mentioned in my last post. At least for this instance, could you get the latest development branch of MySensors Libraries. If i remember correctly, the software guys here have redone the MQTT client gateway sketch in the development branch to now work properly.

This is why i was asking what ethernet module you was using, because they have two mqtt gateway sketches now, one for W5100 (the one i use) and then one for ESP8266. These will be named appropriately for you to see. They also have a serial version in there too, which i have never used. Use the appropriate one for you, all instructions for pin changes SHOULD be inside, i may be wrong though.

In regards to your folder structure again, you're using the program files directory. This is incorrect as this is simply where arduino installs itself. You do not want to be putting your sketches for arduino IDE in there (if the location for your sketches inside of the preference menu for arduino IDE states a different location). Where ever your arduino IDE states is the sketchbook, place your sketches/libraries in there.

[cowen71]

Hi Samuel,

Firstly the folder structure - my previous explanation was poor. Let me try again. Under Program Files I have the normal installation that is done automatically. So far so good I think.
Under a separate dedicated folder (c:\cloudstation\arduino) I have the additional directories I mentioned above. So..
\Hardware
\Documentation
\Sketch\libraries
\Sketch\mqttGateway (for example)
\Sketch\pondController (etc... - folders for other sketches)
\Sketch\mqttClientGateway (I hope to have this!!)
I had already correctly set the sketchbook location in the preferences. I "think" this is correct as installing new libraries seems to install them to the correct location.

And I did find of course the examples folder and the 'GatewayW5100MQTTClient'. But I thought this was actually a sketch providing the framework rather than a complete functional mqttClientGateway. It certainly seems a little sparse!

[tomkxy]

@cowen71 The whole library has been completely refactored. Yes, the sketch looks sparse. You get all the necessary functionality by the define

#define MY_GATEWAY_MQTT_CLIENT

which is as far as I remember already in the sketch.

[Samuel235]

And from what i remember, this then enables it to gather the code required from the files from the #include function. This is the beauty of mysensors layout, enables us to use very clean and easy to read sketches for the newbies to completely understand. This is half of the error your getting when it referes to that .h file that it cant find. That is some of the code needed to run this gateway sketch.

This is one of the reasons i love MySensors so much.

[cowen71]

Guys, firstly, Thank you. You have the patience of saints!
Believe it or not, I am actually a Software developer/Electronics engineer by trade but PC based. The Arduino platform is a new experience for me and my previous uC experience is 20years old!

I now have a working mqttClientGateway. I need to play some more - but first indications are that it is working well.

My problems were a combination of bad folder structure and also bad combinations of sketches and libraries (desperately trying to play down the lack of experience factor here!!). Once I got this sorted, it started working.

A huge 'High 5' to you guys. I owe you a bier or maybe 2!

Not a question for this thread I know, but before I leave you in peace - what are you using for your Home Automation controllers? I have been playing with openHab and this seems to work well, but I have not really investigated the others.

[Samuel235]

@cowen71, you seem to be an awful lot like me I'm using that exact sketch for the gateway with a W5100, modified in an enclosure with incating LEDs on to show the communication. I too use OpenHAB and love it. However i'm yet to experiment with others. Drop me a message, i feel we could have some decent conversations!

Glad you have gotten your gateway working, pleased it was such a simple solution!

[tomkxy]

I am using Openhab. I also did not investigate into a lot of controllers. What I like about Openhab is the powerful rules engine and the tons of bindings it support.
If you are able to program possibilities seem to endless.

[Samuel235]

@tomkxy, it seems that openhab users all love it for the exact same reasons. They sure have a strong client base. Those that like it, love it. I've never tried any of the others and i don't have any inclination to. The only other controller i really like the look of is MYScontroller being created by the software smartass' here ;). When it becomes more feature rich I'll be trialing it for my home

[rollercontainer]

I'am trying to add a local DHT22 sensor but as soon as anything is in the loop(), the gateway stops to work.

even if I delete the sleep() rows. Hasn't it to be wait() ?

[rollercontainer]

Got it to work. Removed serial.print lines and changed sleep() to wait().