SSL support for MQTT on ESP8266

noelgeorgi

@hek @Anticimex First install https://github.com/esp8266/arduino-esp8266fs-plugin and upload the crt file to the ESP8266 file system.

then issue these command from the folder in which MySensors is installed:
sed -i 's/WiFiClient/WiFiClientSecure/g' Arduino/libraries/MySensors/core/MyGatewayTransportEthernet.cpp

sed -i 's/WiFiClient/WiFiClientSecure/g' Arduino/libraries/MySensors/core/MyGatewayTransportMQTTClient.cpp

change your GatewayESP8266MQTTClient to this:

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik Ekblad
 * 
 * DESCRIPTION
 * The ESP8266 MQTT gateway sends radio network (or locally attached sensors) data to your MQTT broker.
 * The node also listens to MY_MQTT_TOPIC_PREFIX and sends out those messages to the radio network
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error  
 * 
 * See http://www.mysensors.org/build/esp8266_gateway for wiring instructions.
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4          
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *            
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 * 
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 * 
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */
#include "FS.h"
#include <EEPROM.h>
#include <SPI.h>

// Enable debug prints to serial monitor
#define MY_DEBUG 

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires usenrame/password
#define MY_MQTT_USER "admin" //your MQTT username
#define MY_MQTT_PASSWORD "admin" //your MQTT password

// Set WIFI SSID and password
#define MY_ESP8266_SSID "yourwifissid"
#define MY_ESP8266_PASSWORD "yourwifipassword"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 172,16,0,120

// If using static ip you need to define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 172,16,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// MQTT broker ip address.  
#define MY_CONTROLLER_IP_ADDRESS 172, 16, 0, 95

// The MQTT broker port to to open 
#define MY_PORT 8883      // note this change its port 8883 as per standard for SSL over MQTT

 /*
// Flash leds on rx/tx/err
#define MY_LEDS_BLINKING_FEATURE
// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Enable inclusion mode
#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
#define MY_INCLUSION_MODE_DURATION 60 
// Digital pin used for inclusion mode button
#define MY_INCLUSION_MODE_BUTTON_PIN  3 

#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED
*/

#include <ESP8266WiFi.h>
#include <MySensor.h>

void setup() { 
  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

  File ca = SPIFFS.open("/ca.crt", "r"); //replace ca.crt with the ca file name
  if (!ca) {
    Serial.println("Failed to open ca file");
  }
  else
  Serial.println("Successfully opened ca file");

if(_ethClient.loadCertificate(ca)) //_ethClient as defined in MyGatewayTransportMQTTClient.cpp
  Serial.println("loaded");
  else
  Serial.println("not loaded");
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  // Send locally attech sensors data here
}

NB: i used https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh for generating the ca files

Anticimex

I have no esp hw. And like I said, ssl and MQTT is specific to a certain architecture (from a gw and controller perspective) so I don't want it being mixed in with the existing security infrastructure that goes on every node.
You are free to implement it as such of course, but I won't bring it into the signing solution.

hek

@Anticimex
This has nothing to do with the message signing.

@noelgeorgi
Good!. Might be good to put this initialization in the optional before()-method instead of setup(), which runs before any other MySensors stuff.

noelgeorgi

@hek I didnt know there was an optional method. Thanks

Anticimex

@hek i know that perfectly well, but since I am adressed, I just want to make clear I do not consider myself involved :)

noelgeorgi

@hek pubsubclient has issues with ssl connection, frequent disconnections. Adafruit's MQTT library works like charm, trying to integrate into MySensors

noelgeorgi

@hek I somewhat succesfully integrated Adafruit MQTT library into MySensors, i am able to sent out data but not able to receive (subscribe). These are the changes i made:

diff output

MySesnors.h

< #if defined(MY_GATEWAY_SERIAL) || defined(MY_GATEWAY_W5100) || defined(MY_GATEWAY_ENC28J60) || defined(MY_GATEWAY_ESP8266) || defined(MY_GATEWAY_MQTT_CLIENT) || defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
---
> #if defined(MY_GATEWAY_SERIAL) || defined(MY_GATEWAY_W5100) || defined(MY_GATEWAY_ENC28J60) || defined(MY_GATEWAY_ESP8266) || defined(MY_GATEWAY_MQTT_CLIENT)
55,59d54
< #if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 	#include "drivers/Adafruit_MQTT_Library/Adafruit_MQTT.cpp"
< 	#include "drivers/Adafruit_MQTT_Library/Adafruit_MQTT_Client.cpp"
< #endif
< 
180c175
< #if defined(MY_GATEWAY_MQTT_CLIENT) || defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
---
> #if defined(MY_GATEWAY_MQTT_CLIENT)

keywords.txt

< MY_GATEWAY_MQTT_SECURE_CLIENT LITERAL 1

MyGatewayTransportMQTTClient.cpp

36,40c36
< 	#if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 		#define EthernetClient WiFiClientSecure
< 	#else 
< 		#define EthernetClient WiFiClient
< 	#endif
---
> 	#define EthernetClient WiFiClientSecure
54,277d49
< #if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 	Adafruit_MQTT_Client mqtt(&_ethClient
< 				 #if defined(MY_CONTROLLER_IP_ADDRESS)
<  				, _brokerIp 
< 				#else
< 				, MY_CONTROLLER_URL_ADDRESS
< 				#endif				
< 				, MY_PORT 
< 				#if defined(MY_MQTT_USER) && defined(MY_MQTT_PASSWORD) 
< 				, MY_MQTT_USER, MY_MQTT_PASSWORD
< 				#endif
< 				);
< 
< Adafruit_MQTT_Subscribe feed = Adafruit_MQTT_Subscribe(&mqtt,MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "/+/+/+/+/+" );
< Adafruit_MQTT_Subscribe *subscription;
< bool _connecting = true;
< bool _available = false;
< char _convBuffer[MAX_PAYLOAD*2+1];
< char _fmtBuffer[MY_GATEWAY_MAX_SEND_LENGTH];
< MyMessage _mqttMsg;
< 
< bool gatewayTransportSend(MyMessage &message) {
< 	if (!mqtt.connected())
< 		return false;
< 	snprintf_P(_fmtBuffer, MY_GATEWAY_MAX_SEND_LENGTH, PSTR(MY_MQTT_PUBLISH_TOPIC_PREFIX "/%d/%d/%d/%d/%d"), message.sender, message.sensor, mGetCommand(message), mGetAck(message), message.type);
< 	debug(PSTR("Sending message on topic: %s\n"), _fmtBuffer);
< 	return mqtt.publish(_fmtBuffer, message.getString(_convBuffer));
< }
< 
< void incomingMQTT(char* topic, byte* payload,
<                         unsigned int length)
< {
< 	debug(PSTR("Message arrived on topic: %s\n"), topic);
< 	char *str, *p;
< 	uint8_t i = 0;
< 	uint8_t bvalue[MAX_PAYLOAD];
< 	uint8_t blen = 0;
< 	uint8_t command = 0;
< 	for (str = strtok_r(topic, "/", &p); str && i <= 5;
< 			str = strtok_r(NULL, "/", &p)) {
< 		switch (i) {
< 			case 0: {
< 				// Topic prefix
< 				if (strcmp(str, MY_MQTT_SUBSCRIBE_TOPIC_PREFIX) != 0) {
< 					// Message not for us or malformed!
< 					return;
< 				}
< 				break;
< 			}
< 			case 1: {
< 				// Node id
< 				_mqttMsg.destination = atoi(str);
< 				break;
< 			}
< 			case 2: {
< 				// Sensor id
< 				_mqttMsg.sensor = atoi(str);
< 				break;
< 			}
< 			case 3: {
< 				// Command type
< 				command = atoi(str);
< 				mSetCommand(_mqttMsg, command);
< 				break;
< 			}
< 			case 4: {
< 				// Ack flag
< 				mSetRequestAck(_mqttMsg, atoi(str)?1:0);
< 				break;
< 			}
< 			case 5: {
< 				// Sub type
< 				_mqttMsg.type = atoi(str);
< 				// Add payload
< 				if (command == C_STREAM) {
< 					blen = 0;
< 					uint8_t val;
< 					while (*payload) {
< 						val = protocolH2i(*payload++) << 4;
< 						val += protocolH2i(*payload++);
< 						bvalue[blen] = val;
< 						blen++;
< 					}
< 					_mqttMsg.set(bvalue, blen);
< 				} else {
< 					char* ca;
< 					ca = (char *) payload;
< 					ca += length;
< 					*ca = '\0';
< 					_mqttMsg.set((const char*) payload);
< 				}
< 				_available = true;
< 			}
< 		}
< 		i++;
< 	}
< }
< 
< bool reconnectMQTT() {
< 	debug(PSTR("Attempting MQTT connection...\n"));
< 	// Attempt to connect
< 		
< int8_t ret;
< 
<   // Stop if already connected.
<   if (mqtt.connected()) {
<     return true;
<   }
< 
<   //Serial.print("Connecting to MQTT... ");
< 
<   uint8_t retries = 3;
<   while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
<        Serial.println(mqtt.connectErrorString(ret));
<        Serial.println("Retrying MQTT connection in 5 seconds...");
<        mqtt.disconnect();
<        delay(5000);  // wait 5 seconds
<        retries--;
<        if (retries == 0) {
<          // basically die and wait for WDT to reset me
<          while (1);
<        }
<   }
<   //Serial.println("MQTT Connected!");
<  
< 
< 		 {
< 		debug(PSTR("MQTT connected\n"));
< 		// Once connected, publish an announcement...
< 		//_client.publish("outTopic","hello world");
< 		// ... and resubscribe
< 		mqtt.subscribe(&feed);
< 		return true;
< 	}
< 	return false;
< }
< 
< bool gatewayTransportInit() {
< 	_connecting = true;
< 	//#if defined(MY_CONTROLLER_IP_ADDRESS)
< 	//	_client.setServer(_brokerIp, MY_PORT);
< 	//#else
< 	//	_client.setServer(MY_CONTROLLER_URL_ADDRESS, MY_PORT);
< 	//#endif
< 
< 	//_client.setCallback(incomingMQTT);
< 
<  if((subscription = mqtt.readSubscription(10))) {
<     if (subscription == &feed) 
<     {
<       Serial.print(F("Got: "));
<       Serial.println((char *)feed.lastread);
<     }}
< 
< 
<   	#if defined(MY_GATEWAY_ESP8266)
< 		// Turn off access point
< 		WiFi.mode (WIFI_STA);
< 		(void)WiFi.begin(MY_ESP8266_SSID, MY_ESP8266_PASSWORD);
< 		#ifdef MY_IP_ADDRESS
< 			WiFi.config(_clientIp, _gatewayIp, _subnetIp);
< 		#endif
< 		while (WiFi.status() != WL_CONNECTED)
< 		{
< 			delay(500);
< 			MY_SERIALDEVICE.print(".");
< 			yield();
< 		}
< 		MY_SERIALDEVICE.print("IP: ");
< 		MY_SERIALDEVICE.println(WiFi.localIP());
< 	#else
< 		#ifdef MY_IP_ADDRESS
< 			Ethernet.begin(_clientMAC, _clientIp);
< 		#else
< 			// Get IP address from DHCP
< 			if (!Ethernet.begin(_clientMAC))
< 			{
< 				MY_SERIALDEVICE.print("DHCP FAILURE...");
< 				_connecting = false;
< 				return false;
< 			}
< 			MY_SERIALDEVICE.print("IP: ");
< 			MY_SERIALDEVICE.println(Ethernet.localIP());
< 		#endif
< 
< 		// give the Ethernet interface a second to initialize
< 		// TODO: use HW delay
< 		wait(1000);
< 	#endif
< 	_connecting = false;
< 	return true;
< }
< 
< bool gatewayTransportAvailable() {
< 	if (_connecting)
< 		return false;
< 
< 	//keep lease on dhcp address
< 	//Ethernet.maintain();
< 	if (!_ethClient.connected()) {
< 		//reinitialise client
< 		if (gatewayTransportInit())
< 			reconnectMQTT();
< 		return false;
< 	}
< 	//_ethClient.loop();
< 
< 
< 
< 	return _available;
< }
< 
< MyMessage & gatewayTransportReceive() {
< 	// Return the last parsed message
< 	_available = false;
< 	return _mqttMsg;
< }
< 
< 
< 
< 
< 
< 
< #else
279d50
< 
456c227
< #endif
---
>

GatewWay Code

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik Ekblad
 * 
 * DESCRIPTION
 * The ESP8266 MQTT gateway sends radio network (or locally attached sensors) data to your MQTT broker.
 * The node also listens to MY_MQTT_TOPIC_PREFIX and sends out those messages to the radio network
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error  
 * 
 * See http://www.mysensors.org/build/esp8266_gateway for wiring instructions.
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4          
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *            
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 * 
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 * 
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */

#include <EEPROM.h>
#include <SPI.h>

// Enable debug prints to serial monitor
#define MY_DEBUG 

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
//#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_SECURE_CLIENT
#define MY_GATEWAY_ESP8266

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires usenrame/password
#define MY_MQTT_USER "user"
#define MY_MQTT_PASSWORD "pass"

// Set WIFI SSID and password
#define MY_ESP8266_SSID "ssid"
#define MY_ESP8266_PASSWORD "password"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 172,16,0,120

// If using static ip you need to define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 172,16,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// MQTT broker ip address.  
#define MY_CONTROLLER_URL_ADDRESS "domain.com"

// The MQTT broker port to to open 
#define MY_PORT 8883      

 /*
// Flash leds on rx/tx/err
#define MY_LEDS_BLINKING_FEATURE
// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Enable inclusion mode
#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
#define MY_INCLUSION_MODE_DURATION 60 
// Digital pin used for inclusion mode button
#define MY_INCLUSION_MODE_BUTTON_PIN  3 

#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED
*/

#include <ESP8266WiFi.h>
#include <MySensor.h>

void setup() { 
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  // Send locally attech sensors data here
}

Any help would be really appreciated???

hek

Isn't incomingMQTT() ever called?

You seem to have commented away this line:
//_client.setCallback(incomingMQTT);

noelgeorgi

@hek i am trying to do that in void loop{} will update once it gets working properly and will do a pull request, the problem is that i can subscribe to a main topic but not its subfeeds

noelgeorgi

@hek Adafruit's MQTT libarary doesnt have wildcard subscriptions switching to https://github.com/256dpi/arduino-mqtt SSL seems to be working and library has callback support

monte

@noelgeorgi so do you have a working code already? It would be great to try it out.

Soloam

Also looking at this... was this ever implemented? And where your trials on server encryption or also in client authentication?

Thank You

monte

@soloam look at this https://forum.mysensors.org/topic/8863/esp8266-mqtt-gateway-ssl-connection/1

SSL support for MQTT on ESP8266

11

11.7k

11.2k

113.1k