SSL support for MQTT on ESP8266

[noelgeorgi]

@hek @Anticimex First install https://github.com/esp8266/arduino-esp8266fs-plugin and upload the crt file to the ESP8266 file system.

then issue these command from the folder in which MySensors is installed:
sed -i 's/WiFiClient/WiFiClientSecure/g' Arduino/libraries/MySensors/core/MyGatewayTransportEthernet.cpp

sed -i 's/WiFiClient/WiFiClientSecure/g' Arduino/libraries/MySensors/core/MyGatewayTransportMQTTClient.cpp

change your GatewayESP8266MQTTClient to this:

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik Ekblad
 * 
 * DESCRIPTION
 * The ESP8266 MQTT gateway sends radio network (or locally attached sensors) data to your MQTT broker.
 * The node also listens to MY_MQTT_TOPIC_PREFIX and sends out those messages to the radio network
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error  
 * 
 * See http://www.mysensors.org/build/esp8266_gateway for wiring instructions.
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4          
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *            
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 * 
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 * 
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */
#include "FS.h"
#include <EEPROM.h>
#include <SPI.h>

// Enable debug prints to serial monitor
#define MY_DEBUG 

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires usenrame/password
#define MY_MQTT_USER "admin" //your MQTT username
#define MY_MQTT_PASSWORD "admin" //your MQTT password

// Set WIFI SSID and password
#define MY_ESP8266_SSID "yourwifissid"
#define MY_ESP8266_PASSWORD "yourwifipassword"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 172,16,0,120

// If using static ip you need to define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 172,16,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// MQTT broker ip address.  
#define MY_CONTROLLER_IP_ADDRESS 172, 16, 0, 95

// The MQTT broker port to to open 
#define MY_PORT 8883      // note this change its port 8883 as per standard for SSL over MQTT

 /*
// Flash leds on rx/tx/err
#define MY_LEDS_BLINKING_FEATURE
// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Enable inclusion mode
#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
#define MY_INCLUSION_MODE_DURATION 60 
// Digital pin used for inclusion mode button
#define MY_INCLUSION_MODE_BUTTON_PIN  3 

#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED
*/

#include <ESP8266WiFi.h>
#include <MySensor.h>

void setup() { 
  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

  File ca = SPIFFS.open("/ca.crt", "r"); //replace ca.crt with the ca file name
  if (!ca) {
    Serial.println("Failed to open ca file");
  }
  else
  Serial.println("Successfully opened ca file");

if(_ethClient.loadCertificate(ca)) //_ethClient as defined in MyGatewayTransportMQTTClient.cpp
  Serial.println("loaded");
  else
  Serial.println("not loaded");
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  // Send locally attech sensors data here
}

NB: i used https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh for generating the ca files

[Anticimex]

I have no esp hw. And like I said, ssl and MQTT is specific to a certain architecture (from a gw and controller perspective) so I don't want it being mixed in with the existing security infrastructure that goes on every node.
You are free to implement it as such of course, but I won't bring it into the signing solution.

[hek]

@Anticimex
This has nothing to do with the message signing.

@noelgeorgi
Good!. Might be good to put this initialization in the optional before()-method instead of setup(), which runs before any other MySensors stuff.

[noelgeorgi]

@hek I didnt know there was an optional method. Thanks

[Anticimex]

@hek i know that perfectly well, but since I am adressed, I just want to make clear I do not consider myself involved

[noelgeorgi]

@hek pubsubclient has issues with ssl connection, frequent disconnections. Adafruit's MQTT library works like charm, trying to integrate into MySensors

[noelgeorgi]

@hek I somewhat succesfully integrated Adafruit MQTT library into MySensors, i am able to sent out data but not able to receive (subscribe). These are the changes i made:

diff output

• MySesnors.h

< #if defined(MY_GATEWAY_SERIAL) || defined(MY_GATEWAY_W5100) || defined(MY_GATEWAY_ENC28J60) || defined(MY_GATEWAY_ESP8266) || defined(MY_GATEWAY_MQTT_CLIENT) || defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
---
> #if defined(MY_GATEWAY_SERIAL) || defined(MY_GATEWAY_W5100) || defined(MY_GATEWAY_ENC28J60) || defined(MY_GATEWAY_ESP8266) || defined(MY_GATEWAY_MQTT_CLIENT)
55,59d54
< #if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 	#include "drivers/Adafruit_MQTT_Library/Adafruit_MQTT.cpp"
< 	#include "drivers/Adafruit_MQTT_Library/Adafruit_MQTT_Client.cpp"
< #endif
< 
180c175
< #if defined(MY_GATEWAY_MQTT_CLIENT) || defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
---
> #if defined(MY_GATEWAY_MQTT_CLIENT)

• keywords.txt

< MY_GATEWAY_MQTT_SECURE_CLIENT LITERAL 1

MyGatewayTransportMQTTClient.cpp

36,40c36
< 	#if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 		#define EthernetClient WiFiClientSecure
< 	#else 
< 		#define EthernetClient WiFiClient
< 	#endif
---
> 	#define EthernetClient WiFiClientSecure
54,277d49
< #if defined(MY_GATEWAY_MQTT_SECURE_CLIENT)
< 	Adafruit_MQTT_Client mqtt(&_ethClient
< 				 #if defined(MY_CONTROLLER_IP_ADDRESS)
<  				, _brokerIp 
< 				#else
< 				, MY_CONTROLLER_URL_ADDRESS
< 				#endif				
< 				, MY_PORT 
< 				#if defined(MY_MQTT_USER) && defined(MY_MQTT_PASSWORD) 
< 				, MY_MQTT_USER, MY_MQTT_PASSWORD
< 				#endif
< 				);
< 
< Adafruit_MQTT_Subscribe feed = Adafruit_MQTT_Subscribe(&mqtt,MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "/+/+/+/+/+" );
< Adafruit_MQTT_Subscribe *subscription;
< bool _connecting = true;
< bool _available = false;
< char _convBuffer[MAX_PAYLOAD*2+1];
< char _fmtBuffer[MY_GATEWAY_MAX_SEND_LENGTH];
< MyMessage _mqttMsg;
< 
< bool gatewayTransportSend(MyMessage &message) {
< 	if (!mqtt.connected())
< 		return false;
< 	snprintf_P(_fmtBuffer, MY_GATEWAY_MAX_SEND_LENGTH, PSTR(MY_MQTT_PUBLISH_TOPIC_PREFIX "/%d/%d/%d/%d/%d"), message.sender, message.sensor, mGetCommand(message), mGetAck(message), message.type);
< 	debug(PSTR("Sending message on topic: %s\n"), _fmtBuffer);
< 	return mqtt.publish(_fmtBuffer, message.getString(_convBuffer));
< }
< 
< void incomingMQTT(char* topic, byte* payload,
<                         unsigned int length)
< {
< 	debug(PSTR("Message arrived on topic: %s\n"), topic);
< 	char *str, *p;
< 	uint8_t i = 0;
< 	uint8_t bvalue[MAX_PAYLOAD];
< 	uint8_t blen = 0;
< 	uint8_t command = 0;
< 	for (str = strtok_r(topic, "/", &p); str && i <= 5;
< 			str = strtok_r(NULL, "/", &p)) {
< 		switch (i) {
< 			case 0: {
< 				// Topic prefix
< 				if (strcmp(str, MY_MQTT_SUBSCRIBE_TOPIC_PREFIX) != 0) {
< 					// Message not for us or malformed!
< 					return;
< 				}
< 				break;
< 			}
< 			case 1: {
< 				// Node id
< 				_mqttMsg.destination = atoi(str);
< 				break;
< 			}
< 			case 2: {
< 				// Sensor id
< 				_mqttMsg.sensor = atoi(str);
< 				break;
< 			}
< 			case 3: {
< 				// Command type
< 				command = atoi(str);
< 				mSetCommand(_mqttMsg, command);
< 				break;
< 			}
< 			case 4: {
< 				// Ack flag
< 				mSetRequestAck(_mqttMsg, atoi(str)?1:0);
< 				break;
< 			}
< 			case 5: {
< 				// Sub type
< 				_mqttMsg.type = atoi(str);
< 				// Add payload
< 				if (command == C_STREAM) {
< 					blen = 0;
< 					uint8_t val;
< 					while (*payload) {
< 						val = protocolH2i(*payload++) << 4;
< 						val += protocolH2i(*payload++);
< 						bvalue[blen] = val;
< 						blen++;
< 					}
< 					_mqttMsg.set(bvalue, blen);
< 				} else {
< 					char* ca;
< 					ca = (char *) payload;
< 					ca += length;
< 					*ca = '\0';
< 					_mqttMsg.set((const char*) payload);
< 				}
< 				_available = true;
< 			}
< 		}
< 		i++;
< 	}
< }
< 
< bool reconnectMQTT() {
< 	debug(PSTR("Attempting MQTT connection...\n"));
< 	// Attempt to connect
< 		
< int8_t ret;
< 
<   // Stop if already connected.
<   if (mqtt.connected()) {
<     return true;
<   }
< 
<   //Serial.print("Connecting to MQTT... ");
< 
<   uint8_t retries = 3;
<   while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
<        Serial.println(mqtt.connectErrorString(ret));
<        Serial.println("Retrying MQTT connection in 5 seconds...");
<        mqtt.disconnect();
<        delay(5000);  // wait 5 seconds
<        retries--;
<        if (retries == 0) {
<          // basically die and wait for WDT to reset me
<          while (1);
<        }
<   }
<   //Serial.println("MQTT Connected!");
<  
< 
< 		 {
< 		debug(PSTR("MQTT connected\n"));
< 		// Once connected, publish an announcement...
< 		//_client.publish("outTopic","hello world");
< 		// ... and resubscribe
< 		mqtt.subscribe(&feed);
< 		return true;
< 	}
< 	return false;
< }
< 
< bool gatewayTransportInit() {
< 	_connecting = true;
< 	//#if defined(MY_CONTROLLER_IP_ADDRESS)
< 	//	_client.setServer(_brokerIp, MY_PORT);
< 	//#else
< 	//	_client.setServer(MY_CONTROLLER_URL_ADDRESS, MY_PORT);
< 	//#endif
< 
< 	//_client.setCallback(incomingMQTT);
< 
<  if((subscription = mqtt.readSubscription(10))) {
<     if (subscription == &feed) 
<     {
<       Serial.print(F("Got: "));
<       Serial.println((char *)feed.lastread);
<     }}
< 
< 
<   	#if defined(MY_GATEWAY_ESP8266)
< 		// Turn off access point
< 		WiFi.mode (WIFI_STA);
< 		(void)WiFi.begin(MY_ESP8266_SSID, MY_ESP8266_PASSWORD);
< 		#ifdef MY_IP_ADDRESS
< 			WiFi.config(_clientIp, _gatewayIp, _subnetIp);
< 		#endif
< 		while (WiFi.status() != WL_CONNECTED)
< 		{
< 			delay(500);
< 			MY_SERIALDEVICE.print(".");
< 			yield();
< 		}
< 		MY_SERIALDEVICE.print("IP: ");
< 		MY_SERIALDEVICE.println(WiFi.localIP());
< 	#else
< 		#ifdef MY_IP_ADDRESS
< 			Ethernet.begin(_clientMAC, _clientIp);
< 		#else
< 			// Get IP address from DHCP
< 			if (!Ethernet.begin(_clientMAC))
< 			{
< 				MY_SERIALDEVICE.print("DHCP FAILURE...");
< 				_connecting = false;
< 				return false;
< 			}
< 			MY_SERIALDEVICE.print("IP: ");
< 			MY_SERIALDEVICE.println(Ethernet.localIP());
< 		#endif
< 
< 		// give the Ethernet interface a second to initialize
< 		// TODO: use HW delay
< 		wait(1000);
< 	#endif
< 	_connecting = false;
< 	return true;
< }
< 
< bool gatewayTransportAvailable() {
< 	if (_connecting)
< 		return false;
< 
< 	//keep lease on dhcp address
< 	//Ethernet.maintain();
< 	if (!_ethClient.connected()) {
< 		//reinitialise client
< 		if (gatewayTransportInit())
< 			reconnectMQTT();
< 		return false;
< 	}
< 	//_ethClient.loop();
< 
< 
< 
< 	return _available;
< }
< 
< MyMessage & gatewayTransportReceive() {
< 	// Return the last parsed message
< 	_available = false;
< 	return _mqttMsg;
< }
< 
< 
< 
< 
< 
< 
< #else
279d50
< 
456c227
< #endif
---
> 

• GatewWay Code

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik Ekblad
 * 
 * DESCRIPTION
 * The ESP8266 MQTT gateway sends radio network (or locally attached sensors) data to your MQTT broker.
 * The node also listens to MY_MQTT_TOPIC_PREFIX and sends out those messages to the radio network
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error  
 * 
 * See http://www.mysensors.org/build/esp8266_gateway for wiring instructions.
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4          
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *            
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 * 
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 * 
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */

#include <EEPROM.h>
#include <SPI.h>

// Enable debug prints to serial monitor
#define MY_DEBUG 

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
//#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_SECURE_CLIENT
#define MY_GATEWAY_ESP8266

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires usenrame/password
#define MY_MQTT_USER "user"
#define MY_MQTT_PASSWORD "pass"

// Set WIFI SSID and password
#define MY_ESP8266_SSID "ssid"
#define MY_ESP8266_PASSWORD "password"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 172,16,0,120

// If using static ip you need to define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 172,16,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// MQTT broker ip address.  
#define MY_CONTROLLER_URL_ADDRESS "domain.com"

// The MQTT broker port to to open 
#define MY_PORT 8883      

 /*
// Flash leds on rx/tx/err
#define MY_LEDS_BLINKING_FEATURE
// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Enable inclusion mode
#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
#define MY_INCLUSION_MODE_DURATION 60 
// Digital pin used for inclusion mode button
#define MY_INCLUSION_MODE_BUTTON_PIN  3 

#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED
*/

#include <ESP8266WiFi.h>
#include <MySensor.h>

void setup() { 
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  // Send locally attech sensors data here
}

Any help would be really appreciated???

[hek]

Isn't incomingMQTT() ever called?

You seem to have commented away this line:
//_client.setCallback(incomingMQTT);

[noelgeorgi]

@hek i am trying to do that in void loop{} will update once it gets working properly and will do a pull request, the problem is that i can subscribe to a main topic but not its subfeeds

[noelgeorgi]

@hek Adafruit's MQTT libarary doesnt have wildcard subscriptions switching to https://github.com/256dpi/arduino-mqtt SSL seems to be working and library has callback support

[monte]

@noelgeorgi so do you have a working code already? It would be great to try it out.

[Soloam]

Also looking at this... was this ever implemented? And where your trials on server encryption or also in client authentication?

Thank You

[monte]

@soloam look at this https://forum.mysensors.org/topic/8863/esp8266-mqtt-gateway-ssl-connection/1