ESP8266 WiFi gateway port for MySensors

[Yveaux]

For everyone wondering what serial output during startup of the ESP8266-NRF24 gateway should look like (all captures taken using link development branch from Oct 6, 2015 (hek just merged gateway-refactoring this morning, so output might have slightly changed)) .

WiFi startup comes first after reset, followed by MySensors (nRF24) startup:

• WiFi startup (not using Serial.setDebugOutput(true) )

ESP8266 MySensors Gateway
Connecting to YOUR_SSID
...........Connected!
IP: 192.168.1.101

• WiFi startup (using Serial.setDebugOutput(true) )

ESP8266 MySensors Gateway
Connecting to YOUR_SSID
scandone
f 0, ....scandone
.add 0
aid 10
pm open phy_2,type:2 0 0
cnt 

connected with YOUR_SSID, channel 1
dhcp client start...
.....ip:192.168.1.101,mask:255.255.255.0,gw:192.168.1.1
.Connected!
IP: 192.168.1.101

• MySensors startup (not using DEBUG and MY_DEBUG_VERBOSE)

0;0;3;0;9;gateway started, id=0, parent=0, distance=0

• MySensors startup (using DEBUG and MY_DEBUG_VERBOSE)

write_register(0x04,0x5F)
write_register(0x06,0x27)
write_register(0x06,0x07)
write_register(0x00,0x0C)
write_register(0x07,0x70)
write_register(0x05,0x4C)
write_register(0x00,0x0E)
write_register(0x00,0x0E)
STATUS		0x0E RX_DR=0 TX_DS=0 MAX_RT=0 RX_P_NO=7 TX_FULL=0
RX_ADDR_P0-1	0x0E0E0E0E0E 0303030303 
RX_ADDR_P2-5	0xC3 C4 C5 C6 
TX_ADDR		0x0E0E0E0E0E 
RX_PW_P0-6	0x00 00 00 00 00 00 
EN_AA		0x3F 
EN_RXADDR	0x03 
RF_CH		0x4C 
RF_SETUP	0x07 
CONFIG		0x0E 
DYNPD/FEATURE	0x00 00 
Data Rate	1MBPS
Model		nRF24L01+
CRC Length	16 bits
PA Power	PA_MAX
write_register(0x01,0x3F)
write_register(0x01,0x3B)
write_register(0x1D,0x06)
FEATURE=0x06
write_register(0x1C,0x03)
write_register(0x05,0x4C)
write_register(0x06,0x03)
write_register(0x06,0x23)
write_register(0x04,0x5F)
write_register(0x00,0x0E)
write_register(0x1D,0x06)
FEATURE=0x06
write_register(0x1C,0x3F)
write_register(0x13,0x20)
write_register(0x02,0x07)
write_register(0x11,0x20)
write_register(0x02,0x07)
write_register(0x12,0x20)
write_register(0x02,0x07)
write_register(0x00,0x0F)
write_register(0x07,0x70)
write_register(0x02,0x06)
0;0;3;0;9;gateway started, id=0, parent=0, distance=0

[Yveaux]

@Pseudomizer Had a better look at your crashdump.
Your ESP seems to crash with a watchdog timeout when MySensors is setting the datarate (or shortly after that) after nRF24 has been initialized.
Default datarate is RF24_250KBPS.

Some questions, trying to rule things out:

• Do you use this default datarate?
• Could it be that you have an nRF24L01+ clone, which in reality is a nRF24L01 (clone)?
• Did you ever try and succeed to connect these radios to a regular Arduino?
• Are all radio's from the same batch/order?

[Pseudomizer]

I tried multiple different data rates when I created the ethernet gateway from mysensors. It didn't work at all in terms of connecting the sensors to the gateway while the gateway was visible in Vera until I made them all the same data rate. If I recall correctly I got them working with both at 250kpbs or 1Mbps.

Since I tried to build the wifi gateway I haven't checked the data rates. I am going to check this next.

In regards to your other questions:
I purchased 4 radios from Amazon:
http://www.amazon.com/nRF24L01-Wireless-Transceiver-Arduino-Compatible/dp/B00E594ZX0/ref=cm_cr_pr_product_top?ie=UTF8

Yes, those radios work totally fine with the ethernet gateway from mysensors and I had two sensors attached. My goal is to have a wireless gateway so I can place it anywhere in the house and your great work would enable me to do so. ;-)

[Pseudomizer]

Thinking about the data rate... at the moment I don't have any other sensors powered on at all which means I am just installing the wifi gateway. There should be no mismatch in the data rates unless I would have at least one sensor with a different data rate causing the issue. The Wifi gateway should boot up successfully without any client sensor.

Is my thinking correct?

[miguelingles]

@Yveaux said:

y but I

it,s works !!!!! you have to do is remove the DIO0 while Bots and put it quickly . If not removed DIO0 not start.

[Pseudomizer]

I have done EXTENSIVE testing and I got it working now but I don't understand why.

It seems like my MyConfig.h is broken which is the one which I copied over from your development package. I used one of my MyConfig.h files and the module boot was successfull and it did not crash anymore. It is stable and fully integrated into Vera with sensors now.

I also tested the different data rates. I tested 250KBPS and 1MBPS and both work fine as long as I configure the gateway and the sensor clients with the same data rate. If I use different data rates between client and gateway, the gateway does NOT crash but the sensor clients simply don't connect.

Now I have a working MyConfig.h and a broken MyConfig.h. Here is the WORKING MyConfig.h file.

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 */


#ifndef MyConfig_h
#define MyConfig_h
#include <stdint.h>

// Enable debug flag for debug prints. This will add a lot to the size of the final sketch but good
// to see what is actually is happening when developing
//#define DEBUG
//#define MY_DEBUG_VERBOSE

// Disable this line, If you are using TX(1), RX(0) as normal I/O pin
#define ENABLED_SERIAL

// Serial output baud rate (for debug prints and serial gateway)
#define BAUD_RATE 115200


/**********************************
*  Over the air firmware updates
***********************************/

// The following define enables the safe over-the-air firmware update feature
// which requires external flash and the DualOptiBoot bootloader.
// Note: You can still have OTA FW updates without external flash but it
// requires the MYSBootloader and disabled MY_OTA_FIRMWARE_FEATURE
//#define MY_OTA_FIRMWARE_FEATURE
// Slave select pin for external flash
#define MY_OTA_FLASH_SS 8
// Flash jdecid
#define MY_OTA_FLASH_JDECID 0x1F65


/**********************************
*  Information LEDs blinking
***********************************/
// This feature enables LEDs blinking on message receive, transmit
// or if some error occured. This was commonly used only in gateways,
// but now can be used in any sensor node. Also the LEDs can now be
// disabled in the gateway.

// #define WITH_LEDS_BLINKING

// The following setting allows you to inverse the blinking feature WITH_LEDS_BLINKING
// When WITH_LEDS_BLINKING_INVERSE is enabled LEDSs are normally turned on and switches
// off when blinking

//#define WITH_LEDS_BLINKING_INVERSE


// default LEDs blinking period in milliseconds
#define DEFAULT_LED_BLINK_PERIOD 300
// The RX LED default pin
#define DEFAULT_RX_LED_PIN 6
// The TX LED default pin
#define DEFAULT_TX_LED_PIN 5
// The Error LED default pin
#define DEFAULT_ERR_LED_PIN 4


/**********************************
*  Message Signing Settings
***********************************/
// Disable to completly disable signing functionality in library
//#define MY_SIGNING_FEATURE

// Define a suitable timeout for a signature verification session
// Consider the turnaround from a nonce being generated to a signed message being received
// which might vary, especially in networks with many hops. 5s ought to be enough for anyone.
#define MY_VERIFICATION_TIMEOUT_MS 5000

// Enable to turn on whitelisting
// When enabled, a signing node will salt the signature with it's unique signature and nodeId.
// The verifying node will look up the sender in a local table of trusted nodes and
// do the corresponding salting in order to verify the signature.
// For this reason, if whitelisting is enabled on one of the nodes in a sign-verify pair, both
// nodes have to implement whitelisting for this to work.
// Note that a node can still transmit a non-salted message (i.e. have whitelisting disabled)
// to a node that has whitelisting enabled (assuming the receiver does not have a matching entry
// for the sender in it's whitelist)
//#define MY_SECURE_NODE_WHITELISTING

// MySigningAtsha204 default setting
#define MY_ATSHA204_PIN 17 // A3 - pin where ATSHA204 is attached

// MySigningAtsha204Soft default settings
#define MY_RANDOMSEED_PIN 7 // A7 - Pin used for random generation (do not connect anything to this)

// Key to use for HMAC calculation in MySigningAtsha204Soft (32 bytes)
#define MY_HMAC_KEY 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00

/**********************************
*  NRF24L01 Driver Defaults
***********************************/
#define RF24_CE_PIN		   9
#define RF24_CS_PIN		   10
#define RF24_PA_LEVEL 	   RF24_PA_MAX
#define RF24_PA_LEVEL_GW   RF24_PA_LOW
// RF channel for the sensor net, 0-127
#define RF24_CHANNEL	   76
//RF24_250KBPS for 250kbs, RF24_1MBPS for 1Mbps, or RF24_2MBPS for 2Mbps
//#define RF24_DATARATE 	   RF24_1MBPS
#define RF24_DATARATE 	   RF24_250KBPS
// This is also act as base value for sensor nodeId addresses. Change this (or channel) if you have more than one sensor network.
#define RF24_BASE_RADIO_ID ((uint64_t)0xA8A8E1FC00LL)

// Enable SOFTSPI for NRF24L01 when using the W5100 Ethernet module
//#define SOFTSPI
#ifdef SOFTSPI
	// Define the soft SPI pins used for NRF radio
	const uint8_t SOFT_SPI_MISO_PIN = 16;
    const uint8_t SOFT_SPI_MOSI_PIN = 15;
    const uint8_t SOFT_SPI_SCK_PIN = 14;
#endif


/**********************************
*  RFM69 Driver Defaults
***********************************/
// Default network id. Use the same for all nodes that will talk to each other
#define RFM69_NETWORKID     100

// Default frequency to use. This must match the hardware version of the RFM69 radio (uncomment one):
// #define RFM69_FREQUENCY   RF69_433MHZ
#define RFM69_FREQUENCY   RF69_868MHZ
//#define FREQUENCY     RF69_915MHZ

// Enable this for encryption of packets
//#define RFM69_ENABLE_ENCRYPTION
#define RFM69_ENCRYPTKEY    "sampleEncryptKey" //exactly the same 16 characters/bytes on all nodes!




#endif

And here is the NON WORKING MyConfig.h file.

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 */


#ifndef MyConfig_h
#define MyConfig_h
#include <stdint.h>

// Enable debug flag for debug prints. This will add a lot to the size of the final sketch but good
// to see what is actually is happening when developing
#define DEBUG

// Enable MY_DEBUG_VERBOSE flag for verbose debug prints. Requires DEBUG to be enabled.
// This will add even more to the size of the final sketch!
#define MY_DEBUG_VERBOSE

// Disable this line, If you are using TX(1), RX(0) as normal I/O pin
#define ENABLED_SERIAL

// Serial output baud rate (for debug prints and serial gateway)
#define BAUD_RATE 115200


/**********************************
*  Over the air firmware updates
***********************************/

// The following define enables the safe over-the-air firmware update feature
// which requires external flash and the DualOptiBoot bootloader.
// Note: You can still have OTA FW updates without external flash but it
// requires the MYSBootloader and disabled MY_OTA_FIRMWARE_FEATURE
//#define MY_OTA_FIRMWARE_FEATURE
// Slave select pin for external flash
#define MY_OTA_FLASH_SS 8
// Flash jdecid
#define MY_OTA_FLASH_JDECID 0x1F65


/**********************************
*  Information LEDs blinking
***********************************/
// This feature enables LEDs blinking on message receive, transmit
// or if some error occured. This was commonly used only in gateways,
// but now can be used in any sensor node. Also the LEDs can now be
// disabled in the gateway.

#define WITH_LEDS_BLINKING

// The following setting allows you to inverse the blinking feature WITH_LEDS_BLINKING
// When WITH_LEDS_BLINKING_INVERSE is enabled LEDSs are normally turned on and switches
// off when blinking

//#define WITH_LEDS_BLINKING_INVERSE


// default LEDs blinking period in milliseconds
#define DEFAULT_LED_BLINK_PERIOD 300
// The RX LED default pin
#define DEFAULT_RX_LED_PIN 6
// The TX LED default pin
#define DEFAULT_TX_LED_PIN 5
// The Error LED default pin
#define DEFAULT_ERR_LED_PIN 4


/**********************************
*  Message Signing Settings
***********************************/
// Disable to completly disable signing functionality in library
//#define MY_SIGNING_FEATURE

// Define a suitable timeout for a signature verification session
// Consider the turnaround from a nonce being generated to a signed message being received
// which might vary, especially in networks with many hops. 5s ought to be enough for anyone.
#define MY_VERIFICATION_TIMEOUT_MS 5000

// Enable to turn on whitelisting
// When enabled, a signing node will salt the signature with it's unique signature and nodeId.
// The verifying node will look up the sender in a local table of trusted nodes and
// do the corresponding salting in order to verify the signature.
// For this reason, if whitelisting is enabled on one of the nodes in a sign-verify pair, both
// nodes have to implement whitelisting for this to work.
// Note that a node can still transmit a non-salted message (i.e. have whitelisting disabled)
// to a node that has whitelisting enabled (assuming the receiver does not have a matching entry
// for the sender in it's whitelist)
//#define MY_SECURE_NODE_WHITELISTING

// MySigningAtsha204 default setting
#define MY_ATSHA204_PIN 17 // A3 - pin where ATSHA204 is attached

// MySigningAtsha204Soft default settings
#define MY_RANDOMSEED_PIN 7 // A7 - Pin used for random generation (do not connect anything to this)

// Key to use for HMAC calculation in MySigningAtsha204Soft (32 bytes)
#define MY_HMAC_KEY 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00

/**********************************
*  NRF24L01 Driver Defaults
***********************************/
#define RF24_CE_PIN		   9
#define RF24_CS_PIN		   10
#define RF24_PA_LEVEL 	   RF24_PA_MAX
#define RF24_PA_LEVEL_GW   RF24_PA_LOW
// RF channel for the sensor net, 0-127
#define RF24_CHANNEL	   76
//RF24_250KBPS for 250kbs, RF24_1MBPS for 1Mbps, or RF24_2MBPS for 2Mbps
#define RF24_DATARATE 	   RF24_250KBPS
// This is also act as base value for sensor nodeId addresses. Change this (or channel) if you have more than one sensor network.
#define RF24_BASE_RADIO_ID ((uint64_t)0xA8A8E1FC00LL)

// Enable SOFTSPI for NRF24L01 when using the W5100 Ethernet module
//#define SOFTSPI
#ifdef SOFTSPI
	// Define the soft SPI pins used for NRF radio
	const uint8_t SOFT_SPI_MISO_PIN = 16;
    const uint8_t SOFT_SPI_MOSI_PIN = 15;
    const uint8_t SOFT_SPI_SCK_PIN = 14;
#endif


/**********************************
*  RFM69 Driver Defaults
***********************************/
// Default network id. Use the same for all nodes that will talk to each other
#define RFM69_NETWORKID     100

// Default frequency to use. This must match the hardware version of the RFM69 radio (uncomment one):
// #define RFM69_FREQUENCY   RF69_433MHZ
#define RFM69_FREQUENCY   RF69_868MHZ
//#define FREQUENCY     RF69_915MHZ

// Enable this for encryption of packets
//#define RFM69_ENABLE_ENCRYPTION
#define RFM69_ENCRYPTKEY    "sampleEncryptKey" //exactly the same 16 characters/bytes on all nodes!

#endif

I don't understand why the second one makes the module reboot constantly. However, I am happy now that it is working but I wanted to share my experience and maybe it can help others to troubleshoot this and understand why this was happening as I am sure I will not be the only one having that issue in the future.

I do appreciate all the help so far and I wouldn't have been able to solve it without your help Yveaux. THANK YOU!!!

[hek]

You cannot mix MyConfig.h between branches.

In the development brach all MySensors related defines has the prefix "MY_".

[Frank Herrmann]

Hi, if it possible to show a sketch with an ESP8266 WIFI => MQTT gateway? I figure out, that the received sensor data will automatic publish to my MQTT Server running on my Raspi. Then i can install this pice of wonderfull work in every room from my house :)

[hek]

I created a simple ESP-MQTT sketch a while ago. I've only done some simple tests using it. Feedback appreciated.

https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/examples/GatewayESP8266MQTTClient/GatewayESP8266MQTTClient.ino

[Frank Herrmann]

@hek Very nice! the code are magic :) Very nice solution, i'll test this next days.

...
#include <ESP8266WiFi.h>
#include <MySensor.h>

void setup() { 
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  // Send locally attech sensors data here
}

[FotoFieber]

@hek said:

I created a simple ESP-MQTT sketch a while ago. I've only done some simple tests using it. Feedback appreciated.

https://github.com/mysensors/Arduino/blob/development/libraries/MySensors/examples/GatewayESP8266MQTTClient/GatewayESP8266MQTTClient.ino

Testing the above sket from the development branch with a new sensbender micro I get:
mygateway/255/255/3/0/3 (null) on MQTT

This is an id request. I answer this request with
mosquitto_pub -h ds -t "mygateway/255/255/3/0/4" -m 8
which should set the ID 8 on the sensbender micro in my undestanding. I can see the message in the serial terminal on the ESP-8266 but the message doesn't arrive at the sensbender micro.

Is my message to set the id correct?

[hek]

Yes, the response message looks ok. But I must confess that I haven't verified the MQTT gateway much yet. Probably something I've missed (=bug).

Do you have the possibility to to see where the message ends up or being thrown away (debug-print-debugging is awesome, yes I know...) in the MQTT gateway?

Or wait a few days until I've had time to look at it.

[FotoFieber]

@hek said:

...
Do you have the possibility to to see where the message ends up or being thrown away (debug-print-debugging is awesome, yes I know...) in the MQTT gateway?
...
There seems to be a bug in incomingMQTT:
replace
for (str = strtok_r(topic, "/", &p); str && i < 5;
with
for (str = strtok_r(topic, "/", &p); str && i <= 5;

or it will never send a message.

There seems to be another problem:
0;0;3;0;9;send: 0-0-0-255 s=255,c=3,t=4,pt=0,l=1,sg=0,st=fail:8

Why is the payload in st=fail?

The messages sent to MQTT seem to make a merry go round and come back.... :)

[hek]

@FotoFieber said:

Why is the payload in st=fail?

The payload (8) seems correct. Gateway just never received any ack from node... which is fine for broadcast messages (to node 255) such as this.

The messages sent to MQTT seem to make a merry go round and come back....

Ok, like you suspected earlier? So we need to subscribe to something else than we publish to then...

[FotoFieber]

@hek
Loaded the ethernet ESP-8266 gateway and entered with telnet:
255;255;3;0;4;8

It is working as it should. Now I have assigned the id 8.

Tried to solve the merry go round but it didn't help. Now I let the stage to the pros.. :)

[Fabien]

Do you think we can use this board with MySensors (with NRF24 or RFM69) ?
https://github.com/hallard/NodeMCU-Gateway

[Mike Cayouette]

Hi,

If I want to use the GatewayESP8266MQTTClient sketch located in the development branch do I also need to use the mysensors libraries in that branch?

Thank you,

Mike

[hek]

@Mike-Cayouette

Yes

[vickey]

@hek Thanks for the code. I have few questions in my mind. Firstly do I just have to upload this ESP8266MQTTClient.ino sketch on esp8266 12-E model? or does it needs anything to add in the sketch except SSID and password? I want to use ESP-8266-E and NRF24L01+ as gateway for openhab. If it is possible then, do I have to adopt the same procedure for configuring openhab as ethernet gateway? Secondly, Can I make two gateways using this sketch, is this possible? As I want to add sensors which are 500 meters away at my farmhouse.

[joshmosh]

This is a bit off topic, but hopefully not too off ;-)
What about a port of the MySensors library /API to ESP8266 ? Instead of two "boards" (Arduino nano plus nRF24L01+) only one would be required. For simple sensors an ESP-01 would be sufficient - about the size of a nRF24, but with much more muscle than an Arduino. Of course, running off a battery probably would not work because of the high power consumption of the ESP.

Or am I asking for something which is already possible ?

Josh