ESP8266 WiFi gateway port for MySensors

[Yveaux]

@tjay4x4 try development version of MySensors. This thread discussions a similar issue:
http://forum.mysensors.org/topic/2118/esp8266-wifi-gateway-issue/2

[Yveaux]

@tjay4x4 Just crossed my reply! Great it's working now!

[robosensor]

Yesterday I switched gateway hardware from serial gateway based on arduino pro mini 3.3V and nrf24l01+ (connected directly to uart port of my mikrotik router) to esp8266 nodemcu and nrf24l01+ (exactly as at this page: http://www.mysensors.org/build/esp8266_gateway, using GatewayESP8266 sketch from 1.6 branch, revision https://github.com/mysensors/Arduino/tree/1035f84a431a8f591d31100c3e73e99a8245e345).

I have 4 nodes with ~27 sensors total, half of sensors sending data every 10 seconds, another sensors send data only if environment changed.

After switching to esp8266 my gateway started loosing packets (messages) from my sensor nodes:

Left side of graphs is esp8266, right side is old pro mini serial gateway, sensors sending data every 10 seconds. Distance between gateways is about 5-10 centimeters. Both nrf24l01+ boards oriented in the same direction.

nrf24l01+ board isn't changed at all (I'm using old board from pro mini serial gateway).

Also, I have tried different nrf24l01+ boards (different PCBs and different chip revisions, but seems like all of them are fake china chips). I have tried soldering 220 uF electrolytic and 0.1 uF ceramic capacitors directly to nrf24l01+ power pins. But nothing changed. Esp8266 gateway is losing messages.

Switching back to pro mini serial gateway is completely solves issue (as shown in graph above).

What else can I try to fix this? Could it be interference between wifi chip and nrf24? Or, maybe, this is software problem? Maybe non-interrupt driven polling of nrf24l01 causes loosing messages coming too often?

[hek]

@robosensor

Interesting graph... But I have no immediate answer to you.

[robosensor]

@hek graphs are drawn by Zabbix. My own Perl script connects directly to gateway, reads sensor data and sends this data directly to Zabbix.

Also, I have tried to write raw data from gateway directly to file and execute "tail -f logfile.txt | grep sensor-specific-id". With old pro mini gateway data from sensor comes every 10 seconds. From esp8266 gateway sensor data comes very irregular.

Also, every send() call in all my nodes followed by wait(25) call to reduce possible power issues and interference issues.

UPDATE: I have recompiled esp8266-gateway sketch with debugging enabled, but I get the same results with loosing messages then I connecting directly to uart port of esp8266 uart-usb converter.

[Yveaux]

@robosensor What strikes me is the fact that most of the time not all sensor values seem to disappear at the same time. A failing WiFi connection would be the most likely cause when all dis/reappear at the same time (like at 13:44 and 13:22).
However, most of the time at least one sensor seems to report its values correctly. IMO This rules out the radio.
Is there anything interesting in the serial logs?

[robosensor]

@Yveaux thank you for your reply. It is not failing wifi connection, as the same problem exists then I using serial connection to esp8266 (via on-board usb-uart converter on nodemcu board).

Nothing interesting in the serial logs (with debug mode enabled), just random absence of sensor data. Seems like I should connect sensor node to collect debug logs from node side.

Also speed is changed - much faster CPU (80 MHz vs 8 MHz) and much faster connection (wifi vs uart 9600 bps). Maybe this affects.

[robosensor]

Seems like problem with loosing messages is gateway firmware related.

Then I connect two controllers (my own perl script and MYSController 0.1.2.282) to esp8266 gateway (#define MY_GATEWAY_MAX_CLIENTS 2), both controllers receiving exactly the same data from gateway, but messages lost even more.

1 on graphs = one controller connected
2 on graphs = two controllers connected.

I'm not familiar with esp8266 network programming, but could it be that firmware looses incoming packets from nrf24l01+ due to blocking call to wifi-related network functions? Seems like during sending data to wifi network firmware does not have time to pick up all incoming packets from nrf24l01+ chip.

[robosensor]

I think I found problem. As I said before, seems like WiFi's clients[i].write() call blocks thread for a very long time.

I added logging of write() time:

		// Send message to connected clients
		#if defined(MY_GATEWAY_ESP8266)
			unsigned long start_time = hwMillis();
			for (uint8_t i = 0; i < ARRAY_SIZE(clients); i++)
			{
				if (clients[i] && clients[i].connected())
				{
					clients[i].write((uint8_t*)_ethernetMsg, strlen(_ethernetMsg));
				}
			}
			debug(PSTR("WiFi transaction time: %u ms\n"), hwMillis() - start_time);
		#else
			_ethernetServer.write(_ethernetMsg);
		#endif

And got following results:
For Perl script from FreeBSD server in another country (ping to server is about 55 milliseconds):

0;0;3;0;9;WiFi transaction time: 141 ms
0;0;3;0;9;read: 2-2-0 s=1,c=1,t=23,pt=2,l=2,sg=0:98
0;0;3;0;9;WiFi transaction time: 141 ms
0;0;3;0;9;read: 2-2-0 s=6,c=1,t=43,pt=3,l=2,sg=0:2
0;0;3;0;9;WiFi transaction time: 140 ms
0;0;3;0;9;read: 1-1-0 s=105,c=1,t=0,pt=7,l=5,sg=0:27.5000
0;0;3;0;9;WiFi transaction time: 136 ms
0;0;3;0;9;read: 1-1-0 s=106,c=1,t=0,pt=7,l=5,sg=0:52.8750
0;0;3;0;9;WiFi transaction time: 140 ms
0;0;3;0;9;read: 1-1-0 s=107,c=1,t=0,pt=7,l=5,sg=0:45.9375
0;0;3;0;9;WiFi transaction time: 142 ms
0;0;3;0;9;read: 1-1-0 s=108,c=1,t=0,pt=7,l=5,sg=0:34.4375
0;0;3;0;9;WiFi transaction time: 137 ms
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=16,pt=0,l=1,sg=0:0
0;0;3;0;9;WiFi transaction time: 143 ms

Connection from LAN (windows telnet client):

0;0;3;0;9;read: 3-3-0 s=1,c=1,t=23,pt=2,l=2,sg=0:61
0;0;3;0;9;WiFi transaction time: 209 ms
0;0;3;0;9;read: 4-4-0 s=1,c=1,t=24,pt=2,l=2,sg=0:152
0;0;3;0;9;WiFi transaction time: 201 ms
0;0;3;0;9;read: 4-4-0 s=2,c=1,t=24,pt=2,l=2,sg=0:43
0;0;3;0;9;WiFi transaction time: 212 ms
0;0;3;0;9;read: 4-4-0 s=3,c=1,t=24,pt=2,l=2,sg=0:129
0;0;3;0;9;WiFi transaction time: 209 ms
0;0;3;0;9;read: 4-4-0 s=4,c=1,t=24,pt=2,l=2,sg=0:147
0;0;3;0;9;WiFi transaction time: 208 ms
0;0;3;0;9;read: 2-2-0 s=102,c=1,t=0,pt=7,l=5,sg=0:4.2500
0;0;3;0;9;WiFi transaction time: 221 ms
0;0;3;0;9;read: 2-2-0 s=103,c=1,t=0,pt=7,l=5,sg=0:27.4375
0;0;3;0;9;WiFi transaction time: 208 ms
0;0;3;0;9;read: 2-2-0 s=104,c=1,t=0,pt=7,l=5,sg=0:23.3750
0;0;3;0;9;WiFi transaction time: 210 ms
0;0;3;0;9;read: 2-2-0 s=3,c=1,t=1,pt=7,l=5,sg=0:99.9
0;0;3;0;9;WiFi transaction time: 208 ms

Seems like many packets coming to NRF24L01+ in this time interval (140-200 milliseconds) and this causes NRF buffer overflow and packet loss.

[hek]

Damn, 200 msec is a pretty long time.

[Yveaux]

@hek Yeah, but is this significantly longer than for a wired connection?
Only real solution would be to switch to interrupt based message handling for the MySensors network (hopefully the ESP stack can handle this...)
@robosensor Are these long delays (only) caused by the fact that this is a long distance connection?
How is your mileage for a server on e.g. the same lan?

[Fabien]

Exactly same problem here with RFM69W Radio. With 2 clients, i loose more packets. Both controller and ESP are on the same LAN. No probelm with Serial Gateway on Jeelink v3.

[robosensor]

@Yveaux seems like delays caused by low-level network/tcp code or settings of receiving side (maybe something as TCP ACK packet delays or Nagle algorithm or something like this).

ESP8266 connected directly to Mikrotik RB2011 router.

Then I connect from external server (ping to server is about 55 milliseconds, FreeBSD 9.3, Perl), delays about 140 milliseconds.
Then I connect from windows-based computer, connected directly to router, delays about 200 milliseconds (both windows telnet and PuTTy).
For 2 parallel above connections delays about 350 milliseconds.

Then I connect from router's internal telnet client, delays about 1-2 milliseconds
Then I connect from android-based phone (phone connected to router via wifi, using ConnectBot software), delays vary from 3-5 milliseconds (mostly) to 100-150 milliseconds (less frequently).

Seems like interrupts for nrf24 is the only solution. Seems like esp8266 gateway with such delays is completely unusable with high packet rates. With ~150 milliseconds delays no more than 6 messages/second packet rate allowed.

Just to reference: I'm using Arduino 1.6.5 with esp8266 libraries version 2.0.0 installed.

[Yveaux]

@robosensor @Fabien Please understand that this issue is not related to ESP8266. I expect delays to be nearly identical when using the Ethernet gateway (wired ethernet) and ATMega microcontroller. Of course the serial gateway will not suffer from these delays as it doesn't use ethernet as transport medium -- it is a direct connection.
Switching to interrupt-based message handling for the MySensorts stack is not a simple task. It will require locking and buffering, and footprint of the library (especially RAM) will increase.
Furthermore I don't know how the ESP's network stack handles interrupts during ethernet processing. Worst case it will just block interrupts or run from a higher-priority interrupt than the nRF and all efforts of making the MySensors library interrupt based will be in vain.
I personally would try to reduce the latency in your server connection by choosing a server nearby or by buffering your traffic by a server in your LAN. MQTT for example can easily be buffered by a local broker which then uses the slow connection to sync with another server.
Another option is switching to a serial gateway, but again you need a local server inbetween.
A simple Raspberry Pi or so could act as a server for these scenarios.

Another option would be to implement a retry mechanism on your sensors. If the gateway does not confirm reception of your message, then resend it.

[robosensor]

@Yveaux I completely agree with you. Rewriting gateway code to support interrupts is not very easy task and it is not clear what problems you will encounter. Furthermore, rewriting will not solve the problem of delays, this only can solve packet loss problem.

Just tried to connect from RPi 2 (connected using WiFi directly to router), average delay is 2-5 milliseconds, sometimes 5-10 ms. Seems like this is good temporary solution.

[D_dude]

@Yveaux said:

@D_dude flush the serial output after each print.
The ESP will output the serial data in the background while your programm continues and then it crashes.

Finally was able to get everything working.
Ordered a fresh nRF from Itead and started from scratch. everything worked fine first time. seems like the nRF from ebay was causing issues.

[robosensor]

Just tried old version of esp8266 libraries (1.6.5-947-g39819f0 instead of 2.0.0) with old version of esp8266 gateway (master branch, commit 9ef2604c81d2fea5d646a9a194f312192833a79b) and got exactly the same results.

WiFi write() time is about 150-200 milliseconds for remote server in another country, 200-210 milliseconds for windows7-based computer in local area network and 0-4 milliseconds for Raspberry Pi 2 in local area network.

To show transaction times just replace original output function in Esp8266Gateway.ino with following code:

void output(const char *fmt, ... )
{
  char serialBuffer[MAX_SEND_LENGTH];
  va_list args;
  va_start (args, fmt );
  vsnprintf_P(serialBuffer, MAX_SEND_LENGTH, fmt, args);
  va_end (args);
  Serial.print(serialBuffer);

  unsigned long start_time = millis();
  
  for (uint8_t i = 0; i < ARRAY_SIZE(clients); i++)
  {
    if (clients[i] && clients[i].connected())
    {
//       Serial.print("Client "); Serial.print(i); Serial.println(" write");
       clients[i].write((uint8_t*)serialBuffer, strlen(serialBuffer));
    }
  }
  
  start_time = millis() - start_time;
  Serial.print("WiFi transaction time: "); Serial.print(start_time); Serial.println("");
}

Also disconnection of client during clients[i].write() call causes reboot by watchdog:

 ets Jan  8 2013,rst cause:4, boot mode:(3,7)

wdt reset
load 0x4010f000, len 1264, room 16 
tail 0
chksum 0x42
csum 0x42
~ld

[Yveaux]

@robosensor said:

WiFi write() time is about 150-200 milliseconds for remote server in another country, 200-210 milliseconds for windows7-based computer in local area network and 0-4 milliseconds for Raspberry Pi 2 in local area network.

So another confirmation of what we knew already.

Do you know how many nRF messages are actually sent during this wifi write call?
The nRF has a rx buffer for 3 messages -- not sure if MySensors uses all 3 of them.

Also disconnection of client during clients[i].write() call causes reboot by watchdog

I guess this is ESP core related. Is this fixed in code 2.0.0?

[robosensor]

@Yveaux said:

So another confirmation of what we knew already.

Yes. It seems that I now understand what is happening. WiFiClient::write() blocks thread until TCP ACK packet received or until timeout (5 seconds). Windows sending TCP ACKs after 200ms timeout. That's why write() delayed for ~200 milliseconds for windows-based controller. More information (and link to non-blocking library) is available here: https://github.com/esp8266/Arduino/issues/922

Do you know how many nRF messages are actually sent during this wifi write call?
The nRF has a rx buffer for 3 messages -- not sure if MySensors uses all 3 of them.

Nodes sending 3-7 (sometimes more) messages with 35 ms delay between messages, so in my case 3-packet hardware buffer in nrf24 is overflowed after 105-140 milliseconds of write() call.

Also disconnection of client during clients[i].write() call causes reboot by watchdog
I guess this is ESP core related. Is this fixed in code 2.0.0?

I did not tried to check this reboots in 2.0.0, but 2.0.0 is much more stable and I never seen any reboots/resets.

[Yveaux]

@robosensor said:

2.0.0 is much more stable and I never seen any reboots/resets.

That sounds promising! I'll have to give the 2.0.0 core a try myself then.

[patrick schaerer]

Thanks for your work!
I made my wifigateway today and added variables for a static IP Adress.
I also tried to add an DHT11 to the wifi gateway, but it didn't work. I cannot include those sensors because I need to startup while inclusion and this would restart the gateway.
Mabe someone has an Idea how to add a sensor to the wifi gateway. edit: I found this thread on the forum ... http://forum.mysensors.org/topic/1387/sensors-on-gateway

/**
 * The MySensors Arduino library handles the wireless radio link and protocol
 * between your home built sensors/actuators and HA controller of choice.
 * The sensors forms a self healing radio network with optional repeaters. Each
 * repeater and gateway builds a routing tables in EEPROM which keeps track of the
 * network topology allowing messages to be routed to nodes.
 *
 * Created by Henrik Ekblad <henrik.ekblad@mysensors.org>
 * Copyright (C) 2013-2015 Sensnology AB
 * Full contributor list: https://github.com/mysensors/Arduino/graphs/contributors
 *
 * Documentation: http://www.mysensors.org
 * Support Forum: http://forum.mysensors.org
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * REVISION HISTORY
 * Version 1.0 - Henrik EKblad
 * Contribution by a-lurker and Anticimex, 
 * Contribution by Norbert Truchsess <norbert.truchsess@t-online.de>
 * Contribution by Ivo Pullens (ESP8266 support)
 * 
 * DESCRIPTION
 * The EthernetGateway sends data received from sensors to the WiFi link. 
 * The gateway also accepts input on ethernet interface, which is then sent out to the radio network.
 *
 * VERA CONFIGURATION:
 * Enter "ip-number:port" in the ip-field of the Arduino GW device. This will temporarily override any serial configuration for the Vera plugin. 
 * E.g. If you want to use the defualt values in this sketch enter: 192.168.178.66:5003
 *
 * LED purposes:
 * - To use the feature, uncomment WITH_LEDS_BLINKING in MyConfig.h
 * - RX (green) - blink fast on radio message recieved. In inclusion mode will blink fast only on presentation recieved
 * - TX (yellow) - blink fast on radio message transmitted. In inclusion mode will blink slowly
 * - ERR (red) - fast blink on error during transmission error or recieve crc error  
 * 
 * See http://www.mysensors.org/build/ethernet_gateway for wiring instructions.
 * The ESP8266 however requires different wiring:
 * nRF24L01+  ESP8266
 * VCC        VCC
 * CE         GPIO4          
 * CSN/CS     GPIO15
 * SCK        GPIO14
 * MISO       GPIO12
 * MOSI       GPIO13
 *            
 * Not all ESP8266 modules have all pins available on their external interface.
 * This code has been tested on an ESP-12 module.
 * The ESP8266 requires a certain pin configuration to download code, and another one to run code:
 * - Connect REST (reset) via 10K pullup resistor to VCC, and via switch to GND ('reset switch')
 * - Connect GPIO15 via 10K pulldown resistor to GND
 * - Connect CH_PD via 10K resistor to VCC
 * - Connect GPIO2 via 10K resistor to VCC
 * - Connect GPIO0 via 10K resistor to VCC, and via switch to GND ('bootload switch')
 * 
  * Inclusion mode button:
 * - Connect GPIO5 via switch to GND ('inclusion switch')
 * 
 * Hardware SHA204 signing is currently not supported!
 *
 * Make sure to fill in your ssid and WiFi password below for ssid & pass.
 */
#define NO_PORTB_PINCHANGES 

#include <SPI.h>  

#include <MySigningNone.h> 
#include <MySigningAtsha204Soft.h>
#include <MyTransportNRF24.h>
#include <MyTransportRFM69.h>
#include <EEPROM.h>
#include <MyHwESP8266.h>
#include <ESP8266WiFi.h>

#include <MyParserSerial.h>  
#include <MySensor.h>  
#include <stdarg.h>
#include "GatewayUtil.h"


const char *ssid =  "yourSSID";    // cannot be longer than 32 characters!
const char *pass =  "yourPassword"; //
IPAddress local_ip(192,168,0,6);
IPAddress dns_address(192,168,0,1);
IPAddress gateway_ip(192,168,0,1);
IPAddress subnet(255,255,255,0);

//Code for DHT Temp and Hum
/*#include <DHT.h>  

#define CHILD_ID_HUM 9
#define CHILD_ID_TEMP 10
#define HUMIDITY_SENSOR_DIGITAL_PIN 1
long intervall = 300000; //read Temp and Hum every 5 minutes
long last_millis = -300000;
DHT dht;
float lastTemp;
float lastHum;
boolean metric = true; 
MyMessage msgHum(CHILD_ID_HUM, V_HUM);
MyMessage msgTemp(CHILD_ID_TEMP, V_TEMP);
//End of Temp and Hum */

#define INCLUSION_MODE_TIME 1 // Number of minutes inclusion mode is enabled
#define INCLUSION_MODE_PIN  5 // Digital pin used for inclusion mode button

#define RADIO_CE_PIN        4   // radio chip enable
#define RADIO_SPI_SS_PIN    15  // radio SPI serial select

#ifdef WITH_LEDS_BLINKING
#define RADIO_ERROR_LED_PIN 7  // Error led pin
#define RADIO_RX_LED_PIN    8  // Receive led pin
#define RADIO_TX_LED_PIN    9  // the PCB, on board LED
#endif


// NRFRF24L01 radio driver (set low transmit power by default) 
MyTransportNRF24 transport(RADIO_CE_PIN, RADIO_SPI_SS_PIN, RF24_PA_LEVEL_GW);
//MyTransportRFM69 transport;


// Message signing driver (signer needed if MY_SIGNING_FEATURE is turned on in MyConfig.h)
#ifdef MY_SIGNING_FEATURE
MySigningNone signer;
//MySigningAtsha204Soft signer;
#endif

// Hardware profile 
MyHwESP8266 hw;

// Construct MySensors library (signer needed if MY_SIGNING_FEATURE is turned on in MyConfig.h)
// To use LEDs blinking, uncomment WITH_LEDS_BLINKING in MyConfig.h
MySensor gw(transport, hw
#ifdef MY_SIGNING_FEATURE
    , signer
#endif
#ifdef WITH_LEDS_BLINKING
  , RADIO_RX_LED_PIN, RADIO_TX_LED_PIN, RADIO_ERROR_LED_PIN
#endif
  );
  

#define IP_PORT 5003         // The port you want to open 
#define MAX_SRV_CLIENTS 5    // how many clients should be able to telnet to this ESP8266

// a R/W server on the port
static WiFiServer server(IP_PORT);
static WiFiClient clients[MAX_SRV_CLIENTS];
static bool clientsConnected[MAX_SRV_CLIENTS];
static inputBuffer inputString[MAX_SRV_CLIENTS];

#define ARRAY_SIZE(x)  (sizeof(x)/sizeof(x[0]))


void output(const char *fmt, ... )
{
  char serialBuffer[MAX_SEND_LENGTH];
  va_list args;
  va_start (args, fmt );
  vsnprintf_P(serialBuffer, MAX_SEND_LENGTH, fmt, args);
  va_end (args);
  Serial.print(serialBuffer);
  for (uint8_t i = 0; i < ARRAY_SIZE(clients); i++)
  {
    if (clients[i] && clients[i].connected())
    {
//       Serial.print("Client "); Serial.print(i); Serial.println(" write");
       clients[i].write((uint8_t*)serialBuffer, strlen(serialBuffer));
    }
  }
}

void setup()  
{ 
  // Setup console
  hw_init();

  Serial.println(); Serial.println();
  Serial.println("ESP8266 MySensors Gateway");
  Serial.print("Connecting to "); Serial.println(ssid);

  (void)WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }
  Serial.println("Connected!");
  Serial.print("IP: "); Serial.println(WiFi.localIP());
  Serial.flush();
  
  (void)WiFi.config(local_ip, dns_address, gateway_ip, subnet);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }
  Serial.println("Connected!");
  Serial.print("IP: "); Serial.println(WiFi.localIP());
  Serial.flush();
  
  
  setupGateway(INCLUSION_MODE_PIN, INCLUSION_MODE_TIME, output);

  // Initialize gateway at maximum PA level, channel 70 and callback for write operations 
  gw.begin(incomingMessage, 0, true, 0);
  
  // start listening for clients
  server.begin();
  server.setNoDelay(true);  
  
 /* //Code for Temp and Hum
  dht.setup(HUMIDITY_SENSOR_DIGITAL_PIN); 

  // Register all sensors to gw (they will be created as child devices)
  gw.present(CHILD_ID_HUM, S_HUM);
  gw.present(CHILD_ID_TEMP, S_TEMP);
  
  metric = gw.getConfig().isMetric;
  //End of Temp and Hum */
}


void loop() {
  gw.process();  
  
  checkButtonTriggeredInclusion();
  checkInclusionFinished();

  // Go over list of clients and stop any that are no longer connected.
  // If the server has a new client connection it will be assigned to a free slot.
  bool allSlotsOccupied = true;
  for (uint8_t i = 0; i < ARRAY_SIZE(clients); i++)
  {
    if (!clients[i].connected())
    {
      if (clientsConnected[i])
      {
        Serial.print("Client "); Serial.print(i); Serial.println(" disconnected");
        clients[i].stop();
      }
      //check if there are any new clients
      if (server.hasClient())
      {
        clients[i] = server.available();
        inputString[i].idx = 0;
        Serial.print("Client "); Serial.print(i); Serial.println(" connected"); 
        output(PSTR("0;0;%d;0;%d;Gateway startup complete.\n"),  C_INTERNAL, I_GATEWAY_READY);
      }
    }
    bool connected = clients[i].connected();
    clientsConnected[i] = connected;
    allSlotsOccupied &= connected;
  }
  if (allSlotsOccupied && server.hasClient())
  {
    //no free/disconnected spot so reject
    Serial.println("No free slot available");
    WiFiClient c = server.available();
    c.stop();
  }
  
  // Loop over clients connect and read available data
  for (uint8_t i = 0; i < ARRAY_SIZE(clients); i++)
  {
    while(clients[i].connected() && clients[i].available())
    {
      char inChar = clients[i].read();
      if ( inputString[i].idx < MAX_RECEIVE_LENGTH - 1 )
      { 
        // if newline then command is complete
        if (inChar == '\n')
        {  
          // a command was issued by the client
          // we will now try to send it to the actuator
          inputString[i].string[inputString[i].idx] = 0;
    
          // echo the string to the serial port
          Serial.print("Client "); Serial.print(i); Serial.print(": "); Serial.println(inputString[i].string);
    
          parseAndSend(gw, inputString[i].string);
    
          // clear the string:
          inputString[i].idx = 0;
          // Finished with this client's message. Next loop() we'll see if there's more to read.
          break;
        } else {  
         // add it to the inputString:
         inputString[i].string[inputString[i].idx++] = inChar;
        }
      } else {
        // Incoming message too long. Throw away 
        Serial.print("Client "); Serial.print(i); Serial.println(": Message too long");
        inputString[i].idx = 0;
        // Finished with this client's message. Next loop() we'll see if there's more to read.
        break;
      }
    }
  }
 /* //Code for Temp and Hum
  if (millis() >= last_millis + intervall) {
    float temperature = dht.getTemperature();
    if (isnan(temperature)) {
      Serial.println("Failed reading temperature from DHT");
    } else if (temperature != lastTemp) {
      lastTemp = temperature;
    if (!metric) {
      temperature = dht.toFahrenheit(temperature);
    }
    gw.send(msgTemp.set(temperature, 1));
    Serial.print("T: ");
    Serial.println(temperature);
    }
  
  float humidity = dht.getHumidity();
  if (isnan(humidity)) {
      Serial.println("Failed reading humidity from DHT");
  } else if (humidity != lastHum) {
      lastHum = humidity;
      gw.send(msgHum.set(humidity, 1));
      Serial.print("H: ");
      Serial.println(humidity);
  }
  last_millis = millis();
 }
  
 //End of Temp and Hum */
  
  
  
}

[Alex B Goode]

Any chance to achieve some success using ESP-01 or I really have to buy ESP-07/12?

I just have some ESP-01 modules already and I wonder if I can use them

[Yveaux]

@Alex-B-Goode the Spi bus is not available on the esp01 header, so you won't be able to connect the radio then...

[hek]

But I guess you could use it as a sensor node (mqtt/ethernet).

[sdndba]

@Yveaux said:

@krajcl Does it only crash when sending this particular message?

It's late (early morning) I did not want to read over all the last three months posts so my question... is there a solution for this problem? I just downloaded the latest library and found the NodeMCU will reboot when a node initializes. Data below. Just point me to the solution if you would please. Thanks very much.

0;0;3;0;9;read: 3-3-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.54
3;1;1;0;0;21.54
0;0;3;0;9;read: 3-3-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
3;2;1;0;38;3.44
0;0;3;0;9;read: 3-3-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.76
3;1;1;0;0;21.76
0;0;3;0;9;read: 3-3-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
3;2;1;0;38;3.44
0;0;3;0;9;read: 1-1-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.11
1;1;1;0;0;21.11
0;0;3;0;9;read: 1-1-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
1;2;1;0;38;3.44
0;0;3;0;9;read: 3-3-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.76
3;1;1;0;0;21.76
0;0;3;0;9;read: 3-3-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
3;2;1;0;38;3.44
0;0;3;0;9;read: 0-0-0 s=255,c=0,t=17,pt=0,l=5,sg=0:1.5.1
0;255;0;0;17;1.5.1
0;0;3;0;9;read: 0-0-0 s=255,c=3,t=6,pt=1,l=1,sg=0:0

Exception (28):
epc1=0x40203911 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: cont
sp: 3ffefe50 end: 3fff00e0 offset: 01a0

stack>>>
3ffefff0: 3ffeef14 3ffeee8b 3ffeee84 402037d5
3fff0000: 000000ff 00000003 00000006 00000001
3fff0010: 00000001 00000000 3ffeeed8 00000030
3fff0020: 00000000 3fffdc20 3ffef0ac 00000030
3fff0030: 00000003 00000001 00000001 00000000
3fff0040: 00000000 00000000 000000ff 00000023
3fff0050: 00000006 40209ee1 3ffeee64 3ffeee64
3fff0060: 00000000 00000000 3ffeee84 3ffeee84
3fff0070: 3fffdc20 00000000 3ffef0a4 40202582
3fff0080: 402020cc 3ffeee64 3fff1720 402051e6
3fff0090: 00000000 00ffffff 0100000a 3ffef0ac
3fff00a0: 00000000 00000000 00000016 40101941
3fff00b0: 40205499 1100000a 00000000 3ffef0ac
3fff00c0: 3fffdc20 00000000 3ffef0a4 402054c1
3fff00d0: 00000000 00000000 3ffef0c0 40100114
<<<stack<<<

ets Jan 8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld

ESP8266 MySensors Gateway
Connecting to TESTNETWORK
....Connected!
IP: 10.0.0.17
0;0;3;0;9;gateway started, id=0, parent=0, distance=0
0;0;3;0;9;read: 0-0-0 s=6,c=0,t=13,pt=0,l=0,sg=0:
0;6;0;0;13;
0;0;3;0;9;read: 0-0-0 s=1,c=1,t=16,pt=2,l=2,sg=0:0
0;1;1;0;16;0
0;0;3;0;9;read: 0-0-0 s=4,c=1,t=2,pt=2,l=2,sg=0:0
0;4;1;0;2;0
0;0;3;0;9;read: 0-0-0 s=5,c=1,t=2,pt=2,l=2,sg=0:0
0;5;1;0;2;0
0;0;3;0;9;read: 3-3-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.76
3;1;1;0;0;21.76
0;0;3;0;9;read: 3-3-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
3;2;1;0;38;3.44
0;0;3;0;9;read: 3-3-0 s=1,c=1,t=0,pt=7,l=5,sg=0:21.97
3;1;1;0;0;21.97
0;0;3;0;9;read: 3-3-0 s=2,c=1,t=38,pt=7,l=5,sg=0:3.44
3;2;1;0;38;3.44

[emc2]

@Yveaux
Is there any specific reason on why CE is plugged on D2?
I'm asking as I want to add a BME280 sensor directly on my gateway and by default the I2C bus is on D2/D1. It can be changed to any pin with Wire.begin(SDA,SCL) so it is not a major problem, but for user friendliness it would be nicer id CE was plugged on D9 or eventually D4 or D3.

If there is no specific reason can I change MyConfig.h line 329 to #define MY_RF24_CE_PIN 3 safely?

Thanks again for this gateway!

[Yveaux]

@emc2 said:

Is there any specific reason on why CE is plugged on D2?

Not that I know of.

[emc2]

Switching it to D9 and adding #define MY_RF24_CE_PIN 3 directly to the program worked indeed.

Now everything seems to work either by moving the I2C bus or the CE. It may be nice to officially free up the I2C port on the next versions of the library by default? It could be easier than to modify each I2C sensor or even display libraries individually.

I will let it run a little while and if I don't see any major problem I will try to put the code in a new topic. Thanks for your fast answer @Yveaux !

[rapzak]

Hi,

I have added som support, where the gateway can upload data to emoncms, transparant to other functions (not fully testet):

in gatewayutil add this in top of file, after the ARDUINO check:

#define EMONCMS

#ifdef EMONCMS
static WiFiClient emon_client;
const char* emon_host = "emoncms.org";  
const int emon_httpPort = 80; 
const int emon_addr_offset = 10;
#endif

---

Change this function to:

void incomingMessage(const MyMessage &message) {
//  if (mGetCommand(message) == C_PRESENTATION && inclusionMode) {
//	gw.rxBlink(3);
//   } else {
//	gw.rxBlink(1);
//   }
   // Pass along the message from sensors to serial line
   serial(PSTR("%d;%d;%d;%d;%d;%s\n"),message.sender, message.sensor, mGetCommand(message), mGetAck(message), message.type, message.getString(convBuf));
#ifdef EMONCMS
     if (!emon_client.connect(emon_host, emon_httpPort)) {
      Serial.println("emoncms connection failed");
      }

      emon_client.print(String("GET /emoncms/input/post.json?&node="+ String(message.sender+emon_addr_offset)+"&json={"+String(message.sensor)+":"+message.getString(convBuf)+"}&apikey=XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n"));
      Serial.println(String("GET /emoncms/input/post.json?&node="+ String(message.sender+emon_addr_offset)+"&json={"+String(message.sensor)+":"+message.getString(convBuf)+"}&apikey=XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n"));
#endif
} 

It will upload data with NODE ID and Sensor ID from the network, with an offset of emon_addr_offset...

/Rapzak

[scalz]

hi.

Has someone already thought about using websockets with wifiGW ??? before I start to reinvent the wheel!
I have already made some tests with websockets but it was not with mysensors. it was not big tests too but I was able to have an html5 client with js. it was a simple test like blink led+serial monitor, receive/sending serial things. very simple!
I did this because I would like to have sort of display on my phone even if there are no controller. and I don't want to use spiffs and webserver. I prefer to have the stuff on client side and esp8266 acting as a simple websocket server. More lightweight I think, to handle something more complex if needed...For the client, then it could be html+boostrap or something more hybrid. I am not expert at this but I should be able to make some poc..but I don't want to waste my time if it already exists! I have lot of other projects so I don't know how long it could take, and maybe someone is faster on this, so it depends..but it's a feature I would like to have in future and I will make it when needed

So what do you think ??? dumb idea?

[hek]

Yeah, thought about it the other day and found this.

https://github.com/Links2004/arduinoWebSockets

Would have been fun to create something cloud.mysensors.org where people could share sensor data (and of course make it embeddable on the forum). Creating a websocket gateway-variant would probably be my first choice if I decide going down that path.

It also seems to support wss which is kind of nice.

[scalz]

@hek : cool so you think it could be useful too.
yes I use this lib, nice features and simple to use.
so, even if someone has already done something, maybe I will dig a little for fun, time to time...

[hek]

I haven't done anything yet. The old head is just constantly spinning ideas you know

So please go ahead and see if you can create a MyGatewayTransportWebsocket.cpp

[ErrK]

@scalz @hek you can look at socket.io if you are thinking of using websockets.

[hek]

@ErrK said:

socket.io

Yep, use it on my dayjob. Allows downgrade to polling which (I guess) the Arduino library doesn't support. Might be better to use raw websocket on the server side in this case.

[scalz]

@ErrK : yes I saw it too. but here a discussion about socket.io with esp websockets https://github.com/Links2004/arduinoWebSockets/issues/42
very interesting

[Yveaux]

@scalz @hek a local server on the ESP serving gateway status overview (connected clients, routing table, sensors presented by clients, data etc) would be a very nice addition to any gateway IMHO.
The ESP has enough power/flash to implement this.

[ErrK]

@hek: Yes, it has fallbacks to use other protocols. I think there is a setting if you don't want to use all the default protocols.
@scalz: interesting discussion. It looks like the best way shuld be to only use the websockets

[scalz]

@Yveaux: exactly what I was thinking for monitoring and it could be fun to be able to show something on the phone without a controller (for instance when I visit my not geek friends, it looks more simple if something on the phone instead of setting up a controller).
I started to look at dev branch last night but was too tired.... I visualize where are the changes to do, but maybe not all, I will ask you if I have some problems A cool thing in websocket lib is broadcasting client is already implemented in it.

[A R]

Hi,
I am working on a sensor system. Sensors feed to ESP8266 which feed to NodeMCU ESP8266. I saw in the intro to GatewayESP8266.ino on GitHub your code which says:

• The EthernetGateway sends data received from sensors to the WiFi link.
• The gateway also accepts input on ethernet interface, which is then sent out to the radio network.

Does that mean Ethernet Gateway receiving data from WiFi?
Thanking you in anticipation.
Riaz Ahmed

[Yveaux]

@A-R said:

Does that mean Ethernet Gateway receiving data from WiFi?

As the topic indicates this is a WiFi gateway for mysensors. That means that it converts sensor data from the MySensors network to WiFi and vice versa.
When sending data to a sensor, a controller will send data over WiFi to the gateway, which then sends the data on to a MySensors sensor (using nRF24L01+ or RFM69HW radio).
Seen from the controller, it will communicate to the WiFi gateway exactly the same way as it would when connecting to a wired ethernet gateway.

[A R]

@Yveaux Thanks for the reply. Let me elaborate please. My sensor sends data to ESP8266 which sends it to NodeMCU ESP8266. From there I want the NodeMCU ESP8266 to send data through a wired Ethernet. (from there i want it be sent to a router).

So my question is can the GatewayESP8266.ino be used to send data from NodeMCU to Ethernet?

[Yveaux]

@A-R said:

can the GatewayESP8266.ino be used to send data from NodeMCU to Ethernet?

How do you plan to connect the wired ethernet to the NodeMCU board?
You could probably connect a W5100 module through SPI, but why bother?

[A R]

@Yveaux I am using W5100. Its a system requirement. I want wifi connectivity with sensors through ESPs but the final data is to be routed through ethernet for security.

[Yveaux]

@A-R But do you use any MySensors parts then?

[A R]

@Yveaux No. I asked a simple question: can the GatewayESP8266.ino be used to send data from NodeMCU to Ethernet? If you have an answer then do reply.

[Yveaux]

@A-R No need to get all grumpy! I'm just trying to understand what you want to achieve, and help you out.
You are posting to a MySensors forum, so I assumed you are using MySensors software in your project.
Answer to your question: No.

[scalz]

@A-R : it depends of what you are connecting to esp8266, you didn't give details. esp8266 lacks of io, so it depends...I think you have not googled?? esp8266 w5100 for instance, you will find things...then if it is to use radio+w5100 with esp8266, sure it's possible but depends of available io...software spi you would have to double io, hardware spi, need manage your cs lines...some work. If it's just esp8266+w5100 there are already things on google...

[Robban]

@rapzak said:

Hi,

I have added som support, where the gateway can upload data to emoncms, transparant to other functions (not fully testet):

in gatewayutil add this in top of file, after the ARDUINO check:

#define EMONCMS

#ifdef EMONCMS
static WiFiClient emon_client;
const char* emon_host = "emoncms.org";  
const int emon_httpPort = 80; 
const int emon_addr_offset = 10;
#endif

---

Change this function to:

void incomingMessage(const MyMessage &message) {
//  if (mGetCommand(message) == C_PRESENTATION && inclusionMode) {
//	gw.rxBlink(3);
//   } else {
//	gw.rxBlink(1);
//   }
   // Pass along the message from sensors to serial line
   serial(PSTR("%d;%d;%d;%d;%d;%s\n"),message.sender, message.sensor, mGetCommand(message), mGetAck(message), message.type, message.getString(convBuf));
#ifdef EMONCMS
     if (!emon_client.connect(emon_host, emon_httpPort)) {
      Serial.println("emoncms connection failed");
      }

      emon_client.print(String("GET /emoncms/input/post.json?&node="+ String(message.sender+emon_addr_offset)+"&json={"+String(message.sensor)+":"+message.getString(convBuf)+"}&apikey=XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n"));
      Serial.println(String("GET /emoncms/input/post.json?&node="+ String(message.sender+emon_addr_offset)+"&json={"+String(message.sensor)+":"+message.getString(convBuf)+"}&apikey=XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n"));
#endif
} 

It will upload data with NODE ID and Sensor ID from the network, with an offset of emon_addr_offset...

/Rapzak

Are there any one that can help me implement this in the dev version? Have a nodemcu 1-0 and got errors uploading the 1.5 version so i cant use that. Tried to fix it but ended up with the dev version.

[bkenobi]

Can anyone provide the correct wiring for NodeMCU (ESP-12E) to RFM69HW? I have seen several different guides that show connections for RFM69, but there are several different patterns and none have worked so far. Perhaps I didn't get it right, but this is what I've seen and tried:

someburner/esp-rfm69

RFM69->	ESP-12E
MISO	GPIO12
MOSI	GPIO13
SCK		GPIO14
CS/SS	GPIO15
DIO0	GPIO5
VCC		3V3
GND		GND
ANA		<antenna>

halburd/NodeMCU-Gateway

RFM69->	NodeMCU
NSS		GPIO2 (J2P5)
SCK		SCK (J1P9)
MISO	MISO (J1P8)
MOSI	MOSI (J1P6)
DIO0	SS (J1P7)
VCC		3V3
GND		GND
ANA		<antenna>

I've also determined that the following should also be correct, though again, they all don't match.

RFM69	ESP-12E	WEMOS D1	NodeMCUv3
MISO	GPIO12	D6/D12		D6
MOSI	GPIO13	D7/D11		D7
SCK		GPIO14	D5/D13		D5
CS/SS	GPIO15	D10			D8
DIO0	GPIO5	D3/D15		D1

I've also tried to figure out how to use the nRF24L0+ wiring, but that doesn't quite match either. If someone could simply post a working config, I could get some traction on my project.

[bkenobi]

Looking at the different charts, I've located some errors (I think).

Looking at someburner's setup, it looks like he is opting to use software SPI rather than hardware as the ESP pins for SPI are not on GPIO12-15.

Looking at halburd's code, it appears that he's opted to use GPIO2 for slave select and connect the actual SS to DIO0. I'm not clear why this is done, but it seems odd.

I lost the reference to the last chart. It also uses GPIO's for SPI rather than the built-in hardware SPI. The guide in this sketch is only documented for nRF24L01+, so it doesn't match exactly. But, it also uses GPIO 4, 12, 13, 14, 15. It seems odd to use those rather than the SPI pins.

[Mark Swift]

I tried to get the ESP8266 gateway working for 2 weeks but kept getting st=fail at various times. Switching to a serial gateway fixed it straight away? Is this a known issue? Is it perhaps related to having the ESP8266 and the nRF24L01's so close together? BTW, I tied both the normal gateway and the MQTT gateway without much success. The who time I thought it was a radio issue

This was all running on a NodeMCU v1.0.

Mark

[Yveaux]

@Mark-Swift Did you follow the connection in the ESP build guide?
Did you make any changes to the sketch? Which MySensors version did you use and which ESP Arduino version?
Just asking all these questions so we can be of better help.

[Mark Swift]

@Yveaux I believe so, the nNF24L01 is connected to a NodeMCU (I've tried feeding the radio off both the onboard 3.3v and via VIN and a 1117 3.3v regulator (I have caps across the radio GND / VCC).

No changes were made to the sketch - I'm using the latest development build (also on the nodes). I've tried with all of the ESP Arduino board versions, doesn't seem to change anything?

[Yveaux]

@Mark-Swift said:

kept getting st=fail at various times

Now I read over your post again I get the impression that it sometimes works, and sometimes you get st=fail. Is this correct?

[Mark Swift]

Example - I'm sat here right now with one of my nodes and the gateway connected to my Macbook via USB.

When using the serial gateway I get =ok after each item no problem; in fact I can't make it fail.

When using the ESP8266 MQTT gateway I get:

Starting repeater (RNNRA-, 2.0.0-beta)
Radio init successful.
send: 3-3-0-0 s=255,c=3,t=11,pt=0,l=13,sg=0,st=ok:R+M+L+T+D+Rpt
send: 3-3-0-0 s=255,c=3,t=12,pt=0,l=3,sg=0,st=ok:1.1
send: 3-3-0-0 s=1,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=2,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=3,c=0,t=16,pt=0,l=0,sg=0,st=fail:
send: 3-3-0-0 s=4,c=0,t=15,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=10,c=0,t=1,pt=0,l=0,sg=0,st=fail:
send: 3-3-0-0 s=11,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=12,c=0,t=23,pt=0,l=0,sg=0,st=fail:
send: 3-3-0-0 s=255,c=3,t=15,pt=0,l=2,sg=0,st=ok:

send: 3-3-0-0 s=255,c=0,t=18,pt=0,l=10,sg=0,st=fail:2.0.0-beta
send: 3-3-0-0 s=255,c=3,t=6,pt=1,l=1,sg=0,st=ok:0
Init complete, id=3, parent=0, distance=1
Moisture Sensor: 0
send: 3-3-0-0 s=1,c=1,t=16,pt=2,l=2,sg=0,st=ok:0
Rain Sensor: 0
send: 3-3-0-0 s=2,c=1,t=16,pt=2,l=2,sg=0,st=ok:0
LUX: 54612
send: 3-3-0-0 s=3,c=1,t=37,pt=3,l=2,sg=0,st=ok:54612
Distance: 0 cm
send: 3-3-0-0 s=4,c=1,t=13,pt=2,l=2,sg=0,st=ok:0
No rain or moisture detected, landroid is not waiting: Status(0)
Landroid is free to go and is now waiting on the schedule: Status(0)
Landroid is home charging!
Sending landroidHome (1) status to gateway: send: 3-3-0-0 s=10,c=1,t=16,pt=1,l=1,sg=0,st=ok:1
Sending landroidWaitingTriggered (0) status to gateway: send: 3-3-0-0 s=11,c=1,t=16,pt=1,l=1,sg=0,st=ok:0
Sending timeElapsed (0) status to gateway: send: 3-3-0-0 s=12,c=1,t=48,pt=2,l=2,sg=0,st=fail:0

Using the serial gateway on a nano (Not connected to anything, just the nano and nRF24L01) I get:

Starting repeater (RNNRA-, 2.0.0-beta)
Radio init successful.
send: 3-3-0-0 s=255,c=3,t=11,pt=0,l=13,sg=0,st=ok:R+M+L+T+D+Rpt
send: 3-3-0-0 s=255,c=3,t=12,pt=0,l=3,sg=0,st=ok:1.1
send: 3-3-0-0 s=1,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=2,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=3,c=0,t=16,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=4,c=0,t=15,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=10,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=11,c=0,t=1,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=12,c=0,t=23,pt=0,l=0,sg=0,st=ok:
send: 3-3-0-0 s=255,c=3,t=15,pt=0,l=2,sg=0,st=ok:

send: 3-3-0-0 s=255,c=0,t=18,pt=0,l=10,sg=0,st=ok:2.0.0-beta
send: 3-3-0-0 s=255,c=3,t=6,pt=1,l=1,sg=0,st=ok:0
Init complete, id=3, parent=0, distance=1
Moisture Sensor: 0
send: 3-3-0-0 s=1,c=1,t=16,pt=2,l=2,sg=0,st=ok:0
Rain Sensor: 0
send: 3-3-0-0 s=2,c=1,t=16,pt=2,l=2,sg=0,st=ok:0
LUX: 54612
send: 3-3-0-0 s=3,c=1,t=37,pt=3,l=2,sg=0,st=ok:54612
Distance: 69 cm
send: 3-3-0-0 s=4,c=1,t=13,pt=2,l=2,sg=0,st=ok:69
No rain or moisture detected, landroid is not waiting: Status(0)
Landroid is free to go and is now waiting on the schedule: Status(0)
Landroid is out cutting the grass!
Sending landroidHome (0) status to gateway: send: 3-3-0-0 s=10,c=1,t=16,pt=1,l=1,sg=0,st=ok:0
Sending landroidWaitingTriggered (0) status to gateway: send: 3-3-0-0 s=11,c=1,t=16,pt=1,l=1,sg=0,st=ok:0
Sending timeElapsed (0) status to gateway: send: 3-3-0-0 s=12,c=1,t=48,pt=2,l=2,sg=0,st=ok:0

[Yveaux]

@Mark-Swift It does work sometimes, so that rules out a lot of possible problems!
How do you power the NodeMCU & rest of the hardware?
Both NodeMCU & nRF are very sensitive to low/noisy supplies.

[Mark Swift]

@Yveaux I power the NodeMCU from either my Macbook Retina, or from a decent USB wall plug, tbh, it seems to make little difference which I use. I presume everyone else is powering them via the USB also?

I've also tried a spare NodeMCU and a bunch of radios. In fact, I've been trying to get this working for 3 weeks, it's literally going to cause a divorce soon

[Yveaux]

@Mark-Swift The log mentions Moisture Sensor, Rain Sensor, LUX, Distance.
Are these real sensor connected (and yes, how) or virtual?

[Mark Swift]

@Yveaux real sensors connected to my robot mower garage node...

[Mark Swift]

@Yveaux Node code is here for reference:

http://pastebin.com/mxkLq1Rp

[Yveaux]

@Mark-Swift To be frank: I don't know.
I'm quite sure it has nothing to do with the ESP port, or software in general.
You could try different orientation of the nRF, or move it closer to the sensor(s).

Are you using an amplified nRF24 btw?

[Mark Swift]

@Yveaux That's a shame, I'd really like to solve this. I've tried the radios orientated different ways but it makes little difference. Not using a PA+LNB, I was but gave up on it as I thought it was the issue (Now realise it wasn't).

Strange that I only see these issues with the ESP, and as soon as I switch to a Nano running the serial gateway they're gone.

[Yveaux]

@Mark-Swift I always use my sniffer in this case to see what's going on on air, but I don't know if your marriage will survive another sniffer build

[Mark Swift]

@Yveaux I doubt it will, seems I'm destined to build a serial gateway, crap, I've invested so much time in this and tried everything!

[Mark Swift]

@Yveaux How about this, for a laugh I just tried rolling the ESP8266 gateway back to v1.5 (master).

Guess what? It works every time with no fails (communicating with my 2.0.0 beta node). So the issue lies with the latest development version somewhere, any ideas?
~~
Insane, I just took the gateway down the other end of the house and not one fail, wow! So okay, what was changed in version 2.0.0?

UPDATE: After more investigation, the reason it was working is because v1.5 only has a straight forward gateway version, and until now I've been using the MQTT client. It seems the gateway versions work fine, but the MQTT client version create the fails!

UPDATE 2: Okay, I can reproduce the fails with the gateway too, let me try and explain. When the gateway is not connected to anything (for example Domoticz) I get OK all of the time, no fails whatsoever. If I connect to Domoticz, I start to get fails against certain child sensors (it's always the same!). I presume it's sketch related, or even the amount of node updates I'm sending? As I believe the fails are the same I'm seeing with the MQTT client.

UPDATE 3: Nearly 3 weeks in and I've just figured out what has been causing me all this pain. I've always commented out the default 9600 serial baud define on the ESP8266 sketches. It seems this combined with the 200ms TCP delay has been causing me my issues when connecting to my Windows 7 machine. S$%&*!

[Yveaux]

@Mark-Swift Well, look on it from the bright side: your marriage is saved now :bowtie:
There have been more reports from issues with connections to external servers (e.g. MQTT, Domotiocz) that take too long.
Maybe we should pull @hek in as he wrote the MQTT client for ESP.

[hek]

What adjustments do you propose to the MQTT sketch/driver?

[Yveaux]

@hek I'm not familiar with the exact implementation at the moment. Just hoped you would have a hunch where these issues might arise from or how we can prevent them...

[hek]

Nope, I don't get why baud rate would affect radio/tcp traffic. Maybe @Mark-Swift could shed some light on it.

[robosensor]

ESP8266 gateway is quite unusable in setups where TCP ACKs can be delayed. This is because thread is blocked until TCP ACK received in ESP wifi code. So all wireless messages will be rejected if NRF24 receives more than 3 messages during this 200ms delay: http://forum.mysensors.org/topic/1870/esp8266-wifi-gateway-port-for-mysensors/235

[hek]

Hmm.. blocking code.. crap.

[robosensor]

The problem is slightly wider than it seems. Wireless messages should be fetched from internal NRF24 FIFO and stored into memory as soon as possible (using interrupts?). But lack of RAM makes the problem difficult to be solved.

The same thing happens not only with ESP8266 gateway and TCP ACKs, but also with all blocking calls during sensor reading (for example for old blocking call to read DS18B20 data with 750ms delay), with improper use of delay() instead of gw.wait() and so on.

[Yveaux]

@robosensor said:

lack of RAM

Not really an issue on ESP I would say.

@hek IMHO it really is time to investigate the impact on the library to support message queueing (fetched & sent from interrupt).
Starting with @tekka rewrite of the nRF driver in 2.0 beta, as it supports SPI transactions.
Probably we can add support incrementally to lower the impact on the library.

[hek]

Yes, at least for the gateway and repeaters this would be preferred. How's your play-time-bandwidth?

[Yveaux]

@hek said:

How's your play-time-bandwidth?

I already feared you we going to ask this
Quite limited at the moment, but such things should grow in my mind first. Let's start with that hehe..

[hek]

Mo ha haha

[scalz]

@hek @Yveaux : I don't know if this could help. but some times ago I found a lib when you were talking about this issue but I have never had enough time to test it. It is a "non blocking" ack with some buffering version for wifiserver. I think that does not solve completely the buffering issue But maybe a small step..
I don't remember where I read this but igr was saying it would be better to have a lib for this than modifiying the wifiserver lib for compatibility with other arduino wifi libs. So maybe this guy did it.
http://www.forward.com.au/pfod/pfodParserLibraries/index.html
In the middle
I also read that for esp, bigger tcp packet were processed faster than small
very tricky issue to debug when so much layers..

[robosensor]

Also it is possible to partially solve this issue by fetching all available messages from NRF24 FIFO and concatenating them into one TCP packet for ESP. It is possible that this will be useful for other gateways too.

@Yveaux said:

IMHO it really is time to investigate the impact on the library to support message queueing (fetched & sent from interrupt).

But no chances that MYS protocol will be changed? I'm talking about reliable delivery and message ids. That topic is very close to this.

[Yveaux]

@robosensor I think message buffering could be the first step to (more) reliable delivery.

[scalz]

this was the thread I read. pfod lib creator and esplibs team were talking about this.
https://github.com/esp8266/Arduino/issues/1430
And it seems there is another lib too for this:
https://github.com/me-no-dev/ESPAsyncTCP

[Yveaux]

@scalz Thanks for the links man!

Now we're at it: anyone knows if a blocked socket connection can be interrupted at all by an external pin interrupt on ESP(Arduino) ? Otherwise buffering on interrupts will make no sense...

[Mark Swift]

Thanks for looking into this guys... I've been doing some tests and basically the ESP gateway is not an option for me right now due to too many radio failures.

Using the normal serial gateway I tested one of my nodes that sends about 15 values during each loop; it was perfect every time!
The same test with the ESP shows failures every few messages or so, introducing a large radio delay between each send fixes it, but the issue can still occur if other nodes call in at the same time.

Another interesting thing, when I don't specify a gateway, or static IP I get no failures, when I do, I get failures, what!?

[Yveaux]

@Mark-Swift Don't know if you already ditched the ESP gateway completely, but you could give my latest experiment a try: https://github.com/Yveaux/Arduino/tree/development
It adds a reception queue to the gateway which will immediately retrieve a message from the radio when it comes in. This process should even continue when the ESP is blocked on some TCP transfer.
Only thing you have to do is connect the IRQ line from the nRF radio to the ESP (e.g. to pin 1), and define the IRQ pin at the top of your gateway sketch:

#define MY_RF24_IRQ_PIN  (1)

I tested this on a UNO, and can easily sleep the gateway for 1 second or more when messages come in at 0.1sec interval -- none gets lost. The buffer is set to 20 by default, but you can increase it when the amount of RAM allows it:

#define MY_RF24_MESSAGE_BUFFER_SIZE  (50)

I don't have a NodeMCU setup ready so I cannot test it on ESP myself...

[Mark Swift]

@Yveaux I'll sure give it a shot when time allows, most likely tomorrow... I've actually worked around the issue by using a regedit hack on my Windows server to remove the 200ms TCP delay. I also fried a Node MCU tonight due to frustration and not concentrating!

Thanks for the efforts, sounds a great solution

BTW, I figured out my main issue was trying to put the nRF24l01+PA+LNA and ESP in a small project box, I guess that also causes issues, especially with the crappy cheap unshielded PA modules (I'm awaiting a proper shielded version).

@robosensor also...

[Mark Swift]

@Yveaux Is there a way I can track dropped messages?

[Yveaux]

@Mark-Swift my queueing code counts how many messages could not be stored due to the queue being full, and therefore got lost. If the radio drops messages because it's FIFO is full we won't know...

[Mark Swift]

@Yveaux I just tried to compile it and get the following error:

In file included from /Users/markswift/Documents/Arduino/libraries/MySensors/core/MyTransportNRF24.cpp:23:0,
from /Users/markswift/Documents/Arduino/libraries/MySensors/MySensor.h:260,
from /Users/markswift/Documents/Personal/Hobbies/Arduino/GatewayESP8266MQTTClient/GatewayESP8266MQTTClient.ino:132:
/Users/markswift/Documents/Arduino/libraries/MySensors/drivers/CircularBuffer/CircularBuffer.h:23:25: fatal error: util/atomic.h: No such file or directory
#include <util/atomic.h>

[Yveaux]

@Mark-Swift Sorry, my bad. This is AVR only code... Have to look for the ESP equivalent...

[Yveaux]

This util/atomic.h is an easy fix, but then the next error occurs: current ESP8266 Arduino port (even tried upto 2.2.0) does not support SPI access from within interrupts (SPI.usingInterrupt is not supported.
See https://github.com/esp8266/Arduino/issues/1943.
This means I cannot reliably get the message data from the nRF from within an interrupt on ESP8266, using the regular Arduino SPI library.
Have to think if there's another solution (or push this issue to be solved )
To be continued...

[Mark Swift]

nightmare!

Let me know how you get on...

[signal15]

Is anyone having luck running the ESP without problems at all? I'm on 1.5 and it's my first gateway, and I cannot get sensors to include. Should I try the development branch? Or, should I build a different gateway altogether? I just need something that works at this point.

[alexsh1]

Let me reiterate what @signal15 asked - did anyone successfully have esp8266 running as the GW on 2.0b? I have just acquired Wemos mini and would like to try it

[Anduril]

@alexsh1 @signal15 I can confirm that the ESP gateway is working fine and stable. I even enabled the ESP OTA update function and that's working fine too.

[signal15]

@Anduril

Can you point me to documentation on how the OTA update stuff works and what the capabilities are?

[Anduril]

@signal15 well there is a doc about OTA here. I discussed this topic with tekka, who is the author of the GatewayESP8266OTA example in the mysensors lib here.
It took me some time to get this working, but now I can upload new firmware to my ESP using wifi. Only drawback is that is seems to be not compatible with the mqtt version of the gateway. At least I was not able to get this working and stopped on that.
If you have further questions, don't hesitate to ask.

[marceloaqno]

I recently built a mysensors node with esp8266 where I needed many I/O pins which made me realize the possibility of using the pin RX (http://www.forward.com.au/pfod/ESP8266/GPIOpins/ESP8266_01_pin_magic.html).
Initializing the serial with:

Serial.begin(MY_BAUD_RATE, SERIAL_8N1, SERIAL_TX_ONLY, 1);

allows to use RX pin for I/O.

So I was wondering if it would be useful to add a way in which the serial port can be initialized:

#define MY_SERIAL_MODE SERIAL_TX_ONLY
#define MY_SERIAL_MODE SERIAL_RX_ONLY
#define MY_SERIAL_MODE SERIAL_FULL   (default)

github.com/marceloaqno/MySensors/commit/687fecc6b4abb782eae8e1abb3b07016bfeac291

Also, to use the esp8266 analog pin, I had to comment the line:

ADC_MODE(ADC_VCC);

from MyHwESP8266.cpp.
Is there another way to use analog readings without messing the code?

Excellent port by the way, I have been using here and it works great!!

[Mark Swift]

Was the blocking code issue ever resolved @Yveaux ?

[Yveaux]

@Mark-Swift no, still open at Esp side, and I couldn't think of a clean way working around it...

[gloob]

I have an ESP using as a gateway and want to send some sensor information each 5 minutes to the controller.

Is it possible to use the wait command in the "loop"?

void loop() 
{
  wait(unsigned long ms);
  statusCounter += 1;
  send(msg.set(statusCounter));
}