ESP8266 WiFi gateway port for MySensors

[Yveaux]

@Mark-Swift The log mentions Moisture Sensor, Rain Sensor, LUX, Distance.
Are these real sensor connected (and yes, how) or virtual?

[Mark Swift]

@Yveaux real sensors connected to my robot mower garage node...

[Mark Swift]

@Yveaux Node code is here for reference:

http://pastebin.com/mxkLq1Rp

[Yveaux]

@Mark-Swift To be frank: I don't know.
I'm quite sure it has nothing to do with the ESP port, or software in general.
You could try different orientation of the nRF, or move it closer to the sensor(s).

Are you using an amplified nRF24 btw?

[Mark Swift]

@Yveaux That's a shame, I'd really like to solve this. I've tried the radios orientated different ways but it makes little difference. Not using a PA+LNB, I was but gave up on it as I thought it was the issue (Now realise it wasn't).

Strange that I only see these issues with the ESP, and as soon as I switch to a Nano running the serial gateway they're gone.

[Yveaux]

@Mark-Swift I always use my sniffer in this case to see what's going on on air, but I don't know if your marriage will survive another sniffer build

[Mark Swift]

@Yveaux I doubt it will, seems I'm destined to build a serial gateway, crap, I've invested so much time in this and tried everything!

[Mark Swift]

@Yveaux How about this, for a laugh I just tried rolling the ESP8266 gateway back to v1.5 (master).

Guess what? It works every time with no fails (communicating with my 2.0.0 beta node). So the issue lies with the latest development version somewhere, any ideas?
~~
Insane, I just took the gateway down the other end of the house and not one fail, wow! So okay, what was changed in version 2.0.0?

UPDATE: After more investigation, the reason it was working is because v1.5 only has a straight forward gateway version, and until now I've been using the MQTT client. It seems the gateway versions work fine, but the MQTT client version create the fails!

UPDATE 2: Okay, I can reproduce the fails with the gateway too, let me try and explain. When the gateway is not connected to anything (for example Domoticz) I get OK all of the time, no fails whatsoever. If I connect to Domoticz, I start to get fails against certain child sensors (it's always the same!). I presume it's sketch related, or even the amount of node updates I'm sending? As I believe the fails are the same I'm seeing with the MQTT client.

UPDATE 3: Nearly 3 weeks in and I've just figured out what has been causing me all this pain. I've always commented out the default 9600 serial baud define on the ESP8266 sketches. It seems this combined with the 200ms TCP delay has been causing me my issues when connecting to my Windows 7 machine. S$%&*!

[Yveaux]

@Mark-Swift Well, look on it from the bright side: your marriage is saved now :bowtie:
There have been more reports from issues with connections to external servers (e.g. MQTT, Domotiocz) that take too long.
Maybe we should pull @hek in as he wrote the MQTT client for ESP.

[hek]

What adjustments do you propose to the MQTT sketch/driver?

[Yveaux]

@hek I'm not familiar with the exact implementation at the moment. Just hoped you would have a hunch where these issues might arise from or how we can prevent them...

[hek]

Nope, I don't get why baud rate would affect radio/tcp traffic. Maybe @Mark-Swift could shed some light on it.

[robosensor]

ESP8266 gateway is quite unusable in setups where TCP ACKs can be delayed. This is because thread is blocked until TCP ACK received in ESP wifi code. So all wireless messages will be rejected if NRF24 receives more than 3 messages during this 200ms delay: http://forum.mysensors.org/topic/1870/esp8266-wifi-gateway-port-for-mysensors/235

[hek]

Hmm.. blocking code.. crap.

[robosensor]

The problem is slightly wider than it seems. Wireless messages should be fetched from internal NRF24 FIFO and stored into memory as soon as possible (using interrupts?). But lack of RAM makes the problem difficult to be solved.

The same thing happens not only with ESP8266 gateway and TCP ACKs, but also with all blocking calls during sensor reading (for example for old blocking call to read DS18B20 data with 750ms delay), with improper use of delay() instead of gw.wait() and so on.

[Yveaux]

@robosensor said:

lack of RAM

Not really an issue on ESP I would say.

@hek IMHO it really is time to investigate the impact on the library to support message queueing (fetched & sent from interrupt).
Starting with @tekka rewrite of the nRF driver in 2.0 beta, as it supports SPI transactions.
Probably we can add support incrementally to lower the impact on the library.

[hek]

Yes, at least for the gateway and repeaters this would be preferred. How's your play-time-bandwidth?

[Yveaux]

@hek said:

How's your play-time-bandwidth?

I already feared you we going to ask this
Quite limited at the moment, but such things should grow in my mind first. Let's start with that hehe..

[hek]

Mo ha haha

[scalz]

@hek @Yveaux : I don't know if this could help. but some times ago I found a lib when you were talking about this issue but I have never had enough time to test it. It is a "non blocking" ack with some buffering version for wifiserver. I think that does not solve completely the buffering issue But maybe a small step..
I don't remember where I read this but igr was saying it would be better to have a lib for this than modifiying the wifiserver lib for compatibility with other arduino wifi libs. So maybe this guy did it.
http://www.forward.com.au/pfod/pfodParserLibraries/index.html
In the middle
I also read that for esp, bigger tcp packet were processed faster than small
very tricky issue to debug when so much layers..

[robosensor]

Also it is possible to partially solve this issue by fetching all available messages from NRF24 FIFO and concatenating them into one TCP packet for ESP. It is possible that this will be useful for other gateways too.

@Yveaux said:

IMHO it really is time to investigate the impact on the library to support message queueing (fetched & sent from interrupt).

But no chances that MYS protocol will be changed? I'm talking about reliable delivery and message ids. That topic is very close to this.

[Yveaux]

@robosensor I think message buffering could be the first step to (more) reliable delivery.

[scalz]

this was the thread I read. pfod lib creator and esplibs team were talking about this.
https://github.com/esp8266/Arduino/issues/1430
And it seems there is another lib too for this:
https://github.com/me-no-dev/ESPAsyncTCP

[Yveaux]

@scalz Thanks for the links man!

Now we're at it: anyone knows if a blocked socket connection can be interrupted at all by an external pin interrupt on ESP(Arduino) ? Otherwise buffering on interrupts will make no sense...

[Mark Swift]

Thanks for looking into this guys... I've been doing some tests and basically the ESP gateway is not an option for me right now due to too many radio failures.

Using the normal serial gateway I tested one of my nodes that sends about 15 values during each loop; it was perfect every time!
The same test with the ESP shows failures every few messages or so, introducing a large radio delay between each send fixes it, but the issue can still occur if other nodes call in at the same time.

Another interesting thing, when I don't specify a gateway, or static IP I get no failures, when I do, I get failures, what!?

[Yveaux]

@Mark-Swift Don't know if you already ditched the ESP gateway completely, but you could give my latest experiment a try: https://github.com/Yveaux/Arduino/tree/development
It adds a reception queue to the gateway which will immediately retrieve a message from the radio when it comes in. This process should even continue when the ESP is blocked on some TCP transfer.
Only thing you have to do is connect the IRQ line from the nRF radio to the ESP (e.g. to pin 1), and define the IRQ pin at the top of your gateway sketch:

#define MY_RF24_IRQ_PIN  (1)

I tested this on a UNO, and can easily sleep the gateway for 1 second or more when messages come in at 0.1sec interval -- none gets lost. The buffer is set to 20 by default, but you can increase it when the amount of RAM allows it:

#define MY_RF24_MESSAGE_BUFFER_SIZE  (50)

I don't have a NodeMCU setup ready so I cannot test it on ESP myself...

[Mark Swift]

@Yveaux I'll sure give it a shot when time allows, most likely tomorrow... I've actually worked around the issue by using a regedit hack on my Windows server to remove the 200ms TCP delay. I also fried a Node MCU tonight due to frustration and not concentrating!

Thanks for the efforts, sounds a great solution

BTW, I figured out my main issue was trying to put the nRF24l01+PA+LNA and ESP in a small project box, I guess that also causes issues, especially with the crappy cheap unshielded PA modules (I'm awaiting a proper shielded version).

@robosensor also...

[Mark Swift]

@Yveaux Is there a way I can track dropped messages?

[Yveaux]

@Mark-Swift my queueing code counts how many messages could not be stored due to the queue being full, and therefore got lost. If the radio drops messages because it's FIFO is full we won't know...

[Mark Swift]

@Yveaux I just tried to compile it and get the following error:

In file included from /Users/markswift/Documents/Arduino/libraries/MySensors/core/MyTransportNRF24.cpp:23:0,
from /Users/markswift/Documents/Arduino/libraries/MySensors/MySensor.h:260,
from /Users/markswift/Documents/Personal/Hobbies/Arduino/GatewayESP8266MQTTClient/GatewayESP8266MQTTClient.ino:132:
/Users/markswift/Documents/Arduino/libraries/MySensors/drivers/CircularBuffer/CircularBuffer.h:23:25: fatal error: util/atomic.h: No such file or directory
#include <util/atomic.h>

[Yveaux]

@Mark-Swift Sorry, my bad. This is AVR only code... Have to look for the ESP equivalent...

[Yveaux]

This util/atomic.h is an easy fix, but then the next error occurs: current ESP8266 Arduino port (even tried upto 2.2.0) does not support SPI access from within interrupts (SPI.usingInterrupt is not supported.
See https://github.com/esp8266/Arduino/issues/1943.
This means I cannot reliably get the message data from the nRF from within an interrupt on ESP8266, using the regular Arduino SPI library.
Have to think if there's another solution (or push this issue to be solved )
To be continued...

[Mark Swift]

nightmare!

Let me know how you get on...

[signal15]

Is anyone having luck running the ESP without problems at all? I'm on 1.5 and it's my first gateway, and I cannot get sensors to include. Should I try the development branch? Or, should I build a different gateway altogether? I just need something that works at this point.

[alexsh1]

Let me reiterate what @signal15 asked - did anyone successfully have esp8266 running as the GW on 2.0b? I have just acquired Wemos mini and would like to try it

[Anduril]

@alexsh1 @signal15 I can confirm that the ESP gateway is working fine and stable. I even enabled the ESP OTA update function and that's working fine too.

[signal15]

@Anduril

Can you point me to documentation on how the OTA update stuff works and what the capabilities are?

[Anduril]

@signal15 well there is a doc about OTA here. I discussed this topic with tekka, who is the author of the GatewayESP8266OTA example in the mysensors lib here.
It took me some time to get this working, but now I can upload new firmware to my ESP using wifi. Only drawback is that is seems to be not compatible with the mqtt version of the gateway. At least I was not able to get this working and stopped on that.
If you have further questions, don't hesitate to ask.

[marceloaqno]

I recently built a mysensors node with esp8266 where I needed many I/O pins which made me realize the possibility of using the pin RX (http://www.forward.com.au/pfod/ESP8266/GPIOpins/ESP8266_01_pin_magic.html).
Initializing the serial with:

Serial.begin(MY_BAUD_RATE, SERIAL_8N1, SERIAL_TX_ONLY, 1);

allows to use RX pin for I/O.

So I was wondering if it would be useful to add a way in which the serial port can be initialized:

#define MY_SERIAL_MODE SERIAL_TX_ONLY
#define MY_SERIAL_MODE SERIAL_RX_ONLY
#define MY_SERIAL_MODE SERIAL_FULL   (default)

github.com/marceloaqno/MySensors/commit/687fecc6b4abb782eae8e1abb3b07016bfeac291

Also, to use the esp8266 analog pin, I had to comment the line:

ADC_MODE(ADC_VCC);

from MyHwESP8266.cpp.
Is there another way to use analog readings without messing the code?

Excellent port by the way, I have been using here and it works great!!

[Mark Swift]

Was the blocking code issue ever resolved @Yveaux ?

[Yveaux]

@Mark-Swift no, still open at Esp side, and I couldn't think of a clean way working around it...

[gloob]

I have an ESP using as a gateway and want to send some sensor information each 5 minutes to the controller.

Is it possible to use the wait command in the "loop"?

void loop() 
{
  wait(unsigned long ms);
  statusCounter += 1;
  send(msg.set(statusCounter));
}

[mfalkvidd]

@gloob yes

[gloob]

Thanks for the answer. So all incoming or outgoing messages of the gateway will be processed in the background, while the "wait" command is active?

[hek]

@gloob Yes

[Eawo]

Hey

Im not really getting it is this a gateway for mysensors nodes or for mtqq Nodes with esp stuff? I dont really getting it. Do you need a esp8266 wifi gateway to get a wireless gateway for mysensors stuff or is like. Mysensors gateway ---> Esp8266 gateway --- Esp8266 node.

Sorry i dont really understand but im intressted in buying 8-10 sonoff for my window lamps, and i want them to work with domoticz

[alexsh1]

@Eawo the sonoff does not require any GW to work with Domoticz.

[hek]

Yes, @alexsh1 is right. You would normally setup ESP nodes (like the sonoff) to act as a gateway, communicating directly with the controller themselves.

But if you're persistent (or have short wifi range), you can get two ESP nodes to communicate with each other using NRF24/RF69 like @mfalkvidd did recently.

[Matthew Ohlson de Fine]

Hi. I have loaded the Sonoff as an Ethernet gateway using the modified sketch in Vera and it recognizes it perfectly. The problem I have is that I dont know how to control the relay with the Vera now. Please help.

[Mark Swift]

@hek @Yveaux I see the blocking issue on the ESP side has been closed... Does this mean we can now use the ESP in high traffic environments? How can we test it, I assume we also need to activate IRQ for the NRF in the ESP gateway code?

[Yveaux]

@Mark-Swift that issue apparently addressed an issue in the vs1053 library, which surfaced because the esp8266 doesn't support interrupts on spi transfers.
The issue was closed because the vs1053 library was modified to support esp8266, unfortunately not because esp interrupt support for spi was implemented...

[Mark Swift]

@Yveaux said in ESP8266 WiFi gateway port for MySensors:

@Mark-Swift that issue apparently addressed an issue in the vs1053 library, which surfaced because the esp8266 doesn't support interrupts on spi transfers.
The issue was closed because the vs1053 library was modified to support esp8266, unfortunately not because esp interrupt support for spi was implemented...

@Yveaux You're right, should have read more before jumping to conclusions

Damn, I thought I could finally dump my serial gateway!

[ahmedadelhosni]

Is it normal to use GPIO15 ? Isn't it used for booting mode ?

[mfalkvidd]

@ahmedadelhosni where is gpio15 used? (This is a very long thread so finding it without your help isn't easy)