ESP8266 OTA and Arduino IDE What Am I Missing?

[GardenShedDev]

Apologies upfront if this has already been asked and worked out but I cannot find the answer in the forum or a good working example.

I have been running MySensors for a few years now with no OTA, I have decided I want to enable it for my gateway only. For simplicity I want to use ESP8266 OTA and avoid custom booloaders for the ESP8266.

I have used the ESP8266ota example sketch as the base and tweaked - see below - the sketch compiles and the gateway works - but - the gateway device cannot found in the network ports section in the Arduino IDE - although my other ESPs are present?!?!?

I know that there are differences between a standard ESP8266 OTA and the MySensors OTA version but I am not sure what, if anything I am missing in the my sensors variant?

// Enable debug prints to serial monitor
#define MY_DEBUG

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_RF24
//#define MY_RADIO_RFM69
//#define MY_RADIO_RFM95

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set WIFI SSID and password
#define MY_WIFI_SSID "SSID"
#define MY_WIFI_PASSWORD "password"

// Set this node's subscribe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires username/password
#define MY_MQTT_USER "user"
#define MY_MQTT_PASSWORD "password"

// Controller ip address. Enables client mode (default is "server" mode).
// Also enable this if MY_USE_UDP is used and you want sensor data sent somewhere.
// MQTT broker ip address.
#define MY_CONTROLLER_IP_ADDRESS 192,168,0,1

// The port to keep open on node server mode
// The MQTT broker port to to open
#define MY_PORT 1883

// Set the hostname for the WiFi Client. This is the hostname
// passed to the DHCP server if not static.
#define MY_HOSTNAME "MYS_GW"

// Enable UDP communication
//#define MY_USE_UDP  // If using UDP you need to set MY_CONTROLLER_IP_ADDRESS below

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
#define MY_IP_ADDRESS 192,168,0,2

// If using static ip you can define Gateway and Subnet address as well
#define MY_IP_GATEWAY_ADDRESS 192,168,0,1
#define MY_IP_SUBNET_ADDRESS 255,255,255,0


// How many clients should be able to connect to this gateway (default 1)
//#define MY_GATEWAY_MAX_CLIENTS 2

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE

// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN  3

// Set blinking period
// #define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
// Led pins used if blinking feature is enabled above
//#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED

#include <ArduinoOTA.h>
#include <MySensors.h>

void setup()
{
	// Setup locally attached sensors
	ArduinoOTA.onStart([]() {
		Serial.println("ArduinoOTA start");
	});
	ArduinoOTA.onEnd([]() {
		Serial.println("\nArduinoOTA end");
	});
	ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {
		Serial.printf("OTA Progress: %u%%\r", (progress / (total / 100)));
	});
	ArduinoOTA.onError([](ota_error_t error) {
		Serial.printf("Error[%u]: ", error);
		if (error == OTA_AUTH_ERROR) {
			Serial.println("Auth Failed");
		} else if (error == OTA_BEGIN_ERROR) {
			Serial.println("Begin Failed");
		} else if (error == OTA_CONNECT_ERROR) {
			Serial.println("Connect Failed");
		} else if (error == OTA_RECEIVE_ERROR) {
			Serial.println("Receive Failed");
		} else if (error == OTA_END_ERROR) {
			Serial.println("End Failed");
		}
	});
	ArduinoOTA.begin();
}

void presentation()
{
	// Present locally attached sensors here
}

void loop()
{
	// Send locally attached sensors data here
	ArduinoOTA.handle();
  yield();                //let the ESP do some internal work
}

[mfalkvidd]

@GardenShedDev the sketch looks good to me. I have a working ESP8266 RFM69 gateway with OTA, and I can't see any differences from my sketch that should affect OTA. My sketch is at https://github.com/mfalkvidd/Arduino-MySensors-ESP8266-RFM69-Gateway if you want to take a look at it.

[GardenShedDev]

@mfalkvidd thanks or that - spent an hour or so refactoring the code to be similar to yours, dhcp address rather than static, ensure all the includes are the same, added MY_USE_UDP and the WiFiUdp.h include etc etc, tried from a windows pc instead of mac, I have even put in the additional ESP OTA include in ESP8266mDNS (this threw a compile error - figure its not compatible with MYS) net result - nothing, it still doesnt show up!!!

I have a spare ESP kicking around I might try swapping it over just to make sure there is nothing screwy going on with the HW!

That said, I feel like I am missing something obvious - but I am not sure where to go next?!?!?

[GardenShedDev]

@mfalkvidd - random thought (and clutching at straws now) does it make a difference on where the includes are placed and the definitions?

The reason I ask is that you have #include <ArduinoOTA.h> at the top of the file then your definitions followed by your includes - i know this is important for MYS just wondering if this is the same for the ArduinoOTA.h when used with MYS? Thoughts?

[mfalkvidd]

@GardenShedDev have you been able to verify that OTA works without MySensors? Just the basic ArduinoOTA example.

[GardenShedDev]

@mfalkvidd - No, i havent tried this specific ESP8266 to ensure the Arduino OTA sketch works BUT I do have 5 other devices on the network all using Arduino OTA and they work fine.

For the 20 minutes it will take me to flash the programs in I will check it to rule out the HW

[GardenShedDev]

@mfalkvidd - checked with the Arduino OTA and it works as it should and appeared straight away in the network ports section of the Arduino IDE.

I went one step further and succesfully re-uploaded the gateway sketch over the Arduino OTA, however, as before the Gateway Sketch OTA doesnt work.

Ok, so this is a step forward as I now know that its not a HW problem - its definitely a SW problem

[mfalkvidd]

Nice work @GardenShedDev

Which version of MySensors are you using? I haven't upgraded my gateway in a while so I suspect it could be running 2.3.1.

[GardenShedDev]

@mfalkvidd running 2.3.2

[GardenShedDev]

@mfalkvidd Have tried downgrading to 2.3.1, at which point the compiler has thrown a shed load of errors, so reinstated the library back to 2.3.2.

I have also worked in my code from my working ArduinoOTA sketch, still not working....?!?!

[BearWithBeard]

@GardenShedDev I just uploaded a basic blinky sketch with ArduinoOTA and also my trusty old WiFi-MQTT-GW sketch with MySensors 2.3.2 and ArduinoOTA - which basically looks just like the one you posted above, minus the yield() in the loop - to a spare NodeMCU (ESP-12E).

OTA-uploading of a new sketch both via PlatformIO and manually from the command line worked right away - with and without MySensors - but it always took the ArduinoIDE (1.8.12 and 1.8.13) a few minutes and a restart (1.8.12) to display the NodeMCU in the Tools > Port menu list as an upload target. After that, one-click uploads through the GUI worked just fine as well.

My guess would be that this is an ArduinoIDE or maybe network issue, but not related to MySensors. Not sure how to address that at the moment. Your sketch seems fine though. Do you get a reply if you try to ping the IP address of the ESP?

If you have or want to try your luck with PlatformIO, here's a minimal config for OTA uploads:

[platformio]
default_envs = ota

[env]
platform = espressif8266
board = nodemcuv2
framework = arduino
lib_deps = MySensors@2.3.2
monitor_speed = 9600

[env:ota]
upload_port=192.168.178.xxx
upload_protocol=espota

[env:uart]
upload_protocol = esptool
upload_port = COM4

Make sure to insert the correct IP and COM port. To upload an initial sketch via serial run pio run -t upload -e uart.

[GardenShedDev]

@BearWithBeard Thank you for having a look into this!!!

I retried minus the yield() - still doesnt work (its the only combination I havent actually tried upto this point!!) - also yes I can ping the device on the network.

I am running Arduino IDE 1.8.13, I have tried restarting, gone to another laptop (windows) as I run OSX, waited a minute, an hour, 3 hours, no joy with the MYS OTA.

When I use the Arduino OTA everything works as it should, the board shows up in the Ports menu straight away.

Through process of elimination its not a HW related problem (ESP or network), but I do definitley think its a SW problem specific to the MYS library.

This has got me thinking a little, I am going to recompile the sketch using the windows laptop to see if this works - my thinking being there may be a library problem on the mac.

Im going to swerve platformio at the moment but it does look intriguing so im going have a look at this in down time over the next few weeks

[mfalkvidd]

@GardenShedDev just to check, when the gateway is running, it prints the normal MySensors debug info (messages received and sent), but none of the serial prints registered in setup() are triggered?

[GardenShedDev]

@mfalkvidd heres a typical output from the gateway serial. Just did a power cycle and captured the startup and then a couple of transactions at the end;

scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 4
cnt 

connected with MYSSIDxxx, channel 1
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
3231 GWT:TPC:IP=192.168.x.x
3396 GWT:RMQ:CONNECTING...
3441 GWT:RMQ:OK
3458 GWT:TPS:TOPIC=mygateway1-out/0/255/0/0/18,MSG SENT
3948 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/30, MSG RECEIVED
4045 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1
4118 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/29, MSG RECEIVED
4215 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=29,pt=0,l=1,sg=0,ft=0,st=NACK:1
4289 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/1/30, MSG RECEIVED
4385 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1
4459 GWT:IMQ:TOPIC=mygateway1-in/3/255/3/0/6, MSG RECEIVED
4556 !TSF:MSG:SEND,0-0-67-3,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4630 GWT:IMQ:TOPIC=mygateway1-in/7/255/3/0/6, MSG RECEIVED
4728 !TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4802 GWT:IMQ:TOPIC=mygateway1-in/9/255/3/0/6, MSG RECEIVED
4899 !TSF:MSG:SEND,0-0-9-9,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4972 GWT:IMQ:TOPIC=mygateway1-in/8/255/3/0/6, MSG RECEIVED
5070 !TSF:MSG:SEND,0-0-8-8,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5143 GWT:IMQ:TOPIC=mygateway1-in/6/255/3/0/6, MSG RECEIVED
5241 !TSF:MSG:SEND,0-0-6-6,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5314 GWT:IMQ:TOPIC=mygateway1-in/4/255/3/0/6, MSG RECEIVED
5412 !TSF:MSG:SEND,0-0-4-4,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5485 GWT:IMQ:TOPIC=mygateway1-in/2/255/3/0/6, MSG RECEIVED
5583 !TSF:MSG:SEND,0-0-2-2,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
7725 TSF:MSG:READ,7-7-255,s=255,c=3,t=7,pt=0,l=0,sg=0:
7782 TSF:MSG:BC
7799 TSF:MSG:FPAR REQ,ID=7
7827 TSF:CKU:OK,FCTRL
7850 TSF:MSG:GWL OK
8576 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=OK:0
8646 TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:19.6
8704 GWT:TPS:TOPIC=mygateway1-out/4/1/1/0/0,MSG SENT
8761 TSF:MSG:READ,4-4-0,s=2,c=1,t=1,pt=7,l=5,sg=0:69.1
8819 GWT:TPS:TOPIC=mygateway1-out/4/2/1/0/1,MSG SENT
8875 TSF:MSG:READ,4-4-0,s=4,c=1,t=0,pt=7,l=5,sg=0:19.4
8933 GWT:TPS:TOPIC=mygateway1-out/4/4/1/0/0,MSG SENT
9481 TSF:MSG:READ,4-4-0,s=3,c=1,t=37,pt=7,l=5,sg=0:0.0
9538 GWT:TPS:TOPIC=mygateway1-out/4/3/1/0/37,MSG SENT
9797 TSF:MSG:READ,7-7-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1
9855 TSF:MSG:PINGED,ID=7,HP=1
9893 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=OK:1
10024 TSF:MSG:READ,4-4-0,s=255,c=3,t=0,pt=1,l=1,sg=0:12
10083 GWT:TPS:TOPIC=mygateway1-out/4/255/3/0/0,MSG SENT
10532 TSF:MSG:READ,4-4-0,s=5,c=1,t=38,pt=7,l=5,sg=0:1.7
10591 GWT:TPS:TOPIC=mygateway1-out/4/5/1/0/38,MSG SENT
11043 TSF:MSG:READ,4-4-0,s=6,c=1,t=38,pt=7,l=5,sg=0:510.0
11103 GWT:TPS:TOPIC=mygateway1-out/4/6/1/0/38,MSG SENT
pm open,type:2 0
17013 TSF:MSG:READ,7-7-0,s=255,c=3,t=0,pt=1,l=1,sg=0:43
17072 GWT:TPS:TOPIC=mygateway1-out/7/255/3/0/0,MSG SENT
17527 TSF:MSG:READ,7-7-0,s=5,c=1,t=38,pt=7,l=5,sg=0:2.4
17586 GWT:TPS:TOPIC=mygateway1-out/7/5/1/0/38,MSG SENT
18040 TSF:MSG:READ,7-7-0,s=6,c=1,t=38,pt=7,l=5,sg=0:730.0
18101 GWT:TPS:TOPIC=mygateway1-out/7/6/1/0/38,MSG SENT
20697 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:3.0
20756 GWT:TPS:TOPIC=mygateway1-out/7/4/1/0/19,MSG SENT
21211 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:0.0

Just to make doubly sure I addded

Serial.print("SETUP MESSAGE TEST PRINT"); to the setup part of the sketch

and here is the revised log snippet;

87 GWT:TPC:CONNECTING...
1421 MCO:BGN:STP
SETUP MESSAGE TEST PRINTReady
IP address: 192.168.x.x
1441 MCO:BGN:INIT OK,TSP=1
1529 TSM:READY:NWD REQ
1556 ?TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=20,pt=0,l=0,sg=0,ft=0,st=OK:
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 4
cnt 

[electrik]

To me it looks like you are not connected to the network at all based on the last output.
Did you check this?

[GardenShedDev]

Yes it is connected, the psu bumps a little and then settles down when initially turned on (despite having extra capacitors strapped to the NRF and ESP) and then starts communicating as expected. I suspect its to do with the length of the leads and the thickness connecting the nrf to the esp, I either need to shorten the leads or put thicker ones in place to compensate for the v drop.

Funny thing is tthe HW 'as' configured works for Arduino OTA but not the MYS OTA?!?!

[mfalkvidd]

@GardenShedDev how about disabling

#define MY_RADIO_RF24

to see if OTA + MySensors - NRF24 works?

If the problem is power related, nrf24 is the largest power consumer.

[BearWithBeard]

@GardenShedDev ArduinoOTA should not be confused with the MySensors-specific OTA features to upload firmware via other transports (like NRF24) to remote nodes. There's nothing special about ArduinoOTA when used alongside MySensors. As far as I know, MySensors doesn't mess with it at all. It's completely separate.

Here's a dependency graph from a basic ESP8266-GW sketch with ArduinoOTA:

|-- <MySensors> 2.3.2
|   |-- <Wire> 1.0
|   |-- <SPI> 1.0
|   |-- <EEPROM> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266SdFat> 1.1.0
|   |   |-- <SPI> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0

That being said, I've now tried to replicate this issue again with different ESP8266 cores from version 2.5.0 to 2.7.4 and MySensors from 2.3.0 to 2.3.2. It's always the same behaviour: It may take the ArduinoIDE a minute, a restart of the IDE or an hardware reset of the NodeMCU to list the device. But in the end, it always connects, uploads and executes the new sketch properly. In any case, I can always OTA-upload manually as soon as the ESP is online, no matter if the ArduinoIDE lists it or not.

I'm not sure what's causing this issue for you, but I honestly doubt that the MySensors framework is the culprit.

The static IP configuration from your earlier attempts might still be retained. Try adding this to the beginning of setup() for test purposes:

WiFi.disconnect(true);
delay(1000);
WiFi.begin(MY_WIFI_SSID, MY_WIFI_PASSWORD);
WiFi.config(0u, 0u, 0u);

Or clear the whole flash, in case some other WiFi related stuff may still be stored (Tools > Erase Flash > All Flash Contents).

You may also try OTA-uploading a sketch a different way. From a terminal, this should look like this:

path/to/python3 path/to/espota.py -i 192.168.esp.ip -p 8266 -f path/to/sketch-binary.ino.bin

On Windows, you should be able to find the ArduinoIDE-managed python3 in %localappdata%\Arduino15\packages\esp8266\tools\python3\xxx-version-string/python3 and espota in %localappdata%\Arduino15\packages\esp8266\hardware\esp8266\xxx-version-string/tools/espota.py. On Linux and macOS, that should be ~/.arduino15/... or ~/Library/Arduino15/...

The ArduinoIDE saves binaries to some temp directory by default, I think, but you can use Sketch > Export Compiled Binary to save it in the same folder as the sketch.

[GardenShedDev]

@BearWithBeard @mfalkvidd

Some good news and progress!!

Seems like the NRF power consumption was indeed the culprit. Disabling the NRF in the software didnt do the trick, however when I disconected the power to the module and restarted, after about 10 minutes it appeared in the IDE Network Ports section and I can upload sketches over OTA.

As a result I have trimmed down the connector leads to 2 inches, which still works in my enclosure, added another 10uF cap across the ESP regulator and replaced the one strapped across the NRF module. I have lost all the 'bounces' in the debug log from the ESP and the OTA is available straight away every time the board is power cycled. I wont know for sure until I can get my scope out of storage but I suspect that the original 10uF Cap had become more of a resistor than a capacitor with use over the last 2 years - one for the list during down time over xmas

So while I have a fix, the elephant in the room is why did the Arduino OTA sketch work and why did the MYS OTA sketch not work?!?!?!

Does the MYS OTA sketch handle power/ comms/ timing in a different way to the Arduino OTA sketch which could cause this problem? or, Is it a timing thing and by this I mean is it a race condition on the power between the ESP core working through its startup and the sketch and the NRF turning on and the causing the power to bounce?

[mfalkvidd]

Nice work @GardenShedDev

About why this happens: creating different scenarios and doing lots and lots of testing is probably the only way to find out. Or connect the scope (or something like Otii) and try to match voltage/current event to parts of code.

[GardenShedDev]

@mfalkvidd - your response got me thinking about my question some more - the effort required to diagnose/ rework would for little gain, i am thinking its simpler to rework the psu and seperate the NRF from the ESP - its likley that the ESP regulator is only spec'd to power the ESP and not a lot else - lets face it we are not dealing with miltary grade components here lol

Thanks for the help and support!