ESP8266 OTA and Arduino IDE What Am I Missing?

GardenShedDev

@mfalkvidd thanks or that - spent an hour or so refactoring the code to be similar to yours, dhcp address rather than static, ensure all the includes are the same, added MY_USE_UDP and the WiFiUdp.h include etc etc, tried from a windows pc instead of mac, I have even put in the additional ESP OTA include in ESP8266mDNS (this threw a compile error - figure its not compatible with MYS) net result - nothing, it still doesnt show up!!!

I have a spare ESP kicking around I might try swapping it over just to make sure there is nothing screwy going on with the HW!

That said, I feel like I am missing something obvious - but I am not sure where to go next?!?!?

GardenShedDev

@mfalkvidd - random thought (and clutching at straws now) does it make a difference on where the includes are placed and the definitions?

The reason I ask is that you have #include <ArduinoOTA.h> at the top of the file then your definitions followed by your includes - i know this is important for MYS just wondering if this is the same for the ArduinoOTA.h when used with MYS? Thoughts?

mfalkvidd

@GardenShedDev have you been able to verify that OTA works without MySensors? Just the basic ArduinoOTA example.

GardenShedDev

@mfalkvidd - No, i havent tried this specific ESP8266 to ensure the Arduino OTA sketch works BUT I do have 5 other devices on the network all using Arduino OTA and they work fine.

For the 20 minutes it will take me to flash the programs in I will check it to rule out the HW

GardenShedDev

@mfalkvidd - checked with the Arduino OTA and it works as it should and appeared straight away in the network ports section of the Arduino IDE.

I went one step further and succesfully re-uploaded the gateway sketch over the Arduino OTA, however, as before the Gateway Sketch OTA doesnt work.

Ok, so this is a step forward as I now know that its not a HW problem - its definitely a SW problem

mfalkvidd

Nice work @GardenShedDev

Which version of MySensors are you using? I haven't upgraded my gateway in a while so I suspect it could be running 2.3.1.

GardenShedDev

@mfalkvidd running 2.3.2

GardenShedDev

@mfalkvidd Have tried downgrading to 2.3.1, at which point the compiler has thrown a shed load of errors, so reinstated the library back to 2.3.2.

I have also worked in my code from my working ArduinoOTA sketch, still not working....?!?!

BearWithBeard

@GardenShedDev I just uploaded a basic blinky sketch with ArduinoOTA and also my trusty old WiFi-MQTT-GW sketch with MySensors 2.3.2 and ArduinoOTA - which basically looks just like the one you posted above, minus the yield() in the loop - to a spare NodeMCU (ESP-12E).

OTA-uploading of a new sketch both via PlatformIO and manually from the command line worked right away - with and without MySensors - but it always took the ArduinoIDE (1.8.12 and 1.8.13) a few minutes and a restart (1.8.12) to display the NodeMCU in the Tools > Port menu list as an upload target. After that, one-click uploads through the GUI worked just fine as well.

My guess would be that this is an ArduinoIDE or maybe network issue, but not related to MySensors. Not sure how to address that at the moment. Your sketch seems fine though. Do you get a reply if you try to ping the IP address of the ESP?

If you have or want to try your luck with PlatformIO, here's a minimal config for OTA uploads:

[platformio]
default_envs = ota

[env]
platform = espressif8266
board = nodemcuv2
framework = arduino
lib_deps = MySensors@2.3.2
monitor_speed = 9600

[env:ota]
upload_port=192.168.178.xxx
upload_protocol=espota

[env:uart]
upload_protocol = esptool
upload_port = COM4

Make sure to insert the correct IP and COM port. To upload an initial sketch via serial run pio run -t upload -e uart.

GardenShedDev

@BearWithBeard Thank you for having a look into this!!!

I retried minus the yield() - still doesnt work (its the only combination I havent actually tried upto this point!!) - also yes I can ping the device on the network.

I am running Arduino IDE 1.8.13, I have tried restarting, gone to another laptop (windows) as I run OSX, waited a minute, an hour, 3 hours, no joy with the MYS OTA.

When I use the Arduino OTA everything works as it should, the board shows up in the Ports menu straight away.

Through process of elimination its not a HW related problem (ESP or network), but I do definitley think its a SW problem specific to the MYS library.

This has got me thinking a little, I am going to recompile the sketch using the windows laptop to see if this works - my thinking being there may be a library problem on the mac.

Im going to swerve platformio at the moment but it does look intriguing so im going have a look at this in down time over the next few weeks :)

mfalkvidd

@GardenShedDev just to check, when the gateway is running, it prints the normal MySensors debug info (messages received and sent), but none of the serial prints registered in setup() are triggered?

GardenShedDev

@mfalkvidd heres a typical output from the gateway serial. Just did a power cycle and captured the startup and then a couple of transactions at the end;

scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 4
cnt 

connected with MYSSIDxxx, channel 1
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
3231 GWT:TPC:IP=192.168.x.x
3396 GWT:RMQ:CONNECTING...
3441 GWT:RMQ:OK
3458 GWT:TPS:TOPIC=mygateway1-out/0/255/0/0/18,MSG SENT
3948 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/30, MSG RECEIVED
4045 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1
4118 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/29, MSG RECEIVED
4215 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=29,pt=0,l=1,sg=0,ft=0,st=NACK:1
4289 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/1/30, MSG RECEIVED
4385 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1
4459 GWT:IMQ:TOPIC=mygateway1-in/3/255/3/0/6, MSG RECEIVED
4556 !TSF:MSG:SEND,0-0-67-3,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4630 GWT:IMQ:TOPIC=mygateway1-in/7/255/3/0/6, MSG RECEIVED
4728 !TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4802 GWT:IMQ:TOPIC=mygateway1-in/9/255/3/0/6, MSG RECEIVED
4899 !TSF:MSG:SEND,0-0-9-9,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
4972 GWT:IMQ:TOPIC=mygateway1-in/8/255/3/0/6, MSG RECEIVED
5070 !TSF:MSG:SEND,0-0-8-8,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5143 GWT:IMQ:TOPIC=mygateway1-in/6/255/3/0/6, MSG RECEIVED
5241 !TSF:MSG:SEND,0-0-6-6,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5314 GWT:IMQ:TOPIC=mygateway1-in/4/255/3/0/6, MSG RECEIVED
5412 !TSF:MSG:SEND,0-0-4-4,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
5485 GWT:IMQ:TOPIC=mygateway1-in/2/255/3/0/6, MSG RECEIVED
5583 !TSF:MSG:SEND,0-0-2-2,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M
7725 TSF:MSG:READ,7-7-255,s=255,c=3,t=7,pt=0,l=0,sg=0:
7782 TSF:MSG:BC
7799 TSF:MSG:FPAR REQ,ID=7
7827 TSF:CKU:OK,FCTRL
7850 TSF:MSG:GWL OK
8576 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=OK:0
8646 TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:19.6
8704 GWT:TPS:TOPIC=mygateway1-out/4/1/1/0/0,MSG SENT
8761 TSF:MSG:READ,4-4-0,s=2,c=1,t=1,pt=7,l=5,sg=0:69.1
8819 GWT:TPS:TOPIC=mygateway1-out/4/2/1/0/1,MSG SENT
8875 TSF:MSG:READ,4-4-0,s=4,c=1,t=0,pt=7,l=5,sg=0:19.4
8933 GWT:TPS:TOPIC=mygateway1-out/4/4/1/0/0,MSG SENT
9481 TSF:MSG:READ,4-4-0,s=3,c=1,t=37,pt=7,l=5,sg=0:0.0
9538 GWT:TPS:TOPIC=mygateway1-out/4/3/1/0/37,MSG SENT
9797 TSF:MSG:READ,7-7-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1
9855 TSF:MSG:PINGED,ID=7,HP=1
9893 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=OK:1
10024 TSF:MSG:READ,4-4-0,s=255,c=3,t=0,pt=1,l=1,sg=0:12
10083 GWT:TPS:TOPIC=mygateway1-out/4/255/3/0/0,MSG SENT
10532 TSF:MSG:READ,4-4-0,s=5,c=1,t=38,pt=7,l=5,sg=0:1.7
10591 GWT:TPS:TOPIC=mygateway1-out/4/5/1/0/38,MSG SENT
11043 TSF:MSG:READ,4-4-0,s=6,c=1,t=38,pt=7,l=5,sg=0:510.0
11103 GWT:TPS:TOPIC=mygateway1-out/4/6/1/0/38,MSG SENT
pm open,type:2 0
17013 TSF:MSG:READ,7-7-0,s=255,c=3,t=0,pt=1,l=1,sg=0:43
17072 GWT:TPS:TOPIC=mygateway1-out/7/255/3/0/0,MSG SENT
17527 TSF:MSG:READ,7-7-0,s=5,c=1,t=38,pt=7,l=5,sg=0:2.4
17586 GWT:TPS:TOPIC=mygateway1-out/7/5/1/0/38,MSG SENT
18040 TSF:MSG:READ,7-7-0,s=6,c=1,t=38,pt=7,l=5,sg=0:730.0
18101 GWT:TPS:TOPIC=mygateway1-out/7/6/1/0/38,MSG SENT
20697 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:3.0
20756 GWT:TPS:TOPIC=mygateway1-out/7/4/1/0/19,MSG SENT
21211 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:0.0

Just to make doubly sure I addded

Serial.print("SETUP MESSAGE TEST PRINT"); to the setup part of the sketch

and here is the revised log snippet;

87 GWT:TPC:CONNECTING...
1421 MCO:BGN:STP
SETUP MESSAGE TEST PRINTReady
IP address: 192.168.x.x
1441 MCO:BGN:INIT OK,TSP=1
1529 TSM:READY:NWD REQ
1556 ?TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=20,pt=0,l=0,sg=0,ft=0,st=OK:
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 4
cnt

electrik

To me it looks like you are not connected to the network at all based on the last output.
Did you check this?

GardenShedDev

Yes it is connected, the psu bumps a little and then settles down when initially turned on (despite having extra capacitors strapped to the NRF and ESP) and then starts communicating as expected. I suspect its to do with the length of the leads and the thickness connecting the nrf to the esp, I either need to shorten the leads or put thicker ones in place to compensate for the v drop.

Funny thing is tthe HW 'as' configured works for Arduino OTA but not the MYS OTA?!?!

mfalkvidd

@GardenShedDev how about disabling

#define MY_RADIO_RF24

to see if OTA + MySensors - NRF24 works?

If the problem is power related, nrf24 is the largest power consumer.

BearWithBeard

@GardenShedDev ArduinoOTA should not be confused with the MySensors-specific OTA features to upload firmware via other transports (like NRF24) to remote nodes. There's nothing special about ArduinoOTA when used alongside MySensors. As far as I know, MySensors doesn't mess with it at all. It's completely separate.

Here's a dependency graph from a basic ESP8266-GW sketch with ArduinoOTA:

|-- <MySensors> 2.3.2
|   |-- <Wire> 1.0
|   |-- <SPI> 1.0
|   |-- <EEPROM> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266SdFat> 1.1.0
|   |   |-- <SPI> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0

That being said, I've now tried to replicate this issue again with different ESP8266 cores from version 2.5.0 to 2.7.4 and MySensors from 2.3.0 to 2.3.2. It's always the same behaviour: It may take the ArduinoIDE a minute, a restart of the IDE or an hardware reset of the NodeMCU to list the device. But in the end, it always connects, uploads and executes the new sketch properly. In any case, I can always OTA-upload manually as soon as the ESP is online, no matter if the ArduinoIDE lists it or not.

I'm not sure what's causing this issue for you, but I honestly doubt that the MySensors framework is the culprit.

The static IP configuration from your earlier attempts might still be retained. Try adding this to the beginning of setup() for test purposes:

WiFi.disconnect(true);
delay(1000);
WiFi.begin(MY_WIFI_SSID, MY_WIFI_PASSWORD);
WiFi.config(0u, 0u, 0u);

Or clear the whole flash, in case some other WiFi related stuff may still be stored (Tools > Erase Flash > All Flash Contents).

You may also try OTA-uploading a sketch a different way. From a terminal, this should look like this:

path/to/python3 path/to/espota.py -i 192.168.esp.ip -p 8266 -f path/to/sketch-binary.ino.bin

On Windows, you should be able to find the ArduinoIDE-managed python3 in %localappdata%\Arduino15\packages\esp8266\tools\python3\xxx-version-string/python3 and espota in %localappdata%\Arduino15\packages\esp8266\hardware\esp8266\xxx-version-string/tools/espota.py. On Linux and macOS, that should be ~/.arduino15/... or ~/Library/Arduino15/...

The ArduinoIDE saves binaries to some temp directory by default, I think, but you can use Sketch > Export Compiled Binary to save it in the same folder as the sketch.

GardenShedDev

@BearWithBeard @mfalkvidd

Some good news and progress!!

Seems like the NRF power consumption was indeed the culprit. Disabling the NRF in the software didnt do the trick, however when I disconected the power to the module and restarted, after about 10 minutes it appeared in the IDE Network Ports section and I can upload sketches over OTA.

As a result I have trimmed down the connector leads to 2 inches, which still works in my enclosure, added another 10uF cap across the ESP regulator and replaced the one strapped across the NRF module. I have lost all the 'bounces' in the debug log from the ESP and the OTA is available straight away every time the board is power cycled. I wont know for sure until I can get my scope out of storage but I suspect that the original 10uF Cap had become more of a resistor than a capacitor with use over the last 2 years - one for the list during down time over xmas :)

So while I have a fix, the elephant in the room is why did the Arduino OTA sketch work and why did the MYS OTA sketch not work?!?!?!

Does the MYS OTA sketch handle power/ comms/ timing in a different way to the Arduino OTA sketch which could cause this problem? or, Is it a timing thing and by this I mean is it a race condition on the power between the ESP core working through its startup and the sketch and the NRF turning on and the causing the power to bounce?

mfalkvidd

Nice work @GardenShedDev

About why this happens: creating different scenarios and doing lots and lots of testing is probably the only way to find out. Or connect the scope (or something like Otii) and try to match voltage/current event to parts of code.

GardenShedDev

@mfalkvidd - your response got me thinking about my question some more - the effort required to diagnose/ rework would for little gain, i am thinking its simpler to rework the psu and seperate the NRF from the ESP - its likley that the ESP regulator is only spec'd to power the ESP and not a lot else - lets face it we are not dealing with miltary grade components here lol :)

Thanks for the help and support!

ESP8266 OTA and Arduino IDE What Am I Missing?

24

11.7k

11.2k

113.1k