ESP8266 OTA and Arduino IDE What Am I Missing?
-
Apologies upfront if this has already been asked and worked out but I cannot find the answer in the forum or a good working example.
I have been running MySensors for a few years now with no OTA, I have decided I want to enable it for my gateway only. For simplicity I want to use ESP8266 OTA and avoid custom booloaders for the ESP8266.
I have used the ESP8266ota example sketch as the base and tweaked - see below - the sketch compiles and the gateway works - but - the gateway device cannot found in the network ports section in the Arduino IDE - although my other ESPs are present?!?!?
I know that there are differences between a standard ESP8266 OTA and the MySensors OTA version but I am not sure what, if anything I am missing in the my sensors variant?
// Enable debug prints to serial monitor #define MY_DEBUG // Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h #define MY_BAUD_RATE 9600 // Enables and select radio type (if attached) #define MY_RADIO_RF24 //#define MY_RADIO_RFM69 //#define MY_RADIO_RFM95 #define MY_GATEWAY_MQTT_CLIENT #define MY_GATEWAY_ESP8266 // Set WIFI SSID and password #define MY_WIFI_SSID "SSID" #define MY_WIFI_PASSWORD "password" // Set this node's subscribe and publish topic prefix #define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out" #define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in" // Set MQTT client id #define MY_MQTT_CLIENT_ID "mysensors-1" // Enable these if your MQTT broker requires username/password #define MY_MQTT_USER "user" #define MY_MQTT_PASSWORD "password" // Controller ip address. Enables client mode (default is "server" mode). // Also enable this if MY_USE_UDP is used and you want sensor data sent somewhere. // MQTT broker ip address. #define MY_CONTROLLER_IP_ADDRESS 192,168,0,1 // The port to keep open on node server mode // The MQTT broker port to to open #define MY_PORT 1883 // Set the hostname for the WiFi Client. This is the hostname // passed to the DHCP server if not static. #define MY_HOSTNAME "MYS_GW" // Enable UDP communication //#define MY_USE_UDP // If using UDP you need to set MY_CONTROLLER_IP_ADDRESS below // Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP) #define MY_IP_ADDRESS 192,168,0,2 // If using static ip you can define Gateway and Subnet address as well #define MY_IP_GATEWAY_ADDRESS 192,168,0,1 #define MY_IP_SUBNET_ADDRESS 255,255,255,0 // How many clients should be able to connect to this gateway (default 1) //#define MY_GATEWAY_MAX_CLIENTS 2 // Enable inclusion mode //#define MY_INCLUSION_MODE_FEATURE // Enable Inclusion mode button on gateway //#define MY_INCLUSION_BUTTON_FEATURE // Set inclusion mode duration (in seconds) //#define MY_INCLUSION_MODE_DURATION 60 // Digital pin used for inclusion mode button //#define MY_INCLUSION_MODE_BUTTON_PIN 3 // Set blinking period // #define MY_DEFAULT_LED_BLINK_PERIOD 300 // Flash leds on rx/tx/err // Led pins used if blinking feature is enabled above //#define MY_DEFAULT_ERR_LED_PIN 16 // Error led pin //#define MY_DEFAULT_RX_LED_PIN 16 // Receive led pin //#define MY_DEFAULT_TX_LED_PIN 16 // the PCB, on board LED #include <ArduinoOTA.h> #include <MySensors.h> void setup() { // Setup locally attached sensors ArduinoOTA.onStart([]() { Serial.println("ArduinoOTA start"); }); ArduinoOTA.onEnd([]() { Serial.println("\nArduinoOTA end"); }); ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) { Serial.printf("OTA Progress: %u%%\r", (progress / (total / 100))); }); ArduinoOTA.onError([](ota_error_t error) { Serial.printf("Error[%u]: ", error); if (error == OTA_AUTH_ERROR) { Serial.println("Auth Failed"); } else if (error == OTA_BEGIN_ERROR) { Serial.println("Begin Failed"); } else if (error == OTA_CONNECT_ERROR) { Serial.println("Connect Failed"); } else if (error == OTA_RECEIVE_ERROR) { Serial.println("Receive Failed"); } else if (error == OTA_END_ERROR) { Serial.println("End Failed"); } }); ArduinoOTA.begin(); } void presentation() { // Present locally attached sensors here } void loop() { // Send locally attached sensors data here ArduinoOTA.handle(); yield(); //let the ESP do some internal work }@GardenShedDev the sketch looks good to me. I have a working ESP8266 RFM69 gateway with OTA, and I can't see any differences from my sketch that should affect OTA. My sketch is at https://github.com/mfalkvidd/Arduino-MySensors-ESP8266-RFM69-Gateway if you want to take a look at it.
-
@mfalkvidd thanks or that - spent an hour or so refactoring the code to be similar to yours, dhcp address rather than static, ensure all the includes are the same, added MY_USE_UDP and the WiFiUdp.h include etc etc, tried from a windows pc instead of mac, I have even put in the additional ESP OTA include in ESP8266mDNS (this threw a compile error - figure its not compatible with MYS) net result - nothing, it still doesnt show up!!!
I have a spare ESP kicking around I might try swapping it over just to make sure there is nothing screwy going on with the HW!
That said, I feel like I am missing something obvious - but I am not sure where to go next?!?!?
-
@mfalkvidd - random thought (and clutching at straws now) does it make a difference on where the includes are placed and the definitions?
The reason I ask is that you have #include <ArduinoOTA.h> at the top of the file then your definitions followed by your includes - i know this is important for MYS just wondering if this is the same for the ArduinoOTA.h when used with MYS? Thoughts?
-
@mfalkvidd - random thought (and clutching at straws now) does it make a difference on where the includes are placed and the definitions?
The reason I ask is that you have #include <ArduinoOTA.h> at the top of the file then your definitions followed by your includes - i know this is important for MYS just wondering if this is the same for the ArduinoOTA.h when used with MYS? Thoughts?
-
@mfalkvidd - No, i havent tried this specific ESP8266 to ensure the Arduino OTA sketch works BUT I do have 5 other devices on the network all using Arduino OTA and they work fine.
For the 20 minutes it will take me to flash the programs in I will check it to rule out the HW
-
@mfalkvidd - checked with the Arduino OTA and it works as it should and appeared straight away in the network ports section of the Arduino IDE.
I went one step further and succesfully re-uploaded the gateway sketch over the Arduino OTA, however, as before the Gateway Sketch OTA doesnt work.
Ok, so this is a step forward as I now know that its not a HW problem - its definitely a SW problem
-
@mfalkvidd - checked with the Arduino OTA and it works as it should and appeared straight away in the network ports section of the Arduino IDE.
I went one step further and succesfully re-uploaded the gateway sketch over the Arduino OTA, however, as before the Gateway Sketch OTA doesnt work.
Ok, so this is a step forward as I now know that its not a HW problem - its definitely a SW problem
-
@mfalkvidd running 2.3.2
-
@mfalkvidd Have tried downgrading to 2.3.1, at which point the compiler has thrown a shed load of errors, so reinstated the library back to 2.3.2.
I have also worked in my code from my working ArduinoOTA sketch, still not working....?!?!
-
@mfalkvidd Have tried downgrading to 2.3.1, at which point the compiler has thrown a shed load of errors, so reinstated the library back to 2.3.2.
I have also worked in my code from my working ArduinoOTA sketch, still not working....?!?!
@GardenShedDev I just uploaded a basic blinky sketch with ArduinoOTA and also my trusty old WiFi-MQTT-GW sketch with MySensors 2.3.2 and ArduinoOTA - which basically looks just like the one you posted above, minus the
yield()in the loop - to a spare NodeMCU (ESP-12E).OTA-uploading of a new sketch both via PlatformIO and manually from the command line worked right away - with and without MySensors - but it always took the ArduinoIDE (1.8.12 and 1.8.13) a few minutes and a restart (1.8.12) to display the NodeMCU in the Tools > Port menu list as an upload target. After that, one-click uploads through the GUI worked just fine as well.
My guess would be that this is an ArduinoIDE or maybe network issue, but not related to MySensors. Not sure how to address that at the moment. Your sketch seems fine though. Do you get a reply if you try to ping the IP address of the ESP?
If you have or want to try your luck with PlatformIO, here's a minimal config for OTA uploads:
[platformio] default_envs = ota [env] platform = espressif8266 board = nodemcuv2 framework = arduino lib_deps = MySensors@2.3.2 monitor_speed = 9600 [env:ota] upload_port=192.168.178.xxx upload_protocol=espota [env:uart] upload_protocol = esptool upload_port = COM4Make sure to insert the correct IP and COM port. To upload an initial sketch via serial run
pio run -t upload -e uart. -
@BearWithBeard Thank you for having a look into this!!!
I retried minus the yield() - still doesnt work (its the only combination I havent actually tried upto this point!!) - also yes I can ping the device on the network.
I am running Arduino IDE 1.8.13, I have tried restarting, gone to another laptop (windows) as I run OSX, waited a minute, an hour, 3 hours, no joy with the MYS OTA.
When I use the Arduino OTA everything works as it should, the board shows up in the Ports menu straight away.
Through process of elimination its not a HW related problem (ESP or network), but I do definitley think its a SW problem specific to the MYS library.
This has got me thinking a little, I am going to recompile the sketch using the windows laptop to see if this works - my thinking being there may be a library problem on the mac.
Im going to swerve platformio at the moment but it does look intriguing so im going have a look at this in down time over the next few weeks :)
-
@BearWithBeard Thank you for having a look into this!!!
I retried minus the yield() - still doesnt work (its the only combination I havent actually tried upto this point!!) - also yes I can ping the device on the network.
I am running Arduino IDE 1.8.13, I have tried restarting, gone to another laptop (windows) as I run OSX, waited a minute, an hour, 3 hours, no joy with the MYS OTA.
When I use the Arduino OTA everything works as it should, the board shows up in the Ports menu straight away.
Through process of elimination its not a HW related problem (ESP or network), but I do definitley think its a SW problem specific to the MYS library.
This has got me thinking a little, I am going to recompile the sketch using the windows laptop to see if this works - my thinking being there may be a library problem on the mac.
Im going to swerve platformio at the moment but it does look intriguing so im going have a look at this in down time over the next few weeks :)
-
@mfalkvidd heres a typical output from the gateway serial. Just did a power cycle and captured the startup and then a couple of transactions at the end;
scandone state: 0 -> 2 (b0) state: 2 -> 3 (0) state: 3 -> 5 (10) add 0 aid 4 cnt connected with MYSSIDxxx, channel 1 ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x 3231 GWT:TPC:IP=192.168.x.x 3396 GWT:RMQ:CONNECTING... 3441 GWT:RMQ:OK 3458 GWT:TPS:TOPIC=mygateway1-out/0/255/0/0/18,MSG SENT 3948 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/30, MSG RECEIVED 4045 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1 4118 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/0/29, MSG RECEIVED 4215 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=29,pt=0,l=1,sg=0,ft=0,st=NACK:1 4289 GWT:IMQ:TOPIC=mygateway1-in/3/4/1/1/30, MSG RECEIVED 4385 !TSF:MSG:SEND,0-0-67-3,s=4,c=1,t=30,pt=0,l=1,sg=0,ft=0,st=NACK:1 4459 GWT:IMQ:TOPIC=mygateway1-in/3/255/3/0/6, MSG RECEIVED 4556 !TSF:MSG:SEND,0-0-67-3,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 4630 GWT:IMQ:TOPIC=mygateway1-in/7/255/3/0/6, MSG RECEIVED 4728 !TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 4802 GWT:IMQ:TOPIC=mygateway1-in/9/255/3/0/6, MSG RECEIVED 4899 !TSF:MSG:SEND,0-0-9-9,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 4972 GWT:IMQ:TOPIC=mygateway1-in/8/255/3/0/6, MSG RECEIVED 5070 !TSF:MSG:SEND,0-0-8-8,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 5143 GWT:IMQ:TOPIC=mygateway1-in/6/255/3/0/6, MSG RECEIVED 5241 !TSF:MSG:SEND,0-0-6-6,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 5314 GWT:IMQ:TOPIC=mygateway1-in/4/255/3/0/6, MSG RECEIVED 5412 !TSF:MSG:SEND,0-0-4-4,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 5485 GWT:IMQ:TOPIC=mygateway1-in/2/255/3/0/6, MSG RECEIVED 5583 !TSF:MSG:SEND,0-0-2-2,s=255,c=3,t=6,pt=0,l=1,sg=0,ft=0,st=NACK:M 7725 TSF:MSG:READ,7-7-255,s=255,c=3,t=7,pt=0,l=0,sg=0: 7782 TSF:MSG:BC 7799 TSF:MSG:FPAR REQ,ID=7 7827 TSF:CKU:OK,FCTRL 7850 TSF:MSG:GWL OK 8576 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=8,pt=1,l=1,sg=0,ft=0,st=OK:0 8646 TSF:MSG:READ,4-4-0,s=1,c=1,t=0,pt=7,l=5,sg=0:19.6 8704 GWT:TPS:TOPIC=mygateway1-out/4/1/1/0/0,MSG SENT 8761 TSF:MSG:READ,4-4-0,s=2,c=1,t=1,pt=7,l=5,sg=0:69.1 8819 GWT:TPS:TOPIC=mygateway1-out/4/2/1/0/1,MSG SENT 8875 TSF:MSG:READ,4-4-0,s=4,c=1,t=0,pt=7,l=5,sg=0:19.4 8933 GWT:TPS:TOPIC=mygateway1-out/4/4/1/0/0,MSG SENT 9481 TSF:MSG:READ,4-4-0,s=3,c=1,t=37,pt=7,l=5,sg=0:0.0 9538 GWT:TPS:TOPIC=mygateway1-out/4/3/1/0/37,MSG SENT 9797 TSF:MSG:READ,7-7-0,s=255,c=3,t=24,pt=1,l=1,sg=0:1 9855 TSF:MSG:PINGED,ID=7,HP=1 9893 TSF:MSG:SEND,0-0-7-7,s=255,c=3,t=25,pt=1,l=1,sg=0,ft=0,st=OK:1 10024 TSF:MSG:READ,4-4-0,s=255,c=3,t=0,pt=1,l=1,sg=0:12 10083 GWT:TPS:TOPIC=mygateway1-out/4/255/3/0/0,MSG SENT 10532 TSF:MSG:READ,4-4-0,s=5,c=1,t=38,pt=7,l=5,sg=0:1.7 10591 GWT:TPS:TOPIC=mygateway1-out/4/5/1/0/38,MSG SENT 11043 TSF:MSG:READ,4-4-0,s=6,c=1,t=38,pt=7,l=5,sg=0:510.0 11103 GWT:TPS:TOPIC=mygateway1-out/4/6/1/0/38,MSG SENT pm open,type:2 0 17013 TSF:MSG:READ,7-7-0,s=255,c=3,t=0,pt=1,l=1,sg=0:43 17072 GWT:TPS:TOPIC=mygateway1-out/7/255/3/0/0,MSG SENT 17527 TSF:MSG:READ,7-7-0,s=5,c=1,t=38,pt=7,l=5,sg=0:2.4 17586 GWT:TPS:TOPIC=mygateway1-out/7/5/1/0/38,MSG SENT 18040 TSF:MSG:READ,7-7-0,s=6,c=1,t=38,pt=7,l=5,sg=0:730.0 18101 GWT:TPS:TOPIC=mygateway1-out/7/6/1/0/38,MSG SENT 20697 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:3.0 20756 GWT:TPS:TOPIC=mygateway1-out/7/4/1/0/19,MSG SENT 21211 TSF:MSG:READ,7-7-0,s=4,c=1,t=19,pt=7,l=5,sg=0:0.0Just to make doubly sure I addded
Serial.print("SETUP MESSAGE TEST PRINT"); to the setup part of the sketch
and here is the revised log snippet;
87 GWT:TPC:CONNECTING... 1421 MCO:BGN:STP SETUP MESSAGE TEST PRINTReady IP address: 192.168.x.x 1441 MCO:BGN:INIT OK,TSP=1 1529 TSM:READY:NWD REQ 1556 ?TSF:MSG:SEND,0-0-255-255,s=255,c=3,t=20,pt=0,l=0,sg=0,ft=0,st=OK: scandone state: 0 -> 2 (b0) state: 2 -> 3 (0) state: 3 -> 5 (10) add 0 aid 4 cnt -
Yes it is connected, the psu bumps a little and then settles down when initially turned on (despite having extra capacitors strapped to the NRF and ESP) and then starts communicating as expected. I suspect its to do with the length of the leads and the thickness connecting the nrf to the esp, I either need to shorten the leads or put thicker ones in place to compensate for the v drop.
Funny thing is tthe HW 'as' configured works for Arduino OTA but not the MYS OTA?!?!
-
Yes it is connected, the psu bumps a little and then settles down when initially turned on (despite having extra capacitors strapped to the NRF and ESP) and then starts communicating as expected. I suspect its to do with the length of the leads and the thickness connecting the nrf to the esp, I either need to shorten the leads or put thicker ones in place to compensate for the v drop.
Funny thing is tthe HW 'as' configured works for Arduino OTA but not the MYS OTA?!?!
-
Yes it is connected, the psu bumps a little and then settles down when initially turned on (despite having extra capacitors strapped to the NRF and ESP) and then starts communicating as expected. I suspect its to do with the length of the leads and the thickness connecting the nrf to the esp, I either need to shorten the leads or put thicker ones in place to compensate for the v drop.
Funny thing is tthe HW 'as' configured works for Arduino OTA but not the MYS OTA?!?!
@GardenShedDev ArduinoOTA should not be confused with the MySensors-specific OTA features to upload firmware via other transports (like NRF24) to remote nodes. There's nothing special about ArduinoOTA when used alongside MySensors. As far as I know, MySensors doesn't mess with it at all. It's completely separate.
Here's a dependency graph from a basic ESP8266-GW sketch with ArduinoOTA:
|-- <MySensors> 2.3.2 | |-- <Wire> 1.0 | |-- <SPI> 1.0 | |-- <EEPROM> 1.0 | |-- <ESP8266WiFi> 1.0 | |-- <ESP8266SdFat> 1.1.0 | | |-- <SPI> 1.0 |-- <ArduinoOTA> 1.0 | |-- <ESP8266WiFi> 1.0 | |-- <ESP8266mDNS> 1.2 | | |-- <ESP8266WiFi> 1.0That being said, I've now tried to replicate this issue again with different ESP8266 cores from version 2.5.0 to 2.7.4 and MySensors from 2.3.0 to 2.3.2. It's always the same behaviour: It may take the ArduinoIDE a minute, a restart of the IDE or an hardware reset of the NodeMCU to list the device. But in the end, it always connects, uploads and executes the new sketch properly. In any case, I can always OTA-upload manually as soon as the ESP is online, no matter if the ArduinoIDE lists it or not.
I'm not sure what's causing this issue for you, but I honestly doubt that the MySensors framework is the culprit.
The static IP configuration from your earlier attempts might still be retained. Try adding this to the beginning of
setup()for test purposes:WiFi.disconnect(true); delay(1000); WiFi.begin(MY_WIFI_SSID, MY_WIFI_PASSWORD); WiFi.config(0u, 0u, 0u);Or clear the whole flash, in case some other WiFi related stuff may still be stored (Tools > Erase Flash > All Flash Contents).
You may also try OTA-uploading a sketch a different way. From a terminal, this should look like this:
path/to/python3 path/to/espota.py -i 192.168.esp.ip -p 8266 -f path/to/sketch-binary.ino.binOn Windows, you should be able to find the ArduinoIDE-managed python3 in
%localappdata%\Arduino15\packages\esp8266\tools\python3\xxx-version-string/python3and espota in%localappdata%\Arduino15\packages\esp8266\hardware\esp8266\xxx-version-string/tools/espota.py. On Linux and macOS, that should be ~/.arduino15/... or ~/Library/Arduino15/...The ArduinoIDE saves binaries to some temp directory by default, I think, but you can use Sketch > Export Compiled Binary to save it in the same folder as the sketch.
-
Some good news and progress!!
Seems like the NRF power consumption was indeed the culprit. Disabling the NRF in the software didnt do the trick, however when I disconected the power to the module and restarted, after about 10 minutes it appeared in the IDE Network Ports section and I can upload sketches over OTA.
As a result I have trimmed down the connector leads to 2 inches, which still works in my enclosure, added another 10uF cap across the ESP regulator and replaced the one strapped across the NRF module. I have lost all the 'bounces' in the debug log from the ESP and the OTA is available straight away every time the board is power cycled. I wont know for sure until I can get my scope out of storage but I suspect that the original 10uF Cap had become more of a resistor than a capacitor with use over the last 2 years - one for the list during down time over xmas :)
So while I have a fix, the elephant in the room is why did the Arduino OTA sketch work and why did the MYS OTA sketch not work?!?!?!
Does the MYS OTA sketch handle power/ comms/ timing in a different way to the Arduino OTA sketch which could cause this problem? or, Is it a timing thing and by this I mean is it a race condition on the power between the ESP core working through its startup and the sketch and the NRF turning on and the causing the power to bounce?
-
Some good news and progress!!
Seems like the NRF power consumption was indeed the culprit. Disabling the NRF in the software didnt do the trick, however when I disconected the power to the module and restarted, after about 10 minutes it appeared in the IDE Network Ports section and I can upload sketches over OTA.
As a result I have trimmed down the connector leads to 2 inches, which still works in my enclosure, added another 10uF cap across the ESP regulator and replaced the one strapped across the NRF module. I have lost all the 'bounces' in the debug log from the ESP and the OTA is available straight away every time the board is power cycled. I wont know for sure until I can get my scope out of storage but I suspect that the original 10uF Cap had become more of a resistor than a capacitor with use over the last 2 years - one for the list during down time over xmas :)
So while I have a fix, the elephant in the room is why did the Arduino OTA sketch work and why did the MYS OTA sketch not work?!?!?!
Does the MYS OTA sketch handle power/ comms/ timing in a different way to the Arduino OTA sketch which could cause this problem? or, Is it a timing thing and by this I mean is it a race condition on the power between the ESP core working through its startup and the sketch and the NRF turning on and the causing the power to bounce?
Nice work @GardenShedDev
About why this happens: creating different scenarios and doing lots and lots of testing is probably the only way to find out. Or connect the scope (or something like Otii) and try to match voltage/current event to parts of code.
-
@mfalkvidd - your response got me thinking about my question some more - the effort required to diagnose/ rework would for little gain, i am thinking its simpler to rework the psu and seperate the NRF from the ESP - its likley that the ESP regulator is only spec'd to power the ESP and not a lot else - lets face it we are not dealing with miltary grade components here lol :)
Thanks for the help and support!
