GatewayESP8266MQTTClient with realtime connection params

[Mickey]

Hi
I would like to alter GatewayESP8266MQTTClient.ino a bit to be able to transfer mqtt broker connection params from an HTML5 app I developed.
I altered the core code in the file MyGatewayTransportMQTTClient.cpp to enable SMARTCONFIG:
WiFi.beginSmartConfig();
To be able to transfer the SSID and PASS dynamically from the app.

All the other params(server IP, user/password for MQTT broker etc.) I would send from the app with JSON and the GATEWAY will save them to local storage (the ESP-12 have 4mb for storage).
I do not want to alter core files too much and would like to implement all of the above in the ino file as much as I can.(with the smart config option I didn't see a way other than altering the MyGatewayTransportMQTTClient file.)

The MyGatewayTransportMQTTClient file looks for defined variables to function but I need to change the code to enable dynamic params.

The parameters I need to change are:

#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"
#define MY_MQTT_CLIENT_ID "mysensors-1"
#define MY_MQTT_USER "username"
#define MY_MQTT_PASSWORD "password"
#define MY_ESP8266_SSID "MySSID"
#define MY_ESP8266_PASSWORD "MyVerySecretPassword"
#define MY_ESP8266_HOSTNAME "mqtt-sensor-gateway"
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68
#define MY_PORT 1883

I know how to code the ino file to listen to JSON and then process the data and put it in storage but I cant seem to grasp how to change the core file as minimally as I can...

[hek]

This is possible to solve like this (pseudo code):

char myTopicPrefix[20]
uint32_t controllerIp;

void before() {
     // Fetch settings from eeprom (or some other place)
    strcpy(myTopicPrefix, <something>); 
    memcpy(&controllerIp, <4 bytes representing ip>, sizeof(uint32_t));
   ....
}

#define MY_MQTT_PUBLISH_TOPIC_PREFIX myTopicPrefix
...
#define MY_CONTROLLER_IP_ADDRESS controllerIp
....

[Mickey]

Hi
So I change the order of the core code as I mention and all the WIFI connection is done before the main sketch setup function instead off the MyGatewayTransportMQTTClient.cpp file.
But when I try to use:
WiFi.mode(WIFI_STA);
or
WiFi.hostname(MY_ESP8266_HOSTNAME);
or
WiFi.begin();
before the setup function I keep getting:
error: 'WiFi' does not name a type.
Can I take WIFI.begin() before setup ? (the original core code seems to do it... )

[Yveaux]

@Mickey The variable named WiFi is not known when you call it. Make sure to #include the right libraries before using WiFi. Have a look at some esp8266 examples to see how to do it.

[Mickey]

I included whats need to be included and this variable known when it is inside setup function but I need to call it before setup().

EDIT
@hek I trying to implement your suggested pseudo code but in an ordinary esp8266 sketch WIFI object is only recognizable in the setup function and on but in the GatewayESP8266MQTTClient.ino the setup is empty and the WIFI object becoming recognizable somewhere in the core code long before the main sketch setup function.
I see that it is put to use already in MyGatewayTransportMQTTClient.cpp and MyGatewayTransportEthernet.cpp but in order to receive the mqtt params from user I need to use the WIFI object before including MySensors (#include <MySensors.h>) much like it is being done by the core.(to use the WIFI object it is not enough to include <ESP8266WiFi.h> but also to initiate it in setup).

So how it is done in the core? what one need to include/implement in order to use WIFI in the GatewayESP8266MQTTClient.ino before including <MySensors.h> and before the setup function.

Hope I made myself clear enough ...
thanks

[Mickey]

So I saw that I cannot use WIFI object out side setup and also if I take all the web based parameters configuration code to a separate library its not so different because library initiation can be done only in setup function also.
So I'm trying to include an ordinary cpp code that will perform this before #include <MySensors.h> and before setup. So how do I go and do that should I include the cpp file in the ino file or do I do it like an ordinary library and only include the .h file?

Also I had another problem
@hek said:

strcpy(myTopicPrefix, <something>);

So if I implement this code :

String uniqueID = (String)ESP.getChipId();
#define MY_MQTT_CLIENT_ID uniqueID
char mqtt_username[20];
char mqtt_password[20];
strcpy(mqtt_username, json["mqttusername"]); 
strcpy(mqtt_password, json["mqttpassword"]); 
#define MY_MQTT_USER mqtt_username
#define MY_MQTT_PASSWORD mqtt_password

I keep getting this error:
error: no matching function for call to 'PubSubClient::connect(String&, char [20], char [20])'
candidates are:
boolean PubSubClient::connect(const char*, const char*, const char*)

Is it mean that I can only use const variables and thus cannot put dynamic parameters that I will get from user at run time?

***EDIT
I finally change the MyGatewayTransportMQTTClient.cpp without really touching the ino file...

[Mickey]

So after seeing this post:
https://forum.mysensors.org/topic/5533/esp8266-gw-node-mqtt-transport-won-t-present-sketch-or-actuator-to-controller

I finally succeeded to change the GatewayESP8266MQTTClient that it can load connection parameters dynamicaly and also use smart config to send SSID and password to the ESP.(I using a android app I developped for it).
This is the source code of GatewayESP8266MQTTClient:

#include <EEPROM.h>
#include <SPI.h>
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <ArduinoJson.h>
#include <FS.h>

const int button = 0;//button for reset config
ESP8266WebServer server (80);
File configFile;

boolean configExist = false; 

struct My_Config_t
{
String WLAN_STA_SSID = "";
String WLAN_STA_PASSWD = "";
String MQTT_Broker_FQDN = "";
//IPAddress MQTT_Broker_IP;
uint16_t MQTT_Broker_Port = 0;
} My_Config;

// Enable debug prints to serial monitor
#define MY_DEBUG 
//#define MY_DEBUG_VERBOSE
//#define TRANSPORT_DEBUG

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set this nodes subscripe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires usenrame/password
//#define MY_MQTT_USER "username"
//#define MY_MQTT_PASSWORD "password"

// Set WIFI SSID and password
#define MY_ESP8266_SSID (My_Config.WLAN_STA_SSID).c_str()
#define MY_ESP8266_PASSWORD (My_Config.WLAN_STA_PASSWD).c_str()



// MQTT broker ip address.  
//#define MY_CONTROLLER_IP_ADDRESS 192, 168, 178, 68
//#define MY_CONTROLLER_URL_ADDRESS "test.mosquitto.org"
#define MY_CONTROLLER_URL_ADDRESS ((My_Config.MQTT_Broker_FQDN).c_str())

// The MQTT broker port to to open 
//#define MY_PORT 1883      
#define MY_PORT (My_Config.MQTT_Broker_Port)


bool loadConfig() {
  configFile = SPIFFS.open("/config.json", "r");
  if (!configFile) {
    return false;
  }
size_t size = configFile.size();
  if (size > 1024) {
    Serial.println("Config file size is too large");
    return false;
  }
  // Allocate a buffer to store contents of the file.
  std::unique_ptr<char[]> buf(new char[size]);
  configFile.readBytes(buf.get(), size);

  StaticJsonBuffer<1024> jsonBuffer;
  JsonObject& json = jsonBuffer.parseObject(buf.get());

  if (!json.success()) {
    Serial.println("Failed to parse config file");
    return false;
  }

  char MQTTURL[40];
  char MQTTPORT[4];
  char wlan_sta_ssid[20];
  char wlan_sta_passwd[20];

  strcpy(wlan_sta_ssid, json["wlan_sta_ssid"]); 
  strcpy(wlan_sta_passwd, json["wlan_sta_passwd"]);
  strcpy(MQTTURL, json["mqttbrockerurl"]); 
  strcpy(MQTTPORT, json["mqttport"]);

  My_Config.WLAN_STA_SSID = String(wlan_sta_ssid); 
  My_Config.WLAN_STA_PASSWD = String(wlan_sta_passwd); 
  My_Config.MQTT_Broker_FQDN = String(MQTTURL);
  My_Config.MQTT_Broker_Port = String(MQTTPORT).toInt();
  configFile.close();
  jsonBuffer = StaticJsonBuffer<1024>();

return true;
}
bool startSmartConfig()
 {
    WiFi.mode(WIFI_STA);
    delay(500);
    // Start SmartConfig if necessary
    WiFi.beginSmartConfig();
    while (WiFi.status() != WL_CONNECTED)
        {
          delay(500);
          Serial.print(".");
          if(WiFi.smartConfigDone())
            Serial.println("smart Config Done");
        }
     WiFi.stopSmartConfig();
     delay(500);
     WiFi.begin();
     return true;
}
void handleJson() {

  StaticJsonBuffer<1024> newBuffer;
  JsonObject& newjson = newBuffer.parseObject(server.arg("plain"));
  server.sendHeader("Access-Control-Max-Age", "10000");
  server.sendHeader("Access-Control-Allow-Methods", "POST,GET,OPTIONS");
  server.sendHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
  if (!newjson.success()) {
    Serial.println("Failed to parse config file from handleJson 1");
    server.send ( 200, "text/json", "{success:false,deviceID:'none'}" );
  }
  else{
 configFile = SPIFFS.open("/config.json", "w");
       if (!configFile) {
      Serial.println("Failed to open config file for writing in handleJson 2");
      server.send ( 200, "text/json", "{success:false,deviceID:'none'}" );
        }
        else{
       newjson.printTo(configFile);
       configFile.close();
       //server.send ( 200, "text/json", "{success:true,deviceID:'bla'}" );*/
       server.send ( 200, "text/json", "{ \"success\": \"OK\",\"deviceID\": \""+String(MY_MQTT_CLIENT_ID)+"\" }" );
       newBuffer = StaticJsonBuffer<1024>();
       }
  
  }
}
void startConfigWebServer()
  {
  server.on("/json",handleJson);
  server.begin(); // Web server start
  Serial.println("HTTP server started");
  }

#include <MySensors.h>


void before() {
Serial.println("Resetting WiFi credentials");
WiFi.disconnect();
ESP.eraseConfig();
//spi_flash_erase_sector(0x7e);
if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
  }
  if(!loadConfig())
  {
      if(startSmartConfig())
      {
        startConfigWebServer();
        server.handleClient();
      }
      while(!loadConfig())
  {
      server.handleClient();
  }
  delay(2000);
  server.stop();
  WiFi.disconnect();
  ESP.eraseConfig();
//  spi_flash_erase_sector(0x7e);
  //loadConfig();
  SPIFFS.end();
  ESP.reset();
  }
  else{Serial.println("Config loaded...");
  delay(2000);
  }

  server.stop();
  WiFi.disconnect();
  ESP.eraseConfig();
//  spi_flash_erase_sector(0x7e);
  //loadConfig();
  SPIFFS.end();
  //ESP.reset();
}

void setup() { 
  ESP.wdtDisable();
  pinMode( button, INPUT_PULLUP );
  digitalRead(button) == HIGH;
}

void presentation() {
  // Present locally attached sensors here    
}


void loop() {
  ESP.wdtDisable();
  //Serial.println("inside loop");
  if ( digitalRead(button) == LOW ) {
    Serial.println("Resetting WiFi credentials");
    SPIFFS.begin();
    SPIFFS.remove("/config.json");
    SPIFFS.end();
    WiFi.disconnect();
    ESP.eraseConfig();
    spi_flash_erase_sector(0x7e);
    configExist = false;
    ESP.restart();
    }
   //Send locally attech sensors data here
}

So after a some time I get this esp reset (I tried to put ESP.wdtDisable(); in loop)
this is the serial output:

connected with MYSSID, channel 1
dhcp client start...
..ip:192.168.0.113,mask:255.255.255.0,gw:192.168.0.1
.IP: 192.168.0.113
0;255;3;0;9;MCO:BGN:STP
0;255;3;0;9;MCO:BGN:INIT OK,TSP=1
IP: 192.168.0.113
0;255;3;0;9;Attempting MQTT connection...
0;255;3;0;9;MQTT connected
0;255;3;0;9;Sending message on topic: mygateway1-out/0/255/0/0/18
pm open,type:2 0

but after a few seconds (mainly after a message is sent from a sensor...but not only . if I send a message from broker that is not sending via radio to sensor its more stable ) I getting this:

bcn_timout,ap_probe_send_start
ap_probe_send over, rest wifi status to disassoc
state: 5 -> 0 (1)
rm 0
pm close 7
?)�‏L®‎D‚

And its seems that the ESP is resetting itself...
What do you think I can do to make this more stable? maybe add "__yield" some where? (I think to adding of ESP.wdtDisable helped a little at list the reset is with out the long stack message...)

[Mickey]

Is the
"bcn_timout,ap_probe_send_start
ap_probe_send over, rest wifi status to disassoc"

message is a known issue? I cant seem to find anything with this string when searching this forum...