Ethernet/WiFi-Client Gateway enhancement

[FotoFieber]

Today I was playing with MY_GATEWAY_ESP32. I like to use several ESP32 talking to node-red with a client connection (MY_CONTROLLER_IP_ADDRESS). To my understanding I have to open one port for every ESP32, to distinguish the different ESP32 gateways. This is not a nice solution.

I suggest to add a define
#define MY_API_PREFIX "example"
and to send this prefix with every message.

Are there better solutions?

[mfalkvidd]

@fotofieber doyou know if controllers in general understand the suggested prefix? Or would all controllers need to be changed to support this feature?

An alternative might be to use mqtt, where you can set one of these (or both)

#define MY_MQTT_CLIENT_ID "mysensors-1" 
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out" 

Documentation https://www.mysensors.org/apidocs/group__GatewaySettingGrpPub.html

[FotoFieber]

@mfalkvidd
Yes, this could be a solution.

When using gateways as 'nodes', I miss the automatic ID handling (I_ID_REQUEST...) and registration in the controller. I have to configure something in the controller for every new node.

This could maybe solved by a new hal/transport/ip.

As the ESP32 can act as AccesPoint and WifiClient at the same time, he could be a gateway offering his service to nodes via his AccessPoint on a separate WLAN.

[Peter Loeffler]

@fotofieber
I am exactely at the same point!

using the MY_GATEWAY_* as a node is a big big issue, when you want to have a "higher" number of those. each of these devices has to be configured as a gateway in whatever controller you use.

following some other posts, the developers seem to have no view on ip-based sensors, and the word "gateway" is a little bit misleading.
the mysensors framework is a perfect framework for sensors though.

the AP-feature of ESP32 ESP8266 e.a. devices is, as you stated correctly, a perfect way to build a wifi-based backbone for such devices.

Some days ago I started to run exactely into this "problems" and as a easy to use "workarround" I tried the following:

I use a ESP* devices as an access-point and run a tenetserver on it. all the traffic is then bridged to one of the serial interfaces, which is (in my case) connected to a "mYsensor Ethernet gateway" , that uses a "fake" rs485 on its inside serial. those two serials are connect via crossed-over TX/RX lines. (see this sketch and activate the swap option)
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/WiFiTelnetToSerial/WiFiTelnetToSerial.ino

next "problem" are the ip-based client-nodes. I also expected that the "gateway" would register itself and all the nodes behind on the "outside" interface, but thats not implemented. SO the only way to gt that working is:
ESP*-IP-Client node (aka bridge):
Start wifi and connection to a AP.
Defina a Softserial device (with loopback)
Start a telnet-client(!!!) on the device and bridge all input/ouput to a that device.
Use MySensors framework (MY_RS485) and define the use of Softserial as the device (the same softserial as defined above).
THESE clients can now connect to

) an MySensors network either with conneting to a MY_ESP_GATEWAY or MY_ETHERNET_GATEWAY doe to the fact, that the IP-connection is nothing else but simple "serial over ip" if you want to name the child like that.

*) to a "bridge" like described above

maybe thats not teh nicest solution but a way to get serial transport over ip running.

CAVEATS:
*) "repeating nodes" could generate a denial-of-service by repeatedly re-transmitting the same packets
*) Wifi-AP and Wifi-Client on the same device is a bad idea: the Wifi itself is something like a hub (and definately not a switch in the ip-network-speaking), what means: you bring additional traffic to that hub (aka HF-channel) and screw down the performence

one more (future) thoughts:
maybe someone should try something like a "IP-Concentrator" or "Arduino based Controller" functionality. That means:
*) IP or mqtt-Gateway-functionality as already possible with mysensors
*) MySensor Gateways can connect to that devices via TCP. Incomming traffic needs to be "rewritten" for example NodeId of the connected GW needs to be altered into the last byte of the IP-address and vice versa

I am very sure that somebody already made this, but the difficulty is (as far as I see) the nameing of funtionality: GATEWAY vs. CONTROLLER vs. NODE vs.BRIDGE ...
The main argument against "a MySensor Sensor is always ONE Gateway" is, as you already stated, the amount of gateway-connections you have to maintain.

a last thought:
I have not played arround much with Nodemanager, but I think that "concentrator" functionality as well as "SerialOverIP" transport could maybe be easyly done there

cheers
Peter
(wenns zu unverständlich war, erklär ich das auch gern nochmal in deutsch)

EDIT: https://github.com/paulo-raca/ArduinoBufferedStreams/blob/master/examples/LoopbackBuffer/LoopbackBuffer.ino

[mfalkvidd]

@peter-loeffler a MySensors ethernet gateway works exactly like a MySensors serial gateway, except that the data (text strings) are sent through TCP instead of through the serial port. To me, this is SerialOverIP. Could you clarify how "your" SerialOverIP is different?

[Peter Loeffler]

@mfalkvidd
its not different, its the same....

maybe its my idea of a "gateway" that makes the problem

maybe you could follow me with this example:
lets say there is a gateway with MySensors and MY_GATEWAY_ETHERNET and MY_RS485.

The RS485 bus runs through some cooling-houses (they are farraday-cages). in every house there is a arduino-based MySensor MY_GATEWAY_RS485 and MY_RF** to collect the local sensors.

(I hope its that, what we both call a tree)

will this work?

[mfalkvidd]

@peter-loeffler I think it will work. Just add all the (MySensors) gateways to your controller. Most controllers support multiple (MySensors) gateways. I have 3 (MySensors) gateways connected to my controller.

[mfalkvidd]

Note: in MySensors, there is no rs485 gateway.

There is a rs485 transport though.

[Peter Loeffler]

@mfalkvidd RS485=serial

as you mention that "multiple gateways" are then presented to the controller, that makes everything clear!!!

what would you name a device, that connects 2 physically different "networks" ...

....as long as this device DOES changes the data-packets (setting parent_id as the devices node_id in the higher network)? (for me this is a GATWEWAY)

....as long as this device DOES NOT change the data-packets? (for me this is a BRIDGE)

so there is to be resumed: MySensors can (out of the box) not buildup TREES
(thats a big big pitty)

[Peter Loeffler]

@fotofieber said in Ethernet/WiFi-Client Gateway enhancement:

This could maybe solved by a new hal/transport/ip.

As the ESP32 can act as AccesPoint and WifiClient at the same time, he could be a gateway offering his service to nodes via his AccessPoint on a separate WLAN.

another solution could be:

add a defintion like MY_GATEWAY_ROUTING

#ifdef MY_GATEWAY_ROUTING
getId() ... at/of the interface of "MY_GATEWAY" and set it as "parent_id"
#endif

[scalz]

@Peter-Loeffler
Not completely sure if there is any overhead in your whole archi, but I think now I better imagine how it looks.

Regarding your previous comments, it seems you didn't know what exactly MySensors provides for the moment, and had maybe too much expectations??

Still there are always different ways to do things !

Let's try to clarify some points (as far a s I remember):

• MySensors is a framework which allows you to build a tree network of RF sensors nodes, repeaters and gateway. That's what we call a tree (vs mesh) https://www.mysensors.org/about/network
  a gateway is sort of concentrator if you prefer (still in people mind a gateway connect something to another thing, same as some others commercial brands do)
• MySensors first gateway was proprietary RF nrf24/rfm69, no wifi, <-> Serial (a gateway from one to another protocol)
• then appeared proprietary RF <-> basic/simple ethernet plain tcp
• then appeared proprietary RF to ethernet tcp MQTT
• esp8266/esp32 can be a gateway (not a node)
• as an extended library/framework MySensors provides examples. Certainly they are too simple for your "professional" usecase but usually frameworks/libs around only provide simple examples. then people do their cooking on top of that. I think MySensors has a lot of examples (those provided by community included).
• as core team is doing this freely, in hobby time, it is hard to cover all usecase and protocols, add new featuresetc.

So users can publish their work when they want to share etc..

What you're asking is an extended usecase which is not covered yet.

I'm sure there is certainly different ways of doing what you need.
for example, here I also use esp32s, some in AP mode too, with http server for webapp, websockets server for comm with ethernet clients (can be used for bridging), serial cli&proxy, RF transports etc.
It's an example of what's possible when coding by yourself (private local code for the moment..as I have no time for providing support/users wishes, also very busy with real life job).

So you can:

• use mfalkvidd solution
• use nodered and do some coding for bridging, if it fits your archi
• code your own bridge on top of MySensors gateway examples
• modify MySensors and submit PR if you need
• wait someone release in future something which fits exactly your need

[Peter Loeffler]

@scalz please don't get me wrong! I choosed MySensors because its a great framework!!! (and I thank every single programmer for his job)

....but:
why is there a "feature requests" section, when you tell me

a) write yourself whatever you want
b) take it as it is

as MANY others already asked in this forum, there is (not only me) a always re-appearing demand for enhanced hardware/tech. (eg IP)
so you would def. make several peoples happy, if we could use THE SAME MY_CONTROLLER_IP_ADDRESS and MY_PORT on a node-only, with the same transport that is used in the MY_GATEWAY_(tcp/ip-network)

[scalz]

@peter-loeffler said in Ethernet/WiFi-Client Gateway enhancement:

@scalz please don't get me wrong!

I just "clarified" things. In your posts you were debating about how this stuff should be named or not, how should work a gateway example firmware, what features should have been provided etc. a picture of your archi needs would have been more useful.

....but:
why is there a "feature requests" section, when you tell me

a) write yourself whatever you want
b) take it as it is

as MANY others already asked in this forum, there is (not only me) a always re-appearing demand for enhanced hardware/tech. (eg IP)

your turn! trying to get me wrong. kidding

what's a feature request section in a non remunerated opensource project with limited resources?
sort of "todolist" !
I don't think the context implies any duty nor deadline.
That's why community is patient when they need something for free. Or they find workarounds if they are in hurry.

Will a feature request fullfill your need in short term. I don't think so..
If you need this for a professional project and are in hurry (you mentioned many greenhouses)
do you have better options than a) or b) ?? maybe
c)if really in hurry, hiring someone
d) if no hurry or you don't feel comfortable with coding, wait for community or future releases, try mfalkvidd solution or nodered

usually people chooses d)

Releasing takes time then support..free time (time==money..) that I do not have for the moment.

I really hope you'll find a solution/workaround for your project

[Peter Loeffler]

@scalz so to be clear at the end: if THIS is the developer's sight of a "discussion" I am really out of it...

(and after all that I know, why people dont share the own developments on some point of the code, because of rude guys stopping "discussions")

so @admin : please close that topic

[tbowmo]

@Peter-Loeffler

What @scalz is trying to explain, is that we are not doing mysensors as our daytime job, we do have a "normal" job to attend to, in order to get money to pay the bills. And we have families that demands our presence when we are at home, and then we have mysensors and other projects (house rebuilding, old cars / tractors etc) during the small sparetime that is left.

In other words, we are not super humans, that can work 30 hours a day

The feature request forum part, is there so others can chip in on the development, and we can discuss features that might be a good idea to implement. But it doesn't say WHO should implement it..

You are welcome to create a PR with the improvements that you need. After all this is an opensource project, which means that the source is available to be hacked on, it doesn't need to be one of the few core group members that do all the code changes. We are happy to do code reviews as well on incomming PR's.

[zboblamont]

@tbowmo @scalz @mfalkvidd Your patience is amazing...