Posts made by ntruchsess

@Aloha
sure. You may use as many gateways in a single instance of fhem as the machine is capable to service in terms of cpu, ram and filehandles. What does not work is to use multiple gateways for the very same sensors to archive higher avaiability (or redundancy), so it would be beneficial to use different channels for each gateway to lessen cpu-load on the gateways ans save bandwith on the connection to fhem. If all sensors and gateways share the same channel you will have to assign each sensor to the gateway that suits best (e.g. is positioned with the least distance to the sensor) - fhem will just ignore messages for a gateway a sensor is not assigned to (if not in inclusion-mode).

• Norbert

@Aloha
fhem (or better 'perl') is unable to do a clean reload of a single module without restart of the perl-process. Hence the warnings. Just do a 'shutdown restart' on the web-console.

@Aloha
VAR_1 needs to be mapped for clientid 5:

attr Vatten mapReading_value15 5 value1

(you may choose to use any other identifier of your choice instead of 'value15' - this is just what the autocreate-mechanism is supposed to choose).

Edit: I just noticed that V_VAR1 was missing in S_WATER definitions for autocreate (that is fixed now: http://sourceforge.net/p/fhem/code/7112/)
With this fix it should be sufficient to just do a 'delete Vatten', enable inclusion-mode and restart the sensor. Autocreate should create all required attributes including the missing mapReading. As I didn't test with the Watersensor-sketch please let me know if it doesn't....

• Norbert

@hek said:

I can't seem to find any install instructions in english at the moment.

The commandref is in english:
http://fhem.de/commandref.html#MYSENSORS
http://fhem.de/commandref.html#MYSENSORS_DEVICE

@Aloha, I've just seen your post on fhem-forum. Seems you are not on the latest code, the kind of crash you experience has been fixed on Nov. 09 2014. Please upgrade:
http://fhem.de/commandref.html#update

• Norbert

@b0rmann
PubSubClient and ENC28J60 require more memory than an Uno provides. It should run on a Mega256 though.

For TCP you are right - the boolean return-value is not required as the library does free memory not before a packet is acknowledged or the connection is closed. I guess here I can remove some code from the lib. UDP does not retransmit and would loose packets just because of collisions. UDP should loose packets only when they time out or get dropped due to physical failure.

@Rachmat-Aditiya said:

fhem is more stable but you need to configure everything manually,

configure 'everything' is not entirely true, in respect to MySensors there's an autocreate-mode in FHEM. You will have to manually configure the gateway (e.g. 'define gw MYSENSOR /dev/ttyUSB0'), then activate autocreate (by pressing the include-button on the gateway, or setting the autocreate-attribute on the gateway device), then restart all MySensor-nodes and fhem will create a MYSENSORS_DEVICE-entry for every Sensor that sends presentation-messages while inclusion-mode is active.

Cannot tell about the others, never tried them. But here is my experience with fhem:
in FHEM you find support for all the sketches that come with MySensors lib and may easily configure any combination and number of current defined sensor-, variable-types. Serial and Ethernet-gateway are supported. (MQTT is supported as well (using the mqtt-client-gateway), but the as the MQTT-gateway is not as configurable as FHEM itself the support for custom sensor-types is limitted). In terms of MySensors-protocol it supports the complete functionality of Serial- and Ethernet-gateway (including configurable Support for Acknowledge, so an actor would get a message resend multiple times until it's acknowledged).
FHEM requires very little resources (it even runs on a Fritz-box, so a RasPi is more than sufficient) and is written in perl. The buildin-web-interface is a bit old-fashioned, but there are mobile clients for Android and iOS as well. Values are persisted on restart, they may be logged and rendered as SVG-Plots. Value-changes may trigger arbitrary Events (as being written in perl you may even run your own perl-code as configurable events occour). And last but not least the fhem-codebase is very mature. I run an fhem-server 24/7 on an alix-board (that is also my router, owncloud and print-server) to control my heating-system since nearly a decade now with very, very little downtime. As I usually do very little changes to the setup availability has been better than 99% during those years including all maintenance. For sure availability has been proven better than for the arduino-based ethernet gateways
But as a last word: as administration is not all point and klick a bit of technical interest (e.g. willingness to use a web-based commandline-interface for configuration and maybe some knowledge of regular expressions and a bit of perl) is helpful to explore the full power of fhem.

• Norbert

@frol Thank you for the data. Bad thing TXABRT is not set when timeout occours. I have to give this workaround a second thought. I hope I can somehow avoid the 1sec busy wait. It's not about the 1 sek of having unresponsive ethernet, but it stalls any other processing during that time as well

Did merge your pullrequest so others may test it easily.

• Norbert

@frol Thank you for the data.
Well, it looks as if the workaround for errata12/13 doesn't work as described. TXERIF is never set in the 4 uip_debug-logs :-(, hence the timeout (BTW: it would make sense to return false in case timeout occurs so a packet might be retransmitted in that case. As TXERIF is not set sendPackage returns true in case of error). Strange thing is that even if a packet is not transmitted (and that is not detected) the next outgoing packet will reset transmitlogic anyway - but this doesn't seem to re-enable transmission, does it? (Any output truncated after the timeout?)
Maybe one should poll ESTAT for TXABRT instead of TXERIF (or both at the same time)?

• Norbert

intention is you may controll MySensor-actuators via mqtt-client-gateway by publishing messages to topic MyMQTT/<radioId>/<childId>/<variableType>

@frol thank you the feedback. If it still responds to ping it indicates enc28j60 transmit-logic does not stall and the changes in fix_errata12 branch seem to work. I think the freeze must have a reason that is unrelated to this low-level patch. What sketch are you using for testing?

that test was using Ethernet-gw on wiz5100 shield. Enc28j60 not tested (yet)

so when you intent to use GNU GPL v2 or higher for all code why would you need the right to change to 'any other license' (even to licenses not compliant with GPL v2 of higher)?

I'd like to contribute but will not sign a CLA that allows this 'change to any other license'.

Feel free to pull any changes that I commit to my fork (as you have done with my UIPEthernet-lib):

https://github.com/ntruchsess/MySensors/tree/ethernet_gw
https://github.com/ntruchsess/MySensors/tree/mqttclient

all changes there are (and will remain) licensed GPL v2 or higher.

Ok, I can reproduce this with Arduino IDE 1.0.6.
It did compile fine on IDE version 1.0.5 though.

EDIT:
go to PubSubClient.h and remove the 'PROGMEM' in line 72:

change:
boolean publish_P(char *, uint8_t PROGMEM *, unsigned int, boolean);
to:
boolean publish_P(char *, uint8_t *, unsigned int, boolean);

see issue on github and pullrequest with fix:
https://github.com/knolleary/pubsubclient/issues/46
https://github.com/knolleary/pubsubclient/issues/47

btw: I did improve reconnect of Ethernet-gw: http://forum.mysensors.org/topic/572/improved-ethernet-gateway

have noticed the same here. Upgrading the Gateway to 1.4.1 would receive from Relay-actuator 1.4 fine but The relay-actuator wouldn't receive messages sent by gw.
Upgrading the actor to 1.4.1 MySensors lib did solve the issue.

I did improve stability of ethernet-gateway in respect to reconnect:

https://github.com/ntruchsess/MySensors/blob/ethernet_gw/libraries/MySensors/examples/EthernetGateway/EthernetGateway.ino#L125

Issue with relying on server.available() + server.write() is that it allows to connect with multiple clients. That might be fine with other protocols, but doesn't make a lot of sense as the gateway is by design connected to a single controller only. It also has issues when (re-)connecting more than 4 times in a row without doing a proper shutdown of previous connections as both WIZ5100 and UIPEthernet support only a limited (4) connections in parallel and TCP-timeouts are quite long so the gateway wouldn't notice not being connected to a valid remote-socket any more if you e.g. do a hard power down and reboot of your controller.

The change I made makes sure that whenever the controller reconnects any previous connection is properly removed by calling client.stop().

Works using Arduino-IDE 1.0.6/1.5.7 (or later) as it makes use of operator== in EthernetClient (a change I added this to EthernetClient some month ago for this very purpose...)

@MagKas I did investigate a bit - the implementation in UIPEthernet was based on an older Version of Silicon Errata. Rev B7 has clarified this issue in more detail, in fact Issue 13 contains pseudocode that also should solve your 'deadlock' on 'while (readOp(ENC28J60_READ_CTRL_REG, ECON1) & ECON1_TXRTS)'
The issue is that eventually TXRTS is not (never) cleared by the transmission-logic after package transmission so the while will never exit. As a workaround the code should wait for either TXIF or TXERIF to be set.

Here is my code that I just commited to UIPEthernet (https://github.com/ntruchsess/arduino_uip/blob/fix_errata12/utility/Enc28J60Network.cpp#L233

// Reset the transmit logic problem. See Rev. B7 Silicon Errata issues 12 and 13
writeOp(ENC28J60_BIT_FIELD_SET, ECON1, ECON1_TXRST);
writeOp(ENC28J60_BIT_FIELD_CLR, ECON1, ECON1_TXRST);
writeOp(ENC28J60_BIT_FIELD_CLR, EIR, EIR_TXERIF | EIR_TXIF);
// send the contents of the transmit buffer onto the network
writeOp(ENC28J60_BIT_FIELD_SET, ECON1, ECON1_TXRTS);
// wait for transmission to complete or fail
while (((eir = readReg(EIR)) & (EIR_TXIF | EIR_TXERIF)) == 0);
writeOp(ENC28J60_BIT_FIELD_CLR, ECON1, ECON1_TXRTS);

The retransmission-logic that is described in Issue 13 is implemented outside of sendPacket-method. On transmission-error it returns false, the packet will not be freed in UIPEthernet::network_send() and transmission will be reattempted on next call to UIPEthernet.tick().

I also added a fix that allocated the 7 bytes of transmit-status-vector to prevent corruption of other outstanding packets. Code is in branch 'fix_errata12' (https://github.com/ntruchsess/arduino_uip/tree/fix_errata12). Maybe you'd like to test before I release?

• Norbert

@MagKas

Thank you for this very detailed analysis. That is really a great finding!

• Norbert

I've just implemented a controller for fhem and must say a 'thick' conjtroller would have made implementation noticibly easyer. It's not about parsing the protocol and pulling the values into suitable entities, it's about implementing the low-level details (managing ID-requests, inclusion-mode, acknowlede and retransmit, ping etc....). And only half of this effort was due to things being not propperly documented.

On the other hand a really 'thick' gateway only makes things really easy when implementing a standard interface (like mqtt) that allows to just access sensor-data without having to deal with any of the node-management stuff.

• Norbert

DEBUG also enables use of Serial which causes the 3.3V Voltage to drop a bit by causing power-consumption of the USB-chip (not relevant when using external power-regulator for the nRF24L01).

@Johnny-B-Good said:

Your links don't work..
@ntruchsess said:

https://github.com/ntruchsess/fhem-mirror/blob/mysensors_unified

Thanks for giving notice, I've just merged into master and deleted branch 'mysensors_unified', links are fixed now.

It uses 27.886 Bytes of flash with Debug enabled, 26.014 Bytes without (using a standard WIZ5100 Ethernet-shield).

BTW: I've used Parts of your Perl-code to implement the fhem-controller-module:

https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/00_MYSENSORS.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/10_MYSENSORS_DEVICE.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Constants.pm
https://github.com/ntruchsess/fhem-mirror/tree/master/fhem/FHEM/lib/Device/MySensors/Message.pm

My plan is to implement a full-featured Device::MySensors module that will be independent of but easy to integrate with Net::MQTT or AnyEvent.

I just took the time this afternoon and implemented a mqtt-gateway that works as mqtt-client:

https://github.com/ntruchsess/MySensors/tree/mqttclient/libraries/MySensors/examples/MQTTClientGateway

it is based on the work of @Damme (http://forum.mysensors.org/topic/260/mysensors-mqtt-gateway) and makes use of nick o'learys PubSubClient-library (http://knolleary.net/arduino-client-for-mqtt/)

The main difference to the existing MyMQTT-gateway is that it will connect to an existing mqtt-broker (e.g. mosquitto). This is beneficial if you want to integrate with other mqtt-client-devices sharing a single broker for your automation solution. It also ensures your mqtt-client that is going to pick up the messages from the gateway will talk to a 'real' broker understanding the whole mqtt protocolls semantics

It works both ways:
on startup or reconnect it subscribes to "MyMQTT/#" (configurable as MQTT_PREFIX). Any publish-messages payload that is delivered by the broker to a topic that complies with the sheme <prefix>/<nodeId>/<childId>/<variable_type> is forwarded as a C_SET-msg to the MySensors radio.
Any C_SET-message received from a node is published to the mqtt-broker using the same sheme.

• Norbert

RFduino uses BLE (Bluetooth Low Energy). While this is also 2.4GHz it's not compatible with nRF24L01 in terms of protocol.

@hek said:

If thin things like the Arduino based MQTT gateway would manage without it we could probably remove it from the normal payload.

As MQTT-protocoll is not stateless (it does Quality-of-service with message-storage and redelivery - though the existing gateway code doesn't support this yet) the MQTT-gateway cannot be a thin thing anyway. I wouldn't mind if it doesn't run on Uno due to memory-constraints, there's the Mega2560 or even the DUE which seems to be affordable as you only need a single gateway per install.

@hek said:

Request for metadata is a good idea. Would require sensor to keep this available at all time in memory.

This is allready in memory, just requires a move of send-presentation-messages code into own method that is called from begin and in response to 'request-metadata'.

@Zeph said:

I think that ChildSensorID is an appropriate concept for identification and addressing (ie: included in packets), while "ChildSensorType" is static metadata which should be collected during configuration but does not need to be passed in every packet. So I would simplify in the other direction - get rid of ChildSensorType in the packet addressing heirarchy, rather than getting rid of ChildSensorID.

100% agree. Sending ChildSensorType is just redundant static metadata. Just having implement a MySensor-conroller for fhem I noticed there's no way to request Presentation-messages from Sensors so a Controller after Sensor startup, so the controller has no way to verify it's persistent 'static' metadata without a user going to restart the sensors. This actually is a point that needs to improove.
But the argument a controller would have to be 'thin' (in terms of not being statefull in respect to sensor details) is a no-arg. You cannot do automation without knowing details about the controlled items.

• Norbert

As long you intent to support both 1.0 and 1.5 IDE you should stay with UIPEthernet 1.0.9 as latest 1.5.x IDE fully supports 1.0 library format. 1.59 is suitable if you intent to compile for Arduino DUE (or YUN but that is another story unrelated to UIPEthernet).

If you have any suggestions how this could be improved please let me know. I'm not very happy with this situation either. Eventually Arduino 1.5.x will overcome beta phase so 1.0.x IDE will become deprecated, but for now it seems we have to live with it.

BTW: I've written a MySensors-module for FHEM: http://forum.fhem.de/index.php/topic,26807.msg210234.html#msg210234

Just to let you know, I've released UIPEthernet Version 1.09 / 1.59: https://github.com/ntruchsess/arduino_uip/releases. This release contains both a fix for a condition that caused the ENC28J60 chip to freeze and an improvement that reduces timer-induced latency from 250ms to 10ms aprox.

Another hint when compiling Ethernet-gateway with UIPEthernet:

Set UIP_CONF_UDP 0 greatly reduces flash memory footprint by not including UDP-code (has been mentioned in this thread before)

Reduce UIP_CONF_MAX_CONNECTIONS will lower RAM-usage. As Ethernet-Gateway works as Server with only a single Client that stays connected to the gateway configuring a value of 1 should work fine.

• Norbert