Ethernet Gateway problem

[hek]

@Dan-S.

Yes, exactly. And un-comment SOFTSPI-define in RF24_config.h

[Anticimex]

Correct. And I forgot to mention that you also need to include digitalio.h in you gateway sketch as well in order to compile.
I have had some promising progress with my software patches. I hope to bring good news by the end of the week.

[Dan S.]

In doing research on this I came across:
https://github.com/nicoverduin/rf24
Didn't know whether you all considered his implementation, but appears to have the same objective of allowing for softSPI on the radio.

[Dan S.]

Am getting a compile error on this line in RF24.h:

SoftSPI<SOFT_SPI_MISO_PIN, SOFT_SPI_MOSI_PIN, SOFT_SPI_SCK_PIN, SPI_MODE> spi;

error: ISO C++ forbids declaration of 'SoftSPI' with no type
error: expected ';' before '<' token

[Anticimex]

@Dan-S. have you included digitalio.h in your sketch?

[Dan S.]

Double checked. Yes I have.

[Anticimex]

Hum...then I cannot think of anything more than that I had to change (because I made that mistake and the error message looks familiar).
But if you are patient a while longer, I will push a patch to githup with a fix for the ethernet gateway sketch so it works with HW SPI. I have confirmed it now.

[Dan S.]

Figured it out and had a successful compile. Did not have my digital IO Lib properly set up. Now to download and test on the gateway!

[Dan S.]

Compiled successfully and downloaded. But doesn't show any signs of operation when plugged in. Guess I will wait for your patch.

[Anticimex]

My patch is now merged to the development branch.
Note that I had to increase the SPI clock divider to get it to work on my breadboard. This might not be needed for everybody, but in case you cannot get it to work, it might be worth a try on that.
For details, see here:

[Dan S.]

I looked at your new gateway sketch and noted you use pin 4 f0r Ethernet enable. According to arduino Ethernet shield documentation, pin 4 is used for writing to the sd card on the shield. You select the W5100 with pin 10 and the sd card with pin 4. So if you have an Ethernet shield, should you be using pin 10 to select and deselect the W5100 instead of pin 4 as you do in your breadboard setup?

[Anticimex]

@Dan-S. You are probably right. I use no shields, just a Nano, so I am free to use any available pin. But I have left this configurable so it is possible to change which pin to use if needed.

[Anticimex]

@Dan-S. I checked again, and I actually think you mixed two things here. I use pin 10, it is the slave select signal for the ethernet shield/module.
It is not to be confused with the SPI_EN signal, which is more of a free-standing GPIO. It could be that having that signal on D4 makes it incompatible with certain shields, but you cannot use pin 10, because that is reserved for SPI slave select on the ethernet shield (and therefore the library).
SPI_EN and SS is not the same thing. And on a Nano D4 is the only one left unless you want to use an analog pin, one of the UART pins or one of the interrupt-pins. I picked D4 since I use a Nano (and the MySensors site reports the Nano as the "officially supported" option). But I did try to make it possible to change this without having to hack the code, so I hope you can still make use of the patch (or create a new pull request for @hek to consider).

[daulagari]

Did get in my W5100 Ethernet Module yesterday and checking some documentation I think the key to get things working is:

• Check if the Slave Select signal is connected to the SPI_EN pin (pin 31) of the W5100 or the /SCS pin (pin 29).
• If connected to /SCS pin (pin 29) sharing the SPI bus will not work
• If connected to SPI_EN pin (pin 31) sharing the SPI bus should work but SPI_EN is active high where /SCS is active low
• Make sure the wires connecting the module are not too long or clock down the SPI clock

I ordered the W5100 Ethernet Module as on the website from alice1101983 and got a red board with funduino on it with pinning:

Checked the board but I see no invertor IC on it so I think NSS is connected to /SCS pin (pin 29). Too bad.

I think the shopping list should be updated to a board that has SPI_EN pin (pin 31) connected.

[Anticimex]

Hm. But I ordered my W5100 modile from the "store" and it has the SPI_EN signal. But I do not think it has POE on the other hand. I think it since then has been replaced with a different module on the store though.
This is the one I got.

[Anticimex]

@daulagari I have seen HW patches that solves the SPI_EN problem by putting an inverter on the SCS signal, thus automatically enables the SPI bus on the W5100 if SCS is asserted. That should have been put on the module in the first place, but for those who are unfortunate to have a W5100 module without the SPI_EN and no "management" of SPI_EN, that might be a option to consider. A description for how to modify the "standard" shield is here.

[daulagari]

@Anticimex : Yes, the module I got is definitely different.

A hardware patch will be very difficult I think because the pins of the devices are spaced very close to each other and I do fear the SPI_EN pin (pin 31) is not going to a net. Will have a look Monday in the office where we have a magnifying binocular.

But, suppose I have access to the SPI_EN pin, then I would hope an additional invertor would not be needed, Arduino should be able to invert the select signal I hope.

[Anticimex]

@daulagari Yep, if you can get SPI_EN on a pin, the Arduino can manage it. That is what my fix has implemented. But it "cost" one Arduino pin and adds some complexity to the sketch (if one does not pay attention, moving around stuff in the sketch might break the signal management with ethernet vs rf).

[daulagari]

On:

I think the shopping list should be updated to a board that has SPI_EN pin (pin 31) connected.

Looking at ebay I think for now the Ethernet Shield W5100 is a better option:

• SPI_EN pin (pin 31) connected to the /SCS pin (pin 29) using an invertor
• Additional micro-SD card slot
• Even cheaper

Drawback is the size but that should not be a problem for a gateway I think. It is a pity shipment takes long.

I you want the same form factor, the WIZnet W5500 Ethernet Module - WIZ550io seems a better choice. Can not find it on ebay but at Antratek the price is reasonable.

@Anticimex : Thanks! Good to know what exactly your fix is doing, did not have a look it yet.

[Dan S.]

@Anticimex Thanks for correcting me, still learning all the hardware idiosyncrasies. I always thought I had some hardware issue since my gateway would work for about an hour and then quit. Yesterday my radio (with antenna) died completely. Replaced it with a standard radio and the gateway has been working continuously since last night. So I must be one of the lucky ones to have a ethernet shield that works with the standard sketch (which I reverted back to)

[Dan S.]

I take that back. Looks like the radio not working again--just lasted longer this time. Will make another try at soft spi.

[Dan S.]

My soft spi version is up and running. Didn't work at first (Vera lua errora) till I read that you moved gw.begin in setup() to be done after the delay(1000). After doing that I was up and running. Will keep my fingers crossed for long term reliability.

[Dan S.]

@Anticimex Was wondering how you knew/diagnosed that it was necessary to move the gw begin to after the delay to get the soft spi to work

[Anticimex]

@Dan-S. I actually stumbled upon that when I was experimenting with the SPI_EN signal and temporarily disabled ethernet or rf24 to see which interfered with what. I never went into why this had to be done because after I went back to HW SPI and rebased my library to get the latest updates, I got it to work without switching the order as well.

[Dan S.]

Well if it wasn't for that stumble, my softSpi Ethernet gateway wouldn't be humming along like it is right now. If the softSpi is a solution for all Ethernet shield users, regardless of the version and SPI_EN hookup, would recommend a more formal documentation of the procedure. Maybe a whole separate softspi version of the library only for creation of the ethernet shield gateway that can then be discarded and replaced with the normal library for all the sensors. Those doing breadboarding or with known compatible shields can opt for the hard wire version.

[Anticimex]

I suspect the re-ordering is only necessary for soft-SPI variants, possibly due to some unclear dependency in the init of the IO. I agree that a soft-SPI alternative RF solution could be a useful complement to the library.

[Dan S.]

After 24 hours of continuous operation with the softSPI version of my ethernet shield based gateway with no hiccups, I'm ready to declare victory and move on!! A special thanks to Anitcimex, without whose help it would not have been possible.

[Anticimex]

Great news Dan
I have also verified my gw on breadboard and am now working on the final product. I'll publish both HW and SW design on the forum. It will feature a new twist on the gateway I don't think anyone have done before

[Dan S.]

Hate to keep this thread going but although I focused mostly on the SPI end of the problem here, I found out I also had a radio problem--in fact all of my problems may have been related to the radio. I thought I was doing good in powering the antenna version of the radio using the AMS117 to supply 3.3 v rather than the board supplied 3.3V. Found out that when I did this the radio would quit working after a while. When connected directly to my UNO 3.3v supply (and the uno has enough power for the radio) it works flawlessly. If anyone has any insights on this would appreciate a response.

[RJ_Make]

@Dan-S.

Noise? Ripple? Overheating? What kind of circuit did you build around it.?

Looking at the datasheet; pages 4 and 5 may be of some help.

ds1117.pdf

[Dan S.]

Using the AMS117 module shown in mysensor store. Vin connected directly to the same power line powering the Uno, ground to uno ground and Vout to the radio whose ground is also to Uno (common ground for uno, AM117 module and radio). Power supplied by 5volt, 2amp wall wart.

[RJ_Make]

@Dan-S.
Capacitor across vcc and gnd on the radio?

[meanpenugin]

@ServiceXp
I believe the part that is at the store already has the supporting circuitry on the board as described in those pages. I don't think there is any external components necessary.
Edward

[Dan S.]

I do have a cap on the radio in addition to whatever is onboard the part. Also the UNO power is connected to through the USB connector if that affects anything.

[RJ_Make]

@meanpenugin
Yep, I thought he was using the chip.

[RJ_Make]

@Dan-S.
What size cap? In testing I found I needed a pretty big cap (47uF) to get mine working with MySensor version 1.4.

[Dan S.]

Thought about this some more. I supplied power to both the radio and the Uno from the same wall wart 5v power source. But when the Uno is supplied 5v from its USB connector, the power from that connection is not regulated by the UNO. Since the radio has significant fluctuations in power output (and demand) it may cause significant ripples in its power source line--the same one which sources the UNO. Because the UNO is not regulating this power, it may have caused problems with the UNO (not the radio). Perhaps I should have capped the UNO power source also or at least should have had 2 separate power supplies, one for the UNO and one for the radio. Thought I was being clever in using one. At any rate the gateway continues to run with no problems with the radio getting power directly from the UNO's 3.3V output pin. The UNO provides enough power from this pin to power the radio with no issues.

[gregl]

Thanks everyone esp. @Anticimex for your troubleshooting on this.
I didnt realise i had an issue with my 1.4 Ethernet GW ( using the iBoard hardware) as i wasn't using it in "production" and since it was just running from my PC's USB port it was restarted each time i woke my PC.

Anyway - to bring stability to the Iboard , i just added to the original sketch the line:

SPI.setClockDivider(SPI_CLOCK_DIV2); //to assist with gateway stability.

eg:

void setup()  
{ 
 SPI.setClockDivider(SPI_CLOCK_DIV2); //to assist with gateway stability.
  // Initialize gateway at maximum PA level, channel 70 and callback for write operations 
  gw.begin(RF24_PA_LEVEL_GW, RF24_CHANNEL, RF24_DATARATE, writeEthernet);
 
  Ethernet.begin(mac, myIp);

  // give the Ethernet interface a second to initialize
    delay(1000);
  
    // start listening for clients
    server.begin();
  }

Thanks again!

[Anticimex]

@gregl You're welcome
Actually, I read somewhere that SPI.setClockDivider(SPI_CLOCK_DIV2) is a default setting in the device, but maybe various AVRs differ.
In any case, I found that when breadboarding, it might actually be needed to lower it to DIV4 as well.
Eventually, SPI.setClockDivider(SPI_CLOCK_DIV2) will be executed when SPI is configured in the library, but possibly some SPI calls are made without this call since you notice a difference when adding the call yourself before calling any library function.

[Dan S.]

Still having an issue with my ethernet gateway (Uno plus shield). Works flawlessly for up to 24 hours, then shuts down. When working, responds without fail to every message from my test light sensor, but then fails to respond to any message. If I take the gateway offline, then restart everything it goes through the same process--working for some length of time, then shutting down. Have tried soft spi, separate power for the radio etc., and each time I think I have found the solution till it shuts down again. Any suggestions. Thinking that it is a hardware issue given that it operates fine for hours before stopping.

[Anticimex]

Hm. Perhaps dodgy USB circuitry? (for powering the Arduino that is) assuming you do that. Have you tried alternative power supply for the Arduino as well?

[Dan S.]

@Anticimex Think I tried more than one power supply, but just to be sure will try again. Also plan on trying another uno board to eliminate that.

[Johnny B Good]

Hey Dan,

Did you got it working ?
Ik had some REALY REALY strange problems here with my MQTT gateway, who hung for the most unexplainable reasons, sometimes it worked a few hours, sometimes a few minutes, sometimes it hung when I added a new node (????) ... so no node worked at all anymore after that ...

I tried a lot of things, I wasn't sure about the power too here so I used a different power circuit with a cheap LD33V for the NRF only, but that didn't help much either (zucht)

I take the project from my hobby-room table to the places it suppose to work and back for some xXx times.
It works well on the table every time, but when installed I got the weirdest problems ... well ...
I'm not sure it's a solution for you too, I can't explain how it helped me, but It's now stable for a time now ...

What did I do ? I just disabled the DEBUG'ing in MyConfig.h O_o for trying out, and it's working for me so far...
Don't ask me how or what, cause I'm not that smart, it was just a desperate try
Maybe the serial was fine when connected to the PC for debugging on the table and was NOT connected when I placed it back, maybe the serial buffer was somehow messed up ... I don't know, maybe the Arduino got just a little bit more memory when debugging is disabled ... really, I can't explain it.
Let's blame it on the ozon and sun flares ...

If you want to know (it wasn't my problem here) how I powered my MQTT, just let me know and I make a Fritz diagram...
And maybe can a more geek then me explain this with this fresh look on the problem.

-= GreetingZzz and may the sensors be with you =-

[hek]

Enabling DEBUG increases program space. Could be just enough to tip things over.

[ntruchsess]

DEBUG also enables use of Serial which causes the 3.3V Voltage to drop a bit by causing power-consumption of the USB-chip (not relevant when using external power-regulator for the nRF24L01).

[Dan S.]

@Johnny-B-Good I'm pretty sure my problems are power supply/radio related as others have suggested. It doesn't seem to lose connection via the Ethernet with Vera. Its the radio connection to the sensor that eventually fails.
Am now testing with the UNO directly supplied by a 9v power supply which says that it is regulated. Was using 5Vusb input before. We'll see how it goes. I do have a cap on the radio, but if this fails new power supply fails I'll look at other alternatives to help the radio operation be more reliable.
I did have debug turned off when uploaded the Ethernet sketch.

[Dan S.]

Looking for some debug help here. Have tried different boards, separate power supply, soft spi, etc. Ethernet controller words for a period from a few hours to up to a day, but eventually stops communicating with test light sensor. The controller still communicates fine with Vera since I receive no error messages on reload or when include button is pressed. But when I serial monitor the sensor it says failed for every message it sends to the controller. The sensor shows an ok for every message up to the failure point and then ir shows fail for every message thereafter. If I unplug the gateway and plug it back in it starts to work--for a while.

[Anticimex]

Do you get anything using netcat?
Assuming you use linux:

   nc <ip of gateway> <port>

It essentially allows you to monitor what the GW sends on ehternet exactly like a serial line. That is, you can expect it to print a message there if you press the inclusion button.

[Dan S.]

@Anticimex Don't use linux but am certainly willing to learn. I do need some way to monitor what is happening with regard to the gateway. It looks like that is what you are proposing.

[Anticimex]

@Dan-S. You can use PuTTY if you like as well.
My proposal is that you establish if it is the GW that stops sending data on Ethernet or the controller that stops receiving it.
Example:
You run your GW until you think it has died (controller stops reacting to sensor input).
Then you connect using your monitor (PuTTY or NC). If you get anything there then you can determine if your GW receives sensordata or not. And you can also determine if GW is alive (if you have a inclusion button). When pressed, it should show in that monitor.

What workaround are you currently using for your ethernet module? SOFT SPI or SW patched management of SPI_EN?
SW patched management of SPI_EN works with this module but not this module.
If you use any variant of W5100 you need some workaround due to the W5100 hogging MISO. Supposedly there are shields that mitigates this flaw in HW (using inverter on CS signal to handle SPI_EN) but I have none of those so I cannot confirm which ones.

The SPI problem can potentially hide (luck and timing play a role here) for a while, and that could be what you experience.
Other factors I have noticed when I debugged my module was that DEBUG flag can have an effect (I am again guessing timing) so if you have that enabled, try disabeling it). Also, the HW SPI bus speed could be too high (if you have "dirty" wires) and that can be worh a try to lower as well (see here).

[hek]

@Anticimex said:

What workaround are you currently using for your ethernet module? SOFT SPI or SW patched management of SPI_EN?
SW patched management of SPI_EN works with this module but not this module.

@Anticimex

What do you say about just have one ethernet sketch always running SOFTSPI for radio. This should be working on all variants of W5100 ethernet moduls/shields?

[Anticimex]

@hek I think it should be a sufficient solution. Personally I am going to stick with my own patch though because I am going to need every byte program memory available. But generally speaking SOFT_SPI should satisfy most ethernet gw users.

[Hausner]

Dan S.
Not to hijack your tread, but I've had similar problems. First the "check wires" thing, next dropping out at random times.

I started with this R3 and this shield which is a ENC28J60, but couldn't get it to work properly.

Then I bought this shield , plugged it in, and uploaded the Ethernet GW without mods, except IP of cause, and it has worked ever since.

[Dan S.]

@Hausner My board looks like the one you got to work. I am back up and running with soft spi and separate power supply for the radio. Will see how long it works. Had stopped running soft spi since it seemed to work without it, but Anticlimax says the SPI problem can hide, so I put the soft spi back in. Wish mine was plug and play like yours. I have a genuine Arduino UNO and W5100 shield, but the knockoff version of each that I have acted the same way. Anticlimax: I do believe its a radio problem since I get the OK at the end of the sensor message line on the serial monitor even if the gateway is not connected to the controller but when it stops it says fail at the end of the sensor line, which to me means either the gateway radio is not receiving and/or sending when it fails.

[Anticimex]

@Dan-S. I assume you refer to me (Anticimex) and yes, the problems I have found with w5100 manifest themselves through the radio. It is the radio that suffer from the misbehaving w5100. But by monitoring the tcp traffic you can see what happens in the gw even with debugging off (which might also be a factor). So by looking on the tcp traffic (NC or putty) you can see if radio messages reach the gw or not. If not then the radio has been knocked out (or your sensor is not getting its messages through).

[Dan S.]

@Anticimex Sorry about the misspelling. I'll blame it on the spell checker. It's up and running now for a couple of hours on soft spi plus separate power for radio, and in the meantime I'll look into NC and putty.

[karenbobiv]

I'm having the same problems described and I'm using the same hardware. Very frustrating. I previously tried the Serial Gateway, but it continuously dropped contact with Vera and now this with the Ethernet Gateway. I'm only trying to get 3 temp sensors connected. I'm a newbie, so hope someone with more experience has some ideas. Thanks.

[karenbobiv]

I'm having the same problems described and I'm using the same hardware. Very frustrating. I previously tried the Serial Gateway, but it continuously dropped contact with Vera and now this with the Ethernet Gateway. I'm only trying to get 3 temp sensors connected. I'm a newbie, so hope someone with more experience has some ideas. Thanks.

[korttoma]

This is probably unrelated but I thought it was worth mentioning. I had similar symptoms in a node (working great for up to 20h then it stoped, restarted it and again working for some time and then again stop). I fond that I had accidentally shorted the primary and secondary side of the regulator for the radio so the radio var running at 5v.

[Johnny B Good]

Hi Dan,

Let me see ... try something like, connect it to another hardware router / switch.
Try change the IP adres on the Ethernet Gateway.
Why ? My experience is that my Fritz-box is not registering/showing ANY ethernet shields with Arduino, (don't ask me why) so if that happens and your DHCP box isn't seeing your Arduino it can give mistakenly the same adres to another device (I have actually see it happend here)
Maybe it's a common problem, I really didn't looked it op because I reserve a static Arduino IP-Range outside the range from the DHCP box.

Just try to help you by looking from other directions to the problem, I know it's very unpleasant dealing with such problems.
(It's still working here btw. disabling debug)

[Dan S.]

Been running fine now for the past 17 hours in the latest test. Have my fingers crossed but have been fooled before in thinking it was fixed. Time will tell.

[Dan S.]

Going on 24 hours of continuous operation (with fingers still crossed). Operating with soft SPI and separate radio power source (the 2 biggest issues in my opinion).

Been investigating rf24 libraries and see that libraries are set up so that radio will stop if an error is detected in radio operation. Can then only be started be initiating the radio begin function. Since this is only in Ethernet startup code, explains why I need to unplug the gateway to get it going when it fails.

There is a debug option in RF24 to allow user action in event of failure. Would help if this could be used to provide notice of failure since this seems to be a common problem. Maybe even allow for restart of the radio with count of failures so that everything does not come to a complete stop and never to start again till you unplug the gateway and restart.

[hek]

@Dan-S. said:

There is a debug option in RF24 to allow user action in event of failure. Would help if this could be used to provide notice of failure since this seems to be a common problem. Maybe even allow for restart of the radio with count of failures so that everything does not come to a complete stop and never to start again till you unplug the gateway and restart.

Yes, i've also noticed the FAILURE_HANDLING option-define in the latest RF24 releases. Would be good if we could make use of it.

[Dan S.]

Going on 27 hours of continuous operation of the ethernet gateway. Close to declaring success, but having been bitten before....

[Dan S.]

After over 2 days of continuous operation without a hitch, I believe all my gateway problems (all of which related to the radio) are solved. If I had known early on that the radio libraries are set up to stop if an error is detected in radio operation, I would have focused on it from the start.

In the interest of others having similar problems, the successful UNO/Ethernet shield gateway includes:

Soft SPI--enabled using the latest iteration of the mysensor library and Ethernet gateway software which includes the proper RF24 componenets and DigitalIO library required for its implementation. See anticimex's instructions above on how to implement. Think this was the biggest factor in stability.

Separate 5 volt power supply fed through the AMS1117 to provide 3.3v power to the radio (with common ground to UNO/shield). The onboard 3.3V feed cannot be depended upon to provide enough peak current to power the radio.

Also downloaded and used latest Arduino IDE 1.5.8

Finally I can move on to sensor building! Thanks for all the help provided by this forum.

[Anticimex]

Great news Dan! I hope it holds up. I have not noticed any hiccups in my gw, but it is Nano-based and hence uses a different ethernet module (but with my software spi_en fix). But I also run the radio off its "own" regulator. I have posted a topic under "your project" with my build (still ongoing).

[Johnny B Good]

Good to hear Dan...

Hope it keeps going on, as last hope you can add a watchdog to your gateway.

Good Luck

[Dan S.]

Thank you all--still going strong without any hiccups.

[gregl]

Gday everyone - i was just looking up something else to do with the etherten board and read this snippet below...it may help apply to somone who reads this thread in future....
We probablky dont have this issue anyway as we startup radios too.... but anyway... it may help someone..
http://www.freetronics.com/pages/usb-power-and-reset#.VGSniPmUceA

<quote>
"Ethernet Reset IC Delay
If your W5100 Ethernet IC is not 'awake' after your board is powered up or reset, the most likely issue is that your Arduino microcontroller has come out of reset sooner than the W5100 IC, and the sketch has sent the W5100's ethernet configuration out before the W5100 140-280ms reset period has expired.
This is because the Arduino microcontroller has a 65 millisecond powerup/reset delay, the longest available to be set by the fuses and bootloader, and the STM811 Reset IC controlling the W5100 Ethernet IC has a longer reset delay of 140-280 milliseconds.

The fix:

Insert this line as the very first line in the void setup() { function in your project:

  delay( 50 );   // allow some time (50 ms) after powerup and sketch start, for the Wiznet W5100 Reset IC to release and come out of reset.

50 ms covered all boards in our testing, though if in doubt make it 200-250 ms for full reliability. Adding this to the start of setup() does not affect any normal operations or running, it just adds a fraction of a second to the one-off powerup or reset of the board.

</quote>

[djolafritte]

Hello,

I had the same problems and I solved it by using an external 12V power supply created with a RAC02-12SC from RECOM Powersupply.

Regards