RS485 Stress test

[AWI]

@LeoDesigner Thanks for you tips. I tried to increase Baud rate but ran into communication problems. At 57600 almost none of the messages came through, at 38400 one of the nano's could not keep up. So 19200 seems to be the max for this breadboard setup.

After removing the debug lines and streamlining my own sketch I am now to the level of 3% failure with random variations (0%-7%). From perspecitive of probability this is explicable.
Reliability is decent compared to radio, but would need another layer of checking where in an environment where no message can be missed.

[LeoDesigner]

@hek said:

@LeoDesigner

You are right, those lines should go away. Will you fix?

It's fixed in my repository, please make changes in mysensors library (in https://github.com/mysensors/MySensors/blob/development/core/MyTransportRS485.cpp)

[AWI]

@LeoDesigner The protocol implementation in your library looks solid. I am wondering if AltSerial could be the cause of the transmission problems. Any clue?

[LeoDesigner]

@AWI
Probably, in my case (I am using hardware serial at 115200) transmission is more reliable because it's not an emulation via AltSerial. Did you try to connect only one node ? I am just curious... In that case should be no lost messages at all.

[AWI]

@LeoDesigner Do you have a serial gateway with hardware serial for RS485?

I have tried the stress test with one device (two on the bus) but still (very limited) errors.

[LeoDesigner]

@AWI
Yes - I am using wired gateway with two nodes (mySensors libs 1.5.2) with my rs485 transport class. I am decided to use only hardware serial for reliability reasons.
I am handling lost messages via NodeRed. Actually I am doing a lot with NodeRed and mqtt. (mySensors<->mqtt gateway, mqtt->influxdb->grafana dashboard, time scheduled and rule based automation, mqtt<-> homeassistant, mqtt <-> custom web dashboard via websockets )

Thanks for testing. I think your errors with one node could be related to AltSerial. RS485 by itself is very reliable.

[AWI]

RS485 Stress test "next level"

I changed my setup to use Hardware Serial (and made it a little cleaner). With the latest changes in development core/MyTransportRS485.cpp that was easy. Baud rate to 115200 and pushed the gas a little...

• 4 nodes (arduino nano) firing 1 message each 200ms (average) on hwserial
• acknowledge on each message
• 1 serial gateway (arduino mega)

observations:

• with 1 node - no errors
• with the 4 nodes - around 5% messages missed with variations between 0 and 9 missed per 100. Not bad considering the speed and chances for collision.

I would like to get to a more (full...) reliable protocol. Looking at the code (MyTransportRS485.cpp) there is some collision detection in place before a message is sent. What I would like to do is have the protocol read the (sent) message and verify if it corresponds to what was sent..

Any suggestions? (@LeoDesigner? )

[LeoDesigner]

@AWI
Thanks for your efforts and persistence :)
Yes, before the packet will be sent, the MyTransport485 will check if there any bytes arrived recently from the serial port. The collision itself can occur during period of the first byte of the packet. (I wish to have time to draw a time diagram for this.)

I think I forgot to mention one more change to the original library:

//The number of SOH to start a message
//some device like Raspberry was missing the first SOH
//Increase or decrease the number to your needs
#define ICSC_SOH_START_COUNT 3

Could you please try to modify this constant ?
It may affect amount of lost packets.
You may also try to increase speed up to 250000 baud.

I have one more idea about collision avoidance:
(In this case we need to make a small modification to the original RS485 module connection, /RE input should be connected to the ground to make receiver always active - then we would be able to receive own messages sent)

We can try to implement the following algorithm:
(Send a packet procedure):
-- check if there any bytes received recently - continue if we have a silence on the bus (done at this moment)
-- send a first SOH (start of header byte)
-- check if it is received by our serial and it's equal SOH.
(so we know at this moment we have no collision on the bus for sure)
-- send next SOH bytes and all packet bytes.

It's like we are sending bytes to the line and reading them at the same time to make sure we had no collision at the same time.
This way we can even resend our packet if collision was detected during packet transmission.

Also we can implement one master and slaves nodes poling mode. But this will not be easy to make in the mySensors protocol architecture.

Let me know what you think about, you probably should have some other ideas too.

[AWI]

@LeoDesigner We are thinking in the same direction. We could do a little tuning to minimize the collisions/ bus errors but I would like to be absolutely sure that a message is sent and could have been received. If we take some measures like you suggested, I think we could guarantee quality even at lower baud rates and using altsoftserial. I "pseudo coded" my thoughts. Pleas feel free to comment / adjust.

// RS485 collision detection
// algorithm to avoid/ detect bus collision
// considerations, assumptions & conditions:
// message size (incl overhead) 40 bytes
// - timing needs to be in line with bit rate 
//		9600 	~> 1 ms/Byte 	-> 40 ms/ message
// 		19200	~> 500 us/Byte 	-> 20 ms/ message
// 		115200	~> 10 us/Byte	-> 0.4 ms/ message
// no master/ slave - all equal rights
//
do{ // until message sent -or- timeout/ max retries
	// Send phase 1 - avoid collision
	If ( bus activity ){
		// 1. either wait for message to complete (we know the protocol) - and/or -
		//		wait until no more activity == no bytes during x bytes period
		// 2. be gentle and don't always want to be first in line 
		delay( random number of bytes );
		// now we are ready to start
		}
	// Send phase 2 - send and detect collision
	send start_byte_out ;
	read start_byte_in ;
	// Assuming all nodes are aware of and oblige the bus protocol, collision detection in start byte would suffice
	if (start_byte_in != start_byte_out){
			// possible collision so restart phase 1
		}
	for (the rest of the bytes){
		send byte_out ;
		read byte_in ;
		// be absolutely sure that all bytes are sent 
		if (byte_in != byte_out){
			// possible collision so restart phase 1
		}
	}
} while ( < max_retries  || ! messageSent || < timeout)

[LeoDesigner]

@AWI
The pseudocode looks correct, with one minor addition.
We may have corrupted incoming bytes received (or in our case just dropped) by the USART. So some kind of incoming byte waiting timeout should be implemented.

Please take a look here, to the bus arbitration process (J1708 NETWORK ACCESS part)

0_1478452964585_AN01230A.pdf

I hope we are not reinventing the wheel :)

One more interesting project to check:
https://github.com/MichaelJonker/HardwareSerialRS485/wiki/software-capabilities

[kimot]

If you all think, that reading message back on "real" (= not 10cm cables on table) RS485 bus by sender guarantee delivering to recipient, you are wrong. RS485 bus do not work like this and message in long distance from sending node with other parallel sending node my looks different on both places. You must use drivers for CAN bus instead of. I wrote about this in different thread.
Then some modified SoftwareSerial library with checking every sent bit and immediately stopping sending when collision detected does best work. But unfortunately nobody wrote this piece of software still ....
I need robust wired network connected to Domoticz via serial gateway too, so I choose CAN protocol on CANbus. CAN controller is not so expensive today. I am using extended 29bit ID, first 7 bits are receivers ID, so I can set mask and filter registers so, that I receive only messages for my node or broadcast.
Others bits in ID and in payload I use for senders ID, sensor ID,message type , data type, data and etc.
I am planning send only short messages from individual sensors, so data length of complete CAN message is enough for me.
But MySensors library is growing and growing and very complex for me, to implement this into it. So I try make my own library for connecting nodes to gateway and use MySensors serial protocol to the controller only.
I am planing Arduino and pure AVR libraries.
I am still coding, when I have got free time, so stay tuned.....

[LeoDesigner]

@kimot
I have the same concerns regarding long RS485 lines. Also SoftwareSerial cannot send and receive at the same time (not sure about AltSoftSerial).
We would like to make the solution as cheap as we can.
How about sending the message and get ACK back from node/gateway ? (Resend if ACK not received) It is like immediate confirmation of received packet from the other side ? I think it's could be reliable solution. CAN bus is a good solution, but one of the reasons why many people choosing mysensors library is a simplicity and cheapness.
Another way is too look to the modbus - but it's not multimaster bus solution.

[AWI]

@LeoDesigner The MySensors protocol allows for an ack from the destination although not all controllers support it. I would be more than satisfied if we could make sure that the message left the source in good shape and check with ack if its critical.
Judging from the links you sent there is a good chance we are reinventing the RS485 collision detection. The hardwareserialRS485 looks like a complete solutions, but a bit out of my league...

[napo7]

I'm also looking for a wired network...
I think, as @kimot said, that we cannot rely on "listen what I just said", since this just ensure that the message we did wanted to say was correctly put on the line, but it doesn't ensure us that it was correctly received !
Perhaps we should use collision AVOIDANCE (be sure that the line is free before speaking), but ALSO use ACKing system.

[LeoDesigner]

I know about mysensors ACK protocol feature, I was talking about small ACK packet with checksum as a confirmation of the received packet.
So basically the easiest way to improve current situation is to add ACK/CONFIRMATION packets to the existing library. Unfortunately I am really busy right now with main projects.

[hausinger]

@kimot

I'm very interested in your development.
What kind of can bus module are you using? I found 2 different modules for very less money. The very small module is called SN65HVD230 (controlled by serial rx, tx) and cost approximately 1,75$ from China. The other module with bigger size is called MCP2515 (controlled by spi) and cost approximately 1,26 $ from China. Both are very cheap.
What is you currently status in this case?
I'm planning to use it for my new house in some month.

Thank you

[pjr]

@kimot @hausinger could you open new topic about can bus transport implementation for MYS? I think there would be a lot of interest about it.

[kimot]

@pjr @hausinger

Ok.
New topic is here:

https://forum.mysensors.org/topic/5327/can-bus-transport-implementation-for-mys

[LeoDesigner]

One more idea regarding cheap collision detection on the bus.
We can almost eliminate collisions on the bus (should be almost no lost packets at all).
Before packet transmission we need to check the bus state during one first byte time using digitalRead(rxPin). In this case collision may occur very rarely, because time between bus checking and actual first byte sending is very short.
Also the code modification will be very small in current library.

[napo7]

Hi,

@LeoDesigner , that's the begining of a way avoiding collisions, but it's not sufficient :
The problem is that if two devices performs that check at exactly the same time (even if it's not so-probable, it can happen), they will start speaking at the same time !

A second step would require HANDLING (and not avoiding) collision : what can (what MUST) we do when two devices speak at the same time ?

That's where CAN drivers become interresting : we can use them on a "simili-RS485" bus : if two devices speak at the same time, the device speaking louder (understand, placing higher bits first...) will be overwriting what second device is saying. So, the first device (the one that is speaking "louder") will not see any problem, it will continue speaking, and the second one (which will see that his bits are not heard) will stop speaking, and try again a few milliseconds later !