I finally got it working reading through google searches. I'd like to provide how I did it for anyone that may run into a similar issue in the future. While looking for this solution, I saw at least one person who had the exact same issue and their problem went unresolved.
Raspberry Pi 3 is connected to Arduino Mega through regular serial link. Raspberry pi uses uart pins that map to Serial0 in raspbian. On Arduino Mega I use Serial2 pins (16,17). There's a voltage divider circuit between the two that reduces 5v to 3.3 for the pi.
I compiled my gateway on pi with below:
./configure --my-transport=rs485 --my-rs485-serial-port=/dev/serial0 --my-rs485-baudrate=9600
On Arduino I have these define lines:
#define MY_DEBUG
#define MY_DEBUGDEVICE Serial
#define MY_RS485
#define MY_RS485_HWSERIAL Serial2
#define MY_NODE_ID 1
#define MY_RS485_BAUD_RATE 9600
#define MY_RS485_DE_PIN -1
I'm not an expert on this by any means, but out of everything I tried, this is what worked for me. There are still some remaining issues:
- Possibly due to line noise or voltage being low on Arduino, I've had one crash that made Arduino print gibberish on serial and it stopped transmitting anything at all.
- I use Home Assistant and it seems to give up on MySensors if it doesn't receive anything in a while. Requires restart of HA core. Not very reliable.
I still welcome any insight into this process and any alternatives/recommendations. I'd like to make this as reliable as possible, but my knowledge of these systems and electronics is limited.