So after running pcaps from four spots along the connection chain between and including the remote client and the RRAS server, I think I managed to figure this out. We noticed that the UDP NAT entry for the tunnel keepalives would randomly just stop working based on the packet captures. It was on the Sophos XG that was protecting the RRAS server (it's in a perimeter network).
I started combing the logs looking for other things that correlated with the timing of the tunnel drops. It could be anywhere from 30 seconds to 30 minutes when it would happen, so it took a few tries. But I found that one of the S2S IPsec tunnels to a vendor was resetting at the same time.
Sophos has a somewhat undocumented "feature" where it resets all UDP connections when an IPsec tunnel comes up. No idea why. But anyway, the command to disable it is:
set vpn conn-remove-tunnel-up disable
You can't even tab-complete that command. It seems to be hidden in the CLI. Once we ran that, the disconnects stopped happening.
Just putting this here in case some future person runs into a similar issue, or I can't remember how I fixed it and I find this myself.