I guess your OPNSense rule from Edit3 is not working because the source is not your mailu instance, because connections are initiated from the outside and mailu only answers (TCP ACK). So you have asynchornous routing.

You may get this working if you set the “reply-to” option to the wg gateway on the firewall rule that allows VPS -> wg -> mailu traffic.

However there is a much cleaner solution using the PROXY protocol, which mailu seems to support: https://mailu.io/master/reverse.html

They are using traefik, but nginx also supports the PROXY protocol.

deleted by creator

thank you, just subsribed

It matters only if “the docker hosts external IP” your dns resolves is a public IP. In that case packets travel to the router which needs to map/send them back to the docker hosts LAN IP (NAT-Reflection). With cgnat this would need to be enabled on the carrier side, where you set up the port forwarding. If that’s not possible, split-DNS may be an alternative.

If “the docker hosts external IP” is actually your docker hosts LAN IP, all of that is irrelevant. Split-DNS would accomplish that.

Are you hosting behind NAT / at home? If so, you may need to enable NAT reflection on your router.