Configuring multiple Ethernet connections in a linux server

I have the following topology

                                                                                        .───────────────.        
   ┌──────────────────┐                                                             _.─'                 `──.    
   │                  │                                                          ,─'                         '─. 
   │    ISP Router    │ Dynamic IP                                              ;                               :
   │     with NAT     ├─────────────────────────────────────────────────────────:           Internet            ;
   │                  │                                                          ╲                             ╱ 
   └─────────┬────────┘                                                           '─.                       ,─'  
             │                                                                       `──.               _.─'     
             │  10.0.0.1                                                                 `──────┬──────'         
             │                                                                                  │                
      .──────┴────.                                                                       .─────┴─────.          
   ,─'             '─.                   ┌─────────────────────────┐                   ,─'             '─.       
 ,'                   `.                 │                         │                 ,'                   `.     
;          LAN          :  Static DHCP┌──┴───┐                  ┌──┴───┐ Static IP  ;      Public Net       :    
:                       ;─────────────┤ eth1 │     My Server    │ eth0 ├────────────:                       ;    
 ╲     10.0.0.0/24     ╱     10.0.0.5 └──┬───┘                  └──┬───┘  1.2.3.5    ╲     1.2.3.4/20      ╱     
  `.                 ,'                  │       Debian Buster     │    mydomain.com  `.                 ,'      
    '─.           ,─'                    │                         │                    '─.           ,─'        
       `─────────'                       └─────────────────────────┘                       `─────────'

Because of historical reasons, my ISP uses a separate network to provide static IPs. This means I have to connect my server directly to a special connection that is directly exposed to the internet (in the diagram this is eth0 with IP 1.2.3.5).

However, I use this same server as a NAS in my home and since that's a separate network altogether I connect it using a different interface eth1 and create a static DHCP allocation with address 10.0.0.5. This way, other devices in the home network can find it and access the network mounts.

Lastly, I have a DNS record with my domain, in this example mydomain.com, pointing to 1.2.3.5.

I'm running into a couple of issues:

• When I try to access mydomain.com from my LAN it doesn't work. I believe this is because the Server is getting the packet from interface eth0 but then replying through eth1 and the other device is dropping the response since it's from a different IP.
• On boot the Server should pick 1.2.3.4 as the network to direct internet traffic to so external access works through mydomain.com. However, since both eth0 and eth1 have internet access through the respective gateways, it sometimes boots and picks 10.0.0.1 as the default gateway and then accessing mydomain.com doesn't work. I believe this is a similar issue with traffic entering through eth0 but leaving through eth1.

While I'm familiar with basic networking stuff, I don't have experience with having a server connected to multiple networks at once like this. Ideally I'd like to configure the server so that:

• All internet traffic exits through eth0
• All traffic from entering eth0 is replied through eth0
• All traffic from entering eth1 is replied through eth1

My server runs Debian, I'd appreciate any help.

EDIT: So after disabling the eth1 gateway and playing a bit with tcpdump I was able to confirm that traffic is entering from eth0 and leaving eth1. The issue is that the ISP is not NATing traffic going to the endpoint 1.2.3.5 and then the server is identifying the 10.0.0.X addresses as located on interface eth1.

This is a routing issue which can be solved with source-based routing or policy routing. This involves two routing tables, one for each source address:

See http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html for more details.

I believe this is because the Server is getting the packet from interface eth0 but then replying through eth1.

No. The ISP Router with NAT will be NATing your client 10.0.0.0 address to an ISP dynamics address, so your server would be replying to that address rather than the internal one. There are exceptions such as FTP that are not necessarily compatible with NAT as they will pass your internal IP address to the other end of the connection, however for the general case your NAT traffic will stay on eth0.

Check your logs to confirm clients are going outbound to 1.2.3.4 and the server is getting connections from the NATed addresses ip.ip.ip.ip.

On boot the Server should pick 1.2.3.4 as the network to direct internet traffic to so external access works through mydomain.com. However, since both eth0 and eth1 have internet access through the respective gateways

Do not do that. Having multiple default gateways in multi-homed configurations rarely works as people intend. Very roughly multiple default gateways should only be used when both paths provide access to identical networks such as for router redundancy/high availability in an enterprise setting. Since your ISP router to the Internet and your direct connection to the public net are not identical you should remove this. eth0 should have a default route, eth1 should have a route to 10.x.x.x. You can still do this with DHCP, however I would approach this as a reservation/exclusion for 10.0.0.5 and configure the server IP manually.

Aside from the extraneous default route, what you described is the normal default operation. Be sure to look for other issues not described above:

• Host files differing from DNS (did you try getting clients to connect to 10.0.0.5 instead of 1.2.3.4? I like to use a technical name internally such as webserver1.internalfqdn.com and a friendly name externally such as www.externalfqdn.com)
• Manual route tables not reflected above (did you try to shortcut 10.0.0.x to 1.2.3.4 through 10.0.0.5 as a router? - this can work, but it requires extra configuration)
• Routing protocols (e.g. RIP) (really a variation on the above, but if you have RIP or other route publishing enabled you may be publishing bad routes