Post

Zero-Trust Mesh Networking: Subnet Routing and Tailscale ACLs Across Hybrid Cloud and Homelabs

How to eliminate exposed bastion hosts and secure hybrid cloud environments using Tailscale subnet routers, Linux kernel tuning, and least-privilege ACL policies.

Zero-Trust Mesh Networking: Subnet Routing and Tailscale ACLs Across Hybrid Cloud and Homelabs

Exposing SSH bastion hosts, maintaining complex IPsec site-to-site VPN tunnels, and managing fragile port-forwarding rules on edge routers are common security anti-patterns. Every exposed port on a public IP address is an invitation to automated credential-stuffing bots and port scanners.

Modern cloud infrastructure demands a Zero-Trust Network Access (ZTNA) model: no device is trusted by default, connections are authenticated and encrypted end-to-end, and internal private subnets remain completely dark to the public internet.

In this guide, we will walk through architecting a private, encrypted mesh network using Tailscale (WireGuard) as a high-performance subnet router across AWS VPCs and on-premise homelab environments, backed by granular, tag-based Access Control Lists (ACLs).


Architecture: Zero-Trust Hybrid Mesh

Rather than piercing firewall holes or managing public jump hosts, a Tailscale Subnet Router acts as a secure bridge into private network segments (e.g., AWS VPC private subnets 10.0.0.0/16 or local LAN subnets 192.168.1.0/24). Devices authenticated to your Tailnet can route traffic to internal targets (such as databases, internal APIs, or Kubernetes control planes) as if they were physically on that local network.

```mermaid flowchart TD subgraph Clients[“💻 Authenticated Endpoints (Tailnet)”] Engineer[“Engineer Workstation\n(Tailscale IP: 100.64.0.10)”] CIRunner[“CI/CD Runner\n(Tag: tag:ci-runner)”] end

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
subgraph TailscaleMesh["🔒 WireGuard Encrypted Overlay Network (Tailnet)"]
    direction TB
    CoordinationServer["Tailscale Control Plane\n(Policy & ACL Engine)"]
end

subgraph AWS_VPC["☁️ AWS Private VPC (10.0.0.0/16)"]
    SubnetRouterAWS["🛡️ AWS Subnet Router (EC2/EKS)\n(Tag: tag:aws-subnet-router)\n(Advertises: 10.0.0.0/16)"]
    InternalRDS[("🐘 Private RDS / Aurora\n(10.0.24.50:5432)")]
    PrivateEKS["☸️ EKS Private API Server\n(10.0.12.10:6443)"]
end

subgraph Homelab_LAN["🏠 Local Infrastructure (192.168.1.0/24)"]
    SubnetRouterHome["🛡️ Homelab Subnet Router\n(Tag: tag:home-subnet-router)\n(Advertises: 192.168.1.0/24)"]
    MonitoringNode["📊 Grafana / Prometheus\n(192.168.1.150:3000)"]
    StorageCluster["🗄️ TrueNAS / S3 MinIO\n(192.168.1.200:9000)"]
end

Engineer -.->|WireGuard Direct Peer-to-Peer| SubnetRouterAWS
Engineer -.->|WireGuard Direct Peer-to-Peer| SubnetRouterHome
CIRunner -.->|WireGuard Direct Peer-to-Peer| SubnetRouterAWS

SubnetRouterAWS -->|Forward Traffic| InternalRDS
SubnetRouterAWS -->|Forward Traffic| PrivateEKS
SubnetRouterHome -->|Forward Traffic| MonitoringNode
SubnetRouterHome -->|Forward Traffic| StorageCluster ```text

Step 1: Linux Kernel Tuning for Subnet Routing

A subnet router forwards packets between the Tailscale network interface (tailscale0) and the local Ethernet interface (eth0 or ens5). For Linux to route traffic across interfaces without dropping packets, IP forwarding and UDP buffer optimizations must be enabled.

Add the following kernel parameters to /etc/sysctl.d/99-tailscale.conf:

```ini

Enable IPv4 and IPv6 packet forwarding

net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1

Optimize network socket buffers for high-bandwidth WireGuard tunnels

net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.udp_rmem_min = 8192 net.ipv4.udp_wmem_min = 8192 ```text

Apply the changes immediately:

bash sudo sysctl -p /etc/sysctl.d/99-tailscale.conf text

AWS VPC Source/Destination Check

If running the subnet router on an AWS EC2 instance, AWS enforces a Source/Destination Check on ENIs by default. This will drop any packet whose destination IP does not match the EC2 instance’s own private IP.

Disable it using the AWS CLI or Terraform:

bash aws ec2 modify-instance-attribute \ --instance-id i-0abcdef1234567890 \ --no-source-dest-check text

In Terraform:

```hcl resource “aws_instance” “tailscale_router” { ami = data.aws_ami.debian.id instance_type = “t4g.small” source_dest_check = false # Required for subnet routing subnet_id = aws_subnet.private_a.id

tags = { Name = “tailscale-subnet-router” Role = “Networking” } } ```text


Step 2: Deploying the Subnet Router

You can run the router either as a native systemd service or as a containerized workload in Docker.

Native Systemd Setup

Authenticate the node, advertise the target CIDR ranges, and assign a management tag:

bash sudo tailscale up \ --advertise-routes=10.0.0.0/16 \ --advertise-exit-node=false \ --accept-dns=true \ --accept-routes=false \ --ssh=true \ --auth-key="tskey-auth-kXXXXX-XXXXXXXXXXXXXXXXXXXXXXXX" text

Containerized Setup with Docker Compose

For micro-servers or Kubernetes worker nodes, deploy Tailscale with network admin capabilities:

yaml services: tailscale-router: image: tailscale/tailscale:latest container_name: tailscale-subnet-router hostname: aws-vpc-router environment: - TS_AUTHKEY=tskey-auth-kXXXXX-XXXXXXXXXXXXXXXXXXXXXXXX - TS_ROUTES=10.0.0.0/16 - TS_EXTRA_ARGS=--accept-dns=true --ssh=true - TS_STATE_DIR=/var/lib/tailscale - TS_USERSPACE=false volumes: - /var/lib/tailscale:/var/lib/tailscale - /dev/net/tun:/dev/net/tun cap_add: - NET_ADMIN - NET_RAW restart: unless-stopped network_mode: host text


Step 3: Hardening with Tailscale ACL Policies

By default, Tailscale allows all nodes to communicate freely. In production, we enforce a strict Least-Privilege Policy using Tag-based ACLs (HuJSON format in the Tailscale Admin Console).

Here is a hardened access policy dividing access between Developers, DevOps Engineers, and CI/CD pipelines:

```json { // Define Groups and Identities “tagOwners”: { “tag:aws-subnet-router”: [“group:devops-leads”], “tag:home-subnet-router”: [“group:devops-leads”], “tag:ci-runner”: [“group:devops-leads”], “tag:developer-device”: [“group:devops-leads”, “group:engineering”] },

“groups”: { “group:devops-leads”: [“admin@example.com”], “group:engineering”: [“dev1@example.com”, “dev2@example.com”] },

// Auto-approve advertised routes for trusted tags “autoApprovers”: { “routes”: { “10.0.0.0/16”: [“tag:aws-subnet-router”], “192.168.1.0/24”: [“tag:home-subnet-router”] } },

// Strict Packet Filter Rules “acls”: [ // 1. DevOps Leads have full administrative access to all subnets { “action”: “accept”, “src”: [“group:devops-leads”], “dst”: [“10.0.0.0/16:”, “192.168.1.0/24:”, “:”] },

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// 2. Developers can only access Private EKS API and internal dev web services
{
  "action": "accept",
  "src":    ["tag:developer-device"],
  "dst": [
    "10.0.12.10:6443",      // Private EKS Kubernetes API
    "10.0.0.0/16:80,443",   // Internal HTTP/HTTPS services
    "192.168.1.150:3000"    // Homelab Grafana dashboard
  ]
},

// 3. CI/CD Runners can only deploy to Kubernetes and push to internal registries
{
  "action": "accept",
  "src":    ["tag:ci-runner"],
  "dst": [
    "10.0.12.10:6443",      // EKS API
    "10.0.24.50:5432"       // Database migration target
  ]
}   ],

// Enable Tailscale SSH with session check controls “ssh”: [ { “action”: “accept”, “src”: [“group:devops-leads”], “dst”: [“tag:aws-subnet-router”, “tag:home-subnet-router”], “users”: [“root”, “ubuntu”, “debian”], “checkPeriod”: “12h” } ] } ```text


Step 4: Handling MTU Issues and MSS Clamping

One of the most elusive networking bugs with overlay VPNs is Path MTU (PMTU) blackholing.

WireGuard encapsulates IP packets inside UDP packets, adding 40-80 bytes of overhead. If a client attempts to send standard 1500-byte Ethernet frames through the tunnel without PMTU Discovery working properly, packets get silently dropped, resulting in TLS handshakes stalling or SSH sessions freezing during large data transfers.

To eliminate this across all subnet clients, add an MSS clamping rule using iptables on the subnet router:

```bash

Clamp TCP MSS to Path MTU for forwarded traffic

sudo iptables -t mangle -A FORWARD -p tcp –tcp-flags SYN,RST SYN -j TCPMSS –clamp-mss-to-pmtu ```text

To persist this rule across reboots on Debian/Ubuntu:

bash sudo apt-get install -y iptables-persistent sudo iptables-save | sudo tee /etc/iptables/rules.v4 text


Step 5: High-Availability Subnet Routers with Auto-Failover

For production workloads, relying on a single EC2 instance or server for all subnet traffic introduces a Single Point of Failure (SPOF).

Tailscale supports Active/Passive High Availability automatically:

  1. Launch two subnet routers in different Availability Zones (us-east-1a and us-east-1b).
  2. Have both instances advertise the exact same CIDR route:

    1
    
     sudo tailscale up --advertise-routes=10.0.0.0/16
    
  3. Tailscale’s coordination server monitors the health of both routers. If Router A becomes unreachable, the control plane automatically re-routes traffic through Router B within seconds, without requiring DNS updates or client reconnection.

Verification and Testing

Verify route propagation from an authorized client machine:

```bash

Check available and active routes

tailscale status

Verify direct connectivity to private VPC IP without a public IP

nc -zv -w3 10.0.24.50 5432

Output: Connection to 10.0.24.50 port 5432 [tcp/postgresql] succeeded!

Test latency across the WireGuard tunnel

ping -c 4 10.0.12.10 ```text


Key Takeaways

  1. Eliminate Public Exposure: Removing SSH bastion hosts and public ingress points shrinks your cloud attack surface to near zero.
  2. Deterministic Identity: Tag-based ACLs decouple firewall rules from volatile IP addresses, tying access directly to cryptographically authenticated device identities.
  3. Seamless Multi-Cloud & Homelab Hybrid Connectivity: A unified overlay network connects AWS VPCs, local hardware, and remote laptops without brittle site-to-site IPsec tunnels.

This post is licensed under CC BY 4.0 by the author.