Dynamic DNS Resolution in Nginx

Introduction

When configuring Nginx as a reverse proxy, one of the most challenging issues engineers face is the problem of dynamic DNS resolution. This becomes particularly relevant when using cloud services like AWS, where load balancer endpoints (ALB) might change their underlying IP addresses without warning.

Understanding the Problem: DNS Resolution in Nginx

How Nginx Handles DNS Resolution

• By default, Nginx resolves DNS names to IP addresses only once during startup or configuration reload.

• Then it caches the ip and reuses it for further requests

This design decision was made for performance reasons, as constant DNS lookups would add overhead to request processing. However, this creates a significant problem when working with dynamic endpoints like AWS Application Load Balancers (ALBs).

For example, consider this basic Nginx configuration:

upstream backend {
    server my-alb-1234567890.us-east-1.elb.amazonaws.com;
}

server {
    listen 80;
    
    location / {
        proxy_pass http://backend;
    }
}

When Nginx starts, it resolves my-alb-1234567890.us-east-1.elb.amazonaws.com to its current IP addresses.

The problem occurs when AWS rotates the IP addresses behind that ALB domain (which happens regularly for maintenance, scaling, or failover).

As Nginx will continue using the cached (and now potentially invalid) IP addresses, then the apis will start failing and they will not reach to backend.

Why AWS ALB DNS Addresses Change

AWS Application Load Balancers are designed for high availability and scaling. To achieve this, AWS:

1. Uses multiple IP addresses for a single ALB

2. Regularly rotates these IPs for maintenance and scaling

3. Manages failover by changing IPs when instances become unhealthy

4. May scale the ALB horizontally, adding new IP addresses during high traffic periods

The ALB DNS name itself remains constant, but the IP addresses it resolves to can change at any time, often without notice.

The Resulting Problems

This mismatch between Nginx's one-time DNS resolution and AWS's dynamic IP assignment leads to several issues:

1. Connection failures: Requests fail when Nginx tries to connect to stale IP addresses

2. Service degradation: Some backends might become unavailable while others still work

3. Manual intervention: Engineers need to regularly reload Nginx to force DNS re-resolution

4. Cascading failures: During high-traffic events when AWS is scaling the ALB, Nginx might miss the new IPs

Let's explore various solutions to address this fundamental mismatch.

Solution 1: Resolver Directive with Valid Parameter

Nginx provides a resolver directive that can be configured to periodically re-resolve DNS names. This is the most straightforward solution: [Need to check with Nginx Version compatibility]

http {
    # Configure DNS resolver with cache validity time
    resolver 10.0.0.2 8.8.8.8 valid=30s ipv6=off;
    resolver_timeout 5s;
    
    # Maintain variable for dynamic resolution
    upstream backend {
        server my-alb-1234567890.us-east-1.elb.amazonaws.com resolve; # may not work in some nginx versions
    }

    server {
        listen 80;
        
        location / {
            proxy_pass http://backend;
        }
    }
}

Let's break down what's happening here:

1. resolver 10.0.0.2 8.8.8.8 valid=30s: Configures Nginx to use AWS VPC dns and Google's DNS (8.8.8.8) as fallback to resolve the ip and cache DNS entries for 30 seconds

2. ipv6=off: Disables IPv6 resolution (optional but helpful if your environment doesn't support IPv6)

3. server my-alb-1234567890.us-east-1.elb.amazonaws.com resolve: The resolve parameter tells Nginx to re-resolve this hostname periodically

This solution works well for many scenarios but has limitations with upstream blocks. Let's look at a more robust approach.

Solution 2: Variable for Backend with Resolver

A more flexible approach uses variables to store the backend address:

http {
    resolver 10.0.0.2 8.8.8.8 valid=30s ipv6=off;
    resolver_timeout 5s;
    
    server {
        listen 80;
        
        # Store the backend address in a variable
        set $backend_server "my-alb-1234567890.us-east-1.elb.amazonaws.com";
        
        location / {
            # Use the variable in proxy_pass - this forces DNS resolution on each request
            proxy_pass http://$backend_server;
            
            # Important headers for AWS ALBs
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Connection handling settings
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

The key difference in this approach is storing the ALB address in a variable ($backend_server) and using that variable directly in the proxy_pass directive.

When Nginx sees a variable in proxy_pass, it re-resolves the DNS name for each request, using the cache duration specified in the resolver directive.

This gives you fine-grained control over how often DNS resolution occurs. But this solution also may not work as variables does not inherit location block uri, but backend expects it.

See final solution.

Solution 3: External Service Discovery

For production environments, consider using a dedicated service discovery tool:

# nginx.conf using Consul for service discovery
http {
    resolver 127.0.0.1:8600 valid=5s;  # Consul DNS interface
    
    server {
        listen 80;
        
        set $backend_service "service.aws-alb.consul";
        
        location / {
            proxy_pass http://$backend_service;
            proxy_set_header Host "my-alb-1234567890.us-east-1.elb.amazonaws.com";
        }
    }
}

This approach relies on an external service discovery system like Consul, which would need additional configuration:

# consul-config.hcl
service {
  name = "aws-alb"
  address = "my-alb-1234567890.us-east-1.elb.amazonaws.com"
  
  checks = [
    {
      id = "dns-check"
      name = "Verify ALB DNS resolution"
      args = ["/usr/local/bin/check-dns.sh", "my-alb-1234567890.us-east-1.elb.amazonaws.com"]
      interval = "30s"
      timeout = "5s"
    }
  ]
}

The service discovery approach:

1. Delegates DNS management to a specialized tool

2. Provides additional health checking capabilities

3. Can handle complex service discovery logic

4. Works well in microservices environments

Difference Between proxy_pass with variable and proxy_pass with hardcoded url

When you use a hardcoded URL in the proxy_pass directive vs proxy_pass with variable:

proxy_pass http://internal-prod-elb-12596.ap-south-1.elb.amazonaws.com:8000/;

set $backend_server "http://internal-prod-elb-12596.ap-south-1.elb.amazonaws.com:8000/";

proxy_pass $backend_server; 

Nginx handles it differently than when using a variable:

1. URI Handling: With a hardcoded URL, Nginx processes the URI differently and preserves certain aspects of the original request that are important for authentication.

   1. With variable it does not inherit the location block that may be important for the backend.

   2. With hardcode url it cachescaches the result for the lifetime of the worker process and reuse it for further requests, that could get change but it will not get resolved untill reload or restarted.

2. Header Preservation: The way Nginx builds the upstream request maintains authentication headers better with hardcoded URLs.

   1. The host header may be inaccurate

• Hardcoded URL: Resolved once at startup or config reload, cached for worker lifecycle

• Variable-based URL: Resolved at request time using the resolver directive settings

Final Solution

To solve all the problems related with dns resolutions, correct request uri and header preservation, we need to combine the solutions.

Here is the final configuration that worked for me.

First we need to add resolver for AWS Dns in http block in nginx.conf file

http {
    # aws dns then fallback to google dns, refreshes in every 30s
    resolver 10.0.0.2 8.8.8.8 valid=30s ipv6=off; 
    resolver_timeout 5s; # timeout of dns resolution

    .... # rest of the config
}

In server block of app.conf

      # location block that will do proxy on /api/
      location /api/ {
      
       # creating the variable with alb dns address and port as my BE need it
        set $elb_dns "http://internal-prod-elb-1259.ap-south-1.elb.amazonaws.com:8000";
        
        # When you use a literal upstream URL with a trailing slash in proxy_pass, nginx automatically strips the matching location prefix. 
        # However, when you use a variable (like $elb_dns), this automatic URI rewriting isn’t performed.
         # Rewrite the URI: Remove the location prefix (/api) from the incoming request.
         # For example, /api/api/auth/verify becomes /api/auth/verify. [ because BE need /api]
        rewrite ^/api(/.*)$ $1 break;

        proxy_pass $elb_dns; # adding variable in proy pass

        proxy_pass_request_headers on;

        proxy_set_header Host $host; # keeping the original host header

        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-NginX-Proxy true;

        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Methods "POST, PUT, GET, DELETE, OPTIONS";

         # error log on debug level for troubleshooting
        error_log /var/log/nginx/api_error.log debug;

    }

Best Practices and Considerations

Regardless of which solution you choose, consider these best practices:

1. Configure Proper Timeouts

Always set appropriate timeouts in your Nginx configuration:

proxy_connect_timeout 3s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;

2. Implement Health Checks

Add active health checks to detect backend issues early:

http {
    upstream backend {
        server my-alb-1234567890.us-east-1.elb.amazonaws.com resolve;
        
        # Active health checks
        health_check interval=10 fails=3 passes=2;
    }
}

3. Monitor DNS Resolution

Add logging to track DNS resolution:

http {
    resolver 8.8.8.8 valid=30s ipv6=off;
    resolver_timeout 5s;
    
    log_format dns '$time_local DNS: $host to $upstream_addr';
    access_log /var/log/nginx/dns.log dns;
}

4. Test Under Load

Before deploying to production, test how your solution performs under load, especially during IP transitions.

Conclusion

Dynamic DNS resolution in Nginx when dealing with AWS ALB endpoints requires careful consideration. While the default behaviour of resolving DNS only at startup is a performance optimization, it creates challenges in dynamic cloud environments.