AWS EKS DNS Services Guide for Network Engineers

Note: This guide is specifically designed for network engineers who may not have extensive container experience. We'll explain EKS (Elastic Kubernetes Service) components and how they relate to traditional networking concepts you already know.

Container and EKS Fundamentals for Network Engineers

What is EKS and How Does it Differ from ECS?

EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes service. Think of it as AWS managing the control plane (like a network controller) while you manage the worker nodes (like switches/routers in your network).

ECS (Elastic Container Service) is AWS's proprietary container orchestration service. It's simpler but less flexible than EKS.

Key Difference: EKS uses Kubernetes (open-source, industry standard), while ECS uses AWS's proprietary orchestration. For DNS services, EKS provides more granular control similar to how you might configure DNS in a traditional network.

EKS Components - Network Engineer Perspective

graph TB A[EKS Cluster
Control Plane] --> B[Worker Node 1] A --> C[Worker Node 2] A --> D[Worker Node 3] B --> E[Pod 1
Application Container] B --> F[Pod 2
Application Container] C --> G[Pod 3
Application Container] D --> H[Pod 4
Application Container] I[AWS Load Balancer
Controller] --> J[Application Load Balancer] J --> K[Target Group] K --> E K --> F K --> G K --> H L[CoreDNS
DNS Service] --> M[kube-dns Service] M --> N[DNS Queries from Pods] style A fill:#ff9900,stroke:#232f3e,stroke-width:3px,color:#fff style L fill:#146eb4,stroke:#232f3e,stroke-width:2px,color:#fff style J fill:#28a745,stroke:#232f3e,stroke-width:2px,color:#fff

EKS Architecture Explained:
• Control Plane: Like your network management system - AWS manages this for you
• Worker Nodes: Similar to your switches/routers - EC2 instances that run your applications
• Pods: Think of these as VLANs or network segments containing your applications
• CoreDNS: The DNS server for your cluster (like your internal DNS servers)
• Load Balancer Controller: Routes traffic to healthy pods (like your F5 or NetScaler)

DNS in EKS - Deep Dive

DNS Traffic Flow in EKS

sequenceDiagram participant Client as External Client participant ALB as Application Load Balancer participant Pod as Application Pod participant CoreDNS as CoreDNS Pod participant ExtDNS as External DNS Client->>ALB: DNS Query for app.example.com ALB->>Pod: Forward to healthy pod Pod->>CoreDNS: Internal DNS Query (service.namespace.svc.cluster.local) CoreDNS->>CoreDNS: Check local cache CoreDNS->>Pod: Return service IP Pod->>ExtDNS: External DNS Query (api.external.com) ExtDNS->>Pod: Return external IP Pod->>ALB: Response with data ALB->>Client: Final response

DNS Flow Explanation:
This sequence shows how DNS queries work in EKS:
1. External clients query your application through Route 53 or external DNS
2. Application Load Balancer receives the request and forwards to healthy pods
3. Pods use CoreDNS for internal service discovery (finding other services in the cluster)
4. CoreDNS resolves internal Kubernetes service names to IP addresses
5. External DNS queries (like third-party APIs) go through the node's DNS configuration

CoreDNS Configuration Architecture

graph LR A[CoreDNS ConfigMap] --> B[Corefile Configuration] B --> C[kubernetes Plugin] B --> D[forward Plugin] B --> E[cache Plugin] B --> F[errors Plugin] C --> G[Service Discovery
cluster.local] D --> H[Upstream DNS
168.63.129.16] E --> I[DNS Cache
30 seconds] F --> J[Error Logging] K[kube-dns Service] --> L[CoreDNS Deployment] L --> M[CoreDNS Pod 1] L --> N[CoreDNS Pod 2] style A fill:#ff9900,stroke:#232f3e,stroke-width:2px,color:#fff style K fill:#146eb4,stroke:#232f3e,stroke-width:2px,color:#fff

CoreDNS Architecture:
• ConfigMap: Stores DNS configuration (like your DNS server config files)
• Corefile: Main configuration file defining DNS behavior
• kubernetes Plugin: Handles internal service discovery
• forward Plugin: Forwards external queries to upstream DNS
• kube-dns Service: ClusterIP service that pods use for DNS (like your internal DNS VIP)

Setup Commands and Configuration Order

🔧 Setup Order Diagram

graph TD A[1Create EKS Cluster] --> B[2Configure kubectl] B --> C[3Install AWS Load Balancer Controller] C --> D[4Configure CoreDNS] D --> E[5Deploy External DNS] E --> F[6Configure Route 53] F --> G[7Deploy Applications] G --> H[8Configure DNS Policies] style A fill:#ff9900,stroke:#232f3e,stroke-width:2px,color:#fff style H fill:#28a745,stroke:#232f3e,stroke-width:2px,color:#fff

Step 1: Create EKS Cluster

# Create EKS cluster with proper DNS configuration aws eks create-cluster \ --name my-cluster \ --version 1.28 \ --role-arn arn:aws:iam::123456789012:role/eks-service-role \ --resources-vpc-config subnetIds=subnet-12345,subnet-67890,securityGroupIds=sg-12345 \ --endpoint-config privateAccess=true,publicAccess=true \ --logging '{"enable":[{"types":["api","audit","authenticator","controllerManager","scheduler"]}]}'

Parameter Explanation:
• --name: Your cluster identifier (like a hostname)
• --version: Kubernetes version (similar to OS version)
• --role-arn: IAM role for EKS service (like service account permissions)
• --resources-vpc-config: Network configuration (your subnets and security groups)
• --endpoint-config: API server accessibility (like management interface access)
• --logging: Enable control plane logging for troubleshooting

What this creates: The EKS control plane in AWS-managed infrastructure. This is like setting up your network management system - AWS handles the underlying infrastructure while you get the API endpoints to manage your cluster.

Step 2: Configure kubectl Access

# Configure kubectl to connect to your cluster aws eks update-kubeconfig --region us-west-2 --name my-cluster # Verify connection kubectl get nodes kubectl get pods -n kube-system

Purpose: This configures your local kubectl tool to authenticate with your EKS cluster. It's similar to setting up SSH keys or SNMP credentials to manage your network devices.

Step 3: Install AWS Load Balancer Controller

# Create IAM role for AWS Load Balancer Controller aws iam create-role \ --role-name AmazonEKSLoadBalancerControllerRole \ --assume-role-policy-document file://load-balancer-role-trust-policy.json # Attach the required IAM policy aws iam attach-role-policy \ --role-name AmazonEKSLoadBalancerControllerRole \ --policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess

# load-balancer-role-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller",
                    "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
        

Trust Policy Parameters:
• Federated: Your EKS cluster's OIDC provider (get this from EKS console)
• StringEquals conditions: Ensures only the load balancer controller can assume this role
• sub: Service account that will use this role
• aud: AWS STS audience for token validation

# Install AWS Load Balancer Controller using Helm helm repo add eks https://aws.github.io/eks-charts helm repo update helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=my-cluster \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/AmazonEKSLoadBalancerControllerRole \ --set region=us-west-2 \ --set vpcId=vpc-12345678

Helm Installation Parameters:
• clusterName: Must match your EKS cluster name
• serviceAccount.annotations: Links the controller to the IAM role
• region: AWS region where your cluster is deployed
• vpcId: VPC where your EKS cluster is running

What this does: Installs a controller that automatically creates and manages AWS Load Balancers when you deploy applications. It's like having an automated system that configures your hardware load balancers based on application requirements.

Step 4: Configure CoreDNS

# Get current CoreDNS configuration kubectl get configmap coredns -n kube-system -o yaml > coredns-config.yaml # Edit the configuration kubectl edit configmap coredns -n kube-system

# CoreDNS ConfigMap - Enhanced Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . 168.63.129.16 8.8.8.8 {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
        log
    }
    company.local:53 {
        errors
        cache 300
        forward . 10.0.0.10 10.0.0.11
    }
        

CoreDNS Configuration Breakdown:
• .:53 - Default zone handling all DNS queries on port 53
• errors - Log DNS errors for troubleshooting
• health - Health check endpoint with 5-second lameduck period
• ready - Readiness probe endpoint
• kubernetes cluster.local - Handle internal cluster DNS
• pods insecure - Enable pod DNS records (A/AAAA)
• ttl 30 - Cache internal DNS records for 30 seconds
• prometheus :9153 - Metrics endpoint for monitoring
• forward . 168.63.129.16 8.8.8.8 - Upstream DNS servers
• cache 30 - Cache external DNS queries for 30 seconds
• company.local:53 - Custom zone for internal company DNS

Network Engineer Note: This configuration is similar to your DNS server zones file. The first block handles internet and cluster DNS, while the second block handles your internal corporate DNS zone.

# Apply the configuration and restart CoreDNS kubectl apply -f coredns-config.yaml kubectl rollout restart deployment coredns -n kube-system kubectl rollout status deployment coredns -n kube-system

Step 5: Deploy External DNS

# Create IAM policy for External DNS aws iam create-policy \ --policy-name ExternalDNSPolicy \ --policy-document file://external-dns-policy.json # Create IAM role for External DNS aws iam create-role \ --role-name ExternalDNSRole \ --assume-role-policy-document file://external-dns-trust-policy.json # Attach policy to role aws iam attach-role-policy \ --role-name ExternalDNSRole \ --policy-arn arn:aws:iam::123456789012:policy/ExternalDNSPolicy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "*"
        }
    ]
}
        

External DNS IAM Policy:
• route53:ChangeResourceRecordSets: Create/update/delete DNS records
• route53:ListHostedZones: Discover available DNS zones
• route53:ListResourceRecordSets: Read existing DNS records
This is like giving your DNS management system permissions to update your authoritative DNS servers.

# external-dns-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.14.0
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=example.com
        - --provider=aws
        - --policy=upsert-only
        - --aws-zone-type=public
        - --registry=txt
        - --txt-owner-id=my-cluster
        - --log-format=json
        - --log-level=info
        - --interval=1m
        env:
        - name: AWS_DEFAULT_REGION
          value: us-west-2
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ExternalDNSRole
        

External DNS Configuration:
• --source=service,ingress: Watch Kubernetes services and ingresses for DNS annotations
• --domain-filter=example.com: Only manage DNS records for this domain
• --provider=aws: Use AWS Route 53 as the DNS provider
• --policy=upsert-only: Only create/update records, never delete
• --registry=txt: Use TXT records to track ownership
• --txt-owner-id: Unique identifier for this cluster's DNS records
• --interval=1m: Check for changes every minute

What this does: Automatically creates DNS records in Route 53 when you deploy services with specific annotations. It's like having an automated system that updates your DNS zone files when you add new services.

Step 6: Configure Route 53 Integration

# Create Route 53 hosted zone (if not exists) aws route53 create-hosted-zone \ --name example.com \ --caller-reference $(date +%s) \ --hosted-zone-config Comment="EKS cluster DNS zone" # Get hosted zone ID aws route53 list-hosted-zones-by-name --dns-name example.com

# Sample service with External DNS annotation
apiVersion: v1
kind: Service
metadata:
  name: my-app
  annotations:
    external-dns.alpha.kubernetes.io/hostname: app.example.com
    external-dns.alpha.kubernetes.io/ttl: "300"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: my-app
        

Service DNS Annotations:
• external-dns.alpha.kubernetes.io/hostname: DNS name to create
• external-dns.alpha.kubernetes.io/ttl: DNS record TTL (300 seconds)
• service.beta.kubernetes.io/aws-load-balancer-type: Load balancer type (nlb/alb)
This automatically creates an A record pointing app.example.com to your load balancer.

Step 7: DNS Policies and Network Policies

# DNS Network Policy - Restrict DNS access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dns-policy
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to: []
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
        

DNS Network Policy:
• podSelector: {}: Applies to all pods in the namespace
• policyTypes: [Egress]: Controls outbound traffic
• ports 53 UDP/TCP: Allow DNS queries
• namespaceSelector kube-system: Allow queries to CoreDNS
This is like creating firewall rules that only allow DNS traffic on ports 53.

Advanced DNS Configuration

Custom DNS Resolution Flow

graph TD A[Application Pod] --> B{DNS Query Type} B -->|service.namespace.svc.cluster.local| C[CoreDNS - Kubernetes Plugin] B -->|company.local| D[CoreDNS - Custom Zone] B -->|external.com| E[CoreDNS - Forward Plugin] C --> F[Return Service ClusterIP] D --> G[Forward to Internal DNS
10.0.0.10] E --> H[Forward to Public DNS
8.8.8.8] G --> I[Internal DNS Response] H --> J[Public DNS Response] F --> K[Pod connects to Service] I --> L[Pod connects to Internal Resource] J --> M[Pod connects to External Resource] style A fill:#ff9900,stroke:#232f3e,stroke-width:2px,color:#fff style C fill:#146eb4,stroke:#232f3e,stroke-width:2px,color:#fff style D fill:#28a745,stroke:#232f3e,stroke-width:2px,color:#fff style E fill:#dc3545,stroke:#232f3e,stroke-width:2px,color:#fff

Advanced DNS Resolution:
This diagram shows how CoreDNS handles different types of DNS queries:
• Internal Service Discovery: Handled by the kubernetes plugin
• Company Internal DNS: Forwarded to your corporate DNS servers
• External Internet DNS: Forwarded to public DNS servers
This allows your applications to resolve both internal Kubernetes services and external dependencies.

DNS Performance Optimization

# Scale CoreDNS for better performance kubectl scale deployment coredns --replicas=3 -n kube-system # Configure DNS caching on nodes kubectl apply -f node-local-dns.yaml

# Node Local DNS Cache Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
            success 9984 30
            denial 9984 5
        }
        reload
        loop
        bind 169.254.20.10
        forward . 10.100.0.10 {
            force_tcp
        }
        prometheus :9253
    }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10
        forward . 10.100.0.10 {
            force_tcp
        }
        prometheus :9253
    }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10
        forward . 10.100.0.10 {
            force_tcp
        }
        prometheus :9253
    }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10
        forward . 8.8.8.8 8.8.4.4 {
            force_tcp
        }
        prometheus :9253
    }
        

Node Local DNS Configuration:
• bind 169.254.20.10: Link-local IP address for DNS cache
• cache success 9984 30: Cache successful responses for 30 seconds
• cache denial 9984 5: Cache NXDOMAIN responses for 5 seconds
• force_tcp: Use TCP for upstream queries (more reliable)
• prometheus :9253: Metrics endpoint for monitoring
This creates a local DNS cache on each node, reducing latency and load on CoreDNS.

Monitoring and Troubleshooting

# Monitor CoreDNS performance kubectl top pods -n kube-system -l k8s-app=kube-dns kubectl logs -n kube-system -l k8s-app=kube-dns # Test DNS resolution from a pod kubectl run debug-pod --image=nicolaka/netshoot --rm -it -- bash nslookup kubernetes.default.svc.cluster.local dig @10.100.0.10 google.com # Check DNS configuration kubectl exec -it debug-pod -- cat /etc/resolv.conf

Troubleshooting Commands:
• kubectl top pods: Monitor CPU and memory usage
• kubectl logs: View DNS query logs and errors
• nslookup/dig: Test DNS resolution from within the cluster
• /etc/resolv.conf: Check DNS configuration in pods
These tools help you diagnose DNS issues similar to how you'd troubleshoot traditional DNS servers.

Best Practices for Network Engineers

Security Considerations:
• Always use Network Policies to restrict DNS access
• Monitor DNS query logs for suspicious activity
• Use separate DNS zones for different environments
• Implement proper RBAC for DNS management
• Regular security audits of DNS configurations

Performance Tips:
• Use Node Local DNS for better performance
• Configure appropriate TTL values for different record types
• Monitor DNS cache hit rates
• Scale CoreDNS replicas based on cluster size
• Use health checks and readiness probes

Summary Configuration Flow

graph TD A[1. Create EKS Cluster] --> B[2. Configure kubectl] B --> C[3. Install Load Balancer Controller] C --> D[4. Configure CoreDNS] D --> E[5. Deploy External DNS] E --> F[6. Configure Route 53] F --> G[7. Apply DNS Policies] G --> H[8. Monitor and Optimize] I[IAM Roles] --> C I --> E J[Network Policies] --> G K[ConfigMaps] --> D L[Helm Charts] --> C style A fill:#ff9900,stroke:#232f3e,stroke-width:2px,color:#fff style H fill:#28a745,stroke:#232f3e,stroke-width:2px,color:#fff style I fill:#146eb4,stroke:#232f3e,stroke-width:2px,color:#fff

Complete Setup Flow:
This diagram shows the complete setup process for EKS DNS services. Each component depends on the previous ones, and supporting elements like IAM roles, network policies, and configurations are applied at the appropriate stages. Follow this sequence to ensure proper DNS functionality in your EKS cluster.

Next Steps:
After completing this setup, you'll have a fully functional DNS system in your EKS cluster that can handle internal service discovery, external DNS management, and integration with your existing network infrastructure. Monitor the logs and metrics to ensure optimal performance and security.