Table of Contents
Overview and Architecture
AWS Direct Connect with Transit Gateway creates a scalable, centralized hub for connecting your on-premises network to multiple VPCs across different AWS regions. This architecture eliminates the need for multiple Direct Connect connections and provides a single point of management.
Key Benefits:
- Centralized connectivity hub
- Cross-region VPC connectivity
- Simplified routing and management
- Cost optimization through shared connectivity
- Enhanced security with segmentation
Traffic Flow Overview
graph TB
subgraph "On-Premises"
OP[On-Premises Network
192.168.0.0/16] CE[Customer Edge Router] end subgraph "AWS Direct Connect" DXL[Direct Connect Location] DXC[Direct Connect Connection
1Gbps/10Gbps] end subgraph "AWS Cloud" DXGW[Direct Connect Gateway] TGW[Transit Gateway] subgraph "VPC-A us-east-1" VPCA[VPC-A
10.1.0.0/16] TGWA[TGW Attachment] end subgraph "VPC-B us-west-2" VPCB[VPC-B
10.2.0.0/16] TGWB[TGW Attachment] end end OP --> CE CE --> DXL DXL --> DXC DXC --> DXGW DXGW --> TGW TGW --> TGWA TGW --> TGWB TGWA --> VPCA TGWB --> VPCB
192.168.0.0/16] CE[Customer Edge Router] end subgraph "AWS Direct Connect" DXL[Direct Connect Location] DXC[Direct Connect Connection
1Gbps/10Gbps] end subgraph "AWS Cloud" DXGW[Direct Connect Gateway] TGW[Transit Gateway] subgraph "VPC-A us-east-1" VPCA[VPC-A
10.1.0.0/16] TGWA[TGW Attachment] end subgraph "VPC-B us-west-2" VPCB[VPC-B
10.2.0.0/16] TGWB[TGW Attachment] end end OP --> CE CE --> DXL DXL --> DXC DXC --> DXGW DXGW --> TGW TGW --> TGWA TGW --> TGWB TGWA --> VPCA TGWB --> VPCB
Virtual Interface Types & VGW vs Transit Gateway
Key Point: You DON'T need a VGW if you use Transit Gateway, even with just 1 VIF!
Modern AWS architecture recommends Transit Gateway over VGW for scalability and simplicity.
Modern AWS architecture recommends Transit Gateway over VGW for scalability and simplicity.
Does Transit Gateway Require a Direct Connect Gateway?
NO - Transit Gateway does NOT always require a Direct Connect Gateway!
It depends on your VIF type and architecture choice.
It depends on your VIF type and architecture choice.
graph TB
subgraph "Option A: Private VIF (Requires DX Gateway)"
A1[On-Premises] --> A2[Private VIF] --> A3[Direct Connect Gateway] --> A4[Transit Gateway] --> A5[VPCs]
A3 -.->|Required| A4
end
subgraph "Option B: Transit VIF (NO DX Gateway Needed!)"
B1[On-Premises] --> B2[Transit VIF] --> B3[Transit Gateway] --> B4[VPCs]
B2 -.->|Direct Connection| B3
end
style A3 fill:#ffeb3b
style B3 fill:#4caf50
When You DO Need Direct Connect Gateway
Private VIF → DX Gateway → Transit Gateway
- Using Private VIF type
- Private VIFs cannot connect directly to Transit Gateway
- Must go through Direct Connect Gateway as an intermediary
- More hops = slightly more latency
- Traditional/legacy approach
When You DON'T Need Direct Connect Gateway
Transit VIF → Transit Gateway (Direct)
- Using Transit VIF type
- Transit VIFs connect directly to Transit Gateway
- No Direct Connect Gateway required
- Fewer hops = better performance
- Modern/recommended approach
- Supports higher bandwidth (up to 100G)
Side-by-Side Comparison
| Aspect | Private VIF + DX Gateway | Transit VIF (Direct) |
|---|---|---|
| DX Gateway Required? | ✅ YES - Required | ❌ NO - Not needed |
| Network Hops | VIF → DX GW → TGW → VPC | VIF → TGW → VPC |
| Maximum Bandwidth | 10 Gbps per VIF | 100 Gbps per VIF |
| Latency | Higher (more hops) | Lower (fewer hops) |
| Complexity | More complex (more components) | Simpler (fewer components) |
| When to Use | Legacy deployments, existing Private VIFs | New deployments, high performance needs |
Configuration Examples: With vs Without DX Gateway
Option A: Private VIF (Requires DX Gateway)
# This approach REQUIRES Direct Connect Gateway
# 1. Create Transit Gateway
aws ec2 create-transit-gateway --description "TGW with DX Gateway"
# 2. Create Direct Connect Gateway (REQUIRED for Private VIF)
aws directconnect create-direct-connect-gateway \
--name "Required-DX-Gateway" \
--amazon-side-asn 64512
# 3. Associate DX Gateway with Transit Gateway
aws directconnect create-direct-connect-gateway-association-proposal \
--direct-connect-gateway-id dxgw-12345678 \
--gateway-id tgw-0abcd1234efgh5678
# 4. Create Private VIF pointing to DX Gateway
aws directconnect create-private-virtual-interface \
--connection-id dxcon-xxxxxxxxx \
--new-private-virtual-interface '{
"virtualInterfaceName": "Private-VIF-via-DXGW",
"vlan": 100,
"asn": 65001,
"amazonAddress": "192.168.254.2/30",
"customerAddress": "192.168.254.1/30",
"directConnectGatewayId": "dxgw-12345678"
}'
Option B: Transit VIF (NO DX Gateway Needed)
# This approach does NOT need Direct Connect Gateway!
# 1. Create Transit Gateway
aws ec2 create-transit-gateway --description "TGW with Transit VIF"
# 2. Create Transit VIF directly to Transit Gateway
# NO Direct Connect Gateway creation needed!
aws directconnect create-transit-virtual-interface \
--connection-id dxcon-xxxxxxxxx \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Direct-Transit-VIF",
"vlan": 200,
"asn": 65001,
"amazonAddress": "192.168.254.6/30",
"customerAddress": "192.168.254.5/30",
"transitGatewayId": "tgw-0abcd1234efgh5678"
}'
# That's it! No DX Gateway needed.
Important Note: In the Transit VIF command above, notice we use
transitGatewayId directly instead of directConnectGatewayId. This creates a direct connection from the VIF to the Transit Gateway.
Architecture Options for Single VIF
graph TB
subgraph "Option 1: Direct VIF to VGW (Legacy)"
O1A[On-Premises] --> O1B[Private VIF] --> O1C[Virtual Gateway VGW] --> O1D[Single VPC]
end
subgraph "Option 2: VIF via DX Gateway to TGW (Recommended)"
O2A[On-Premises] --> O2B[Private VIF] --> O2C[DX Gateway] --> O2D[Transit Gateway] --> O2E[Single VPC]
end
subgraph "Option 3: Transit VIF to TGW (Most Efficient)"
O3A[On-Premises] --> O3B[Transit VIF] --> O3C[Transit Gateway] --> O3D[Single VPC]
end
style O1D fill:#ffcccc
style O2E fill:#ccffcc
style O3D fill:#ccffff
Architecture Comparison Table
| Architecture | Components Needed | When to Use | Limitations |
|---|---|---|---|
| Direct to VGW | Private VIF → VGW → VPC | Legacy deployments, single VPC only | No multi-VPC, no cross-region |
| Via DX Gateway | Private VIF → DX Gateway → TGW → VPC(s) | Multiple VPCs, cross-region connectivity | Additional hop, more complex |
| Transit VIF | Transit VIF → TGW → VPC(s) | Modern deployments, highest performance | Requires newer DX infrastructure |
When You DON'T Need a VGW
Modern Recommendation: Skip VGW entirely and use Transit Gateway
Even for a single VPC, Transit Gateway provides better scalability and future-proofing.
Even for a single VPC, Transit Gateway provides better scalability and future-proofing.
# Single VPC Scenarios - VGW vs TGW Decision
CHOOSE VGW ONLY IF:
❌ Legacy deployment that can't be changed
❌ Very simple setup with no growth plans
❌ Cost is extremely sensitive (VGW is cheaper for single VPC)
CHOOSE TRANSIT GATEWAY IF:
✅ Any possibility of adding more VPCs in future
✅ Need cross-region connectivity
✅ Want simplified routing management
✅ Plan to connect to other AWS services (VPN, other DX)
✅ Modern deployment with best practices
Configuration Examples: Single VIF, No VGW
# Example: Single VIF to Single VPC via Transit Gateway
# NO VGW required!
# 1. Create Transit Gateway
aws ec2 create-transit-gateway \
--description "Single VPC Transit Gateway" \
--options DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable
# 2. Create Direct Connect Gateway
aws directconnect create-direct-connect-gateway \
--name "Single-VPC-DX-Gateway" \
--amazon-side-asn 64512
# 3. Associate DX Gateway with Transit Gateway
aws directconnect create-direct-connect-gateway-association-proposal \
--direct-connect-gateway-id dxgw-12345678 \
--gateway-id tgw-0abcd1234efgh5678 \
--add-allowed-prefixes-to-direct-connect-gateway Cidr=10.1.0.0/16
# 4. Create Private VIF pointing to DX Gateway (NOT VGW)
aws directconnect create-private-virtual-interface \
--connection-id dxcon-xxxxxxxxx \
--new-private-virtual-interface '{
"virtualInterfaceName": "Single-VPC-VIF",
"vlan": 100,
"asn": 65001,
"amazonAddress": "192.168.254.2/30",
"customerAddress": "192.168.254.1/30",
"directConnectGatewayId": "dxgw-12345678"
}'
# 5. Attach your single VPC to Transit Gateway
aws ec2 create-transit-gateway-vpc-attachment \
--transit-gateway-id tgw-0abcd1234efgh5678 \
--vpc-id vpc-0123456789abcdef0 \
--subnet-ids subnet-0123456789abcdef0
Alternative: Even Simpler with Transit VIF
# Most efficient: Transit VIF directly to Transit Gateway
# Eliminates DX Gateway entirely!
# 1. Create Transit Gateway (same as above)
aws ec2 create-transit-gateway \
--description "Single VPC Transit Gateway"
# 2. Create Transit VIF (NO DX Gateway needed!)
aws directconnect create-transit-virtual-interface \
--connection-id dxcon-xxxxxxxxx \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Direct-Transit-VIF",
"vlan": 200,
"asn": 65001,
"amazonAddress": "192.168.254.6/30",
"customerAddress": "192.168.254.5/30",
"directConnectGatewayId": "dxgw-12345678"
}'
# 3. Attach VPC to Transit Gateway (same as above)
aws ec2 create-transit-gateway-vpc-attachment \
--transit-gateway-id tgw-0abcd1234efgh5678 \
--vpc-id vpc-0123456789abcdef0 \
--subnet-ids subnet-0123456789abcdef0
3 Types of Virtual Interfaces (VIFs) - Complete Overview
YES - There are exactly 3 types of VIFs in AWS Direct Connect:
- Private VIF - Connects to VPCs via VGW or DX Gateway
- Public VIF - Connects to AWS public services
- Transit VIF - Connects directly to Transit Gateway
graph TB
subgraph "3 Types of Virtual Interfaces"
subgraph "1. Private VIF"
P1[On-Premises] --> P2[Private VIF
RFC 1918 IPs] P2 --> P3A[Option A: VGW → Single VPC] P2 --> P3B[Option B: DX Gateway → TGW → Multiple VPCs] end subgraph "2. Public VIF" PU1[On-Premises] --> PU2[Public VIF
Public IPs] --> PU3[AWS Public Services
S3, DynamoDB, etc.] end subgraph "3. Transit VIF" T1[On-Premises] --> T2[Transit VIF
RFC 1918 IPs] --> T3[Transit Gateway] --> T4[Multiple VPCs
Cross-Region] end end style P2 fill:#e3f2fd style PU2 fill:#fff3e0 style T2 fill:#e8f5e8
RFC 1918 IPs] P2 --> P3A[Option A: VGW → Single VPC] P2 --> P3B[Option B: DX Gateway → TGW → Multiple VPCs] end subgraph "2. Public VIF" PU1[On-Premises] --> PU2[Public VIF
Public IPs] --> PU3[AWS Public Services
S3, DynamoDB, etc.] end subgraph "3. Transit VIF" T1[On-Premises] --> T2[Transit VIF
RFC 1918 IPs] --> T3[Transit Gateway] --> T4[Multiple VPCs
Cross-Region] end end style P2 fill:#e3f2fd style PU2 fill:#fff3e0 style T2 fill:#e8f5e8
Complete VIF Type Comparison
| VIF Type | Purpose | IP Addressing | Connects To | Max Bandwidth | Use Case |
|---|---|---|---|---|---|
| Private VIF | VPC access | Private (RFC 1918) | VGW or DX Gateway | 10 Gbps | Traditional VPC connectivity |
| Public VIF | AWS public services | Public IPs | AWS Public Zone | 10 Gbps | S3, DynamoDB, CloudFront, etc. |
| Transit VIF | Multi-VPC via TGW | Private (RFC 1918) | Transit Gateway directly | 100 Gbps | Modern multi-VPC architecture |
Important Clarification About Transit VIF
Key Point: Transit VIF is a newer VIF type introduced specifically for Transit Gateway. In my earlier example, I made an error - you still need to create a Direct Connect Gateway even for Transit VIF, but the Transit VIF connects directly to the Transit Gateway through that DX Gateway.
Corrected Transit VIF Configuration
# CORRECTED: Transit VIF still needs a DX Gateway
# But it's optimized for Transit Gateway connectivity
# 1. Create Transit Gateway
aws ec2 create-transit-gateway --description "TGW for Transit VIF"
# 2. Create Direct Connect Gateway (still needed)
aws directconnect create-direct-connect-gateway \
--name "Transit-VIF-DX-Gateway" \
--amazon-side-asn 64512
# 3. Associate DX Gateway with Transit Gateway
aws directconnect create-direct-connect-gateway-association-proposal \
--direct-connect-gateway-id dxgw-12345678 \
--gateway-id tgw-0abcd1234efgh5678
# 4. Create Transit VIF (optimized for TGW)
aws directconnect create-transit-virtual-interface \
--connection-id dxcon-xxxxxxxxx \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Transit-VIF-for-TGW",
"vlan": 300,
"asn": 65001,
"amazonAddress": "192.168.254.10/30",
"customerAddress": "192.168.254.9/30",
"directConnectGatewayId": "dxgw-12345678"
}'
Why Use Each VIF Type?
1. Private VIF - When to Use
- Single VPC: Connect directly via VGW
- Multiple VPCs: Use DX Gateway → Transit Gateway
- Legacy environments: Already established Private VIF infrastructure
- Simple setups: Basic VPC connectivity needs
2. Public VIF - When to Use
- AWS Public Services: S3, DynamoDB, SQS, SNS access
- Bypass Internet: Private connection to public AWS services
- Compliance: Avoid internet routing for public service access
- Performance: Dedicated bandwidth for public service access
3. Transit VIF - When to Use
- High Performance: Need more than 10G bandwidth
- Modern Architecture: New deployments with Transit Gateway
- Multiple VPCs: Optimized for TGW routing
- Cross-Region: TGW peering across regions
- Future-Proof: Latest AWS networking technology
Can You Have Multiple Transit VIFs?
YES! You can have multiple Transit VIFs on the same Direct Connect connection:
- Multiple Transit VIFs per connection: Each using different VLANs
- Different Transit Gateways: Each Transit VIF can connect to different TGWs
- Different regions: Transit VIFs can connect to TGWs in different AWS regions
- Bandwidth aggregation: Combine multiple Transit VIFs for higher total bandwidth
- Redundancy: Multiple paths for high availability
Multiple Transit VIF Scenarios
graph TB
subgraph "Single Direct Connect Connection"
DC[Direct Connect
dxcon-123456789] subgraph "Multiple Transit VIFs" TVIF1[Transit VIF 1
VLAN 100
Production] TVIF2[Transit VIF 2
VLAN 200
Development] TVIF3[Transit VIF 3
VLAN 300
Cross-Region] end end subgraph "Multiple Transit Gateways" TGW1[TGW Production
us-east-1] TGW2[TGW Development
us-east-1] TGW3[TGW Staging
us-west-2] end DC --> TVIF1 DC --> TVIF2 DC --> TVIF3 TVIF1 --> TGW1 TVIF2 --> TGW2 TVIF3 --> TGW3
dxcon-123456789] subgraph "Multiple Transit VIFs" TVIF1[Transit VIF 1
VLAN 100
Production] TVIF2[Transit VIF 2
VLAN 200
Development] TVIF3[Transit VIF 3
VLAN 300
Cross-Region] end end subgraph "Multiple Transit Gateways" TGW1[TGW Production
us-east-1] TGW2[TGW Development
us-east-1] TGW3[TGW Staging
us-west-2] end DC --> TVIF1 DC --> TVIF2 DC --> TVIF3 TVIF1 --> TGW1 TVIF2 --> TGW2 TVIF3 --> TGW3
Use Cases for Multiple Transit VIFs
| Scenario | Why Multiple Transit VIFs | Configuration | Benefits |
|---|---|---|---|
| Environment Separation | Isolate Prod/Dev/Test | 1 Transit VIF per environment | Security isolation, separate routing |
| Cross-Region | Connect to multiple AWS regions | 1 Transit VIF per region | Regional redundancy, compliance |
| High Bandwidth | Aggregate bandwidth >100G | Multiple Transit VIFs to same TGW | Higher total throughput |
| Redundancy | Active/passive or load balancing | 2+ Transit VIFs, different paths | High availability, failover |
| Different Customers | Service provider scenario | 1 Transit VIF per customer account | Customer isolation, billing |
Configuration Example: Multiple Transit VIFs
# Example: 3 Transit VIFs for different environments
# Get Direct Connect connection ID
DX_CONNECTION_ID="dxcon-123456789abcdef0"
# Create 3 different Direct Connect Gateways
aws directconnect create-direct-connect-gateway \
--name "Production-DX-Gateway" \
--amazon-side-asn 64512
aws directconnect create-direct-connect-gateway \
--name "Development-DX-Gateway" \
--amazon-side-asn 64513
aws directconnect create-direct-connect-gateway \
--name "Cross-Region-DX-Gateway" \
--amazon-side-asn 64514
# Create Transit VIF #1 - Production (VLAN 100)
aws directconnect create-transit-virtual-interface \
--connection-id $DX_CONNECTION_ID \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Production-Transit-VIF",
"vlan": 100,
"asn": 65001,
"amazonAddress": "192.168.254.2/30",
"customerAddress": "192.168.254.1/30",
"directConnectGatewayId": "dxgw-prod123456"
}'
# Create Transit VIF #2 - Development (VLAN 200)
aws directconnect create-transit-virtual-interface \
--connection-id $DX_CONNECTION_ID \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Development-Transit-VIF",
"vlan": 200,
"asn": 65001,
"amazonAddress": "192.168.254.6/30",
"customerAddress": "192.168.254.5/30",
"directConnectGatewayId": "dxgw-dev123456"
}'
# Create Transit VIF #3 - Cross-Region (VLAN 300)
aws directconnect create-transit-virtual-interface \
--connection-id $DX_CONNECTION_ID \
--new-transit-virtual-interface '{
"virtualInterfaceName": "CrossRegion-Transit-VIF",
"vlan": 300,
"asn": 65001,
"amazonAddress": "192.168.254.10/30",
"customerAddress": "192.168.254.9/30",
"directConnectGatewayId": "dxgw-xregion123456"
}'
Customer Router Configuration for Multiple Transit VIFs
! Configure multiple sub-interfaces for different Transit VIFs
! Transit VIF 1 - Production (VLAN 100)
interface GigabitEthernet0/0/1.100
description Production Transit VIF to AWS
encapsulation dot1Q 100
ip address 192.168.254.1 255.255.255.252
no ip redirects
! Transit VIF 2 - Development (VLAN 200)
interface GigabitEthernet0/0/1.200
description Development Transit VIF to AWS
encapsulation dot1Q 200
ip address 192.168.254.5 255.255.255.252
no ip redirects
! Transit VIF 3 - Cross-Region (VLAN 300)
interface GigabitEthernet0/0/1.300
description Cross-Region Transit VIF to AWS
encapsulation dot1Q 300
ip address 192.168.254.9 255.255.255.252
no ip redirects
! BGP Configuration for multiple neighbors
router bgp 65001
bgp log-neighbor-changes
! Production Transit VIF neighbor
neighbor 192.168.254.2 remote-as 64512
neighbor 192.168.254.2 description AWS-Production-TGW
neighbor 192.168.254.2 password prodSecretKey123
! Development Transit VIF neighbor
neighbor 192.168.254.6 remote-as 64513
neighbor 192.168.254.6 description AWS-Development-TGW
neighbor 192.168.254.6 password devSecretKey456
! Cross-Region Transit VIF neighbor
neighbor 192.168.254.10 remote-as 64514
neighbor 192.168.254.10 description AWS-CrossRegion-TGW
neighbor 192.168.254.10 password xregionSecretKey789
! Address family configuration
address-family ipv4
! Advertise different networks to different environments
network 192.168.0.0 mask 255.255.0.0
! Activate all neighbors
neighbor 192.168.254.2 activate
neighbor 192.168.254.6 activate
neighbor 192.168.254.10 activate
! Apply different route policies per environment
neighbor 192.168.254.2 prefix-list PROD-OUT out
neighbor 192.168.254.6 prefix-list DEV-OUT out
neighbor 192.168.254.10 prefix-list XREGION-OUT out
exit-address-family
! Different prefix lists for different environments
ip prefix-list PROD-OUT seq 10 permit 192.168.0.0/16
ip prefix-list DEV-OUT seq 10 permit 192.168.0.0/16
ip prefix-list XREGION-OUT seq 10 permit 192.168.0.0/16
Bandwidth Aggregation with Multiple Transit VIFs
Important: Multiple Transit VIFs to the same Transit Gateway can provide:
- Bandwidth Aggregation: 2x 100G Transit VIFs = 200G total bandwidth
- Load Distribution: Traffic automatically load-balances across VIFs
- Redundancy: If one VIF fails, traffic continues on remaining VIFs
- ECMP (Equal Cost Multi-Path): BGP supports multiple equal-cost paths
Limitations and Considerations
Keep in Mind:
- VLAN Limits: Each VIF needs a unique VLAN ID (1-4094)
- BGP Sessions: Each Transit VIF creates a separate BGP session
- Routing Complexity: More VIFs = more complex routing policies
- Cost: Each VIF has hourly charges, plan accordingly
- Port Capacity: Total bandwidth limited by DX connection speed
YES! You can have multiple VIFs of different types on the same Direct Connect connection:
- VLAN 100: Private VIF for legacy VPC
- VLAN 200: Public VIF for S3 access
- VLAN 300: Transit VIF for new multi-VPC architecture
Private VIFs connect to VPCs through various paths depending on your architecture choice.
Private VIF Characteristics:
- Uses private IP addressing (RFC 1918)
- Connects to VPCs via VGW, DX Gateway, or directly to TGW (Transit VIF)
- Supports BGP routing
- Can connect to Transit Gateway via Direct Connect Gateway OR directly
2. Public VIF
Public VIFs provide access to AWS public services using public IP addresses.
Public VIF Characteristics:
- Uses public IP addressing
- Accesses AWS public services (S3, DynamoDB, etc.)
- Does NOT connect to VPC resources
- Requires public ASN and public IP prefixes
3. Transit VIF
Transit VIFs are specifically designed to connect directly to Transit Gateway, providing the most efficient path for multi-VPC connectivity.
Transit VIF Characteristics:
- Connects directly to Transit Gateway (no Direct Connect Gateway needed)
- Supports up to 100 Gbps bandwidth
- Provides the most efficient routing path
- Supports advanced routing features
graph LR
subgraph "VIF Types Comparison"
subgraph "Private VIF via DX Gateway"
P1[On-Premises] --> P2[Private VIF] --> P3[DX Gateway] --> P4[Transit Gateway]
end
subgraph "Transit VIF (Direct)"
T1[On-Premises] --> T2[Transit VIF] --> T3[Transit Gateway]
end
subgraph "Public VIF"
PU1[On-Premises] --> PU2[Public VIF] --> PU3[AWS Public Services]
end
end
Detailed Architecture Diagrams
Complete Network Architecture
graph TB
subgraph "On-Premises Data Center"
DC[Data Center
192.168.0.0/16] CE[Customer Edge Router
BGP ASN: 65001] SW[Core Switch] SRV[Servers/Workloads] end subgraph "AWS Direct Connect Location" DXL[Direct Connect Location
Equinix/CoreSite/etc.] DXP[Direct Connect Port
1G/10G/100G] AWSR[AWS Router] end subgraph "AWS us-east-1" DXGW[Direct Connect Gateway
dxgw-12345678] TGW1[Transit Gateway
tgw-0abcd1234efgh5678] subgraph "Production VPC" PROD[VPC-Production
10.1.0.0/16] PRODSUB1[Private Subnet
10.1.1.0/24] PRODSUB2[Public Subnet
10.1.2.0/24] PRODEC2[EC2 Instances] end subgraph "Development VPC" DEV[VPC-Development
10.2.0.0/16] DEVSUB1[Private Subnet
10.2.1.0/24] DEVEC2[EC2 Instances] end end subgraph "AWS us-west-2" TGW2[Transit Gateway
tgw-0wxyz9876abcd1234] subgraph "Staging VPC" STAGE[VPC-Staging
10.3.0.0/16] STAGESUB1[Private Subnet
10.3.1.0/24] STAGEEC2[EC2 Instances] end end %% Connections DC --> SW SW --> SRV SW --> CE CE -.->|Cross Connect| DXL DXL --> DXP DXP --> AWSR AWSR --> DXGW DXGW --> TGW1 TGW1 --> PROD TGW1 --> DEV PROD --> PRODSUB1 PROD --> PRODSUB2 PRODSUB1 --> PRODEC2 DEV --> DEVSUB1 DEVSUB1 --> DEVEC2 %% Cross-region peering TGW1 -.->|TGW Peering| TGW2 TGW2 --> STAGE STAGE --> STAGESUB1 STAGESUB1 --> STAGEEC2
192.168.0.0/16] CE[Customer Edge Router
BGP ASN: 65001] SW[Core Switch] SRV[Servers/Workloads] end subgraph "AWS Direct Connect Location" DXL[Direct Connect Location
Equinix/CoreSite/etc.] DXP[Direct Connect Port
1G/10G/100G] AWSR[AWS Router] end subgraph "AWS us-east-1" DXGW[Direct Connect Gateway
dxgw-12345678] TGW1[Transit Gateway
tgw-0abcd1234efgh5678] subgraph "Production VPC" PROD[VPC-Production
10.1.0.0/16] PRODSUB1[Private Subnet
10.1.1.0/24] PRODSUB2[Public Subnet
10.1.2.0/24] PRODEC2[EC2 Instances] end subgraph "Development VPC" DEV[VPC-Development
10.2.0.0/16] DEVSUB1[Private Subnet
10.2.1.0/24] DEVEC2[EC2 Instances] end end subgraph "AWS us-west-2" TGW2[Transit Gateway
tgw-0wxyz9876abcd1234] subgraph "Staging VPC" STAGE[VPC-Staging
10.3.0.0/16] STAGESUB1[Private Subnet
10.3.1.0/24] STAGEEC2[EC2 Instances] end end %% Connections DC --> SW SW --> SRV SW --> CE CE -.->|Cross Connect| DXL DXL --> DXP DXP --> AWSR AWSR --> DXGW DXGW --> TGW1 TGW1 --> PROD TGW1 --> DEV PROD --> PRODSUB1 PROD --> PRODSUB2 PRODSUB1 --> PRODEC2 DEV --> DEVSUB1 DEVSUB1 --> DEVEC2 %% Cross-region peering TGW1 -.->|TGW Peering| TGW2 TGW2 --> STAGE STAGE --> STAGESUB1 STAGESUB1 --> STAGEEC2
BGP Routing Flow
sequenceDiagram
participant CE as Customer Edge
participant AWS as AWS Router
participant DXGW as DX Gateway
participant TGW as Transit Gateway
participant VPC as VPC Route Tables
Note over CE,VPC: BGP Session Establishment
CE->>AWS: BGP OPEN (ASN 65001)
AWS->>CE: BGP OPEN (ASN 64512)
Note over CE,VPC: Route Advertisement from On-Premises
CE->>AWS: BGP UPDATE: 192.168.0.0/16
AWS->>DXGW: Route: 192.168.0.0/16
DXGW->>TGW: Route: 192.168.0.0/16
TGW->>VPC: Route: 192.168.0.0/16
Note over CE,VPC: Route Advertisement from AWS
VPC->>TGW: Route: 10.1.0.0/16, 10.2.0.0/16
TGW->>DXGW: Route: 10.1.0.0/16, 10.2.0.0/16
DXGW->>AWS: Route: 10.1.0.0/16, 10.2.0.0/16
AWS->>CE: BGP UPDATE: 10.1.0.0/16, 10.2.0.0/16
Prerequisites and Planning
Network Planning
| Component | Requirement | Example |
|---|---|---|
| On-Premises CIDR | Non-overlapping with AWS VPCs | 192.168.0.0/16 |
| BGP ASN (Customer) | Private or Public ASN | 65001 (Private) |
| VLAN ID | Unique per VIF (100-4094) | 100 |
| BGP Authentication Key | MD5 key for BGP security | mySecretKey123 |
| Connection Speed | 1G, 10G, or 100G | 10G |
IP Addressing Scheme
# On-Premises Network
On-Premises CIDR: 192.168.0.0/16
- Management Network: 192.168.1.0/24
- Server Network: 192.168.10.0/24
- Point-to-Point Link: 192.168.254.0/30
# AWS VPC Networks
Production VPC: 10.1.0.0/16
- Private Subnets: 10.1.1.0/24, 10.1.2.0/24
- Public Subnets: 10.1.10.0/24, 10.1.11.0/24
Development VPC: 10.2.0.0/16
- Private Subnets: 10.2.1.0/24, 10.2.2.0/24
# BGP Peering Network (for VIF)
Customer Side: 192.168.254.1/30
AWS Side: 192.168.254.2/30
Transit Gateway Setup
Create Transit Gateway
# Create Transit Gateway
aws ec2 create-transit-gateway \
--description "Main Transit Gateway for Direct Connect" \
--options DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable,AutoAcceptSharedAttachments=enable \
--tag-specifications 'ResourceType=transit-gateway,Tags=[{Key=Name,Value=Main-TGW},{Key=Environment,Value=Production}]' \
--region us-east-1
Expected Output:
Save the Transit Gateway ID (e.g., tgw-0abcd1234efgh5678) - you'll need this for subsequent commands.
Save the Transit Gateway ID (e.g., tgw-0abcd1234efgh5678) - you'll need this for subsequent commands.
Transit Gateway Parameters Explained
| Parameter | Description | Default |
|---|---|---|
| DefaultRouteTableAssociation | Automatically associate attachments with default route table | enable |
| DefaultRouteTablePropagation | Automatically propagate routes to default route table | enable |
| AutoAcceptSharedAttachments | Automatically accept shared resource attachments | disable |
| DnsSupport | Enable DNS resolution for VPC attachments | enable |
Create VPC Attachments
# Get VPC IDs first
PROD_VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Production-VPC" --query 'Vpcs[0].VpcId' --output text)
DEV_VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=Development-VPC" --query 'Vpcs[0].VpcId' --output text)
# Get subnet IDs for attachments
PROD_SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$PROD_VPC_ID" "Name=tag:Name,Values=*Private*" --query 'Subnets[0].SubnetId' --output text)
DEV_SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$DEV_VPC_ID" "Name=tag:Name,Values=*Private*" --query 'Subnets[0].SubnetId' --output text)
# Create VPC attachments
aws ec2 create-transit-gateway-vpc-attachment \
--transit-gateway-id tgw-0abcd1234efgh5678 \
--vpc-id $PROD_VPC_ID \
--subnet-ids $PROD_SUBNET_ID \
--tag-specifications 'ResourceType=transit-gateway-attachment,Tags=[{Key=Name,Value=Production-VPC-Attachment}]'
aws ec2 create-transit-gateway-vpc-attachment \
--transit-gateway-id tgw-0abcd1234efgh5678 \
--vpc-id $DEV_VPC_ID \
--subnet-ids $DEV_SUBNET_ID \
--tag-specifications 'ResourceType=transit-gateway-attachment,Tags=[{Key=Name,Value=Development-VPC-Attachment}]'
Important: VPC attachments require at least one subnet per Availability Zone. For high availability, attach subnets from multiple AZs.
Direct Connect Gateway Configuration
Create Direct Connect Gateway
# Create Direct Connect Gateway
aws directconnect create-direct-connect-gateway \
--name "Main-DX-Gateway" \
--amazon-side-asn 64512
Expected Output:
Save the Direct Connect Gateway ID (e.g., dxgw-12345678) - you'll need this for VIF creation and TGW association.
Save the Direct Connect Gateway ID (e.g., dxgw-12345678) - you'll need this for VIF creation and TGW association.
Associate Direct Connect Gateway with Transit Gateway
# Create association proposal
aws directconnect create-direct-connect-gateway-association-proposal \
--direct-connect-gateway-id dxgw-12345678 \
--direct-connect-gateway-owner-account 123456789012 \
--gateway-id tgw-0abcd1234efgh5678 \
--add-allowed-prefixes-to-direct-connect-gateway Cidr=10.1.0.0/16 \
--add-allowed-prefixes-to-direct-connect-gateway Cidr=10.2.0.0/16
# Accept the association (if in same account)
PROPOSAL_ID=$(aws directconnect describe-direct-connect-gateway-association-proposals \
--direct-connect-gateway-id dxgw-12345678 \
--query 'directConnectGatewayAssociationProposals[0].proposalId' --output text)
aws directconnect accept-direct-connect-gateway-association-proposal \
--direct-connect-gateway-association-proposal-id $PROPOSAL_ID \
--associated-gateway-owner-account 123456789012 \
--override-allowed-prefixes-to-direct-connect-gateway Cidr=10.1.0.0/16 \
--override-allowed-prefixes-to-direct-connect-gateway Cidr=10.2.0.0/16
Direct Connect Gateway Parameters
| Parameter | Description | Value/Range |
|---|---|---|
| amazon-side-asn | AWS side BGP ASN for the gateway | 64512 (default) or 4200000000-4294967294 |
| allowed-prefixes | CIDR blocks allowed to be advertised from AWS | VPC CIDR blocks (e.g., 10.1.0.0/16) |
| gateway-id | Transit Gateway ID to associate | tgw-xxxxxxxxxxxxxxxxx |
Virtual Interface Configuration
Create Private Virtual Interface
# Get your Direct Connect connection ID first
DX_CONNECTION_ID=$(aws directconnect describe-connections --query 'connections[0].connectionId' --output text)
# Create Private VIF
aws directconnect create-private-virtual-interface \
--connection-id $DX_CONNECTION_ID \
--new-private-virtual-interface '{
"virtualInterfaceName": "Production-Private-VIF",
"vlan": 100,
"asn": 65001,
"authKey": "mySecretKey123",
"amazonAddress": "192.168.254.2/30",
"customerAddress": "192.168.254.1/30",
"directConnectGatewayId": "dxgw-12345678",
"tags": [
{
"key": "Name",
"value": "Production-Private-VIF"
},
{
"key": "Environment",
"value": "Production"
}
]
}'
Expected Output:
Save the Virtual Interface ID (e.g., dxvif-abcd1234) for monitoring and troubleshooting.
Save the Virtual Interface ID (e.g., dxvif-abcd1234) for monitoring and troubleshooting.
Virtual Interface Parameters Detailed
| Parameter | Description | Example/Notes |
|---|---|---|
| virtualInterfaceName | Friendly name for the VIF | Production-Private-VIF |
| vlan | VLAN ID for traffic separation | 100 (range: 1-4094, must be unique per connection) |
| asn | Customer BGP ASN | 65001 (private: 64512-65534, public: registered ASN) |
| authKey | BGP MD5 authentication key | Optional but recommended for security |
| amazonAddress | AWS side IP address | 192.168.254.2/30 (must be /30 or /31) |
| customerAddress | Customer side IP address | 192.168.254.1/30 (must be in same subnet as amazonAddress) |
Alternative: Create Transit Virtual Interface
# Create Transit VIF (direct to Transit Gateway - more efficient)
aws directconnect create-transit-virtual-interface \
--connection-id $DX_CONNECTION_ID \
--new-transit-virtual-interface '{
"virtualInterfaceName": "Production-Transit-VIF",
"vlan": 200,
"asn": 65001,
"authKey": "mySecretKey123",
"amazonAddress": "192.168.254.6/30",
"customerAddress": "192.168.254.5/30",
"directConnectGatewayId": "dxgw-12345678",
"tags": [
{
"key": "Name",
"value": "Production-Transit-VIF"
},
{
"key": "Type",
"value": "Transit"
}
]
}'
Transit VIF vs Private VIF:
- Transit VIF: Connects directly to Transit Gateway, supports higher bandwidth (up to 100G), more efficient routing
- Private VIF: Connects via Direct Connect Gateway, traditional approach, maximum 10G per VIF
Customer Router Configuration (Cisco Example)
! Configure the sub-interface for VLAN 100
interface GigabitEthernet0/0/1.100
description Direct Connect to AWS Production VIF
encapsulation dot1Q 100
ip address 192.168.254.1 255.255.255.252
no ip redirects
no ip proxy-arp
! Configure BGP
router bgp 65001
bgp log-neighbor-changes
bgp graceful-restart
! Neighbor configuration for AWS
neighbor 192.168.254.2 remote-as 64512
neighbor 192.168.254.2 description AWS-DirectConnect-VIF
neighbor 192.168.254.2 password mySecretKey123
neighbor 192.168.254.2 timers 10 30
neighbor 192.168.254.2 soft-reconfiguration inbound
! Address family configuration
address-family ipv4
network 192.168.0.0 mask 255.255.0.0
neighbor 192.168.254.2 activate
neighbor 192.168.254.2 prefix-list ADVERTISE-TO-AWS out
neighbor 192.168.254.2 prefix-list ACCEPT-FROM-AWS in
maximum-paths 4
exit-address-family
! Prefix lists for route filtering
ip prefix-list ADVERTISE-TO-AWS seq 10 permit 192.168.0.0/16
ip prefix-list ADVERTISE-TO-AWS seq 20 deny 0.0.0.0/0 le 32
ip prefix-list ACCEPT-FROM-AWS seq 10 permit 10.1.0.0/16
ip prefix-list ACCEPT-FROM-AWS seq 20 permit 10.2.0.0/16
ip prefix-list ACCEPT-FROM-AWS seq 30 deny 0.0.0.0/0 le 32
! Static route for redundancy (higher metric)
ip route 10.1.0.0 255.255.0.0 192.168.254.2 200
ip route 10.2.0.0 255.255.0.0 192.168.254.2 200
BGP Timers Explanation:
- Keepalive Timer: 10 seconds (how often to send keepalive messages)
- Hold Timer: 30 seconds (how long to wait before declaring neighbor down)
- AWS Default: 30/90 seconds - adjust based on your requirements
Routing Configuration
Transit Gateway Route Tables
# Create custom route table for segmentation
aws ec2 create-transit-gateway-route-table \
--transit-gateway-id tgw-0abcd1234efgh5678 \
--tag-specifications 'ResourceType=transit-gateway-route-table,Tags=[{Key=Name,Value=Production-Routes},{Key=Environment,Value=Production}]'
# Get the route table ID from output
TGW_RT_ID="tgw-rtb-0123456789abcdef0"
# Associate VPC attachments with route table
PROD_ATTACHMENT_ID=$(aws ec2 describe-transit-gateway-vpc-attachments \
--filters "Name=tag:Name,Values=Production-VPC-Attachment" \
--query 'TransitGatewayVpcAttachments[0].TransitGatewayAttachmentId' --output text)
aws ec2 associate-transit-gateway-route-table \
--transit-gateway-attachment-id $PROD_ATTACHMENT_ID \
--transit-gateway-route-table-id $TGW_RT_ID
# Create routes for on-premises traffic
aws ec2 create-route \
--route-table-id $TGW_RT_ID \
--destination-cidr-block 192.168.0.0/16 \
--transit-gateway-attachment-id $DX_ATTACHMENT_ID
VPC Route Table Updates
# Get VPC route table IDs
PROD_RT_ID=$(aws ec2 describe-route-tables \
--filters "Name=vpc-id,Values=$PROD_VPC_ID" "Name=tag:Name,Values=*Private*" \
--query 'RouteTables[0].RouteTableId' --output text)
# Add route to on-premises network via Transit Gateway
aws ec2 create-route \
--route-table-id $PROD_RT_ID \
--destination-cidr-block 192.168.0.0/16 \
--transit-gateway-id tgw-0abcd1234efgh5678
# Add route to other VPCs via Transit Gateway
aws ec2 create-route \
--route-table-id $PROD_RT_ID \
--destination-cidr-block 10.2.0.0/16 \
--transit-gateway-id tgw-0abcd1234efgh5678
Route Propagation Configuration
# Enable route propagation from Direct Connect Gateway
aws ec2 enable-transit-gateway-route-table-propagation \
--transit-gateway-route-table-id $TGW_RT_ID \
--transit-gateway-attachment-id $DX_ATTACHMENT_ID
# Enable route propagation from VPC attachments
aws ec2 enable-transit-gateway-route-table-propagation \
--transit-gateway-route-table-id $TGW_RT_ID \
--transit-gateway-attachment-id $PROD_ATTACHMENT_ID
# View propagated routes
aws ec2 get-transit-gateway-route-table-propagations \
--transit-gateway-route-table-id $TGW_RT_ID
Route Priority and Path Selection
graph TD
subgraph "Route Selection Process"
A[Packet Arrives at TGW] --> B{Destination Match?}
B -->|Yes| C[Check Route Priority]
B -->|No| D[Drop Packet]
C --> E{Static Route?}
E -->|Yes| F[Use Static Route - Highest Priority]
E -->|No| G{Propagated Route?}
G -->|Yes| H[Check AS Path Length]
G -->|No| I[Default Route]
H --> J{Shorter AS Path?}
J -->|Yes| K[Select Route with Shorter Path]
J -->|No| L[Use Local Preference/MED]
F --> M[Forward Packet]
K --> M
L --> M
I --> M
end
Route Selection Priority (Highest to Lowest):
- Static Routes: Manually configured routes in TGW route tables
- BGP Routes: Routes learned via BGP with shortest AS path
- Local Preference: BGP attribute for policy-based routing
- MED (Multi-Exit Discriminator): BGP attribute for path preference
Troubleshooting and Verification
Verify Direct Connect Status
# Check Direct Connect connection status
aws directconnect describe-connections \
--connection-id dxcon-xxxxxxxxx
# Check Virtual Interface status
aws directconnect describe-virtual-interfaces \
--virtual-interface-id dxvif-abcd1234
# Check BGP peer status
aws directconnect describe-virtual-interfaces \
--virtual-interface-id dxvif-abcd1234 \
--query 'virtualInterfaces[0].bgpPeers'
Expected BGP Status Output
{
"bgpPeers": [
{
"bgpPeerId": "dxpeer-xxxxxxxx",
"asn": 65001,
"authKey": "mySecretKey123",
"addressFamily": "ipv4",
"amazonAddress": "192.168.254.2/30",
"customerAddress": "192.168.254.1/30",
"bgpPeerState": "established",
"bgpStatus": "up",
"awsDeviceV2": "EqDC2-19y7z3w8uk"
}
]
}
Transit Gateway Verification
# Check Transit Gateway attachments
aws ec2 describe-transit-gateway-attachments \
--transit-gateway-id tgw-0abcd1234efgh5678
# Check Transit Gateway route tables
aws ec2 describe-transit-gateway-route-tables \
--transit-gateway-route-table-ids $TGW_RT_ID
# Search routes in Transit Gateway route table
aws ec2 search-transit-gateway-routes \
--transit-gateway-route-table-id $TGW_RT_ID \
--filters Name=state,Values=active
# Check specific route
aws ec2 search-transit-gateway-routes \
--transit-gateway-route-table-id $TGW_RT_ID \
--filters Name=route-search.exact-match,Values=192.168.0.0/16
Network Connectivity Testing
# Test from on-premises to AWS VPC
ping 10.1.1.100 # Replace with actual EC2 private IP
# Test with traceroute to verify path
traceroute 10.1.1.100
# Test from AWS EC2 to on-premises
# (Run this from EC2 instance)
ping 192.168.10.100 # Replace with on-premises server IP
# Check routing table on EC2 instance
ip route show
route -n
BGP Route Verification (Customer Router)
! Check BGP neighbor status
show ip bgp summary
! Check received routes from AWS
show ip bgp neighbors 192.168.254.2 received-routes
! Check advertised routes to AWS
show ip bgp neighbors 192.168.254.2 advertised-routes
! Check BGP table
show ip bgp
! Check specific route
show ip route 10.1.0.0
! Debug BGP (use carefully in production)
debug ip bgp updates
debug ip bgp events
Common Issues and Solutions
| Issue | Symptoms | Solution |
|---|---|---|
| BGP Not Establishing | bgpStatus: down, no routes learned | Check VLAN config, IP addresses, ASN, auth key |
| Routes Not Propagating | BGP up but no routes in TGW table | Check allowed prefixes in DX Gateway association |
| Connectivity Issues | Can ping AWS side IP, but not VPC resources | Check VPC route tables, security groups, NACLs |
| Asymmetric Routing | Traffic goes one way but not return path | Verify route propagation in both directions |
| High Latency | Slow response times | Check for suboptimal routing, MTU issues |
Monitoring and Logging
# Enable VPC Flow Logs for troubleshooting
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids $PROD_VPC_ID \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name VPCFlowLogs \
--deliver-logs-permission-arn arn:aws:iam::123456789012:role/flowlogsRole
# Monitor Direct Connect metrics
aws cloudwatch get-metric-statistics \
--namespace AWS/DX \
--metric-name ConnectionBpsEgress \
--dimensions Name=ConnectionId,Value=dxcon-xxxxxxxxx \
--start-time 2025-07-01T00:00:00Z \
--end-time 2025-07-01T23:59:59Z \
--period 300 \
--statistics Average,Maximum
Key Metrics to Monitor:
- ConnectionBpsEgress/Ingress: Bandwidth utilization
- ConnectionPpsEgress/Ingress: Packet rate
- ConnectionCRCErrorCount: Physical layer errors
- ConnectionLightLevelLow: Optical signal issues
- VirtualInterfaceBpsEgress/Ingress: Per-VIF bandwidth
Performance Optimization
# MTU Optimization (set on customer router interface)
interface GigabitEthernet0/0/1.100
ip mtu 9000
# Enable jumbo frames if supported
# AWS supports up to 9000 byte MTU on Direct Connect
# TCP MSS Clamping (if needed)
ip tcp adjust-mss 8960
# Quality of Service (example)
class-map match-all CRITICAL
match dscp ef
class-map match-all HIGH
match dscp af31
policy-map DX-QOS
class CRITICAL
priority percent 20
class HIGH
bandwidth percent 60
class class-default
bandwidth percent 20
interface GigabitEthernet0/0/1.100
service-policy output DX-QOS