aliyuncontainerservice/terway-e2e

Terway E2E Test Workflow

Complete workflow for running Terway end-to-end tests on ACK clusters.

Overview

Terway is Alibaba Cloud's CNI plugin for Kubernetes. E2E tests validate:

• Prefix Mode: /28 IPv4 or /80 IPv6 prefix allocation on ENIs
• Connectivity: Pod-to-Pod, Pod-to-Service, NodePort, LoadBalancer
• PodNetworking: Custom network configurations via CRD
• SecurityGroup: Security group isolation and rules
• MultiNetwork: Multiple network interfaces per pod
• Trunk Mode: Member ENI attachment via trunk
• Exclusive ENI: One pod per ENI mode

Phase 1: Build & Push Image

Registry: Replace registry.cn-hangzhou.aliyuncs.com/acs in all examples below with your own image registry namespace (e.g., registry.cn-hangzhou.aliyuncs.com/<your-namespace>). Make sure you have push access to that namespace before building.

# Record the image tag (git short commit)
IMAGE_TAG=$(git rev-parse --short HEAD)
echo "Image tag: $IMAGE_TAG"

# If running directly on the Linux dev server:
cd /root/opensource_terway
make BUILD_PLATFORMS=linux/amd64,linux/arm64 \
  REGISTRY=registry.cn-hangzhou.aliyuncs.com/acs \
  build-push-terway build-push-terway-controlplane

# If building from macOS/remote, sync first then build:
PROJECT_ROOT=$(git rev-parse --show-toplevel)
PROJECT_NAME=$(basename "$PROJECT_ROOT")
rsync -ar --exclude=bin/ --delete "$PROJECT_ROOT/" root@dev:"/root/$PROJECT_NAME/"
ssh dev "bash -l -c 'cd /root/$PROJECT_NAME && make BUILD_PLATFORMS=linux/amd64,linux/arm64 REGISTRY=registry.cn-hangzhou.aliyuncs.com/acs build-push-terway build-push-terway-controlplane'"

CRITICAL — Registry Endpoint: Always use the public registry endpoint registry.cn-hangzhou.aliyuncs.com (NOT the VPC endpoint registry-vpc.cn-hangzhou.aliyuncs.com). The VPC endpoint causes dial tcp ...: i/o timeout from dev servers that are not inside the same VPC as the registry. The public endpoint works from any ECS in the same region.

Phase 2: Create ACK Cluster via Terraform

cd hack/terraform/ack/

# Initialize (first time only)
terraform init

# Preview changes
terraform plan

# Create cluster (takes 10-20 minutes)
terraform apply -auto-approve

# Verify kubeconfig was generated
ls kubeconfig-tf-ack-*
export KUBECONFIG=$(ls kubeconfig-tf-ack-* | head -1)

# Verify cluster is accessible
kubectl get nodes

Cluster Configuration

The Terraform configuration creates:

• Default node pool: Standard ECS nodes for shared ENI mode
• Exclusive ENI node pool: Nodes labeled for exclusive ENI mode
• IP Prefix node pool: Nodes labeled with k8s.aliyun.com/ip-prefix=true and k8s.aliyun.com/ignore-by-terway=true for dedicated prefix E2E testing. Initially isolated from terway scheduling; the E2E test removes the ignore label to bring them online.

IP Prefix Node Pool: The ip-prefix node pool starts with k8s.aliyun.com/ignore-by-terway=true so terway does not schedule onto these nodes at cluster creation time. The E2E test setup flow (setupIPPrefixNodes) removes this label and waits for terway to become ready before running tests. Teardown restores the label.

Key configuration files:

• terraform.tfvars: Cluster configuration (region, instance types, CIDRs)
• terraform_e2e.tf: Main Terraform configuration

Phase 3: Deploy Terway

cd hack/terraform/ack/

# Deploy with ENIMultiIP mode (required for Prefix/Trunk)
# --pull-policy Always ensures nodes re-pull the image on every restart (recommended for test builds)
./deploy-terway.sh \
  --tag <IMAGE_TAG> \
  --registry registry.cn-hangzhou.aliyuncs.com/acs \
  --daemon-mode ENIMultiIP \
  --pull-policy Always

# Or with Datapath V2 (kube-proxy replacement)
./deploy-terway.sh \
  --tag <IMAGE_TAG> \
  --registry registry.cn-hangzhou.aliyuncs.com/acs \
  --daemon-mode ENIMultiIP \
  --pull-policy Always \
  --enable-dp-v2 \
  --enable-network-policy

# Verify all pods are Running
kubectl get pods -n kube-system | grep -E 'terway'
# Expected:
#   terway-controlplane-xxx   2/2   Running
#   terway-eniip-xxx          2/2   Running  (one per node)

Note: deploy-terway.sh reads cluster metadata (clusterID, VPC, security group, vSwitches) automatically from terraform.tfstate. No manual values.yaml editing needed. The generated terway-values.yaml is overwritten on each run.

Deployment Options

| Option | Description | |---------------------------|------------------------------------------------------------| | --daemon-mode | ENIMultiIP (shared ENI) or ENIDirectIP (exclusive ENI) | | --ip-stack | ipv4, ipv6, or dual | | --enable-dp-v2 | Enable Datapath V2 (kube-proxy replacement) | | --enable-network-policy | Enable eBPF-based network policy |

Phase 4: Run E2E Tests

Test Categories

| Test | File | Description | |-------------------|-------------------------------------------|-----------------------------------------------| | Prefix | prefix_basic_test.go, prefix_boundary_test.go, prefix_helpers_test.go | Prefix allocation, boundary conditions, scale | | Prefix E2E | prefix_e2e_test.go | End-to-end prefix validation on dedicated node pool | | Prefix E2E | prefix_e2e_test.go | End-to-end Pod IP validation on dedicated prefix node pool | | Connectivity | connective_test.go, connectivity_*.go | Pod-to-Pod, Service connectivity | | PodNetworking | pod_networking_test.go | Custom network via CRD | | SecurityGroup | security_group_test.go | SG isolation and rules | | MultiNetwork | multi_network_test.go | Multiple ENIs per pod | | Upgrade | upgrade_test.go | Upgrade compatibility | | ERDMA | erdma/erdma_test.go | ERDMA device support |

Running Tests

# Set kubeconfig (e2e tests read ~/.kube/config by default)
cp hack/terraform/ack/kubeconfig-tf-ack-* ~/.kube/config
# OR: export KUBECONFIG=hack/terraform/ack/kubeconfig-tf-ack-*

# Run all prefix tests
cd /root/opensource_terway
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix' -region-id cn-hangzhou

# Run specific prefix suites
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_Basic' -region-id cn-hangzhou

go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_Boundary' -region-id cn-hangzhou

# Run connectivity tests
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestConnectivity' -region-id cn-hangzhou \
  -repo registry.cn-hangzhou.aliyuncs.com/acs

IMPORTANT: Use ./tests (not ./tests/...) — the e2e tests live directly in the tests package. The -region-id flag is required for prefix tests (used to query ECS API). The test binary reads ~/.kube/config automatically.

Prefix Tests

# Basic prefix allocation
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_Basic' -region-id cn-hangzhou

# Boundary conditions (max prefixes, exhaustion)
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_Boundary' -region-id cn-hangzhou

# Dual stack and EFLO tests
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_DualStack|TestPrefix_EFLO' -region-id cn-hangzhou

# State machine and scale tests
go test -v -count=1 -timeout 120m -tags e2e ./tests \
  -run 'TestPrefix_State|TestPrefix_Scale' -region-id cn-hangzhou

# Prefix E2E tests (dedicated node pool, end-to-end pod IP validation)
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestPrefix_E2E' -region-id cn-hangzhou

Prefix Node Capacity Reference (ECS g7ne)

| Instance | ENI adapters | Prefixes/ENI | Max prefix capacity | |--------------|--------------|--------------|---------------------| | g7ne.2xlarge | 6 | 14 | 84 | | g7ne.4xlarge | 8 | 14 | 112 |

Known Prefix Test Behavior

• Cleanup timeout warnings (resetNodePrefixState): Between tests, the 3-minute cleanup wait sometimes expires while Deleting prefixes are still being unassigned from ENIs via the ECS API. This is expected — tests emit a warning and proceed. The underlying IPPrefixStatusDeleting controller handles the ECS API call asynchronously.

• TestPrefix_Boundary_MaxValue: This test requests more prefixes than node capacity (e.g., 134 requested, 84 max). The test verifies no over-allocation occurs. If the system hasn't reached max capacity within the wait window, the test accepts any count ≤ maxCapacity and still passes.

• TestPrefix_Boundary_ENICapacity: Requests 17 prefixes when max-per-ENI is 14, verifying the system spreads them across 2+ ENIs.

Connectivity Tests

# All connectivity tests
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestConnectivity' -region-id cn-hangzhou \
  -repo registry.cn-hangzhou.aliyuncs.com/acs

# Specific scenarios
go test -v -count=1 -timeout 60m -tags e2e ./tests \
  -run 'TestConnectivity_NodePort' -region-id cn-hangzhou \
  -repo registry.cn-hangzhou.aliyuncs.com/acs

Phase 5: Issue Discovery & Analysis

When tests fail or when validating feature implementations, analyze both the test code and the implementation to discover issues.

Check Test Implementation

1. Test Prerequisites: Check if tests verify cluster configuration correctly

   // Example: Check IPAM type
   func checkIPAMType(t *testing.T) {
       cm := &corev1.ConfigMap{}
       err := client.Resources().Get(ctx, "eni-config", "kube-system", cm)
       // Verify ipam_type is "crd" for centralized IPAM
   }
   

2. Node Selection: Tests should use proper node affinity

   // Tests need correct node type selection
   nodeInfo, _ := DiscoverNodeTypesWithCapacity(ctx, client)
   if len(nodeInfo.ECSSharedENINodes) == 0 {
       t.Skip("No ECS Shared ENI nodes available")
   }
   

3. Resource Cleanup: Verify tests clean up resources properly

   • PodENI CRs
   • PodNetworking CRs
   • ConfigMaps (Dynamic Config)
   • Node labels

Check Feature Requirements

1. Version Requirements: Some features require specific Terway versions

   // Example from tests
   if !RequireTerwayVersion("v1.17.0") {
       t.Skip("Requires Terway >= v1.17.0")
   }
   

2. Cluster Configuration:

   • Centralized IPAM (ipam_type: crd) required for Prefix
   • Trunk mode enabled for member ENI tests
   • Dual-stack enabled for IPv6 tests

3. Node Labels:

   • terway-config: <configmap-name> for Dynamic Config
   • ip-prefix: "true" for prefix mode
   • k8s.aliyun.com/exclusive-mode-eni-type for exclusive ENI

Common Issues

| Issue | Cause | Solution | |------------------------------------------------|------------------------------------------------------|---------------------------------------------------------------------------------------| | dial tcp ...: i/o timeout during push | Using VPC registry endpoint from wrong network | Switch to public endpoint: registry.cn-hangzhou.aliyuncs.com (NOT registry-vpc.*) | | Tests skip with "ipam type is not crd" | Cluster uses non-centralized IPAM | Ensure ipam_type: crd in eni-config | | Tests skip with "No ECS Shared ENI nodes" | Only Lingjun/EFLO nodes present | Add standard ECS nodes | | Prefix not allocated | Missing node labels | Add ip-prefix: "true" and terway-config labels | | Pod stuck in Creating | ENI quota exceeded | Check ENI quota or use prefix mode | | Connectivity failures | Security group rules | Check SG allows pod traffic | | resetNodePrefixState cleanup timeout warning | Deleting prefixes take >3m to unassign via ECS API | Expected behavior — test continues; no fix needed | | Nodes in NotReady state after cluster creation | CNI not yet installed | Deploy Terway via deploy-terway.sh first |

Analyzing Test Failures

1. Check Pod Status:

   kubectl get pods -n <namespace> -o wide
   kubectl describe pod <pod-name> -n <namespace>
   kubectl logs <pod-name> -n <namespace>
   

2. Check Terway Logs:

   kubectl logs -n kube-system <terway-eniip-pod> -c terway
   kubectl logs -n kube-system <terway-controlplane-pod>
   

3. Check CR Status:

   kubectl get podeni -A
   kubectl describe podeni <name> -n <namespace>
   kubectl get podnetworking
   kubectl describe podnetworking <name>
   

4. Check Node Resources:

   kubectl describe node <node-name> | grep -A 10 "Allocatable"
   # Check for aliyun/eni, aliyun/member-eni resources
   

Phase 6: Cleanup

cd hack/terraform/ack/
terraform destroy -auto-approve

Key Files Reference

| File | Purpose | |---------------------------------------|-------------------------------------------| | hack/terraform/ack/ | Terraform configs for ACK cluster | | hack/terraform/ack/deploy-terway.sh | Helm-based Terway deployment script | | tests/main_test.go | Test initialization and cluster discovery | | tests/utils_test.go | Test utilities (Pod, Service builders) | | tests/node_utils_test.go | Node type discovery and capacity checks | | tests/prefix_*.go | Prefix mode tests | | tests/connective*.go | Connectivity tests | | tests/pod_networking_test.go | PodNetworking CRD tests | | tests/security_group_test.go | Security group tests | | types/daemon/dynamicconfig.go | Dynamic Config implementation |

Node Types

| Type | Label | Description | |-------------------------|--------------------------------------------------|-------------------------------------| | ecs-shared-eni | (default) | Standard ECS nodes, shared ENI mode | | ecs-exclusive-eni | k8s.aliyun.com/exclusive-mode-eni-type=eniOnly | ECS nodes, exclusive ENI | | ecs-ip-prefix | k8s.aliyun.com/ip-prefix=true | ECS nodes, dedicated IP Prefix E2E pool | | lingjun-shared-eni | alibabacloud.com/lingjun-worker=true | Lingjun/EFLO nodes, shared ENI | | lingjun-exclusive-eni | Both labels above | Lingjun nodes, exclusive ENI |

Test Flags

| Flag | Default | Description | |-----------------------|------------------------------------------------|-------------------------------| | -repo | registry.cn-hangzhou.aliyuncs.com/build-test | Image registry | | -timeout | 2m | Default test timeout | | -enable-trunk | true | Enable trunk mode tests | | -region-id | cn-hangzhou | Alibaba Cloud region | | -vswitch-ids | "" | Additional vSwitch IDs | | -security-group-ids | "" | Additional security group IDs |