**Help improve this page** To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page. # Set up Amazon EKS cluster for AI/ML workloads using Terraform **Tip** [Register](https://awsworkshop.io/eks/genai?trk=4a9b4147-2490-4c63-bc9f-f8a84b122c8c&sc_channel=el) for upcoming Amazon EKS AI/ML workshops. This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster with EKS Auto Mode or Karpenter for node auto-scaling, load balancing for in-cluster endpoints, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. You create the GPU NodePool that provisions GPU-enabled nodes in the [Create a GPU NodePool for AI/ML inference](ml-inference-gpu-nodepool.md) section. See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest/userguide/automode.html) and [Karpenter](https://karpenter.sh/docs/) for more information on how those features provision and auto-scale EC2 instances in EKS clusters. The following diagram shows the AWS high-level architecture for this section’s setup. Inside your AWS account, the EKS cluster uses a NodeClass and NodePool to provision an On-Demand Capacity Reservation (ODCR) G instance and a Spot or On-Demand G instance. In the monitoring stack, Prometheus remote writes metrics to Amazon Managed Service for Prometheus, and Grafana reads them. An Amazon S3 bucket stores model weights. The NodeClass and NodePool are shown for context; you create them in the [Create a GPU NodePool for AI/ML inference](ml-inference-gpu-nodepool.md) section. ![The EKS cluster, monitoring stack, and Amazon S3 bucket.](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-tf-architecture.png) ## Prerequisites **Important** The resources you create in this tutorial, including EKS clusters, GPU instances, Application Load Balancers, and Amazon Managed Service for Prometheus, incur charges. Delete resources when you finish to avoid ongoing charges. + Terraform >= 1.15.0. For setup instructions, see [Installing Terraform](https://developer.hashicorp.com/terraform/install). + `kubectl` >= 1.37. For setup instructions, see [Set up `kubectl` and `eksctl`](install-kubectl.md). + AWS CLI >= 2.27. For setup instructions, see [Installing](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). + `jq`. For setup instructions, see [Download jq](https://jqlang.github.io/jq/download/). + `git`, to clone the sample repository. + At least one unused VPC in your AWS Region’s quota. The Terraform code always creates its own VPC, and the default quota is 5 VPCs per Region. Check how many VPCs the Region already has against your quota: ``` aws ec2 describe-vpcs --region [.replaceable]`region-code` --query 'length(Vpcs)' aws service-quotas get-service-quota \ --service-code vpc \ --quota-code L-F678F1CE \ --query 'Quota.Value' \ --region [.replaceable]`region-code` ``` **Important** If the Region is already at its VPC quota, the apply fails partway through with `VpcLimitExceeded` after it has already created IAM roles, KMS keys, the S3 bucket, and the AMP workspace. Run `terraform destroy` to remove those resources before you retry in another Region, or request a quota increase. For more information, see [Requesting a quota increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html). Verify your tool versions: ``` terraform --version aws --version kubectl version --client jq --version ``` This walkthrough uses the Terraform code in the [sample-eks-docs](https://github.com/aws-samples/sample-eks-docs) AWS Samples GitHub repository. Clone the repository into a working directory: ``` git clone https://github.com/aws-samples/sample-eks-docs.git cd sample-eks-docs/ai-ml/set-up-cluster ``` The repository has the following structure under the `ai-ml/set-up-cluster/` directory you changed into: ``` set-up-cluster/ ├── scripts/ │ └── cleanup.sh └── terraform/ ├── README.md ├── auto-mode/ │ ├── helm-values/ │ └── nodepools/ └── karpenter/ ├── helm-values/ └── nodepools/ ``` ## Understand the Terraform code The `terraform/auto-mode/` and `terraform/karpenter/` directories are each a self-contained root module. They create the same infrastructure, but differ in how much of it they manage explicitly: EKS Auto Mode provides node auto-scaling, the ALB controller, and GPU drivers as managed capabilities, whereas the Karpenter variant installs those components itself. Each directory contains the following files. Review them before you apply, so you know what the module creates in your account. | File | What it creates | | --- | --- | | `main.tf`, `versions.tf`, `variables.tf`, `outputs.tf` | Provider configuration, input variables, and the outputs printed after an apply. | | `vpc.tf` | The VPC, subnets, and subnet tags the ALB and EKS require. | | `eks.tf` | The EKS cluster, its add-ons, and (on the Karpenter path) the `system` managed node group. | | `monitoring.tf` | The Amazon Managed Service for Prometheus workspace, the kube-prometheus-stack Helm release, the NVIDIA DCGM Exporter, and the EKS Pod Identity associations they use. Chart values live in `helm-values/`. | | `s3.tf` | The model weights S3 bucket, the `model-storage-sa` ServiceAccount, and its Pod Identity association. | | `storage-class.tf` | The default `gp3` `StorageClass`. | | `nodepools.tf` | GPU NodePools, selected by `var.nodepools`. On the Karpenter path, also the always-on `general-purpose` NodePool for non-GPU workloads. Manifests live in `nodepools/`. | | `karpenter.tf`, `nvidia-device-plugin.tf`, `alb-controller.tf`, `fsx-subnet.tf` | Karpenter path only. EKS Auto Mode covers this functionality natively, so it has `alb-ingress-class.tf` in place of `alb-controller.tf` and no Karpenter or device plugin installs. | | `efa-device-plugin.tf`, `mpi-operator.tf`, `fsx-lustre.tf` | Optional add-ons for distributed training, disabled by default. See the following table. | ### What a default apply creates Several components are controlled by input variables. With no `-var` overrides, `terraform apply` creates the cluster, the monitoring stack, and the S3 bucket, and does not create any GPU capacity: | Variable | Default | Effect | | --- | --- | --- | | `nodepools` | `{}` | No GPU NodePools, so no GPU nodes and no GPU charges. You enable a strategy in [Create a GPU NodePool for AI/ML inference](ml-inference-gpu-nodepool.md). | | `enable_amazon_prometheus` | `true` | Creates the AMP workspace and wires Prometheus remote-write to it. | | `enable_dcgm_exporter` | `true` | Installs the NVIDIA DCGM Exporter. Its Pods stay at zero until GPU nodes join the cluster. | | `enable_efa` | `false` | Skips the EFA Kubernetes device plugin and the MPI Operator. | | `enable_fsx` | `false` | Skips the FSx for Lustre file system, CSI driver, and static `PersistentVolume`. | | `my_cidr` | `0.0.0.0/0` | Source CIDR allowed to reach the Grafana ALB. Override this, as described in the following warning. | Other variables set versions and names, including `region` (`us-east-2`), `cluster_name` (`ai-eks-docs`), and `kubernetes_version`. For the full list with descriptions, see `variables.tf` in your chosen directory. For usage examples beyond this guide, see the [README](https://github.com/aws-samples/sample-eks-docs/blob/main/ai-ml/set-up-cluster/terraform/README.md) on GitHub in the `terraform/` directory. ## Step 1: Deploy Terraform to create the EKS cluster, load balancing, monitoring, and S3 bucket A single `terraform apply` creates all of the infrastructure for this section: the EKS cluster, load balancing, the monitoring stack, and the Amazon S3 bucket for model weights. The repository provides two deployment paths. **Choose only one and use it throughout the guide**. + **EKS Auto Mode** (`terraform/auto-mode/`) — In addition to the core [networking, storage, and load balancing add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html#addon-consider-auto), EKS Auto Mode includes and manages the following capabilities for training and inference workloads: EKS node monitoring agent, automatic node repair, [SOCI](https://github.com/awslabs/soci-snapshotter) snapshotter for fast container pulls, and GPU readiness for the default NodeClass. The NVIDIA device plugin is included in the Bottlerocket accelerated AMI that EKS Auto Mode uses for GPU-enabled nodes. + **Self-managed Karpenter** (`terraform/karpenter/`) — On an EKS cluster without EKS Auto Mode, the Terraform code installs and configures the components required for training and inference workloads. This includes networking add-ons (VPC CNI, CoreDNS, kube-proxy), Karpenter, the EKS node monitoring agent, the NVIDIA device plugin, and SOCI snapshotter for fast container pulls. ![Side-by-side comparison of the two cluster options: an EKS Auto Mode cluster with a NodePool, and an EKS standard cluster with self-managed Karpenter, CoreDNS, VPC CNI, NVIDIA device plugin, EKS Pod Identity agent, Node Monitoring Agent, kube-proxy, and a NodeClass and NodePool](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-cli-cluster-options.png) **Grafana is publicly accessible over HTTP with default credentials** The Grafana ALB `Ingress` defaults `var.my_cidr` to `0.0.0.0/0`, which exposes Grafana to the public internet over plain HTTP with default admin credentials. Automated scanners discover public load balancers within minutes. You **must** restrict access by overriding `var.my_cidr` with your own IP address: ``` export MY_CIDR="$(curl -s https://checkip.amazonaws.com)/32" terraform apply -var "my_cidr=${MY_CIDR}" ``` Treat source-IP allowlisting as a minimum safeguard, not a complete one. Also change the default Grafana admin password after first login. For a stronger posture, change the `alb.ingress.kubernetes.io/scheme` to `internal` (reachable only from within your VPC or a connected VPN) and add a TLS certificate. Change into the directory for your chosen path, initialize Terraform, and apply: To deploy in a different Region, add `-var "region={{region-code}}"` to the `terraform apply` command that follows, where {{region-code}} is the AWS Region you want to deploy in. The Terraform code uses all available Availability Zones in the target Region, excluding `use1-az3`, `usw1-az2`, and `cac1-az3` because [Amazon EKS does not support control plane placement in those zones](https://repost.aws/knowledge-center/eks-cluster-creation-errors). **Important** Run `terraform apply` directly, as shown in the following steps. Do not save a plan with `terraform plan -out` and apply the saved plan file. The Kubernetes, Helm, and kubectl providers authenticate with a token from the `aws_eks_cluster_auth` data source, and a saved plan freezes that token at plan time. By the time you apply the plan, the token has expired, and every Helm release and Kubernetes object fails with `Unauthorized` or `the server has asked for the client to provide credentials` after the cluster itself is already created. ------ #### [ EKS Auto Mode ] ``` cd terraform/auto-mode terraform init terraform apply ``` This command takes a few minutes to complete. ------ #### [ Self-managed Karpenter ] ``` cd terraform/karpenter terraform init terraform apply ``` This command takes about 30 minutes. It creates an EKS cluster with a managed node group dedicated to hosting add-ons and the Karpenter controller. Terraform installs Karpenter with the Spot interruption queue enabled and the `NodeRepair` and `StaticCapacity` feature gates turned on. It also installs the NVIDIA device plugin, the AWS Load Balancer Controller, and the monitoring stack. ------ ### Review the Terraform outputs When the apply completes, Terraform prints the following outputs (values vary based on your configuration): ``` Apply complete! Resources: 74 added, 0 changed, 0 destroyed. Outputs: cluster_name = "ai-eks-docs" configure_kubectl = "aws eks update-kubeconfig --region us-east-2 --name ai-eks-docs --alias ai-eks-docs" configure_model_bucket = "export MODEL_BUCKET=ai-eks-docs-models-20260612084512345600000004" model_bucket = "ai-eks-docs-models-20260612084512345600000004" node_iam_role_name = "ai-eks-docs-eks-auto-20260612..." region = "us-east-2" ``` The `configure_kubectl` output is a ready-to-run command that points `kubectl` at the cluster. The `model_bucket` output contains the S3 bucket name for model weights. The `node_iam_role_name` output shows the IAM role that nodes use. The preceding resource count is for the EKS Auto Mode path. The self-managed Karpenter path creates roughly 115 resources, because it provisions Karpenter, its interruption queue and event rules, the `system` managed node group, the AWS Load Balancer Controller, and the NVIDIA device plugin itself. The count also differs in Regions with a different number of Availability Zones, because the code creates a subnet, a route table, and their associations per zone. ### Configure kubectl Point `kubectl` at the new cluster. The `configure_kubectl` output is a ready-to-run command: ``` eval "$(terraform output -raw configure_kubectl)" ``` ### Verify the cluster ------ #### [ EKS Auto Mode ] ``` kubectl get pods --all-namespaces ``` Expected output: ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system metrics-server-5db89f9ffd-h4mlr 1/1 Running 0 3m kube-system metrics-server-5db89f9ffd-nd748 1/1 Running 0 3m monitoring kube-prometheus-stack-grafana-ff9b5fd57-dh562 3/3 Running 0 3m monitoring kube-prometheus-stack-kube-state-metrics-5dcbfdf69b-wd6qn 1/1 Running 0 3m monitoring kube-prometheus-stack-operator-548c4f4485-m5wh2 1/1 Running 0 3m monitoring kube-prometheus-stack-prometheus-node-exporter-p2z7l 1/1 Running 0 3m monitoring prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 3m ``` In EKS Auto Mode, the VPC CNI, kube-proxy, and CoreDNS run as managed components and do not appear as pods in `kube-system`. ------ #### [ Self-managed Karpenter ] ``` kubectl get pods --all-namespaces ``` Expected output includes Karpenter, CoreDNS, kube-proxy, aws-node (VPC CNI), the Amazon EBS CSI driver, the EKS Pod Identity Agent, the EKS node monitoring agent, the AWS Load Balancer Controller, and the NVIDIA device plugin: ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-load-balancer-controller-56c454df76-bbt9h 1/1 Running 0 5m kube-system aws-load-balancer-controller-56c454df76-drdp6 1/1 Running 0 5m kube-system aws-node-bzdcz 2/2 Running 0 5m kube-system aws-node-vkbhb 2/2 Running 0 5m kube-system coredns-7dbb8998cf-9b9wk 1/1 Running 0 5m kube-system coredns-7dbb8998cf-pwtjd 1/1 Running 0 5m kube-system ebs-csi-controller-748f54b69-8h7jv 6/6 Running 0 5m kube-system ebs-csi-controller-748f54b69-v2mv8 6/6 Running 0 5m kube-system ebs-csi-node-ks8vq 3/3 Running 0 5m kube-system ebs-csi-node-ps9g8 3/3 Running 0 5m kube-system eks-node-monitoring-agent-5qw4l 1/1 Running 0 5m kube-system eks-node-monitoring-agent-7lrtm 1/1 Running 0 5m kube-system eks-pod-identity-agent-ddvlv 1/1 Running 0 5m kube-system eks-pod-identity-agent-q4g29 1/1 Running 0 5m kube-system karpenter-898ff78-cndbd 1/1 Running 0 5m kube-system karpenter-898ff78-hfbhn 1/1 Running 0 5m kube-system kube-proxy-gcfnh 1/1 Running 0 5m kube-system kube-proxy-tktcf 1/1 Running 0 5m kube-system metrics-server-5b789db597-cm9qd 1/1 Running 0 5m kube-system metrics-server-5b789db597-w57b6 1/1 Running 0 5m kube-system nvidia-device-plugin-node-feature-discovery-gc-66cb7f5dc-xvftc 1/1 Running 0 5m kube-system nvidia-device-plugin-node-feature-discovery-master-854d6b5hnqtp 1/1 Running 0 5m monitoring kube-prometheus-stack-grafana-6797bcb59f-2zlwg 3/3 Running 0 5m monitoring kube-prometheus-stack-kube-state-metrics-8446bd549c-q6gjc 1/1 Running 0 5m monitoring kube-prometheus-stack-operator-5f499b784-vf5xb 1/1 Running 0 5m monitoring kube-prometheus-stack-prometheus-node-exporter-4c6wq 1/1 Running 0 5m monitoring kube-prometheus-stack-prometheus-node-exporter-l5t25 1/1 Running 0 5m monitoring prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 5m ``` Verify the NVIDIA device plugin is installed. Zero device plugin Pods appear until a GPU NodePool with the matching `amiFamily=al2023` label is provisioned: ``` kubectl get daemonset nvidia-device-plugin -n kube-system ``` Expected output: ``` NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nvidia-device-plugin 0 0 0 0 0 amiFamily=al2023 5m ``` ------ ## Step 2: Verify the monitoring stack Terraform already provisioned the full monitoring stack during the `terraform apply` in [Step 1](#cluster-setup-tf-deploy). The stack includes an Amazon Managed Service for Prometheus (AMP) workspace, IAM policies and EKS Pod Identity Associations for Prometheus remote-write and Grafana query access, the kube-prometheus-stack Helm chart (Prometheus, Grafana, kube-state-metrics, node-exporter), and the NVIDIA DCGM Exporter for GPU metrics. This section covers verification of the deployed monitoring components. ### Verify monitoring pods Wait for all monitoring pods to be ready: ``` kubectl wait --for=condition=Ready pod --all -n monitoring --timeout=300s kubectl get pods -n monitoring ``` Expected output: ``` NAME READY STATUS RESTARTS AGE kube-prometheus-stack-grafana-7c58f54f77-rftrj 3/3 Running 0 5m kube-prometheus-stack-kube-state-metrics-d68dcbc84-5smxq 1/1 Running 0 5m kube-prometheus-stack-operator-5895df479f-ttm47 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-t9q7s 1/1 Running 0 5m kube-prometheus-stack-prometheus-node-exporter-x6vfb 1/1 Running 0 5m prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 5m ``` ### Access Grafana Grafana is exposed through an internet-facing AWS Application Load Balancer (ALB), restricted to the CIDR you set in `var.my_cidr`. Print the load balancer URL (allow a minute or two for the ALB to provision): ``` echo "http://$(kubectl get ingress kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')" ``` Open the URL in your browser. Log in with username `admin` and the password from the following command: ``` kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo ``` ### Verify the metrics pipeline To verify the metrics pipeline is working end to end: 1. Navigate to **Connections > Data sources** and confirm **Amazon-Managed-Prometheus** is listed as the default datasource. **Validate the AMP datasource in Grafana** ![Grafana Connections page showing Amazon-Managed-Prometheus listed as the default data source](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-cli-prometheus-ds-validate.png) 1. Navigate to **Drilldown > Metrics** and search for the `up` metric. You should see results from your cluster’s scrape targets. **Validate the `up` metric in Grafana** ![Grafana Drilldown Metrics page showing the up metric with green status bars indicating active scrape targets](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-cli-prometheus-metrics-validate.png) If `up` shows results, the pipeline (cluster → Prometheus → AMP → Grafana) is working. ### Validate DCGM GPU metrics The DCGM Exporter DaemonSet runs on GPU nodes and reports GPU utilization, memory, temperature, power draw, NVLink bandwidth, and tensor activity metrics. Verify the DCGM exporter DaemonSet: ``` kubectl get daemonset dcgm-exporter -n monitoring ``` After a GPU node starts running (which you trigger by creating the GPU NodePool in the [Create a GPU NodePool for AI/ML inference](ml-inference-gpu-nodepool.md) section), you should see one or more ready Pods. To validate DCGM metrics, navigate to **Drilldown > Metrics** in Grafana and search for `DCGM_`. The search returns the GPU metrics the exporter collects, including `DCGM_FI_DEV_ECC_SBE_VOL_TOTAL` for single-bit ECC errors, `DCGM_FI_DEV_ENC_UTIL` for encoder utilization, and `DCGM_FI_DEV_FB_FREE` and `DCGM_FI_DEV_FB_USED` for free and used framebuffer memory. Any results mean Prometheus is scraping the exporter and writing to Amazon Managed Service for Prometheus. **Validate DCGM metrics in Grafana** ![Grafana Drilldown Metrics page filtered by DCGM_ showing GPU metrics including DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, DCGM_FI_DEV_ENC_UTIL, DCGM_FI_DEV_FB_FREE, and DCGM_FI_DEV_FB_USED](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-cli-dcgm-metrics-validate.png) To view the pre-provisioned GPU dashboard, navigate to **Dashboards > GPU Monitoring > NVIDIA DCGM Exporter Dashboard**. The dashboard has panels for GPU Utilization, GPU Avg Temp, GPU Framebuffer Mem Used, and GPU Power Total. Non-zero values in those panels confirm the full path is working, from the exporter on the GPU node through Prometheus to Amazon Managed Service for Prometheus. An idle GPU node reports 0% utilization and 0 MiB framebuffer memory used, but still reports a temperature and power draw. **NVIDIA DCGM Exporter Dashboard in Grafana** ![Grafana NVIDIA DCGM Exporter Dashboard showing GPU Utilization, GPU Avg Temp, GPU Framebuffer Mem Used, and GPU Power Total panels](https://docs.aws.amazon.com/eks/latest/userguide/images/ml-cluster-setup-cli-dcgm-dashboard.png) ## Step 3: Verify the model weights S3 bucket Terraform already created an Amazon S3 bucket for storing model weights, a `model-storage-sa` ServiceAccount in the `default` namespace, an IAM policy scoped to the bucket, and an EKS Pod Identity Association that links them. Workload Pods that set `serviceAccountName: model-storage-sa` can read from and write to the bucket. ### Verify the bucket Retrieve the bucket name from Terraform outputs: ``` MODEL_BUCKET=$(terraform output -raw model_bucket) echo ${MODEL_BUCKET} ``` Verify the bucket exists: ``` aws s3api head-bucket --bucket ${MODEL_BUCKET} ``` Expected output: ``` { "BucketArn": "arn:aws:s3:::ai-eks-docs-models-20260612084512345600000004", "BucketRegion": "us-east-2", "AccessPointAlias": false } ``` #### Validate S3 access from a Pod Run a one-off Pod with the AWS CLI image, using the `model-storage-sa` ServiceAccount, to confirm EKS Pod Identity is wired up and S3 access works: ``` cat << EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: s3-test labels: guide: ai-eks-docs spec: serviceAccountName: model-storage-sa containers: - name: aws-cli image: public.ecr.aws/aws-cli/aws-cli:2.27.0 command: - sh - -c - | echo "=== Caller Identity ===" aws sts get-caller-identity echo "" echo "=== S3 Write Test ===" echo "pod identity works" | aws s3 cp - s3://${MODEL_BUCKET}/test.txt echo "" echo "=== S3 List Test ===" aws s3 ls s3://${MODEL_BUCKET}/ echo "" echo "=== S3 Delete Test ===" aws s3 rm s3://${MODEL_BUCKET}/test.txt restartPolicy: Never EOF ``` Wait for the Pod to complete and check the logs: ``` kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/s3-test --timeout=300s kubectl logs s3-test ``` **Note** On the self-managed Karpenter path, this Pod usually waits for a new node. The `system` managed node group is tainted for cluster-critical add-ons, so the Pod stays `Pending` until Karpenter launches a node from the `general-purpose` NodePool, which takes two to three minutes. If `kubectl wait` times out, check `kubectl describe pod s3-test` for a `Nominated` event and run the `wait` command again. Expected output: ``` === Caller Identity === { "UserId": "AROA...:eks-ai-eks-doc-s3-test-...", "Account": "123456789012", "Arn": "arn:aws:sts::123456789012:assumed-role/ai-eks-docs-models-.../eks-ai-eks-doc-s3-test-..." } === S3 Write Test === === S3 List Test === 2026-07-15 12:00:00 19 test.txt === S3 Delete Test === delete: s3://ai-eks-docs-models-20260612084512345600000004/test.txt ``` The caller identity confirms the Pod assumed the model storage role through EKS Pod Identity. The S3 commands confirm read and write access. The write test prints no output of its own, because the AWS CLI suppresses its `upload:` progress line when the source is standard input; the object listed by the next command is the proof that the write succeeded. Clean up the test Pod: ``` kubectl delete pod s3-test ``` ## Next steps With your cluster ready, continue to [Create a GPU NodePool for AI/ML inference](ml-inference-gpu-nodepool.md) to create a GPU NodePool that dynamically provisions GPU-enabled nodes for your workloads. ## Cleanup **Tip** If you plan to continue with the next sections of this guide, skip the full cleanup. Only run it when you are done. ### Destroy the cluster and all Terraform-managed resources Delete the test workloads from the GPU NodePool walkthrough first, so no Pod holds a GPU node while the nodes drain: ``` kubectl delete pod nvidia-smi --ignore-not-found kubectl delete deployment gpu-overflow-test --ignore-not-found ``` Drain the Karpenter-managed nodes before destroying, so no in-flight node lifecycle blocks the destroy. Delete any PodDisruptionBudgets that would prevent a drain, then delete the NodeClaims: ``` kubectl delete pdb -A --all --ignore-not-found kubectl delete nodeclaim --all --wait=true --timeout=900s ``` Next, delete the Grafana `Ingress` and wait for its load balancer to disappear. The AWS Load Balancer Controller creates that load balancer and its security groups in response to the `Ingress`, so Terraform does not track them. If they still exist when Terraform reaches the VPC, their network interfaces hold the public subnets, which stalls the destroy on the subnets, internet gateway, and VPC: ``` kubectl delete ingress kube-prometheus-stack-grafana -n monitoring --ignore-not-found aws elbv2 describe-load-balancers \ --region [.replaceable]`region-code` \ --query "LoadBalancers[?LoadBalancerName=='grafana-ai-eks-docs'].LoadBalancerArn" \ --output text ``` The load balancer name is `grafana-ai-eks-docs` even if you overrode `cluster_name`. The `alb.ingress.kubernetes.io/load-balancer-name` annotation in `helm-values/kube-prometheus.yml` sets it to that literal value, and Terraform does not template it from `cluster_name`. If you changed that annotation, query for the name you set instead. Wait until the second command returns an empty result, then destroy everything Terraform created, including the EKS cluster, the VPC, the monitoring stack, the NodePools and NodeClasses, the S3 model bucket, and any ODCR: ``` terraform destroy ``` **Note** A destroy can report `Error: Error uninstalling release …​ context deadline exceeded` for the `kube-prometheus-stack` or `karpenter-crd` Helm release and leave the cluster and VPC in place. Run `terraform destroy` again with the same `-var` overrides; the second run picks up where the first stopped. **Important** Pass `terraform destroy` the same `-var` overrides you passed to `terraform apply`, including `region` and `nodepools`. Variables are not persisted between commands, so a `destroy` that omits them falls back to the defaults, targets a different Region than the one you deployed to, and cannot find your resources. **Warning** The model weights S3 bucket is created with `force_destroy = true`, so `terraform destroy` deletes the bucket along with any model weights you uploaded to it. Copy anything you want to keep to another location first. **Note** The repository also ships a `scripts/cleanup.sh` helper that deletes the PodDisruptionBudgets, NodeClaims, NodePools, and NodeClasses and then runs `terraform destroy`. Run it from inside the `terraform//` directory you applied from, and pass `--auto-approve` to skip the Terraform confirmation prompt. Read the script first, and if it does not delete the `Ingress` resources, delete the Grafana `Ingress` yourself as shown previously. If the destroy stalls on `module.vpc.aws_vpc.this[0]` for more than a few minutes, a security group that the AWS Load Balancer Controller created is still in the VPC. The controller tags every security group it creates with `elbv2.k8s.aws/cluster`, so list only those, and then delete them: ``` aws ec2 describe-security-groups \ --region [.replaceable]`region-code` \ --filters "Name=tag:elbv2.k8s.aws/cluster,Values=[.replaceable]`cluster-name`" \ --query 'SecurityGroups[].[GroupId,GroupName]' \ --output text aws ec2 delete-security-group --region [.replaceable]`region-code` --group-id [.replaceable]`sg-id` ``` The groups this returns are named `k8s-traffic-{{cluster-name}}- ` and `k8s-{{namespace-prefix}}-{{ingress-prefix}}-`. Delete only these. The cluster and node security groups in the same VPC belong to Terraform, which deletes them itself once nothing holds them. ### Verify the reservation is gone Confirm no active Capacity Reservation remains for the cluster: ``` aws ec2 describe-capacity-reservations \ --filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand" \ --query 'CapacityReservations[].CapacityReservationId' \ --output text \ --region [.replaceable]`region-code` ``` An empty result means no reservation is active and no further charges apply. Run the command against the Region you deployed to, because `terraform destroy` has already removed the `region` output by this point. **Important** If you ever interrupted a `terraform apply` while it was creating the ODCR, Terraform never recorded the reservation in its state and `terraform destroy` cannot remove it. Drop the `tag:nodepool` filter to list every active reservation in the Region, then cancel any that belong to this walkthrough with `aws ec2 cancel-capacity-reservation --capacity-reservation-id {{cr-1234567890abcdef0}} `. ### Verify no cluster resources remain Confirm the cluster, its nodes, its VPC, and the model bucket are gone. Run each command against the Region you deployed to, replacing {{cluster-name}} with the value of `cluster_name` you deployed with, which defaults to `ai-eks-docs`: ``` aws eks list-clusters --region [.replaceable]`region-code` aws ec2 describe-instances \ --region [.replaceable]`region-code` \ --filters "Name=tag:PartOf,Values=[.replaceable]`cluster-name`" \ "Name=instance-state-name,Values=pending,running,stopping,stopped" \ --query 'Reservations[].Instances[].InstanceId' \ --output text aws ec2 describe-vpcs \ --region [.replaceable]`region-code` \ --filters "Name=tag:PartOf,Values=[.replaceable]`cluster-name`" \ --query 'Vpcs[].VpcId' \ --output text aws s3api list-buckets \ --query "Buckets[?starts_with(Name, '[.replaceable]`cluster-name`-models-')].Name" \ --output text ``` Each command should return an empty result, other than any unrelated clusters in your account. Terraform tags every resource it creates with `PartOf` set to the cluster name, so these filters find anything left behind. **Important** If you overrode `cluster_name` and these commands still use the default, they return an empty result even when resources remain, which looks like a successful cleanup while the resources keep billing. Confirm the name you deployed with before you rely on the output.