Skip to content

Cloud TPU Setup

This guide provides information on setting up and provisioning Google Cloud TPUs for use with tpu-inference.

TPU Generations

Tensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs are available in different versions each with different hardware specifications. For more information about TPUs, see TPU System Architecture.

The following TPU versions are compatible with tpu-inference. Selecting a topology allows you to configure the physical arrangements of the TPU chips, improving throughput and networking performance.

Recommended

Experimental

Quota and Pricing

In order for you to use Cloud TPUs you need to have TPU quota granted to your Google Cloud project. For more information, see TPU quota.

For TPU pricing information, see Cloud TPU pricing.

Provisioning Cloud TPUs

Google Cloud supports two primary APIs for provisioning TPUs:

  • Compute Engine API (gcloud compute instances create): Recommended for modern TPU generations starting with TPU v6e (Trillium) and TPU v5p.
  • Cloud TPU API (gcloud alpha compute tpus queued-resources create): Legacy API used for earlier generations like TPU v5e. Note that the Cloud TPU API is no longer under active development.

TPU v7x (Ironwood)

TPU v7x (Ironwood) is in preview status and is provisioned via Google Kubernetes Engine (GKE) rather than standalone on-demand VM commands. Note that TPU v7x does not support Flex-start (DWS). For v7x provisioning and cluster orchestration, see About TPUs in GKE and the official TPU v7x documentation.

You can also choose between two capacity models:

  • Standard (On-Demand): Immediate allocation at standard pay-as-you-go rates.
  • Flex-start (DWS): Discounted capacity via Dynamic Workload Scheduler (DWS) that runs uninterrupted for up to 7 days. Note that Flex-start is supported on TPU v5e, TPU v5p, and TPU v6e (TPU v7x, as well as older generations like v3 and v4, do not support Flex-start). For more details, see About Flex-start VMs and DWS Pricing.

Select your desired TPU hardware, number of chips, and capacity model to generate the exact provisioning command. Be sure to replace placeholder variables (like PROJECT_ID and SERVICE_ACCOUNT) with your own values before running.

Hardware
Chips
Model
Parameter Description
PROJECT_ID Your Google Cloud project ID.
ZONE The Google Cloud zone where you have TPU quota (e.g., us-east5-a, europe-west4-a, us-central2-b). See TPU regions and zones for availability.
SERVICE_ACCOUNT The email address for your service account, found in the Cloud Console under IAM Service Accounts (e.g., tpu-service-account@<your_project_ID>.iam.gserviceaccount.com). Required for legacy Cloud TPU API calls.
RUNTIME_VERSION Automatically populated by the generator above based on your selected TPU hardware generation.

Connecting and Checking Status

Connect to your TPU VM using SSH:

  • For GCE VM Instances (TPU v6e, v5p):

    gcloud compute ssh my-tpu-vm --zone ZONE

  • For Legacy TPU VMs (TPU v5e, v4, v3):

    gcloud compute tpus tpu-vm ssh my-tpu-name --project PROJECT_ID --zone ZONE

Check Provisioning Status:

To check whether your TPU VM or Flex-start request has been allocated and is running:

  • For GCE VM Instances (TPU v6e, v5p):

    gcloud compute instances describe my-tpu-vm --zone ZONE

  • For Queued Resources (TPU v5e, v4, v3):

    gcloud alpha compute tpus queued-resources describe my-queued-resource --zone ZONE