Develop TensorRT LLM on Runpod#

Runpod is a popular cloud platform among many researchers. This doc describes how to develop TensorRT LLM on Runpod.

Prepare#

Create a Runpod account#

Please refer to the Runpod Getting Started.

Configure SSH Key#

Note that we can skip the step of “Start your Pod. Make sure of the following things” here as we will introduce it below.

Build the TensorRT LLM Docker Image and Upload to DockerHub#

Please refer to the Build Image to DockerHub.

Note that the docker image must enable ssh access. See on Enable ssh access to the container.

Create a Pod Template#

Click the “Template” button in the menu, then click “Create Template”.

Fill the docker image link of DockerHub such as docker.io/<your_dockerhub_username>/tensorrt_llm:devel on “Docker Image” field.

Fill “22” into “Expose TCP Ports” field.

Fill

sleep infinity

into ‘Container Start Command’ field.

Connect to the Pod#

Please refer to the Connect to the Pod.

You can connect the pod with SSH or Web Terminal.

If you want to connect the pod with SSH, you can copy the command from “SSH over exposed TCP” field and run it on your host.

In some scenarios such as using a team account, your public key has not been added to the pod successfully. You can directly add this command to the ‘Container Start Command’ field as:

bash -c 'echo "<your_public_key>" >> ~/.ssh/authorized_keys;sleep infinity'

Enjoy your development!