メインコンテンツへスキップ
見出し画像

Home Workstation for Local LLMs: The Journey from Tesla P40 to Tesla V100



    Introduction

    I have successfully deployed a home workstation dedicated to running local LLMs. The heart of this system—the GPUs—started with four Tesla P40s (totaling 96GB VRAM) and has recently been upgraded to four Tesla V100 32GB PCIe cards (totaling 128GB VRAM). I would like to record the journey of this transition from the Tesla P40 to the Tesla V100.

    Current Evaluation of Tesla P40 and Tesla V100

    Writing this in May 2026, the cutting edge of consumer video cards is the NVIDIA RTX 5090. The Tesla P40 and V100 are already ancient by these standards, dating back nearly a decade. Why write about this now? Because I feel that, at this point in time, these cards are no longer highly regarded.

    However, in my experience, they are still remarkably capable.

    I recently collected and integrated the Tesla V100 (32GB PCIe version) into my system, replacing the P40s. While they cannot handle the “latest and greatest” features, I don’t use this machine for gaming, image generation, or video production. My sole purpose is LLM inference, and for that specific use case, they are incredibly useful. I am writing this to leave a record for those who seek massive VRAM for inference on a budget; for them, the Tesla P40 and V100 remain viable options.

    I don’t know how many people will read this, but I want to document the specifics of my system.

    My System

    Purpose of the Workstation

    The goal is to run LLMs locally at home. Specifically, I have targeted the Gemma 4:31B-it-Q8_0 model because of its proficiency in Japanese. After testing various models, I found that this 31B parameter count produces almost no errors in Japanese. With other models, I occasionally encountered unfamiliar conjunctions; searching Google revealed they were actually Chinese conjunctions.

    When I mentioned this to Google’s AI mode as a joke, it pointed out that models not specifically trained in Japanese may treat Asian languages as a single cluster, leading to a mix of Chinese, Korean, and Japanese. This may be the case. Regardless, I wanted to run Gemma 4:31B locally, which led to the creation of this workstation.

    Software for LLM Execution

    I am running LLMs via Ollama on Ubuntu 22.04. Using Ollama has allowed these older GPUs to function without issues. On Reddit and other forums, the common consensus is that the Tesla P40 is too old and the Tesla V100 is not worth buying now. From a general-purpose perspective, that may be true. However, I already own multiple RTX 3090s and RTX 4060 Ti 16GB cards for image and video generation. My current focus is strictly LLMs. I want as much VRAM as possible on a low budget, and I am interested in inference, not training. My goal is to feed large amounts of text into the LLM and have it reason through the data. For this, the Tesla P40 and V100 are more than sufficient.

    Another major factor was the hardware I already had on hand. I possessed motherboards, memory, and CPUs from the same era as the Tesla P40/V100—specifically, dual Xeon server setups. In May 2026, memory prices have skyrocketed, and I cannot afford new DDR5. Fortunately, I had a significant amount of used DDR3 server ECC memory (192GB). I wanted to utilize this. I reasoned that since the Tesla P40 and V100 were the pinnacle of their generation, they would be most compatible and easiest to manage on a motherboard and CPU from that same era.

    Non-GPU Components

    Motherboards

    • ASUS X9PE-D8 WS (Failure)
      Initially, I tried using the ASUS X9PE-D8 WS because I already owned it. However, I couldn’t get the Tesla P40s to be recognized correctly. Specifically, the system failed to POST only when the Tesla P40 was installed. Although the BIOS had an “Above 4G Decoding” option, enabling it caused the motherboard to stop functioning entirely. Not knowing the exact cause, I abandoned this board and moved to another. In hindsight, downgrading the BIOS might have worked, as the next motherboard I used also failed to POST initially but worked after a BIOS downgrade.

    • Supermicro X9DRG-QF (Current)
      Since I planned to run four GPUs, I purchased a used Supermicro X9DRG-QF, which has a proven track record as a GPU server. It features four PCIe x16 slots capable of supporting dual-slot GPUs (like the P40 and V100), plus three additional PCIe slots. I use those additional slots for NVMe SSDs for the boot drive and model storage. Note for others: This generation does not support direct booting from NVMe. I achieve this using Clover on a USB flash drive. This motherboard has an onboard display output, which is very useful for monitoring Ollama and running nvidia-smi or ollama ps without needing a separate graphics card.

    CPU

    I am using dual Xeon E5-2650L v2 processors. During the pandemic, when generative AI first became popular, I stockpiled used workstation CPUs and memory. I started with dual Xeon E5 v1 and v2, eventually setting up two dual v4 systems on ASUS X10PE-D16 WS boards for video generation and daily work.

    Because of my familiarity and affection for ASUS boards, I hoped the X9PE-D8 would work, especially since the board and DDR3 memory were sitting idle. I felt it would be a waste not to use them.

    Furthermore, the dual Xeon E5 configuration provides an abundance of PCIe lanes, easily supporting four x16 slots, making it ideal for multi-GPU setups.

    The primary concern was power consumption. Dual Xeons from this era can generate enough heat to warm a room in winter. To mitigate this, I chose the low-voltage E5-2650L v2 (70W x 2 = 140W). Since the LLM runs on the GPU, the CPU doesn’t need to be exceptionally fast. I also intend to apply power limits to the GPUs. Even with these limits, the system gets extremely hot during inference.

    Memory

    I utilized 192GB of surplus DDR3 server memory (ECC RDIMM). I bought these in bulk during the pandemic, and ironically, they are now more expensive than when I first purchased them. I am glad I bought them when I did.

    My core philosophy for server construction is to provide RAM in abundance. I believe that having more capacity, even if the generation is old, provides operational flexibility. I apply the same logic to VRAM. To me, RAM and VRAM capacity are like the “size of a workshop.” If the space is too small, there are many things you simply cannot do. I would rather have a large, slightly dilapidated barn than a tiny, modern studio, because the larger space allows for more possibilities. I chose the ability to do more over raw speed.

    Power Supply

    A single 1300W power supply handles everything. This is sufficient because I have set power limits of 140W (now 150W since switching to V100) for the four GPUs. With the dual Xeon E5-2650L (max 70W each), and factoring in the motherboard, case fans, and CPU fans, the total load remains well within the PSU’s capacity.

    Shroud (Airflow Guide)

    The biggest challenge was that Tesla P40 and V100 GPUs use passive cooling, meaning I had to design my own cooling solution. I built a custom shroud using cardboard and duct tape to channel air through all four GPUs. It is essentially a rectangular tube with 140mm Noctua fans installed in a “push-pull” configuration at both ends.

    While Noctua fans are quiet, I was concerned about static pressure. To address this, I used a dual-fan setup on both the push and pull sides (four fans total), leaving a small gap of about 3cm between the pairs. I consulted Google AI on this; while some on Reddit argue that vertical dual-fans are meaningless, I believe they help maintain the static pressure required to push air through the dense cooling fins of the Tesla cards. I can feel a steady stream of air exiting the shroud, so I am satisfied with the result.

    Since the motherboard is too large for a standard case, the system sits directly on my desk, and the cardboard shroud works perfectly in this open-air environment. I tried a blower fan, but the noise was unbearable; I don’t have a dedicated server room, and this machine sits on the edge of my dining table.

    To further combat heat during inference, I added two heatsinks on top of the four GPUs with an additional case fan blowing over them, which seems to help significantly.

    GPU Implementation

    The Tesla P40 Era

    I started with four Tesla P40s. I verified the system with one card first before scaling up to four, primarily to utilize the 96GB of VRAM. I took a gradual approach to avoid the panic of a total system failure—which almost happened when the X9PE-D8 failed to POST.

    Once the system was stable on the Supermicro board, the P40s performed well, though inference speed was slow—roughly 4 to 5 tokens/s for Gemma 4:31B-it-Q8_0.

    Why Q8_0 Works on Tesla P40

    Gemma 4:31B-it-Q8_0 requires about 32GB of VRAM for the model. However, to run it 100% on the GPU with a full context size of 256K, you need about 30GB additional VRAM, totaling over 62GB.

    Although the Tesla P40 is old, it supports INT8 calculations and can perform them efficiently, making it highly compatible with Q8_0. Thus, with three or more Tesla P40s (72GB+), you can run Gemma 4:31B-it-Q8_0 at full context.

    The speed is modest, but for conversational use, it matches my own reading speed. Even if the generation were instantaneous, the bottleneck would simply shift to how fast I can read the text.

    Purchasing Tesla P40s

    I primarily used eBay, mostly buying from Chinese vendors. Shipping was fast, and I never encountered import taxes. Since the P40 has few variations, I always received the correct card. However, I warn against sites like AliExpress, where “Tesla P40” might actually be a “Tesla P4” in the default selection. They look almost identical, and it’s easy to make a mistake if you only look at the price.

    The Path to Upgrade

    Direction of the Upgrade

    I knew I couldn’t stay with the P40s forever, but I had to decide on the direction of the upgrade. The obvious choice would be the RTX 5090, but that would require replacing the entire workstation, including the motherboard and DDR3/Xeon E5 v2 system. I also couldn’t justify the cost of multiple RTX 5090s.

    RTX 3090s were an option, but their used prices had risen significantly. This led me to a more cost-effective path for increasing VRAM: transitioning from the Tesla P40 to the Tesla V100.

    Purchasing Tesla V100 (32GB)

    I gradually purchased Tesla V100 (32GB) cards over time to avoid hitting my credit card limits.

    With the V100, the color scheme is distinct, so identification is easier. However, the real challenge is that non-PCIe versions (SXM2, SXM3) exist. These are cheaper and widely available, but since I was using the Supermicro PCIe board, I avoided them entirely.

    I used eBay again. According to Reddit, eBay has small-scale sellers and occasional “bulk sellers” who dump inventory from data centers. I waited for a bulk seller to appear and managed to snag two sets. I wish I could have bought all four at once, but I couldn’t wait indefinitely.

    To save on shipping and taxes, I used a forwarding service called Planet Express in Oregon (a tax-free state). The US seller shipped for free within the US, and I only paid the shipping from Oregon to Japan.

    Interestingly, I later discovered that one of my four V100s was a modified SXM2 card disguised as a PCIe version. I found this out via nvidia-smi. This specific card has a higher idle power draw (about 50W compared to 30W for the others), which is my only grievance.

    Transitioning to Tesla V100 (32GB) x4

    Once I had all four cards, the replacement was simple: remove the cardboard shroud, swap the P40s for V100s, and boot. It worked immediately.

    One detail regarding power limits: while the P40s were fine at 140W, only one of my V100s (likely an original PCIe version) accepted a 140W limit. The other three rejected it as “out of range.” However, they all accepted a 150W limit, so I standardized all four at 150W. I kept the NVIDIA drivers and CUDA settings from the P40 era to avoid stability issues.

    With this, my local LLM system with 128GB of VRAM (32GB x 4) was complete.


    (Addendum: 2026.06.08)
    Updated NVIDIA drivers from version 535 to 570 during the Ollama 0.30 series update.

    (Addendum: 2026.07.09)
    Had a detailed conversation with Gemma 4:31B-it-Q8_0 about the advantages of using the Tesla V100 for LLM inference.

    (Addendum: 2026.09.06)
    After verifying BF16 compatibility on the Tesla V100, I have switched to Gemma 4:31B-it-FP16 as my main model. I can now run it at full context (256K) 100% on the GPU.

    (Addendum: 2026.10.02)
    Conducted an analysis on who would be satisfied with a Tesla V100 setup and how to maximize its utility. (English article created based on LLM translation).


    (To be continued)

    Original article in Japanese

     
     

    sai

     
     
    ❤️ 優先 ⚠️ 必ずしもフォロバはしません ⚠️ 自宅で勉強してます。その勉強にLLMを利用してます。でも勉強部屋のWSには、動画高画質化、動画立体視ソフトもあるので、直ぐ脇道にそれます。そして人類の叡知に向けて全力疾走してます。そんな自分に気づいたら勉強に戻ります。その繰り返しです

    あなたへのおすすめ