Build an LLM Gateway for 4 AI Models in 13 Steps [2026]

Nadia Dubois
October 14, 2026
22 min read

Four new AI models landed inside a ten-day window this October: OpenAI’s Ultrafast tier for GPT-6.1 Sol on October 8, Anthropic’s Claude Haiku 5.5 on October 7, StepFun’s Step 5 Preview on October 8, and Ant Group’s Ling-3.1-Flash in the final days of September. Each one ships a different SDK, a different pricing table, and a different context limit. Picking just one and hardcoding it into your app is how you end up re-writing integration code every six weeks. An LLM gateway solves that by putting a single, OpenAI-compatible endpoint in front of every provider you use, so switching models is a one-line config change instead of a weekend refactor.

This tutorial walks through building that gateway from scratch, then wiring in all four of this month’s newest releases. By the end you will have a working project that routes requests, streams tokens, falls back automatically when a provider rate-limits you, and logs cost and latency so you can compare models on your own workload instead of trusting a vendor’s marketing page.

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

Why Developers Are Building LLM Gateways in Late 2026

The 2025 Stack Overflow Developer Survey found that “81.4% of developers used OpenAI GPT models for development work in the past year,” while “42.8% of developers used Claude Sonnet models” and “35.3% of developers used Gemini Flash models” in the same period, according to Stack Overflow’s survey data. Those numbers overlap heavily. Most teams are not choosing one provider and sticking with it; they are running two or three side by side, often for different tasks inside the same product. That is exactly the pattern an LLM gateway is built to support.

October 2026 made the case even stronger. A single tracking site logged 12 new AI models released that month from 12 different providers as of mid-October, according to a roundup of October 2026 AI model updates. Specialist models, agentic upgrades, and regional challengers are shipping faster than any one team can integrate by hand. A gateway turns “add a new model” into a config edit rather than a new SDK dependency and a new auth flow.

There is also a practical cost angle. Hosted gateway providers now expose large model catalogs through one API surface, letting you swap a model string without touching client code. That matters when pricing shifts, when a provider rate-limits you mid-incident, or when a newer, cheaper model beats your current default on the exact task you care about. This guide builds that capability yourself, so you understand every moving part instead of depending on a black box.

Vendor lock-in is the quieter reason teams reach for a gateway. A codebase that calls openai.chat.completions.create() directly in forty different files is a codebase that cannot move when a better or cheaper model shows up next quarter. Engineers who lived through a provider outage know the pattern: a single company has a bad afternoon, and every product that hardcoded that one SDK call goes down with it. Routing every model call through one function, as this tutorial does, turns that kind of outage into a one-line config change instead of an emergency deploy.

Prerequisites: Tools, Accounts, and API Keys You Need

Gather these before you start. None of them cost more than a few dollars in test credits, and some providers offer free trial access.

  • Node.js 22 or newer (LTS) with the built-in fetch API, so you don’t need an extra HTTP client library.
  • An OpenAI API key with access to gpt-6.1-sol and its Ultrafast service tier.
  • An Anthropic API key with access to Claude Haiku 5.5.
  • A StepFun API key for Step 5 Preview, available through StepFun’s developer console.
  • Access to Ling-3.1-Flash through Vercel’s AI Gateway, where Ant Group’s inclusionAI lab made the model available as a free two-week trial hosted by Novita.
  • A code editor, a terminal, and roughly $10 to $20 in provider credits for testing across four accounts.
  • Basic familiarity with environment variables and async JavaScript (Promises, async/await).

One note on model identifiers: the slugs used in this guide, such as gpt-6.1-sol, match each provider’s public documentation as of October 14, 2026. Always confirm the exact string in your own dashboard before shipping to production, since providers occasionally rename a model slug after a preview period ends.

Budget for testing matters more than it sounds. Running the full tutorial end to end, including the benchmark script in Step 13 and a handful of retries while you debug streaming, typically burns well under a dollar in token costs across all four providers, since the sample prompts are short. The real cost driver is time spent per account on sign-up and key generation, not API usage, so set aside twenty to thirty minutes just for account creation before you write any code.

Step 1: Understand How an LLM Gateway Routes Requests

Strip away the branding and every LLM gateway does the same four things. First, it accepts a request in one consistent shape, usually the OpenAI chat-completions or Responses format, because that is the de facto standard most SDKs already speak. Second, it reads a model field to decide which provider adapter should handle the call. Third, it attaches the correct authentication header for that provider, whether that is a bearer token, an x-api-key header, or a provider-specific signing scheme. Fourth, it normalizes the response back into a shape your application code already understands, so switching providers never means rewriting your parsing logic.

Everything else, like streaming, retries, caching, and cost tracking, sits on top of that core routing layer. Hosted products such as LiteLLM’s proxy, Portkey, TrueFoundry, and llmgateway.io all follow this pattern; a guide published by LLM Gateway describes exactly this setup: “get a key, send your first chat completion… switch models by changing one string, stream tokens, pin or let the gateway pick the provider, and keep using the OpenAI and Anthropic SDKs you already” have. Building your own lightweight version of this, rather than adopting a vendor’s full product, gives you full control over fallback logic and keeps you from being locked into one gateway’s pricing.

Step 2: Choose a Self-Hosted or Hosted Gateway Approach

You have three realistic paths. The first is a fully self-hosted proxy you run yourself, which keeps all traffic and credentials inside your own infrastructure but means you own patching and uptime. The second is a hosted gateway product that manages provider adapters for you in exchange for a usage fee or markup. The third, which this tutorial builds, is a thin custom layer: a single Node.js module that implements just enough routing logic for your four target models, with no external gateway dependency at all. It is the fastest way to understand the mechanics, and you can graduate to a hosted product later without changing how your application code calls it.

ApproachExampleSelf-hosted?Pricing modelBest for
Custom thin layerThe project built in this tutorialYesFree — you pay only each provider’s own API costSmall teams who want full control and no markup
Self-hosted proxyLiteLLM proxyYesFree, open source; you host the serverTeams needing data residency with many providers
Hosted gatewayLLM Gateway (llmgateway.io)NoUsage-based, pay-as-you-goTeams wanting hundreds of models with zero infra
Hosted gatewayPortkeyNoFree tier plus paid plansTeams wanting built-in observability and caching
Cloud platform gatewayVercel AI GatewayNoUsage-based; some launch models offered on free trialTeams already on Vercel, including early access to Ling-3.1-Flash

Notice that Ling-3.1-Flash’s own launch channel is a hosted gateway, not a direct API. Ant Group’s inclusionAI lab made the model available as “a two-week free trial through Vercel’s AI Gateway, with hosting provided by Novita,” and as of its launch the model was “not yet self-hostable… because its open weights, license details, and post-trial pricing had not been posted,” according to tracking by LLM Releases’ Ling family page. That is a useful real-world example of why a gateway-first architecture pays off: you can add a brand-new, not-yet-fully-public model to your routing table on day one without waiting for a direct SDK integration.

Step 3: Scaffold the Project and Install Dependencies

Create a new directory and initialize it as a standard Node.js project. Because Node 22 ships a native fetch implementation, this project needs zero runtime dependencies for the core gateway, only a small dotenv-style loader for convenience.

mkdir llm-gateway-tutorial && cd llm-gateway-tutorial
npm init -y
mkdir src
touch src/gateway.js src/providers.js src/index.js .env
echo "node_modules/\n.env" > .gitignore
npm pkg set type="module"

The folder structure stays deliberately flat: providers.js holds the per-provider adapters, gateway.js holds the routing and fallback logic, and index.js is where you call the gateway from your application code. Keeping the provider adapters in their own file means adding a fifth model later is a matter of appending one object, not editing the core routing function.

Step 4: Store API Keys for Four Providers Securely

Never hardcode credentials into providers.js. Put every key in .env, keep that file out of version control, and load it once at process start. This also makes it trivial to run the same codebase against a staging set of keys versus production keys.

# .env
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
STEPFUN_API_KEY=your-stepfun-key
VERCEL_AI_GATEWAY_KEY=your-vercel-ai-gateway-key

# Optional: override base URLs if a provider changes its endpoint
OPENAI_BASE_URL=https://api.openai.com/v1
ANTHROPIC_BASE_URL=https://api.anthropic.com/v1
STEPFUN_BASE_URL=https://api.stepfun.com/v1
VERCEL_AI_GATEWAY_BASE_URL=https://gateway.ai.vercel.app/v1

Loading these values is a two-line job once you add a minimal env loader at the top of src/index.js, importing from Node’s built-in node:process module after reading .env into the process environment with a lightweight parser or Node’s native --env-file flag (available in Node 20.6 and later): run the project with node --env-file=.env src/index.js instead of pulling in an external package at all.

Step 5: Write the Model Routing Configuration

This is the heart of the gateway: one object per model that tells the router which base URL to hit, which auth header format to use, and how to translate your request into that provider’s expected payload shape.

// src/providers.js
export const providers = {
  "gpt-6.1-sol": {
    baseUrl: process.env.OPENAI_BASE_URL,
    authHeader: () => ({ Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }),
    path: "/responses",
    buildBody: (prompt, opts = {}) => ({
      model: "gpt-6.1-sol",
      input: prompt,
      service_tier: opts.ultrafast ? "ultrafast" : "default",
      stream: opts.stream ?? false,
    }),
  },
  "claude-haiku-5.5": {
    baseUrl: process.env.ANTHROPIC_BASE_URL,
    authHeader: () => ({
      "x-api-key": process.env.ANTHROPIC_API_KEY,
      "anthropic-version": "2023-06-01",
    }),
    path: "/messages",
    buildBody: (prompt, opts = {}) => ({
      model: "claude-haiku-5.5",
      max_tokens: opts.maxTokens ?? 1024,
      messages: [{ role: "user", content: prompt }],
      stream: opts.stream ?? false,
    }),
  },
  "step-5-preview": {
    baseUrl: process.env.STEPFUN_BASE_URL,
    authHeader: () => ({ Authorization: `Bearer ${process.env.STEPFUN_API_KEY}` }),
    path: "/chat/completions",
    buildBody: (prompt, opts = {}) => ({
      model: "step-5-preview",
      messages: [{ role: "user", content: prompt }],
      stream: opts.stream ?? false,
    }),
  },
  "ling-3.1-flash": {
    baseUrl: process.env.VERCEL_AI_GATEWAY_BASE_URL,
    authHeader: () => ({ Authorization: `Bearer ${process.env.VERCEL_AI_GATEWAY_KEY}` }),
    path: "/chat/completions",
    buildBody: (prompt, opts = {}) => ({
      model: "ling-3.1-flash",
      messages: [{ role: "user", content: prompt }],
      stream: opts.stream ?? false,
    }),
  },
};

The table below summarizes what you are routing between. Treat any cell marked “not published” as a signal to check the provider’s own docs before you finalize a production budget, since this tutorial only reports figures confirmed in current provider documentation and release tracking.

ModelProviderPublic releaseContext windowPrice per million tokens (in / out)
GPT-6.1 Sol (Ultrafast tier)OpenAISept. 29, 2026; Ultrafast added Oct. 8, 2026Not published in this comparisonNot published in this comparison
Claude Haiku 5.5AnthropicOct. 7, 20261,000,000 tokens$0.10 / $0.50
Step 5 PreviewStepFunOct. 8, 20261,000,000 tokens$1.00 / $2.70
Ling-3.1-FlashAnt Group (inclusionAI)Late Sept. 2026 (Sept. 29–30)Not published in this comparisonFree during 2-week launch trial; post-trial pricing not yet posted

Note that GPT-6.1 Sol’s context window and standard per-token pricing were not disclosed in OpenAI’s public changelog entry for the Ultrafast tier, only that the feature “is available to all API users, subject to rate limits, with global processing and US and EU data residency,” per the OpenAI API changelog. Pull the current figures from your own OpenAI dashboard before estimating cost at scale.

Step 6: Send Your First Call to GPT-6.1 Sol

With the provider map in place, the router itself is short: look up the model, build the request, attach the right headers, and send it. Here is the minimal version before fallback or streaming logic gets added.

// src/gateway.js
import { providers } from "./providers.js";

export async function callModel(model, prompt, opts = {}) {
  const provider = providers[model];
  if (!provider) throw new Error(`Unknown model: ${model}`);

  const res = await fetch(`${provider.baseUrl}${provider.path}`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      ...provider.authHeader(),
    },
    body: JSON.stringify(provider.buildBody(prompt, opts)),
  });

  if (!res.ok) {
    const errText = await res.text();
    throw new Error(`${model} returned ${res.status}: ${errText}`);
  }

  return res.json();
}

Call it from src/index.js with await callModel("gpt-6.1-sol", "Summarize this changelog in two sentences."). The first time you run this, expect a few minutes of debugging around response shape differences, since OpenAI’s Responses API, Anthropic’s Messages API, and the OpenAI-compatible chat-completions format used by StepFun and the Vercel AI Gateway are all slightly different. That mismatch is exactly what the normalization step in Step 11 exists to fix.

Step 7: Turn On GPT-6.1 Sol’s Ultrafast Service Tier

OpenAI’s October 8, 2026 changelog entry describes the Ultrafast tier plainly: use gpt-6.1-sol “with service_tier: \"ultrafast\" to reduce the time between generated output tokens.” In the provider config from Step 5, that is already wired up through the opts.ultrafast flag, so turning it on from your calling code is a one-line change.

const fast = await callModel("gpt-6.1-sol", "Draft a release note.", {
  ultrafast: true,
});
console.log(fast);

Because Ultrafast is a service tier rather than a different model, your cost accounting still needs to track it separately. Log the service_tier value alongside every response so your benchmark data in Step 13 can tell you whether the latency gain is worth any premium OpenAI applies to that tier on your account.

Step 8: Route a Request to Claude Haiku 5.5

This is the step that proves the gateway is actually doing its job. Change nothing in gateway.js, change nothing in your application logic, swap one string, and the exact same function call now talks to a completely different company’s infrastructure.

const claudeResponse = await callModel(
  "claude-haiku-5.5",
  "Draft a release note.",
  { maxTokens: 512 }
);
console.log(claudeResponse);

Claude Haiku 5.5 is worth defaulting to for high-volume, latency-sensitive tasks: it carries a 1-million-token context window at $0.10 per million input tokens and $0.50 per million output tokens, among the cheapest rates of the four models in this guide. That price point makes it a natural default for chat support, summarization, and other jobs where you are running thousands of calls a day and GPT-6.1 Sol’s agentic depth is overkill.

Step 9: Add StepFun Step 5 Preview for Long-Context Jobs

StepFun’s Step 5 Preview launched on October 8, 2026 with the same 1-million-token context window as Claude Haiku 5.5, but at a noticeably higher rate: $1.00 per million input tokens and $2.70 per million output tokens, as tracked by a live AI model releases timeline. Route to it when a task genuinely needs that full context window and you want a second provider’s output to sanity-check Claude’s, rather than as your default route.

const stepResponse = await callModel(
  "step-5-preview",
  "Read this 400-page contract and flag any auto-renewal clauses.",
);
console.log(stepResponse);

Because Step 5 Preview is explicitly labeled a preview release, treat it the way you would any preview-tier model from any provider: fine for prototyping and comparison testing, but worth re-checking before you lock it into a production default, since preview models can change behavior or pricing once they graduate to general availability.

Step 10: Wire In Ling-3.1-Flash as a Low-Cost Trial Route

Ling-3.1-Flash is the successor to Ant Group’s Ling-3.0-Flash, unveiled by the inclusionAI lab in the final days of September 2026. Unlike the other three models in this guide, it did not launch with a direct API: it shipped as a two-week free trial routed through Vercel’s AI Gateway and hosted on Novita’s infrastructure. That makes it a good stress test for your gateway’s flexibility, since you are routing through a gateway-of-a-gateway rather than hitting the model’s own servers directly.

const lingResponse = await callModel(
  "ling-3.1-flash",
  "Translate this product description into formal Mandarin.",
);
console.log(lingResponse);

Treat this route as explicitly temporary in your code comments. Because open weights, a license, and post-trial pricing had not been published at launch, build your fallback chain (Step 12) so that if Ling-3.1-Flash access expires or changes shape, traffic automatically drops to Claude Haiku 5.5 instead of failing outright.

Step 11: Stream Responses From Any of the Four Models

Non-streaming calls are fine for batch jobs, but any chat-style product needs tokens arriving incrementally. Each provider streams via server-sent events, but the chunk format differs slightly, so the gateway needs a small normalization layer that yields plain text regardless of which provider is on the other end.

// src/gateway.js (continued)
export async function* streamModel(model, prompt, opts = {}) {
  const provider = providers[model];
  const res = await fetch(`${provider.baseUrl}${provider.path}`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      ...provider.authHeader(),
    },
    body: JSON.stringify(provider.buildBody(prompt, { ...opts, stream: true })),
  });

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    const lines = buffer.split("\n");
    buffer = lines.pop();
    for (const line of lines) {
      if (!line.startsWith("data: ") || line.includes("[DONE]")) continue;
      try {
        const chunk = JSON.parse(line.slice(6));
        const text =
          chunk.choices?.[0]?.delta?.content ??
          chunk.delta?.text ??
          chunk.output_text_delta ??
          "";
        if (text) yield text;
      } catch {
        // partial JSON chunk, wait for more data
      }
    }
  }
}

Consume it the same way regardless of model: for await (const token of streamModel("gpt-6.1-sol", prompt)) { process.stdout.write(token); }. The three-way fallback chain inside the text lookup covers OpenAI-compatible delta chunks, Anthropic’s delta format, and the Responses API’s output-text delta field, which is what lets one streaming function serve all four providers.

Step 12: Build Automatic Fallback for Rate Limits and Outages

A gateway that still goes down whenever one provider has a bad day has not actually solved the reliability problem. Wrap callModel in a small retry chain that tries a primary model, catches rate-limit or 5xx errors, and falls through to a backup model before giving up.

// src/gateway.js (continued)
export async function callWithFallback(chain, prompt, opts = {}) {
  const errors = [];

  for (const model of chain) {
    try {
      const result = await callModel(model, prompt, opts);
      return { model, result, errors };
    } catch (err) {
      errors.push({ model, message: err.message });
      const isRetryable = /429|500|502|503|504/.test(err.message);
      if (!isRetryable) throw err; // don't silently fall back on bad requests
    }
  }

  throw new Error(
    `All models in fallback chain failed: ${JSON.stringify(errors)}`
  );
}

Call it with an ordered list, cheapest or fastest first: callWithFallback(["claude-haiku-5.5", "step-5-preview", "gpt-6.1-sol"], prompt). The regex check matters more than it looks: a malformed request (a 400 error) should surface immediately as a bug in your code, not get silently retried against three other providers while you wonder why nothing works.

Step 13: Benchmark Latency, Cost, and Quality Side by Side

Once every model answers through the same function signature, comparing them is just a loop with a timer. Run the identical prompt against all four, record wall-clock time and reported token usage, and divide by each model’s published per-token price to get a real cost-per-request figure for your workload instead of a vendor’s benchmark.

// src/benchmark.js
import { callModel } from "./gateway.js";

const prompt = "Explain the tradeoffs of a self-hosted LLM gateway in 3 bullet points.";
const models = ["gpt-6.1-sol", "claude-haiku-5.5", "step-5-preview", "ling-3.1-flash"];

for (const model of models) {
  const start = Date.now();
  const res = await callModel(model, prompt);
  const elapsedMs = Date.now() - start;
  console.log(`${model}: ${elapsedMs}ms`, res.usage ?? "usage not reported");
}

The table below shows the shape of output you should expect from a single sample run, not a verified cross-model ranking. Your own numbers will shift with region, time of day, and prompt length, which is exactly why you run this script yourself rather than trusting a static chart.

Field loggedWhat it tells youExample console line
Model nameWhich route produced this resultclaude-haiku-5.5
Elapsed timeWall-clock latency for this exact prompt, on your networkclaude-haiku-5.5: 842ms
Usage objectInput/output token counts reported by the providerusage: { input_tokens: 24, output_tokens: 96 }
Computed costUsage multiplied by that model’s published ratecost: $0.00005 for this single call
Fallback trailWhether the primary model succeeded or a backup kicked inerrors: [] (primary succeeded)

Run this nightly against a fixed prompt set and you build your own historical record of how each provider’s latency and pricing drift over time, which is far more useful for planning than a single launch-day comparison.

Matching Each Model to the Right Job

With four working routes and real benchmark numbers in hand, the next question is which model should actually sit at the front of each fallback chain. There is no single right answer, but the four models in this guide do split cleanly along a few lines once you look at what each one was built for.

GPT-6.1 Sol earns its place as an agentic-work upgrade over the earlier GPT-6 Sol release, and its Ultrafast service tier is explicitly built to cut the gap between generated tokens, not to cut raw cost. That makes it the right default for interactive, tool-calling workloads such as coding agents or multi-step task runners, where a user is watching the screen and every extra second of latency is felt directly. It is a weaker choice as a default for high-volume batch summarization, where its per-token economics have not been published and Claude Haiku 5.5’s confirmed $0.10 input rate is hard to beat.

Claude Haiku 5.5 and StepFun’s Step 5 Preview both carry a 1-million-token context window, which puts them in direct competition for long-document work such as contract review, codebase-wide search, or multi-file refactoring prompts. The deciding factor in this guide is price: Claude Haiku 5.5’s $0.10 input and $0.50 output rate per million tokens is roughly a tenth of Step 5 Preview’s $1.00 and $2.70. That gap is large enough that Step 5 Preview makes more sense as a secondary opinion you consult occasionally, not a route you send every request through by default, unless your own quality testing shows it consistently outperforms Claude on your specific documents.

Ling-3.1-Flash sits in a category of its own: a model you adopt opportunistically while it is free, with a hard deadline attached. Treat it as a bonus route for non-critical, easily-redone tasks during its trial window, such as draft translations or low-stakes classification, rather than anything your product depends on. The moment the trial period described by Ant Group’s inclusionAI lab ends, that route either needs a confirmed price you are willing to pay or it needs to disappear from your fallback chain entirely.

Common Pitfalls When Running a Multi-Model Gateway

  • Treating every “stream” field the same way. OpenAI’s Responses API, Anthropic’s Messages API, and OpenAI-compatible chat-completions endpoints all format streaming chunks differently. Skipping the normalization step in Step 11 means your UI breaks the moment you add a second provider.
  • Hardcoding a fallback chain that never changes. A chain built in October around Step 5 Preview being a cheap backup stops making sense the day that preview graduates to general availability pricing. Revisit fallback order monthly, not once at launch.
  • Ignoring provider-specific rate limit headers. A blanket retry-on-429 strategy without reading the Retry-After header burns through your limit faster and can extend a temporary block into a longer one.
  • Building for a trial model as if it were permanent. Ling-3.1-Flash launched as a two-week free trial with no confirmed post-trial pricing. Routing production traffic to it without a tested fallback is a guaranteed outage the day the trial ends.
  • Comparing models on a vendor’s benchmark instead of your own prompts. Published benchmark scores rarely reflect your actual system prompts, document lengths, or output format requirements. The benchmark script in Step 13 exists specifically so you are not making routing decisions off someone else’s marketing chart.
  • Forgetting that service tiers are not separate models. GPT-6.1 Sol’s Ultrafast tier is a flag on the same model, not a different one. Log it separately in your cost tracking or your per-model cost figures will be wrong.

Troubleshooting Your LLM Gateway Setup

  • 401 Unauthorized on every request. Confirm .env is actually being loaded. Running node src/index.js without --env-file=.env leaves process.env empty and every auth header comes through blank.
  • Anthropic calls fail but OpenAI calls succeed. Check the anthropic-version header. Anthropic’s Messages API rejects requests missing that header, even when the API key itself is valid.
  • Streaming hangs with no output. Verify the response body actually supports getReader() in your runtime. Some HTTP client wrappers buffer the full response before returning it, which defeats streaming entirely.
  • Fallback chain always exhausts to the last model. Log the raw error message before your regex check runs. A provider that changed its error response format may no longer match /429|500|502|503|504/, so every call looks “non-retryable” and throws immediately.
  • Ling-3.1-Flash requests return 404. Double check you are hitting Vercel’s AI Gateway base URL, not a direct Ant Group endpoint. The model is not available outside that gateway during its trial period.
  • Cost tracking numbers don’t match your provider invoice. Confirm you are reading usage.input_tokens and usage.output_tokens correctly; OpenAI, Anthropic, and StepFun each place these fields at slightly different depths in the response object.
  • Ultrafast tier requests get silently downgraded. Some SDK wrappers strip unrecognized fields. If you built providers.js with a wrapper library instead of raw fetch, confirm service_tier survives serialization.
  • Intermittent timeouts only on long-context prompts. Both Claude Haiku 5.5 and Step 5 Preview support up to a 1-million-token context window, but very large payloads can exceed a default client timeout. Raise your fetch timeout or chunk the input before sending it.

Advanced Tips: Caching, Semantic Routing, and Cost Guardrails

Once the basic gateway works, three upgrades pay off quickly. The first is response caching: hash the prompt plus the model name and opts, store the result for a short TTL, and skip the network call entirely on a repeat request. This matters most for support bots and documentation assistants where the same handful of questions get asked constantly.

The second is semantic routing: instead of a static fallback chain, classify the incoming prompt first (short classification call to your cheapest model) and route long-context document analysis to Step 5 Preview or Claude Haiku 5.5, quick factual lookups to whichever model is fastest that hour, and anything agentic or tool-calling-heavy to GPT-6.1 Sol. This turns your gateway from a dumb switch into something closer to a router that optimizes for both cost and task fit.

The third is a hard cost guardrail. Track cumulative spend per provider per day in a simple counter, and once a model crosses a threshold you set, automatically drop it from the fallback chain until the next billing cycle. This single change prevents the most common gateway failure mode: a runaway loop or a traffic spike that burns through a month’s budget on one provider in a single afternoon.

A fourth upgrade worth the extra hour is structured logging. Instead of printing plain strings to the console, emit one JSON line per request containing the model name, elapsed time, token counts, whether a fallback fired, and a request ID you also return to the caller. That single change makes it possible to pipe gateway logs into any log aggregator you already run and answer questions like “which model caused last night’s latency spike” without adding a dedicated observability product. Because every provider adapter in this project returns through the same callModel function, there is exactly one place to add that logging line, which is the kind of payoff a thin custom gateway gives you that a thicker, more opinionated hosted product sometimes makes harder to customize.

If you later decide a custom layer is more than your team wants to maintain, the architecture transfers directly: a production-focused AI system design guide covering October 2026’s model lineup, including Claude Opus 5.5, GPT-6 Astra/Sol/Luna, GPT-6.1 Sol, and DeepSeek V4.1-Flash, works through the same routing, fallback, and caching patterns at a larger scale, which is a useful next read once your prototype outgrows a single Node.js file.

The Complete Working Gateway Project

Putting every piece together, your finished project has four files. providers.js holds the four model adapters from Step 5. gateway.js holds callModel, streamModel, and callWithFallback from Steps 6, 11, and 12. benchmark.js holds the comparison loop from Step 13. The entry point ties them together for real use.

// src/index.js
import { callWithFallback, streamModel } from "./gateway.js";

async function main() {
  // Non-streaming call with automatic fallback
  const { model, result } = await callWithFallback(
    ["claude-haiku-5.5", "step-5-preview", "gpt-6.1-sol"],
    "Write a one-paragraph status update for a release that is on schedule."
  );
  console.log(`Answered by: ${model}`);
  console.log(result);

  // Streaming call direct to a specific model
  console.log("\nStreaming from gpt-6.1-sol (Ultrafast):");
  for await (const token of streamModel("gpt-6.1-sol", "List 3 risks in this plan.", {
    ultrafast: true,
  })) {
    process.stdout.write(token);
  }
}

main().catch((err) => {
  console.error("Gateway failed:", err.message);
  process.exit(1);
});

Run it with node --env-file=.env src/index.js. A healthy run prints which model answered the fallback call, the response text, and then a live-streamed answer from GPT-6.1 Sol’s Ultrafast tier. From here, the project is ready to drop behind an internal API route, a Slack bot, or any service that currently calls one provider’s SDK directly. Swap that direct call for callWithFallback and you have already solved the next six months of “which model should we use for this” conversations.

Frequently Asked Questions

Do I need a paid gateway product to do this, or can I really just build it myself?
For four providers, a self-built layer like the one in this tutorial is genuinely simpler than onboarding a hosted gateway product. Hosted options such as LLM Gateway or Portkey earn their keep once you are juggling dozens of models or need built-in observability dashboards you do not want to build yourself.

Is Ling-3.1-Flash safe to use in production given it is only a trial?
Not on its own. Since open weights, licensing, and post-trial pricing had not been published at launch, route any production traffic through the fallback chain from Step 12 so a backup model picks up the work automatically once the trial window closes or terms change.

Why does GPT-6.1 Sol’s Ultrafast tier not show pricing in the comparison table?
OpenAI’s public changelog entry for the feature confirms its existence and availability but does not publish a separate price for the tier. Check your own OpenAI account dashboard for current rates before budgeting around it.

Can I add a fifth model, like Gemini 4 Argon or DeepSeek V4.1-Flash, to this same gateway?
Yes. Add one new object to the providers map in providers.js with that model’s base URL, auth header, and body format, then include its name in any fallback chain. Nothing else in gateway.js needs to change.

Which of these four models is cheapest for high-volume use?
Based on published rates, Claude Haiku 5.5 is the cheapest of the three with confirmed pricing in this guide, at $0.10 per million input tokens and $0.50 per million output tokens, against Step 5 Preview’s $1.00 and $2.70. Ling-3.1-Flash is currently free during its trial, but that is not a durable long-term price.

My streaming function returns garbled text with some models. What’s wrong?
You are likely missing one of the three delta field fallbacks in Step 11. OpenAI-compatible endpoints use choices[0].delta.content, Anthropic uses delta.text, and OpenAI’s Responses API uses output_text_delta. Confirm all three lookups are present in your parsing logic.

How often should I re-run the benchmark in Step 13?
Monthly at minimum, and immediately after any provider announces a new model or pricing change. Preview-labeled models like Step 5 Preview are especially likely to shift behavior once they reach general availability.

Does this gateway work with Python instead of Node.js?
The pattern translates directly. Replace the native fetch calls with Python’s httpx or requests library, keep the same provider map structure, and use an async generator for the streaming function if you are on Python 3.12 or newer with asyncio support.

Related Coverage

Nadia Dubois

Nadia Dubois

AI & Innovation Editor

Nadia Dubois is the AI & Innovation Editor at Tech Insider, where she tracks the rapid evolution of artificial intelligence, from foundation models to real-world enterprise deployment. She previously covered AI and startups for La Tribune and contributed to MIT Technology Review's European coverage. Nadia specializes in generative AI, AI regulation, and the intersection of technology and European industrial policy. She holds a dual degree in Computational Linguistics and Journalism from Sciences Po Paris.

View all articles

Related Posts

Tech Insider delivers in-depth coverage of the technologies shaping the future: AI, cybersecurity, cloud computing, hardware, and the trends that matter.

[email protected]

Company

Explore

Categories

© 2026 Tech Insider Media AB. All rights reserved.
EN AU CA IE IT DE NL SV FR FI ES