The sgalng api specs by rainj-me · Pull Request #41585 · sgl-project/sglang

rainj-me · 2026-09-28T20:18:37Z

Motivation

The Rust server and the Python server each hand-wrote their own /generate request and response types, and the Rust gRPC adapter served a third vocabulary (sglang.runtime.v1) with its own conversion layer. Nothing held the three in sync, and drift showed up as silent behavior differences: fields accepted by one server and dropped by the other, and meta_info shapes that only matched by convention.

This PR makes proto/sglang/api/v1 the single contract for both servers. One generator emits the Rust types, their serde implementations, the tonic service, and the Python types from the schema; the Rust HTTP, Rust gRPC, and Python HTTP entrypoints all decode and encode through those generated types; and parity tests pin the three against each other and against the hand-written Python structs.

Modifications

Schema and generator

proto/sglang/api/v1/ defines the native API: GenerateRequest, SamplingParams, the scalar-or-list carriers, typed media inputs (MediaInput, MediaItem, MediaRef), GenerateResponse / GenerateMetaInfo with typed logprob entries and finish reasons, GenerateStreamItem, and the SglangService with Generate, HealthCheck, GetModelInfo, and GetServerInfo.
proto/sglang/json/v1/options.proto is the JSON-fidelity vocabulary: the MessageShape, OneofShape, NullPolicy, and UnknownFields enums plus per-field flags (emit_null_when_absent, raw_json, defaults). Proto3's canonical JSON mapping is not used; these options are the contract. Enums replace the earlier boolean trios so two shapes cannot be set at once.
rust/sglang-api-codegen (cargo gen-api) runs protoc once and emits rust/sglang-api-types/src/generated/* (prost, tonic, and schema-driven serde for api.v1; prost and tonic for runtime.v1) and python/sglang/api/v1/api_types.py (frozen msgspec structs with from_json_value / to_json_value codecs that mirror the Rust serde byte for byte). All output is checked in.
rust/sglang-api-types replaces the old proto/ crate (sglang-grpc-types); sglang-grpc now depends on it. prost and tonic move to 0.14.

Rust server

message/wire.rs is the one bridge from the generated request to the internal types, and into_requests fans out the generated GenerateRequest. The hand-written GenerateBody, SamplingParamsInput, processor_extensions, and the Qwen packing round trip are gone. Wire token ids are int64.
api_server is reorganized as core (the transport-neutral handle, events, errors, and the CoreOutput to api.v1 frame builder), http, and grpc. The gRPC adapter now serves api.v1 over the same fan-out and admission path as HTTP; the runtime.v1 conversion layer and its 26 stubs are deleted. Batch items fail independently as in-stream error items on both transports.
HTTP responses are the generated GenerateResponse serialized through serde. The hand-written meta_info JSON and the OutputAccumulator cache are removed; a small memo for the two growing columns (text, output_ids) keeps cumulative text-only streams O(1) per frame, pinned byte for byte against the typed frame.
ModelInfo / ServerInfo moved to message/info.rs; CoreError::http_status and CoreError::stream_error map errors for both transports.
gRPC applies schema defaults to unset protobuf fields (apply_absent_defaults) after the preferred-sampling fill, so an unset max_new_tokens is 128, not unbounded.

Python server

/generate validates the parsed body against the generated GenerateRequest before the dataclass path and answers a 400 with the schema's error text on a contract violation (entrypoints/api_contract.py).

Tests

rust/sglang-api-types/tests/json_contract.rs and test/registered/unit/api/v1/test_json_contract.py: per-option golden tests, mirrored case for case across the two emitters.
rust/sglang-api-types/testdata/json_parity.json with tests/json_parity.rs and test/registered/unit/api/v1/test_parity.py: 17 shared fixtures that both generators must decode to the same canonical bytes, plus field-set parity against GenerateReqInput and SamplingParams, dataclass normalization of schema bodies, Python meta_info decoding to the schema subset, and key-order pins.
gRPC adapter tests cover streaming, unary folding, drop-abort, in-stream errors, timeouts, batch indexing, preferred params, defaults, and health.

Three contract bugs the parity tests found are fixed in the same change: null_is_none fields now round-trip an explicit null; preferred_sampling_params is raw_json to match the HTTP shape; raw passthroughs keep insertion order regardless of which crate links serde_json.

Behavior changes to note

The Rust server's gRPC endpoint speaks sglang.api.v1, not sglang.runtime.v1.
Python clients sending a /generate body the schema rejects now get a 400 where the dataclass path previously accepted it.
hidden_states values serialize through double, matching Python's tolist() output.
max_new_tokens: null is preserved as null on re-encode.

Howto

cd rust && cargo gen-api

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #37553797497
Latest PR Test (Extra): ❌ Run #37553797318
Latest PR Test (AMD ROCm 10): ❌ Run #37553797573

rainj-me · 2026-09-30T22:09:10Z

/rerun-group rust-server

github-actions · 2026-09-30T22:09:36Z

Results for /rerun-group rust-server:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/rust/test_run_rust_tests.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py

🚀 1-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/vlm/test_rust_native_mm_e2e.py
cd test/ && python3 registered/vlm/test_rust_native_mm_mmmu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cpu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cuda_ipc.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_tp2_shm.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_2node_cpu.py

merrymercy

Two tests, can be in follow up prs:

delete most items in sglang/rust/sglang-server/src/message and use the generated rust api definition
run parity checks. python io_struct.py <-> generated python types <-> generated rust types. check fields, serialization orders, round trips.

merrymercy · 2026-10-03T04:08:51Z

+package sglang.json.v1;
+
+import "google/protobuf/descriptor.proto";
+


simplify and unify the custom option annotations.

why do we need 3? MessageJson, FieldJson, OneofJson? any ways to simplify them?

It seems bare_list and transparent are the same thing.

Checks:

check whether we can simplify them.

check whether they match the current things used in rust (sglang/rust/sglang-server/src/message) and python
(io_struct.py). It can be a subset but needs to match against them.

check whether everything is strongly and correctly typed.

rainj-me · 2026-10-06T21:15:14Z

/rerun-group rust-server

github-actions · 2026-10-06T21:15:43Z

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit 60878c3fba50.

Re-run was not dispatched. What to do:

Rebase your branch onto the latest main and push again
Follow issue #21065 for context
CI-fix PRs may request the bypass-maintenance label to skip this check

rainj-me · 2026-10-06T22:43:40Z

/rerun-group rust-server

github-actions · 2026-10-06T22:44:07Z

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/rust/test_run_rust_tests.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py

🚀 1-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/vlm/test_rust_native_mm_e2e.py
cd test/ && python3 registered/vlm/test_rust_native_mm_mmmu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cpu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cuda_ipc.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_tp2_shm.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_2node_cpu.py

rainj-me · 2026-10-06T23:21:47Z

Two tests, can be in follow up prs:

delete most items in sglang/rust/sglang-server/src/message and use the generated rust api definition

run parity checks. python io_struct.py <-> generated python types <-> generated rust types. check fields, serialization orders, round trips.

delete the serialization related structs that replaced by the api definition, non-serialization related structs are kept. Removed all hand-written json deserialization code
add a parity check test that compare the current rust generated structs by the api definition along with python generated classes and io_struct.py (22 fields)

For followups

Move all api related (de-)serialization classes in io_struct.py to schema
OpenAI apis implementation within grpc
Abstract tower layers for the api-server

…parity test_parity.py::TestRequestFieldParity::test_schema_request_fields_are_a_subset_of_the_dataclass (added on main by sgl-project#41585) requires every GenerateReqInput field to be in the v1 API schema or listed in PYTHON_ONLY_REQUEST_FIELDS. This PR adds GenerateReqInput.parent_session_id for --allow-subagent-keepalive, which the HTTP contract has not adopted, so list it next to session_id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added the dependencies Pull requests that update a dependency file label Sep 28, 2026

rainj-me requested a review from merrymercy September 28, 2026 20:18

rainj-me force-pushed the rainj-me/api-spec branch from 07c17e3 to 9a5ef85 Compare September 30, 2026 22:06

rainj-me requested review from Fridge003, HaiShaw, ishandhanani, ispobock, sogalin and yctseng0211 as code owners September 30, 2026 22:06

github-actions Bot added the documentation Improvements or additions to documentation label Sep 30, 2026

rainj-me mentioned this pull request Sep 30, 2026

[Rust] Extract shared runtime.v1 protobuf bindings #41726

Merged

5 tasks done

merrymercy reviewed Oct 3, 2026

View reviewed changes

Comment thread proto/sglang/api/v1/generate.proto Outdated

Comment thread python/sglang/api/v1/api_types.py

merrymercy reviewed Oct 3, 2026

View reviewed changes

rainj-me requested review from CatherineSue, JustinTong0323 and slin1237 as code owners October 6, 2026 01:10

github-actions Bot added the Multi-modal multi-modal language model label Oct 6, 2026

rainj-me force-pushed the rainj-me/api-spec branch from d1824e3 to 73cd657 Compare October 6, 2026 21:06

merrymercy approved these changes Oct 6, 2026

View reviewed changes

rainj-me force-pushed the rainj-me/api-spec branch from 73cd657 to 235aa9c Compare October 6, 2026 22:42

rainj-me added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026

sglang generate api spec

9789f02

rainj-me added 12 commits October 6, 2026 23:41

change media type from raw_json to MediaInput type

09d9c39

use the generated data type

d3fdb23

rename to wire.rs

4d67724

fix the error after rebase

60e747a

fix the grpc implementation

6c6d4d1

refactor the grpc server to use the schema generate apis

74963ac

refactor the code

7fb2bd9

refactor

9651095

refactor

3f94b2a

refactor

fd4a195

avoid copy token_ids and text

cd7917c

add parity tests

5f65014

rainj-me force-pushed the rainj-me/api-spec branch from bbe6f6f to 5f65014 Compare October 6, 2026 23:49

rainj-me added 3 commits October 7, 2026 00:19

lint

bed6534

fix tests

39aa16e

fix tests

1ff0611

merrymercy merged commit c52dcc5 into main Oct 7, 2026
159 of 184 checks passed

merrymercy deleted the rainj-me/api-spec branch October 7, 2026 08:55

nvpohanh mentioned this pull request Oct 7, 2026

[Perf] hicache: Add serialized load-back and subagent keepalive to raise usable prefix-cache tokens #41865

Open

5 tasks

bingxche mentioned this pull request Oct 8, 2026

[CI Monitor] Daily Report - 2026-10-08 bingxche/sglang-ci-bot#219

Open

kchatr mentioned this pull request Oct 8, 2026

[Feat] runtime: Support tokenized multimodal generation #42833

Merged

2 of 5 tasks

bingxche mentioned this pull request Oct 9, 2026

[CI Monitor] Daily Report - 2026-10-09 bingxche/sglang-ci-bot#220

Open

OrangeRedeng mentioned this pull request Oct 9, 2026

[NPU] Diffusion CI fixes #43210

Merged

2 of 5 tasks

jain-ria mentioned this pull request Oct 12, 2026

[Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1 #42118

Merged

ShangmingCai mentioned this pull request Oct 14, 2026

[Fix] sglang-processor: Send watermark only when the request sets it #44269

Merged

adenzhou1350 mentioned this pull request Oct 16, 2026

[Perf] rust-stream: Build cumulative frames in one buffer #39220

Closed

Navigation Menu

Repository navigation

The sgalng api specs - #41585

The sgalng api specs#41585
merrymercy merged 23 commits into
mainfrom
rainj-me/api-spec

rainj-me commented Sep 28, 2026 •

edited by github-actions Bot

Loading

Uh oh!

rainj-me commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

merrymercy left a comment

Uh oh!

merrymercy Oct 3, 2026

Uh oh!

rainj-me commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

rainj-me commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

rainj-me commented Oct 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		package sglang.json.v1;

		import "google/protobuf/descriptor.proto";

Conversation

rainj-me commented Sep 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Schema and generator

Rust server

Python server

Tests

Behavior changes to note

Howto

Checklist

Review and Merge Process

CI States

Uh oh!

rainj-me commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merrymercy left a comment

Choose a reason for hiding this comment

Uh oh!

merrymercy Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

rainj-me commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

⚠️ Rebase Required Before Re-run

Uh oh!

rainj-me commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rainj-me commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rainj-me commented Sep 28, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Oct 6, 2026 •

edited

Loading

rainj-me commented Oct 6, 2026 •

edited

Loading