Core Java

In-Process Inference vs Calling a Service: ONNX Runtime and DJL Against a Remote Model

Every Java team that adds a machine learning model faces the same early choice. Do you load the model inside your application, or do you call it over the network? The answer depends on one number that most teams never measure.

Two ways to put a model into a Java application

The first option is in-process inference. Your JVM loads the model file and runs it directly. For ONNX models, the ONNX Runtime Java API does this. It is a binding for Java 8 or newer, and the library documentation explains that the jar bundles the native runtime together with the JNI glue.

Alternatively, you can use the Deep Java Library (DJL). It is an engine-agnostic Java framework, and it ships an ONNX Runtime engine. The DJL team even advises against using the engine classes directly, because that would tie your code to one engine.

The second option is a remote service. The model lives behind an HTTP endpoint, and your application sends it requests. DJL Serving is one example. It is a stand-alone model server with a REST inference API and dynamic batching.

What each choice buys you

Both approaches work. However, they trade different things, as Table 1 shows.

AspectIn-process (ONNX Runtime or DJL)Remote service
Latency per callNo network hopAdds serialization and a round trip
ScalingScales with your app instancesModel tier scales separately
MemoryEvery app instance holds a model copyOne copy per model server
ReleasesModel ships with the appModel changes without an app deploy
FailureA native crash hits your appIsolated, but network errors appear
GPUsEvery app instance needs accessConcentrated in one tier

Table 1. The main trade-offs between the two designs.

What a call really costs

Most of Table 1 is easy to reason about. Latency is not, so I measured it. I built three synthetic ONNX models of different sizes and ran each one in two ways: directly in the process, and behind a local HTTP service.

One honest limit first. Maven Central was unreachable from my test environment, so I could not run the Java ONNX Runtime API itself. Instead, I used the Python binding of ONNX Runtime 1.24.4 with one thread per session. Later, I cross-checked the service side with a Java HTTP client.

Self-run benchmark, one request at a time, median of seven interleaved rounds. Synthetic models with random weights (9 KB, 1.6 MB, and 25 MB). Single-core sandbox with client and server on loopback, and a plain Python server, so absolute service numbers are pessimistic. The vertical axis is logarithmic.

The gap is striking for small models. The small model took about 0.007 ms in process, but 0.303 ms behind a JSON service. In other words, the call became roughly 43 times slower, even though the network was only loopback. As the model grows, the fixed cost matters less, because the compute dominates. Table 2 shows the ratios.

ModelIn-process p50JSON service vs in-processBinary service vs in-process
Small (9 KB)0.007 ms43 times slower32 times slower
Medium (1.6 MB)0.054 ms12 times slower6 times slower
Large (25 MB)1.010 ms3.3 times slower1.4 times slower

Table 2. Median latency ratios from the same run as Figure 1.

The wire format matters more than the network

Notice the third series in the chart. I also sent the same input as raw bytes instead of JSON. For the medium model, that cut the median from 0.630 ms to 0.332 ms. The difference grows with batch size, since JSON turns every float into text.

Self-run benchmark with the medium model (256 inputs), median of seven interleaved rounds. Cost per sample is the median call time divided by the batch size. The vertical axis is logarithmic.

At a batch of 128, the JSON service needed 32.1 ms, while the binary service needed 1.3 ms, and in-process inference needed 0.86 ms. Therefore, batching helps a remote service only when the payload stays compact. Fortunately, binary formats are a design choice, not a law of nature.

Measure your own service from Java

The numbers above come from my machine, so yours will differ. The snippet below times repeated POST calls with the JDK’s built-in HttpClient. It needs JDK 11 or newer, and it sends a 256-value JSON input like the one my medium model expects. Point it at your own endpoint and adjust the payload.

cat > Ping.java <<'EOF'
import java.net.URI;
import java.net.http.*;
import java.util.Arrays;

public class Ping {
  public static void main(String[] a) throws Exception {
    String url = a[0];
    int n = Integer.parseInt(a[1]);
    String json = "{\"x\":[[" + "0.1,".repeat(255) + "0.1]]}";
    HttpClient c = HttpClient.newHttpClient();
    HttpRequest r = HttpRequest.newBuilder(URI.create(url))
      .header("Content-Type", "application/json")
      .POST(HttpRequest.BodyPublishers.ofString(json)).build();
    long[] t = new long[n];
    for (int i = 0; i < n + 200; i++) {
      long s = System.nanoTime();
      c.send(r, HttpResponse.BodyHandlers.ofString());
      if (i >= 200) t[i - 200] = System.nanoTime() - s;
    }
    Arrays.sort(t);
    System.out.printf("p50 %.3f ms, p99 %.3f ms%n",
      t[n / 2] / 1e6, t[(int) (n * 0.99)] / 1e6);
  }
}
EOF
java Ping.java http://localhost:8080/predict 2000

Against my local service, this Java client reported a median between 0.84 and 0.98 ms across three runs. That is the same order of magnitude as the Python client’s 0.63 ms, although its 99th percentile was higher, between 7.5 and 8.6 ms, on a single shared core.

When each design wins

Now the decision becomes practical. Compare the call overhead with the compute time of your model. When the overhead is a large share of the total, the network hop dominates, and in-process inference wins. When compute dominates, the hop barely matters, and a remote service becomes attractive for its other benefits. This is a rule of thumb, not a law, but it matches the trend in Table 2.

Favor in-process for small, hot models

Small models in a latency-sensitive path, such as scoring a request before it is handled, belong in the process. Moreover, there is no extra service to deploy, monitor, or secure. DJL helps here when you want to keep the option of switching engines later.

Favor a service for heavy or shared models

Large models, GPU workloads, and models used by many applications fit a service better. Memory is also a concern, since every replica of your application holds its own copy when the model runs in-process. My large test model alone was about 25 MB per instance. In addition, a service lets data scientists release new model versions without redeploying the Java application.

Decision checklist

  • Measure the call overhead of your own stack with a simple timer like Ping.
  • Compare it with the model’s compute time at your real batch size.
  • Prefer a compact binary payload when you call a service with batches.
  • Keep inference behind a small interface, so you can switch designs later.
  • Count the model copies you will hold in memory across all replicas.

What we have learned

In-process inference and remote model services are both valid, but they cost very different amounts per call. In my benchmark, a small model ran in about 0.007 ms in process, while the same model behind a JSON service needed about 0.3 ms, roughly 43 times more, even on loopback. The penalty shrinks as the model grows, down to about 3.3 times for a 25 MB model. The wire format mattered more than the network: raw bytes beat JSON by a wide margin, and at a batch of 128 the binary service came within about 1.5 times of in-process speed. Therefore, choose in-process for small, latency-critical models, and a service for heavy, GPU-bound, or widely shared ones. Above all, measure the overhead on your own stack before you decide.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button