The Latency Lie: How Ready Pods Were Quietly Costing a FinTech

Sep 5, 2026 cloud-native-architecture performance-engineering
Share on:

The Scene

A CTO of a small FinTech company had scheduled a meeting with me along with senior folks from his engineering team.

CTO : (shares screen) "Look at this. Every 1st and 15th of month - 9:00 to 9:30 PM - Our p99 latency spikes from 50 ms to 800 ms. After 4 mins - back to normal. Surprisingly, no 5xx errors, no pods crashing. Our APM says everything is green. But our enterprise customers? They're constantly asking why settlement APIs feels so 'sluggish.'"

Me : "1st and 15th - are these settlement days?"

CTO : "Bingo - Thousands of our customers hit our settlement APIs. HPA scales us from 10 pods to 40. The new pods come online. The load balancer sends them traffic. And somehow, for those first four minutes, the newest pods in the fleet are the slowest ones."

Me : "How do you know it's the new pods specifically?"

CTO : (switches to another dashboard) "Because I have been capturing metrics for last 2 months along with their request IDs. Look - requests landing on pods older than 5 minutes: 50ms p99. Requests landing on pods younger than 60 seconds: 780ms p99. Same code. Same image. Same node pool."

He sat back. Rubbed his eyes.

CTO : "Fix provided — We have kept a warm pool of 15 pods running 24/7. We never let the cluster drop below that. And our HPA scale-up stabilization window is so conservative that we're basically pre-paying for capacity we need maybe 4% of the time. Do you know what 15 idle pods cost us per month? Its 5,200 $ / month - and that too for avoiding a 4 min latency hiccup twice a month."

The call went quiet. Someone from the platform team unmuted, then muted again without speaking.

Me : "Show me your readiness probe."

CTO : "It's standard. HTTP GET on /health. Returns 200, pod is Ready, load balancer routes traffic. Why?"

Me : "Because I think your pods are lying to Kubernetes. They're saying 'I'm open for business', & that too before they've actually finished starting up. And that IMHO is the root cause of high latency.

CTO : (pause) "That's... specific. But why only do we see this behavior in newly added pods?"

Me : "That's my hypothesis. Let me prove it to you and thereby answer your above question!"

The Measurement

I don't recommend solutions until I've measured the behavior with actual data points. So I instrumented the service with two independent timestamps:

  1. T1 (serverStartedAt): The moment the embedded web server finishes starting and the port is accepting TCP connections. I captured this using a WebServerInitializedEvent
  2. T2 (initializationCompletedAt): The moment all business-logic initialization is actually done - captured via custom StartupMetrics
 1@Component
 2public class StartupMetrics {
 3    private volatile long serverStartedAt;
 4    private volatile long initializationCompletedAt;
 5    private final AtomicBoolean trulyReady = new AtomicBoolean(false);
 6
 7    @EventListener(WebServerInitializedEvent.class)
 8    public void onServerStarted(WebServerInitializedEvent event) {
 9        this.serverStartedAt = System.currentTimeMillis();
10        log.info("SERVER_STARTED:  port = {} timestamp = {}", 
11                 event.getWebServer().getPort(), this.serverStartedAt);
12    }
13
14    public void markInitializationComplete() {
15        this.initializationCompletedAt = System.currentTimeMillis();
16        this.trulyReady.set(true);
17        long gap = this.initializationCompletedAt - this.serverStartedAt;
18        log.info("INIT_COMPLETED : gapMs = {} serverStartedAt = {} initCompletedAt = {}", 
19                 gap, this.serverStartedAt, this.initializationCompletedAt);
20    }
21}

To ensure I was measuring cold starts, I ran 20 fresh container instances, not 20 restarts inside the same pod.

I compared medians across those 20 runs. The gap was consistent:

1Baseline (port-open vs. actually-ready): median 280.5 ms | stdev 23.2 ms

The 23.2 ms standard deviation was operationally not acceptable - as it was too high to tune initialDelaySeconds . Some pods were "almost ready" around ~260 ms. Others needed ~310 ms.

My first instinct was AppCDS. It's well-understood, zero code changes, with just two commands:

1  java -XX:ArchiveClassesAtExit=set-app-cds.jsa -jar set-app.jar
2  java -XX:SharedArchiveFile=set-app-cds.jsa -jar set-app.jar

I ran the same readiness-gap test. The result were:

1  AppCDS: median 280.5 ms | stdev 30.7 ms -> 0.0% improvement on this metric

AppCDS made overall JVM startup faster, but it did nothing for the specific window between port open and actually ready. My hypothesis - AppCDS's benefit (faster parsing of already-loaded JDK classes) is spent entirely during the JVM's early bootstrap phase, before the HTTP port ever opens. The remaining work after port-open is dominated by application logic and framework initialization.

The Mechanism

JDK 25 ships with Project Leyden's AOT cache. While AppCDS pre-parses classes into a shared archive, Leyden's AOT cache pre-loads and links them. The commands look like this:

1  java -XX:AOTCacheOutput=set-app.aot -jar set-app.jar   # training run
2  java -XX:AOTCache=set-app.aot -jar set-app.jar         # production run

When I ran the identical readiness-gap test:

1  Leyden AOT: median 254.0 ms | stdev 7.2 ms -> 9.4% improvement, far more consistent

The median improvement was not that substantial, but the standard deviation dropping from 23.2 ms to 7.2 ms was more valuable. A pod that starts in more predictable manner is always easy to tune from HPA standpoint

Training at Build Time

The AOT cache is only as good as the training run that produced it. If your training run exercises a different code path than production, you're caching the wrong thing. I worked with client's platform team to implement a multi-stage Docker build that performed the training run at image build time:

 1  # syntax=docker/dockerfile:1
 2  FROM eclipse-temurin:25-jdk AS build
 3  WORKDIR /app
 4  COPY . .
 5  RUN ./mvnw clean package -DskipTests
 6
 7  # Training run — exercises real startup paths once, during the build
 8  RUN java -XX:AOTCacheOutput=set-app.aot -jar target/set-app.jar --training-mode
 9
10  FROM eclipse-temurin:25-jre
11  WORKDIR /app
12  COPY --from=build /app/target/set-app.jar /app/set-app.aot ./
13  ENTRYPOINT ["java", "-XX:AOTCache=set-app.aot", "-jar", "set-app.jar"]

The --training-mode flag is a convention - your application should interpret it to mean "exercise your real startup code paths, then exit cleanly," so the cache captures genuinely representative behavior equivalent to that in production.

CI / CD Pipeline

 1# .github/workflows/build.yml
 2name: Build and Package
 3on:
 4  push:
 5    branches: [main]
 6
 7jobs:
 8  build:
 9    runs-on: ubuntu-latest
10    steps:
11      - uses: actions/checkout@v4
12
13      - uses: actions/setup-java@v4
14        with:
15          distribution: 'temurin'
16          java-version: '25'
17
18      - name: Compile and package
19        run: |
20          javac -d out $(find src -name "*.java")
21          jar cfe set-app.jar com.sti.Settlement -C out .          
22
23      - name: Build AOT cache from a representative training run
24        run: |
25                    java -XX:AOTCacheOutput=set-app.aot -jar set-app.jar --training-mode
26
27      - name: Verify the cache actually loads before shipping it
28        run: |
29          java -Xlog:aot=info -XX:AOTCache=set-app.aot -jar set-app.jar 2>&1 | \
30            grep -q "Opened AOT cache" || (echo "AOT cache failed to load — failing build" && exit 1)          
31
32      - name: Build and push container image
33        run: |
34          docker build -t <> .
35          docker push <>

The verification step is the one most real pipelines skip - it turns "the cache silently didn't load" from a production mystery into a failed build.

The Outcome

Key benefits of deploying realistically trained Leyden-enabled image:

  1. Warm pool reduction - Because new pods were consistent and predictably ready, client dropped their over-provisioned warm pool from 20 pods to 5. That alone saved huge amount in compute costs.
  2. Aggressive HPA tuning - With startup standard deviation dropping from ~23 ms to ~7 ms, they tightened their HPA scale-up stabilization window. The cluster scaled faster and more predictably during the settlement window.
  3. Latency stability - No p99 spikes during scale-up events. The latency gap was still there, but it was more predictable and no longer catching the load balancer off-guard.

Conclusion

The most important finding in this entire engagement wasn't that Leyden was faster. It was that AppCDS - the obvious, easy, zero-code-change answer didn't solve client's problem. Had I not adopted data driven approach of identifying hotspots - client would still be having P99 latency spikes on Settlement day along with wastage of few thousand of dollars every month on warm pool.

That's the standard I hold myself to

Measure the thing that hurts, not the thing that's easy to benchmark

P.S - If your platform is facing any Performance / Resiliency / Scalability / High Availability issues, reach out.

comments powered by Disqus