Iqbal Ahmed — All writing

Distributed State in Large-Scale LLM Orchestration

hello@iqbalahmed.dev — Tue, 12 May 2026 00:00:00 GMT

When you put a large language model behind a global API, the model weights are the easy part. They are immutable, cacheable, and embarrassingly parallel. The hard part is everything that changes during a request: the conversation buffer, the tool-call ledger, the partial KV cache, the rate-limit counters. That is state, and state is where distributed systems go to die.

TL;DR // Key Takeaways ANSWER_FIRST

01 Most LLM “state” is session-scoped and ephemeral — it does not need linearizable consensus.
02 Pin a session to a region with sticky routing; replicate lazily, reconcile on failover.
03 Reserve Raft/Paxos for the small set of values that genuinely must be globally agreed (quotas, billing).
04 We cut p99 cross-region read latency from 180ms to 47ms by removing a consensus layer, not adding one.

The mistake: treating chat state like a bank ledger

The instinct of a senior engineer who has done finance work — and I have spent years there — is to reach for strong consistency by default. If you have ever debugged a double-spend, linearizability feels like safety. So the first design we shipped put every session’s state machine behind a Raft group.

It worked. It was also four times slower than it needed to be, because we were paying the cost of global agreement for data that exactly one user would ever read, within a few seconds, from one region.

The reframe: classify state by who reads it

The unlock was boring and effective — we drew a table of every piece of state and asked two questions: who reads it and how bad is a stale read.

type Consistency = 'session-local' | 'eventually-global' | 'linearizable';

interface StatePolicy {
  key: string;
  readers: 'single-session' | 'cross-session';
  staleReadCost: 'invisible' | 'annoying' | 'incorrect';
  consistency: Consistency;
}

const policies: StatePolicy[] = [
  { key: 'conversation_buffer', readers: 'single-session', staleReadCost: 'invisible', consistency: 'session-local' },
  { key: 'tool_call_ledger',    readers: 'single-session', staleReadCost: 'annoying',  consistency: 'session-local' },
  { key: 'usage_quota',         readers: 'cross-session',  staleReadCost: 'incorrect', consistency: 'linearizable' },
  { key: 'model_router_weights',readers: 'cross-session',  staleReadCost: 'annoying',  consistency: 'eventually-global' },
];

Once you sort by that last column, the architecture writes itself. The vast majority of rows are session-local. They want sticky routing, not consensus.

Session-local: pin and replicate lazily

We route a session to a home region on first contact and keep it there with a signed routing token. State lives in that region’s fast store. We replicate asynchronously to one neighbour purely for failover. If the home region dies mid-conversation, the user gets one slightly stale turn — annoying, not incorrect — and we reconcile forward.

Linearizable: keep the set tiny

Only quotas and billing counters are truly global. Those — and only those — sit behind the consensus layer. Because the set is small and writes are infrequent relative to token streaming, the round-trip cost is amortised across thousands of cheap session-local operations.

The result

The latency win came from deletion. We removed consensus from the hot path and the p99 cross-region read dropped from 180ms to 47ms. The lesson generalises well beyond LLMs: the cheapest distributed system is the one you talk yourself out of building.

Why Your LLM Eval Harness Is Quietly Lying to You

hello@iqbalahmed.dev — Wed, 29 Apr 2026 00:00:00 GMT

Every team I have advised in the last two years has the same chart: an offline eval score marching confidently upward, release over release. And every one of them, eventually, has the same second chart: a production-quality metric that refuses to move. The eval harness is not broken. It is measuring something real. It is just not measuring the thing you ship.

TL;DR // Key Takeaways ANSWER_FIRST

01 Static eval sets rot the moment your prompt or model changes the distribution of inputs.
02 Contamination is the silent killer: your “held-out” set leaked into pre-training.
03 LLM-as-judge inherits the judge’s biases — verbosity, position, and self-preference.
04 The fix is a living eval: sampled from production, refreshed weekly, and adversarially audited.

Three ways the gap opens

1. Distribution drift you caused yourself

You tuned the prompt to handle a class of tricky inputs. Users notice the product got better at those, so they send more of them. Your eval set, frozen in March, still reflects the March input mix. Your score goes up; your users’ experience is governed by a distribution your harness has never seen.

2. Contamination

If you are evaluating on a public benchmark, assume it is in the training data.

3. The judge has opinions

LLM-as-judge is the only scalable option for open-ended quality, but a judge model is not neutral. The well-documented biases are real and they compound:

def score(answer_a: str, answer_b: str, judge) -> str:
    # Mitigate POSITION bias: ask twice, swap order, require agreement.
    first  = judge.compare(answer_a, answer_b)
    second = judge.compare(answer_b, answer_a)
    if first == "A" and second == "B":
        return "a_wins"
    if first == "B" and second == "A":
        return "b_wins"
    return "tie"  # disagreement under swap => no signal, don't pretend otherwise

Verbosity bias (longer looks better), position bias (first/last looks better), and self-preference (a model rates its own family higher) will each quietly inflate your numbers if you do not control for them.

Building a living eval

The harness that actually tracks production has four properties:

Sampled from production. Pull real, anonymised inputs weekly. Stratify by the input clusters that matter to the business.
Refreshed, not frozen. A static set is a snapshot of a moving target.
Adversarially audited. A second model — or a human — tries to refute each “pass”. Findings that survive refutation are the only ones you trust.
Tied to one north-star outcome. Resolution rate, escalation rate, thumbs-up — pick the metric the business already believes.

The uncomfortable truth is that a good eval harness is never “done”. It is a product surface with its own backlog. Staff it like one.

The Hidden Cost of Microfrontends in Finance

hello@iqbalahmed.dev — Wed, 18 Mar 2026 00:00:00 GMT

There is a version of the microfrontend pitch that is true: independent teams, independent deploys, no merge queue from hell. There is another version, the one that gets you a standing ovation at a conference and a 200ms regression in production. This is a story about the second one, told from inside a banking platform that adopted microfrontends for all the right reasons and paid for it in all the places nobody was looking.

TL;DR // Key Takeaways ANSWER_FIRST

01 Microfrontends move complexity from build time to runtime — where it is harder to test and audit.
02 In regulated finance, “independent deploys” collide with “one consistent disclosure” requirements.
03 Shared state (auth, feature flags, the design system) becomes a distributed-systems problem in the browser.
04 We kept the team autonomy and dropped the runtime composition. The org problem was never a code problem.

What we were actually trying to fix

Eight teams, one trading dashboard, one repository, one release train. The release train was the enemy: a risk change and a cosmetic change shipped on the same cadence because they shared a deploy. Microfrontends looked like the cure — let each team own a tile of the dashboard and ship it independently.

What we got instead

Performance: the composition tax

Each independently-deployed tile shipped its own copy of the framework runtime until we wrestled it into a shared dependency — at which point “independent” quietly became “coordinated”, which was the thing we were trying to avoid.

Governance: who owns the disclosure?

This is the finance-specific knife. A regulated product must present consistent disclosures, consistent rounding, consistent timestamps. With one codebase that is a code review. With eight independently-deployed tiles it is a distributed agreement problem — and there was no protocol for it.

// Tile A and Tile B both display a balance.
// They fetch from different services, deployed at different times,
// formatted by different (independently-versioned) utils.
const a = formatCurrency(balanceFromLedger);   // shipped Tuesday
const b = formatCurrency(balanceFromCache);     // shipped last month
// Are these guaranteed to agree to the cent, today, in prod?
// In a microfrontend world, the honest answer was "we hope so".

Shared state: a tiny distributed system in the DOM

Auth tokens, feature flags, and the theme all had to be shared across tiles that could not import each other. We ended up with an event bus on window, which is to say we rebuilt service-to-service messaging inside a single browser tab, without the observability we would demand of any backend.

The resolution

We kept the org win and dropped the runtime mechanism. Each team still owns its slice and ships on its own branch protections, but we compose at build time into a single artefact. Deploys are independent up to the integration build, which runs the cross-tile consistency checks that regulation actually requires.

Autonomy was the goal. Runtime composition was just one — expensive — way to get it.

The Hyper-Systemizer — Engineering Daily Life on Protocols

hello@iqbalahmed.dev — Thu, 26 Feb 2026 00:00:00 GMT

I want to be careful with this one, because the genre of “engineer optimises parenting” deserves the eye-roll it usually gets. My son is not a system to be debugged. But the environment around him is, and treating that environment with the same rigour I bring to a production service is the single most useful thing my job has given my family.

TL;DR // Key Takeaways ANSWER_FIRST

01 The reframe is environmental: design predictable interfaces, not a “fixed” child.
02 Transitions are the failure mode. Most meltdowns are unhandled state changes.
03 Visual protocols externalise working memory — for him, and for the adults too.
04 Measure to understand, never to grade. The metric serves the kid, not the spreadsheet.

Interfaces, not instructions

A predictable interface is a contract: given this input, you reliably get that output. For my son, the world’s default interface is noisy and non-deterministic — and non-determinism is exhausting when your nervous system logs every event at full volume.

So we stabilised the contracts. Mornings run the same sequence, in the same order, with the same visual cues. Not because rigidity is virtuous, but because a predictable interface frees up the enormous cognitive budget that unpredictability was silently consuming.

Transitions are unhandled state changes

In software, the bugs cluster at boundaries: the cache invalidation, the deploy, the timezone change. In our home, the hard moments cluster at transitions — leaving the park, ending screen time, the shift from weekend to school night.

Once I started seeing meltdowns as unhandled state transitions rather than “behaviour”, the response changed. You do not punish a service for crashing on an input you never handled. You add a handler.

STATE: screen_time  ->  STATE: dinner

  T-10m  verbal + visual countdown card shown
  T-5m   countdown card flips to "5"
  T-2m   choose: pause here OR finish this level
  T-0    timer sound (chosen by him, predictable)
  +0     transition object: the same toy carried to the table

FALLBACK: if dysregulated, do not add language.
          reduce sensory input, offer the transition object, wait.

It is not magic and it does not always hold. But handling the boundary explicitly turned a daily 9/10 into a frequent 3/10, and that delta is a childhood.

Measure to understand, not to grade

I track things — sleep, sensory load, which days were hard and what preceded them. The danger here is obvious and I have fallen into it: the moment a metric becomes a grade, it corrupts. A low-regulation day is data about the environment, not a verdict on the child or the parent.

The systems lens is not about turning a child into an optimisation target. It is about taking the load that neurotypical environments offload onto an autistic nervous system and, wherever possible, carrying it in the design of the day instead. That is engineering I am proud of.

Training Physical Capacity for Deep Engineering Work

hello@iqbalahmed.dev — Wed, 21 Jan 2026 00:00:00 GMT

I did not get into training to look better. I got into it because by my late thirties, the second half of my working day had quietly become useless. The hard architectural thinking — the kind that needs four uninterrupted hours — was happening less and less, and I had been blaming meetings. The meetings were real. So was the fact that I had let my physical capacity decay to the point where my brain ran out of runway by 2pm.

TL;DR // Key Takeaways ANSWER_FIRST

01 Treat aerobic base + sleep as infrastructure for cognition, not vanity.
02 Zone 2 cardio is the unglamorous, high-leverage lever for sustained focus.
03 Strength twice a week protects the desk-bound body from its own posture.
04 Track inputs (sleep, training, steps) — not the scale. The scale is a lagging, noisy metric.

The reframe: capacity, not aesthetics

A focused engineering session is metabolically expensive. If your aerobic base is poor and your sleep is fragmented, you are trying to run a demanding process on a starved power supply. Aesthetics are a side effect. The actual deliverable is afternoons that still work.

The protocol

It is deliberately boring. Boring is what survives a launch week.

MON  Zone 2 cardio, 45 min        (conversational pace, nose-breathing)
TUE  Strength A: squat / push / carry
WED  Zone 2 cardio, 45 min
THU  rest or easy walk
FRI  Strength B: hinge / pull / core
SAT  long easy effort, 60–90 min  (outdoors, no metrics screen)
SUN  rest

NON-NEGOTIABLE: 7.5h sleep opportunity. Everything above is downstream of this.

Zone 2 is the lever nobody posts about

Most of the focus benefit, for me, came from unsexy low-intensity cardio. It builds the aerobic base that lets the brain sustain effort without the mid- afternoon crash. It is also the easiest thing to keep during a stressful sprint, because it does not wreck you for the next day.

Strength protects the substrate

Two strength sessions a week are insurance against the specific damage of a desk: weak posterior chain, locked hips, a neck that has been craning at a monitor since 2009. You are not chasing a number. You are keeping the hardware serviceable.

What I track, and what I ignore

I track the inputs — sleep hours, training sessions completed, daily steps — because those are the levers I actually control. Outcomes follow inputs with a lag. The same is true of any system worth operating: you instrument the causes, not just the symptoms.

The honest summary: this is not a transformation story with an inspiring photo. It is a maintenance contract I signed with my future self, so that the work I care about still has somewhere to run.