When signals disagree
Investigate the measurement boundary.
A reassuring aggregate should not erase a concrete failure. First ask whether the instruments answer the same question.
Senior ML Engineer · Microsoft
Mrityunjay Kumar
I build ML systems that remain useful after the demo.
My work is moving from solving hard model problems to shaping the systems around them: product intent, evaluation, reliability, cost, and the engineering leverage that lets teams improve all five together.
Read my operating approach- 01Product intentThe decision or workflow that must improve
- 02EvaluationEvidence that separates progress from theatre
- 03ML executionModels, data, tools, and bounded autonomy
- 04OperationsReliability, latency, cost, and recovery
- 05Team leverageInterfaces and paved paths that compound
Outcome A useful capability that can be measured, operated, and improved.
01 / What I care about
Capability matters. Durable value matters more.
A model can be impressive and still leave the product fragile. I care about the whole system that turns intelligence into a dependable user outcome, including the parts that are less visible than the model itself.
- 01
Useful over theatrical
Start with the decision or workflow that should improve, not with a model looking for a use case.
- 02
Reliability is product work
Evaluation, fallbacks, latency, observability, and recovery shape trust as directly as model quality.
- 03
Economics are architecture
Token use, compute, human review, and operational load belong in design decisions from the beginning.
- 04
Clarity creates leverage
Explicit interfaces, ownership, and evidence let more engineers change the system without losing its intent.
02 / How I think
Make the system legible before making it larger.
Staff-level judgment begins before implementation. I frame the decision, map the boundaries, and identify the evidence that would justify investment or stop it.
- Intent
Whose outcome changes, and how will we know?
- System
Where do state, data, models, and ownership meet?
- Risk
Which failure is costly, silent, or hard to reverse?
- Evidence
What is the cheapest observation that could change the decision?
- Commit
Build, narrow, or stop with the rationale recorded.
When investment grows
Raise the evidence bar.
The more expensive and irreversible the direction, the more explicit its assumptions, rollback, and success criteria should become.
When a hypothesis fails
Change the plan, not the story.
A negative result is useful when it prevents further waste. Technical leadership includes stopping work that has not earned scale.
03 / How I develop
Move from uncertain capability to an operable system.
I work in thin, observable slices. Each stage reduces a different risk, and each leaves behind an interface or check the next engineer can use.
- 01
Frame
Define the outcome and invariants.
Turn an ambiguous request into a decision, a baseline, and constraints that survive implementation choices.
- 02
Slice
Prove the riskiest path end to end.
Connect real data, model behavior, and user interaction before optimizing isolated components.
- 03
Instrument
Make quality and failure observable.
Capture task success, regressions, cost, latency, and traces at the boundary where decisions are made.
- 04
Harden
Design for imperfect models and dependencies.
Add validation, retries only where bounded, fallbacks, idempotency, isolation, and explicit unknown states.
- 05
Operationalize
Turn the solution into team leverage.
Ship paved paths, ownership, runbooks, release gates, and telemetry so improvement no longer depends on one person.
04 / Scaling in the LLM era
Scale the learning system, not only the inference fleet.
LLM products change faster than traditional software and fail less deterministically. Sustainable scale comes from shortening the path from production evidence to a safe, measurable improvement.
01
Intent
Task contracts
Define success in user and domain terms.
02
Evaluation
Evidence loops
Pair offline suites with production signals.
03
Execution
Bounded autonomy
Constrain tools, state, retries, and side effects.
04
Operations
Observable economics
Track quality, latency, tokens, and recovery together.
05
Organization
Paved roads
Make safe patterns easier to reuse than bypass.
The evolution I am pursuing
From owning components to improving how teams make technical decisions.
The staff role I am growing toward is not “more implementation.” It is broader judgment: finding the constraint that matters, aligning interfaces and ownership, creating reusable engineering paths, and making trade-offs visible early enough for the organization to act on them.
05 / Three studies
Evidence changed the conclusion each time.
The studies are useful together because they do not tell one convenient story. One establishes a bounded condition, one exposes an audit boundary, and one shows when an expensive technical direction has not earned another run.
Study A · Geometric contracts
Bounded positive result
What can an item-local rule actually certify about flexbox?
Finite models, exact arithmetic, and browser perturbation across real pages.
Inspect the TLA+ model- Corpus
- 1,200 rendered pages
- Observed ceiling
- 45.76% no coupling detected
- Certified
- 28.07% by the local predicate
- Decision
- Separate what finite probes observed from what the predicate can establish.
- Finding
- The predicate is sound and maximal on eight checked finite domains, not for arbitrary CSS. The 17.69-point corpus gap remains an open information-boundary question.
- Staff lesson
- Choose the abstraction boundary explicitly. A precise bounded claim is more useful than a broad guarantee the evidence cannot carry.
Study B · Disclosure contracts
Single-repository finding
Can a policy account for what a build actually publishes?
An abstract disclosure model paired with inspection of emitted files.
- Sinks
- 12 declared outputs
- Flows
- 11 source-to-sink edges
- Gaps
- 2 audit defects exposed
- Decision
- Check both policy coherence and emitted bytes; neither side alone establishes publication safety.
- Finding
- A reversible representation bypassed a literal check, and two published PDFs sat outside the prior audit set. The case establishes existence, not prevalence.
- Staff lesson
- System boundaries fail where ownership and inventories drift. Build the handoff between abstractions into the operating path.
Study C · Generated formal feedback
Negative feasibility result
Does TLA+-guided repair improve an LLM-generated web artifact?
One paid task, generated behavioral contracts, browser replay, and bounded repair.
- Provider requests
- 13 no retries
- Recorded tokens
- 349,968 input plus output
- Formal score
- None blocked before repair
- Decision
- Compare the formal path with a serious non-formal search before spending on another generated run.
- Finding
- The fresh baseline received 58/100, while the TLA+-guided run remained unscored. Offline, ordinary depth-12 search matched the synthetic TLC witness with the same 12 browser actions.
- Staff lesson
- Complexity must earn its place. Stop an expensive direction when its incremental value remains unestablished, and preserve the negative result.
What changed in how I operate
From model quality to system quality. From passing checks to understanding their coverage. From implementing the next idea to deciding whether it deserves organizational scale.