Most Agent Problems Are Actually Process Problems

What building infrastructure-native AI taught us about the difference between a conversation and a process.

The most consequential architecture decision we made building Nora, Inorsa’s AI interface, was a decision not to build something  

We didn’t build a chatbot. 

That isn’t a criticism of conversational AI. Chat is the right interface for a huge class of problems, and the agent loop of thinking, calling a tool, observing the result, and thinking again is one of the most important advances in AI application development over the past few years. But within the first few weeks of building AI for critical infrastructure, we ran into a different class of problem.  

The work we were automating didn’t have turns. It had stages. It had checkpoints. It had a licensed engineer who needed to sign off on Tuesday for something the system produced on Monday. 

The orchestration threshold 

As we built Nora, we started noticing the same pattern across more and more workflows. 

There’s a point where an AI problem stops being a conversation and becomes a process. We’ve started calling it the orchestration threshold, because naming the pattern turned out to be useful. Once you can recognize it, you stop trying to solve process problems with conversation tools. 

Three characteristics usually appear together.  

The work outlives the session. A telecom deliverable package is a dense stack of engineering drawings, technical reports, and equipment schedules. Dozens of documents for a single site, running several minutes each end to end. Getting through one deliverable generation isn’t a long answer. It’s a pipeline that has to survive a restart. 

Correctness depends on reconciliation, not on a good response. Equipment listed in a data sheet has to match equipment listed in the associated engineering analyses, construction drawings, leases, and related artifacts. Telecom carriers like AT&T, T-Mobile, and Verizon each publish their own documentation standards and validation rules, and the system has to apply the right one automatically. It isn’t being asked what it thinks. It’s being asked to prove that two artifacts agree, and to say precisely where they don’t. 

A human has to interrupt it in the middle, not at the end. Certain engineering determinations legally require a licensed engineer’s sign-off before anything downstream can proceed. Not review-after-the-fact. Review-as-a-stage. 

Where the threshold isn’t 

If the work fits inside one sitting, an agent loop is almost always the better tool. A support-triage assistant, a research helper, a coding agent working through a task: these finish while someone is watching, and if they go wrong the human simply asks again. That retry loop is the error handling. Wrapping it in a state machine buys durability nobody needed. 

What tips a problem over the threshold is the combination of all three signals. Any one of them is survivable inside an agent loop if you’re willing to build scaffolding around it. All three at once means the scaffolding is the product, and you should stop pretending otherwise. 

“We didn’t need an agent that could hold a conversation. We needed one that could hold state, reliably, across a process that might pause for a human reviewer and pick back up an hour later exactly where it left off.”
— Xinning Wang, Principal Architect, Inorsa 

That’s why we built Nora’s validation workflow as a LangGraph state machine rather than a top-level agent loop. The framing shift is small and the consequences are not: the source of truth is workflow state, not conversation history. 

A deterministic spine with ephemeral limbs 

The shape that emerged looks less like an agent and more like a factory floor. 

The parent graph is a hand-built LangGraph StateGraph with 23 parent nodes and four compiled subgraphs mounted directly as nodes inside it. It’s deterministic: ingest, verify, classify, extract, reconcile, validate, render, in that order, every time. We can draw it on a whiteboard and a new engineer can follow it. That predictability isn’t a limitation we accepted reluctantly. It’s the thing that makes the system auditable, and auditability is the product in this domain. 

The intelligence lives in the limbs. Several of those stages fan out dynamically at runtime using LangGraph’s Send() API to spin up one worker per document or per section and run them in parallel. Each worker is its own small ReAct loop rather than a single model call: it calls tools, hands its output to an evaluator, and routes to an optimizer if the evaluator isn’t satisfied. A document extraction worker classifies content, extracts structured fields, and can delegate to a dedicated vision subgraph when a page needs a different kind of understanding. A scanned drawing is not a text extraction problem. 

So a single run is one long-lived state machine coordinating what can be dozens of short-lived agents underneath it. Not one big brain. A lot of small, focused ones, inside a structure that remembers. 

Every node’s output is checkpointed to Postgres and every worker’s reasoning is traceable independently. When something goes wrong, we don’t reconstruct intent from a transcript. We look at the state. 

Human review is a state, not an exception 

The most elegant simplification in the system came from refusing to treat human review as an integration problem. 

When the workflow reaches a determination that needs a licensed engineer, the graph pauses using LangGraph’s interrupt mechanism and hands control to a reviewer. There is no separate approval service, no side database of pending tasks, no polling. The interrupt and its eventual resume live in the same thread state as everything else the graph is doing. 

The practical effect on the team building the review interface was larger than we expected. The frontend reads the interrupt payload out of the thread, renders whatever it describes, and resumes the run with the reviewer’s response. That single contract covers both a one-click approval and a multi-field engineering form: the graph describes what it needs, the UI renders it, the state carries the answer back. 

Getting the contract right wasn’t free. An early version of the review loop raced a static edge against an explicit end command: both fired, neither won, and the graph spun in place instead of terminating. The fix reinforced the design rather than undermining it. Route on state, not on parallel edges that each assume they’re the only one firing. 

Observability found the problem before production did 

The most instructive incident in Nora’s history started as a tracing complaint and ended as an architecture decision. 

LangSmith caps trace payloads at 25MB, and we started blowing past it. Loaded documents and extracted chunks meant some runs were carrying 30MB or more of state. The first instinct was to treat it as an observability constraint and disable tracing on the noisy parts of the pipeline. 

That was the wrong read. The same bloated state was being written into every LangGraph checkpoint, and Postgres checkpoint writes at that size were destabilizing production. The observability layer had flagged the failure mode well before customers would have felt it. We just hadn’t connected the two signals yet. 

The fix was structural: strip evidence and image payloads out of checkpointed state, offload them to external storage, cut a pass-through node that existed only to shuttle data between steps. Typical checkpoint sizes dropped by more than half. 

That held until concurrency scaled further and connection-pool pressure on the LangGraph deployment started surfacing as intermittent timeouts. Intermittent is the worst failure mode there is: you can’t reproduce it on demand and you can’t ignore it either. Inorsa and LangChain worked through it together over several days, trading traces and logs in real time. LangChain tracked down and fixed a resource-management issue on the platform side while we kept slimming our own state. 

Scaling an agentic pipeline is as much an infrastructure problem as a graph-design one. Building production AI systems is rarely just about prompts or models. 

Observability is not a monitoring feature. It’s an architectural instrument. It tells you what your system is actually carrying, which is rarely what you designed it to carry. 

Evals run the graph that ships 

Nora’s evaluation suite runs the same compiled graph that serves production. It just interrupts it early. 

For extraction-quality checks, the graph runs with an in-memory checkpointer and interrupts immediately before section validation begins, so every upstream stage is exercised exactly as it would be in a live run. A separate LangSmith dataset scores full end-to-end reports against human-validated ground truth using an LLM-as-judge rubric with five distinct error categories: omission, hallucination, commission, misinterpretation, source misattribution, each weighted differently depending on how costly that kind of mistake is for a telecom QA team to miss. 

We run the same judge through two independent harnesses, which has caught scoring inconsistencies neither surfaced on its own. Nobody writes a blog post about their CI pipeline. But it’s the reason changes to a 23-node graph can ship with confidence instead of hope. 

What to take from this 

Don’t start from the model. Start from three questions about the work itself. 

Does it outlive a session? If the process can pause, restart, or run longer than anyone will sit and watch, durable state is a first-class design concern, not scaffolding you bolt on later. 

Is correctness a matter of reconciliation? If the output has to be consistent across many artifacts rather than merely plausible in isolation, you need explicit stages and explicit checks. “The model usually gets it right” is not a validation strategy. 

Does a human belong in the middle? If so, model the pause as a state the workflow occupies, not as an external system the workflow calls out to. The architecture gets simpler, not more complex, when the interrupt lives inside the graph. 

Telecom infrastructure is not a glamorous place to build AI products, and that’s most of the point. The documents are dense, the validation rules are unforgiving, and there is no shortcut around getting a rigorous engineering determination right. Those constraints don’t reward clever prompting. They reward state machines instead of prompts, checkpoints instead of chat history, and evals that exercise the same graph that ships. 

Everyone is building agents that can think. The harder problem, the one this domain forced on us, was building one that could wait. 

We’re hiring

 

If the shape of this problem is the kind of thing you’d rather build than read about, we’re hiring. The work is unglamorous in exactly the ways described above: dense documents, unforgiving validation rules, and infrastructure problems that surface at the worst possible time. We think that’s a feature. 

Careers Page