LLM orchestration is the layer that coordinates everything around a language model on a single request: which model and provider to call, how the prompt and context get assembled, when to invoke a tool, how state and memory carry across steps, and what to do on a retry or a failure. The model generates text. The orchestration layer decides what the model sees, what happens with what it returns, and how the whole thing behaves when something goes wrong.
What an orchestration layer actually handles
It helps to separate the model from the work around it. A raw model call takes some text in and gives some text back. In production that call sits inside a set of responsibilities that someone has to own. You either write that code yourself or you adopt a framework that provides it. Either way, these are the jobs that make up the orchestration layer.
Model and provider routing
Picking which model handles a given request, and which provider serves it. A cheap model for simple work, a stronger one for hard work, and a route to a second provider when the first is slow or down.
Prompt and context assembly
Building the actual input from system instructions, the user message, retrieved documents, and prior turns, then trimming it to fit the context window without dropping what matters.
Tool invocation
Reading a tool call from the model, running the function or API behind it, and feeding the result back so the model can continue. This is where most real work happens.
State and memory across steps
Carrying conversation history, intermediate results, and any working memory from one step to the next, so a multi-step request stays coherent.
Retries and fallbacks
Catching timeouts, rate limits, and malformed output, then retrying, backing off, or failing over to another model so one bad call does not take down the request.
Observability
Logging prompts, responses, token counts, latency, and cost per request. Without this you cannot debug a bad answer or explain a spend, so treat it as part of the layer, not an add-on.
None of this is exotic. It is the plumbing that turns one model call into a system you can run, debug, and pay for. The question is never whether you need these jobs done. It is whether you reach for a framework to do them or write the small amount of code yourself.
Orchestration vs an AI agent
These two terms get used as if they mean the same thing, and they do not. The distinction is about who decides the steps.
Orchestration is the coordination layer. You, the engineer, define the flow: call this model, assemble this context, run this tool, then format the answer. The steps are fixed in code. The layer carries out a plan you wrote.
An AI agent sits on top of that layer and decides the steps itself. It looks at a goal, picks the next action, reads the result, and decides what to do next, looping until it judges the task done. The agent still relies on orchestration underneath to make each call, run each tool, and keep state between steps. The difference is that the model chooses the path rather than following one you hard-coded.
The short version
- Orchestration coordinates a flow you defined. The control logic lives in your code.
- An agent decides the flow at runtime. The control logic lives in the model's reasoning.
- Every agent needs orchestration underneath it. Not every orchestration is an agent.
This matters when you scope a project. A fixed pipeline (classify, retrieve, answer) is orchestration and nothing more. If you genuinely need the model to choose its own steps, that is agent territory, and it brings more cost, more latency, and harder debugging. We cover that end of the spectrum in our work on AI agents and LLM products, where the decision of whether to give the model that freedom is one of the first things we settle.
When you need a framework, and when plain code is clearer
Here is the honest part. For a single prompt with no tools and no memory, you do not need an orchestration framework. You need an HTTP call to a provider, a try-catch, and a log line. Reaching for a framework at that point adds an abstraction you have to learn, debug through, and keep updated, in exchange for solving a problem you do not have yet.
Plain code is clearer when
The flow is one or two steps and you can read the whole path in a single file. You call a model, maybe run one tool, return the answer. A framework here hides logic that would otherwise be obvious. When something breaks, you want to step through your own function, not through layers of someone else's callbacks.
A framework starts to earn its place when
You are juggling several of the orchestration jobs at once across many steps: routing between providers, assembling context from retrieval, chaining tool calls, carrying state, and wanting consistent retries and tracing across all of it. At that point a framework gives you tested building blocks and a shared vocabulary, and writing it all yourself starts to look like rebuilding the framework badly. Choosing between supplying knowledge through retrieval or baking it into the model also shapes how much orchestration you need, which is the subject of our piece on RAG vs fine-tuning.
The trap is adopting a framework on day one because the demos look tidy, then spending week three fighting its abstractions to do something a plain function would have done in ten lines. Start with the smallest thing that works and let the complexity pull you towards a framework, rather than the framework pushing complexity onto you.
Build vs framework: the tradeoffs
When the work does grow past a single call, the real decision is build your own thin orchestration or adopt an existing one. Both are defensible. The choice turns on how much of the layer you need and how much you want to own.
Building your own keeps the system small and legible. You add only the jobs you actually use, the code reads top to bottom, and there is no third-party release cycle to track. The cost is that you own every edge case: the retry logic, the token trimming, the provider quirks. For a focused product with two or three providers and a known flow, that ownership is often cheaper than it looks.
Adopting a framework gives you those edge cases pre-solved and a path others have walked. The cost is a dependency that moves on its own schedule, an abstraction layer between you and the model, and the risk that your needs drift away from what the framework assumes. Frameworks also tend to be young and churn fast, so factor in the upgrade tax.
On the British Council's AiBC platform we ran multi-provider LLM orchestration in production, and the routing, fallback, and observability mattered far more than any single clever prompt. You can read how that system was built in the British Council AiBC case study. The lesson that carried over was to keep the orchestration thin and explicit, and to add a piece only when a concrete need asked for it.
A decision guide for when to reach for orchestration
If you are deciding how much orchestration a project needs, work down this list. The further you go, the stronger the case for a dedicated layer, and eventually for a framework.
When to reach for orchestration
- One prompt, no tools, no memory: plain code. A provider call, error handling, and logging. Skip the framework.
- A fixed two or three step flow with one tool: a small orchestration layer you write yourself, still readable in one file.
- Multiple providers, retrieval, chained tools, and shared state: a real orchestration layer, built thin or adopted, with retries and tracing across all of it.
- The model needs to choose its own steps towards a goal: agent territory, which assumes solid orchestration underneath before you start.
- Across all of these: log prompts, tokens, latency, and cost from the first request. Observability is the part teams skip and regret.
The rule that has held up for us: name the orchestration jobs you actually have, do the smallest thing that covers them, and let real need, not the tidiness of a demo, decide when to add the next piece.