The Problem With Black-Box Orchestration (And What Transparent Routing Actually Looks Like)
Why orchestration you can't see, audit, or steer is just a fancier single model
The Promise Is Real. The Problem Comes Later.
The pitch for black-box multi-model orchestration is genuinely compelling. Send one request. The system figures out which AI model is best suited for each part of the task, routes accordingly, and returns a result that's better than any single model would have produced. You don't have to think about model selection, context management, or task decomposition. It just works.
And for a while, it does. The first few times you use a system like this, the results feel like magic. You get outputs that are noticeably better than what you'd get from a single model, without any of the overhead of managing the routing yourself. The abstraction is clean. The integration is simple. The benchmark numbers are impressive.
The problem doesn't show up on day one. It shows up at scale — when you're running dozens of agent tasks a day, when a result comes back wrong and you need to understand why, when your API bill is higher than expected and you can't figure out which tasks drove the cost, when you need to demonstrate to a stakeholder that the system made the right call. That's when the black box stops feeling like magic and starts feeling like a liability.
This is a category-level argument, not a product takedown. The same dynamics apply to any orchestration system that hides its routing decisions from the user. The question is: what does transparent orchestration actually require, and why does it matter more as your workloads scale?
What Black-Box Orchestration Looks Like in Practice
A black-box orchestration system has a simple interface: input goes in, output comes out. The routing decisions — which model handled which sub-task, why, at what cost, with what context — happen inside the system and are not exposed to the user.
Some systems give you partial controls. You might be able to opt out of specific model providers. You might be able to set a preference for speed vs. quality. But you cannot see the execution plan before it runs, you cannot inspect the routing decisions after it runs, and you cannot attribute cost to specific sub-tasks.
Sakana AI's Fugu is a current example of this architecture. Its TRINITY system (Thinker, Worker, Verifier roles) and Conductor routing layer are technically sophisticated — and the benchmark results on coding tasks are strong, with Fugu Ultra scoring 73.7 on SWE-Bench Pro and 93.2 on LiveCodeBench. But the routing is opaque. You can opt out of specific providers, but you cannot see which model handled which role in a given task, why the Conductor made that assignment, or what each step cost. The system is designed to be a drop-in API endpoint, and that design prioritizes simplicity over visibility.
Fugu is not alone in this. Many orchestration products — including others in the emerging "AI agent API" category — make the same tradeoff: hide the complexity, surface the result, let the user trust the system.
What You Lose When You Can't See Inside
Cost Control
When multiple models run in parallel or in sequence on a single task, the cost is not simply the sum of individual model calls. Orchestration layers add overhead, and billing rules for multi-agent stacking can be complex and poorly documented. If you can't attribute cost to specific sub-tasks, you can't identify which tasks are expensive, which routing decisions are driving your bill, or where to optimize.
At small scale, this is a minor inconvenience. At scale — dozens of tasks per day, multiple projects running in parallel — it becomes a real budget problem. You're flying blind on cost, and the only lever you have is turning the whole system off.
Debugging
When a black-box orchestration system produces a wrong result, your debugging options are limited. You have the input, you have the output, and you have nothing in between. You don't know which model handled which part of the task, what context it was given, what intermediate outputs were produced, or where the reasoning went wrong.
This is not a hypothetical problem. AI models make mistakes. Orchestration systems make routing mistakes. When they do, the ability to inspect the execution trace is the difference between a five-minute fix and a multi-hour investigation — or, worse, a wrong result that you don't catch because you have no way to audit the reasoning.
Trust and Auditability
For many teams, the ability to audit AI decisions is not optional. Regulated industries, enterprise environments, and any context where AI outputs affect real decisions require an audit trail. "The model said so" is not an acceptable answer when a stakeholder asks why a particular decision was made.
Black-box orchestration makes auditability structurally impossible. If you don't know which model made which decision, you can't explain the reasoning, you can't verify it against your standards, and you can't demonstrate compliance. The system is a black box not just to you but to anyone who needs to review its outputs.
The Ability to Improve
Perhaps the most underappreciated cost of black-box orchestration is that you can't improve it. If you can't see the routing decisions, you can't identify which ones are suboptimal. If you can't attribute cost to sub-tasks, you can't optimize the routing for cost efficiency. If you can't inspect the execution trace, you can't build better prompts, better context, or better task decompositions.
A system you can't see is a system you can't steer. Over time, that means you're permanently dependent on the vendor's routing decisions — with no ability to tune the system to your specific workload, your team's standards, or your cost constraints.
Four Things Transparent Orchestration Actually Requires
Transparency in orchestration is not just about showing a log after the fact. It requires four structural elements:
1. A Visible Execution Plan (DAG)
Before execution starts, you should be able to see the full task decomposition: every sub-task, every agent assignment, every dependency. This is typically represented as a directed acyclic graph (DAG) — a visual map of the work to be done.
The DAG serves two purposes. First, it lets you catch problems before they happen — if the system has decomposed your goal incorrectly, you can fix it before any model calls are made. Second, it gives you a reference point for understanding the output — you know what the system was trying to do at each step, which makes the result interpretable.
2. Per-Task Cost Receipts
Every sub-task should produce a cost receipt: which model was used, how many tokens were consumed, and what the cost was. This is not just a billing feature — it is a debugging and optimization feature. When you can see that a particular sub-task is consuming 40% of your budget, you can make an informed decision about whether to use a cheaper model for that step, simplify the prompt, or restructure the task decomposition.
Per-task cost attribution also makes it possible to compare routing strategies. If you want to know whether routing a particular type of task to a smaller model produces acceptable results at lower cost, you need per-task data to answer that question.
3. A Decision Log
Every routing decision — which model was assigned to which sub-task, why, and with what context — should be logged and accessible. This is the audit trail that makes the system explainable, debuggable, and improvable.
A decision log is also the foundation for learning. If you can see which routing decisions produced good results and which produced bad ones, you can build a feedback loop that improves the system over time. Without a decision log, there is no feedback loop — just a stream of inputs and outputs with no connective tissue.
4. A Trust-Compounding Loop
The most powerful form of transparency is not just visibility into what happened — it is the ability to influence what happens next. A trust-compounding loop records your approvals and rejections with context, learns from them, and applies that learning forward.
This is how orchestration systems should get better over time. Not by the vendor updating the routing model in the background, but by the system learning your specific preferences, your team's standards, and your project's constraints — and applying that knowledge to reduce the decisions you need to make manually.
In week one, you review most decisions. By week six, the system handles routine decisions automatically and surfaces only the genuinely novel ones. The system earns autonomy through demonstrated alignment with your judgment.
What Transparent Routing Looks Like in Practice
Medley is built around all four of these requirements.
When you describe a goal — a "mission" — Medley decomposes it into a visible, editable DAG before execution starts. You can see every sub-task, every agent assignment, every dependency. You can edit the plan, reassign sub-tasks to different agents, or add steps before anything runs.
During execution, every sub-task produces a cost receipt. You can see what each step cost, which model handled it, and what context it was given. After execution, the full decision log is available for inspection and audit.
And every approval or rejection you make is recorded with context and fed into Medley's decision memory. Over time, the system learns your preferences and applies them automatically — reducing the overhead of oversight without reducing your control.
This is not a more complex system than a black-box API. In many ways it is simpler — because when something goes wrong, you know exactly where to look.
Why This Matters More as Agent Workloads Scale
The case for transparency is not equally strong at every scale. If you're running a handful of agent tasks per week, the cost of black-box routing is low. You can absorb the debugging overhead, the cost opacity, and the lack of auditability.
But agent workloads are not staying small. As teams move from experimenting with AI agents to running them as a core part of their workflow — dozens of tasks per day, multiple projects in parallel, outputs that feed into real decisions — the cost of opacity compounds. Every wrong result that takes hours to debug, every unexplained cost spike, every audit request you can't answer: these are not edge cases at scale. They are the normal operating conditions.
Transparent orchestration is not a nice-to-have for power users. It is the foundation for running agent workloads reliably, cost-effectively, and accountably at any meaningful scale.
The Bottom Line
Black-box orchestration is not bad technology. It is a reasonable tradeoff for teams that want simplicity and are willing to accept opacity in exchange. But it is a tradeoff — and the costs of that tradeoff grow as your workloads scale.
Orchestration you can't see, audit, or steer is, in the end, just a fancier single model. You get better results, but you don't get the ability to understand, improve, or account for them.
If you're building agent workflows that need to be debuggable, cost-controlled, and auditable — and that should get better over time as they learn your preferences — transparent orchestration is not optional. It is the architecture.
Medley is built on that architecture. It's free, local, globally available, and designed to earn your trust one decision at a time. Start at medley.sh.