Why Your Agent Ignores the Skill You Gave It

Your agent had the skill and still reached for the wrong tool, or nothing at all. That is not the model refusing to work. It is a selection failure, and it gets worse in a specific, measured way as you add capability. The published research is precise enough to give you a decision rule you can apply this afternoon.

Why does my agent ignore a tool it already has?

The agent is failing a routing decision, not rejecting the skill. Choosing among many tools means picking one option out of several, and every option you add is one more thing competing for that choice. This is why the usual remedies do not help: a larger context window does not repair a wrong choice, because the tool description was never missing. It was outvoted.

That distinction matters for debugging. If the tool is present in context and still unused, adding more context, more instructions, or a more capable model addresses the wrong layer. The decision itself is the thing under strain.

What happens to accuracy as the tool count grows?

It degrades, and there is production data on how much. Scaling Enterprise Agent Routing evaluated a 110-agent, 584-tool catalog from a deployed enterprise productivity assistant. As the system scaled from 10 to 110 agents, routing F1 on under-specified requests dropped 16 to 23 percentage points across models.

Two details are worth holding onto. The degradation was measured on under-specified requests, which is the realistic case: users rarely name the tool they want. And the same paper reports the recovery, which is the useful half. Embedding-based shortlisting won back 10 to 11 points of F1 at full scale across all three models tested, and a production annotation study on real traffic confirmed a 10 to 17 point recovery. The problem is real and it has a known remedy.

How few tools can you get away with showing?

Far fewer than most people expose. How Many Tools Should an LLM Agent See? treats shortlist depth as the thing being optimized rather than a fixed setting, across registries ranging from 20 to 3,251 tools. On BFCL, a benchmark with 370 tools, a learned policy that adapts depth per query reached 90.3% coverage while showing an average of 7 tools. Always showing 50 reached 90.8%.

Read those two numbers together. Roughly a seventh of the tools bought you effectively the same coverage. The gap on the hardest cases is larger and runs the other way: on ToolBench's most difficult queries, the adaptive approach found the correct tools 16.7% of the time against 0% for a fixed depth-5 shortlist. Depth should follow the query, not a constant you picked once.

Why can a high selection score hide a failure you cannot see?

Because a raw score ignores how many options the agent faced. With 20 tools, random guessing succeeds one time in twenty. A model scoring 20% raw on that catalog has done nothing a coin could not. Chance correction subtracts what random selection would have achieved and reports only the remainder.

The 99% Success Paradox shows how far apart those two views can sit. On the 20 Newsgroups retrieval set, BM25 and SPLADE both report over 99% success at K=100, while their chance-corrected score sits at roughly zero, meaning the result is indistinguishable from drawing at random. If you evaluate tool selection without correcting for the number of options, you can ship a metric that looks excellent and measures nothing.

How much does multi-step work make this worse?

It compounds. If each call in a sequence is 90% accurate, a five-call pipeline finishes end to end about 59% of the time. That is arithmetic, 0.9 to the fifth power, not a benchmark result. It is worth doing on your own numbers, because it explains how a per-step degradation that looks tolerable becomes a task-level failure rate that is not.

The practical consequence is that reducing the number of decisions per task beats improving the accuracy of each one. A workflow with three steps at 90% is more reliable than one with six steps at 95%.

What should you do instead of exposing more tools?

Consolidate, retrieve, or hide. Consolidation merges narrow tools behind a parameter so the model makes one decision instead of five. Retrieval searches the catalog and passes forward only the candidates relevant to this request, which is what won back those 10 to 17 points in production. Hiding puts a single well-described entry point in front of a capability that does its own work internally.

Retrieval is the strongest published answer for large catalogs, and it carries an honest tradeoff: a retriever that misses the right tool produces exactly the same wrong outcome as a model that overlooked it. You have swapped a selection failure for a retrieval failure. Evaluate the retriever with the same chance-corrected lens, or you will hide the problem one layer down instead of fixing it.

The hiding pattern also suits capability that cannot be shipped to the caller at all. A packaged skill exposed as one entry point is easier to route to than six primitives, and loading it in layers keeps the description short enough to compete. Askpert works this way for expertise its authors do not want to hand over: the caller sees one tool, and the instructions behind it stay on the server.

How many tools should an agent actually see?

Expose the minimum set that can finish the task, keep the rest of the catalog out of context, and pull in the relevant few at call time. Define a handful of tools directly. Past roughly ten, add a retrieval step rather than a longer list. Let shortlist depth vary per query instead of fixing it, since the research found adaptive depth matched a much larger fixed shortlist and beat it badly on hard queries.

Then measure the right thing. Judge each step by selection accuracy corrected for the number of options it faced, and check the end-to-end completion rate rather than the per-call average. Those two numbers will tell you whether your agent is reliable. The raw per-call score will not. If you are choosing an architecture on the back of this, the case against splitting into multiple agents rests on the same compounding math.