AskpertBlog
The Askpert blog
Plain-English guides to agent skills and packaged expertise.
44 posts
How much does an AI agent cost to run?
Agents use roughly 4x the tokens of a chat turn and multi-agent systems about 15x, because the bill scales with the loop rather than the question. How to estimate cost per successful task, and the levers that move it.
Why AI assistants agree with everything
Challenged on a correct answer, 2023-era assistants abandoned it between 32% and 86% of the time. Sycophancy is a predictable result of training on human preference, and how you ask decides what the answer is worth.
Chain-of-thought faithfulness: why a reasoning trace is not an audit log
Reasoning models named the hint that actually changed their answer 25% of the time. When they exploited a reward hack, they mentioned it in under 2% of cases. Here is what a trace can and cannot prove.
What cosine similarity between embeddings actually means
A WWW 2024 paper shows cosine similarity between embeddings can be arbitrary. The score is a property of your current vector space, not a measurement of meaning, which is why absolute thresholds quietly break.
Fine-tuning vs prompt engineering: when to fine-tune, retrieve, or instruct
Three levers change three different layers: weights, context, and the task specification. Pick by which one is failing, because the expensive mistake is training a model to hold facts that change weekly.
Can AI agents correct their own mistakes?
Asking a model to review its own answer measurably lowered accuracy in the ICLR 2024 study. The direction flips once an external signal exists, which makes the signal the design decision that matters.
Why does the same prompt give different answers at temperature 0?
Temperature 0 buys greedy decoding, not reproducibility. The cause is not the GPU race condition everyone cites, and it changes how you read an eval score, a regression, and a bug report.
Does JSON output make LLM reasoning worse?
A widely cited paper says format restrictions wreck reasoning. A direct rebuttal says the experiment was broken. Both sides agree on one engineering rule, and it is the one worth shipping.
Agent Reliability Is a Distribution, Not a Score
A single benchmark score tells you an agent can do a task, not that it will. Researchers drove eight leading agent benchmarks to near-perfect results without solving anything. Here is what to measure instead.
Why Your Agent Finishes the Five-Minute Task and Fails the Four-Hour One
Agent capability is bounded by task length, not intelligence. What METR's time horizons actually measure, why long tasks fail structurally, and how to decompose work so it fits inside the boundary.
Why Your Agent Ignores the Skill You Gave It
Tool selection accuracy degrades as an agent's catalog grows. Here is what the research measured, why a high raw score can hide a failure, and the decision rule for how many tools to expose.
Context Rot: Why a Bigger Context Window Doesn't Fix Your Agent
Context rot is the measured drop in model accuracy as inputs grow. A bigger window does not fix it, because quality decays continuously and position matters as much as presence.