AskpertBlog
The Askpert blog
Plain-English guides to agent skills and packaged expertise.
44 posts
Why RAG distractor documents hurt answer quality
The worst context is not random noise. It is the top-ranked passage that looks like the query and does not contain the answer, measured across 18 distractors.
Repeated sampling needs a verifier to pay off
Repeated sampling took SWE-bench Lite from 15.9% to 56% — but that is coverage, not delivered accuracy. Without a verifier you cannot collect the difference.
What the Reversal Curse means for storing facts in a model
GPT-4 answered the forward question 79% of the time and the reversed one 33%. A fact written into weights is not automatically queryable from both ends.
Why language models hallucinate
Models guess because the grading pays them to. A survey of ten leading benchmarks found most award nothing at all for saying 'I don't know'.
What the AI coding assistant security study actually found
Developers with an assistant wrote less secure code and were more likely to believe it was secure. The rates are from a 2022 model; the confidence gap belongs to the reviewer.
Why a benchmark score does not transfer to your workload
GSM1k rebuilt a famous benchmark with new problems and matched difficulty. Some model families dropped up to 13%. The gap is the part of the score that was never generalization.
What few-shot examples actually teach a model
Replacing gold labels with random ones cost 0-5%. Changing the inputs cost 3-16%. A 2022 result that inverts where most teams spend their prompt budget.
When chain-of-thought prompting actually helps
A meta-analysis of over 100 papers found CoT's gains concentrated in math and symbolic tasks. On MMLU, 95% of the improvement rides on questions containing an equals sign.
BM25 vs vector search: why production RAG usually needs both
Keyword search is a stronger baseline than most teams assume, and the two methods fail in opposite directions. A decision rule keyed to query shape, not to fashion.
Why LLMs get worse in multi-turn conversations
Across 15 models, multi-turn performance fell 39% against the same task asked in one shot. Aptitude dropped 16%; unreliability rose 112%. That split decides what the fix has to be.
Why LLM prompt sensitivity makes model comparisons unreliable
Separators, casing and field order change nothing about what you asked, and can change the score enormously. That breaks prompt tuning, and it quietly breaks model comparisons run on one template.
How to detect LLM hallucinations with semantic entropy
Semantic entropy spots a real class of hallucination by asking whether the model means the same thing each time. It reached 0.790 AUROC in Nature. It also cannot see a model that is confidently and consistently wrong.