Skill Issue
TL;DR - in a real agentic workflow, the agent rarely loaded knowledge management skills on trigger; and when loaded, didn't follow them. Targeted experimentation shows that the problem correlates with context size, and is largely unaffected by optimizing prompt wordings. Force-loading the skill doesn't reliably lead to it being used. The problem is more evident on OpenAI models. Agentic workflows should be tested under load and their traces should be inspected. Maybe skills could be embedded into the model itself via fine-tuning?
I was researching better ways to give my coding agents persistent knowledge using Database Exploration set of tasks from the Continual Learning bench. Each of the 40 questions in the benchmark starts a fresh conversation related to the same database, but Markdown notes survive between questions. My goal was to get the agent to avoid rediscovering schema details, thus saving on tokens and database queries.
I was comparing a plain, single-file Markdown memory against three more structured approaches - a skill for authoring the Open Knowledge Format (OKF) wiki, which is a Google's effort to standardize Karpathy's LLM Wiki skill; okfcli, a CLI (and an accompanying skill) that I had built for maintaining local OKF- and Obsidian-compatible agentic knowledge bases; and the combination of the two.
There was just one problem: as I discovered, the agents weren't using the skills.
In one of the benchmark runs, across 120 agentic sessions with the skills installed, the traces showed only one explicit read of a skill's full instructions and zero invocations. In that one case where the skill was loaded, Codex ignored it. The agent did still retain knowledge, but avoided the workflow that I thought I was benchmarking.
Maybe I was just bad at writing skills. Plausible. But Codex also ignored the community-maintained OKF skill. At least my questionable skill-writing abilities weren't the only suspect.
Nudging the Agents
First, I tried adjusting prompts and skill descriptions.
I tried making the persistent-memory requirement more explicit in the prompt: first pointing out the directory that survives between conversations, then encouraging durable notes, saying the agent must retain reusable knowledge, eventually just openly using triggers knowledge bundle and knowledge base for agents from the OKF and okfcli skill descriptions:
Read all 5 prompt revisions
Update the memory store if useful.Update the supplied memory store if useful. Persistent memory: /workspace/wiki.Update the supplied memory store if useful. Persistent memory: /workspace/wiki. This directory persists between questions; other workspace files and conversation history do not.Update the supplied memory store if useful. Persistent memory: /workspace/wiki. This directory persists between questions; other workspace files and conversation history do not. You MUST use persistent memory to retain durable, reusable knowledge.Persistent knowledge bundle is located at: /workspace/wiki. This directory persists between questions; other workspace files and conversation history do not. You MUST create and maintain a knowledge base for agents.
I also rewrote my CLI skill's YAML frontmatter description several times with an increasing emphasis on discoveries, prior findings, Markdown memory, and validation, and refined its instructions:
Read all 4 skill description revisions
Search, read, and edit local Markdown wiki knowledge bases using okfcli. Use for wiki pages, sections, links, and validation.Create, edit, and maintain local Markdown wikis and knowledge bases. Use for wiki exploration, retrieving prior findings, recording learned knowledge, correcting outdated notes, and searching or validating memory pages. Use as a standalone persistent knowledge skill or alongside another knowledge base skill, such as OKF or karpathy-llm-wiki.Proactively or reactively create, edit, and maintain local Markdown wikis and retain durable knowledge. Use for wiki creation, exploration, retrieving prior findings, recording learned knowledge, correcting outdated notes, and searching or validating memory pages. Use as a standalone persistent knowledge skill or alongside another knowledge base skill, such as OKF or karpathy-llm-wiki.Proactively or reactively create, edit, and maintain local Markdown wikis and retain durable knowledge. Use for wiki creation, exploration, retrieving prior findings, recording learned knowledge, correcting outdated notes, and searching or validating memory pages. Use as a standalone persistent knowledge skill or alongside another knowledge base skill, such as `okf-open-knowledge-format` or `karpathy-llm-wiki`. Invoke this skill when the user mentions 'OKF', 'Open Knowledge Format', 'knowledge bundle', 'OKF bundle', creating a knowledge base for agents, 'agent-readable knowledge', 'LLM wiki', 'Markdown wiki', 'knowledge catalog', or wants to structure knowledge as markdown files for AI agent consumption.
Along the way, the agent started reliably persisting knowledge, but it didn't reliably make the agent select the relevant skills. Runs after runs, it just used cat and ordinary Markdown files instead of okfcli; the created wikis did not follow the OKF standard. In one of the more successful runs, the prompt that explicitly mentioned multiple skill triggers resulted in the skill being loaded in 23 out of 120 sessions (19%); out of these 23, the skill was followed 9 times; at the same time, persistent knowledge was read or written in almost every session.
Of course, I could have made the next step and simply prompted the agent directly to use the OKF or okfcli skill but that would defeat the goal: I want the agent to proactively recognize when something is worth remembering, find the appropriate knowledge-management workflow, and maintain that knowledge as a natural part of doing other work. Users shouldn't have to micromanage agent's memory retention.
But out of curiosity I did do a benchmark run with forcibly loading the OKF skill and... while the agent retained knowledge in a Markdown wiki, the wiki did not follow the OKF guidelines at all, i.e., the loaded skill was ignored in 40 out of 40 sessions. Interestingly, forcibly loading my okfcli skill did make the agent use it properly and - to my joy - that reduced the token usage considerably, but that's a story for another day.
A Smaller, Controlled Experiment
To understand what was failing, I stripped away most of the complexity.
All that was left was a synthetic skill with a tiny set of instructions on maintaining local knowledge base. Agent was tasked with significantly simplified, compact SQLite questions while being told to retain reusable knowledge, but never told to invoke the skill. It was a very similar setup that I had for the CL Database Exploration bench but with a much smaller context and skill.
I ran 30 independent trials across 10 configurations, varying the skill descriptions, instruction length, the structured-output setting, and whether the knowledge directory was pre-created and initialized with an empty note.
In the session traces I checked that the skill was properly advertised; verified whether or not it was read; and if read, then whether or not it was followed.
This synthetic experiment produced a striking contrast, especially considering that a knowledge base was much less needed for these simpler tasks:
- Skill was loaded in 12 out of 30 runs (as opposed to 1 out of 120 in the real benchmark).
- Every time the skill was loaded, it was executed (the one time it was loaded in the real benchmark, it was not executed).
- Reducing context size seemed to be the only impactful change; changing other variables didn't have a clear effect.
I postponed publishing this by several days and did a second synthetic experiment with 1,050 agentic sessions going through a collection of latest OpenAI and Anthropic models at different reasoning levels, of which 840 were supposed to trigger a skill.
| Model | Effort | Didn't load or follow the skill | Didn't follow a force-loaded skill | Overall non-compliance |
|---|---|---|---|---|
| Claude Haiku 5.5 | Max | 0 | 1 | 1/40 (2.5%) |
| GPT 5.6 Terra | Light | 1 | 0 | 1/40 (2.5%) |
| GPT 6 Luna | High | 0 | 2 | 2/40 (5.0%) |
| GPT 6 Sol | Light | 3 | 0 | 3/40 (7.5%) |
| GPT 6 Sol | High | 5 | 3 | 8/40 (20.0%) |
| GPT 6.1 Sol | Light | 5 | 0 | 5/40 (12.5%) |
| GPT 6.1 Sol | High | 5 | 0 | 5/40 (12.5%) |
| Other 14 model/effort configurations | — | 0 | 0 | 0/560 (0%) |
| Total | 19 | 6 | 25/840 (3.0%) |
Although, the results are not as damning in this experiment, there are still patterns emerging. All 25 misses occurred with the longer, wordier skill and the longer, more complex prompt; and overwhelmingly with OpenAI models.
Across three experiments, there is clearly an issue of agents not selecting and not applying skills. An issue that manifests itself across a variety of setups, and seemingly is aggravated by increased context.
From Benchmarks to Real Agentic Workflows
While the experiments imply it, they do not necessarily prove that context length causes the difference. Skill complexity, task framing, and several other factors changed too; behavior changes from model to model. But what it does show is that agentic workflow evaluations need to incorporate extensive trials under realistic load, as well as that agents can produce seemingly correct results while not adhering to the intended workflows.
A possibility I want to investigate is using fine-tuning for skill memorization, selection, and adherence: teaching a model to recognize when a skill applies, get the model to inherently know the skill, and have it follow its workflow even in a noisy context. Embed the skills directly into the model, thus elevating the issue above the context completely.
That is a hypothesis. I wish I had the time to thoroughly test it.
For now, the takeaways are:
- An available and relevant skill would not necessarily be loaded; a loaded relevant skill would not necessarily be followed.
- One should inspect traces of agentic workflows.
- One should test agentic workflows under load.