Engineering Agent Skills at Scale

I work in a huge organization in a team that owns the developer productivity and developer experience of a large monorepo with hundreds of active developers working in it. Before AI was a thing, we mostly focused our efforts on large migrations, CI improvements and faster inner loop experiences. Around February 2026, there seemed to be an inflection point where agentic engineering suddenly worked so well that I found myself not writing much, if any, code at all anymore. The AI productivity gains were real, but agents needed context to be effective in a codebase as large as ours. So in addition to our pre-AI focus areas for developer productivity we are now also heavily considering AI-driven engineering and I think I found a pretty nice sweet spot for our agent skills. Here is my philosophy to managing high quality agent skills:

#The structure of good agent skills

Even though an agent skill is extremely easy to create as it is just markdown with a bit of frontmatter, I came to conclude how important it is to structure skills carefully, such that they are easily discoverable and do not bloat the context unnecessarily.

In order to make a skill easily discoverable, you really need to nail the frontmatter name and description. It should obviously be descriptive and precise, but more importantly it should semantically not overlap much with other skills, such that the agent does not read the wrong skill and bloat the context window. We used to have three distinct skills related to E2Es, one dedicated to migrating Cypress tests to Playwright, one for our style guides in playwright projects, one for running playwright in headed mode. The agent was most of the time confused on which one to pick and often went off the rails. The solution here was to create one playwright skill where the main SKILL.md had no content other than references to reference markdown files.

We started treating SKILL.md files as index files with pointers to references with minimal but precise instruction on which reference to look up in which case. Think of it like a table of contents, where the agent can just pick the one reference it really needs.

/.github
└── skills
    └── ado
        ├── references
        │   ├── pr-create.md
        │   ├── pr-threads.md
        │   ├── pr-update.md
        │   ├── wiki.md
        │   └── workitems.md
        └── SKILL.md

This avoids having too many skills and conflicting skills, but more importantly it is also a great pattern to keep context minimal which is absolutely crucial for getting good results out of agents. Generally, context matters most, which is a great segue into the next section which further focuses on reducing context.

#Context Matters

Another issue, especially in large repos with multiple teams actively developing in, the domain specific skills of each team should not affect the context bloat of other teams. For example a team working on the build system should not get skills loaded into the context related to product domain code as it is simply not useful in this context. Therefore, putting all skills into the repo quickly explodes to dozens and hundreds of skills, which will lead to context inefficiencies even when unrelated skills are not loaded into the context. That’s because each skill is accumulated into a skill index that is fed into the system prompt of an agent by the harness like copilot, claude or codex. This skill index is basically just an enumeration of all the frontmatter of all skills (so mostly just name and description of each skill). On a sidenote, it would therefore be wise to keep the frontmatter lean and precise as well.

But the core problem is having too many skills discoverable in a large monorepo or organization. I already liked the lazy-loading concept in skills with references a lot and built on the idea on a higher level, using plugins. We are using GitHub Copilot CLI which supports custom plugins and marketplaces, which are basically a registry of multiple plugins. Think of it as the npm for skills and agents.

Each team creates its own team specific copilot plugin, which is registered in a local copilot marketplace. Those teams are maintaining their skills in their plugins and it won’t affect anyone else, as none of the plugins are installed by default. In order to use a plugin and its skills they include, you must opt-in and actually manually install the copilot plugin. Only the very most generic skills applying to all engineers in the repo stay in the root skills, which are hoisted into the skill index by default (those can’t be opted out). For example those skills would be our ADO skill, or style guide skills.

In practice, teams just install their plugin once and never have to do it again afterwards.

/.github
├── plugin
│   └── marketplace.json
├── plugins
│   └── <plugin-name>
│       ├── skills
│       │   └── <skill-name>
│       │       ├── references
│       │       │   └── <reference>.md
│       │       └── SKILL.md
│       └── plugin.json
└── skills
    └── ado
        ├── references
        │   ├── pr-create.md
        │   ├── pr-threads.md
        │   ├── pr-update.md
        │   ├── wiki.md
        │   └── workitems.md
        └── SKILL.md

#Skills love CLIs

Although skills make agents much more useful and context-aware, LLMs by nature are non deterministic and every token you burn is expensive. Trying to hand the agent tools and scripts to perform deterministic actions is crucial for managing token usage and sometimes it is the only sane way of producing good results.

As an example, I used an agent to move a file from one project to another in a TypeScript monorepo and the agent naively started using grep to replace all consumers to point to the path alias of the new file’s project. However, this is not sound as moving a file needs to update all project internal relative imports to path alias imports, should remove the export in the barrel file (index.ts), update all import declarations, which is also tricky as sometimes there are alias property names on exported identifiers, or a consumer is also importing other exported identifiers of the old project in the same import declaration.

In order to solve that mechanical file move once and for all, I used AI to vibe code a Rust based CLI that uses SWC to parse the entire monorepo pretty quickly and it transforms the Abstract Syntax Tree (AST) deterministically considering all edge cases. I wrapped this CLI in an agent skill and now my agents are perfectly able to mechanically move files in the repo around quickly, reliably and without burning tokens.

#Enforcing Standards in CI

One thing I have learnt in big tech is that a style guide, best practices and good will are never enough. Engineers will almost always try to take the path of least resistance and quality standards are usually not upheld just by good will and code reviews. Instead, the best way to lift up the quality bar is to actually enforce these style guides and best practices by automated tools.

We have Vitest, Jest, Karma and more tools to unit test our code for correctness. We have linters and custom lint rules to enforce style guides and avoid anti patterns. We have formatters to enforce unified formatting rules. So the logical next step for agent skills was to build a skills linter that is cheap to run and validates all SKILL.md files either locally by running a CLI or integrated into our CI pipeline, running the validation on each PR.

I built a skills linter, which does a few things:

1. Uses tiktoken as a tokenizer to convert a skill into raw tokens and then compares each skill’s token count against a global token budget. If the skill has too many tokens, the linter fails.
2. Validates for correctness in the frontmatter. If description or name properties are missing, the linter fails. Empty name? The linter fails.
3. Checks for unique names. If there are duplicate skill names, the linter fails.
4. Checks for unique descriptions. If there are duplicate skill descriptions, the linter fails.
5. Budgets the tokens of the skill index (aggregated frontmatter).

Here is the configuration to give you an idea:

{
  "patterns": ["./.github/**/SKILL.md"],
  "rules": {
    "token-limit": {
      "error": 4000
    },
    "skill-structure": true,
    "unique-name": true,
    "unique-description": true,
    "frontmatter-limit": {
      "error": 50
    },
    "skill-index-budget": {
      "error": 1000
    }
  }
}

It’s really nice as it has already proven to have caught a few instances where engineers would have checked in skills that are out of token budget and additionally to blocking those from getting checked in, such CI-gated enforcements act as self-service documentation as it guides engineers to follow our standards.

If this sounds interesting, check out this repo I used to spike an initial version (which is just a reference point, not production ready): https://github.com/HaasStefan/skills-lint

#Eliminating Blind Spots

All of which I so far wrote in this article was based on my personal assumptions on what could be good engineering practices for agent skills. In big tech, assumptions are not enough. Data talks. It is my due diligence to track and analyze real data and make conclusions based on that, to avoid my personal bias.

Therefore, we are using copilots OpenTelemetry monitoring to track sessions and usage of skills, failure rates, token usage and response durations. This data helped me identify areas where we could use a skill, or which skills are used a lot, or which skills are not used much and might have discoverability issues.

#Conclusion and Outlook

Agent skills are a means to feed context into an LLM. Context is expensive and the wrong context in the wrong session will degrade your AI experience. Therefore, my philosophy to authoring skills is to reduce context, which means making non generic skills opt-in and to slice skills into lazy-loadable references. Working in a large organization has taught me that good will is never an acceptable strategy, hence I am all for enforcing standards using tools in PR pipelines, such as the skill linter, which lints for token budgets.

While I am pretty happy with our skills and tools around them, I still would like to explore Evals to take this engineering effort to the next level. Evals are hard, because they are expensive in three ways:

1. Token burn —> Actual $ cost
2. They are slow —> Slowing down developers
3. They are brittle —> Flakiness in pipelines will block engineers from delivering work

However, having evals is interesting in two distinct ways to me:

1. Evals should help you to measure how token efficient/inefficient tasks are and how changes to agent skills correlate to token efficiency and task score.
2. They act as a real benchmark to compare different models in your codebase, not some arbitrary random terminal SWE benchmarks. I am just waiting for token cost to have huge price hikes once the hyperscalers can no longer subsidize the cost to run LLMs and when this day comes, you would benefit from having benchmarks to find the best price-performance ratio LLM.

So far, I have dabbled with evals and tried to build two eval frameworks. The first one mocked tool calls and is comparable to a unit test, ensuring the LLM is not calling unexpected tools and invokes expected tools. The second one was a classic LLM-as-a-judge where you have a set of tasks/prompts and another LLM reads the session and outputs and gives a score. Honestly, both approaches had major flaws. The first one was flaky as hell, but that is no surprise given I was trying to write deterministic tests for a non deterministic model. The second one seemed just too arbitrary to give scores on difficult tasks. I have not had success yet, but I still have evals on my todo list and want to sit on this some more in the future.