When a coding skill tells the AI to leave vulnerabilities alone

24 April 2026 · 8 min read

An adoption-risk analysis of the andrej-karpathy-skills Claude Code plugin, focused on the behavioral guidelines it injects.

Executive summary

andrej-karpathy-skills is one of the most-installed Claude Code skills. It is a set of coding guidelines, distilled from Andrej Karpathy’s advice, that get injected as system-level instructions into every session. Structurally it is about as safe as a plugin gets: no hooks, no MCP server, no subprocesses, no bundled binaries. It is pure text. And the text is the problem.

We threat-modeled the skill with Oplane. The supply chain checked out. What the model flagged was the usage context: three of the injected guidelines, each sensible on its own, combine to build a quiet wall around insecure code. Together they tell a capable agent to leave adjacent vulnerabilities in place, match insecure style when it writes, and treat security hardening as a feature nobody asked for. The fix is one line, and we’ll get to it. First, what the skill actually says.

What the skill actually says

The skill (forrestchang/andrej-karpathy-skills) is a single SKILL.md that injects four behavioral principles into every Claude Code session. Paraphrased lightly, they are:

• Think Before Coding. State assumptions, ask when uncertain, present alternatives instead of silently picking one, push back when a simpler approach exists.
• Simplicity First.No features beyond what was asked, no abstractions for single-use code, no “flexibility” nobody requested, “no error handling for impossible scenarios.”
• Surgical Changes.“Don’t improve adjacent code, comments, or formatting,” “don’t refactor things that aren’t broken,” “match existing style, even if you’d do it differently.”
• Goal-Driven Execution. Turn tasks into verifiable goals with tests, state a brief plan, loop until the success criteria are met.

As coding advice, this is good. It is the kind of discipline that keeps an eager agent from rewriting your whole module because you asked it to add a field. Three of the four principles are unambiguously helpful. The trouble is entirely in the interaction between a few of these rules and the one thing the skill never carves out: security.

Reasonable rules, bad in aggregate

Take three guidelines, quoted verbatim from the skill, and put them together against a real codebase:

• “Don’t improve adjacent code, comments, or formatting”
• “Match existing style, even if you’d do it differently”
• “No features beyond what was asked”

Now give the agent a routine task. It is editing one function and notices a SQL injection in the function right next to it. Watch the rules fire in sequence:

# Task: add an "email" field to get_user().
# The function right next to it, untouched by the task:

def list_orders(uid):
    q = "SELECT * FROM orders WHERE user_id = " + uid   # SQL injection
    return db.execute(q)

# The skill's rules, applied in order:
#   "Don't improve adjacent code"        -> leave list_orders alone
#   "Match existing style"               -> if you must touch it, concatenate too
#   "No features beyond what was asked"  -> "add validation" = an unasked feature
#
# Result: the agent sees the bug, and is instructed to walk past it.

Each rule is defensible. The first keeps diffs small. The second keeps the codebase coherent. The third stops scope creep. But the sum tells a capable coding agent to look away from a live vulnerability, to perpetuate an insecure pattern if it does write nearby, and to reclassify “add input validation” as a feature nobody requested. The agent that would have flagged the injection on its own is now holding a note that says don’t.

This is not a hypothetical about a malicious plugin. The skill is benign and well intentioned. That is exactly what makes it a good example: the gap is not malice, it is a side effect nobody modeled when they wrote three reasonable rules.

How we looked at it: threat modeling a prompt

There is nothing to fuzz here and no binary to inspect. The entire artifact is instructions. So we modeled it the way you would model any input that shapes behavior: what does it change, what could that change cause, and what is missing that ought to be present.

The supply-chain pass came back clean, and it is worth being precise about that. At the reviewed commit the plugin has no hooks, no .mcp.json, no agents, no bin/ executables, no settings.json. It is a manifest and one Markdown skill file. Nothing executes. If your only question was “can this run code on my machine,” the answer is no, and most review would stop there.

Oplane’s first pass did roughly that, and the model itself flagged the gap: it had assessed the installation and supply chain but not the security impact of the behavioral guidelines on the code the agent goes on to write. That second question is the one that matters for a skill whose entire purpose is to change how code gets written, and it is where the real findings came from.

What Oplane flagged

Modeling the usage context, not just the install, produced four requirements. The two Critical ones are the core of this writeup:

1. 1

   Security vulnerabilities are always in scope for remediation, even when adjacent to the task

   Critical

2. 2

   Carve security hardening out of "no features beyond what was asked"

   Critical

3. 3

   Never match existing style when it is insecure: use parameterized queries, strong crypto, and a CSPRNG regardless

   High

4. 4

   Flag security-relevant dead code (unused endpoints, legacy auth) for review

   Medium

The severities describe the gap, not the skill’s intent. A guideline that quietly suppresses remediation of an SQL injection, an unsafe deserialization, or a missing auth check earns a Critical because of what the suppressed class can do, regardless of how reasonable the guideline sounds in isolation. The skill’s own overall risk is moderate; these are the sharp edges inside it.

The one-line fix

None of this means forking the skill or arguing with its philosophy. It needs one carve-out, added to the skill or to your own CLAUDE.md:

Security vulnerabilities are always in scope for remediation,
regardless of whether they are adjacent to the current task.

That single line restores the security instinct the guidelines quietly remove, and leaves everything else about the skill intact. If you want to close the related gaps the model flagged, three more lines do it:

Security hardening (input validation, authentication, authorization,
output encoding) is never a "feature beyond what was asked." Always include it.

Never match existing style when it is insecure. Use parameterized queries,
subprocess arrays, strong cryptography, and a CSPRNG regardless of surrounding style.

If dead code has security implications (unused endpoints, legacy auth,
debug routes), flag it as a security concern requiring review.

Four lines, no behavioral cost. The skill keeps doing the useful thing it was written to do; it just stops telling the agent that security is out of scope.

A second gap: the skill is not pinned

One smaller finding, separate from the guidelines. The skill is installed from a marketplace definition that uses a bare relative source with no commit pinning, so an install tracks the default branch rather than a reviewed revision. The text you audited today is not guaranteed to be the text you run tomorrow. Claude Code supports pinning; the repo just doesn’t use or document it.

/plugin marketplace add forrestchang/andrej-karpathy-skills@<commit-sha>

Pinning to a SHA you have actually read turns “a set of instructions that can change under me” back into “a set of instructions I reviewed.” For an artifact whose entire job is to shape what your agent writes, that is worth doing.

The bigger picture: instruction dependencies

The broader point is the uncomfortable one. Every skill you install is a set of instructions you didn’t write, running with the same authority as the prompts you do. We have learned, painfully, to review code dependencies: to pin them, scan them, and read the diff before we pull a new version.

Instruction dependencies, the skills, system prompts, and agent guidelines that shape what the model builds, are just as load-bearing, and almost nobody reviews them. They don’t show up in a lockfile. They don’t trip a scanner. They change the output of every session they touch, and the only way to catch a gap like this one is to read them and ask what they tell the agent to not do.

This skill is benign and well intentioned, which is exactly why it is a good example: the gap was not malice, it was a side effect nobody modeled. The instructions you adopt deserve the same scrutiny as the code you import, because they are shaping the code you ship.