The AI Coding Agent Arms Race — Why Model Portability Matters More Than Benchmarks

A coding-agent migration often starts with a model comparison, then fails somewhere much less glamorous: a test command was stored in a chat preset, a deployment warning existed only in one tool’s dashboard, or a local override quietly changed what the agent was allowed to touch.

That is the wrong place to discover the real dependency. The durable asset is not a model setting. It is the repository-owned release policy: which paths are protected, which command verifies a change, who approves an exception, what evidence must survive review, and how the team returns to safety when a tool behaves badly.

The question is not whether a coding agent demos well; it is whether its work survives maintenance, handoff, and local constraints.

The portability problem is larger than a provider switch

Switching a model is easy to describe. Change an identifier, update an API key, rerun a prompt. Switching a coding-agent workflow is harder because the workflow includes much more than a request to a model. It contains instructions, directory conventions, allowed tools, review rules, test commands, CI status checks, deployment permissions, and the place where a reviewer looks for proof.

The major coding-agent products expose different mechanisms for keeping that context near the project. Current OpenAI Codex documentation says Codex reads AGENTS.md before work, combines global and project guidance, and lets files closer to the working directory appear later in the instruction chain. GitHub documents repository-wide Copilot instructions, path-specific instruction files, and AGENTS.md files whose nearest location takes precedence for an agent. Claude Code documents project CLAUDE.md files and path-scoped rules; it also makes an important boundary explicit: instructions are context, while a hook or other technical control is needed when an action must be enforced regardless of model judgment. OpenAI Codex AGENTS.md GitHub Copilot instructions Claude Code memory

Those are useful capabilities. They are not, by themselves, a release system.

A repository becomes portable when the contracts it needs do not disappear with a product change. If tomorrow’s team moves from one coding agent to another, it should still be able to answer five questions without opening an old chat transcript:

1. Which files or actions require additional review?
2. What is the smallest verification command for this change?
3. What evidence must a pull request carry?
4. Who can approve an exception?
5. How do we stop or reverse a failed change?

That is a different standard from “the new tool can read our instructions.” A tool can read them and still misinterpret them, skip a command, or produce a plausible result that lacks the artifact the next engineer needs.

Put the control plane in the repository

Repository instruction files work best when they identify the operating contract rather than trying to become an encyclopedia. A root file should establish common expectations. A scoped file should add a local hazard. CI should enforce the controls that are mechanical. A pull request should carry the human evidence.

The following example is an illustrative repository contract, not a vendor-specific configuration. The commands and owners are intentionally explicit so a reviewer can check them.

 1# AGENTS.md
 2
 3## Delivery contract
 4
 5- Keep edits inside the requested boundary.
 6- Run `composer test` for PHP application changes.
 7- Run `npm run build` for Vue application changes.
 8- Record commands run and failures in the pull request.
 9- Do not apply production migrations or change deployment configuration.
10- Escalate any conflict between this file and a task request.
11
12## Protected paths
13
14- `database/migrations/**`: require a migration plan and rollback note.
15- `infra/**`: require platform-owner approval.
16- `resources/js/**`: require the Vue build and visual-check evidence.

The point is not that every repository should use these exact commands. The point is that the decision is visible, reviewable, and versioned. A team can replace a model or agent runtime without translating hidden dashboard settings into a new tool’s vocabulary.

Make the root contract short enough to read. It should name the default verification route, prohibited actions, source of authority, and escalation behavior. Long instruction files lose the property that makes them useful: an agent and a reviewer can locate the relevant rule before work starts. Claude Code’s guidance recommends specific, concise, structured project instructions and recommends path-scoped rules when a rule only matters in part of a codebase. That is sound repository design even when the team does not use Claude Code. Claude Code memory

A concise policy also makes product differences manageable. Codex uses its own discovery chain. Copilot offers repository-wide and path-specific instruction types. Claude Code reads CLAUDE.md rather than AGENTS.md, while its documentation provides an @AGENTS.md import pattern for sharing a common source. Do not pretend their loading rules are identical. Keep the high-value contract in a canonical project document, then add small adapters where a tool needs a different filename or format.

Scope rules like code, not prose

A broad instruction such as “be careful with billing changes” is not a control. It does not identify a path, action, reviewer, proof, or response when the rule conflicts with delivery pressure. A scoped instruction can do that work because it has a boundary.

GitHub’s current documentation gives a concrete model: a repository-wide instruction can apply to all requests, path-specific instruction files apply to matching paths, and multiple relevant instruction types can be used together. It also documents AGENTS.md discovery for agents. Codex documents a root-to-working-directory chain and an override filename. Claude Code documents project instructions plus rules that can be scoped with YAML paths frontmatter. These are three different implementations of the same architectural concern: local context must narrow a general rule without silently erasing it. GitHub Copilot instructions OpenAI Codex AGENTS.md Claude Code memory

Use that flexibility to make rules stricter near risk, not to hide a bypass. A directory policy should say what it adds and who owns it. It should point back to the root contract instead of copying everything. Repetition creates drift: a secret-handling rule copied into five files eventually changes in only one.

 1# database/migrations/AGENTS.md
 2
 3This directory inherits the repository delivery contract.
 4
 5## Additional migration controls
 6
 7- Describe the forward and rollback path in the pull request.
 8- Do not edit a migration that has already been released.
 9- Attach the targeted test command and its result.
10- Ask `database-owners` to review an irreversible data change.

This file is intentionally narrow. It does not restate the whole project. It names a hazard that the root policy cannot safely ignore.

A scope review should test the policy itself. Change a file inside a protected directory and one outside it. Confirm that the intended instruction source, reviewer requirement, and verification expectation differ as designed. Then move or rename a directory in a branch and look for stale paths. The policy is part of the system architecture; it deserves fixtures and negative cases.

Make instructions a guide and enforcement a separate layer

The most expensive mistake is treating a model-facing sentence as a hard stop. It is not. An agent can misunderstand a rule, a prompt can omit relevant context, and a tool can run from a directory that activates a different local instruction.

Claude Code’s documentation states this boundary clearly: CLAUDE.md is context, not enforced configuration, and a PreToolUse hook is appropriate when an action must be blocked regardless of what the model decides. The general lesson applies everywhere. Use instruction files for intent, rationale, and workflow context. Use branch protection, CI, permission controls, deployment approvals, and hooks for requirements that must not be optional. Claude Code memory

For a Laravel/Vue repository, a useful division looks like this:

• The instruction file tells the agent which test to run, which directories are sensitive, and what record to attach.
• The pull-request template asks the author to name the commands, evidence, and rollback.
• CI checks a deterministic condition, such as required files or a policy manifest.
• Branch and deployment controls decide whether a protected change can merge or release.
• A human reviewer judges the case that cannot be reduced to a shell command.

Here is a small, working Python validator for an illustrative YAML policy manifest. It does not claim to understand the entire repository. It deliberately catches malformed control records before they become trusted documentation.

 1#!/usr/bin/env python3
 2from pathlib import Path
 3import sys
 4import yaml
 5
 6policy = yaml.safe_load(Path("policy/agent-policy.yaml").read_text())
 7errors = []
 8
 9for control in policy.get("controls", []):
10    name = control.get("id", "<unnamed>")
11    for field in ("owner", "paths", "evidence", "rollback"):
12        if not control.get(field):
13            errors.append(f"{name}: missing {field}")
14
15if errors:
16    print("policy validation failed")
17    print("\n".join(f"- {error}" for error in errors))
18    raise SystemExit(1)
19
20print("policy validation passed")

Pair it with a minimal policy document:

 1controls:
 2  - id: database-change
 3    owner: database-owners
 4    paths:
 5      - database/migrations/**
 6    evidence:
 7      - migration-plan
 8      - rollback-plan
 9    rollback: revert-policy-and-follow-release-runbook
10  - id: frontend-build
11    owner: application-maintainers
12    paths:
13      - resources/js/**
14    evidence:
15      - npm-run-build
16      - visual-check
17    rollback: revert-change-and-rebuild-assets

This is not a governance product. It is a cheap tripwire. The value is that a policy edit now has an executable structural check, an owner, a path set, an evidence requirement, and a recovery reference. That is enough to turn an instruction change from an unreviewed prose edit into a change someone can reason about.

Evaluate an agent migration with a repository harness

Do not benchmark a migration only by asking two tools to implement the same clean feature. A production evaluation should exercise your actual boundaries: project discovery, a constrained edit, a targeted test, a rejected action, an inspectable artifact, and a rollback route.

Build a small harness before you change the default agent. It need not be complex. It needs to make differences visible and repeatable.

 1#!/usr/bin/env sh
 2set -eu
 3
 4policy_check="python3 scripts/validate_agent_policy.py"
 5app_check="composer test -- --filter=Invoice"
 6ui_check="npm run build"
 7
 8printf '%s\n' "1. Read the applicable repository instructions"
 9printf '%s\n' "2. Propose files to change before editing"
10printf '%s\n' "3. Run: $policy_check"
11printf '%s\n' "4. Run the targeted check: $app_check or $ui_check"
12printf '%s\n' "5. Record output path, changed files, and rollback command"

The comparison should score evidence, not a leaderboard claim. Did the agent identify the correct local policy? Did it respect a protected path? Did it run the named command? Did it distinguish a failing test from a successful release? Can a reviewer find the output without opening the original session? Can the patch be reverted without losing the record of what happened?

This gives model portability a practical meaning. The team is not trying to make every agent behave identically. It is testing whether each candidate can participate in the same controlled delivery system. A weaker tool can still be useful for low-risk discovery. A stronger tool still should not acquire deployment authority merely because it produces convincing code.

Preserve the evidence after the chat disappears

A chat session is a poor system of record. It is hard to search during an incident, inaccessible to many reviewers, and easily detached from the commit that eventually shipped. A release control needs artifacts that live where the engineering work lives.

Keep a small evidence record in the pull request, commit trailer, build artifact, or a versioned runbook. The record should say which instruction sources applied, which commands ran, which results failed, which exception was approved, and what rollback path applies. A tool-generated summary is useful only after a reviewer can locate its supporting evidence.

A simple pull-request section is enough to start:

1## Agent-assisted change record
2
3- Applicable policy: `AGENTS.md`, `database/migrations/AGENTS.md`
4- Changed paths: `app/Invoices`, `database/migrations`
5- Verification: `composer test -- --filter=Invoice`
6- Evidence: CI job URL and migration-plan attachment
7- Approval: `database-owners`
8- Rollback: revert commit; follow invoice-release runbook

Do not require the same evidence for a documentation typo and a schema change. Proportion matters. Excess ceremony teaches teams to route around controls. Missing evidence for an externally visible action leaves a gap that no later model comparison can repair.

The repository should also retain a way to disable an agent path without deleting the facts needed to diagnose it. Separate policy rollback from product rollback. Reverting a rule that required a migration plan does not undo a migration. Disabling an agent workflow does not retract an already-sent message. Name both actions in the relevant runbook so the next maintainer does not confuse them during a failure.

Treat the policy migration as a release of its own

When a team adopts a new coding agent, it commonly migrates prompts first and controls last. Reverse that order. Start by moving the repository-owned contract, exercise it in a disposable branch, and verify that the target tool reaches the correct instruction source from the directories where developers actually work. Only then make the new tool a default.

The first rollout should be deliberately narrow. Give the agent a task with an obvious boundary and no irreversible release authority. Ask it to name the instruction sources it used, propose paths before editing, run the required check, and package the resulting evidence. Run one negative case too: place a task in a protected directory and confirm the workflow stops for approval rather than treating the request as a normal patch. The negative case tells you more about operational safety than a polished happy-path demo.

Keep the old route available until the new route has produced repeatable artifacts across maintenance work. This is not indecision. It is a rollback design. A team should be able to turn off the new agent integration, preserve its run records for review, and continue delivery through the existing protected branch and deployment process. The model may be new. The release control should remain familiar.

What you should do Monday morning

1. Inventory instruction surfaces in one repository: AGENTS.md, CLAUDE.md, Copilot files, CI workflows, approval configuration, and pull-request templates.
2. Choose one risky boundary: migrations, deployment configuration, secrets, a payment workflow, or a customer-visible queue action.
3. Write a short root delivery contract that names a verification command, evidence expectation, protected action, and escalation behavior.
4. Add one local rule that tightens the control for the chosen directory. Do not duplicate the root file.
5. Put required fields—owner, paths, evidence, rollback—into a small policy manifest and validate its shape in CI.
6. Run the same bounded repair task with the current agent and a candidate replacement. Compare the policy discovery, changed paths, test output, and rollback record.
7. Keep deployment authority in the existing protected path while the agent proves it can produce dependable artifacts.
8. Review the harness after a real maintained change. If another engineer cannot locate the rule and evidence quickly, fix the contract before expanding agent authority.

The winning coding-agent setup is not the one with the most impressive comparison chart. It is the one a team can replace without rewriting its definition of a safe change.

Further reading

• Source OpenAI Codex — AGENTS.md discovery, layering, and verification

• Source GitHub Docs — repository, path-specific, and agent instructions

• Source Claude Code Docs — CLAUDE.md, path-scoped rules, and enforcement boundary

Continue with AI Agent Operations, Developer Tools, and Laravel + Vue SaaS for the surrounding delivery practices.