The Refactor That Passes Every Test

Overview

Ask a developer which task they would most comfortably hand to an AI, and refactoring comes up early. You are not adding a feature or changing a contract. You are moving code around, and behavior is not supposed to change at all.

That feels safe. It is not.

Refactoring legacy code is especially risky because success means “nothing changed”, and that is the hardest thing to prove. The system may be years old, the people who wrote it may be gone, and tests may already exist without anyone being sure they still describe what it actually does today.

Most development runs forward: you know what correct should look like, then you build toward it. Refactoring runs backward. The system as it runs today is the specification. If you are fixing a bug, you can usually show how to reproduce it. If you are adding a feature, you can describe what done looks like, as I did when building a new application with AI. A legacy refactor has neither: only what the system did yesterday, much of it never written down, including quirks that stuck around for years because nobody wanted to touch them. Tests you already have only help if they pass against the code as it stands today, before you change anything.

So when every test passes after an AI-assisted refactor, the honest question is not “did it work?” It is: passed against what?

“Every test passed” can mean two different things. Neither proves as much as it sounds.

A simple example: a shop that updates every night

Picture a small shop that updates every product each night. Each item has two numbers: a sell-by countdown and a quality score. How those numbers change depends on the product type.

• Bread (and most groceries): lose one quality point per night; the sell-by countdown ticks down.
• Cheese: gains quality as it ages.
• Concert tickets: gain value as the event nears, then drop to zero afterward.
• Display vase: a shop fixture, not really for sale. Its numbers never change.

There are two more product types with their own quirks, but you get the idea: six types, six rules, all crammed into one long piece of code with lots of if-then-else. Nobody wrote tests. Nobody wrote down the rules in a place you can trust. The code is the specification.

That is legacy code in miniature: valuable behavior trapped in a shape that is hard to change safely.

Now ask an AI to “clean this up.” It gives each product type its own small piece of code and, each night, looks up which piece to run for each item instead of branching through one long block. The result looks organized. Add tests afterward and they all pass.

Here is the trap: passed against what?

Passed against the code before the refactor?

The trap has two layers. This shop example makes each one visible.

Layer 1: when were the tests written?

If tests were written after the refactor, they only describe the refactored code. What they cannot tell you is whether behavior still matches the code before the refactor, because they never ran against that version.

That is not a small mistake you might catch later. You can fail completely and still see a success screen: every step marked complete, every test passing, the diff looking tidy. Nothing flags a problem, so nobody asks the question that matters.

The fix is ordering. Before anything moves, write characterization tests that capture what the code actually does today, including quirks, wrong answers, and bugs nobody has fixed yet. Run them against the unchanged code first. Michael Feathers introduced them in Working Effectively with Legacy Code: tests that photograph what is there rather than judging it. Fixing a recorded bug is a separate decision with the product owner, not part of the refactor.

A suite that never ran against the old code is not a safety net. It is a notary for whatever the AI just produced.

Figure: Depending on when the tests were written, the same passing report proves two very different things.

The second layer only shows up once you run a real cleanup on this shop example.

Two AI cleanups, every test passing

The original shop code had no tests. I wrote a test for every behavior I could find, ran them all against the untouched original, and only then asked an AI to clean it up. Two fresh sessions, two prompts:

1. Refactor-1 (one-line prompt). “Refactor this file to fix nested if/else blocks.”
2. Refactor-2 (detailed prompt). Split into one rule file per product type, picked from a lookup table, so a new product is one new file and one lookup line. Keep a clean module interface and hide the rest. Do not touch the tests. Show a numbered plan before changing anything.

Both reached the same layout: one file per product type, chosen from a lookup table each night. Both passed every test. From the test report alone, they looked identical.

Another model, or even the same model on another day, could produce different code. This is an example, not a ranking of prompts.

Figure: Refactor-1 and Refactor-2 reached this shape. Same layout, every test passing. The differences are not in the test report.

Passed against behavior, not structure

Layer 2: the tests cannot grade structure. Both runs already had tests in place before any cleanup, so Layer 1 is not what went wrong here. That is the separate mistake of writing the tests after the refactor.

Refactor-1 did not produce broken code. It produced the same overall layout as Refactor-2: one rule file per product, picked from a lookup table. Both passed every test. The tests still could not grade two gaps.

Gap 1: internals visible to the whole project. Refactor-1 opened the internals: interface and implementation both visible to the whole project. Refactor-2 kept a clean module interface and hid the rest. Nothing outside that module needs those files. The tests only call the nightly shop update, so they stay green either way. The code is different. The test report is not.

Gap 2: a rule written nowhere. The display vase must never change: no countdown, no quality shift. Both refactors use a rule file that does nothing, which is correct. Refactor-2 had to show a numbered plan first, so it put a comment in that file: this is meant to do nothing. Refactor-1 went straight to code and left the file empty, with no comment.

An empty file looks unfinished. The next developer, or the next AI asked to “complete anything incomplete,” will fill it in. The tests will fail, which is correct. After Refactor-1 there is no comment to stop the next person, so the empty file looks like the bug. After Refactor-2 the comment is already there.

The lesson here is not only “write tests first.” Tests check that the numbers still match. They do not check visibility, or whether an empty file is on purpose. Gap 1 came from Refactor-2 asking to keep a clean module interface and hide the rest. Gap 2 came from asking for a numbered plan first. A one-line prompt will not ask for things the tests cannot see.

A reminder is not enough: stop the AI before it edits

Knowing you should write tests first is not enough. You will skip it. “Clean it up, then test” always sounds faster. The AI will skip it too. Ask it to refactor and it starts editing.

A note in a prompt does not fix that. “Remember to write tests first” is a reminder. The model will agree, then change files anyway.

So this is not a discipline speech. I block the AI. The rule lives in the project files the coding tool reads at the start of every session, such as AGENTS.md or CLAUDE.md, not in a one-off chat prompt. If my message contains refactor, restructure, or extract, those instructions say: do not edit files, do not write a plan, and do not sketch an approach. The rule reads:

Will this change behavior that already exists? If yes, refuse to start. The next step is tests that capture what the code does today, run against the unchanged code. Only after those tests pass may the refactor begin. Do not offer to “add the tests later.” Tests written later are still tests written after the refactor. That is Layer 1 again.

The developer still answers the question. The AI is not allowed to write a plan or touch a file until that path is followed.

Models will keep getting better at moving code around. Knowing what your tests actually ran against is what keeps that safe.

Key Takeaways

1. Refactoring legacy code is among the highest-risk tasks to delegate, not the lowest. Success means “nothing changed” in a system whose real behavior may be unclear even when tests already exist. Everything else can be checked against something outside the code. A refactor can only be checked against what the code did before.

2. Ordering. Tests must pass against the unchanged code before anything moves. Tests written after the refactor only describe the refactored code. They cannot detect drift, and they will still all pass.

3. Blind spots. Refactor-1 and Refactor-2 both passed every test and reached the same layout. The gaps were internals visible to the whole project, and a rule with no comment. Check those after any AI cleanup, not just the test report.

4. Enforcement. A reminder will not hold. Put the stop in the project files the coding tool reads every session. Trigger a hard refusal on refactoring vocabulary before any plan or code is generated.