Playwright's Planner, Generator and Healer on a real codebase: what I kept and what I rejected

Playwright now ships with three AI agents: a Planner that explores your app and writes a test plan, a Generator that turns the plan into Playwright tests, and a Healer that repairs tests when they fail. The demos run them against a todo app. I wanted to know what happens on a real site, so I pointed them at this one.

Everything below comes from a single run on 13 October 2026, with Playwright 1.64, against the live fahriddin.com.

The setup

npm i -D @playwright/test@1.64.0
npx playwright install chromium
npx playwright init-agents --loop=claude

init-agents creates three agent definitions, an MCP config for Playwright's test server, a specs/ folder for plans and a seed test. The seed test is the starting point every agent reuses. Mine just opens the home page and checks the title.

Because this is a live site, every prompt carried the same hard rules: browse read-only, never submit a form, never visit the admin.

Step 1: the Planner

I asked the Planner to cover the home page, navigation, the writing list, a post, the work page, the CV, the contact page, the theme toggle and the mobile menu.

• Time: 33 minutes
• Cost: about $0.66
• Output: a 449-line Markdown plan with 49 scenarios in 9 groups

The plan was good. It respected the rules on its own: every scenario that touches a form is labelled "NO SUBMIT", and it checked validation by reading the browser's validation state instead of pressing Send.

The best part was not the plan. It was what the Planner noticed while exploring:

• The footer topic links don't filter anything. They point to /writing?topic=Agentic+QA, but the page ignores the parameter and shows every post. I checked the code and it is a real bug.
• A topic count was wrong. The "Testing" chip showed a different number from the Testing tab.
• The work page was empty, so it wrote the project-dialog scenario as conditional instead of inventing one.

That is exploratory testing, and it found a real defect on a site I thought I knew well.

Step 2: the Generator

Generating all 49 scenarios would have taken hours, so I picked 10 that cover every area.

• Time: 5 minutes
• Cost: about $1.22
• Output: 10 test files, 367 lines
• First run: 7 of 10 passed

The three failures were instructive. Two mobile-menu tests failed because the Generator could not resize its browser while writing them, so it wrote selectors from the plan without seeing the mobile layout. The third expected the exact browser message "Please fill in this field." while Chromium says "Please fill out this field."

Step 3: the Healer

To test the Healer honestly, I added one break of my own: I changed a test to expect the writing page at /blog instead of /writing. That gave it four failing tests.

• Time: under 2 minutes
• Cost: about $0.38
• Result: 4 of 4 green, 16 lines added and 10 removed

Three of the heals were correct. The validation-message test now accepts either browser wording. The mobile tests had failed because the menu's outer element has zero size (only its children are drawn), so Playwright called it hidden. The Healer switched to checking the menu's open state and kept the checks that every link inside is visible.

The fourth heal is the one to think about. It "fixed" my /blog break by changing the test back to /writing:

-    await header.locator('a[href="/blog"]').click();
-    await expect(page).toHaveURL(/\/blog$/);
+    await header.locator('a[href="/writing"]').click();
+    await expect(page).toHaveURL(/\/writing$/);

Here that was right, because /writing is the real route. But imagine the opposite: the product team really did rename the route to /blog, and the deploy silently failed. The Healer would have made the test agree with the broken site, and the suite would have stayed green. It cannot know which side is the source of truth. Only a human reviewing the diff can.

What I kept and what I rejected

Of the 10 generated tests:

• Kept as written (4): header navigation, opening a post from the list, theme persistence across reloads, and contact-form validation (after the heal).
• Kept after reviewing the heal (2): both mobile-menu tests.
• Rewritten (3): the post list, the category tabs and the post header. All three hardcode CMS content: exact titles, dates, reading times and post counts. They broke during the run itself, because a new post was published halfway through. Tests for a CMS-driven page should assert structure, not today's content.
• Rejected (1): the footer topic test. The Generator saw that the topic filter is ignored and wrote a test that asserts exactly that:

// POTENTIAL DEFECT (known behaviour): the 'topic' query param is currently ignored.
// The page does not filter by "Agentic QA"; the 'All' tab stays selected
await expect(page.getByRole('tab', { name: 'All' })).toHaveAttribute('aria-selected', 'true');

It even left a comment calling it a defect, then encoded the defect as the expected result. That test passes today and will fail on the day I fix the bug. The correct version asserts the filtered list and is marked as a known failure until the fix ships.

The scorecard

• Planner: genuinely useful. Slow, but it explores like a curious tester and found a real bug. I would run it on every new feature.
• Generator: a fast first draft. Expect to rewrite anything that touches dynamic content, and never trust a test that asserts current behaviour without checking it is correct behaviour.
• Healer: good at mechanical fixes (wording, locators, timing). Dangerous on anything where the test might be right and the app wrong. Every heal should arrive as a reviewable diff, never an automatic commit.

Total cost for the whole experiment was about $2.25 and roughly 40 minutes of agent time. That is cheap. The expensive part, and the part that stays human, is deciding which of those tests deserve to exist.

Have you let a self-healing tool change your tests without a review step, and did it ever hide a real bug?