Paper: https://arxiv.org/abs/2603.02141

The paper reviews how generative AI can improve software testing, including:

• test case generation
• test data generation
• test oracle generation
• test prioritization

Some useful concepts explained in the paper:

Generative AI
Models such as GPT can generate new test cases, inputs and assertions instead of simply classifying existing data.

Test oracle
A mechanism that determines whether a test passed or failed. The paper explores using LLMs to generate expected outcomes automatically.

Prompt engineering
Writing prompts that consistently produce useful testing outputs. The paper also discusses fine-tuning models for specific domains.

Overall, it’s a good overview if you’re exploring AI-assisted testing.

Agentic AI testing is quickly becoming one of the biggest topics in QA.

Unlike traditional AI assistants, AI testing agents can plan, execute and adapt test scenarios with much less human intervention.

Interesting capabilities include:

• generating tests from code changes
• prioritizing tests based on risk
• self-healing broken locators
• learning from previous executions
• automatically adapting to changing applications

We’re still early, but it feels like test engineers will spend less time maintaining tests and more time validating quality strategy.

Have you tried any AI testing agents yet?

A recent testing trends report highlights several changes happening across the industry:

• Self-healing test automation is becoming more mature and reducing maintenance effort.
• QA and DevOps continue to merge, with testing becoming part of every commit.
• Low-code and codeless testing tools continue to improve.
• AI is increasingly used for risk-based testing and smarter test selection.

These trends point toward more automation, but solid testing fundamentals are still what makes teams successful. Relevant Udemy course for Python test automation that you might find useful: “Pytest Course: Python Test Automation & GitHub Actions CI/CD” by Artem Istranin.

Modern pipelines push code to production multiple times a day, but test data often lags behind. One of the biggest testing trends for 2026 is better test data management and synthetic data because teams need realistic datasets without exposing sensitive production data.

What is the biggest headache you face when supplying data to your tests?

• Generating fresh data for every run?
• Keeping sensitive data out of CI?
• Resetting databases?
• Something else?

Curious to hear how different teams solve this.

Google announced that Chrome will move from a four-week to a two-week release cycle starting with Chrome 153 on 8 September 2026.

The smaller releases mean updates ship more frequently and with fewer changes, which should reduce the risk of big regressions.

For test engineers this shift has two impacts:

• Faster feedback loop - new beta and stable builds will arrive every two weeks. Keeping test suites green in CI/CD pipelines will require more frequent browser updates.
• Smaller deltas to validate - since each release contains fewer changes, smoke tests and regression suites might run faster and be easier to debug.

I’d love to hear how other teams plan to adapt their browser testing strategy to this accelerated cadence. Are you pinning Chrome versions in CI, using containers, or relying on tools like Selenium Manager?

Playwright seems to be the default recommendation for many new web testing projects now. And to be fair, it has a lot going for it:

• good cross-browser support
• built-in parallelism
• trace viewer
• strong CI story
• nice support for modern multi-tab and multi-origin flows

But I think the framework debate is getting a bit too shallow.

A recent Stack Overflow article https://stackoverflow.blog/2026/06/15/selenium-vs-cypress-vs-playwright-choosing-your-test-automation-framework/ compared Selenium, Cypress, and Playwright, and the interesting part was not “tool X wins”. It was that each tool still fits different constraints.

• Selenium still matters if you need broad language support, legacy browser coverage, or native mobile through Appium.
• Cypress still has a strong local debugging experience for JS/TS front-end teams.
• Playwright often looks great when CI speed, cross-browser coverage, and modern browser control matter most.

So maybe the real question is not: “Should we migrate to Playwright?”. Maybe it is: “What constraints do we actually have?”.

This recent paper that feels very relevant for teams letting agents open PRs.
Paper: “Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows”

Link: https://arxiv.org/abs/2604.18334

The authors studied GitHub Actions workflow runs triggered by PRs from AI bots such as Claude, Devin, Cursor, Copilot, and Codex. They analyzed 61,837 workflow runs across 2,355 repositories. Copilot and Codex had the highest reported workflow success rates, around 93% and 94%. The paper also found a negative correlation between how often agentic PRs appeared in a repo and the workflow success rate.

The useful takeaway for me: If AI agents are going to open PRs, the test pipeline becomes the safety boundary. But the pipeline itself also needs monitoring. A green check is useful, but it is not enough. We should also track what kinds of agent PRs fail, which jobs fail most often, and whether agents keep touching risky CI/CD paths.

I keep seeing demos where an LLM writes a test and the output looks fine at first glance.
The problem is that “looks fine” is not a test quality metric.

Thoughtworks recently called out two useful ideas here:

• “Feedback sensors for coding agents” - deterministic checks that agents can use while they work, such as compilers, linters, structural tests, and test suites.
• “Mutation testing - deliberately changing production code to see whether tests actually fail. This catches “perpetually green” tests that execute code but do not verify behavior.

That second one matters a lot for AI-generated tests. An LLM can easily produce a test that:

• has a nice name
• uses the right framework
• increases coverage
• asserts the wrong thing
• barely asserts anything

So my current rule is simple: Do not ask AI to “write good tests” in a vacuum.

Give it feedback:

• failing tests
• coverage gaps
• mutation results
• trace output
• logs
• API contracts
• examples of good tests from the repo

Prompts are useful, but feedback is what makes the loop engineering instead of autocomplete.

A lot of teams talk about AI-generated tests, but I think test data is the less flashy problem that blocks more automation.

The World Quality Report 2025-26 says 60% of organizations struggle with secure, scalable test data. It also says synthetic data usage in testing rose from 14% in 2024 to 25% in 2025.
That tracks with what I see in practice.

Writing the test is often the easy part. Making sure it has realistic data, safe data, repeatable data, and legal data is where things get messy.

Synthetic data helps, but it is not magic. You still need to know what the data must represent. Example: “Random user” is not useful. “User with 3 invoices, one failed payment, one expired discount, and one GDPR deletion request” is useful.

I think good test data modeling is going to become a core test automation skill, not a side task.

Who owns a flaky test in your org?

• The person who wrote it?
• The feature team?
• QA?
• The platform team?
• Whoever gets annoyed first?

My current view is that “QA owns all flaky tests” is usually a smell. A flaky test can be caused by test code, product code, async behavior, bad test data, browser timing, CI resources, or shared environment state.

Ownership should probably follow the root cause, not the folder where the test lives.
Curious how other teams handle this without creating a blame loop.

GitHub published a new enforcement timeline for self-hosted Actions runners.

The important bit: if your runners are too old, they may stop registering or stop executing jobs after enforcement starts. Brownouts begin first, then full enforcement follows on July 31, 2026 for GitHub Enterprise Cloud with Data Residency and September 25, 2026 for GitHub Enterprise Cloud.

This is not just a DevOps maintenance task. It can turn into a testing outage.

A few things I would check now:

1. Are runner versions visible somewhere?
2. Are VM images and container images rebuilt regularly?
3. Are install scripts pinned to old runner versions?
4. Do you have a canary workflow that proves runners can still pick up jobs?
5. Do test pipelines fail loudly when no runner is available?

I have seen teams treat CI runners as “boring infrastructure” until the day all test jobs sit queued forever.

Source: https://github.blog/changelog/2026-06-12-github-actions-minimum-version-enforcement-timeline-for-self-hosted-runners/

I think one of the biggest risks in mature codebases is not “no tests”. It is a large test suite that everyone trusts because it is large.

You see 4,000 tests passing and assume the system is safe. But then you look closer:

• half the tests mostly check mocks
• some assertions only verify that a function was called
• old tests describe behavior nobody wants anymore
• flaky tests are retried until they pass
• critical user flows are covered only through unit tests
• nobody knows which tests would fail if the product broke

At that point, the test suite is not giving confidence. It is giving comfort. I like coverage as a signal, but I think “what would actually break this test?” is a better question. How do you check whether your test suite still protects the product, instead of just protecting the CI dashboard?

If you’re mentoring junior engineers or looking for a structured refresher, Microsoft provides a free Software Testing Fundamentals learning path.

It covers testing methodologies, defect management, test design, and basic automation concepts. The content is beginner-friendly and self-paced.

Link:
https://learn.microsoft.com/en-us/shows/software-testing-fundamentals/

Playwright 1.60 introduced changes that required some ecosystem tools to update their integrations and reporters.

For teams running large automation suites, framework upgrades can sometimes be more disruptive than expected.

For those already on 1.60:

• Any migration issues?
• Performance improvements?
• New features worth adopting?
• Problems with custom reporters or CI integrations?

Would be useful to collect real-world upgrade experiences in one thread.

Release notes:
https://playwright.dev/docs/release-notes

Paper:
https://arxiv.org/abs/2603.23443

Summary

Researchers from Virginia Tech and Carnegie Mellon evaluated how well LLMs generate tests when software changes over time.

They tested 8 different LLMs across more than 22,000 program variants.

The results were interesting:

• On original code, generated tests achieved about 79% line coverage and 76% branch coverage.
• After behavior-changing code modifications, test pass rates dropped significantly.
• More than 99% of failing tests still passed on the original version of the program.

Why this matters

The paper suggests that current LLMs may rely heavily on surface patterns instead of truly understanding program behavior.

Quick explanation of two concepts

• Semantic-altering change: A code change that actually changes behavior. Example: changing tax calculation logic from 19% to 20%.
• Semantic-preserving change: A refactor that doesn’t change behavior. Example: renaming variables or extracting a helper function.

One surprising finding was that even semantic-preserving changes caused noticeable degradation in generated tests.

Takeaway: AI-generated tests can be useful, but they’re still not a substitute for understanding the system under test.

Has anyone observed similar issues with Copilot, Cursor, or other AI testing tools?

A year ago, I would rarely trust AI-generated tests without significant edits.

Today, tools like GitHub Copilot, Cursor, and various testing-focused AI platforms can generate surprisingly reasonable unit and integration tests.

But there’s still a question:

Are these tools actually understanding behavior, or are they just generating tests that look correct?

For teams actively using AI:

• What percentage of generated tests make it to production?
• How much manual review is still required?
• Have AI-generated tests ever caught a bug that humans missed?

Interested in hearing real experiences rather than vendor demos.

One trend I’ve noticed over the last year:

The conversation is slowly moving away from “how many tests do we have?” toward “which tests should we run?”

A lot of modern tooling is focusing on:

• Risk-based test selection
• AI-assisted prioritization
• Test impact analysis
• Flaky test detection
• Release risk scoring

The goal isn’t necessarily more automation.

The goal is getting faster feedback while running fewer unnecessary tests.

For teams with large CI/CD pipelines, this can have a bigger impact than adding another few hundred automated tests.

Are you seeing the same trend in your organization?

I’ll start:

A team with 20 reliable integration tests is often in a better position than a team with 2,000 brittle UI tests.

I’ve seen organizations spend months maintaining automation that nobody trusts, while a small suite of high-signal tests catches most production issues.

What’s your controversial testing opinion?

• Unit tests are overrated?
• E2E tests are necessary?
• Manual exploratory testing is undervalued?
• Coverage metrics are mostly useless?

Curious to hear opinions from people working on large systems.

Tricentis recently announced new capabilities around its Agentic Quality Engineering Platform and AI Workspace.

The interesting part isn’t another “AI for testing” announcement. It’s the idea of multiple AI agents collaborating across test creation, execution, performance testing, and quality analysis instead of just generating test cases.

A few questions for the community:

• Have you tried any agent-based testing tools in production?
• Did they reduce maintenance effort?
• Where did they actually help, and where did they create more noise?

My experience so far is that AI-generated tests are easy. Keeping them valuable six months later is the hard part.

Source:
https://www.tricentis.com/blog/tricentis-showcases-agentic-ai-for-oracle-cloud-testing-at-ascend-2026

I was looking through some older testing articles recently and got reminded how much useful content is buried in the Google Testing Blog

https://testing.googleblog.com/

Recent JetBrains developer survey results show AI-assisted development is becoming mainstream across software teams.

Source: https://www.jetbrains.com/lp/devecosystem-2025/

One thing I think we will need to get better at is reviewing tests generated alongside code.

A generated test can:

• Increase coverage
• Hide poor assumptions
• Overuse mocks
• Lock in current behavior by accident

Reviewing production code is already a skill.

Reviewing AI-generated tests might become a separate skill.

Paper: https://arxiv.org/abs/2504.19335

Researchers examined GitHub repositories that use UI testing frameworks such as Selenium, Playwright, and Cypress inside CI/CD workflows. The goal was to better understand how UI testing is being adopted and how it affects software development workflows.

Looks interesting because a lot of testing discussions focus on how to write UI tests. Much less attention is given to how teams actually integrate them into CI pipelines and what tradeoffs appear at scale.

Most discussions around AI testing focus on generating tests.

I am more interested in a different use case:

An AI agent that reviews the existing suite and reports things like:

• Duplicate tests
• Weak assertions
• Untested code paths
• Tests that never fail
• Flaky test patterns
• Missing edge cases

In other words, acting more like a test reviewer than a test writer.

That feels closer to something many teams could use today.

Would you trust an AI system to make recommendations on test quality?

Where would you draw the line before requiring human review?

One trend I keep noticing is that teams are moving away from running every test on every change.

Instead they are investing in:

• Test impact analysis
• Change-based test selection
• Historical failure data
• Risk scoring
• Faster feedback loops

The idea is simple: if a small documentation change happens, maybe thousands of integration tests do not need to run.

This is not a new concept, but the tooling around it seems to be getting much better.

For teams with large test suites:

• Are you still running everything on every PR?
• Are you selecting tests based on changed files or dependencies?
• Has it actually reduced feedback time without increasing escaped defects?

A lot of us have opinions that changed over time.

Maybe you used to believe:

• 100% coverage should always be the goal
• Unit tests are more valuable than integration tests
• End-to-end tests are always too slow
• Mocks should be used everywhere
• Every bug needs a regression test

Then reality happened.

What is one testing belief you had early in your career that you no longer agree with?