Why E2E automated tests still can't (completely) solve software testing problems

Recently, we’ve been discussing how to introduce automated testing into products. A common thought comes up: AI is so powerful now! It can generate browser automation scripts quickly and accurately—shouldn’t it be able to replace manual testing right away and cover every scenario?

Running E2E (End-to-End) tests has an undeniable appeal:

• Black-box testing is easy to execute
• It provides the highest level of confidence
• It validates the entire codebase in one go
• AI-generated test scripts can be stable, consistent, and efficient

However, in real-world environments, the challenges are hard to ignore:

• Unclear specifications: If the expected system behavior isn’t well-defined, what exactly are you testing?
• Side effects: Test actions can affect the test environment
• Test environments: Building a fully realistic environment is slow
• Instability: Code changes can easily break tests
• …

Take a simple “delete role” feature as an example: “Click the delete button and the role disappears.” What does this actually imply?

Feature: Delete Role

  Scenario: Successful deletion
    Given an authenticated admin
    When deleting a role
    Then the role is successfully deleted

  Scenario: Missing parameters
    Given an authenticated admin
    When deleting a role (with missing required parameters)
    Then an invalid parameter error is shown

  Scenario: No admin privileges
    Given an authenticated non-admin user
    When deleting a role
    Then a permission denied error is shown

  Scenario: Deleting oneself
    Given an authenticated admin
    When deleting their own role
    Then an error is shown indicating self-deletion is not allowed

  Scenario: Deleting a non-existent role
    Given an authenticated admin
    When deleting a non-existent role
    Then a role not found error is shown

  Scenario: Deleting a super admin
    Given an authenticated admin
    When deleting a super admin role
    Then an error is shown indicating the super admin cannot be deleted

1. Even a seemingly simple feature can involve multiple preconditions and failure. AI does not inherently understand how your business logic should behave, and most legacy systems lack clearly defined specifications.
2. Six scenarios mean restarting the entire service six times for a single feature. If you don’t, test side effects will inevitably interfere with each other, leading to false confidence and false alarms.
3. Every test is an asset that requires maintenance to remain effective—otherwise, it becomes a burden. E2E tests are the most fragile and hardest to maintain among all testing types.

AI solves the problem of generating automated test scripts, but that’s only one bottleneck. Achieving effective testing requires much more:

• A deep understanding of how the product works and the ability to clearly define specifications
• The ability to choose and apply the right testing strategies and tools (static testing, unit testing, component testing, integration testing, E2E testing, mock data, CI feedback loops, layered architecture, etc.)