Skip to content

test(e2e): real-lane all-testMatch still runs legacy chat-real API bypass #2357

Description

@DeliciousBuding

Summary

scripts/e2e/run-real-e2e-lane.sh currently defaults to every playwright.real.config.ts testMatch spec. On the 2026-09-09 huawei-dev real stack at 2090a420, the full lane produced 7 passed / 17 failed and manifest status failed, while the supported Golden Flows target (AGENTHUB_E2E_SPEC_TARGET=golden-flows.spec.ts) passed 4/4.

The full-lane failures mix two categories; this issue tracks the stale legacy lane rather than a product regression.

Verified causes

  • app/e2e/chat-real.spec.ts direct API tests still mint a hard-coded local JWT (agenthub-local-dev-secret-key-32chars-min) and expect the legacy Hub envelope code: "OK" / old Edge health envelope. Current Hub returns lowercase ok; current Edge health is flat (status, http_status), and direct Hub APIs use real Hub sessions.
  • With a fresh non-dev Edge process, the auto-generated local auth token made Hub→Edge and direct Edge API requests 401. The huawei-dev restart script was missing AGENTHUB_DEV=1; fixing that restored GF4 and Edge API access.
  • The chat-real browser flow also selected the composer <form> instead of its textbox.
  • Running real-oidc-login.spec.ts concurrently with the rest hit TokenDance ID rate limiting (too many requests); this is separate from the stale direct-API assertions.

Acceptance

  • Decide whether chat-real.spec.ts direct API tests are retired, rewritten to use real OIDC/Hub sessions, or split into a separate explicitly-supported lane.
  • Remove hard-coded JWT secret and stale OK/Edge-envelope assumptions from any retained tests.
  • Make the default real-E2E lane either green end-to-end or fail-fast with a documented supported spec set; do not claim L3 evidence from a failed manifest.
  • Re-run on huawei-dev and record the manifest/commit.

Evidence

  • Full lane manifest: /root/agenthub-dev/AgentHub/tests/artifacts/manifest-20260909-073510.json (failed)
  • Targeted Golden Flows manifest: /root/agenthub-dev/AgentHub/tests/artifacts/manifest-20260909-073823.json (passed, 4/4)
  • Current source: 2090a420f733f14accc6036175145f6b9d8874c2

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions