# AWT — AI Watch Tester

> AI-powered E2E testing tool. Just enter a URL — AI generates test scenarios, executes them with Playwright, and reports results. No code required.

## Key Features
- AI auto-generates E2E test scenarios from any URL
- Real browser testing with live screenshot streaming
- Self-healing DevQA Loop — AI fixes broken tests automatically
- Cloud mode (no install) + Local mode (full browser visibility)
- CI/CD integration with one-line curl command
- Document upload for spec-based scenario generation

## Links
- GitHub: https://github.com/ksgisang/AI-Watch-Tester
- Documentation: https://github.com/ksgisang/AI-Watch-Tester/tree/main/docs
- Quick Start: https://github.com/ksgisang/AI-Watch-Tester/blob/main/docs/QUICK_START.md
- API Reference: https://github.com/ksgisang/AI-Watch-Tester/blob/main/docs/API_REFERENCE.md
- Comparison vs Playwright/Cypress: https://github.com/ksgisang/AI-Watch-Tester/blob/main/docs/COMPARISON.md

## Pricing
- Free: $0/month, 10 tests/month (cloud), unlimited (local)
- Pro: $29/month, 100 tests/month

## Tech Stack
- Python, FastAPI, Playwright, Next.js, Supabase, Ollama/OpenAI/Claude

---

# Quick Start Guide

Get AWT running in under 5 minutes.

---

## Local Mode

### 1. Install

```bash
pip install aat-devqa
```

### 2. Configure AI Provider

Pick one of the supported providers:

**Ollama (Free, Local)**

```bash
# Install and start Ollama
brew install ollama    # macOS
ollama serve           # Start the server
ollama pull codellama  # Download the model
```

No API key needed. AWT connects to `http://localhost:11434` by default.

**OpenAI**

```bash
export OPENAI_API_KEY="sk-..."
```

**Anthropic (Claude)**

```bash
export ANTHROPIC_API_KEY="sk-ant-..."
```

### 3. Start AWT

```bash
aat serve
```

Open **http://localhost:9500** in your browser.

### 4. Run Your First Test

1. Enter the URL you want to test (e.g. `https://example.com`)
2. Click **Generate Scenarios** — AI analyzes the page and creates test steps
3. **Review** the generated scenarios, edit if needed
4. Click **Run Test** — watch the browser execute each step
5. View results with pass/fail status per step

### 5. View Results

After the test completes, AWT generates a detailed report with:

- Pass/fail status for each step
- Screenshots (before and after actions)
- Error details for failed steps
- AI-suggested fixes (if DevQA Loop is enabled)

---

## Cloud Mode

> Cloud mode is coming soon at [awt.dev](https://awt.dev).

### How It Will Work

1. **Sign up** at https://awt.dev
2. **Enter a URL** — the target site you want to test
3. **AI generates scenarios** — review and approve them
4. **Watch tests run** — live screenshot streaming in your browser
5. **View results** — detailed reports with full history

### Cloud Advantages

- No installation required
- Tests run on cloud infrastructure
- Real-time screenshot streaming
- Team collaboration and shared test history
- CI/CD integration with API keys

---

## CLI Quick Reference

| Command | Description |
|---------|-------------|
| `aat init` | Initialize project (creates `.aat/` config) |
| `aat config show` | Display current configuration |
| `aat config set <key> <value>` | Update a config value |
| `aat validate <scenario>` | Validate a YAML scenario file |
| `aat run <scenario>` | Run a single test scenario |
| `aat loop <scenario>` | Run DevQA Loop (fail → AI fix → retest) |
| `aat analyze <document>` | Analyze a document with AI |
| `aat generate <spec>` | Auto-generate scenarios from a spec |
| `aat start` | Interactive guided mode |
| `aat serve` | Start the web dashboard |

---

## Next Steps

- [API Reference](API_REFERENCE.md) — Full endpoint documentation
- [FAQ](FAQ.md) — Common questions and answers
- [CI/CD Guide](../cloud/docs/CI_CD_GUIDE.md) — Pipeline integration

---

# Frequently Asked Questions

## What is AWT?

AWT (AI Watch Tester) is an AI-powered end-to-end testing tool. You give it a URL, and AI generates test scenarios, executes them in a real browser, and reports the results. If a test fails, the DevQA Loop can automatically analyze the failure and re-run.

## How is AWT different from Playwright/Cypress?

Playwright and Cypress are browser automation *frameworks* — you write test code manually. AWT is a testing *orchestrator* that uses AI to:

- **Generate** test scenarios automatically from a URL or document
- **Execute** them using Playwright under the hood
- **Self-heal** via the DevQA Loop (AI analyzes failures and retries)

Think of it this way: Playwright is the engine, AWT is the driver.

## Which AI providers are supported?

| Provider | Models | Cost |
|----------|--------|------|
| **Ollama** | codellama, llama3, etc. | Free (runs locally) |
| **OpenAI** | gpt-4o, gpt-4o-mini | Pay per token |
| **Anthropic** | claude-sonnet | Pay per token |

Ollama is recommended for getting started since it's free and runs entirely on your machine.

## Is AWT free? What are the pricing tiers?

**Local mode** is free and open source (MIT License). Run it on your own machine with no limits.

**Cloud mode** (coming soon) will have:

| Tier | Monthly Tests | Price |
|------|--------------|-------|
| Free | 10 | $0 |
| Pro | 100 | TBD |

## Can I use AWT in CI/CD?

Yes. Use the REST API with an API key:

```bash
curl -X POST "https://awt.dev/api/v1/tests?wait=true" \
  -H "X-API-Key: awt_your_key" \
  -H "Content-Type: application/json" \
  -d '{"target_url": "https://staging.example.com", "mode": "auto"}'
```

The `?wait=true` parameter makes it synchronous — the request blocks until the test finishes, so your pipeline can check the result and fail the build if needed.

See the [CI/CD Guide](../cloud/docs/CI_CD_GUIDE.md) for GitHub Actions examples.

## Can I test localhost URLs?

**Local mode:** Yes, AWT runs on your machine and can access `localhost` directly.

**Cloud mode:** Not directly. Cloud tests run on remote servers that cannot reach your local network. You would need to expose your local server via a tunnel (e.g. ngrok) or deploy to a staging environment.

## What browsers does AWT support?

AWT uses **Playwright** as its browser engine, which supports:

- **Chromium** (default)
- Firefox and WebKit are supported by Playwright but not yet enabled in AWT

AWT also offers two engine modes:

| Engine | Description |
|--------|-------------|
| **WebEngine** | Playwright-only. Controls the browser viewport. Default. |
| **DesktopEngine** | PyAutoGUI + Playwright. OS-level mouse/keyboard with full screen capture. |

## How does the DevQA Loop work?

The DevQA Loop is an automated test-fix-retest cycle:

```
1. Run test scenario
       ↓
2. Test fails
       ↓
3. AI analyzes the failure (screenshots, error logs, page state)
       ↓
4. AI suggests or applies a code fix
       ↓
5. Re-run the test
       ↓
6. Repeat until pass or max iterations reached
```

Use it via CLI:

```bash
aat loop scenarios/SC-001_login.yaml --max-iterations 5
```

## Can I self-host AWT?

Yes. AWT is fully open source. For the cloud backend:

```bash
cd cloud
pip install -r requirements.txt
export AWT_SUPABASE_JWT_SECRET="your-secret"
uvicorn app.main:app --host 0.0.0.0 --port 8000
```

For the frontend:

```bash
cd cloud/frontend
npm install && npm run build && npm start
```

See [cloud/README.md](../cloud/README.md) for full setup instructions including Supabase Auth configuration.

## Is AWT open source? What license?

Yes, AWT is open source under the **MIT License**. You can use, modify, and distribute it freely. See the [LICENSE](../LICENSE) file for details.

Contributions are welcome — check [CONTRIBUTING.md](../CONTRIBUTING.md) to get started.

---

# AWT vs Competitors

How does AWT compare to other E2E testing tools? This is an honest, objective look at the landscape.

---

## At a Glance

|  | **AWT** | **Playwright Test** | **Cypress Cloud** | **Testim** | **Katalon** | **Mabl** |
|---|---------|--------------------|--------------------|------------|-------------|----------|
| **Test authoring** | AI auto-generates from URL | Manual code + codegen recorder | Manual code + recorder + AI prompt (experimental) | Visual editor + record + AI + code | Low-code + record + code + TrueTest AI | No-code trainer + AI agent |
| **Install required** | No (cloud) or `pip install` (local) | Yes (npm + browser binaries) | Yes (npm + app) | Cloud SaaS + Chrome ext | Yes (desktop app) | Cloud SaaS + optional desktop |
| **AI usage** | Core — generates, executes, and heals | None native | Assistive (experimental) | Core | Core (TrueTest) | Core (AI-native) |
| **Self-healing** | Yes (DevQA Loop) | No | Experimental | Yes (Smart Locators) | Yes (built-in) | Yes (Adaptive Auto-Heal) |
| **Free tier** | Yes (local unlimited, cloud 10/mo) | Fully free (OSS) | 500 results/mo | Limited | Free forever (core) | No (14-day trial only) |
| **Starting price** | $0 | $0 | $67/mo | ~$450/mo | $84/mo | Contact sales |
| **Open source** | Yes (MIT) | Yes (MIT) | Partial (runner only) | No | No | No |
| **CI/CD** | REST API + one-line curl | CLI in any CI | CLI + Cloud parallelization | CLI + webhooks | Runtime Engine + plugins | Deployment Events API |
| **Local + Cloud** | Both | Both | Both (cloud = dashboard) | Both | Both (TestCloud) | Both (local = limited) |
| **Doc-based generation** | Yes (PDF/DOCX/MD) | No | No | No | Partial (Jira only) | No |
| **Real-time observation** | Yes (live screenshot streaming) | Headed mode + Trace Viewer | Open mode + Test Replay | Local debug + screenshots | Studio IDE | Real-time cloud screenshots |

---

## What Makes AWT Different

### 1. Zero to Test in 30 Seconds

Most tools require you to write code, record flows, or configure a project before running your first test. With AWT, you enter a URL and AI does the rest — scenario generation, execution, and reporting. No code. No setup. No learning curve.

### 2. AI Writes Tests, AI Fixes Tests

Other tools use AI as an assistant — suggesting locators or healing broken selectors. AWT puts AI at the center of the entire workflow. The **DevQA Loop** means that when a test fails, AI analyzes the failure (screenshots, DOM state, error logs), suggests or applies a fix, and re-runs. This goes beyond locator healing into full test-level self-repair.

### 3. Watch AI Test Your Site

AWT streams live screenshots to your browser as tests execute. You see exactly what AI sees, step by step. No waiting for a video file after the run — you observe in real time.

---

## Detailed Comparisons

### vs Playwright Test

Playwright is an excellent browser automation framework and the foundation AWT is built on. It gives you fine-grained control, multi-browser support, and a powerful API for complex scenarios.

The key difference is **who writes the tests**. With Playwright, you write code (or use codegen as a starting point and then refine). With AWT, AI generates the entire scenario from a URL or specification document. If you have a team of test engineers who want full control, Playwright is the better fit. If you want AI to handle test creation and maintenance, AWT fills that gap.

Playwright is also the better choice for testing complex application logic, custom protocols, or scenarios that require precise programmatic control.

### vs Cypress Cloud

Cypress pioneered the developer-friendly test runner with its interactive Test Runner and time-travel debugging. Cypress Cloud adds parallelization, Test Replay, and team analytics. The recently introduced `cy.prompt()` brings experimental AI capabilities.

AWT takes a fundamentally different approach: instead of developers writing tests that get recorded to the cloud, AWT has AI generate and execute tests entirely. Cypress excels when your team wants to write and own test code with excellent DX. AWT is designed for teams that want test coverage without writing test code.

Cypress also has a mature ecosystem of plugins and a large community, which is a significant advantage for teams invested in the JavaScript testing ecosystem.

### vs Testim (Tricentis)

Testim is a strong AI-powered testing platform with Smart Locators, visual test editing, and Agentic AI for test generation. Its acquisition by Tricentis positions it well for enterprise QA workflows.

Both Testim and AWT use AI as a core component. The differences are in approach and accessibility: Testim is a full enterprise platform with a visual editor, team management, and deep integrations. AWT is lightweight — enter a URL, get tests. Testim starts at ~$450/mo with enterprise-level onboarding, while AWT is open source and free to self-host.

Testim is the better choice for large QA teams that need a comprehensive platform. AWT is for developers and small teams who want fast, simple, AI-driven testing without platform overhead.

### vs Katalon

Katalon is a feature-rich platform covering web, mobile, desktop, and API testing with a low-code approach. TrueTest generates tests by observing real production user behavior, and the self-healing system reduces maintenance.

AWT and Katalon differ in scope and philosophy. Katalon is an all-in-one test management platform with its own desktop IDE, while AWT is focused purely on AI-driven web E2E testing. Katalon requires installing a desktop application and has a steeper learning curve, but rewards you with broader coverage across platforms.

If you need cross-platform testing (web + mobile + desktop + API) in a single tool, Katalon covers more ground. AWT is purpose-built for web testing with the simplest possible onboarding.

### vs Mabl

Mabl is arguably the closest competitor to AWT in philosophy — both put AI at the center. Mabl's Adaptive Auto-Healing, Test Creation Agent, and Auto TFA represent a mature AI-native testing platform.

The main differences are openness and simplicity. Mabl is a proprietary SaaS with no free tier, no public pricing, and no self-hosting option. AWT is open source (MIT), free to self-host, and designed to get you from URL to results in seconds. Mabl provides richer enterprise features (visual regression, PDF/email testing, advanced analytics), while AWT focuses on the core AI testing loop.

For enterprises that need a fully managed platform with dedicated support, Mabl delivers. For teams that want an open, self-hostable AI testing tool, AWT is the alternative.

---

## When to Choose AWT

### AWT is a great fit when you:

- Want to test a web app quickly without writing test code
- Need AI to generate test scenarios from a URL or specification document
- Want a self-healing test loop (DevQA Loop) that fixes failures automatically
- Prefer open source with the option to self-host
- Need CI/CD integration via a simple REST API
- Are a small team or solo developer who wants test coverage without a QA platform

### AWT may not be the best fit when you:

- Need cross-platform testing (mobile, desktop, API) — AWT is web-only
- Require pixel-perfect visual regression testing — use Mabl or Applitools
- Want a mature enterprise platform with SSO, RBAC, and audit trails — use Testim or Katalon
- Need fine-grained programmatic control over every test step — use Playwright directly
- Prefer writing and owning test code as part of your codebase — use Playwright or Cypress
