# Whiteline Phone Agent - Full Technical & Agent Integration Specification

- Canonical Showcase: https://phonectl.dev/
- Canonical BYO Cloud Gateway / MCP Domain: https://mcp.phonectl.dev
- Gateway Version: 0.5.0 (Dual MCP: Streamable HTTP + stdio + OpenAPI 3.1.0)
- Android APK Version: v0.4.0 (`com.whiteline.controller`, 13,532,155 bytes, SHA-256 `01f69caff000d209c548efc9f1e2756f5efcfa7e0989a02f5471ce6e7f69c3ed`)
- Minimum Android Version: Android 8.0+ (API 26)
- Deployment Model: Open-Core BYO-Gateway (Zero multi-tenant screen sharing; zero embedded secrets in APK)

## 1. Architecture Overview

Whiteline Phone Agent splits control across three layers:

1. On-Device Executor (`com.whiteline.controller`, Android AccessibilityService):
   - Zero root, zero unlocked bootloader, zero USB/ADB cable required.
   - Polls owner's HTTPS Cloud Gateway when a mission or direct action is queued.
   - Captures compressed JPEG viewport + hierarchical Accessibility UI tree (`node_id`, `class_name`, `view_id`, `text`, `content_description`, `bounds`).
   - Enforces `SafetyPolicyEngine` boundaries before dispatching gestures and renders an instant Emergency Stop notification/control.
2. Personal Cloud Gateway (`v0.5.0`, pure Python 3 standard library + SQLite WAL):
   - Exposes REST `/api/v1/*`, browser operations monitor `/ui`, OpenAPI schema `GET /openapi.json`, and Streamable HTTP MCP `POST /mcp`.
   - Includes `index_ui_tree()` to collapse raw Android Accessibility dumps into compact `@e1..@eN` clickable/editable refs with `[cx, cy]` center coordinates and a deterministic `screen_hash` (`sha256:...`).
   - Supports optional Gemini vision planning (`GEMINI_API_KEY`) for multi-step goals via `phone_run` and deterministic direct-action directives (`__APC_OBSERVE_ONLY__`, `__APC_DIRECT_ACTION__`) requiring zero planner LLM tokens.
3. Dual MCP Client Surface:
   - Remote Streamable HTTP: `POST https://mcp.phonectl.dev/mcp` (or your personal Railway gateway URL) with `Authorization: Bearer <token>`.
   - Local stdio bridge: `whq-phone-mcp` speaking newline-delimited JSON-RPC 2.0 over stdin/stdout and forwarding authenticated calls via `GatewayClient`.

## 2. Quickstart: Deploy Gateway & Pair Your Device

1. Deploy Cloud Gateway from `https://github.com/YuraWhitelineUK/whiteline-android-gateway` (`main` branch) to Railway or Docker, configure a persistent volume on `/data`, set `APC_GATEWAY_TOKEN`, and optionally set `GEMINI_API_KEY`.
2. Download and install the zero-secret APK from `https://phonectl.dev/assets/downloads/whiteline-phone-agent-v0.4.0.apk`.
3. Pair the phone by opening a deep link on Android:
   `whiteline://pair?url=https://mcp.phonectl.dev&token=<YOUR_GATEWAY_BEARER_TOKEN>`
4. Enable Whiteline Phone Agent under Android Settings -> Accessibility -> Installed Services (if sideloaded on Android 13+, first enable "Allow restricted settings" in App Info).
5. Save client config locally in `~/.config/whiteline/android-gateway.json` (`gateway_url` and `token`) with `chmod 600`.

## 3. MCP Tools (JSON-RPC 2.0 `tools/call`)

### `phone_status`
Check Cloud Gateway health, planner provider/model, last device poll age, `device_name`, installed apps count, and `status_code` (`OK` or `DEVICE_OFFLINE`).
- Arguments: `{}` (no parameters).

### `phone_observe`
Inspect the current phone screen as compact `@e1..@eN` refs plus `screen_hash` (`sha256:...`).
- Arguments:
  - `mode`: `"cached"` (default, reads latest gateway snapshot) or `"live"` (queues `__APC_OBSERVE_ONLY__` mission and wakes phone for fresh frame + UI tree).
  - `timeout_sec`: number (`1..120`, default `20`).
  - `include_screenshot`: boolean (default `false`; when `true` returns local `screenshot_path` + `screenshot_bytes` in stdio/in-process mode).
  - `max_elements`: integer (`1..200`, default `60`).

### `phone_act`
Execute one deterministic Android action without calling an external planner LLM.
- Arguments:
  - `action_type` (required): `"click_ref" | "set_text_ref" | "click_xy" | "click_node" | "set_text" | "launch_app" | "global_action" | "scroll"`.
  - `target_ref`: string such as `"@e1"` (required for `click_ref` and `set_text_ref`).
  - `expected_screen_hash`: string (recommended `"sha256:..."` returned by `phone_observe` or prior `phone_act`; triggers `STALE_SCREEN_HASH` if UI changed before execution).
  - `text`: string (required for `set_text_ref` and `set_text`).
  - `x`, `y`: numbers (for `click_xy`).
  - `node_id`: string or integer (for `click_node`, `set_text`, optional for `scroll`).
  - `package_name`: string (for `launch_app`, e.g. `"org.telegram.messenger"`).
  - `global_action`: `"BACK" | "HOME" | "RECENTS" | "NOTIFICATIONS" | "QUICK_SETTINGS"` (for `global_action`).
  - `direction`: `"up" | "down" | "left" | "right"` (for `scroll`).
  - `timeout_sec`: number (`1..120`, default `20`).
  - `include_screenshot`: boolean (default `false`).
  - `max_elements`: integer (`1..200`, default `60`).

### `phone_run`
Submit an autonomous multi-step natural-language goal to the Gemini vision planner.
- Arguments:
  - `prompt` (required): natural-language goal.
  - `max_steps`: integer (`1..50`, default `15`).
  - `wait`: boolean (default `true`; set `false` to enqueue asynchronously and receive `mission_id` immediately).
  - `timeout_sec`: number (`1..180`, default `25`).
  - `include_screenshot`: boolean (default `false`).

### `phone_apps`
List or search launchable Android packages reported by the paired device.
- Arguments:
  - `filter`: substring filter (default `""`).
  - `refresh`: boolean (default `false`; set `true` to trigger a live refresh from device).
  - `timeout_sec`: number (`1..90`, default `20`).

### `phone_cancel`
Emergency stop: abort the active `RUNNING`/`PENDING` mission or a specific `mission_id`.
- Arguments:
  - `mission_id`: optional mission UUID string.
  - `reason`: optional string (default `"Cancelled via MCP"`).

## 4. MCP Resources (`resources/read`)

- `phone://status` (`application/json`): Live gateway health, planner metadata, and device poll state.
- `phone://screen/compact` (`application/json`): Latest cached screen indexed as compact `@e1..@eN` refs plus `screen_hash`.
- `phone://apps` (`application/json`): Launchable Android packages reported by the paired phone.
- `phone://openapi` (`application/json`): Full OpenAPI 3.1.0 specification served by Cloud Gateway.

## 5. Compact `@e1..@eN` Optimistic Loop & Error Codes

Recommended agent pattern:
1. Call `phone_observe` with `{"mode": "live"}` to receive `screen_hash` and `elements` (for example `@e1` Send button, `@e2` text input).
2. Call `phone_act` with `{"action_type": "click_ref", "target_ref": "@e1", "expected_screen_hash": "<hash>"}`.
3. `phone_act` executes deterministically via `__APC_DIRECT_ACTION__` and returns the post-action observation + updated `screen_hash` in the same response.
4. Machine error codes returned in MCP tool error payloads (`code`):
   - `STALE_SCREEN_HASH`: UI changed between `phone_observe` and `phone_act`. Re-inspect returned `elements` and retry with updated `screen_hash`.
   - `DEVICE_OFFLINE`: Paired phone has not polled recently or live observation timed out.
   - `FLAG_SECURE_PROTECTED`: Foreground Android window sets `FLAG_SECURE` blocking screenshot capture.
   - `REF_NOT_FOUND` / `REF_NO_NODE_ID` / `REF_UNRESOLVABLE_TARGET`: Target `@eN` is absent or lacks node/coordinate binding.
   - `INVALID_PARAMS` / `MISSION_TIMEOUT` / `ACTION_FAILED` / `MISSION_FAILED`: Validation or runtime mission errors.

## 6. Streamable HTTP Protocol (`POST /mcp`)

- URL: `https://mcp.phonectl.dev/mcp`
- Headers:
  - `Authorization: Bearer <YOUR_GATEWAY_BEARER_TOKEN>`
  - `Content-Type: application/json`
  - `Accept: application/json`
- Supported methods: `initialize` (protocol versions `2024-11-05`, `2025-03-26`), `notifications/initialized` (returns HTTP `202`), `ping`, `tools/list`, `tools/call`, `resources/list`, `resources/read`.
- `GET /mcp` intentionally returns HTTP `405` (server-initiated SSE notifications are not used in stateless HTTP mode).

## 7. Roadmap (APC-005 .. APC-008)

- Wave 1 - `v0.5.0` (`APC-005`, Shipped): Custom domains `phonectl.dev` & `mcp.phonectl.dev`, AEO static & HTTP discovery, Dual MCP (`stdio` + Streamable HTTP), `@e1..@eN` + `screen_hash` optimistic locking, OpenAPI 3.1.0.
- Wave 2 - `v0.6.0` (`APC-006`, Next): `Idempotency-Key` request deduplication, SQLite `Device Lease` concurrency locks, `semantic_ui_diff` delta observations, and Human-in-the-Loop `AWAITING_APPROVAL` confirmation gate.
- Wave 3 - `v0.7.0` (`APC-007`, Planned): 1-Click Railway BYO-Gateway deploy template with `/ui` QR generator, APK Readiness Checklist wizard, Per-App Privacy Allowlist/Denylist in APK, signed reproducible GitHub Releases.
- Wave 4 - `v0.8.0` (`APC-008`, Horizon): Bidirectional WebSocket push (`150-300 ms`), clipboard + allowlisted notification OTP primitives, native Android Intent/Deep-Link runner, multi-device `device_id` fleet routing & Direct LAN / Tailscale mode.