> ## Documentation Index
> Fetch the complete documentation index at: https://upstash-dx-2913.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Actions

Beyond reading pages, a tab can act. A DOM-aware browser agent runs inside the box and resolves natural-language instructions against the live page. It can find elements and execute single actions.

<Note>
  LLM-resolved AI actions are metered and need an API key for the model's
  provider (Anthropic, OpenAI, OpenRouter, Vercel, or OpenCode) on the box or
  your account. Every method accepts a provider-prefixed `model` override such
  as `"openai/gpt-4o"`. Without an override, the call uses the model the box
  was configured with. If the box has no model, it falls back to
  `anthropic/claude-sonnet-4-5`. Replaying a pre-resolved action with
  `act(action)` is the exception: it uses no LLM and needs no key (see
  [Replay an action without an LLM](#replay-an-action-without-an-llm)).
</Note>

## Observe

`observe()` finds actionable elements matching an instruction. Use it to check the page before acting, or to build your own action loop. Each element carries a `selector` plus a suggested `method` and `arguments`, so you can replay it directly with `act(el)` (see [below](#replay-an-action-without-an-llm)):

<CodeGroup>
  ```typescript box.ts theme={"system"}
  const { elements } = await tab.observe("find the login and signup buttons")

  for (const el of elements) {
    console.log(el.description, el.selector, el.method)
  }
  ```

  ```python box.py theme={"system"}
  result = tab.observe("find the login and signup buttons")

  for el in result.elements:
      print(el.description, el.selector, el.method)
  ```
</CodeGroup>

## Act

`act()` resolves and executes exactly one action described in natural language. It also accepts a pre-resolved action from `observe()` to replay without an LLM (see [Replay an action without an LLM](#replay-an-action-without-an-llm)):

<CodeGroup>
  ```typescript box.ts theme={"system"}
  const action = await tab.act("click the primary call-to-action")

  console.log(action.success, action.actionDescription)
  console.log(action.inputTokens, action.outputTokens)
  ```

  ```python box.py theme={"system"}
  action = tab.act("click the primary call-to-action")

  print(action.success, action.action_description)
  print(action.input_tokens, action.output_tokens)
  ```
</CodeGroup>

The result reports what was done (`actions` with the resolved selectors), whether it succeeded, and the token usage of the call.

### Replay an action without an LLM

`observe()` returns each element's resolved `selector` plus a suggested `method` and `arguments`. Pass that element straight back into `act()` to replay it deterministically: no LLM call, no tokens, and no model provider key required. Resolve once with the model, then reuse the action as many times as you like.

<CodeGroup>
  ```typescript box.ts theme={"system"}
  // Resolve once (metered, needs a model key)
  const { elements } = await tab.observe("the primary call-to-action")
  const action = elements[0]

  // Replay as many times as you like: no LLM, no key
  await tab.act(action)
  ```

  ```python box.py theme={"system"}
  # Resolve once (metered, needs a model key)
  result = tab.observe("the primary call-to-action")
  action = result.elements[0]

  # Replay as many times as you like: no LLM, no key
  tab.act(action)
  ```
</CodeGroup>

Observe narrowly (or check the element) before relying on a fixed index like `elements[0]`. Cache the returned action (in your own store or on the box filesystem) and replay it across pages or runs. This is the built-in path for turning an AI-discovered step into a fast, repeatable one. The action must carry a resolved `selector`: `act()` throws if it is missing (an `observe()` element it could not resolve). The replay form runs no model, so a `model` override does not apply. If the page changes and the selector no longer matches, `observe()` again to re-resolve.

## Which one to use

| Method    | Does                                                                              | Best for                                                                |
| --------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `observe` | Finds elements, executes nothing                                                  | Inspecting a page, building custom loops                                |
| `act`     | Executes one action (natural language, metered; or a pre-resolved action, no LLM) | Flows where your code decides each step, or replaying a resolved action |

To turn a single AI-resolved step into a no-LLM one, replay an `observe()` result through `act()` (see [Replay an action without an LLM](#replay-an-action-without-an-llm)). For fully scripted control with no LLM anywhere in the loop, [connect over CDP](/box/overall/browser/connect) with Playwright or Puppeteer instead. Both drive the same tabs, so you can mix scripted steps with AI steps. To watch or replay what the agent did, see [Live View](/box/overall/browser/live-view) and [Recordings](/box/overall/browser/recordings).