> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e2b.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Use OpenAI computer-use helpers with E2B Desktop
> Run OpenAI's computer-use helpers against a private E2B desktop sandbox you can watch live.
The OpenAI SDKs ship computer-use helpers: `ComputerToolset` runs the [computer tool](https://developers.openai.com/api/docs/guides/tools-computer-use) loop on the Responses API, and you supply a driver that executes each action the model asks for. E2B Desktop is that driver's target: a private Ubuntu desktop in the cloud, isolated from your machine, that you can watch live while the model clicks and types.
OpenAI documents two ways to connect a model to a screen, and both run on E2B Desktop:
* **Code execution**, which OpenAI recommends for `gpt-6-astra`: the model writes PyAutoGUI or Playwright code, and the desktop runs it. See [Use code execution](#use-code-execution).
* **The computer tool**: the model returns structured clicks and keystrokes, and your driver performs them. See [Run an agent on a desktop](#run-an-agent-on-a-desktop).
The E2B cookbook has runnable examples of both, including Excel to OrangeHRM, in [Python](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-python) and [TypeScript](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-js).
## How it works
1. **Create a desktop.** `Sandbox.create()` from E2B Desktop starts an XFCE desktop with LibreOffice, a file manager and other common apps.
2. **Write a driver.** One method per computer action (`click`, `type`, `scroll`, and so on) plus `screenshot`, each mapped to an E2B Desktop call.
3. **Run the toolset.** `ComputerToolset(driver).run(client, params)` calls the Responses API, dispatches every action to your driver, sends back a screenshot, and repeats until the model answers.
4. **Kill the desktop.** The helper never creates or closes the environment. You own it.
## Install
The helpers need `openai` 3.31.0 or newer in Python and 7.33.0 or newer in Node.
```bash JavaScript & TypeScript theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
npm i openai @e2b/desktop
```
```bash Python theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
pip install openai e2b-desktop
```
Set `E2B_API_KEY` ([get one](https://e2b.dev/dashboard?tab=keys)) and `OPENAI_API_KEY` in your environment. Your OpenAI project needs access to a model that supports the `computer` tool.
## Run an agent on a desktop
This script seeds two order notes on a fresh desktop, opens their folder, and asks the model for the order number of the priority one. It prints a live view URL first, so you can watch the run.
```typescript JavaScript & TypeScript expandable theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
import { Sandbox } from '@e2b/desktop'
import OpenAI, { ComputerToolset, type ComputerDriver } from 'openai'
const MODEL = 'gpt-6-luna'
// OpenAI key names that e2b-desktop does not map on its own
const KEYS: Record = {
ARROWUP: 'up', ARROWDOWN: 'down', ARROWLEFT: 'left', ARROWRIGHT: 'right',
PAGEUP: 'page_up', PAGEDOWN: 'page_down',
}
function noHeldKeys(keys?: string[] | null) {
if (keys?.length) throw new Error('Holding keys during mouse actions is not supported')
}
// Runs each action the model asks for on an E2B desktop.
function e2bDesktopDriver(desktop: Sandbox): ComputerDriver {
return {
async screenshot() {
const png = await desktop.screenshot()
return {
type: 'computer_screenshot',
image_url: `data:image/png;base64,${Buffer.from(png).toString('base64')}`,
}
},
async click({ button, x, y, keys }) {
noHeldKeys(keys)
if (button === 'left') await desktop.leftClick(x, y)
else if (button === 'right') await desktop.rightClick(x, y)
else if (button === 'wheel') await desktop.middleClick(x, y)
else throw new Error(`Unsupported button: ${button}`)
},
async double_click({ x, y, keys }) {
noHeldKeys(keys)
await desktop.doubleClick(x, y)
},
async drag({ path: [start, ...rest], keys }) {
noHeldKeys(keys)
await desktop.moveMouse(start.x, start.y)
await desktop.mousePress()
for (const point of rest) await desktop.moveMouse(point.x, point.y)
await desktop.mouseRelease()
},
async move({ x, y }) {
await desktop.moveMouse(x, y)
},
async scroll({ x, y, scroll_x, scroll_y, keys }) {
noHeldKeys(keys)
if (scroll_x) throw new Error('Horizontal scroll is not supported')
await desktop.moveMouse(x, y)
// The model sends pixels; one wheel step scrolls about 100
const steps = Math.max(1, Math.floor(Math.abs(scroll_y) / 100))
await desktop.scroll(scroll_y > 0 ? 'down' : 'up', steps)
},
async keypress({ keys }) {
await desktop.press(keys.map((key) => KEYS[key.toUpperCase()] ?? key))
},
async type({ text }) {
await desktop.write(text)
},
async wait() {
await new Promise((resolve) => setTimeout(resolve, 1000))
},
}
}
const desktop = await Sandbox.create({ resolution: [1280, 800], timeoutMs: 300_000 })
try {
await desktop.stream.start({ requireAuth: true })
const authKey = desktop.stream.getAuthKey()
console.log('Live view:', desktop.stream.getUrl({ authKey, viewOnly: true }))
await desktop.files.write('/home/user/orders/priority-order.txt', 'Order number: PO-7731\nPriority: high\n')
await desktop.files.write('/home/user/orders/standard-order.txt', 'Order number: PO-1001\nPriority: standard\n')
await desktop.open('/home/user/orders')
await desktop.wait(3000)
const response = await new ComputerToolset(e2bDesktopDriver(desktop)).run(
new OpenAI(),
{
model: MODEL,
store: false,
tools: [{ type: 'computer' }],
input: 'Open priority-order.txt and reply with only its order number.',
},
{
maxSteps: 15,
beforeAction: (action) => {
console.log('action:', action.type)
return true // approve every action; put your own policy here
},
},
)
console.log('Answer:', response.output_text)
} finally {
await desktop.kill()
}
```
```python Python expandable theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
import base64
import time
from e2b_desktop import Sandbox
from openai import OpenAI
from openai.lib.computer import ComputerToolset, ComputerUseError
MODEL = "gpt-6-luna"
# OpenAI key names that e2b-desktop does not map on its own
KEYS = {"ARROWUP": "up", "ARROWDOWN": "down", "ARROWLEFT": "left", "ARROWRIGHT": "right",
"PAGEUP": "page_up", "PAGEDOWN": "page_down"}
class E2BDesktopDriver:
"""Runs each action the model asks for on an E2B desktop."""
def __init__(self, desktop: Sandbox):
self.desktop = desktop
def screenshot(self):
png = self.desktop.screenshot()
return {"type": "computer_screenshot",
"image_url": "data:image/png;base64," + base64.b64encode(png).decode()}
def click(self, action):
no_held_keys(action)
if action.button == "left":
self.desktop.left_click(action.x, action.y)
elif action.button == "right":
self.desktop.right_click(action.x, action.y)
elif action.button == "wheel":
self.desktop.middle_click(action.x, action.y)
else:
raise ComputerUseError(f"Unsupported button: {action.button}")
def double_click(self, action):
no_held_keys(action)
self.desktop.double_click(action.x, action.y)
def drag(self, action):
no_held_keys(action)
start, *rest = action.path
self.desktop.move_mouse(start.x, start.y)
self.desktop.mouse_press()
for point in rest:
self.desktop.move_mouse(point.x, point.y)
self.desktop.mouse_release()
def move(self, action):
self.desktop.move_mouse(action.x, action.y)
def scroll(self, action):
no_held_keys(action)
if action.scroll_x:
raise ComputerUseError("Horizontal scroll is not supported")
self.desktop.move_mouse(action.x, action.y)
# The model sends pixels; one wheel step scrolls about 100
steps = max(1, abs(action.scroll_y) // 100)
self.desktop.scroll("down" if action.scroll_y > 0 else "up", steps)
def keypress(self, action):
self.desktop.press([KEYS.get(key.upper(), key) for key in action.keys])
def type(self, action):
self.desktop.write(action.text)
def wait(self, action):
time.sleep(1)
def no_held_keys(action):
if getattr(action, "keys", None):
raise ComputerUseError("Holding keys during mouse actions is not supported")
def approve(action):
print("action:", action.type)
return True # approve every action; put your own policy here
desktop = Sandbox.create(resolution=(1280, 800), timeout=300)
try:
desktop.stream.start(require_auth=True)
auth_key = desktop.stream.get_auth_key()
print("Live view:", desktop.stream.get_url(auth_key=auth_key, view_only=True))
desktop.files.write("/home/user/orders/priority-order.txt", "Order number: PO-7731\nPriority: high\n")
desktop.files.write("/home/user/orders/standard-order.txt", "Order number: PO-1001\nPriority: standard\n")
desktop.open("/home/user/orders")
desktop.wait(3000)
with OpenAI() as client:
response = ComputerToolset(E2BDesktopDriver(desktop)).run(
client,
{
"model": MODEL,
"store": False,
"tools": [{"type": "computer"}],
"input": "Open priority-order.txt and reply with only its order number.",
},
max_steps=15,
before_action=approve,
)
print("Answer:", response.output_text)
finally:
desktop.kill()
```
Run it with `python agent.py`, or `npx tsx agent.ts` in an ES module project. A run takes a few model turns and ends with `Answer: PO-7731`.
For a full run, see the Excel to OrangeHRM example in the E2B cookbook, in [Python](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-python) and [TypeScript](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-js): `gpt-6-luna` reads new hires from a spreadsheet in LibreOffice Calc, enters each one into the OrangeHRM web app in Chrome, and writes the assigned Employee Ids back into the spreadsheet. It is the same task as the [Claude toolsets](/agents/claude-toolsets) example, driven by the computer tool alone.
The driver covers every action the computer tool can send. It refuses what E2B Desktop has no direct call for, such as clicks with held modifier keys and horizontal scrolling: a driver must raise instead of silently skipping an action, and the helper stops the run.
The examples use `gpt-6-luna`. In our runs of multi-step LibreOffice Calc and web app tasks on E2B Desktop, it finished about twice as fast as Claude Haiku 5.5 at about a tenth of the cost, mostly because OpenAI caches the repeated screenshot history automatically.
## Use code execution
With code execution, the model gets one function tool instead of the computer tool: `exec_py` for Python with [PyAutoGUI](https://pyautogui.readthedocs.io), or `exec_js` for JavaScript with [Playwright](https://playwright.dev). One call can batch several actions, read text from the screen or the page, and check its own result before the model looks again. Your application runs the code and returns its text and screenshots.
E2B Desktop is where that code runs. It executes inside the sandbox, never on your machine, and the desktop stays up between calls, so windows, variables and logins carry over. The E2B side has three parts:
1. **A template with the library.** Build it once per E2B team on top of the stock `desktop` template:
```typescript JavaScript & TypeScript (exec_js) theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
import { Template, defaultBuildLogger } from 'e2b'
const template = Template()
.fromTemplate('desktop')
.setUser('root')
.runCmd(['curl -fsSL https://deb.nodesource.com/setup_22.x | bash -', 'apt-get install -y nodejs'])
.setUser('user')
.setWorkdir('/home/user')
// playwright-core only: Playwright drives the desktop's own Chrome over CDP
.runCmd('mkdir -p /home/user/.exec_js && cd /home/user/.exec_js && npm install playwright-core@1.64.0')
await Template.build(template, 'openai-computer-use-exec-js', {
cpuCount: 8,
memoryMB: 8192,
onBuildLogs: defaultBuildLogger(),
})
```
```python Python (exec_py) theme={"theme":{"light":"github-light","dark":"github-dark-default"}}
from e2b import Template, default_build_logger
# python-xlib replaces the python3-Xlib fork PyAutoGUI asks for: the desktop's
# X server listens only on an abstract socket, which python3-Xlib cannot reach.
PACKAGES = [
"pyautogui==0.9.54", "mouseinfo==0.1.3", "pygetwindow==0.0.9", "pymsgbox==2.0.1",
"pyperclip==1.11.0", "pyrect==0.2.0", "pyscreeze==1.0.1", "pytweening==1.2.0",
"python-xlib==0.33", "six==1.17.0", "pillow==11.3.0",
]
template = (
Template()
.from_template("desktop")
.set_user("root")
.apt_install(["python3-tk"]) # importing PyAutoGUI exits without tkinter
.run_cmd("pip install --no-deps " + " ".join(PACKAGES))
.set_user("user")
.set_workdir("/home/user")
)
Template.build(template, "openai-computer-use-exec-py", cpu_count=8, memory_mb=8192,
on_build_logs=default_build_logger())
```
2. **A persistent runtime inside the desktop.** A small process you start in the background keeps one Python namespace (or one Playwright context connected to the desktop's Chrome) for the whole run. It gives the code `log` or `console.log` for text and `display` for screenshots, and answers each call with the `input_text` and `input_image` items the Responses API takes as a `function_call_output`. Write each call to a file in the sandbox, and have the runtime run that file when `curl` posts to its Unix socket through `commands.run`, so nothing listens outside the sandbox.
3. **OpenAI's loop.** The `exec_py` or `exec_js` tool definition and the Responses loop from [OpenAI's guide](https://developers.openai.com/api/docs/guides/tools-computer-use#use-code-execution), with each `function_call` sent to the runtime and its output returned unchanged.
The E2B cookbook has both as a hello world, next to the computer-tool examples and with the same order-number task as above. In [Python](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-python), `gpt-6-astra` opens the file in the file manager with PyAutoGUI. In [TypeScript](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/openai-computer-use-js), it opens the same folder in Chrome with Playwright. Each run took Astra two calls. The runtimes are adapted from [OpenAI's CUA sample app](https://github.com/openai/openai-cua-sample-app), which runs the generated code on your own computer; on E2B it runs in a throwaway desktop with no internet access.
PyAutoGUI drives the whole desktop by screen coordinates, so it works with any app. Playwright drives only the browser, but it can read page text and click elements by role, so it needs fewer screenshots.
## Watch live
`stream.start()` serves the desktop over noVNC. With `require_auth=True` (`requireAuth: true`) the URL carries a generated password, and `view_only` keeps viewers from moving the mouse under the model. Anyone with the URL can watch until the desktop is killed, so do not share it.
## Approve actions
`before_action` (`beforeAction`) runs before every action, including the final screenshot, and must return exactly `True` to let it through. The example approves everything and logs the action type; this is where an application checks or asks a person before consequential clicks and keystrokes.
Calls that carry pending safety checks stop the run unless you also pass `approve_safety_checks` (`approveSafetyChecks`) and return `True`.
## Sandbox ownership
* **Cleanup:** the helper never kills the desktop. Call `kill()` in `finally` so a failed or cancelled run does not leave it running until its timeout.
* **Timeout:** `timeout=300` (`timeoutMs: 300_000`) is a backstop. Keep it longer than a run of `max_steps` model turns can take.
* **Cancellation:** pass `cancelled` (Python) or `signal` (Node) to stop between model calls and actions.
* **Egress:** the desktop has internet access by default. Restrict it with [`allow_internet_access` or `network.allow_out`](/network/internet-access) when the task does not need the open web.
## Related guides
`gpt-6-luna` copies new hires from a spreadsheet into an HR system on one desktop
How computer use agents drive E2B Desktop sandboxes
What the desktop sandbox ships with and how to customize it
OpenAI's references: the [computer use guide](https://developers.openai.com/api/docs/guides/tools-computer-use), the [Node helper guide](https://github.com/openai/openai-node/blob/main/docs/computer.md) and the [Python helper source](https://github.com/openai/openai-python/blob/main/src/openai/lib/computer.py).