Title
Build Persistent AI Agents with OpenAI’s Responses API + Computer Use Tool
Why this matters for builders
The Responses API with the new Computer Use tool turns a stateless LLM into a persistent agent that can directly control a secure shell environment inside an OpenAI-hosted container. It lets you give the model a real terminal, filesystem, installed tools, and long-lived state so it can read, edit, run, and debug code autonomously — moving from “chat assistant” to “managed compute process.”
This is the first time OpenAI has shipped a production-grade agent runtime that includes a full shell tool, automatic container provisioning (container_auto), file persistence across turns, and safety guardrails designed for agentic behavior. For vibe coders and indie builders it means you can finally ship real autonomous workflows without managing your own VMs, Docker, or orchestration layer.
When to use it
- Automate repetitive devops or data-engineering tasks that require shell access
- Build internal tools that edit configuration files, run tests, and commit changes
- Create research agents that scrape, analyze, and write results to persistent storage
- Prototype “AI teammate” features that maintain project state across sessions
- Replace brittle script + cron jobs with a natural-language-controlled agent
Do not use it yet for high-stakes production systems that touch customer data or financial transactions — the OSWorld benchmark score is only 38.1 %, so human oversight is mandatory.
The full process
1. Define the goal (30 minutes)
Start with a one-paragraph spec. Good example:
“I want an agent that can take a GitHub issue URL, clone the repo into its container, install dependencies, run the test suite, fix any failing tests using the code editor, and push a new branch with the fixes. The agent must keep the repo state between turns and log every command it runs.”
Write success criteria:
- Agent can survive at least 20 tool calls without losing context
- All actions are logged and reviewable
- Human can intervene with a simple approval step before
git push
2. Shape the spec & prompt (prompt engineering)
Use this starter system prompt (tested with the new Responses API):
You are an expert software engineer living inside a secure Ubuntu container.
You have full sudo access inside the container but cannot access the internet except through the provided web_search tool.
Rules:
- Think step-by-step before every command
- Use the shell tool for everything that touches the filesystem or runs code
- Never run destructive commands without first explaining what you will do
- After every major change, run relevant tests and show output
- Keep a running log file at ~/agent.log with timestamped entries
- When you need human input, output exactly: [HUMAN: your question here]
3. Scaffold the project
Create a minimal Next.js or Python FastAPI wrapper that calls the Responses API.
from openai import OpenAI
import json
client = OpenAI()
def run_agent_turn(input_text: str, previous_response_id: str = None):
tools = [
{"type": "web_search"},
{"type": "file_search"},
{
"type": "computer_use",
"container": "container_auto", # OpenAI provisions and manages the container
"persist": True
}
]
response = client.responses.create(
model="gpt-4o-2024-08-06", # or newer computer-use capable snapshot
input=input_text,
tools=tools,
previous_response_id=previous_response_id,
temperature=0.2,
max_output_tokens=4096
)
return response
4. Implement the agent loop
def agent_loop(goal: str, max_turns=30):
response_id = None
history = []
for turn in range(max_turns):
user_msg = goal if turn == 0 else "Continue. Current log:\n" + open("agent.log").read()[-800:]
resp = run_agent_turn(user_msg, response_id)
response_id = resp.id
# Extract shell actions for logging and review
for output in resp.output:
if output.type == "computer_use":
print(f"🖥️ {output.action} {output.command}")
# Optional: send to Slack/approval queue here
if "HUMAN:" in output.text:
print(output.text)
human_reply = input("Your input: ")
# feed human_reply back in next turn
history.append(resp)
if "TASK_COMPLETE" in resp.text.upper():
break
return history
5. Validate safely
Follow this checklist before every run:
- Start with a throwaway container (use
container_autowith short TTL) - Run the agent against a public test repo first
- Add a human-in-the-loop gate before any
git push,rm -rf, or network calls - Capture full response trace using the new Agents SDK observability
- Measure cost per successful task (Responses API is priced per input/output token + container runtime seconds)
- Review the
~/agent.logand container filesystem diff after completion
Benchmark reality check: OpenAI reports 38.1 % success on OSWorld benchmark. Expect the agent to get stuck or make dangerous suggestions roughly 6 out of 10 complex tasks. Always treat it as a very capable junior engineer that still needs code review.
6. Ship it safely
Production deployment pattern:
- Wrap the agent in a queue-backed worker (Celery, Temporal, or simple Redis queue)
- Store
response_idand container metadata in your database so you can resume sessions - Expose a review UI that shows every shell command before execution (use the new Agents SDK tracing)
- Add explicit allow-list of safe commands or directories
- Set hard timeout and cost budgets per agent run
Example safety middleware:
def safe_shell_command(cmd: str):
dangerous = ["rm -rf /", "sudo rm", ":(){ :|:& };:"] # fork bomb etc.
if any(x in cmd.lower() for x in dangerous):
raise ValueError("Dangerous command blocked")
return cmd
Copy-paste prompts or snippets
Resume prompt for multi-turn sessions:
Previous session ended with response_id {id}. The current directory contains these files: {ls_output}. Continue working toward: {original_goal}
Debugging prompt when agent is stuck:
You tried to run
{last_command}and received this output: {error}. Diagnose the problem, propose a fix, and continue.
Pitfalls and guardrails
### What if the agent keeps running the same failing command?
Force it to write its reasoning to ~/agent.log every turn and include the last 15 lines of the log in the next prompt. Add temperature=0 for more deterministic behavior.
### What if container costs explode?
Set max_output_tokens low and add a hard timeout in your wrapper. Monitor usage via OpenAI’s usage dashboard — container runtime is billed separately from tokens.
### What if the agent tries to exfiltrate data?
The Computer Use tool runs inside a sandboxed container with network restrictions. Still, never give the agent secrets or production credentials. Use the official system card guidance on adversarial risks.
### What if I need persistent storage across different agent runs?
Mount an external volume or use the File Search tool + your own vector store for long-term memory. The container itself is ephemeral even with persist=True unless you explicitly save artifacts.
What to do next
- Ship your first scoped agent against a toy repo today
- Add a simple approval UI for shell commands
- Measure success rate and average turns per task
- Explore combining Computer Use with the new built-in File Search and web_search tools
- Contribute your agent pattern to the open-source Agents SDK examples
The Responses API + Computer Use combination is the clearest signal yet that OpenAI is moving from model provider to full agent platform. Builders who learn the safe patterns now will have a massive advantage when reliability improves.
Sources
- Original announcement: https://openai.com/index/equip-responses-api-computer-environment
- Microsoft Azure blog on Responses API and Computer-Using Agent
- VentureBeat coverage of Responses API + Shell Tool
- GitHub guide by Dicklesworthstone: https://github.com/Dicklesworthstone/guide_to_openai_response_api_and_agents_sdk
- OpenAI System Card updates on agentic safety and OSWorld benchmark (38.1 %)
Word count: 1,248