Screen Agents (UI Navigation, OmniParser)

“The ultimate API: The User Interface.”

1. Introduction: The Universal API

In the world of software automation, we usually rely on APIs. If you want to automate Salesforce, you use the Salesforce REST API. If you want to automate AWS, you use the Boto3 SDK. But what if there is no API?

• “Can your agent book a flight on Ryanair?” (No public API, aggressive bot protection).
• “Can your agent change a setting in a Legacy Windows 95 ERP app?” (No API exists).
• “Can your agent use Photoshop to remove a background?” (API requires complex scripting).

But all software has a User Interface (UI). A Screen Agent is an AI that interacts with software exactly like a human: It looks at the screen (Pixel input), moves the mouse (Coordinate output), and types on the keyboard. This turns every application on earth into an API.

This represents the next generation of RPA (Robotic Process Automation).

• Old RPA (UiPath): “Click at (500, 200).” If the window moves or the button changes color, the bot breaks.
• New AI Agents: “Find the Submit button and click it.” If the button moves, the agent sees it and adapts.

---

2. Representations: How to “Read” a Screen

To an agent, a computer screen is just data. But what format should that data take? There are three main approaches, each with massive trade-offs.

2.1 The Visual Approach (Pixels)

We take a screenshot (JPEG). We send it to a Multimodal LLM (GPT-4o / Claude 3.5 Sonnet).

• Pros:
  • Universal: Works on Web, Desktop (Windows/Mac/Linux), Games, and even remote video feeds (Citrix/VNC).
  • Robust: It sees what the user sees. If a CSS bug makes a button invisible, the agent knows it’s invisible.
• Cons:
  • Expensive: High-res screenshots cost many tokens ($0.01 per step). A 50-step workflow costs $0.50.
  • Slow: Vision processing takes 2-3 seconds.
  • Coordinate Hallucination: As discussed in Day 22, models struggle to give exact (x,y) coordinates for clicks.

2.2 The Structural Approach (DOM / Accessibility Tree)

Web browsers have the DOM (Document Object Model). Operating Systems have the Accessibility Tree (Microsoft UI Automation). Instead of pixels, we scrape this tree and feed a text representation to the LLM.

[
  {"id": 12, "role": "button", "name": "Submit", "bbox": [100, 200, 150, 230]},
  {"id": 13, "role": "input", "label": "Email", "value": ""}
]

• Pros:
  • Precise: We get exact coordinates from the OS. No hallucination.
  • Cheap: Text is cheaper than images.
• Cons:
  • Noise: A modern webpage (like Amazon) has 5,000 DOM nodes. 95% of them are <div> wrappers with no semantic meaning. Using the raw DOM bloats the context window (“Context Poisoning”).
  • Blindness: The DOM might say a button exists, but visually it might be covered by a popup or hidden by opacity: 0. The agent clicks it, but nothing happens.

2.3 The Hybrid Approach (OmniParser)

The state-of-the-art approach (used by Microsoft’s OmniParser or Apple’s Ferret-UI) combines both. We act like a human: We use vision to identify where things are, and structure to identify what they are.

The OmniParser Pipeline:

1. Icon Detection: Run a specialized Yolo-based model to detect actionable regions (Buttons, Icons, Inputs) purely visually.
2. Captioning: Run a micro-LLM to describe each region (“Gear Icon”, “Search Bar”, “Profile Pic”).
3. Structuring: Output a simplified XML/JSON that contains only the actionable elements detected visually.
   • Result: The LLM receives a list of 20 relevant items instead of 5,000 DOM nodes.

---

3. Set-of-Marks (SoM) Prompting

The biggest challenge in Screen Agents is Grounding: The gap between “I want to click the search bar” (Intent) and “Move Mouse to 500, 50” (Action).

Set-of-Marks (SoM) is a prompting technique that solves this by modifying the input image.

1. Pre-processing: We assume we have a list of bounding boxes (either from the DOM or an Object Detector).
2. Overlay: We programmatically draw bright, high-contrast Bounding Boxes with numeric labels ([1], [2], [3]) directly onto the screenshot.
3. Prompt: “What is the ID of the search bar?”
4. LLM Input: The LLM sees the screenshot with numbers plastered all over it.
5. LLM Output: “The search bar is labeled [15].”
6. Action: The system (Python glue code) looks up the center coordinates of Box #15 and executes the click.

This offloads the “Spatial Reasoning” (finding where pixels are) to the pre-processor, allowing the LLM to focus on “Semantic Reasoning” (deciding which number to click).

---

4. Navigation Strategy: The ReAct Loop

Navigating a UI is a Graph Search problem. The “States” are screens. The “Edges” are clicks. The agent operates in a ReAct Loop (Reason -> Act -> Observe).

Goal: “Buy a pair of black running shoes on Amazon.”

Step 1:

• Observation: Screenshot of Amazon Home Page.
• Thought: I do not see shoes. I see a search bar. It is Box [5].
• Action: Click(5), Type("Black running shoes"), Press(Enter).

Step 2: (Wait for page load - simple wait or check “Loading” spinner).

• Observation: Screenshot of Search Results.
• Thought: I see many shoes. I need to filter for specific ones. I see a “Black” color filter on the left sidebar (Box [22]).
• Action: Click(22).

The “Scroll” Problem: Websites are taller than the screen (The “Fold”). The agent cannot see the footer or results below the fold.

• Heuristic: If the target isn’t found in the current viewport, the agent must output a ScrollDown() action.
• Memory: The agent must track “I already scrolled down twice, and I still don’t see it.” Otherwise, it enters an infinite scroll loop.

---

5. Security and Safety

Giving an AI control of your mouse and keyboard is inherently dangerous.

1. The “Delete All” Risk:
   • User: “Clean up my temporary files.”
   • Agent: Opens Explorer. Selects C:/Windows. Hits Delete.
   • Why? The agent optimizes for “Cleaning”, not “OS Stability” unless explicitly constrained.
2. PII Reduction:
   • Agent takes a screenshot of your bank portal.
   • Sends it to OpenAI/Anthropic cloud.
   • Risk: Financial data is now logging in a third-party server.
3. Recursive Spawning: A generic “Open Terminal” command might lead the agent to open 500 terminal windows until the machine crashes (Fork Bomb).

Mitigations:

• Human in the Loop: Critical actions (Delete, Send Email, Transfer Money, Terminal Command) require explicit user confirmation.
• Sandboxing: Run Screen Agents inside a Docker Container or a disposable Virtual Machine (e.g., E2B, Firecracker). Never run them directly on your host OS with admin privileges.

---

6. Summary

Screen Agents are the bridge from “Chatting” to “Doing”.

• Hybrid Sensing: DOM for precision, Vision for robustness.
• Set-of-Marks: The standard for accurate clicking.
• Navigation: A graph traversal problem requiring memory (scrolling).
• Safety: Sandboxing is not optional; it is mandatory.

In the next post, we will generalize this from “Screens” (2D digital worlds) to the “Physical World” (3D). We will look at Object Detection (YOLO) and Segmentation, establishing the foundations for agents that interact with physical reality (Robotics).