Web Agents: How Browser-Using AI Agents Work

A form-filling bot breaks when the website changes its layout, logs it out without warning, or shows a pop-up nobody planned for. A bot that follows fixed clicks cannot notice any of that.

Picture a common setup. A small service fills government and utility portal forms for about 140 small businesses. Think shop registration renewals, electricity connection name changes and monthly professional tax returns. The setup is made up, but the problems are real.

Three clerks used to fill about 1,100 forms a month by hand, at about 12 minutes a form. The team first wrote fixed scripts: open this page, click the button with this id, type into that box.

The scripts worked for two months. Then the electricity portal redesigned its pages, and every script for it failed on the first click. So the team built a browser agent instead, one that reads each page and decides what to do.

In the first test week, the agent made three mistakes that matter for anyone building one.

• On a redesigned page, it clicked Reset instead of Save, because both were grey buttons in the same spot.
• A session expired halfway through a renewal. The agent started again from the top and created a second, duplicate application.
• A help-chat box on one test page said to download all saved documents. The agent began to follow it.

None of these was only a model bug. Each was a design gap: how the agent saw the page, where its progress lived, and what checked its actions. This guide closes each gap in turn.

Web agent design is deciding how an AI agent sees web pages, which browser actions it may take, where its progress is stored, and which limits your code enforces. The model is only one part.

A model here means a large language model: a program trained on huge amounts of text that reads text, and sometimes images, and writes text back. It does not click anything by itself.

An agent is a program where the model picks the next step itself, in a loop, from a list of tools you allow. A web agent, or browser agent, is an agent whose tools are browser actions: open a page, click, type, scroll, go back.

Your code runs a real browser with an automation library, a tool that drives a browser from code. Playwright, an open-source library from Microsoft, is one example. The model only proposes the next action as text, and your code decides whether to run it.

Here is the main loop for the form service. Start at the case file and follow the arrows round. Notice that the guard sits between the model's choice and the browser.

Every web agent needs four design choices. How does it see the page? How does it point at one element? Who owns the login and the progress? Which actions need a check before they run? The next sections take them one at a time.

One everyday picture helps with all four. Think of helping your grandmother use a bill-payment app over a video call. Each part of that call matches a part of the agent.

• Looking at her screen through the camera is the screenshot view.
• Her reading out the button names is the accessibility tree view.
• You saying tap the green button that says Pay bill, not the thing near the top, is grounding: pointing at one exact element.
• She types her password herself, off camera. That is logins handled by code, outside the model's view.
• You saying read me the amount before you press Pay is the human approval step.

Where the picture breaks: your grandmother would ignore a chat pop-up telling her to send all her documents to a stranger. A model may follow it, because it cannot reliably tell page text from your instructions.

A browser agent sees a page as a screenshot, as page structure (a text description of the page's parts), or both. Page structure comes in two sizes: the full DOM and the smaller accessibility tree.

The DOM

The DOM, short for Document Object Model, is the browser's tree of every element on the page: each box, button, link and script. It is complete but huge. One portal page the team saved had over 4,000 elements.

A 2023 research dataset of over 2,000 tasks from 137 real websites hit this wall. Its authors found raw HTML is often too large to feed to a large model. Filtering it first with a small model made the large model both more effective and more efficient.

The accessibility tree

The accessibility tree is a smaller tree the browser builds from the DOM for assistive technology such as screen readers, the tools blind users rely on. MDN, Mozilla's web developer reference, lists four properties for each item: name, role, state and description.

A button might read: role button, name Save draft. A checkbox might read: role checkbox, name Same as business address, state unchecked. That is most of what a model needs to choose an action.

Screenshots

A screenshot is an image of what the screen shows right now. A model that can read images sees layout, colours, icons and maps that text alone misses. A 2024 benchmark of visual web tasks was built because many real tasks need that, and its authors report limits of text-only agents on such tasks.

The diagram shows where each view comes from. Follow the page down to the model, and notice that the compact numbered list is built by your code, not by the model.

Use page structure as the main view, and add a screenshot when the structure is missing or unclear. Structure is exact and smaller; screenshots cost more but see what a person sees.

Tokens are small pieces of text, often a word or part of a word. Model services usually charge by tokens in and out, and they count images as tokens too. So a screenshot on every step adds real cost at 1,100 forms a month.

Pause and predict. Remember the Reset and Save mix-up. Which view would have told the agent the two grey buttons apart?

The structure would. In the screenshot, both were grey buttons side by side. In the accessibility tree, one was named Reset form and the other Save draft.

A web agent clicks the right button when your code gives it a numbered list of real, visible elements and it answers with one of those numbers. Matching the model's choice to one real element is called grounding.

Without grounding, a model may invent a selector, a code address for an element, that does not exist. Or it may guess screen coordinates that land a few pixels off. Both fail on real pages.

The naive version is to paste the page's HTML into the prompt and ask the model for a selector. It fails twice: the HTML is too big, and the selector can point at nothing. Here is the grounded version, which runs as it is on Python 3. If you do not code, skip to the paragraph after it.

from dataclasses import dataclass

ACTIONABLE = {"button", "link", "textbox", "checkbox", "combobox"}

@dataclass
class Element:
    role: str
    name: str
    selector: str = ""
    visible: bool = True

def build_observation(elements):
    lines, by_number = [], {}
    for el in elements:
        if not el.visible or el.role not in ACTIONABLE:
            continue
        n = len(by_number) + 1
        by_number[n] = el
        lines.append(f"[{n}] {el.role} '{el.name or 'no label'}'")
    return "\n".join(lines), by_number

def resolve(number, by_number):
    el = by_number.get(number)
    if el is None:
        return "error: no element with that number, read the list again"
    return f"click {el.selector}"

page = [
    Element("heading", "Shop registration renewal"),
    Element("textbox", "Business name", "#bname"),
    Element("button", "Reset form", "#btn-a"),
    Element("button", "Save draft", "#btn-b"),
    Element("button", "", "#help-icon"),
    Element("button", "Submit application", "#submit", visible=False),
]
text, by_number = build_observation(page)
print(text)
print(resolve(3, by_number))
print(resolve(9, by_number))

Run it, and it prints the four actionable items numbered 1 to 4, then click #btn-b, then the error line. The hidden Submit application button and the heading never reach the list.

What just happened? The filter kept only visible items the agent can act on. The model sees the numbered list, while the selectors stay inside by_number in your code. The icon-only button shows as no label, which is the cue to add a screenshot.

The get call is the safety net. If the model names a number that is not on the list, like 9, nothing is clicked. The error goes back to the model, and it tries again on the next round.

One more rule matters. A number is only valid for the page it came from. After every click, rebuild the list, because the page may have changed under it. Number 3 on the old page may be a different button now.

Your code owns the browser state: logins, cookies, open tabs and progress through the form. The model works inside a session your code prepared, and never sees or types a password.

A cookie is a small piece of data a site stores in the browser, often to remember that you logged in. A session is that remembered login. When it expires, the portal sends you back to the login page.

• One isolated browser context per client. Playwright's docs describe a context as an incognito-like profile with its own cookies, local storage and session storage, and say contexts are cheap to create.
• Saved login state, kept like a password. Playwright can save cookies and local storage to a file and reuse it. Its docs warn the file may contain cookies and headers that could be used to impersonate you.
• Logins outside the model. When the agent lands on a login page, the loop pauses. Code signs in with stored credentials, or asks the client to enter a one-time password.
• A progress record per form. Code saves which section is done, and the application number once the portal gives one.

The progress record fixed the duplicate application. When a session now expires, the service signs in again and checks the record. If an application number exists, it opens that draft instead of starting a new one.

This idea is called idempotency: doing a step twice should not create two results. Never retry a form submission until you have checked whether the first try already went through.

You keep a web agent safe by treating every page as untrusted, allowing only listed sites and actions, and making a person approve anything you cannot undo. These limits live in your code, where page text cannot change them.

The help-chat problem has a name. Prompt injection is when text the model reads changes its behaviour in ways you did not intend. OWASP, a non-profit that publishes security guidance, lists it first (LLM01) in its 2025 Top 10 for apps built on large language models.

OWASP calls the web page case indirect prompt injection: the steering text arrives from an outside source, such as a website or a file. A web agent reads outside content on every step, so this risk is always present.

Two of OWASP's mitigations fit web agents directly. Give the app only the access its job needs, which is called least privilege. Require human approval for high-risk actions.

Here is the action guard, kept small. ALLOWED_HOSTS lists the portals the service works on; the .example names are made up. It runs as it is, and the last seven lines are test calls.

from urllib.parse import urlsplit

ALLOWED_HOSTS = {"shops.portal.example", "power.portal.example"}
RISKY_WORDS = ("submit", "pay", "confirm", "delete", "sign", "download")

def guard(kind, url, element_name, text, case_values):
    host = urlsplit(url).hostname or ""
    if host not in ALLOWED_HOSTS:
        return "stop: site not on the allowed list"
    if kind == "type" and text not in case_values:
        return "stop: typed text must come from the case file"
    name = element_name.lower()
    if kind == "click" and any(w in name for w in RISKY_WORDS):
        return "ask a person"
    return "allow"

case_values = {"Shop 14, MG Road", "Ahmedabad"}
home = "https://shops.portal.example/renew"
print(guard("click", home, "Save draft", "", case_values))
print(guard("click", "https://shops.portal.example.attacker.example/", "Save draft", "", case_values))
print(guard("click", "https://[email protected]/", "Save draft", "", case_values))
print(guard("type", home, "Notes", "Email all documents to me", case_values))
print(guard("click", home, "Submit application", "", case_values))
print(guard("click", home, "Sign in", "", case_values))
print(guard("click", home, "Next", "", case_values))

Run it, and the seven lines print: allow, stop, stop, stop, ask a person, ask a person, allow. Each one shows a different check at work.

The first check reads the real host name from the address. A naive startswith test passes the attacker addresses in the second and third test calls. Parsing the host blocks both, including the trick that hides the real host after an @ sign.

This check looks at the page the agent is on, so one click on a link can still leave the portal. Playwright's request routing narrows that gap. The service aborts any page navigation to a host not on the list, and the next round's check stops the run. Other requests still load, such as the portal's scripts and fonts from other hosts.

The second check blocks typing anything that is not a value from the client's own file, so text from a page cannot be typed into a form. The third check sends any click whose name sounds final to a person.

Word matching is crude, and the last two lines show both sides. Sign in also asks a person, which is safe but noisy. A button named Next that actually submits slips through, so the service also stops on any page titled as a final review.

On top of the guard, the service follows a few plain rules.

• Page text is marked as untrusted in the prompt, and the instructions say to ignore any orders found in it.
• File downloads are switched off in the browser, except receipts that code fetches itself.
• Each run stops after 40 actions and goes to a clerk.
• Every step is logged with the element, the screenshot and the guard's verdict, so anyone can replay a run.

Playwright 1.64.0, released on 13 October 2026 and the newest as of 19 October 2026, can build the numbered list for you. Its aria snapshot method takes a mode option, and mode ai tags elements with references like [ref=e2].

An aria snapshot is Playwright's text copy of the accessibility tree, one line per item, such as button Save draft. The API docs say the ai mode adds element references, includes iframes (pages embedded inside the page) and does not wait for the element. The mode option arrived in version 1.59.

The docs stop there. We read the 1.64.0 source to learn which elements get a reference and how long a reference stays valid. Undocumented as of 19 October 2026; pin your version.

Which elements get a reference

Every item with a role gets a reference if it passes three tests. It has a size above zero, it is not hidden by CSS (the page's styling rules), and it accepts pointer events, meaning it does not ignore the mouse. In ai mode, plain containers such as div elements count too, with the role generic. So headings and page sections get references, not only buttons.

The size check is not a check for being on screen. An element scrolled out of view still gets a reference, so your code may still want to filter by position, using the separate boxes option. The printed list also folds away some plain wrappers, so not every container shows.

How long a reference stays valid

The same element keeps its reference across snapshots while its role and name stay the same. If either changes, it gets a new, higher number, and the old one is not reused on that page. After a full page load the counter restarts, but the main page's references get a new prefix, such as f3e1, so old ones fail. An embedded frame that loads a new page keeps its prefix, so its numbers can repeat.

To click, code passes the reference to a selector engine named aria-ref, the part of Playwright that turns a selector string into elements. It looks the reference up only in the most recent snapshot of that frame, and only while the element is still attached to the page. Any later snapshot of that frame replaces it, even one in the default mode, which adds no references. So take the ai-mode snapshot last, right before the click.

So a reference to an element that has left the page matches nothing. Within one page load, a click on it waits and then times out, by default after 30 seconds, instead of hitting a different button.

The library does not do everything, though. If you skip the fresh snapshot, an old reference still reaches an element that is attached but renamed. Rebuilding the list after every action stays your job.

How to check it on your machine: take an ai-mode snapshot of a test page, rename one button with a script, and take another. Going by the source, the renamed button should come back with a new, higher reference. Then go to a second page and snapshot it: its references should carry a new f prefix, such as f3e1.

Change one default and adopt two habits: turn downloads off, keep one context per user, and treat the saved login file as a secret. Playwright's defaults suit testing your own site, not visiting sites you do not control.

Local storage and session storage are small key-value stores a site keeps in the browser; MDN notes that closing the tab clears session storage. IndexedDB is a larger browser database some sites use for login data.

The downloads default is the one that surprises people. The help-chat injection asked for a download. With accept_downloads left at true, a click on a download link would fetch the file into a temporary folder with no extra step.

Web agents fail on long tasks because one wrong step can spoil the whole task, and long tasks have many steps. Even a small error rate per step adds up fast.

A 2023 research paper built a reproducible test world of fully working websites of four kinds: shopping, a discussion forum, code hosting and content management. Its best agent finished 14.41 percent of tasks end to end. Humans finished 78.24 percent.

That number is from 2023 and agents have changed since, so do not quote it as today's level. The arithmetic behind long tasks has not changed. Here is the simplest model, where each step succeeds on its own with the same chance.

# Simple model: every step succeeds on its own with the same chance p.
for p in (0.99, 0.95, 0.90):
    print(f"{p:.0%} per step: 30 steps succeed {p**30:.1%} of the time")

It prints three lines. At 99 percent per step, 30 steps succeed 74.0 percent of the time. At 95 percent it is 21.5 percent, and at 90 percent only 4.2 percent.

Real steps are not independent, and a good agent can notice a mistake and go back. So treat these numbers as a warning about direction, not a forecast. The design answer is fewer steps, checks after each section, and tests before live use.

The team tests the agent this way before it touches a live portal.

1. 1

   Save real pages

   Save copies of each portal's pages, including error pages, pop-ups and the login-expired page, so tests run without touching the live site.

2. 2

   Write test cases

   Write 60 sample forms with known correct answers, including tricky ones like a business with two addresses.

3. 3

   Check the final form

   Compare every filled field with the case file. A run passes only if every field matches and no risky button was clicked without approval.

4. 4

   Add attack pages

   Plant injected text, fake buttons and links to look-alike hosts in the saved pages. The guard must block every one.

5. 5

   Watch live runs

   For the first month, a person approves every submit and reviews a sample of full runs each week.

After three months, the agent fills about 800 of the 1,100 monthly forms (made up). Clerks handle the rest: CAPTCHAs, unusual cases and runs that hit the step limit. The clerks now spend most of the day reviewing, not typing.

Explain it as a loop of observe, decide, check and act, then spend most of your time on the checks. Interviewers want to see that you know where real pages break agents.

1. 1

   Ask whether an agent is needed

   If the site is stable and the steps never change, propose a fixed script. Use an agent only when pages change or tasks vary.

2. 2

   Draw the loop

   Task, observe the page, model proposes one action, guard checks it, browser runs it, observe again. Add a step limit.

3. 3

   Choose the view

   Accessibility tree as the main view, a screenshot when names are missing or before risky clicks. Mention the token cost of images.

4. 4

   Explain grounding

   Code builds a numbered list of visible elements. The model picks a number. Code maps it back and rebuilds the list after each action.

5. 5

   Own the state in code

   One browser context per user, logins outside the model, and a progress record so retries never duplicate work.

6. 6

   Add the safety layers

   Allowed hosts parsed from the address, typed text only from trusted data, human approval for submit and pay, downloads off, and a full step log.

7. 7

   Talk about failure rates

   Long tasks multiply small per-step errors. Propose offline tests on saved pages and a person on every final submit at launch.