Mohamed Kafsi

Head of AI Innovation

Dan Tomozei

Data Engineer

AI agents can now use a web browser well enough to be impressive at task automation, but this is not the hard part. The hard part is making them reliable enough to complete company-specific workflows despite customized UIs. This is where context is key.

Systems like Claude Cowork and OpenAI CUA show that browser automation is becoming real: models can observe a UI, reason about the next step, act, and recover when something goes wrong. 

That creates a big automation opportunity in the enterprise where many important workflows still depend on humans navigating web apps by hand. AI agents could take over much of this repetitive work and free employees to focus on higher-value tasks. 

But enterprise work follows processes, conventions, exceptions, and implicit rules that employees learn over time. This constitutes the essential context for an AI agent. Without it, the agent is unable to perform highly specialized tasks. 

 build this context in two complementary ways (1) by observing how employees work, and (2) by letting agents explore sandboxed environments to build a semantic representation of web apps. 

Building the context with our approach unlocks agents’ capability to perform specialized enterprise workflows. 

Traditional browser automation relies on hard-coded scripts: fixed steps with pre-defined selectors. This approach is brittle: a small UI change, such as a button ID changing from 

, can break the workflow. That makes it costly to maintain and hard to scale across dynamic enterprise environments.  

A better alternative is automation that can interpret the page itself, rather than blindly replay fixed scripts. 

AI agents offer the adaptability classic browser automation lack because they (1) see the page layout and content and build a semantic representation that makes them robust to UI changes (2) understand goals in natural language and operate without requiring painstakingly detailed instructions and (3) use the agentic loop observe -> think -> act to increase their likelihood of success through trial and error. 

Web apps are built for humans, not agents. They prioritize UX and customization, not machine readability. That makes automation harder. More machine-readable representations, such as 

, may help over time. However, these approaches are still emerging and will take time to become broadly adopted across enterprise software. 

Feeding raw HTML to the model is not a solution. Page source is too verbose and too noisy: scripts, styles, and deeply nested structures consume LLM context introducing noise and confusion.  

The real opportunity is that agents can now see a web page more like humans do: through screenshots. This gives them access to the semantic structure of the interface, often more stable and informative than raw DOM structure. 

browser_use.png

For browser automation, that is a major shift. 

We can also provide a more compact representation of the page itself. Instead of raw HTML, structured page summaries can expose the key interactive elements in a form that is much easier for the model to use. The Accessibility Tree is a good example thereof, since it captures a comprehensive page layout. The path from its root to a page element provides a clear way to identify the element within the page structure. 

Combined with screenshots, this gives agents both visual understanding and a clear structural view of the page. 

AI agents interacting with web apps typically operate in an 

 loop. First, they have a goal expressed in natural language (Create a new opportunity in Salesforce). Then, they inspect the current page state, reason about the next step, take one action, and repeat until the goal is achieved. 

This makes them more robust than traditional scripts. When the environment changes (a pop-up appears, a page shifts, a step fails) the agent can reassess and try another path instead of breaking immediately. Browser Use agents are a notable example of a successful implementation. 

Why enterprise AI agents need expert context 

General purpose agents do not know the full logic of enterprise applications. This is especially true for dense and highly customized systems such as Salesforce or SAP, where companies adapt workflows, fields, and terminology to their specific business processes. 

Experienced employees can navigate these environments because they have built knowledge over time, often through trial and error, and learnt the company jargon.  

A vanilla agent may handle a large share of common tasks thanks to the model being trained on public datasets, but the long tail of enterprise tasks requires additional context to be provided. To bridge this gap, the most relevant knowledge stems from the employees who perform these tasks every day. 

We ran an internal experiment on a popular web app inside Nexthink. We built a browser extension that collects detailed information about the way Nexthink employees use this specific web application. Over several months, we collected anonymized timestamped clicks and navigation trajectories from 400 unique users. 

We analyzed these trajectories and automatically identified tens of specific task types frequently performed by our employees. These canonical tasks can last from tens of seconds up to several minutes. Most often, a task will consist of filling in compulsory and optional records in a series of forms. The provided records are validated through the UI. The task is typically completed through a final click on a “Save” or “Submit” button, although this is not always the case for more exotic tasks. 

Our method for automatic task discovery relies on detecting changes in the context of a user’s session and we have validated it through extensive benchmarks on manually labeled traces. 

This is the key point: the system learns not just how the application works, but how expert employees use it in the company. The output is a set of task guides that can be injected into an AI agent’s context. 

We then built an AI agent that can infer, from an e-mail, which task to perform and the required information. Using the browser extension and the discovered task guides as context, the agent can automatically complete the workflow. 

task discovery.png

Figure 2: Architecture of the task automation agent. 

What if employee traces are unavailable, sparse, or incomplete? We can generate synthetic ones by sending AI agents into a sandbox version of the application and asking them to complete tasks with only a high-level goal and a limited step budget.

browser-crawler.gif

Figure 3: One hundred agents in action, instructed to complete various tasks. The successful ones contribute to building the semantic graph. 

We applied this approach to Salesforce. Multiple agents, each running in its own browser instance, were asked to complete loosely specified tasks. Their exploration is not exhaustive like classical web crawling. It is goal-driven: at each step, the agent chooses the action that seems most likely to move it toward task completion. Through this approach we capture how tasks can be completed in practice, even without human demonstrations.  

This offline exploration phase mitigates possible failures to complete a task. Among the agents that attempt to complete a task we expect that at least a few of them will succeed. We thus explore the web app and then exploit the successful trajectories. 

In the first phase, we aggregate successful trajectories into a semantic graph of the application that captures the relation between various pages in the application (the “nodes”) through interactions that trigger transitions to the next step (the “edges”). This graph provides to the automation agent useful context in the form of a guiding map of decisions and action paths that get tasks done.  

In the second phase, we correlate the computed semantic graph with user traces to generate aggregated insights on task completion times, struggles, and other useful metrics for business process owners. 

agent crawling.png

Figure 4: The two phases of the agent exploration approach. We build the Semantic Graph of the app from successful synthetic agent task runs, and then we use it to understand tasks performed by employees and to build aggregated insights 

Context matters, and it depends on the task. 

To test this hypothesis, we considered a fixed number of typical Salesforce tasks performed by Nexthink employees. We manually inspected several hundred traces generated by real users and extracted examples of successfully completed tasks. Next, for each task we built a training data set and a balanced test data set from the manually labeled traces. The test data set contains positive examples of successful task completions, and negative examples that either show a different task being performed or an incomplete sequence of interactions where the task was abandoned midway. 

For each task we have built the following task classifier types: 

The first two classifiers identify the most distinguishing feature present in the positive training traces, but not in the negative ones, e.g., the name of the last pressed button, or the URL of the last API call. These classifiers are inherently limited by the training set. They can fail, e.g., if an employee uses Spanish and all training traces are in English, or if the API format is modified in a subsequent web app release. 

The last classifier uses so-called “few-shot learning”: it includes positive traces from the training set when prompting the LLM to classify a new trace. In addition to the provided traces, it exploits the deep LLM knowledge. 

The LLM classifier outperforms the other two classifiers. We show the mean F1-score per task below:  

classifiers2.png

Figure 5 - Mean F1-score per task per classifier type.
⁠

Then, we investigated whether using few-shot learning really helps performance, or whether 0-shot is enough. We again show the mean F1-score per task sorted by the benefit of using few-shot:

context2.png

Figure 6: Comparison between 0-shot and few-shot LLM classifier per-task mean F1-score. The first 3 tasks benefit greatly from context because they are company-specific. 

We notice that with no context, the LLM classifier exhibits dismal performance for the first two tasks. However, as soon as it is provided with a few examples, its performance is boosted by more than 70%. Why? These tasks are company-specific: the web app was modified to include custom business logic for specific business requirements. The LLM has no way of knowing the specifics beforehand. For the other tasks there is no clear benefit of providing examples, the model has enough intuition baked in for nearly perfect performance with no additional context. 

That is the key result. Context does not matter equally for every task, but it matters enormously for the enterprise tail. Generic tasks can often be handled from model knowledge alone. Specialized tasks cannot. There, a small amount of task-specific context is enough to unlock a large gain in reliability. 

The hard part of enterprise automation is building the right context to teach the AI agent how your company works. In this post, we presented two approaches to automatically build this context at scale, from employee traces and from goal-driven exploration in sandbox environments.  Our empirical results show that this context matters most where enterprise automation is hardest: not on generic workflows, but on the company-specific tail, where a small amount of task knowledge added to the agent context can produce large gains in reliability. 

AI agents can now use a web browser well enough to be impressive at task automation, but this is not the hard part. The hard part is making them reliable enough to complete company-specific workflows despite customized UIs. This is where context is key.\nSystems like Claude Cowork and OpenAI CUA show that browser automation is becoming real: models can observe a UI, reason about the next step, act, and recover when something goes wrong. \nThat creates a big automation opportunity in the enterprise where many important workflows still depend on humans navigating web apps by hand. AI agents could take over much of this repetitive work and free employees to focus on higher-value tasks. \nBut enterprise work follows processes, conventions, exceptions, and implicit rules that employees learn over time. This constitutes the essential context for an AI agent. Without it, the agent is unable to perform highly specialized tasks. \nIn this post, we will show how we \nautomatically\n build this context in two complementary ways (1) by observing how employees work, and (2) by letting agents explore sandboxed environments to build a semantic representation of web apps. \nBuilding the context with our approach unlocks agents’ capability to perform specialized enterprise workflows. \nScripts break. Agents adapt. \nTraditional browser automation relies on hard-coded scripts: fixed steps with pre-defined selectors. This approach is brittle: a small UI change, such as a button ID changing from \nlogin-btn-1 \nto \nsubmit-2\n, can break the workflow. That makes it costly to maintain and hard to scale across dynamic enterprise environments.  \nA better alternative is automation that can interpret the page itself, rather than blindly replay fixed scripts. \nAI agents offer the adaptability classic browser automation lack because they (1) see the page layout and content and build a semantic representation that makes them robust to UI changes (2) understand goals in natural language and operate without requiring painstakingly detailed instructions and (3) use the agentic loop observe -> think -> act to increase their likelihood of success through trial and error. \nFirst, give the agents eyes. \nWeb apps are built for humans, not agents. They prioritize UX and customization, not machine readability. That makes automation harder. More machine-readable representations, such as \nllms.txt\n and \nwebMCP\n, may help over time. However, these approaches are still emerging and will take time to become broadly adopted across enterprise software. \nFeeding raw HTML to the model is not a solution. Page source is too verbose and too noisy: scripts, styles, and deeply nested structures consume LLM context introducing noise and confusion.  \nThe real opportunity is that agents can now see a web page more like humans do: through screenshots. This gives them access to the semantic structure of the interface, often more stable and informative than raw DOM structure. \n\nFigure 1: Screenshot taken by Browser Use, a library for browser automation. Page elements are labeled by the agent for better identification. The labels are highlighted and rendered in the browser. This allows the agent to correlate page elements found in the DOM with their position in the page layout.\n\nFor browser automation, that is a major shift. \nWe can also provide a more compact representation of the page itself. Instead of raw HTML, structured page summaries can expose the key interactive elements in a form that is much easier for the model to use. The Accessibility Tree is a good example thereof, since it captures a comprehensive page layout. The path from its root to a page element provides a clear way to identify the element within the page structure. \nCombined with screenshots, this gives agents both visual understanding and a clear structural view of the page. \nObserve-think-act\nAI agents interacting with web apps typically operate in an \nobserve–think–act\n loop. First, they have a goal expressed in natural language (Create a new opportunity in Salesforce). Then, they inspect the current page state, reason about the next step, take one action, and repeat until the goal is achieved. \nThis makes them more robust than traditional scripts. When the environment changes (a pop-up appears, a page shifts, a step fails) the agent can reassess and try another path instead of breaking immediately. Browser Use agents are a notable example of a successful implementation. \nWhy enterprise AI agents need expert context \nGeneral purpose agents do not know the full logic of enterprise applications. This is especially true for dense and highly customized systems such as Salesforce or SAP, where companies adapt workflows, fields, and terminology to their specific business processes. \nExperienced employees can navigate these environments because they have built knowledge over time, often through trial and error, and learnt the company jargon.  \nA vanilla agent may handle a large share of common tasks thanks to the model being trained on public datasets, but the long tail of enterprise tasks requires additional context to be provided. To bridge this gap, the most relevant knowledge stems from the employees who perform these tasks every day. \nTask Discovery: learning from humans \nWe ran an internal experiment on a popular web app inside Nexthink. We built a browser extension that collects detailed information about the way Nexthink employees use this specific web application. Over several months, we collected anonymized timestamped clicks and navigation trajectories from 400 unique users. \nWe analyzed these trajectories and automatically identified tens of specific task types frequently performed by our employees. These canonical tasks can last from tens of seconds up to several minutes. Most often, a task will consist of filling in compulsory and optional records in a series of forms. The provided records are validated through the UI. The task is typically completed through a final click on a “Save” or “Submit” button, although this is not always the case for more exotic tasks. \nOur method for automatic task discovery relies on detecting changes in the context of a user’s session and we have validated it through extensive benchmarks on manually labeled traces. \nThis is the key point: the system learns not just how the application works, but how expert employees use it in the company. The output is a set of task guides that can be injected into an AI agent’s context. \nWe then built an AI agent that can infer, from an e-mail, which task to perform and the required information. Using the browser extension and the discovered task guides as context, the agent can automatically complete the workflow. \n\nFigure 2: Architecture of the task automation agent. \nLearning from synthetic traces\nWhat if employee traces are unavailable, sparse, or incomplete? We can generate synthetic ones by sending AI agents into a sandbox version of the application and asking them to complete tasks with only a high-level goal and a limited step budget.\n\nFigure 3: One hundred agents in action, instructed to complete various tasks. The successful ones contribute to building the semantic graph. \nWe applied this approach to Salesforce. Multiple agents, each running in its own browser instance, were asked to complete loosely specified tasks. Their exploration is not exhaustive like classical web crawling. It is goal-driven: at each step, the agent chooses the action that seems most likely to move it toward task completion. Through this approach we capture how tasks can be completed in practice, even without human demonstrations.  \nThis offline exploration phase mitigates possible failures to complete a task. Among the agents that attempt to complete a task we expect that at least a few of them will succeed. We thus explore the web app and then exploit the successful trajectories. \nIn the first phase, we aggregate successful trajectories into a semantic graph of the application that captures the relation between various pages in the application (the “nodes”) through interactions that trigger transitions to the next step (the “edges”). This graph provides to the automation agent useful context in the form of a guiding map of decisions and action paths that get tasks done.  \nIn the second phase, we correlate the computed semantic graph with user traces to generate aggregated insights on task completion times, struggles, and other useful metrics for business process owners. \n\nFigure 4: The two phases of the agent exploration approach. We build the Semantic Graph of the app from successful synthetic agent task runs, and then we use it to understand tasks performed by employees and to build aggregated insights \n\nContext enables specific workflows\nContext matters, and it depends on the task. \nTo test this hypothesis, we considered a fixed number of typical Salesforce tasks performed by Nexthink employees. We manually inspected several hundred traces generated by real users and extracted examples of successfully completed tasks. Next, for each task we built a training data set and a balanced test data set from the manually labeled traces. The test data set contains positive examples of successful task completions, and negative examples that either show a different task being performed or an incomplete sequence of interactions where the task was abandoned midway. \nFor each task we have built the following task classifier types: \nLast-click classifier\n learns clicks on specific page elements as a signal of task completion (e.g., a “Save” button in a specific page). \nNetwork-request classifier\n learns network API calls to specific endpoints and containing a specific payload as a signal for task completion. \nLLM classifier\n Feeds the sequence of user interactions to an LLM that acts as a classifier. \nThe first two classifiers identify the most distinguishing feature present in the positive training traces, but not in the negative ones, e.g., the name of the last pressed button, or the URL of the last API call. These classifiers are inherently limited by the training set. They can fail, e.g., if an employee uses Spanish and all training traces are in English, or if the API format is modified in a subsequent web app release. \nThe last classifier uses so-called “few-shot learning”: it includes positive traces from the training set when prompting the LLM to classify a new trace. In addition to the provided traces, it exploits the deep LLM knowledge. \nThe LLM classifier outperforms the other two classifiers. We show the mean F1-score per task below:  \n\nFigure 5 - Mean F1-score per task per classifier type.
⁠\nThen, we investigated whether using few-shot learning really helps performance, or whether 0-shot is enough. We again show the mean F1-score per task sorted by the benefit of using few-shot:\n\nFigure 6: Comparison between 0-shot and few-shot LLM classifier per-task mean F1-score. The first 3 tasks benefit greatly from context because they are company-specific. \nWe notice that with no context, the LLM classifier exhibits dismal performance for the first two tasks. However, as soon as it is provided with a few examples, its performance is boosted by more than 70%. Why? These tasks are company-specific: the web app was modified to include custom business logic for specific business requirements. The LLM has no way of knowing the specifics beforehand. For the other tasks there is no clear benefit of providing examples, the model has enough intuition baked in for nearly perfect performance with no additional context. \nThat is the key result. Context does not matter equally for every task, but it matters enormously for the enterprise tail. Generic tasks can often be handled from model knowledge alone. Specialized tasks cannot. There, a small amount of task-specific context is enough to unlock a large gain in reliability. \nConclusion\nThe hard part of enterprise automation is building the right context to teach the AI agent how your company works. In this post, we presented two approaches to automatically build this context at scale, from employee traces and from goal-driven exploration in sandbox environments.  Our empirical results show that this context matters most where enterprise automation is hardest: not on generic workflows, but on the company-specific tail, where a small amount of task knowledge added to the agent context can produce large gains in reliability. 

Reliable Enterprise AI Automation Starts with Context