Gyoktu Dyonmez

AI Engineer Intern

Alaa Abboud

AI Engineer

Yassine Benyahia

AI Engineering Lead

Moe Haidar

Head of Agentic AI Engineering

AI agents can solve thousands of IT problems and remember none of them. We built a memory system for Spark that turns past support conversations into knowledge the agent can reuse on future incidents. In our evaluation, memory increased resolution rate by 9.8% while reducing reasoning steps by 10.1%.

Without memory, an agent can diagnose a recurring Microsoft Teams problem on Monday, encounter the same problem on Tuesday, and begin again with no recollection of what worked the day before. The model brings general knowledge, but the experience accumulated by the deployed agent disappears unless we deliberately preserve it.

Our system converts Spark’s past support conversations into compact, category-specific 

 that evolve as new cases arrive. These skills capture recurring relationships between symptoms, root causes, and solutions, giving the agent a distilled view of what has worked in its own environment.

This raises a harder problem than simply storing conversations. Production data is noisy: Users describe the same symptoms in different ways, successful workarounds can look like permanent fixes, and the agent’s earlier conclusions might be wrong. A useful memory system therefore has to do more than remember: it has to extract, consolidate, rank, and sometimes forget.

In this post, we explain how we built that system, why we split what an agent learns broadly from what it learns about one specific case, and what changed once Spark began learning from its own experience.

For each IT category and customer, we keep a structured memory artifact, modeled internally as a weighted tripartite graph. Symptoms, root causes, and solutions are defined as nodes. Edges connect symptoms to root causes and root causes to solutions that have worked. Each connection carries the set of conversations where that link held true, and its weight increases as more conversations confirm it.

skill-graph-animation.mp4

Figure 1: Memory updates, represented as a tripartite graph accumulating over time. 

At inference time, we render this graph into a compact skill document for Spark. Root causes are listed in order of their cumulative evidence, each followed by its linked symptoms and solutions. The graph is the memory and the skill is the interface Spark actually reads.

Skill Rendered Figure.jpg

Figure 2: Simplified view of a rendered skill, organized by root cause, with weighted symptom and solution links.

This format solves two problems with retrieving raw conversations. First

full transcripts are long and contain irrelevant details which

crowd out the signal Spark actually needs. Second, retrieval over raw conversations is fragile: two cases can share vocabulary while requiring different fixes, so the most similar transcript is not always the most useful one.

A rendered skill gives Spark the compressed advice an experienced technician might give a new colleague: 

When these symptoms show up, check these causes first, and try the fixes that have worked here before”. Spark already has broad knowledge from pretraining; memory adds experience from the customer environment in which it is operating.

We classify each conversation into an IT category, with low-value cases filtered out. 
⁠From there, two systems take over: one proposes what to remember; the other judges it against what's already learned.

 takes in a past conversation and turns it into a proposal: a list of symptoms, a root cause, and a solution. It does not see the existing memory, which reduces the chance that old entries will anchor what the extractor finds.

 compares each proposal with the current memory. It can add a new entry, edit an existing one, reinforce an entry that is already present, or reject the proposal. These are small, scoped operations: each affects a single entry, rather than rewriting the entire skill. Every entry keeps the conversations that support it, so we can inspect why it exists and how often it has appeared.

Separating extraction from judgement reduces self-confirmation. A single model asked to extract knowledge and approve its own extraction tends to accept its first interpretation. The two-model loop creates a second decision point before the memory changes.

The loop can run continuously as new conversations arrive. Repeated symptom-to-root-cause and root-cause-to-solution links gain weight over time, while timestamps let us remove stale information. 

skill-update-loop.mp4

Figure 3: The Two-Stage memory update loop

A rendered skill captures the reusable part of past resolutions. It keeps likely root causes, working fixes, and user symptoms that help Spark choose the correct path. Compression is useful, but it removes details that may matter later, such as the exact sequence of troubleshooting steps, or which device and software version were involved. We keep those details in episodes: a structured, distilled record of each past conversation, still granular enough to preserve the specifics that compression drops

 because they have different retrieval objectives. A graph-based skill is compact enough to load whenever its category is active; it gives Spark a reliable starting point without crowding

the context window with case-specific detail. Episodes, by contrast, are retrieved only when Spark needs more detail than the skill provides for a specific case.

. The model itself acts as the CPU, reasoning over whatever is currently in the context window. Skills are the RAM: fast, compact, per-category knowledge kept close at hand. Episodes are the disk: distilled conversation instances stored and read into context only when a specific past case is worth fetching. However, unlike RAM and disk, everything here ultimately competes for the same resource: tokens in a single context window, shared between the conversation, tool outputs, and Spark's reasoning.

Memory Hierarchy Diagram v3.jpg

Figure 4: Memory hierarchy of skills and episodes, including a simplified example of an episode's content.

There is no simple way to measure how a feature changes the agent's behavior. A troubleshooting investigation

runs over many turns, there is rarely one exact response that counts as “correct”, and the outcome depends on a back-and-forth with a user who reacts to what the agent does. To evaluate the downstream effects of our memory layer, we built a simulated environment that reproduces a real troubleshooting investigation. A simulated user stands in for the employee reporting an issue, while Spark runs its agentic loop against mocked tools and a device state that updates realistically after each action.

Each test scenario is distilled from a real production conversation and defined by the user's issue, its root cause, and the fixes that count as a resolution. Here, we focus on a single category: Microsoft Teams.

and the final outcome using a mix of deterministic metrics and LLM-based judges. To separate signal from run-to-run noise, we ran the full set six times and report the average over those repeats. The individual metrics are presented in the plot below.

Across those six runs, memory made a measurable difference. 

 on the Teams set, while the agent became more targeted:  

10.1% fewer reasoning steps and a clear jump in tool relevance (+16.6%)

, calling for the right tools more often. The improvement is not only that the agent succeeds more often but that it reaches the same answer more directly, and more consistently.

skill-vs-noskill-lift-3metric_2026-06-29.png

Figure 5: Relative improvement in the evaluation environment with skills exposed to Spark.

In addition to the offline simulations, we enabled

memory for six high-volume categories and deployed it internally at Nexthink. Over the following month,

we analyzed 1,338 IT-incident conversations across two months, comparing resolution rates before the rollout to resolution rates after it. We counted an

incident as resolved when the user explicitly confirmed the outcome, or 

an LLM-based classifier judged the conversation's final state to be resolved

The comparison is shown in the figure below.

online-evaluation.png

Figure 6: Relative improvement in incident resolution after enabling memory internally at Nexthink.

Agents should learn and improve as they operate in a new environment. Today, most agents do not. 

Our memory system gives Spark a way to accumulate experience. Each resolved issue can strengthen a known path or add a new one. The evidence behind an entry remains inspectable, while time decay and size limits prevent the memory from growing without control.

There are still hard questions. IT support data is noisy, and resolution is not always a clean label. A workaround may be useful even when the underlying problem remains. Users sometimes leave without confirming whether a fix worked. Human reviewers can disagree about the correct root cause. These ambiguities affect both what the system learns and how we measure it.

We are now extending the system in two directions: bringing episodes into production, and expanding memory coverage across more categories and more of Spark's workload. We also want production outcomes to play a larger role in deciding 

, with evaluation and decay protecting the memory from stale or isolated cases. A mature Spark should enter each conversation with the accumulated experience of the work it has already done, making reliable paths easier to recall.

This direction extends beyond IT support. Foundation models provide broad capability, but a long-running agent also needs to learn the environment in which it operates. Agents that work for months or years will be judged partly by how well they convert experience into better decisions. Model improvements will continue to raise what agents can do on day one. Memory gives them a path to perform better on day 100.

AI agents can solve thousands of IT problems and remember none of them. We built a memory system for Spark that turns past support conversations into knowledge the agent can reuse on future incidents. In our evaluation, memory increased resolution rate by 9.8% while reducing reasoning steps by 10.1%.\nWithout memory, an agent can diagnose a recurring Microsoft Teams problem on Monday, encounter the same problem on Tuesday, and begin again with no recollection of what worked the day before. The model brings general knowledge, but the experience accumulated by the deployed agent disappears unless we deliberately preserve it.\nOur system converts Spark’s past support conversations into compact, category-specific \nagent skills\n that evolve as new cases arrive. These skills capture recurring relationships between symptoms, root causes, and solutions, giving the agent a distilled view of what has worked in its own environment.\nThis raises a harder problem than simply storing conversations. Production data is noisy: Users describe the same symptoms in different ways, successful workarounds can look like permanent fixes, and the agent’s earlier conclusions might be wrong. A useful memory system therefore has to do more than remember: it has to extract, consolidate, rank, and sometimes forget.\nIn this post, we explain how we built that system, why we split what an agent learns broadly from what it learns about one specific case, and what changed once Spark began learning from its own experience.\nA graph-based memory structure\nFor each IT category and customer, we keep a structured memory artifact, modeled internally as a weighted tripartite graph. Symptoms, root causes, and solutions are defined as nodes. Edges connect symptoms to root causes and root causes to solutions that have worked. Each connection carries the set of conversations where that link held true, and its weight increases as more conversations confirm it.\n\n\nFigure 1: Memory updates, represented as a tripartite graph accumulating over time. \nAt inference time, we render this graph into a compact skill document for Spark. Root causes are listed in order of their cumulative evidence, each followed by its linked symptoms and solutions. The graph is the memory and the skill is the interface Spark actually reads.\n\nFigure 2: Simplified view of a rendered skill, organized by root cause, with weighted symptom and solution links.\nThis format solves two problems with retrieving raw conversations. First\n, \nfull transcripts are long and contain irrelevant details which\n \ncrowd out the signal Spark actually needs. Second, retrieval over raw conversations is fragile: two cases can share vocabulary while requiring different fixes, so the most similar transcript is not always the most useful one.\nA rendered skill gives Spark the compressed advice an experienced technician might give a new colleague: \n“\nWhen these symptoms show up, check these causes first, and try the fixes that have worked here before”. Spark already has broad knowledge from pretraining; memory adds experience from the customer environment in which it is operating.\nA Two-Stage memory update loop\nWe classify each conversation into an IT category, with low-value cases filtered out. 
⁠From there, two systems take over: one proposes what to remember; the other judges it against what's already learned.\nThe \nextractor\n takes in a past conversation and turns it into a proposal: a list of symptoms, a root cause, and a solution. It does not see the existing memory, which reduces the chance that old entries will anchor what the extractor finds.\nThe \ncurator\n compares each proposal with the current memory. It can add a new entry, edit an existing one, reinforce an entry that is already present, or reject the proposal. These are small, scoped operations: each affects a single entry, rather than rewriting the entire skill. Every entry keeps the conversations that support it, so we can inspect why it exists and how often it has appeared.\nSeparating extraction from judgement reduces self-confirmation. A single model asked to extract knowledge and approve its own extraction tends to accept its first interpretation. The two-model loop creates a second decision point before the memory changes.\nThe loop can run continuously as new conversations arrive. Repeated symptom-to-root-cause and root-cause-to-solution links gain weight over time, while timestamps let us remove stale information. \n\n\nFigure 3: The Two-Stage memory update loop\nSkills and episodes: A memory hierarchy\nA rendered skill captures the reusable part of past resolutions. It keeps likely root causes, working fixes, and user symptoms that help Spark choose the correct path. Compression is useful, but it removes details that may matter later, such as the exact sequence of troubleshooting steps, or which device and software version were involved. We keep those details in episodes: a structured, distilled record of each past conversation, still granular enough to preserve the specifics that compression drops\n.\nThe two form separate \nmemory layers\n because they have different retrieval objectives. A graph-based skill is compact enough to load whenever its category is active; it gives Spark a reliable starting point without crowding\n \nthe context window with case-specific detail. Episodes, by contrast, are retrieved only when Spark needs more detail than the skill provides for a specific case.\nThis can be thought of as a \nmemory hierarchy\n. The model itself acts as the CPU, reasoning over whatever is currently in the context window. Skills are the RAM: fast, compact, per-category knowledge kept close at hand. Episodes are the disk: distilled conversation instances stored and read into context only when a specific past case is worth fetching. However, unlike RAM and disk, everything here ultimately competes for the same resource: tokens in a single context window, shared between the conversation, tool outputs, and Spark's reasoning.\n\n\nFigure 4: Memory hierarchy of skills and episodes, including a simplified example of an episode's content.\nEvaluation\nThere is no simple way to measure how a feature changes the agent's behavior. A troubleshooting investigation\n \nruns over many turns, there is rarely one exact response that counts as “correct”, and the outcome depends on a back-and-forth with a user who reacts to what the agent does. To evaluate the downstream effects of our memory layer, we built a simulated environment that reproduces a real troubleshooting investigation. A simulated user stands in for the employee reporting an issue, while Spark runs its agentic loop against mocked tools and a device state that updates realistically after each action.\nWe manually curated our golden dataset\n. \nEach test scenario is distilled from a real production conversation and defined by the user's issue, its root cause, and the fixes that count as a resolution. Here, we focus on a single category: Microsoft Teams.\nWe scored improvements\n \non both the troubleshooting sequence\n \nand the final outcome using a mix of deterministic metrics and LLM-based judges. To separate signal from run-to-run noise, we ran the full set six times and report the average over those repeats. The individual metrics are presented in the plot below.\nAcross those six runs, memory made a measurable difference. \nResolution rate increased +9.8%\n on the Teams set, while the agent became more targeted:  \n10.1% fewer reasoning steps and a clear jump in tool relevance (+16.6%)\n, calling for the right tools more often. The improvement is not only that the agent succeeds more often but that it reaches the same answer more directly, and more consistently.\n\nFigure 5: Relative improvement in the evaluation environment with skills exposed to Spark.\nIn addition to the offline simulations, we enabled\n \nmemory for six high-volume categories and deployed it internally at Nexthink. Over the following month,\n \nwe analyzed 1,338 IT-incident conversations across two months, comparing resolution rates before the rollout to resolution rates after it. We counted an\n \nincident as resolved when the user explicitly confirmed the outcome, or \nwhen \nan LLM-based classifier judged the conversation's final state to be resolved\n. \nThe comparison is shown in the figure below.\n\nFigure 6: Relative improvement in incident resolution after enabling memory internally at Nexthink.\nConclusion \nAgents should learn and improve as they operate in a new environment. Today, most agents do not. \nOur memory system gives Spark a way to accumulate experience. Each resolved issue can strengthen a known path or add a new one. The evidence behind an entry remains inspectable, while time decay and size limits prevent the memory from growing without control.\nThere are still hard questions. IT support data is noisy, and resolution is not always a clean label. A workaround may be useful even when the underlying problem remains. Users sometimes leave without confirming whether a fix worked. Human reviewers can disagree about the correct root cause. These ambiguities affect both what the system learns and how we measure it.\nWe are now extending the system in two directions: bringing episodes into production, and expanding memory coverage across more categories and more of Spark's workload. We also want production outcomes to play a larger role in deciding \nwhich knowledge survives\n, with evaluation and decay protecting the memory from stale or isolated cases. A mature Spark should enter each conversation with the accumulated experience of the work it has already done, making reliable paths easier to recall.\nThis direction extends beyond IT support. Foundation models provide broad capability, but a long-running agent also needs to learn the environment in which it operates. Agents that work for months or years will be judged partly by how well they convert experience into better decisions. Model improvements will continue to raise what agents can do on day one. Memory gives them a path to perform better on day 100.

Building Self-Improving IT Agents with Memory