Giving AI Agents Memory That Learns — Jake Broekhuizen, LangChain

Testing. Perfectly. Good morning everyone. I Good morning everyone. I know this is the last know this is the last know this is the last day of the conference, day of the conference, day of the conference, so I'm really glad so I'm really glad so I'm really glad to see you here. to see you here. to see you here. I hope I can I hope I can I hope I can hold your attention for the hold your attention for the hold your attention for the next 15-18 minutes. next 15-18 minutes. next 15-18 minutes. My name is Jake, I My name is Jake, I My name is Jake, I lead the lead the lead the lab team here at lab team here at lab team here at LangChain. Today I want to LangChain. Today I want to LangChain. Today I want to talk about a topic I've talk about a topic I've discussed a lot with our discussed a lot with our team and many team and many team and many agent developers agent developers — how to give your — how to give your — how to give your agents memory. Whether agents memory. Whether agents memory. Whether you're building an agent you're building an agent you're building an agent for coding, for coding, for coding, support, or support, or support, or research, research, research, most teams most teams most teams come to the come to the come to the same conclusion: same conclusion: same conclusion: it should get it should get it should get better with each better with each better with each run. So run. So run. So let me give you an let me give you an let me give you an example that you can example that you can example that you can remember for remember for remember for today's conversation. today's conversation. today's conversation. Recently, the Recently, the Recently, the LangChain Lab team LangChain Lab team LangChain Lab team worked with a worked with a worked with a company in the company in the company in the financial sector financial sector financial sector that is developing an agent that is developing an agent that is developing an agent for spending, for spending, for spending, budgeting, and budgeting, and budgeting, and providing (non- providing (non- providing (non- investment) advice. investment) advice. investment) advice. Given the strict Given the strict Given the strict regulation of regulation of regulation of financial services, you financial services, you financial services, you understand that he must understand that he must understand that he must adhere to certain adhere to certain adhere to certain clear language clear language clear language guidelines. So, the way guidelines. So, the way guidelines. So, the way he speaks, his he speaks, his he speaks, his vocabulary, and the vocabulary, and the vocabulary, and the way he interacts with way he interacts with way he interacts with different different different users are users are users are indicators of whether indicators of whether indicators of whether he is maintaining the he is maintaining the he is maintaining the proper tone. He proper tone. He proper tone. He develops this tone develops this tone develops this tone based on based on based on context, memories, and context, memories, and context, memories, and references to his references to his references to his rules and policies.

rules and policies. You can imagine that in You can imagine that in certain contexts, which is what certain contexts, which is what certain contexts, which is what we encountered, the we encountered, the we encountered, the agent's tone could agent's tone could agent's tone could change. He would change. He would change. He would shift from being shift from being shift from being advisory and advisory and advisory and helpful to being helpful to being helpful to being direct and direct and direct and directive, saying, directive, saying, directive, saying, "Users in your "Users in your "Users in your situation usually situation usually situation usually set a set a set a spending limit," or even, "Oh spending limit," or even, "Oh , you should actually , you should actually , you should actually cancel a few cancel a few cancel a few subscriptions and subscriptions and subscriptions and put the money toward put the money toward put the money toward savings." savings." savings." The company we The company we The company we work with considers this a work with considers this a work with considers this a violation of tone, and there have violation of tone, and there have violation of tone, and there have been other similar been other similar been other similar cases. To cases. To cases. To fix this, fix this, fix this, you have to perform a you have to perform a you have to perform a manual check. manual check. manual check. Someone needs to Someone needs to Someone needs to analyze the logs, analyze the logs, analyze the logs, understand where the tone was understand where the tone was understand where the tone was broken, make broken, make broken, make changes to the context the changes to the context the changes to the context the agent agent agent is referring to, and then is referring to, and then is referring to, and then make sure no make sure no make sure no regressions have occurred. regressions have occurred. regressions have occurred. Obviously, this doesn't Obviously, this doesn't Obviously, this doesn't scale, or scale, or scale, or it's it's it's quite difficult to scale. Therefore, quite difficult to scale. Therefore, the system around the the system around the agent should become an agent should become an agent should become an environment where it environment where it environment where it can continuously can continuously can continuously learn and learn and learn and develop develop develop based on its based on its based on its interactions and experiences. interactions and experiences. interactions and experiences. And it is this And it is this And it is this version of memory that I version of memory that I version of memory that I want to want to want to talk about today. To talk about today. To talk about today. To start, I want start, I want start, I want to talk about a to talk about a to talk about a problem that I problem that I problem that I think think think many teams have encountered. I think many teams have encountered. I think many teams have encountered. I think we're all here today, we're all here today, we're all here today, at this conference, at this conference, at this conference, because we're curious, we're because we're curious, we're because we're curious, we're creating agents.

creating agents. creating agents. I think we learned to I think we learned to observe them very well. But observe them very well. But observing and observing and observing and turning those turning those turning those observations and observations and observations and agent experiences into agent experiences into agent experiences into lessons for future lessons for future lessons for future launches are launches are launches are obviously very obviously very obviously very different things. And that's where different things. And that's where different things. And that's where memory memory comes into play. Agents are generating Agents are generating more signals and more signals and more signals and traces than ever traces than ever traces than ever before. And this, of course, will before. And this, of course, will before. And this, of course, will only grow as only grow as more agents are created. This is more agents are created. This is great for us great for us great for us because every run because every run because every run creates a trace, and that creates a trace, and that creates a trace, and that trace trace trace is stored somewhere. This is is stored somewhere. This is is stored somewhere. This is incredibly useful. incredibly useful. incredibly useful. This means we This means we This means we can understand what can understand what can understand what tools tools tools were used, were used, were used, how the agent reasoned on its how the agent reasoned on its how the agent reasoned on its way to a solution, and way to a solution, and way to a solution, and what artifacts it what artifacts it what artifacts it took into account. took into account. Observability is obviously a is obviously a is obviously a critical critical critical foundation for creating foundation for creating foundation for creating reliable agents. But reliable agents. But reliable agents. But I believe the I believe the I believe the real challenge real challenge real challenge comes after the comes after the comes after the trace, the log or record of the trace, the log or record of the trace, the log or record of the agent's actions, has already agent's actions, has already agent's actions, has already been saved. If the been saved. If the been saved. If the future context future context future context to which the to which the to which the agent refers never agent refers never agent refers never changes, then a changes, then a changes, then a faulty skill or faulty skill or faulty skill or instruction instruction instruction will cause the will cause the will cause the error the error the error the agent made agent made agent made today to today to today to persist in persist in persist in future runs.

future runs. The goal or system The goal or system we are aiming for is we are aiming for is we are aiming for is to transform a trace from a to transform a trace from a to transform a trace from a simple log into a simple log into a simple log into a signal source. signal source. Memory. Agents that Memory. Agents that can be observed can be observed can be observed let you know what let you know what let you know what happened. Adaptive happened. Adaptive happened. Adaptive agents are able agents are able agents are able to change and to change and to change and direct themselves direct themselves direct themselves during subsequent during subsequent during subsequent runs. So, I runs. So, I runs. So, I mentioned memory. mentioned memory. mentioned memory. I think it's worth I think it's worth I think it's worth defining what exactly I mean defining what exactly I mean defining what exactly I mean by the by the by the word "memory." This is a word "memory." This is a word "memory." This is a persistent context that the persistent context that the agent references to manage agent references to manage future launches future launches . A trace, transcript, or . A trace, transcript, or . A trace, transcript, or log is just evidence of log is just evidence of log is just evidence of what happened. It what happened. It what happened. It only becomes a memory only becomes a memory only becomes a memory when that lesson when that lesson when that lesson becomes a becomes a becomes a persistent context persistent context persistent context that the agent can that the agent can that the agent can refer to during refer to during refer to during future launches. future launches. Now on the slide Now on the slide behind me you see behind me you see behind me you see six points. Facts, six points. Facts, six points. Facts, preferences, patterns, preferences, patterns, preferences, patterns, previous examples, previous examples, previous examples, skills and skills and skills and instructions. I find it instructions. I find it instructions. I find it helpful helpful helpful to divide them into three to divide them into three to divide them into three separate categories. This is a separate categories. This is a separate categories. This is a taxonomy that we taxonomy that we taxonomy that we borrowed from borrowed from borrowed from cognitive science, and cognitive science, and cognitive science, and it is very relevant it is very relevant it is very relevant when we talk about when we talk about when we talk about language agents.

language agents. First, it is what the agent First, it is what the agent First, it is what the agent knows. This is often knows. This is often knows. This is often called called called semantic semantic semantic memory. This memory. This memory. This includes things includes things includes things like facts and like facts and like facts and preferences that preferences that preferences that guide how the agent guide how the agent guide how the agent responds. Oh. Next is what the responds. Oh. Next is what the responds. Oh. Next is what the agent experienced firsthand agent experienced firsthand agent experienced firsthand . This is . This is . This is episodic memory. episodic memory. episodic memory. These are learned patterns These are learned patterns These are learned patterns he has seen, past he has seen, past he has seen, past interactions and interactions and interactions and examples. And finally, examples. And finally, examples. And finally, how the agent should how the agent should how the agent should behave. This is also behave. This is also behave. This is also known as procedural known as procedural known as procedural memory. These are memory. These are memory. These are instructions, skills instructions, skills instructions, skills that you are probably that you are probably that you are probably familiar with, and familiar with, and familiar with, and rules that rules that rules that determine his determine his determine his behavior. behavior. Procedural memory is where is where most of the noticeable most of the noticeable results actually come from. So, in the results actually come from. So, in the financial aid example I gave at I gave at I gave at the beginning, the tone changes the beginning, the tone changes the beginning, the tone changes that we wanted to that we wanted to that we wanted to impose on the impose on the impose on the agent and make it agent and make it agent and make it follow follow follow were done by were done by were done by updating its updating its updating its procedural memory. procedural memory. procedural memory. We didn't tell We didn't tell We didn't tell him any new facts or him any new facts or him any new facts or forbid him from forbid him from forbid him from using using using bad words. We bad words. We bad words. We gave him a rule to gave him a rule to follow follow when interacting with when interacting with when interacting with users. This is the users. This is the users. This is the procedural memory procedural memory procedural memory that drives our that drives our that drives our agent. Another agent. Another agent. Another distinction I distinction I distinction I want to point out, and which want to point out, and which want to point out, and which I think is very I think is very I think is very important when important when important when thinking about the thinking about the thinking about the general paradigm of general paradigm of general paradigm of agent memory, is the agent memory, is the agent memory, is the separation between separation between separation between working or working or working or short-term short-term short-term memory and memory and memory and long-term long-term long-term memory. Working memory. Working memory. Working memory is what the memory is what the memory is what the agent refers to agent refers to agent refers to during current during current during current execution. These could execution. These could execution. These could be things like be things like be things like intermediate drafts. These intermediate drafts. These intermediate drafts. These may be the may be the may be the results of the results of the results of the tools. These tools. These tools. These may be files may be files may be files obtained during its

obtained during its obtained during its operations. Everything he operations. Everything he operations. Everything he needs to needs to needs to do his current do his current do his current job. Long-term job. Long-term job. Long-term memory is what memory is what memory is what typically exists in typically exists in typically exists in some some some data structure or storage where data structure or storage where instructions, skills, instructions, skills, and information are stored that and information are stored that and information are stored that it needs to it needs to it needs to access in access in access in future iterations future iterations future iterations and for ongoing and for ongoing and for ongoing behavior. Different behavior. Different behavior. Different agent systems agent systems agent systems use and use and use and implement implement implement long-term memory long-term memory long-term memory differently, regardless differently, regardless differently, regardless of whether of whether of whether you are building your own SDK or you are building your own SDK or you are building your own SDK or using using using popular agent popular agent popular agent frameworks. Some frameworks. Some frameworks. Some implement it implement it implement it directly into the directly into the directly into the request (prompt). Some request (prompt). Some request (prompt). Some use use use tools to tools to tools to obtain it. Some obtain it. Some obtain it. Some pull it pull it pull it directly from directly from directly from storage. Some storage. Some storage. Some load via load via load via files, and some files, and some files, and some change state at change state at change state at runtime. runtime. runtime. Implementations may Implementations may Implementations may vary, but this vary, but this vary, but this distinction is distinction is distinction is really important. really important. really important. Some of the memory Some of the memory Some of the memory or context or context or context accessed by the accessed by the accessed by the agent is temporary, agent is temporary, agent is temporary, while others are persistent while others are persistent while others are persistent and long-lasting, and long-lasting, and long-lasting, useful for useful for useful for guiding guiding guiding future launches. And future launches. And future launches. And I think this I think this I think this distinction is very distinction is very distinction is very important when we important when we important when we think about the think about the think about the traces, logs, and traces, logs, and traces, logs, and artifacts that an agent artifacts that an agent artifacts that an agent creates as it creates as it creates as it runs. Tracing runs. Tracing is not just recording the is not just recording the is not just recording the final final final result. This is a record of result. This is a record of result. This is a record of how our working how our working how our working memory changed memory changed memory changed during execution.

during execution. during execution. Some part of this Some part of this Some part of this has to remain as has to remain as has to remain as history. Some history. Some history. Some part of this needs part of this needs part of this needs to become a sustainable to become a sustainable to become a sustainable long-term long-term long-term context. And that's what context. And that's what context. And that's what creates this very creates this very creates this very useful mental useful mental useful mental model that I model that I model that I or our team at or our team at or our team at LangChain has, and which I LangChain has, and which I LangChain has, and which I think is a very convenient think is a very convenient think is a very convenient way to think about the way to think about the way to think about the relationship between relationship between relationship between working or working or working or short-term short-term short-term memory and memory and memory and long-term long-term long-term memory. This is a memory. This is a memory. This is a read-write cycle. You read-write cycle. You read-write cycle. You may have heard of " may have heard of " sleep computing" or "dream computing. " These terms " These terms " These terms refer to how refer to how refer to how memory is updated memory is updated memory is updated or transferred from or transferred from or transferred from long-term to long-term to long-term to short-term while short-term while short-term while the agent is working, and in the agent is working, and in the agent is working, and in some cases some cases even when it is not even when it is not even when it is not working. This is working. This is working. This is computing while you computing while you computing while you sleep. If we start from the sleep. If we start from the sleep. If we start from the moment moment moment the agent starts, it needs to the agent starts, it needs to the agent starts, it needs to read the appropriate read the appropriate read the appropriate context from context from context from long-term long-term long-term memory into memory into memory into short-term memory. This is short-term memory. This is any information that is any information that is any information that is important for the important for the important for the current task: current task: current task: it could be it could be it could be skills or skills or skills or instructions. You've instructions. You've instructions. You've probably seen probably seen probably seen progressive progressive progressive disclosure in agents disclosure in agents disclosure in agents who hone the who hone the who hone the appropriate skills appropriate skills appropriate skills to perform a to perform a to perform a specific task specific task . This is the phase of transition . This is the phase of transition . This is the phase of transition from long-term from long-term from long-term memory to memory to memory to short-term memory. Then, short-term memory. Then, short-term memory. Then, as it works, the agent as it works, the agent as it works, the agent creates a trail. This creates a trail. This creates a trail. This evidence is very important evidence is very important evidence is very important to us because to us because to us because we now understand we now understand we now understand how it extracts how it extracts how it extracts context, what context, what context, what tool calls it makes, tool calls it makes, tool calls it makes, what decisions it makes, what decisions it makes, what decisions it makes, or how it involves sub- or how it involves sub- agents in its agents in its agents in its work. As I work. As I work. As I mentioned, most mentioned, most mentioned, most evidence should evidence should evidence should remain history.

remain history. remain history. If we were to convert If we were to convert If we were to convert all this evidence into all this evidence into all this evidence into long-term long-term long-term memory, it would only make the memory, it would only make the agent's job more difficult and prevent agent's job more difficult and prevent him from reflecting him from reflecting him from reflecting on future on future on future tasks. Therefore, in tasks. Therefore, in tasks. Therefore, in my opinion, the my opinion, the my opinion, the filtration stage is filtration stage is filtration stage is extremely interesting. extremely interesting. extremely interesting. How do you determine what is a How do you determine what is a How do you determine what is a useful signal and useful signal and useful signal and what should become a what should become a what should become a permanent permanent permanent long-term long-term long-term context? And how do context? And how do context? And how do you decide what should you decide what should you decide what should remain just remain just remain just history? Once history? Once history? Once you've made that you've made that you've made that decision—and we'll decision—and we'll decision—and we'll talk about that in a talk about that in a talk about that in a moment. Once moment. Once moment. Once a decision is made, this a decision is made, this a decision is made, this signal is recorded signal is recorded signal is recorded back into a permanent back into a permanent back into a permanent long-term long-term long-term context. In the case of context. In the case of context. In the case of our financial our financial our financial assistant, this should assistant, this should assistant, this should be a measured and non- be a measured and non- be a measured and non- prescriptive approach prescriptive approach prescriptive approach so that he can so that he can so that he can adjust his adjust his adjust his behavior in behavior in behavior in the future. This is the the future. This is the the future. This is the main cycle. main cycle. main cycle. Memory informs the Memory informs the Memory informs the execution process. execution process. execution process. Execution creates Execution creates Execution creates evidence. We filter evidence. We filter evidence. We filter this evidence to this evidence to this evidence to get a signal. And get a signal. And get a signal. And then we use then we use then we use this signal to this signal to this signal to refresh the memory. refresh the memory. refresh the memory. If you can If you can If you can build such a build such a build such a flywheel, we flywheel, we flywheel, we will be closer to will be closer to will be closer to creating very creating very creating very reliable and powerful reliable and powerful reliable and powerful agents that agents that agents that will only get will only get will only get better over time. So, with better over time. So, with better over time. So, with that mental that mental that mental model, I think an model, I think an model, I think an even easier way to even easier way to even easier way to break it down or break it down or break it down or abstract it is to abstract it is to abstract it is to capture events, capture events, capture events, analyze what might analyze what might analyze what might be important, and be important, and be important, and then update then update then update the context so that the context so that the context so that future launches future launches future launches can can can use that.

use that. The implementations of how The implementations of how different people build different people build different people build different frameworks are different frameworks are different frameworks are still evolving, but still evolving, but still evolving, but this three-part this three-part this three-part structure is a very structure is a very structure is a very useful starting useful starting useful starting point. One point. One point. One implementation of this, and I'm implementation of this, and I'm implementation of this, and I'm not claiming it's the not claiming it's the not claiming it's the only one, but just to only one, but just to only one, but just to give some give some give some concrete structure concrete structure concrete structure to what I'm talking about, to what I'm talking about, to what I'm talking about, is the way we is the way we is the way we do it with do it with do it with LangSmith. The LangSmith. The LangSmith. The fixation stage, to be fixation stage, to be fixation stage, to be able to understand and able to understand and able to understand and see the see the see the agent's trajectory and actions, is agent's trajectory and actions, is agent's trajectory and actions, is where our observation comes into play where our observation comes into play where our observation comes into play . You . You . You can see can see can see tooltips tooltips . You can see how . You can see how . You can see how agents have called agents have called agents have called subagents. You can subagents. You can subagents. You can see the context that see the context that see the context that was received. The was received. The was received. The analysis stage in the middle analysis stage in the middle is something new that I'm is something new that I'm is something new that I'm really excited about, but really excited about, but really excited about, but essentially it's an essentially it's an essentially it's an intelligent intelligent intelligent process that does process that does process that does background analysis and background analysis and background analysis and background work to background work to background work to extract signals extract signals extract signals from traces from traces from traces based on things that are important to based on things that are important to based on things that are important to you, and then it you, and then it you, and then it moves that moves that moves that signal into a signal into a signal into a memory store or memory store or memory store or reference point that the agent reference point that the agent reference point that the agent will use will use will use for future for future for future runs. And this is the runs. And this is the LangSmith context hub. So you LangSmith context hub. So you can imagine that can imagine that can imagine that now there's a place where now there's a place where now there's a place where you can do this you can do this you can do this analysis, extract analysis, extract analysis, extract signals, a remote signals, a remote signals, a remote repository where you can repository where you can repository where you can update the content so the update the content so the update the content so the agent can agent can agent can load it on the load it on the load it on the next next next run. This is exactly what the run. This is exactly what the LangChain context hub does . And this abstract . And this abstract . And this abstract cycle is clearly cycle is clearly cycle is clearly superimposed on superimposed on superimposed on it. You capture it. You capture it. You capture the experience, extract the experience, extract the experience, extract the signal, and then the signal, and then the signal, and then store that store that store that reliable context reliable context reliable context for future for future for future launches. To launches. To launches. To talk about this in talk about this in talk about this in more detail, and also to more detail, and also to more detail, and also to outline this in

outline this in outline this in the context of the the context of the the context of the financial financial financial assistant that assistant that assistant that I talked about at the beginning I talked about at the beginning I talked about at the beginning and how we and how we and how we built this cycle. built this cycle. When this agent When this agent is launched, its is launched, its is launched, its context context context is loaded from the is loaded from the is loaded from the context hub, and context hub, and context hub, and this may include its this may include its this may include its instructions, skills, instructions, skills, instructions, skills, related related related markdown files, markdown files, markdown files, policies, and rules. policies, and rules. policies, and rules. When it runs or When it runs or When it runs or when we call when we call when we call this agent, a this agent, a trace is created that trace is created that shows us failed shows us failed shows us failed tool calls tool calls tool calls or or or user fixes, or as user fixes, or as user fixes, or as in this case, it in this case, it in this case, it indicated where our tone indicated where our tone indicated where our tone deviated from what was deviated from what was deviated from what was desired. The engine then desired. The engine then desired. The engine then looks for patterns looks for patterns looks for patterns and determines where and determines where and determines where to change or to change or to change or update the various update the various update the various static markdown skill files static markdown skill files in the context hub, in the context hub, and it makes those and it makes those and it makes those changes. And then, when changes. And then, when changes. And then, when our agent our agent our agent runs the runs the runs the next time, we next time, we next time, we can actually can actually can actually see and understand see and understand see and understand that its tone that its tone that its tone changes as it changes as it changes as it interacts with interacts with interacts with users. So users. So users. So some of the things that some of the things that some of the things that my team and I my team and I my team and I are thinking about right now, or are constantly are thinking about right now, or are constantly are thinking about right now, or are constantly thinking about as we thinking about as we thinking about as we design design design memory—and this memory—and this memory—and this should also ideally should also ideally should also ideally guide the way guide the way guide the way you design you design you design and build some of and build some of and build some of these abstractions—is that these abstractions—is that agents create agents create agents create so much of what we so much of what we so much of what we call "waste call "waste " or kind of " or kind of " or kind of feedback.

feedback. feedback. Not everything has to Not everything has to Not everything has to become a become a become a memory refresher. And in fact, memory refresher. And in fact, memory refresher. And in fact, the devil is in the devil is in the devil is in the details of what it the details of what it the details of what it should become, as we have should become, as we have should become, as we have already discussed in part already discussed in part already discussed in part throughout throughout throughout this presentation. It's this presentation. It's this presentation. It's understanding and understanding and understanding and deciding what's deciding what's deciding what's important to you. important to you. important to you. You know, most of the You know, most of the You know, most of the trace data trace data trace data should actually should actually should actually remain as remain as remain as history, as history history, as history history, as history that can be that can be that can be referenced. Some of it could referenced. Some of it could referenced. Some of it could become datasets become datasets become datasets for for for offline evaluations and offline evaluations and offline evaluations and things like that. things like that. things like that. A small part or A small part or A small part or subset of this should subset of this should subset of this should eventually become what eventually become what eventually become what alters your alters your alters your agent's memory. Now, this agent's memory. Now, this agent's memory. Now, this second point is second point is second point is kind of a trap kind of a trap kind of a trap that I've run into a lot that I've run into a lot , and I think you're kind of , and I think you're kind of , and I think you're kind of walking a razor's edge walking a razor's edge trying to trying to trying to create a system that's create a system that's create a system that's optimized but optimized but optimized but at the same time has at the same time has at the same time has memory storage or updates memory storage or updates available in a " available in a " hot" way. And what hot" way. And what hot" way. And what I found, especially I found, especially I found, especially for background agents for background agents for background agents that run for a long that run for a long that run for a long time, right? Imagine time, right? Imagine time, right? Imagine you have an agent that you have an agent that you have an agent that runs for a runs for a runs for a long time and you long time and you long time and you do some do some do some memory updates at memory updates at memory updates at the beginning of its run, the beginning of its run, the beginning of its run, but because of the but because of the but because of the execution state and the way the execution state and the way the execution state and the way the memory is accessed, you memory is accessed, you memory is accessed, you don't actually don't actually don't actually get to get to get to access it, or it becomes access it, or it becomes access it, or it becomes unavailable for unavailable for unavailable for future runs.

future runs. future runs. So, you know, So, you know, So, you know, understanding what understanding what understanding what can and can and can and can't be cached to can't be cached to can't be cached to make sure that make sure that make sure that memory updates memory updates memory updates are available for are available for are available for future launches is future launches is future launches is very important. I had been banging very important. I had been banging my head against the wall for a while my head against the wall for a while trying to trying to trying to understand why understand why understand why behavior didn't behavior didn't behavior didn't change, and I change, and I change, and I needed to needed to needed to realize and realize and realize and consider this second consider this second consider this second principle. And then the principle. And then the principle. And then the third. I think Open third. I think Open third. I think Open Core and, you know, the Core and, you know, the Core and, you know, the Hermes agent, and a lot of the agents Hermes agent, and a lot of the agents Hermes agent, and a lot of the agents that are becoming widely that are becoming widely that are becoming widely popular, that popular, that popular, that do this do this do this self-updating process, right? I self-updating process, right? I self-updating process, right? I think this is another think this is another think this is another very important aspect very important aspect very important aspect to consider: what to consider: what to consider: what to make to make to make self-updating and what self-updating and what self-updating and what needs needs needs human verification? So, if we're human verification? So, if we're human verification? So, if we're talking about talking about talking about procedural memory with procedural memory with procedural memory with instructions, instructions, instructions, policies, and policies, and tone guidelines, they're probably going tone guidelines, they're probably going to determine to determine to determine about 75% of about 75% of about 75% of my agent's behavior. And so it my agent's behavior. And so it probably has to probably has to probably has to involve some involve some involve some level of human participation level of human participation level of human participation in the process and in the process and in the process and interaction with it. You interaction with it. You interaction with it. You have to have a place have to have a place have to have a place where it can be where it can be where it can be covered. Changes covered. Changes covered. Changes you make to the you make to the you make to the procedural parts of the procedural parts of the procedural parts of the agent's memory before agent's memory before agent's memory before applying them applying them applying them and adding them to the and adding them to the agent's hot restart path. So, protecting agent's hot restart path. So, protecting important behavior important behavior important behavior with with with valves is probably valves is probably valves is probably another, third another, third another, third level of importance, level of importance, level of importance, which I think is very which I think is very which I think is very essential. So I'll essential. So I'll essential. So I'll go back to that go back to that go back to that opening refrain. The opening refrain. The opening refrain. The next launch should next launch should next launch should be better. Memory be better. Memory be better. Memory allows us to allows us to allows us to do this. The word "memory do this. The word "memory " is used very " is used very " is used very often. I think in often. I think in often. I think in the context of building the context of building the context of building agents, working with agents, working with agents, working with tools, and tools, and tools, and providing context to providing context to providing context to your agent, your agent, your agent, memory really gives memory really gives memory really gives us that opportunity.

us that opportunity. us that opportunity. And it's more than just a And it's more than just a And it's more than just a place to place to place to store store store information. This is the way in information. This is the way in which experience becomes which experience becomes which experience becomes context. And then context. And then context. And then this context provides this context provides this context provides an opportunity to improve an opportunity to improve an opportunity to improve our next our next our next launches. By the way, if launches. By the way, if launches. By the way, if you're interested in a real-world you're interested in a real-world you're interested in a real-world implementation of this, there implementation of this, there 's a video in the QR code where I 's a video in the QR code where I 's a video in the QR code where I show how I created an show how I created an show how I created an agent that does agent that does agent that does exactly what I just exactly what I just exactly what I just described. I hope described. I hope described. I hope this makes the concept this makes the concept this makes the concept tangible and helps tangible and helps tangible and helps connect certain things. connect certain things. connect certain things. If you would like to If you would like to If you would like to discuss this discuss this discuss this in more detail, in more detail, in more detail, please come to please come to please come to me. We are at Eugene booth me. We are at Eugene booth me. We are at Eugene booth 19. I will be happy to.