Stop Telling Agents How — Dan Barrett, Raymond Weitekamp & Jose Montes de Oca, OpenProse

My name is Dan. I am the My name is Dan. I am the creator of Open Pros. creator of Open Pros. creator of Open Pros. This year I have Raymond with me This year I have Raymond with me This year I have Raymond with me , and also , and also , and also Raymond and Jose—two Raymond and Jose—two Raymond and Jose—two amazing amazing amazing teammates, teammates, teammates, colleagues, and just colleagues, and just colleagues, and just phenomenal phenomenal phenomenal creators and engineers. creators and engineers. creators and engineers. I'll tell you a little about I'll tell you a little about I'll tell you a little about them a little later. them a little later. A little bit about what we are A little bit about what we are going to going to going to show you. Five years show you. Five years show you. Five years ago, in April 2021 ago, in April 2021 ago, in April 2021 , I was named the , I was named the , I was named the best prompter in the best prompter in the best prompter in the Eleuther AI Backrooms Discord server Eleuther AI Backrooms Discord server Eleuther AI Backrooms Discord server . At the time, I thought . At the time, I thought . At the time, I thought that this skill would be that this skill would be that this skill would be incredibly valuable for incredibly valuable for incredibly valuable for many years to come, and in many years to come, and in many years to come, and in fact, it fact, it fact, it lasted for lasted for lasted for probably about two or probably about two or three years, maybe three years, maybe three years, maybe its entire its entire its entire lifespan. I lifespan. I lifespan. I say this because, say this because, say this because, as you all as you all as you all know, a lot has changed in the know, a lot has changed in the know, a lot has changed in the last 5 years. But last 5 years. But last 5 years. But one thing has remained one thing has remained one thing has remained unchanged, and today unchanged, and today unchanged, and today I have attended many I have attended many I have attended many master classes. We have master classes. We have master classes. We have many many many amazing amazing amazing performances ahead. Uh, but performances ahead. Uh, but performances ahead. Uh, but there's one thing there's one thing there's one thing that seems to that seems to that seems to always be present always be present always be present in all of our work, in all of our work, in all of our work, and that and that and that doesn't really change with the doesn't really change with the doesn't really change with the latest latest latest hardware or the hardware or the hardware or the latest model: we latest model: we latest model: we all still all still all still need to communicate need to communicate need to communicate our intentions to the our intentions to the our intentions to the device. And device. And device. And no matter no matter no matter how smart the how smart the device becomes, there's still a device becomes, there's still a problem: problem: problem: How do I get what How do I get what How do I get what I want from my brain to a I want from my brain to a superintelligence superintelligence that will just that will just that will just take it over and take it over and take it over and do it for me? Uh

do it for me? Uh , Open Pros is a language, it , Open Pros is a language, it , Open Pros is a language, it 's a framework, it's a 's a framework, it's a 's a framework, it's a project started out of project started out of project started out of my love for my love for my love for language and my strong language and my strong language and my strong belief that language belief that language is a truly is a truly is a truly powerful creative powerful creative powerful creative force. And I mean force. And I mean force. And I mean this; I know it's this; I know it's this; I know it's a bit cliché, but a bit cliché, but a bit cliché, but it's true in the most it's true in the most it's true in the most literal sense, literal sense, literal sense, given the given the given the opportunities opportunities opportunities we have. we have. we have. You can write a You can write a You can write a sentence and sentence and sentence and the device the device the device will simply make that will simply make that will simply make that sentence true sentence true . And then the . And then the . And then the question arises: what question arises: what question arises: what sentence will you sentence will you sentence will you write now write now write now that you have this that you have this that you have this incredibly powerful incredibly powerful incredibly powerful machine that can machine that can machine that can make it true? make it true? make it true? The skill of creating The skill of creating The skill of creating tips that tips that tips that existed 5 years existed 5 years existed 5 years ago is very ago is very ago is very different from the skill of different from the skill of different from the skill of creating or creating or creating or drafting drafting drafting contracts that contracts that contracts that we call we call we call such today. When you such today. When you such today. When you were working with a model were working with a model were working with a model that simply that simply that simply finished processing the finished processing the finished processing the next token, next token, next token, there were good there were good there were good and bad patterns for and bad patterns for and bad patterns for generating hints. generating hints. generating hints. A few years later, A few years later, A few years later, you would fill in you would fill in you would fill in the context manually the context manually the context manually using using using search results, search results, search results, tool calls, and tool calls, and tool calls, and so on, and paste so on, and paste so on, and paste them into the context window them into the context window —this is called —this is called —this is called context context context engineering. Then a special tool engineering. Then a special tool engineering. Then a special tool began to do this for you began to do this for you , and you , and you created it, created it, created it, perceiving it as a perceiving it as a perceiving it as a machine that machine that machine that automatically automatically automatically filled in the context.

filled in the context. Over the last 8-10 Over the last 8-10 months, a ton of articles have appeared months, a ton of articles have appeared that take that take things to another things to another things to another level of abstraction, and the level of abstraction, and the level of abstraction, and the tools themselves tools themselves tools themselves are changing quietly. are changing quietly. This morning we This morning we watched a great watched a great watched a great presentation from Anthropic about presentation from Anthropic about presentation from Anthropic about threads, and it threads, and it threads, and it talks about talks about talks about the idea of ​​a the idea of ​​a the idea of ​​a self-modifying self-modifying self-modifying topology of topology of topology of auxiliary auxiliary auxiliary equipment equipment equipment that manages that manages that manages slave slave slave agents in a way agents in a way agents in a way that would have been that would have been that would have been unthinkable just a year unthinkable just a year unthinkable just a year ago. So if ago. So if ago. So if the problem is solved, the the problem is solved, the the problem is solved, the context is solved, and the context is solved, and the context is solved, and the supporting supporting supporting systems are solved, systems are solved, systems are solved, what's the next step? That's what what's the next step? That's what what's the next step? That's what we'd we'd we'd like to like to like to talk a little bit about today and talk a little bit about today and talk a little bit about today and show you show you show you some of the things some of the things some of the things we're we're we're developing to developing to developing to support this idea. support this idea. support this idea. I imagine it this way I imagine it this way : once you can : once you can : once you can give a name to what you give a name to what you give a name to what you need, any need, any need, any supporting supporting supporting system can system can system can self-organize self-organize , expand, , expand, , expand, perform any perform any perform any cycles to achieve the cycles to achieve the cycles to achieve the goal. And the question is, goal. And the question is, goal. And the question is, what are the best what are the best what are the best practices for naming what practices for naming what practices for naming what you need? And you need? And you need? And we see this we see this we see this as a declarative as a declarative as a declarative approach. We will approach. We will approach. We will borrow borrow borrow concepts from concepts from concepts from classical classical software development. I have software development. I have been working as a been working as a been working as a programmer for 12 years.

programmer for 12 years. programmer for 12 years. Jose and Raymond also Jose and Raymond also Jose and Raymond also have a lot of experience in have a lot of experience in have a lot of experience in this field. Jose this field. Jose this field. Jose was an early was an early was an early developer at Google Cloud developer at Google Cloud and influenced and influenced and influenced many aspects of many aspects of many aspects of our approach to the our approach to the our approach to the topic. topic. topic. Raymond is simply Raymond is simply Raymond is simply phenomenal... He phenomenal... He phenomenal... He asked me to asked me to asked me to introduce myself as an introduce myself as an introduce myself as an outstanding outstanding programmer, but programmer, but that greatly that greatly that greatly underestimates his underestimates his underestimates his talents. He has a talents. He has a talents. He has a PhD, is a two-time PhD, is a two-time PhD, is a two-time company founder, company founder, company founder, and has been and has been and has been designing designing designing user user user interfaces for interfaces for interfaces for almost as long as anyone almost as long as anyone I know. And here's the I know. And here's the idea: idea: idea: borrow old borrow old borrow old engineering patterns and engineering patterns and engineering patterns and apply them to this apply them to this apply them to this new world. We want to new world. We want to new world. We want to write our queries write our queries write our queries compositionally. We compositionally. We compositionally. We want to use want to use want to use composition instead of composition instead of composition instead of inheritance. We inheritance. We inheritance. We want to write our want to write our want to write our queries declaratively queries declaratively , not procedurally or , not procedurally or , not procedurally or imperatively. We want to imperatively. We want to imperatively. We want to tell the system what tell the system what tell the system what to do, not how to to do, not how to to do, not how to do it, because do it, because do it, because the system, in its the system, in its the system, in its infinite wisdom infinite wisdom infinite wisdom and infinite and infinite and infinite intelligence, will figure it out itself intelligence, will figure it out itself intelligence, will figure it out itself . And if we . And if we . And if we tell her how to tell her how to tell her how to do it, we're do it, we're do it, we're actually losing actually losing actually losing sight of the cost and the sight of the cost and the sight of the cost and the intelligence because intelligence because intelligence because we could we could we could use a use a use a cheaper model. A more cheaper model. A more cheaper model. A more intelligent intelligent intelligent model will be able to figure out model will be able to figure out model will be able to figure out how to do this. So how to do this. So how to do this. So the idea is to the idea is to the idea is to write down write down write down what you want, to what you want, to what you want, to state the desired state the desired state the desired outcome, the goal, outcome, the goal, outcome, the goal, the end state.

the end state. the end state. We see this We see this We see this as a contract. And as a contract. And as a contract. And write it in a write it in a write it in a way that allows you to way that allows you to combine it, combine it, because when you because when you because when you create a system, create a system, create a system, when you create an when you create an when you create an open structure open structure open structure like this, when like this, when like this, when you create a system you create a system you create a system that has that has that has many parts, many parts, many parts, maybe within maybe within maybe within your company or your company or your company or your team, you your team, you your team, you want to do things in a want to do things in a want to do things in a way that way that way that allows you to allows you to combine combine elements elegantly and manage elements elegantly and manage elements elegantly and manage complexity more complexity more complexity more locally. With that, I'll locally. With that, I'll locally. With that, I'll hand over to hand over to hand over to Raymond, who'll Raymond, who'll Raymond, who'll tell you a little bit tell you a little bit tell you a little bit about how it about how it about how it works, and then we'll works, and then we'll works, and then we'll show you show you show you some some some demos. We demos. We demos. We will be open to will be open to will be open to questions throughout the questions throughout the presentation. So presentation. So please please please raise your hand if raise your hand if raise your hand if you have any questions. you have any questions. Amazing. Amazing. Thanks, Dan. Okay, Thanks, Dan. Okay, I just wanted to help I just wanted to help you understand Open Pros and you understand Open Pros and you understand Open Pros and think of it as a think of it as a think of it as a quadrant. So quadrant. So quadrant. So if you if you if you only remember two things, here are the only remember two things, here are the only remember two things, here are the two things two things two things I want you to I want you to I want you to remember: it's just remember: it's just remember: it's just Markdown. We sometimes Markdown. We sometimes Markdown. We sometimes call it a call it a call it a programming language for programming language for programming language for agents. Sometimes we agents. Sometimes we agents. Sometimes we call it a call it a call it a framework. But all framework. But all framework. But all this is just text.

this is just text. this is just text. This is just one This is just one This is just one big clue. And big clue. And big clue. And the second thing to the second thing to the second thing to help you understand help you understand help you understand where this all where this all where this all fits in is that this is fits in is that this is fits in is that this is event- event- driven driven software. This is not a software. This is not a chat. We do not communicate with chat. We do not communicate with chat. We do not communicate with these devices. I these devices. I these devices. I will be communicating with an will be communicating with an will be communicating with an agent who will agent who will agent who will later create later create later create Open Pros programs. But Open Pros programs. But Open Pros programs. But if you remember if you remember if you remember just two things about us, that's it just two things about us, that's it just two things about us, that's it . It's all . It's all . It's all text, and we text, and we text, and we use event- use event- driven driven driven multi-agent multi-agent multi-agent workflows, workflows, workflows, not chat. This is not chat. This is open source software open source software . You can . You can . You can run it run it run it locally using locally using locally using NPX skills. I'll show NPX skills. I'll show NPX skills. I'll show you the hosted you the hosted you the hosted version I version I version I created: run.pros.md. And we will created: run.pros.md. And we will created: run.pros.md. And we will do it for do it for do it for you too. If you want to you too. If you want to you too. If you want to contact us, we will contact us, we will contact us, we will help you. So, help you. So, help you. So, quick question: quick question: quick question: how many people here are how many people here are how many people here are from the financial industry? from the financial industry? Fine. And how many of Fine. And how many of them are AI engineers them are AI engineers them are AI engineers or developers? or developers? or developers? Fine. And how many Fine. And how many project managers project managers or specialists are there or specialists are there or specialists are there who use who use who use AI but don’t know how to AI but don’t know how to AI but don’t know how to program? program? Okay, less, but Okay, less, but okay. So why okay. So why okay. So why should I should I should I care about these three care about these three care about these three audiences? As audiences? As audiences? As for smaller for smaller for smaller regulated regulated regulated industries, as I industries, as I industries, as I said, said, said, some banks and some banks and some banks and other financial other financial other financial companies are showing a companies are showing a companies are showing a lot of interest in lot of interest in lot of interest in everything being everything being everything being done in text done in text done in text format. We don't need a format. We don't need a format. We don't need a purchasing department. We don't purchasing department. We don't purchasing department. We don't need an IT department. You need an IT department. You need an IT department. You can use can use can use any any any encoding agent, any encoding agent, any encoding agent, any output engine, output engine, output engine, any any any constraints you already have, constraints you already have, constraints you already have, any any any tech stack, and we tech stack, and we tech stack, and we can help you can help you can help you build build build multi-agent multi-agent multi-agent workflows without workflows without workflows without providing you with providing you with providing you with anything more than

anything more than anything more than a few a few a few Markdown files. This is one of the Markdown files. This is one of the advantages of the advantages of the text format. text format. text format. For those who don't write For those who don't write For those who don't write code themselves, code themselves, code themselves, if you can read if you can read if you can read logical English, logical English, logical English, you can read you can read you can read Open Prose. Therefore, we Open Prose. Therefore, we Open Prose. Therefore, we will deliberately not will deliberately not will deliberately not delve into the delve into the delve into the syntax of the language. syntax of the language. Technically it is a Technically it is a programming language and it programming language and it programming language and it has a syntax, has a syntax, has a syntax, but it is more but it is more but it is more designed to be designed to be designed to be read and read and read and edited edited edited by humans than to be by humans than to be by humans than to be written from scratch written from scratch written from scratch by hand. We don't want by hand. We don't want by hand. We don't want users to users to users to worry about worry about worry about learning the schema. learning the schema. learning the schema. So you can So you can So you can describe the work, describe the work, describe the work, ask your ask your ask your agent to write a agent to write a Prose program, that might give Prose program, that might give you some more you some more you some more ideas about how you ideas about how you ideas about how you might might might edit it, or edit it, or edit it, or you might see you might see something you want something you want something you want to change, and then that to change, and then that Prose program becomes a Prose program becomes a reusable reusable reusable workflow. And for workflow. And for AI engineers and developers, I AI engineers and developers, I think this think this think this approach can be approach can be approach can be useful because useful because useful because it's not a framework, and we'll it's not a framework, and we'll it's not a framework, and we'll talk about talk about talk about this a little bit later, but this a little bit later, but this a little bit later, but even a compiler for a even a compiler for a even a compiler for a framework, even a framework, even a framework, even a virtual machine virtual machine virtual machine for a framework, for a framework, for a framework, is itself is itself is itself just a huge just a huge just a huge help, and you're help, and you're help, and you're literally injecting literally injecting literally injecting Claude code to Claude code to Claude code to compile a program compile a program , injecting Codex to do , injecting Codex to do , injecting Codex to do that. So that. So that. So you can express you can express you can express very, very complex very, very complex very, very complex multi-agent multi-agent multi-agent workflows, workflows, workflows, like like like anthropomorphic, uh, anthropomorphic, uh, anthropomorphic, uh, dynamic ones. What do they dynamic ones. What do they dynamic ones. What do they call them?

call them? call them? Dynamic Dynamic Dynamic agent workflows. Um, agent workflows. Um, agent workflows. Um, that would probably sound like an that would probably sound like an that would probably sound like an analogy. analogy. analogy. You can do this You can do this You can do this in any coding agent in any coding agent using just our using just our Markdown contract. Fine Markdown contract. Fine . So, first of all, . So, first of all, . So, first of all, based on the idea that based on the idea that based on the idea that this is just Markdown, uh, this is just Markdown, uh, this is just Markdown, uh, it could be a whole it could be a whole it could be a whole program, just program, just program, just as an example. And as an example. And as an example. And again, I don't want to again, I don't want to get too hung up on the get too hung up on the syntax, um, but you syntax, um, but you syntax, um, but you can essentially can essentially can essentially say, "Here's the goal, and say, "Here's the goal, and say, "Here's the goal, and here's the contract that here's the contract that here's the contract that supports it. Here is supports it. Here is supports it. Here is the definition of what the definition of what the definition of what supports this supports this supports this contract." Um, and again contract." Um, and again contract." Um, and again , this is just a very , this is just a very , this is just a very simple example. Um, simple example. Um, this idea here, well, this idea here, well, what can you what can you what can you announce? So again announce? So again announce? So again , I don't want to , I don't want to , I don't want to get into schemas and get into schemas and get into schemas and types and syntax, but types and syntax, but types and syntax, but something that you or your something that you or your something that you or your agent can agent can agent can declare on your declare on your declare on your behalf using a behalf using a behalf using a writer function on the writer function on the writer function on the command line command line command line is called a is called a is called a writer function. In our writer function. In our writer function. In our application, it's application, it's application, it's essentially just an agent essentially just an agent essentially just an agent that manages the that manages the that manages the entire application for entire application for entire application for you. You can you. You can you. You can create subagents create subagents . Um, you can . Um, you can . Um, you can assign specific assign specific assign specific skills to those skills to those skills to those specific sub- specific sub- agents. You may agents. You may agents. You may have certain have certain tools available.

tools available. For example, for searching For example, for searching For example, for searching I really like Exa. I really like Exa. I really like Exa. I think it's I think it's I think it's amazing and I don't amazing and I don't amazing and I don't want it to want it to want it to do regular web do regular web searches. I always want searches. I always want searches. I always want him to use him to use him to use Exa because it's Exa because it's Exa because it's faster and cheaper for faster and cheaper for faster and cheaper for me. You can me. You can me. You can say, "Here's a skill say, "Here's a skill say, "Here's a skill for that. Here's the CLI for for that. Here's the CLI for for that. Here's the CLI for that." If you can't that." If you can't that." If you can't find these things find these things , just don't , just don't , just don't run the program run the program . We don't do it the other way around . We don't do it the other way around . We don't do it the other way around . . . So you may have So you may have So you may have skills that are skills that are skills that are specific to specific to specific to your company or your company or your company or your organization. And your organization. And your organization. And then essentially you then essentially you then essentially you can think of can think of can think of it as putting it as putting it as putting this program together, and this program together, and this program together, and if it doesn't have all the if it doesn't have all the if it doesn't have all the dependencies, it doesn't dependencies, it doesn't dependencies, it doesn't compile, right? compile, right? compile, right? Um, and then obviously Um, and then obviously Um, and then obviously what is the input of what is the input of what is the input of this this this program, what is the program, what is the program, what is the output output output that this that this that this program is supposed to program is supposed to program is supposed to provide, this provide, this provide, this contract is supposed to contract is supposed to contract is supposed to provide. Um, and provide. Um, and provide. Um, and again, it might again, it might again, it might be interesting if be interesting if be interesting if people want to people want to people want to go into more detail and go into more detail and go into more detail and ask Dan about ask Dan about ask Dan about how he created this. The how he created this. The how he created this. The runtime itself is runtime itself is runtime itself is also a also a huge clue. huge clue. So, um, you know, So, um, you know, So, um, you know, one of my favorite one of my favorite one of my favorite phrases in a skill or phrases in a skill or phrases in a skill or a tip is, "You are a a tip is, "You are a compiler." This is not compiler." This is not compiler." This is not an analogy. You are a an analogy. You are a compiler. And, um, compiler. And, um, compiler. And, um, it seems to be working it seems to be working . Um, so, um, yeah, and . Um, so, um, yeah, and . Um, so, um, yeah, and then this idea of ​​an then this idea of ​​an then this idea of ​​an event- event- driven driven driven approach. So we approach. So we approach. So we want to get away from the idea want to get away from the idea want to get away from the idea that everything is a conversation that everything is a conversation , that everything is a chat, um, , that everything is a chat, um, , that everything is a chat, um, and build and build and build event- event- driven driven driven systems. So, um, systems. So, um, systems. So, um, again, just kind of again, just kind of again, just kind of comparing these comparing these comparing these different different different workflow trees, workflow trees, workflow trees, it's like something it's like something it's like something happens, it could happens, it could happens, it could just be a just be a just be a scheduled cron job, it scheduled cron job, it scheduled cron job, it could be opening a could be opening a could be opening a new issue on GitHub, new issue on GitHub, new issue on GitHub, or a new or a new or a new pull request, or you pull request, or you pull request, or you have some kind of have some kind of have some kind of webhook system, right? Um,

webhook system, right? Um, webhook system, right? Um, if you have, for example, if you have, for example, if you have, for example, Stripe not working, or Stripe not working, or Stripe not working, or something like that, then something like that, then something like that, then you can run you can run you can run that workflow that workflow that workflow that you described, that you described, that you described, so that it handles the so that it handles the so that it handles the whole process from whole process from whole process from start to finish and start to finish and start to finish and fulfills the contract. fulfills the contract. So the So the idea is to idea is to idea is to start start start building a building a building a task-based system so that task-based system so that task-based system so that you can set you can set you can set it up yourself with it up yourself with it up yourself with your your your agents in your agents in your agents in your system. In run.prose.md I system. In run.prose.md I system. In run.prose.md I created a concept where a created a concept where a created a concept where a task is a task is a task is a contract plus a contract plus a contract plus a trigger. The trigger trigger. The trigger trigger. The trigger could be something could be something could be something very simple, very simple, very simple, like a like a like a schedule, and here schedule, and here schedule, and here I'm essentially saying, " I'm essentially saying, " Okay, run Okay, run Okay, run this every day." But right this every day." But right this every day." But right now, the two main now, the two main now, the two main triggers that we've triggers that we've triggers that we've implemented in the implemented in the implemented in the app are app are webhooks, which essentially webhooks, which essentially cover cover cover a lot of different a lot of different a lot of different things: the Slack app things: the Slack app is just webhooks, is just webhooks, is just webhooks, GitHub is just GitHub is just webhooks, and cron jobs. We'd be webhooks, and cron jobs. We'd be webhooks, and cron jobs. We'd be happy to add happy to add happy to add more if you'd more if you'd more if you'd like to leave like to leave like to leave feedback on features feedback on features feedback on features we can we can we can make easier for you. And make easier for you. And make easier for you. And again, based on again, based on again, based on that trigger, you that trigger, you that trigger, you sort those sort those sort those workflows.

workflows. workflows. So you So you So you can start can start can start creating these, you know creating these, you know , code factories, , code factories, , code factories, creating these creating these creating these ongoing tasks in ongoing tasks in ongoing tasks in Prose programs. What I Prose programs. What I Prose programs. What I want to show, and want to show, and want to show, and what I'll come back to in a what I'll come back to in a what I'll come back to in a second, is that second, is that second, is that one of the things one of the things one of the things I'm I'm I'm developing to developing to developing to make sure it's make sure it's make sure it's ready for this ready for this ready for this event is event is webhooks and using webhooks and using webhooks and using GitHub events, and I'm GitHub events, and I'm GitHub events, and I'm forcing myself to forcing myself to forcing myself to use use use the program to the program to the program to develop the program develop the program . And here is one of the . And here is one of the . And here is one of the examples that examples that examples that we have in the we have in the we have in the system now - these are the two system now - these are the two system now - these are the two main ones. The first is that main ones. The first is that main ones. The first is that any any any GitHub issue that comes in, you write a GitHub issue that comes in, you write a GitHub issue that comes in, you write a Prose program about Prose program about Prose program about how it will how it will how it will try to solve that try to solve that try to solve that problem. You can problem. You can problem. You can make it make it make it as complex as complex as complex or as simple or as simple or as simple as you want. You as you want. You as you want. You can run this can run this can run this as a multi-agent as a multi-agent as a multi-agent workflow. workflow. workflow. He may have He may have He may have certain skills certain skills certain skills that he must that he must that he must follow, or models follow, or models . Basically, the . Basically, the . Basically, the input is the problem, the input is the problem, the input is the problem, the output is the commit, output is the commit, output is the commit, and it has to go and it has to go and it has to go all the way to all the way to all the way to completion of that completion of that completion of that task. Similarly, and task. Similarly, and task. Similarly, and I'll show this in more I'll show this in more I'll show this in more detail, this is how to detail, this is how to detail, this is how to do this using do this using Pull Request (PR) review.

Pull Request (PR) review. So you can So you can So you can do this do this do this event- event- driven driven driven testing. And I don't know if testing. And I don't know if testing. And I don't know if you heard you heard you heard at the Anthropic seminar this morning at the Anthropic seminar this morning at the Anthropic seminar this morning about the problems of about the problems of about the problems of self-verification, self-verification, self-verification, which I which I which I found found found interesting. So interesting. So interesting. So you're working in Claude Code you're working in Claude Code you're working in Claude Code and you say, "Okay. I'll and you say, "Okay. I'll and you say, "Okay. I'll let you let you let you push the changes push the changes push the changes to production, but to production, but to production, but could you please check could you please check could you please check it again? And, it again? And, it again? And, you know, it is biased you know, it is biased , it thinks that it has , it thinks that it has , it thinks that it has done its done its done its job well. And people too, job well. And people too, job well. And people too, right? So, I'll show a right? So, I'll show a right? So, I'll show a workflow where, workflow where, workflow where, essentially, the agents essentially, the agents essentially, the agents who build this who build this who build this application get application get application get backpressure from the backpressure from the backpressure from the application itself, application itself, application itself, through through through pull requests. Fine. So, pull requests. Fine. So, pull requests. Fine. So, as we move on to as we move on to as we move on to that, are there that, are there any questions as any questions as any questions as we switch to the we switch to the we switch to the web app? web app? Please, no Please, no stupid questions. Is this a stupid questions. Is this a stupid questions. Is this a hand or a phone? I don't hand or a phone? I don't hand or a phone? I don't see any light bulbs here. see any light bulbs here. see any light bulbs here. Maybe none. Maybe none. Maybe none. Yes, go ahead. There's Yes, go ahead. There's Yes, go ahead. There's one over there.

one over there. one over there. Excellent question. I Excellent question. I Excellent question. I will answer this one. will answer this one. will answer this one. Yes, you should Yes, you should Yes, you should answer. Yes. answer. Yes. answer. Yes. So we have this So we have this So we have this idea that I idea that I idea that I mentioned earlier, that " mentioned earlier, that " what" and "how" are two what" and "how" are two what" and "how" are two different ideas. different ideas. We announce the result, We announce the result, but “how” is a bit of an but “how” is a bit of an but “how” is a bit of an open question; open question; open question; let the agent let the agent let the agent figure it out for himself. On this figure it out for himself. On this figure it out for himself. On this spectrum, as you spectrum, as you spectrum, as you rightly point out, in rightly point out, in rightly point out, in certain places certain places certain places you actually you actually you actually need to be told how to need to be told how to need to be told how to do it. And we do it. And we do it. And we have a function—a have a function—a syntax that syntax that syntax that lets us lets us lets us define exactly how to define exactly how to define exactly how to do this. This do this. This do this. This way you can way you can way you can set a set a set a very flexible approach from the start. very flexible approach from the start. very flexible approach from the start. You can let a You can let a You can let a very very very intelligent intelligent intelligent agent agent figure out on its own how figure out on its own how to get to what to get to what to get to what you've declared. And you've declared. And you've declared. And then once it then once it then once it does that, you does that, you does that, you can say: here's the can say: here's the can say: here's the agent trace. agent trace. agent trace. Let's take this Let's take this Let's take this trace and kind of trace and kind of trace and kind of encode it or, I encode it or, I encode it or, I don't know, wrap it in a don't know, wrap it in a don't know, wrap it in a description of how to description of how to description of how to do it. And this do it. And this do it. And this allows us to allows us to allows us to reduce the cost of reduce the cost of reduce the cost of executing the work executing the work executing the work process. You process. You process. You start with a very start with a very start with a very expensive model, expensive model, expensive model, you define everything, and you define everything, and you define everything, and then, as you then, as you then, as you say, you make it say, you make it say, you make it more specific, more specific, more specific, more more more deterministic.

deterministic. So you kind of So you kind of shift the spectrum from shift the spectrum from shift the spectrum from just "what" to "how to just "what" to "how to just "what" to "how to do it." And do it." And do it." And the cost goes down the cost goes down the cost goes down as it goes. So as it goes. So as it goes. So when we say when we say when we say everything is written in everything is written in everything is written in Markdown, that's where you Markdown, that's where you Markdown, that's where you can start can start can start , but then you , but then you , but then you can move on to can move on to can move on to more specific more specific more specific instructions in a instructions in a instructions in a procedural procedural procedural imperative style. imperative style. Yeah, I think a couple of Yeah, I think a couple of comments on comments on comments on that. that. First, we have a First, we have a First, we have a system where you system where you system where you can take, can take, can take, for example, a “golden for example, a “golden for example, a “golden session” and turn session” and turn session” and turn it into a reusable it into a reusable it into a reusable workflow. This workflow. This workflow. This was the best was the best was the best cloud code session I've cloud code session I've cloud code session I've ever ever had. I wanted to had. I wanted to had. I wanted to do this because do this because do this because we we we encounter encounter encounter situations like this all the time situations like this all the time , right? And I think there is a , right? And I think there is a , right? And I think there is a part where these part where these part where these things are not things are not things are not deterministic. I deterministic. I deterministic. I think a lot of the think a lot of the think a lot of the so-called " so-called " fixed" and fixed" and fixed" and stable endpoints stable endpoints stable endpoints that that that people have been showing are people have been showing are people have been showing are actually changing actually changing actually changing because all these because all these because all these companies have limited companies have limited companies have limited computing computing computing resources, but I also resources, but I also resources, but I also think there's a think there's a think there's a shift in expectations, shift in expectations, shift in expectations, right? Like right? Like right? Like when you first when you first when you first tried something tried something tried something like Fable or something like Fable or something like Fable or something like that, it like that, it like that, it blew you away and you were like, blew you away and you were like, blew you away and you were like, 'I want that.

'I want that. 'I want that. This workflow is This workflow is This workflow is simply amazing. I simply amazing. I simply amazing. I want it every time." want it every time." want it every time." And then you're like And then you're like And then you're like boiling a frog over a boiling a frog over a boiling a frog over a fire, you have different fire, you have different fire, you have different expectations, and suddenly it expectations, and suddenly it expectations, and suddenly it doesn't doesn't doesn't happen again, and happen again, and happen again, and part of that has to do with part of that has to do with part of that has to do with reliability. So reliability. So reliability. So what you can do is what you can do is what you can do is take a really take a really take a really good session and good session and good session and codify it. But codify it. But codify it. But there's another point there's another point there's another point to give to give to give Dan credit for creating this Dan credit for creating this Dan credit for creating this structure: it's not structure: it's not structure: it's not just the markings just the markings just the markings you see— you see— that's just the tip of the that's just the tip of the that's just the tip of the iceberg of what iceberg of what iceberg of what she sees. For example, the she sees. For example, the she sees. For example, the last time last time last time I checked, a full I checked, a full I checked, a full system request, system request, system request, skill or whatever it's skill or whatever it's skill or whatever it's called (it's a skill called (it's a skill , it's a system , it's a system , it's a system request), contained request), contained request), contained about 70,000, 80,000 about 70,000, 80,000 about 70,000, 80,000 tokens or tokens or tokens or more. We are trying more. We are trying more. We are trying to organize it. We're to organize it. We're to organize it. We're trying... trying... trying... Well, now that Well, now that Well, now that the agents are smarter, the agents are smarter, the agents are smarter, we can make it we can make it we can make it a little shorter. But a little shorter. But a little shorter. But we essentially have a we essentially have a we essentially have a full-fledged full-fledged full-fledged type system that, as it were type system that, as it were type system that, as it were , , , redefines Markdown with a redefines Markdown with a redefines Markdown with a very, very clear very, very clear very, very clear type system and a type system and a type system and a very specific very specific very specific specification of how the specification of how the specification of how the agent is going to agent is going to agent is going to provide this provide this provide this incredibly complex incredibly complex incredibly complex multi-agent multi-agent multi-agent product. But perhaps product. But perhaps product. But perhaps to clarify where to clarify where to clarify where we don't interfere, we don't interfere, we don't interfere, for example, the Claude code for example, the Claude code , when it , when it , when it runs, has runs, has runs, has its own its own its own system for system for system for subagents. We don't subagents. We don't subagents. We don't want want want to interfere with this. We don't to interfere with this. We don't to interfere with this. We don't want our want our want our directive to directive to directive to be be be specific to the specific to the specific to the Claude code in any way, because the Codex has a Claude code in any way, because the Codex has a Claude code in any way, because the Codex has a completely different completely different completely different set of subagents.

set of subagents. set of subagents. So we have, I So we have, I So we have, I don't know, a meta- don't know, a meta- tool, something tool, something tool, something like that. We are one like that. We are one like that. We are one layer, we want layer, we want layer, we want you to use your you to use your you to use your own own own tool. We don't tool. We don't tool. We don't want to be a want to be a want to be a tool. We tool. We tool. We want to be something that is want to be something that is want to be something that is compiled by a compiled by a compiled by a tool. So tool. So , I think it's a couple of , I think it's a couple of , I think it's a couple of parts. Does parts. Does parts. Does this answer your this answer your this answer your question? Fine. question? Fine. question? Fine. So, if you want to So, if you want to So, if you want to give it a try, here is the QR give it a try, here is the QR code: run.prose.md. The QR code code: run.prose.md. The QR code code: run.prose.md. The QR code leads there. If you leads there. If you leads there. If you want to start experimenting with it want to start experimenting with it want to start experimenting with it . . In this video I want to show a few show a few things. First of all, this is things. First of all, this is Prose in edit mode, edit mode, right? Technically this is right? Technically this is right? Technically this is Markdown, and Dan Markdown, and Dan Markdown, and Dan has overridden the has overridden the has overridden the H3 headings with all H3 headings with all H3 headings with all types of this code. But types of this code. But types of this code. But forcing someone forcing someone forcing someone to learn this to learn this to learn this syntax and write syntax and write syntax and write it from scratch is not it from scratch is not it from scratch is not quite what is needed.

quite what is needed. So what do we want to So what do we want to do? Although everything that do? Although everything that do? Although everything that runs this runs this runs this program is program is program is here, and in here, and in here, and in fact, for the real fact, for the real fact, for the real geeks in the room, you geeks in the room, you geeks in the room, you can see that I can see that I can see that I took the metadata from the took the metadata from the took the metadata from the Markdown file, and that's Markdown file, and that's Markdown file, and that's actually all that's actually all that's actually all that's happening in the cloud. happening in the cloud. So, I could take So, I could take this program, this program, this program, export it export it export it as a pure Markdown file, as a pure Markdown file, as a pure Markdown file, and send it to a friend, and and send it to a friend, and and send it to a friend, and they'll upload it to their they'll upload it to their they'll upload it to their cloud in exactly the same way, cloud in exactly the same way, cloud in exactly the same way, so that it's so that it's composable and stays composable and stays true to the true to the true to the pure Markdown idea. So the pure Markdown idea. So the pure Markdown idea. So the whole idea is that whole idea is that whole idea is that while this is the point while this is the point while this is the point and we want you to and we want you to and we want you to read this, it's not the read this, it's not the read this, it's not the entry point. So entry point. So entry point. So I created an agent I created an agent I created an agent that I hope that I hope that I hope will communicate with me will communicate with me will communicate with me . . Hi, this is Prose. What do you Hi, this is Prose. What do you want to do want to do want to do today? today?

Okay, every time Okay, every time I I I have a new have a new have a new pull request, I pull request, I pull request, I want you to want you to want you to write me a write me a write me a Prose program in my Prose program in my Prose program in my run-prose repository run-prose repository run-prose repository that will that will that will check it, as if check it, as if check it, as if you were a grumpy you were a grumpy you were a grumpy senior senior senior developer developer developer who doesn't like the who doesn't like the AI-generated code. Of course I can Of course I can do it. Wait. do it. Wait. Fine. I'll Fine. I'll turn off the sound now. So, turn off the sound now. So, turn off the sound now. So, he says. The program he says. The program he says. The program will try to will try to will try to find out. The whole idea is find out. The whole idea is that whatever you that whatever you do using the do using the do using the draft card in the draft card in the draft card in the app, I'll app, I'll app, I'll let you know when it's let you know when it's let you know when it's ready. ready. ready. Therefore, anything that Therefore, anything that Therefore, anything that may require may require may require payment or payment or payment or expenditure of funds expenditure of funds must be confirmed. I must be confirmed. I must be confirmed. I think this is best think this is best think this is best practice. practice. practice. But constructive, But constructive, evidence-based. evidence-based. Oh, that's it. Oh, that's it. Oh, that's it. Currently in draft. I'll Currently in draft. I'll Currently in draft. I'll let you know when it's let you know when it's let you know when it's ready. ready. ready. Great. I had to Great. I had to Great. I had to confirm because confirm because confirm because this action this action this action costs money. So these are costs money. So these are costs money. So these are the benefits of having the the benefits of having the the benefits of having the right function. right function. right function. Basically, it takes my Basically, it takes my Basically, it takes my request and will request and will request and will generate a new generate a new generate a new program, including program, including program, including all the wiring. This all the wiring. This all the wiring. This step will take a little step will take a little step will take a little longer.

longer. longer. So while this So while this So while this works, I'll show you the works, I'll show you the works, I'll show you the manual way to manual way to manual way to do it. It is located do it. It is located do it. It is located on the Tasks tab. on the Tasks tab. on the Tasks tab. I don't know why it keeps I don't know why it keeps I don't know why it keeps interrupting. In the interrupting. In the interrupting. In the Tasks tab, we Tasks tab, we Tasks tab, we can go here and can go here and can go here and create a new create a new create a new task. A task. A task. A scheduled job is scheduled job is scheduled job is simply a cron job. simply a cron job. simply a cron job. We have an We have an We have an open-ended webhook. open-ended webhook. open-ended webhook. So you or your agent So you or your agent So you or your agent can either can either can either use our use our use our dedicated webhook dedicated webhook dedicated webhook or configure our or configure our or configure our device to device to device to listen for your listen for your webhook. But for Git and GitHub I webhook. But for Git and GitHub I webhook. But for Git and GitHub I will do the same. will do the same. will do the same. For For For GitHub pull requests, we've GitHub pull requests, we've GitHub pull requests, we've simplified this simplified this simplified this process. We will do process. We will do process. We will do this in the run.pros file. I have this in the run.pros file. I have this in the run.pros file. I have pre- pre- pre- configured this configured this configured this contract. It goes like this contract. It goes like this contract. It goes like this : "Okay, you : "Okay, you : "Okay, you will check my will check my will check my program." Then I program." Then I program." Then I can choose which can choose which can choose which model they model they model they use. We use. We use. We will add more later. will add more later. will add more later. We currently have three We currently have three We currently have three top models from OpenAI. top models from OpenAI. Then I'll just create Then I'll just create and deploy it. And one more and deploy it. And one more and deploy it. And one more thing thing thing I want to show you while I want to show you while I want to show you while this is working is this is working is this is working is how I use this as a how I use this as a back pressure mechanism.

back pressure mechanism. So, the main thing is not So, the main thing is not So, the main thing is not to get stuck on to get stuck on the content that is the content that is presented here. I presented here. I presented here. I want to show you want to show you want to show you the scope of work. This all the scope of work. This all the scope of work. This all started three days started three days started three days ago, but if I ago, but if I ago, but if I go here, I go here, I go here, I can see that all these can see that all these can see that all these requests are still requests are still requests are still dated three or dated three or dated three or four days earlier. four days earlier. four days earlier. Over the course of several days, Over the course of several days, numerous numerous pull requests (PRs) were submitted. And pull requests (PRs) were submitted. And pull requests (PRs) were submitted. And every time I every time I every time I pull a new pull a new pull a new feature request to GitHub—the feature request to GitHub—the webhooks that I want to webhooks that I want to webhooks that I want to prepare for that prepare for that prepare for that event—each one event—each one event—each one runs a whole runs a whole runs a whole series of checks, and series of checks, and series of checks, and some of them are some of them are some of them are simple, like, " simple, like, " Okay, this request Okay, this request Okay, this request clearly fired right away, clearly fired right away, clearly fired right away, so this is just so this is just so this is just one round." And one round." And one round." And some went through, some went through, some went through, you know, 14 rounds of you know, 14 rounds of you know, 14 rounds of testing before it testing before it testing before it seemed like the tests seemed like the tests seemed like the tests were successful, and that were successful, and that were successful, and that allowed the project to allowed the project to allowed the project to go through. And then today go through. And then today go through. And then today I noticed something that I noticed something that I noticed something that I didn't like about I didn't like about I didn't like about how it was working, how it was working, how it was working, and I was able to basically and I was able to basically and I was able to basically say, "Okay, on say, "Okay, on say, "Okay, on our test our test our test server, don't touch server, don't touch server, don't touch production. I'm production. I'm production. I'm going to give a presentation now.

going to give a presentation now. going to give a presentation now. On the test server I On the test server I On the test server I want you to want you to want you to change all change all change all the parameters to make the parameters to make the parameters to make sure this runs, but sure this runs, but sure this runs, but you have to you have to you have to use the system yourself use the system yourself . So, both our . So, both our development version and our development version and our production version give production version give production version give different reviews. And different reviews. And different reviews. And now the agent that's now the agent that's now the agent that's running here (we running here (we running here (we see all the different see all the different see all the different jobs in the jobs in the Runs tab) basically can't Runs tab) basically can't Runs tab) basically can't consider its consider its consider its task complete task complete task complete until it gets until it gets feedback from feedback from my my developer agents. developer agents. developer agents. And to highlight And to highlight And to highlight what makes what makes what makes this system different from this system different from this system different from any other chat any other chat any other chat or agent or agent or agent process you've process you've process you've seen, all of this seen, all of this seen, all of this data is compared data is compared data is compared to the contract. So, to the contract. So, to the contract. So, you could say that you could say that you could say that every time a new session is created and every time a new session is created and compared with the compared with the contract contract contract you wrote you wrote you wrote declaratively. As declaratively. As declaratively. As Raymond mentioned, this Raymond mentioned, this Raymond mentioned, this takes models out of the takes models out of the overconfidence loop overconfidence loop where you don't want the where you don't want the where you don't want the same same same model to check an model to check an model to check an action that has already been performed action that has already been performed . So you . So you . So you create new create new create new sessions and sessions and sessions and map them to the map them to the map them to the contract, which I think contract, which I think contract, which I think is the is the is the best approach to best approach to best approach to implementation. Yes, implementation. Yes, implementation. Yes, go ahead.

go ahead. go ahead. But isn't this a But isn't this a But isn't this a great opportunity great opportunity great opportunity for one-time for one-time for one-time classifiers like classifiers like classifiers like Jeff? Yes. Jeff? Yes. We have a lot of good We have a lot of good developments in developments in developments in progress. So progress. So progress. So the idea of the idea of contract compliance and contract compliance and confirmation... confirmation... Sorry, Raymond. Yes. Sorry, Raymond. Yes. This is very interesting This is very interesting for these for these for these decision models, where decision models, where decision models, where you have a contract that you have a contract that you have a contract that represents the represents the represents the ideal state, ideal state, ideal state, and your not-quite- and your not-quite- and your not-quite- correct state correct state correct state in the real world, and you in the real world, and you in the real world, and you can apply a can apply a can apply a general general general decision model to decision model to decision model to classes, like classes, like classes, like Jeff's, to Jeff's, to Jeff's, to say: does say: does say: does this test pass or fail this test pass or fail ? ? ? I think I think I think it will appear soon. it will appear soon. it will appear soon. Yeah, and I think without, Yeah, and I think without, Yeah, and I think without, like, Dan, like, Dan, like, Dan, we're we're we're going way off going way off going way off script, but it's a script, but it's a script, but it's a great question, and, uh great question, and, uh , the great Jeffification is upon us , the great Jeffification is upon us . So we . So we need to talk about this need to talk about this need to talk about this . Um, I . Um, I . Um, I think Dan and I think Dan and I think Dan and I thought about this a lot thought about this a lot thought about this a lot and realized that OpenPros itself and realized that OpenPros itself , the language that , the language that Markdown is, I think Markdown is, I think has no place in has no place in has no place in Jeff's world because it's Jeff's world because it's Jeff's world because it's not generative. That not generative. That not generative. That is, the whole idea of ​​language— is, the whole idea of ​​language— generating language and generating language and generating language and understanding language—is not understanding language—is not understanding language—is not true. But I think there true. But I think there true. But I think there are two places where it are two places where it would fit really well, and in would fit really well, and in particular, with particular, with particular, with run.pros hosted, it's the run.pros hosted, it's the run.pros hosted, it's the backend. So backend. So backend. So the flaw or the flaw or the flaw or weakness of OpenPros is that weakness of OpenPros is that you allow you allow the agent to say, "I've the agent to say, "I've the agent to say, "I've fulfilled the contract.

fulfilled the contract. fulfilled the contract. Here it is. Right". And Here it is. Right". And Here it is. Right". And so I often so I often so I often tell people that tell people that tell people that the goal of OpenPros is similar to the goal of OpenPros is similar to the goal of OpenPros is similar to something like the DS Pi, where something like the DS Pi, where something like the DS Pi, where you want to have you want to have you want to have this abstract this abstract this abstract layer on top of which layer on top of which layer on top of which you can express very you can express very you can express very complex complex complex workflows. The implementation is workflows. The implementation is workflows. The implementation is exactly the opposite of the exactly the opposite of the exactly the opposite of the DS Pi. In DS Pi, every DS Pi. In DS Pi, every DS Pi. In DS Pi, every LLM call is a black box, and LLM call is a black box, and LLM call is a black box, and you isolate it to the you isolate it to the you isolate it to the extreme. This is all extreme. This is all extreme. This is all deterministic deterministic deterministic code. These are all code. These are all code. These are all typed typed typed interfaces that interfaces that interfaces that allow the black allow the black allow the black box to do box to do box to do just that. just that. just that. Dan's approach with Open Prose Dan's approach with Open Prose is the complete is the complete is the complete opposite: opposite: opposite: AI becomes AI becomes AI becomes smart enough smart enough smart enough to be a to be a to be a compiler. Let compiler. Let compiler. Let everything be in the fuzzy everything be in the fuzzy everything be in the fuzzy area. However, its area. However, its area. However, its weakness is weakness is weakness is that if you that if you that if you run it on Haku run it on Haku run it on Haku or on Fable, you will or on Fable, you will or on Fable, you will get a completely get a completely get a completely different result. And different result. And different result. And you also allow you also allow you also allow him to say, "I'm him to say, "I'm him to say, "I'm done." So I done." So I done." So I think a very think a very think a very natural place natural place natural place for Jev or these new for Jev or these new for Jev or these new decision models decision models decision models , or as we , or as we , or as we call them, System call them, System call them, System One models, is, as you One models, is, as you One models, is, as you said, as a said, as a said, as a verifier at the end of the verifier at the end of the verifier at the end of the work. Very fast work. Very fast work. Very fast and cheap. And one more and cheap. And one more and cheap. And one more thing I want to thing I want to thing I want to build that's build that's build that's related to related to related to this is...

this is... A router. So A router. So A router. So why should I why should I why should I click a button click a button click a button that supposedly that supposedly that supposedly sends tasks to the sends tasks to the task processing task and task processing task and merge requests to the merge merge requests to the merge merge requests to the merge request review task request review task request review task ? Why do ? Why do ? Why do n't I just say, n't I just say, "Okay, all the "Okay, all the "Okay, all the GitHub events that GitHub events that GitHub events that come in are solving come in are solving come in are solving issues, reviewing issues, reviewing issues, reviewing pull requests, pull requests, pull requests, right?" One more right?" One more right?" One more question. question. question. Don't give them any more Don't give them any more Don't give them any more code ideas. Yes, code ideas. Yes, code ideas. Yes, I was going to say, I was going to say, I was going to say, "Thank you." Because the "Thank you." Because the hot topic on Twitter right now is: hot topic on Twitter right now is: why not why not why not use RL (Registered use RL (Registered use RL (Registered Learning)? If I Learning)? If I Learning)? If I have all these have all these have all these great great contract performance results , why don't I just , why don't I just , why don't I just use RL? use RL? use RL? So one of the ideas, So one of the ideas, So one of the ideas, going back to the going back to the going back to the question of language, is the question of language, is the question of language, is the question: why question: why question: why do we do this with do we do this with do we do this with language at all? From a language at all? From a language at all? From a human human human perspective, you can perspective, you can perspective, you can commit it to Git, commit it to Git, commit it to Git, use a use a use a version control system, Notion, version control system, Notion, version control system, Notion, or Jira, and anyone on or Jira, and anyone on or Jira, and anyone on your team can your team can your team can use it. But use it. But use it. But from an engineering from an engineering from an engineering perspective, the fact that perspective, the fact that perspective, the fact that it's all text is it's all text is it's all text is perfect and wonderful perfect and wonderful perfect and wonderful for RL. We're for RL. We're for RL. We're not doing that right now, not doing that right now, not doing that right now, but it's something we but it's something we but it's something we definitely talk about definitely talk about definitely talk about with various with various with various potential potential potential clients: when clients: when clients: when you have everything written out in you have everything written out in you have everything written out in text, it's text, it's text, it's much better for RL.

much better for RL. Yeah, we haven't reached Yeah, we haven't reached the volumes where the volumes where the volumes where it matters yet, but if you're it matters yet, but if you're it matters yet, but if you're hosting a model and hosting a model and hosting a model and you have a you have a you have a token prefix of around 100,000, then token prefix of around 100,000, then token prefix of around 100,000, then you can just you can just you can just use that or, you use that or, you use that or, you know, create know, create know, create more token-efficient more token-efficient representations. representations. So, yes, if anyone So, yes, if anyone So, yes, if anyone wants to use a wants to use a wants to use a trillion tokens in Open trillion tokens in Open trillion tokens in Open Pros, we'll be Pros, we'll be Pros, we'll be happy to happy to happy to do that for you. do that for you. do that for you. So anyway, to So anyway, to So anyway, to get back to the get back to the get back to the beginning, thank you. These are beginning, thank you. These are beginning, thank you. These are really really really good suggestions. good suggestions. good suggestions. Now we have our Now we have our Now we have our grumpy senior grumpy senior grumpy senior review of the review of the review of the merge request. And again, the merge request. And again, the merge request. And again, the key idea is that the key idea is that the key idea is that the agent did all this agent did all this agent did all this for us. The key for us. The key for us. The key idea here is that once we idea here is that once we idea here is that once we deploy this task, deploy this task, deploy this task, it will start spending it will start spending it will start spending money whenever money whenever money whenever we receive a we receive a we receive a new new new pull request. To click pull request. To click pull request. To click on this. Then I on this. Then I on this. Then I will confirm it. And will confirm it. And will confirm it. And once this is once this is once this is deployed, when I deployed, when I deployed, when I go back to my go back to my go back to my assignments, it will assignments, it will assignments, it will look exactly like look exactly like look exactly like ...let's ...let's ...let's see. So if see. So if see. So if we just...Oh, I we just...Oh, I we just...Oh, I switched that.

switched that. switched that. Once this Once this Once this loads we should loads we should loads we should see grumpy ones at the see grumpy ones at the see grumpy ones at the top or bottom. Yes, top or bottom. Yes, top or bottom. Yes, that's it. A PR review by a that's it. A PR review by a that's it. A PR review by a grumpy grumpy grumpy AI skeptic. So, as you can see, I AI skeptic. So, as you can see, I AI skeptic. So, as you can see, I have trained before. have trained before. Fine. So let's Fine. So let's see what it is see what it is see what it is . I think these are the . I think these are the . I think these are the main points. Yes, main points. Yes, main points. Yes, I am in charge of run.pros.md, I am in charge of run.pros.md, I am in charge of run.pros.md, so if you so if you so if you have any have any have any problems, don't tell problems, don't tell problems, don't tell my boss, my boss, my boss, just send it just send it just send it directly to me and my directly to me and my directly to me and my agents will take care agents will take care agents will take care of it for you. But yeah, of it for you. But yeah, of it for you. But yeah, I'm really I'm really I'm really excited about it excited about it excited about it because it's a because it's a because it's a simple and easy simple and easy simple and easy way to build way to build way to build these these event-driven systems. If you event-driven systems. If you want to see want to see want to see other triggers...Or other triggers...Or other triggers...Or other features, that's other features, that's other features, that's really really really cool for me. But cool for me. But cool for me. But again, you again, you again, you don't have to don't have to don't have to use the use the use the hosted version at all, and hosted version at all, and you can you can use the NPX skill use the NPX skill like OpenProse. But like OpenProse. But like OpenProse. But now Dan's now Dan's now Dan's going to going to go through a go through a specific example that's specific example that's specific example that's aimed at aimed at aimed at our friends in the our friends in the our friends in the financial sector, financial sector, financial sector, and we'll show it in the Codex, and we'll show it in the Codex, and we'll show it in the Codex, and then he'll and then he'll and then he'll use that use that use that as an opportunity to as an opportunity to go into a little more detail go into a little more detail about how it all about how it all about how it all works under the works under the works under the hood. Yes.

hood. Yes. Thank you. Thank you. Certainly. So Certainly. So Certainly. So , making this , making this , making this work in the cloud work in the cloud is interesting. One is interesting. One is interesting. One of the more of the more of the more interesting use cases interesting use cases we've talked about with we've talked about with various clients various clients various clients and and and prospects is prospects is prospects is when you can when you can when you can use a use a use a framework, language, or framework, language, or framework, language, or template that you template that you template that you can build can build can build in-house and in-house and in-house and run locally. The run locally. The run locally. The whole idea here is that whole idea here is that whole idea here is that you can you can you can run it run it run it locally. You don't locally. You don't locally. You don't need to use need to use need to use our cloud version. our cloud version. our cloud version. We'd love for you to We'd love for you to We'd love for you to try it out in try it out in try it out in the cloud and the cloud and the cloud and experiment experiment experiment with it. Feel with it. Feel with it. Feel it. But if you want to it. But if you want to it. But if you want to implement this in your implement this in your implement this in your organization, organization, organization, the most interesting the most interesting use case use case we see we see is using is using is using open models open models open models on top of your on top of your on top of your own own own development environment and then development environment and then development environment and then using that using that using that as an as an as an application layer. application layer. application layer. This allows you This allows you This allows you to build the entire stack to build the entire stack to build the entire stack within the company. You do within the company. You do within the company. You do not need to send not need to send not need to send your data to your data to your data to laboratories. More laboratories. More laboratories. More importantly than your importantly than your importantly than your data, you don't need to data, you don't need to data, you don't need to send your send your send your institutional institutional institutional knowledge. We knowledge. We knowledge. We prefer prefer prefer to think of to think of to think of open source open source open source as as as institutional institutional institutional operational knowledge, operational knowledge, operational knowledge, as documentation, as documentation, as documentation, right? Think of right? Think of right? Think of it as it as it as standard standard standard operating operating operating procedures. So, procedures. So, procedures. So, even more important than your even more important than your even more important than your data, you don't data, you don't data, you don't want to send want to send want to send your standard your standard your standard operating operating operating procedures to big procedures to big procedures to big labs. And labs. And labs. And so it so it so it fits very well into fits very well into fits very well into this stack of open source, this stack of open source, open source applications open source applications . We are publishing . We are publishing . We are publishing this open core.

this open core. this open core. He has a permit He has a permit He has a permit license. You can license. You can license. You can use it use it use it within the company. within the company. within the company. We have a number of large We have a number of large We have a number of large companies that are already companies that are already companies that are already doing this. And you doing this. And you doing this. And you can bring your can bring your can bring your own system. own system. own system. You can You can You can use it with use it with use it with approved approved approved models that models that models that your company already has your company already has . This way, you . This way, you . This way, you won't have to won't have to won't have to feel like you're feel like you're feel like you're throwing throwing your organization's data or knowledge your organization's data or knowledge over the wall. It's over the wall. It's over the wall. It's just...well, we can just...well, we can just...well, we can go into this a little bit more go into this a little bit more go into this a little bit more , and I want to answer some , and I want to answer some , and I want to answer some additional additional additional questions. This is a questions. This is a questions. This is a workflow we are workflow we are workflow we are developing for a developing for a developing for a client client client interested in interested in interested in using Open Prose using Open Prose using Open Prose to document to document to document their quantitative their quantitative their quantitative models. So, for those models. So, for those models. So, for those of you who are familiar with of you who are familiar with of you who are familiar with quantitative quantitative quantitative models, there are models, there are models, there are various various various regulatory regulatory regulatory obligations that require you obligations that require you to update to update the documentation for these the documentation for these the documentation for these quantitative quantitative quantitative models every 3 years, every 1 year, or whenever there are certain changes. Typically, models every 3 years, every 1 year, or whenever there are certain changes. Typically, models every 3 years, every 1 year, or whenever there are certain changes. Typically, this is a process that, this is a process that, this is a process that, in our in our in our experience, requires a experience, requires a large number of large number of highly qualified highly qualified highly qualified specialists, right? specialists, right? specialists, right? These people, maybe These people, maybe These people, maybe some of you, some of you, some of you, probably spend probably spend probably spend part of your year part of your year part of your year writing writing writing documentation documentation documentation that that that documents documents documents what what what your model is used for, what your model is used for, what your model is used for, what conditions it conditions it conditions it was tested under, was tested under, was tested under, what conditions... Um, who what conditions... Um, who what conditions... Um, who , who, what other , who, what other , who, what other models reference models reference models reference it upstream, and a it upstream, and a it upstream, and a whole host of other whole host of other whole host of other necessary necessary necessary requirements. Obviously, these are requirements. Obviously, these are requirements. Obviously, these are requirements of requirements of requirements of regulators regulators , your internal , your internal , your internal auditors and your auditors and your auditors and your internal internal internal compliance system. So this is the compliance system. So this is the compliance system. So this is the flow we have flow we have flow we have used successfully used successfully . It's perfect

. It's perfect . It's perfect because in some because in some because in some ways the ways the ways the regulations are like the regulations are like the regulations are like the Open Prose program, Open Prose program, Open Prose program, right? And these are right? And these are right? And these are demands demands demands written in written in written in prose form. So, I'm going to prose form. So, I'm going to prose form. So, I'm going to begin. You begin. You begin. You can see here can see here can see here that we are in Codex. You can that we are in Codex. You can that we are in Codex. You can run this on Cloud Code run this on Cloud Code . You can run . You can run . You can run this on your this on your this on your own test own test own test environment. The whole point is that environment. The whole point is that environment. The whole point is that it is it is it is independent of the test independent of the test independent of the test environment. So we just environment. So we just environment. So we just load the kernel. Then load the kernel. Then load the kernel. Then we launch the program, we launch the program, we launch the program, and then a specific and then a specific and then a specific program. So I'm program. So I'm program. So I'm going to run going to run going to run this, and while it's this, and while it's this, and while it's running, I can running, I can running, I can show you what show you what show you what the kernel looks like, and the kernel looks like, and the kernel looks like, and then what the then what the then what the program itself looks like. program itself looks like. program itself looks like. Just go back to Just go back to Just go back to Chrome. And here she is. Chrome. And here she is. Chrome. And here she is. This is the core. This is Markdown. It is This is the core. This is Markdown. It is This is the core. This is Markdown. It is not intended to be not intended to be not intended to be read. It was done read. It was done read. It was done by you, that's why it's by you, that's why it's by you, that's why it's designed this way. designed this way. designed this way. Initially we had a Initially we had a Initially we had a beautiful format, but beautiful format, but beautiful format, but people complained. They people complained. They people complained. They said, "What is said, "What is said, "What is this?" "It's this?" "It's this?" "It's meant for meant for meant for agents to read. It's agents to read. It's agents to read. It's just Markdown. This is an just Markdown. This is an operating manual that operating manual that describes what to describes what to describes what to expect. You can expect. You can expect. You can think of it as a think of it as a think of it as a loader or a loader or a loader or a resolution path, an resolution path, an resolution path, an index system, index system, index system, a way to find other a way to find other a way to find other documents that documents that documents that the agent needs. And the agent needs. And the agent needs. And the program itself looks, and the program itself looks, and the program itself looks, and again, this is what again, this is what again, this is what I referred to in the Codex, I referred to in the Codex, I referred to in the Codex, very similar. So it's very similar. So it's very similar. So it's Markdown again. It Markdown again. It Markdown again. It contains a number of contains a number of contains a number of requirements requirements requirements imposed by imposed by imposed by regulators, regulators, regulators, compliance agencies

compliance agencies compliance agencies and auditors. And that and auditors. And that and auditors. And that determines, you know determines, you know , the ideal state of the , the ideal state of the , the ideal state of the end result end result end result will be will be will be in accordance with in accordance with in accordance with that contract. And those are the that contract. And those are the that contract. And those are the only two only two only two things you things you things you convey. You simply convey. You simply convey. You simply transfer the core, transfer the core, transfer the core, which is published which is published which is published publicly. You publicly. You publicly. You can all download it can all download it can all download it right now. And you right now. And you right now. And you can try it out can try it out can try it out , and then write , and then write , and then write any program, any program, any program, any program you any program you any program you want, any want, any want, any contract that contract that contract that requires a requires a requires a complex set of complex set of complex set of conditions to be met, and then the conditions to be met, and then the conditions to be met, and then the agents will just agents will just agents will just work. And, you know work. And, you know , the better the , the better the , the better the development environment, the better the development environment, the better the development environment, the better the model, the faster and model, the faster and model, the faster and cheaper they will cheaper they will cheaper they will produce produce produce results. But the idea is results. But the idea is results. But the idea is that you that you that you are declaring the result are declaring the result , not how to , not how to , not how to do it. Let's do it. Let's do it. Let's see, we can see, we can see, we can go back and go back and go back and see. We see. We see. We launched this with a launched this with a launched this with a relatively relatively relatively inexpensive model so that inexpensive model so that inexpensive model so that it would run it would run it would run faster. But it still faster. But it still faster. But it still works. Oh, works. Oh, works. Oh, looks like it's looks like it's looks like it's ending. So, ending. So, ending. So, this is, I mean, this is the this is, I mean, this is the this is, I mean, this is the document that she document that she document that she created. In this created. In this created. In this case, we are creating case, we are creating case, we are creating documentation for the documentation for the documentation for the distribution curve distribution curve . This is a publicly available . This is a publicly available . This is a publicly available model. Therefore, we do not model. Therefore, we do not model. Therefore, we do not show show show any internal any internal any internal banking data here.

banking data here. banking data here. I don't know that either. I don't know that either. I don't know that either. Raise your hand. Is there Raise your hand. Is there Raise your hand. Is there any any any quantitative quantitative quantitative analyst here to analyst here to analyst here to check this? What does it check this? What does it check this? What does it look like? Yes, good. look like? Yes, good. look like? Yes, good. We will come, we We will come, we We will come, we will come, we will find will come, we will find will come, we will find you later. But just so you you later. But just so you you later. But just so you know, this is know, this is know, this is documentation documentation documentation that needs to be that needs to be that needs to be created regularly created regularly created regularly for each model. A for each model. A large bank has large bank has about 4000 models. They about 4000 models. They about 4000 models. They need to be need to be need to be re-documented annually, a re-documented annually, a re-documented annually, a problem problem problem that costs that costs that costs some of the some of the some of the largest banks between largest banks between $10 million and $50 million a year. And we're making $10 million and $50 million a year. And we're making great great great strides in doing it strides in doing it strides in doing it for them at a much for them at a much for them at a much lower cost. Of course, lower cost. Of course, lower cost. Of course, there are some there are some there are some difficulties. Getting difficulties. Getting difficulties. Getting approval from approval from approval from regulators regulators , auditors and , auditors and , auditors and regulators is regulators is regulators is another part of the struggle, another part of the struggle, another part of the struggle, as I'm sure you all as I'm sure you all as I'm sure you all know, but we can know, but we can know, but we can compare that to compare that to compare that to historical historical historical results. We results. We results. We can compare this to can compare this to can compare this to how long it how long it how long it takes teams takes teams takes teams to do this. And we to do this. And we to do this. And we have quantitative have quantitative have quantitative analysts who analysts who analysts who work work work directly in the directly in the directly in the office and can evaluate office and can evaluate office and can evaluate these models and give us an these models and give us an these models and give us an idea of idea of idea of how well they're doing how well they're doing . And we . And we find that it's find that it's , I mean, , I mean, , I mean, pretty shockingly pretty shockingly pretty shockingly good. Much good. Much good. Much cheaper than cheaper than cheaper than traditional manual traditional manual traditional manual work. Any work. Any work. Any questions? Do you have any questions? Do you have any questions? Do you have any other other other quantitative quantitative quantitative analysts? Do what analysts? Do what quantitative quantitative analysts are good at.

analysts are good at. analysts are good at. Create models. Create models. Create models. Avoid dealing with Avoid dealing with Avoid dealing with regulatory regulatory regulatory documentation, documentation, documentation, which is probably not which is probably not which is probably not very pleasant. I'm not a very pleasant. I'm not a very pleasant. I'm not a quantitative quantitative quantitative analyst, but have analyst, but have analyst, but have questions? Excellent questions? Excellent questions? Excellent question. Yes. And so, question. Yes. And so, question. Yes. And so, actually, Raymond actually, Raymond demonstrates a little bit of the chat agent as part of the core agent as part of the core , hidden in the , hidden in the , hidden in the Markdown markup along the way, we Markdown markup along the way, we Markdown markup along the way, we have best practices. have best practices. These are the things These are the things you should do you should do you should do when you announce when you announce when you announce results. These are results. These are results. These are anti-patterns, things you should anti-patterns, things you should anti-patterns, things you should n't do n't do n't do . And yes, there is a . And yes, there is a world where you don't world where you don't print out the entire print out the entire print out the entire contract, but rather contract, but rather create it iteratively, and that's exactly what create it iteratively, and that's exactly what Raymond was Raymond was Raymond was demonstrating with another demonstrating with another demonstrating with another agent following agent following agent following our our contract creation best practices guide. But yeah, I mean, But yeah, I mean, getting the getting the getting the intentions out of intentions out of intentions out of people's heads is the hard people's heads is the hard people's heads is the hard part, and I think that's part, and I think that's part, and I think that's part of it. This is, part of it. This is, part of it. This is, of course, not the whole of course, not the whole of course, not the whole process. Yes. I have process. Yes. I have process. Yes. I have I don't I don't I don't want to drag this out, want to drag this out, want to drag this out, we only have 5 minutes. But we only have 5 minutes. But we only have 5 minutes. But here's another point here's another point here's another point I'd like to I'd like to I'd like to discuss: I think discuss: I think discuss: I think this goes a long way toward this goes a long way toward this goes a long way toward explaining why chat has explaining why chat has explaining why chat has become so popular so quickly become so popular so quickly become so popular so quickly . To get . To get . To get started with ChatGPT, started with ChatGPT, started with ChatGPT, you don't need to know you don't need to know you don't need to know what exactly you're looking for, and what exactly you're looking for, and what exactly you're looking for, and the magic of generative the magic of generative the magic of generative AI allows it AI allows it AI allows it to explain everything itself, to explain everything itself, to explain everything itself, right? So I'm not right? So I'm not right? So I'm not sure OpenProse is the sure OpenProse is the best place if you're best place if you're best place if you're at the

at the at the stage where you do stage where you do stage where you do n't even know what you're n't even know what you're n't even know what you're talking about or aren't talking about or aren't talking about or aren't sure what exactly you're sure what exactly you're sure what exactly you're looking for. Here's another looking for. Here's another looking for. Here's another thing: I'm a very thing: I'm a very thing: I'm a very old old old DSPy user, and my joke about DSPy DSPy user, and my joke about DSPy DSPy user, and my joke about DSPy was something like, was something like, was something like, "Oh no, now I "Oh no, now I "Oh no, now I have to know what I have to know what I have to know what I want, right?" Because want, right?" Because want, right?" Because the whole idea of ​​DSPy is that the whole idea of ​​DSPy is that you have to create a you have to create a signature of what you signature of what you signature of what you want to get. And want to get. And want to get. And if you see something if you see something if you see something like that, you think, like that, you think, like that, you think, 'This is so cool!' And 'This is so cool!' And 'This is so cool!' And then you sit back and then you sit back and then you sit back and think, 'Holy [ __ ] think, 'Holy [ __ ] .' I don't really .' I don't really .' I don't really know what I want to do, know what I want to do, know what I want to do, do I? So I do I? So I do I? So I think Open Process is think Open Process is think Open Process is much less strict much less strict much less strict than, say, DSPR, where than, say, DSPR, where than, say, DSPR, where you literally had to you literally had to you literally had to speak speak speak the signature of the the signature of the the signature of the returned returned returned object, otherwise nothing object, otherwise nothing object, otherwise nothing happened. But yes, happened. But yes, happened. But yes, I would say that the I would say that the I would say that the built-in agent will built-in agent will built-in agent will hopefully be able to hopefully be able to hopefully be able to help in some help in some help in some cases. You can cases. You can cases. You can simply turn on simply turn on simply turn on your voice and ask, "What your voice and ask, "What your voice and ask, "What can you do?" And can you do?" And can you do?" And it can do it can do it can do almost everything you almost everything you almost everything you can do in an can do in an can do in an app. app. app. You know, we have these You know, we have these You know, we have these hard- hard- hard- coded coded coded approvals for approvals for approvals for anything that requires anything that requires anything that requires money. Obviously we money. Obviously we money. Obviously we want people to want people to want people to click the button for click the button for click the button for that. But yeah, I think, that. But yeah, I think, that. But yeah, I think, I think that...I don't I think that...I don't I think that...I don't know. I think it's know. I think it's know. I think it's also because also because also because chat isn't the best chat isn't the best chat isn't the best interface, and agents interface, and agents interface, and agents that are good at that are good at communicating convincingly communicating convincingly with people with people with people aren't necessarily the aren't necessarily the aren't necessarily the best type of AI for best type of AI for best type of AI for workflows that do workflows that do workflows that do n't involve n't involve n't involve human human human interaction." in them at all

interaction." in them at all interaction." in them at all . . . And let me just And let me just And let me just throw in a quick throw in a quick throw in a quick remark that remark that remark that just addresses just addresses use cases. We use cases. We tell people that tell people that tell people that it's not worth it's not worth it's not worth using if you're using if you're using if you're doing it as a one-off, doing it as a one-off, doing it as a one-off, if chat is perfect if chat is perfect if chat is perfect for that. for that. for that. If you're in If you're in If you're in research research research mode, this is good mode, this is good mode, this is good for things for things for things that need to be done that need to be done that need to be done frequently. It must frequently. It must frequently. It must be reliable, be reliable, be reliable, reproducible and reproducible and reproducible and verifiable. verifiable. Open Process was created for exactly these types of workflows Open Process was created for exactly these types of workflows . So the . So the implication here is that implication here is that you know what you you know what you you know what you need here. need here. need here. Oh, we got a question Oh, we got a question Oh, we got a question from a real from a real quantitative expert quantitative expert . . . And I mentioned it earlier, and And I mentioned it earlier, and And I mentioned it earlier, and I would call it not I would call it not I would call it not just inheritance just inheritance , but composition. , but composition. In fact, this is the In fact, this is the core function core function core function of the framework, and the of the framework, and the of the framework, and the template itself is built for template itself is built for template itself is built for composition. Yes, composition. Yes, composition. Yes, the implementation of multiple the implementation of multiple the implementation of multiple contracts in the contracts in the contracts in the same process, same process, same process, where the agent must where the agent must where the agent must satisfy many satisfy many satisfy many of them, is a of them, is a of them, is a key element. key element.

Right here. The other Right here. The other half is event- half is event- driven, driven, driven, which is perhaps which is perhaps which is perhaps a bit like a bit like a bit like inheritance. I think inheritance. I think inheritance. I think of it more as of it more as React-style memoization, where you have React-style memoization, where you have specific specific specific components. It's still a components. It's still a components. It's still a composition. You composition. You composition. You have certain have certain have certain components that components that components that have dependencies on have dependencies on components above them, and when components above them, and when the data in one of them the data in one of them the data in one of them changes, it of changes, it of changes, it of course course course propagates to propagates to propagates to everything everything everything below it. This is another below it. This is another below it. This is another place where Jev is very place where Jev is very place where Jev is very interesting, and the decision model is interesting, and the decision model is also very interesting, also very interesting, because we built because we built because we built this system and we this system and we this system and we had agents that had agents that had agents that were trying to determine were trying to determine were trying to determine if if if the state had changed, and if the state had changed, and if the state had changed, and if the state in an the state in an the state in an upstream upstream upstream dependency changes dependency changes , do we fire the , do we fire the , do we fire the downstream downstream downstream dependency? But that's dependency? But that's dependency? But that's very expensive to very expensive to very expensive to maintain, and Jev maintain, and Jev maintain, and Jev suddenly becomes this suddenly becomes this suddenly becomes this very cheap thing, very cheap thing, very cheap thing, like a semantic like a semantic like a semantic classifier or a classifier or a classifier or a semantic semantic semantic memoizer, which memoizer, which memoizer, which you can think of you can think of you can think of as a cache, and Jev decides as a cache, and Jev decides as a cache, and Jev decides when something is out of date when something is out of date , and now it , and now it , and now it runs all the runs all the runs all the downstream downstream downstream components. So this is components. So this is components. So this is another core another core another core component of the framework component of the framework . Except for . Except for . Except for the part about Jev. We are still the part about Jev. We are still the part about Jev. We are still working. We are still working. We are still working. We are still implementing this implementing this implementing this part. Yes, and there is no part. Yes, and there is no part. Yes, and there is no magic here. It's magic here. It's magic here. It's all just all just all just English. It's all English. It's all English. It's all just language. Uh, yeah, just language. Uh, yeah, just language. Uh, yeah, but yeah, and that's where but yeah, and that's where the composability the composability of the composition comes into play.

of the composition comes into play. of the composition comes into play. Another aspect of your Another aspect of your Another aspect of your point point point is that is that is that this is not just a this is not just a this is not just a template. So template. So template. So obviously, even before LLM, obviously, even before LLM, obviously, even before LLM, you might have had, let's you might have had, let's you might have had, let's say, not very say, not very say, not very early LLMs, or even early LLMs, or even early LLMs, or even coders might be coders might be coders might be filling out the template for filling out the template for filling out the template for you. You add a you. You add a you. You add a prefix to the template and prefix to the template and prefix to the template and fill it in. But fill it in. But fill it in. But here everything is different: here everything is different: here everything is different: what can be what can be what can be encoded? And you encoded? And you encoded? And you can code the can code the can code the entire multi-agent entire multi-agent entire multi-agent workflow. Here's a workflow. Here's a workflow. Here's a simple example simple example simple example I'll give: you I'll give: you I'll give: you might say might say might say I need the output in the I need the output in the I need the output in the form of an article. You know, and form of an article. You know, and maybe it maybe it maybe it will be this type of will be this type of will be this type of article, this article, this article, this length, something like that length, something like that length, something like that , right? There are , right? There are , right? There are some some some restrictions around restrictions around restrictions around this. And the entrance is the this. And the entrance is the this. And the entrance is the topic. It's like you topic. It's like you topic. It's like you have a Good Article have a Good Article have a Good Article for Me, and the for Me, and the for Me, and the parent agent parent agent parent agent decides when to exit, decides when to exit, decides when to exit, so with a so with a so with a stronger model you'll stronger model you'll stronger model you'll get a better get a better get a better result than with a result than with a result than with a weaker one. I weaker one. I weaker one. I have a sub-agent - a have a sub-agent - a researcher. researcher. researcher. This agent has This agent has This agent has access to the CLI and CLI skills access to the CLI and CLI skills because I don't want because I don't want him to, you know, him to, you know, him to, you know, use any use any use any random web search.

random web search. random web search. I have a sub-agent I have a sub-agent writer who writer who writer who has my special has my special has my special list of 10 list of 10 list of 10 really really really good articles good articles good articles that I've written that I that I've written that I that I've written that I want you to write like want you to write like want you to write like . I have a . I have a . I have a proofreading subagent, and proofreading subagent, and proofreading subagent, and all this is is probably all this is is probably all this is is probably just just 100 really 100 really important lines of important lines of important lines of markup, and it markup, and it markup, and it might take 10 hours. The might take 10 hours. The might take 10 hours. The important things to look at important things to look at is drafting a is drafting a contract where you, you, you contract where you, you, you contract where you, you, you , you, you, you, you, I , you, you, you, you, I , you, you, you, you, I mean, I mean mean, I mean mean, I mean these things have been around for a these things have been around for a these things have been around for a very very very long time. Any any long time. Any any long time. Any any any any any workflow builder, workflow builder, drag and drop workflow builder. This is a drag and drop workflow builder. This is a different way to think different way to think different way to think about it, because about it, because about it, because when you think about when you think about when you think about it in terms of In a it in terms of In a directed directed acyclic acyclic acyclic contract graph like a DAG, there contract graph like a DAG, there contract graph like a DAG, there can be can be can be intermediate steps intermediate steps intermediate steps that can be that can be that can be thought of as thought of as thought of as checkpoints, checkpoints, checkpoints, or as a situation or as a situation or as a situation where an intermediate where an intermediate where an intermediate step matches step matches step matches a contract, a process a contract, a process a contract, a process continues, continues, continues, reaches that reaches that reaches that contract and contract and contract and fulfills that one, and fulfills that one, and fulfills that one, and then that causes the then that causes the then that causes the next next next process to continue process to continue process to continue until until until that that that contract is fulfilled. contract is fulfilled. contract is fulfilled. So you So you So you can still, in a can still, in a can still, in a sense, determine sense, determine sense, determine how to achieve the goal by how to achieve the goal by how to achieve the goal by just writing these just writing these just writing these intermediate intermediate intermediate contracts along the way, contracts along the way, contracts along the way, while still allowing while still allowing while still allowing agents to be a little agents to be a little agents to be a little flexible in how flexible in how flexible in how to achieve that to achieve that to achieve that intermediate step.

intermediate step. intermediate step. Yes, and the form of this Yes, and the form of this Yes, and the form of this process. For example, process. For example, process. For example, when Claude when Claude when Claude introduced introduced introduced dynamic dynamic dynamic workflows, they had a workflows, they had a workflows, they had a blog post that blog post that blog post that talked about the six talked about the six talked about the six types, right? And one types, right? And one types, right? And one of them is fan- of them is fan- of them is fan- out, out, out, fan- fan- fan- out and out and out and MapReduce with subagents. MapReduce with subagents. MapReduce with subagents. One is One is One is adversarial adversarial adversarial analysis, and the third one is analysis, and the third one is analysis, and the third one is looping until looping until condition X is met, right? And all condition X is met, right? And all this can be expressed in a this can be expressed in a this can be expressed in a text program text program text program and then run. On and then run. On and then run. On any agent. This any agent. This any agent. This way, you can, way, you can, way, you can, for example, create a for example, create a for example, create a template for a template for a template for a multi-agent multi-agent multi-agent workflow workflow workflow without code. And again, without code. And again, without code. And again, this is only possible this is only possible this is only possible because agents are because agents are because agents are good at managing good at managing good at managing themselves, and because themselves, and because themselves, and because most test most test most test environments have environments have environments have subagents. So subagents. So subagents. So if you're using the if you're using the if you're using the regular version of Pie, regular version of Pie, regular version of Pie, which, at least which, at least which, at least last time last time last time I checked, doesn't have I checked, doesn't have I checked, doesn't have subagents, you'll subagents, you'll subagents, you'll need need need subagents to subagents to subagents to run more run more run more complex complex complex workflows. But essentially workflows. But essentially , any test , any test , any test environment that environment that environment that meets one meets one meets one of the requirements of the requirements of the requirements requires requires requires subagents. subagents. subagents. Subagents are Subagents are Subagents are practically the practically the practically the only option only option only option for most for most for most contexts.

contexts. contexts. Yes, for example, bash and Yes, for example, bash and Yes, for example, bash and subagents—in subagents—in subagents—in fact, you don't even fact, you don't even fact, you don't even need bash. We can need bash. We can need bash. We can run them in run them in run them in isolated isolated isolated virtual machines virtual machines virtual machines in code mode. in code mode. in code mode. Any other questions? We Any other questions? We Any other questions? We can give can give can give everyone 10 minutes. This is everyone 10 minutes. This is everyone 10 minutes. This is based on my based on my based on my many years of experience many years of experience many years of experience in engineering and in engineering and in engineering and interacting with interacting with interacting with people who are people who are people who are sometimes not very sometimes not very sometimes not very good at good at good at formulating formulating formulating requirements, and with requirements, and with requirements, and with other people other people other people who are very good at who are very good at who are very good at it. And that's it. And that's it. And that's also, I mean, also, I mean, also, I mean, we're doing we're doing we're doing internal internal internal testing and testing and testing and launching them. In launching them. In launching them. In contrast, contrast, contrast, we have a kind of we have a kind of we have a kind of evolutionary evolutionary evolutionary algorithm where algorithm where algorithm where we can we can we can reduce the kernel reduce the kernel reduce the kernel by checking by checking by checking whether they fit whether they fit whether they fit it. Thus, it. Thus, it. Thus, it is based on it is based on it is based on practice and practice and practice and empirical empirical empirical results. Yes, results. Yes, results. Yes, you can just you can just you can just use that as a use that as a use that as a specification, and you specification, and you specification, and you could have, for example, could have, for example, Matt Pocock's specification. You know, do Matt Pocock's specification. You know, do you like Matt? I you like Matt? I you like Matt? I created a funny created a funny created a funny spec to sort of spec to sort of mess with him, mess with him, it was sort of an it was sort of an it was sort of an auto-version of Pocock, and auto-version of Pocock, and auto-version of Pocock, and it was my way of it was my way of it was my way of proving that my proving that my proving that my way of connecting way of connecting way of connecting skills worked. So skills worked. So skills worked. So he has a he has a he has a workflow workflow workflow that goes like this: that goes like this: that goes like this: "Ask me questions "Ask me questions ," you know? And then ," you know? And then ," you know? And then you ask all these you ask all these you ask all these questions, and it's like TDD, questions, and it's like TDD, questions, and it's like TDD, and all these other ones. And you and all these other ones. And you and all these other ones. And you can put all of can put all of can put all of that into one program that into one program that into one program and just say, " and just say, " Make Make Make Matt Pocock's workflow Matt Pocock's workflow ," and "every sub-agent ," and "every sub-agent ," and "every sub-agent should be loaded should be loaded should be loaded at every step of the at every step of the at every step of the program. You must program. You must program. You must load this skill load this skill load this skill in sequence in sequence ." So yes, you ." So yes, you ." So yes, you can Absolutely can Absolutely can Absolutely right, we have right, we have right, we have people who people who people who use Open Prose, I use Open Prose, I use Open Prose, I didn't call it didn't call it spectrum-driven development , but by its nature , but by its nature , but by its nature it is it is spectrum-driven development . I would like you to . I would like you to . I would like you to try this.

try this. try this. Try it or Try it or Try it or find me later, and find me later, and find me later, and whatever whatever spectrum-driven development specification you have, we you have, we you have, we could could could hook it up to hook it up to hook it up to boot the kernel, boot the kernel, boot the kernel, run Open Prose, and run Open Prose, and run Open Prose, and then see how then see how then see how that works. I that works. I that works. I think this think this think this will work pretty will work pretty will work pretty well. This is good well. This is good well. This is good inspiration. I haven't inspiration. I haven't inspiration. I haven't tried this with tried this with spectrum-driven development contracts . I'd be happy to . I'd be happy to . I'd be happy to sit down and try it out and sit down and try it out and sit down and try it out and see if it see if it see if it overrides it overrides it overrides it or starts to or starts to or starts to conflict. conflict. conflict. That would be interesting That would be interesting . . . Here is one of the options Here is one of the options . . . Let's see how different Let's see how different Let's see how different skills affect skills affect skills affect this. this. this. That's what we have, and That's what we have, and software engineers software engineers use Open Prose to use Open Prose to use Open Prose to develop develop software. More and software. More and more of our more of our more of our users users users are using Open Prose for are using Open Prose for are using Open Prose for operations, operations, operations, product management, product management, product management, documenting documenting documenting quantitative quantitative quantitative models, and non- models, and non- software software engineering purposes. I engineering purposes. I engineering purposes. I think it's just a think it's just a think it's just a combination of, for example, combination of, for example, Markdown documentation, which is Markdown documentation, which is actually more actually more actually more interesting to interesting to interesting to non-engineers. non-engineers. non-engineers. Result. Yes. Any Result. Yes. Any Result. Yes. Any other questions?

other questions? other questions? Thank you all. I appreciate your Thank you all. I appreciate your Thank you all. I appreciate your time.