Refinement-First Processes for AI Development | Stacy Cashmore | .NET Day 2026

We have Stacey Cashmore, We have Stacey Cashmore, and she talks about the and she talks about the improvement processes for improvement processes for developing AI, which developing AI, which developing AI, which sounds very sounds very sounds very sophisticated. sophisticated. Oh, yes. We are trying to Oh, yes. We are trying to make make make improvements here. improvements here. improvements here. That's good. This class is That's good. This class is That's good. This class is Oh, Stacey. Oh, Stacey. Oh, Stacey. Yes. Let's Yes. Let's Yes. Let's see if see if see if I can keep this accent going I can keep this accent going I can keep this accent going for 30 minutes. for 30 minutes. for 30 minutes. I doubt it. I doubt it. I doubt it. Just don't stoop Just don't stoop Just don't stoop to...Anyway, to...Anyway, to...Anyway, um, yeah, I'm um, yeah, I'm um, yeah, I'm looking forward to looking forward to looking forward to it. Hi, Stacey. I'm it. Hi, Stacey. I'm it. Hi, Stacey. I'm really looking forward to this. Hmm, really looking forward to this. Hmm, really looking forward to this. Hmm, hmm, I think the hmm, I think the hmm, I think the processes around how processes around how processes around how we work these we work these we work these days, hmm, are days, hmm, are days, hmm, are rare, and I'd like to rare, and I'd like to rare, and I'd like to understand a little understand a little understand a little more. So, um, yeah, I'm more. So, um, yeah, I'm more. So, um, yeah, I'm really looking forward to it. We really looking forward to it. We really looking forward to it. We can see your can see your can see your slides. Hmm, so if slides. Hmm, so if slides. Hmm, so if you're ready, then you're ready, then you're ready, then great. This is some kind of great. This is some kind of great. This is some kind of name. From boredom and name. From boredom and profanity to profanity to professional... professional... professional... Oh, there was Oh, there was Oh, there was a lot of a lot of a lot of profanity on this trip. profanity on this trip. This is my report. I'm here This is my report. I'm here for her. for her. You see, it will either be You see, it will either be improved or improved or improved or non-standard. We...We're non-standard. We...We're non-standard. We...We're going to figure it going to figure it going to figure it out. out. Perfectly. So thank you to Perfectly. So thank you to everyone who joined everyone who joined everyone who joined me, for me, for me, for watching every time. Yes, watching every time. Yes, watching every time. Yes, improving the first improving the first improving the first processes for processes for processes for developing AI. And this developing AI. And this developing AI. And this foul language foul language , as I said, has , as I said, has , as I said, has been heard frequently over the been heard frequently over the been heard frequently over the past 12 months.

past 12 months. Because for several Because for several years now, so many years now, so many years now, so many people on the Internet people on the Internet people on the Internet have been telling us that AI have been telling us that AI is the way to go, and developing AI is the way to go, and developing AI is the way to go, and developing AI will solve everything we will solve everything we will solve everything we need. If you need. If you need. If you use AI, use AI, use AI, anything is possible. But I anything is possible. But I anything is possible. But I never really never really found that found that promise fulfilled promise fulfilled , certainly not at the , certainly not at the , certainly not at the beginning. I always beginning. I always beginning. I always fell into a vicious fell into a vicious fell into a vicious circle. So I saw circle. So I saw circle. So I saw people shouting people shouting people shouting online about how online about how online about how good something was, whether good something was, whether good something was, whether it was a it was a it was a blog post, a blog post, a blog post, a YouTube video, a LinkedIn post, YouTube video, a LinkedIn post, YouTube video, a LinkedIn post, or whatever. You or whatever. You or whatever. You have to do it. have to do it. have to do it. This solves all your This solves all your This solves all your problems. So I problems. So I problems. So I tried. And tried. And tried. And I I I got fantastic got fantastic got fantastic results on simple things. And it was results on simple things. And it was results on simple things. And it was like: okay, like: okay, like: okay, maybe this will be maybe this will be maybe this will be good. And then I good. And then I good. And then I tried it on tried it on tried it on difficult things. And difficult things. And difficult things. And invariably the results invariably the results invariably the results were terrible. I was were terrible. I was were terrible. I was getting poor getting poor getting poor code quality. I ended up with a code quality. I ended up with a code quality. I ended up with a terrible terrible user interface. And user interface. And functionality functionality functionality that I just couldn't that I just couldn't that I just couldn't implement the way I implement the way I implement the way I wanted. And I finally wanted. And I finally wanted. And I finally decided: you know what?

decided: you know what? decided: you know what? For simple things, it's For simple things, it's For simple things, it's great. For the complex ones, great. For the complex ones, great. For the complex ones, maybe not yet. And maybe not yet. And maybe not yet. And then, you know, then, you know, then, you know, I go to 10, I go to 10, I go to 10, I start over. And I I start over. And I went around in circles many times. Every time I Every time I did it, I saw the same did it, I saw the same did it, I saw the same things. The things. The things. The good was always the same good was always the same . No more . No more . No more annoying annoying annoying templates. I saved a templates. I saved a templates. I saved a ton of time by just ton of time by just ton of time by just writing down everything I didn't writing down everything I didn't writing down everything I didn't want to write. And I could want to write. And I could want to write. And I could do that for a do that for a do that for a wide range of things. wide range of things. I also find I also find converting between converting between converting between programming languages programming languages programming languages very useful. A good very useful. A good very useful. A good example of this is example of this is example of this is that we that we that we used this used this used this when converting from when converting from when converting from TypeScript Playwriter to C#Playwriter TypeScript Playwriter to C#Playwriter , because we have , because we have , because we have mostly mostly mostly C# developers at work. And it C# developers at work. And it C# developers at work. And it was very simple. was very simple. was very simple. Not only could this Not only could this translate the tests well enough, but it translate the tests well enough, but it also helped me also helped me also helped me understand the understand the understand the base classes, which were base classes, which were base classes, which were different in TypeScript different in TypeScript different in TypeScript than in C#. And we than in C#. And we than in C#. And we turned it around in a turned it around in a turned it around in a couple of weeks, a couple of couple of weeks, a couple of couple of weeks, a couple of months, over time, months, over time, months, over time, just doing it in the just doing it in the just doing it in the background. It would background. It would background. It would take much take much take much longer if we longer if we longer if we did it manually, yes, did it manually, yes, did it manually, yes, much longer. And it much longer. And it much longer. And it became became became much easier much easier much easier to work with unknown APIs. The example to work with unknown APIs. The example to work with unknown APIs. The example I have for this is I have for this is I have for this is when I first started when I first started when I first started playing around with Whisper AI, I

playing around with Whisper AI, I playing around with Whisper AI, I wrote code in Python because wrote code in Python because wrote code in Python because AI loves Python. I've never AI loves Python. I've never written on this before. And written on this before. And within a day I was within a day I was within a day I was able to get able to get able to get an app up and running that did an app up and running that did an app up and running that did what I what I what I wanted, as a proof of wanted, as a proof of wanted, as a proof of concept, to concept, to concept, to make sure I make sure I make sure I could move forward and could move forward and could move forward and know that I wasn't wasting know that I wasn't wasting know that I wasn't wasting time wasting time. time wasting time. The most interesting thing here is that The most interesting thing here is that he was wrong he was wrong he was wrong more often than he was more often than he was more often than he was right. But it still right. But it still helped me immensely helped me immensely because he was wrong in a way because he was wrong in a way because he was wrong in a way that I could that I could that I could see what I see what I really needed to do. This really needed to do. This pointed in the direction, pointed in the direction, pointed in the direction, not solved not solved not solved the problem. And then the problem. And then the problem. And then we have the disadvantages. It we have the disadvantages. It we have the disadvantages. It cost a huge cost a huge cost a huge amount of tokens amount of tokens amount of tokens to do some to do some to do some things, and it still didn't things, and it still didn't things, and it still didn't work in the end work in the end work in the end when I tried it when I tried it when I tried it on the most difficult on the most difficult on the most difficult tasks. When I tasks. When I tasks. When I was going through those was going through those was going through those circles, it wasn't as much of circles, it wasn't as much of circles, it wasn't as much of an issue as it is an issue as it is an issue as it is today, because tokens today, because tokens today, because tokens were at the "we were at the "we were at the "we want you want you want you to be interested in this, so to be interested in this, so to be interested in this, so we'll charge you a we'll charge you a we'll charge you a penny" stage. So it was penny" stage. So it was penny" stage. So it was more of a waste of more of a waste of more of a waste of my time and resources my time and resources than anything else.

than anything else. But it was a little But it was a little annoying. This annoying. This annoying. This generated a huge generated a huge generated a huge amount of code, stacks amount of code, stacks amount of code, stacks and stacks of code. And I and stacks of code. And I and stacks of code. And I needed to needed to needed to review it review it review it before I could before I could before I could move on. And if move on. And if move on. And if you've ever you've ever you've ever looked at a 500- looked at a 500- file pull request file pull request file pull request for 500 files, you for 500 files, you for 500 files, you know it's not know it's not know it's not fun. And if a person fun. And if a person fun. And if a person makes me such a request, makes me such a request, makes me such a request, I usually regret I usually regret rejecting it on principle. So I just spent more spent more time watching it than I really time watching it than I really time watching it than I really wanted to. And if you're wanted to. And if you're wanted to. And if you're watching something watching something watching something that big and you that big and you that big and you get so bored, you're get so bored, you're get so bored, you're not really not really not really getting into it the way you getting into it the way you getting into it the way you should. And it should. And it should. And it also led to a also led to a also led to a feeling that I was no feeling that I was no feeling that I was no longer longer longer involved. I was typing involved. I was typing involved. I was typing hints, but I was hints, but I was hints, but I was typing hints, typing hints, typing hints, then typing then typing correction hints, then correction hints, then typing typing typing correction hints, and I just correction hints, and I just correction hints, and I just felt felt felt completely uninvolved completely uninvolved completely uninvolved in either the problem or the in either the problem or the in either the problem or the solution. I was just a solution. I was just a solution. I was just a middleman who middleman who middleman who didn't really didn't really didn't really add anything. And, of course, add anything. And, of course, add anything. And, of course, we had an unpleasant we had an unpleasant we had an unpleasant problem. Artificial problem. Artificial problem. Artificial intelligence can intelligence can intelligence can do too much and do too much and do too much and go off the rails. He go off the rails. He go off the rails. He does things you does things you does things you don't expect from him. It don't expect from him. It don't expect from him. It can provide a lot can provide a lot can provide a lot more code than you more code than you more code than you ever want, ever want, ever want, provide a lot provide a lot provide a lot more more more functionality functionality functionality that doesn't work, and it's that doesn't work, and it's that doesn't work, and it's just annoying at just annoying at just annoying at that level. He that level. He that level. He also never also never also never checked to see if anything was checked to see if anything was checked to see if anything was working or not. So

working or not. So working or not. So I had a lot of "hey, I had a lot of "hey, I had a lot of "hey, here it is, it's done. here it is, it's done. here it is, it's done. Now you can Now you can Now you can run it." And it either run it." And it either run it." And it either doesn't compile, or doesn't compile, or doesn't compile, or hangs during hangs during hangs during startup, or simply startup, or simply startup, or simply has functional has functional has functional errors. And in such errors. And in such errors. And in such situations, I too situations, I too situations, I too often went in circles, often went in circles, often went in circles, repeating the same repeating the same repeating the same two changes. So it was two changes. So it was two changes. So it was giving me a change, and giving me a change, and giving me a change, and it wasn't working. And it wasn't working. And it wasn't working. And you tell him it did you tell him it did you tell him it did n't work and you n't work and you n't work and you get the same get the same get the same AI, you're right. AI, you're right. AI, you're right. Sorry about that. Here, Sorry about that. Here, Sorry about that. Here, take this change. This one will take this change. This one will take this change. This one will suit you now. This one didn't suit you now. This one didn't suit you now. This one didn't work either. You work either. You work either. You would say so, a good would say so, a good would say so, a good substitution. I'm glad substitution. I'm glad substitution. I'm glad you saw it. And that would you saw it. And that would you saw it. And that would bring everything back to the bring everything back to the bring everything back to the original. And at that original. And at that original. And at that point it would just point it would just point it would just transition between these transition between these transition between these two states and two states and two states and not really not really not really solve the solve the solve the problem. This is the problem. This is the problem. This is the moment when the profanity really moment when the profanity really moment when the profanity really intervened intervened . And the last of the . And the last of the worst was that worst was that worst was that I just found it I just found it I just found it boring. I realized that boring. I realized that boring. I realized that this didn't satisfy this didn't satisfy this didn't satisfy the need I the need I the need I get from get from get from creating code creating code creating code solutions to things solutions to things solutions to things happening in the world.

happening in the world. happening in the world. And it took away my And it took away my And it took away my motivation from what motivation from what motivation from what I was doing. So, I was doing. So, I was doing. So, hello. I'm Stacey hello. I'm Stacey hello. I'm Stacey Cashmore. I am a Tech Explorer DevOps Cashmore. I am a Tech Explorer DevOps Cashmore. I am a Tech Explorer DevOps at Omniplan, a financial advisory at Omniplan, a financial advisory at Omniplan, a financial advisory software development company software development company based in based in Amsterdam, Amsterdam, Amsterdam, Netherlands. And one of Netherlands. And one of Netherlands. And one of my responsibilities in my responsibilities in my responsibilities in this position is as an this position is as an artificial intelligence product manager. artificial intelligence product manager. Where I'm trying to Where I'm trying to Where I'm trying to figure out how we figure out how we figure out how we can can can use use use artificial intelligence in both artificial intelligence in both artificial intelligence in both our process and our process and our process and our application in a our application in a our application in a way that makes way that makes way that makes sense both sense both sense both financially financially financially and in and in and in terms of the resources terms of the resources terms of the resources it consumes. I have it consumes. I have it consumes. I have been working in the been working in the been working in the .NET space since 2019, and since 2020 I .NET space since 2019, and since 2020 I .NET space since 2019, and since 2020 I feel extremely feel extremely feel extremely honored and respected to be honored and respected to be honored and respected to be able to call myself a able to call myself a able to call myself a Microsoft MVP in Microsoft MVP in Microsoft MVP in development technologies. development technologies. We needed to We needed to get around these problems get around these problems get around these problems because more and more people because more and more people because more and more people were talking about how were talking about how were talking about how well it worked. More and well it worked. More and well it worked. More and more people more people more people were showing that were showing that were showing that it worked for them, and obviously what it worked for them, and obviously what we were doing we were doing we were doing was wrong. As was wrong. As I read and watched I read and watched the videos, I saw that the videos, I saw that the videos, I saw that there were tools there were tools there were tools and processes that we and processes that we and processes that we needed to needed to needed to implement and implement and implement and experiment with experiment with experiment with to get around some of the to get around some of the to get around some of the issues. Not all issues. Not all issues. Not all at once, because if you at once, because if you at once, because if you implement two implement two implement two solutions at the same time, you solutions at the same time, you solutions at the same time, you have no idea have no idea have no idea what impact each one will have what impact each one will have what impact each one will have . And just . And just . And just step by step see step by step see what has been solved, what hasn't, and what has been solved, what hasn't, and what has been solved, what hasn't, and what new pain what new pain what new pain points have emerged

points have emerged points have emerged thanks to one of these thanks to one of these thanks to one of these solutions. The first thing we solutions. The first thing we solutions. The first thing we did was start did was start did was start using using using agents. Initially, we agents. Initially, we agents. Initially, we used Squad used Squad used Squad by Brady Gaster. by Brady Gaster. This is an This is an open source NPM package that open source NPM package that open source NPM package that you can install. you can install. And it essentially provides And it essentially provides you with specialized you with specialized you with specialized agents for your agents for your agents for your codebase. So, codebase. So, codebase. So, you might have an you might have an you might have an architect, you'll have a architect, you'll have a architect, you'll have a developer, a developer, a developer, a tester, a tester, a tester, a stakeholder, stakeholder, stakeholder, whatever that means whatever that means whatever that means for your for your for your codebase. This, sorry, codebase. This, sorry, codebase. This, sorry, helped us helped us helped us do what we do what we do what we needed to needed to needed to do. It’s important that do. It’s important that do. It’s important that when people hear about when people hear about when people hear about multiple agents, multiple agents, multiple agents, a lot of people I a lot of people I a lot of people I know thought, “Cool, know thought, “Cool, know thought, “Cool, multitasking. multitasking. multitasking. If we have If we have If we have five agents working on it, five agents working on it, five agents working on it, we can do we can do we can do five times as many things five times as many things .” But the reason .” But the reason .” But the reason agents worked so agents worked so agents worked so well was because they were well was because they were well was because they were less less less multitasking and multitasking and multitasking and more more more specialized. You specialized. You specialized. You no longer entrusted no longer entrusted no longer entrusted your very limited your very limited your very limited problem to this problem to this problem to this supermassive LLM.

supermassive LLM. Each agent had his Each agent had his own very strict own very strict own very strict set of responsibilities, and this set of responsibilities, and this set of responsibilities, and this allowed him allowed him allowed him to focus only on what to focus only on what to focus only on what he he he needed to do. needed to do. Now I will say that we Now I will say that we no longer no longer no longer use squad, use squad, use squad, not because there is anything wrong with squad not because there is anything wrong with squad not because there is anything wrong with squad , it is still a , it is still a , it is still a great tool, great tool, great tool, but for our but for our but for our use case use case use case it has become too it has become too it has become too expensive. There were expensive. There were expensive. There were too many agents out there, and too many agents out there, and too many agents out there, and we started creating our we started creating our we started creating our own agents to own agents to own agents to find our own middle find our own middle find our own middle ground. This is ground. This is ground. This is better, but not better, but not better, but not perfect. We still perfect. We still perfect. We still had a hard time had a hard time had a hard time keeping the whole keeping the whole keeping the whole process on track for what process on track for what process on track for what we were doing. There was still we were doing. There was still either either too much or too little code being generated. There too much or too little code being generated. There was still broken code in the compiler runtime, and was still broken code in the compiler runtime, and yes, we still yes, we still yes, we still went around in circles sometimes. The went around in circles sometimes. The went around in circles sometimes. The next next next tool we tool we tool we introduced to introduced to introduced to try to help try to help try to help with this was the with this was the with this was the open open open specification. Another specification. Another open source tool that open source tool that can be installed can be installed can be installed via npm, and it via npm, and it via npm, and it provides specific provides specific provides specific commands and skills, as well as commands and skills, as well as commands and skills, as well as providing providing providing protective barriers for protective barriers for protective barriers for your agents.

your agents. your agents. For example, the process is that For example, the process is that For example, the process is that you you you can research can research can research your codebase and the your codebase and the your codebase and the problem. And based on problem. And based on problem. And based on this research, you this research, you this research, you can suggest can suggest can suggest changes. The first two changes. The first two changes. The first two commands in commands in commands in the tool...When you the tool...When you the tool...When you propose changes, propose changes, propose changes, OpenSpec creates multiple OpenSpec creates multiple OpenSpec creates multiple documents. You documents. You documents. You receive a suggestion receive a suggestion that we want to change that we want to change . It is usually . It is usually . It is usually quite short. A quite short. A quite short. A design that can design that can describe in sufficient detail what describe in sufficient detail what needs to be done. And needs to be done. And needs to be done. And it creates a Gherkin-like list of it creates a Gherkin-like list of it creates a Gherkin-like list of specifications specifications specifications that will allow that will allow that will allow you to see you to see you to see what criteria what criteria what criteria OpenSpec thinks it will be OpenSpec thinks it will be OpenSpec thinks it will be tested against. Now, this tested against. Now, this tested against. Now, this could be one could be one could be one specification, or there specification, or there specification, or there could be multiple could be multiple could be multiple if your application is if your application is if your application is about different about different about different things. And finally, you things. And finally, you things. And finally, you are given a list of are given a list of are given a list of tasks so that you can tasks so that you can tasks so that you can actually see this actually see this actually see this change, AI agents change, AI agents change, AI agents will perform steps 1, 2, 3, will perform steps 1, 2, 3, will perform steps 1, 2, 3, 2.1, 2.2, 2.3, etc. At this 2.1, 2.2, 2.3, etc. At this 2.1, 2.2, 2.3, etc. At this point, you can point, you can point, you can change this manually.

change this manually. change this manually. If you see something If you see something If you see something that clearly doesn't need to that clearly doesn't need to that clearly doesn't need to be changed, change it be changed, change it be changed, change it manually and let manually and let manually and let OpenSpec know you changed it OpenSpec know you changed it . Or you can, . Or you can, . Or you can, for example, submit a for example, submit a for example, submit a request to change OpenSpec. request to change OpenSpec. One is not better than the other. This is more of a question like ... How much time do ... How much time do ... How much time do you think it would you think it would you think it would take you to complete each take you to complete each take you to complete each option? Remember to option? Remember to option? Remember to keep keep keep both in mind. This both in mind. This both in mind. This allowed us to really allowed us to really allowed us to really go through the process, go through the process, go through the process, make sure make sure make sure we had the necessary we had the necessary we had the necessary specs to specs to specs to verify that everything was verify that everything was verify that everything was working, follow working, follow working, follow this to-do list, this to-do list, this to-do list, which allowed us to get everything which allowed us to get everything which allowed us to get everything done according to plan. done according to plan. But that wasn't But that wasn't enough. This enough. This enough. This revealed another problem for us revealed another problem for us revealed another problem for us . And it's the . And it's the . And it's the same as same as same as with with with human developers: human developers: human developers: having having having specs doesn't specs doesn't specs doesn't mean you have a mean you have a mean you have a good way good way good way to know if something to know if something to know if something actually works. This actually works. This actually works. This gives agents a bigger gives agents a bigger gives agents a bigger clue, but doesn't clue, but doesn't clue, but doesn't solve the problem of solve the problem of solve the problem of them thinking them thinking them thinking something is fixed something is fixed something is fixed when it actually isn't when it actually isn't when it actually isn't . And, as I said, . And, as I said, . And, as I said, people have this people have this people have this problem too. The amount of problem too. The amount of problem too. The amount of code that is not code that is not code that is not tested because tested because tested because people don't like people don't like people don't like writing tests or people writing tests or people writing tests or people don't think about tests don't think about tests don't think about tests before writing before writing before writing code, agents do the code, agents do the code, agents do the same thing. So, the way same thing. So, the way same thing. So, the way to solve this is to solve this is to solve this is to get tested first.

to get tested first. to get tested first. Make sure Make sure we have green we have green tests after reading the tests. At this point, tests after reading the tests. At this point, tests after reading the tests. At this point, the agents know the agents know the agents know they are done. they are done. Any time the Any time the Any time the test is still red, test is still red, test is still red, he knows he he knows he he knows he needs to needs to needs to work harder. This work harder. This work harder. This prevents prevents prevents broken compilation, broken compilation, broken compilation, functionality functionality functionality that doesn't do what that doesn't do what that doesn't do what you expect, or it you expect, or it you expect, or it freezing on freezing on freezing on startup. Then you startup. Then you startup. Then you just need to just need to just need to figure out, do figure out, do figure out, do we want people to write we want people to write we want people to write tests, or do we want tests, or do we want tests, or do we want agents to write agents to write agents to write tests? 6 months ago tests? 6 months ago tests? 6 months ago I was talking about people. I was talking about people. I was talking about people. These days it's completely These days it's completely These days it's completely up to the agents. up to the agents. up to the agents. People don't like People don't like People don't like writing tests. There is a writing tests. There is a writing tests. There is a reason why quite a reason why quite a reason why quite a few tests are not few tests are not few tests are not written. Involving written. Involving written. Involving your agents in your agents in your agents in writing tests writing tests writing tests can be beneficial can be beneficial can be beneficial if you check if you check if you check what tests they are writing what tests they are writing . Make sure you . Make sure you . Make sure you agree to the agree to the agree to the test before it test before it test before it is written and is written and is written and always always always use use use different agents. My different agents. My different agents. My development agents don't development agents don't development agents don't write tests. All write tests. All write tests. All this context disappears. The this context disappears. The this context disappears. The developer simply developer simply developer simply solves these tests. This solves these tests. This solves these tests. This opens up another opens up another opens up another problem that problem that problem that arises again with us arises again with us arises again with us humans. Many of humans. Many of humans. Many of the problems we are the problems we are the problems we are addressing here are exactly the same as those faced by addressing here are exactly the same as those faced by addressing here are exactly the same as those faced by humans.

humans. Essentially, AI detects Essentially, AI detects problems that already exist in problems that already exist in problems that already exist in your processes. So, just your processes. So, just your processes. So, just because you have green because you have green because you have green tests doesn't mean tests doesn't mean tests doesn't mean they're good. As I they're good. As I they're good. As I said, developers said, developers said, developers should always should always should always check tests to check tests to check tests to make sure they are make sure they are make sure they are correct. I'm a firm correct. I'm a firm correct. I'm a firm believer that, um, people are the believer that, um, people are the ones who code ones who code . They are . They are . They are responsible for responsible for responsible for this code. "I don't know that this code. "I don't know that this code. "I don't know that AI wrote this" is not a AI wrote this" is not a AI wrote this" is not a valid excuse valid excuse valid excuse for bad code. The for bad code. The way we're way we're tackling this with humans is the same way tackling this with humans is the same way we've we've we've started tackling this started tackling this started tackling this with with with AI, which is by AI, which is by testing new testing new patients. patients. patients. We use We use We use Stryker.NET for this, hence the Stryker.NET for this, hence the Stryker.NET for this, hence the image. William image. William image. William Stryker, I think that's why Stryker, I think that's why Stryker, I think that's why Stryker Stryker Stryker got his name, got his name, got his name, because William Stryker in the because William Stryker in the because William Stryker in the X-Men world X-Men world X-Men world is trying is trying is trying to eradicate mutants, to eradicate mutants, to eradicate mutants, and that's what we're doing with and that's what we're doing with and that's what we're doing with Stryker. You can Stryker. You can Stryker. You can take a piece of code that is take a piece of code that is take a piece of code that is 100% 100% green, run green, run green, run it through Stryker and it through Stryker and it through Stryker and find that find that find that 5% of the tests are actually useful, and 5% of the tests are actually useful, and 5% of the tests are actually useful, and I had that happen I had that happen I had that happen once. I took an application once. I took an application once. I took an application that was completely that was completely that was completely green, and when you green, and when you green, and when you ran it through ran it through ran it through Stryker, you found Stryker, you found Stryker, you found that as long as it didn't throw that as long as it didn't throw that as long as it didn't throw an error and an error and an error and compile, all the compile, all the compile, all the tests would be tests would be tests would be green. In the end, I green. In the end, I green. In the end, I removed 80% of the code from removed 80% of the code from removed 80% of the code from this application and this application and this application and still had completely still had completely still had completely correct tests. So correct tests. So correct tests. So adding this adding this

mutation testing just mutation testing just ensures that the tests ensures that the tests ensures that the tests that are there are actually doing what they're that are there are actually doing what they're that are there are actually doing what they're supposed to do. supposed to do. supposed to do. So, at the review stage, So, at the review stage, So, at the review stage, we have our we have our we have our review agent. He runs review agent. He runs review agent. He runs Stryker based on the code and Stryker based on the code and Stryker based on the code and makes sure the makes sure the makes sure the tests still tests still tests still provide the provide the provide the expected coverage. expected coverage. But there are pitfalls here But there are pitfalls here . Artificial . Artificial . Artificial intelligence loves Python, intelligence loves Python, intelligence loves Python, and Stryker.NET generates JSON and Stryker.NET generates JSON output. Claude constantly output. Claude constantly output. Claude constantly wants to run wants to run wants to run Python scripts to Python scripts to Python scripts to parse JSON, and this sometimes parse JSON, and this sometimes parse JSON, and this sometimes took up 23% of our took up 23% of our took up 23% of our tokens that were tokens that were tokens that were available to us. Now we available to us. Now we available to us. Now we force it force it force it to use JQ to use JQ to use JQ so it can simply so it can simply so it can simply perform a direct perform a direct perform a direct query on the resulting query on the resulting query on the resulting JSON. This saves a JSON. This saves a JSON. This saves a huge amount of huge amount of huge amount of tokens and a huge tokens and a huge tokens and a huge amount of time. As amount of time. As amount of time. As with coverage, with coverage, with coverage, 100% is not required here. We are 100% is not required here. We are 100% is not required here. We are aiming for 70-80%. aiming for 70-80%. aiming for 70-80%. Any more than that and you'll Any more than that and you'll Any more than that and you'll get brittle get brittle get brittle doughs that aren't very doughs that aren't very doughs that aren't very good. And good. And good. And the return on the return on the return on investment you investment you investment you get by get by get by spending that spending that spending that extra time extra time extra time disappears. As long as you have disappears. As long as you have disappears. As long as you have something you can something you can something you can trust, everything is fine.

trust, everything is fine. At this point, we have made At this point, we have made real real real progress. This was the progress. This was the progress. This was the foundation of what we foundation of what we foundation of what we do today. We're do today. We're do today. We're just setting just setting just setting it up now. But we did it up now. But we did it up now. But we did find one thing that was find one thing that was find one thing that was causing us causing us causing us too much too much too much cognitive cognitive cognitive load and load and load and actually preventing actually preventing actually preventing us from using us from using us from using this new tool as much as this new tool as much as we should. And these are we should. And these are annoying annoying annoying interruptions. We interruptions. We interruptions. We will not run will not run will not run agents in YOLO mode agents in YOLO mode agents in YOLO mode on our laptops. I would on our laptops. I would on our laptops. I would n't do this on n't do this on n't do this on my home my home my home computer. I definitely would computer. I definitely would computer. I definitely would n't do this on a n't do this on a n't do this on a work computer. work computer. work computer. Because of this, every time it Because of this, every time it Because of this, every time it needed to needed to needed to do something that do something that do something that went outside of went outside of went outside of normal normal normal parameters, it would parameters, it would parameters, it would ask your ask your ask your permission to permission to permission to continue. Now, continue. Now, continue. Now, whether whether whether it was a request for permission to it was a request for permission to it was a request for permission to run a script or run a script or run a script or to read a file, to read a file, to read a file, sometimes these requests sometimes these requests sometimes these requests came literally came literally came literally every second for 10 every second for 10 every second for 10 seconds. Sometimes it was seconds. Sometimes it was seconds. Sometimes it was every 2-3 seconds. every 2-3 seconds. every 2-3 seconds. Sometimes it was Sometimes it was Sometimes it was about 5-6. Just about 5-6. Just about 5-6. Just when you thought you when you thought you when you thought you could do could do could do something else, no, it something else, no, it something else, no, it stopped you again. And stopped you again. And stopped you again. And that meant we were that meant we were that meant we were spending a lot of spending a lot of spending a lot of time not only time not only time not only keeping track of these keeping track of these keeping track of these things, but also not things, but also not things, but also not doing other doing other doing other things that we would like to do things that we would like to do things that we would like to do . The solution we . The solution we . The solution we have is that have is that have is that all our agents all our agents all our agents now run in now run in now run in Docker containers as a Docker containers as a Docker containers as a sandbox. We give sandbox. We give sandbox. We give them access to our them access to our them access to our code directory, and that's it.

code directory, and that's it. code directory, and that's it. They have no other They have no other They have no other access to the machine. This access to the machine. This access to the machine. This means we can means we can means we can bypass bypass bypass permissions dangerously. The machine can permissions dangerously. The machine can permissions dangerously. The machine can do whatever it do whatever it do whatever it wants. If it wants. If it wants. If it breaks the code, we breaks the code, we breaks the code, we just need to just need to just need to revert to the revert to the revert to the previous version previous version previous version or delete it and or delete it and or delete it and then re- then re- then re- clone. Agents do not clone. Agents do not clone. Agents do not have permission to have permission to have permission to submit changes. And submit changes. And submit changes. And if it manages to if it manages to if it manages to break the break the break the container, we simply container, we simply container, we simply delete the container delete the container delete the container and recreate it and recreate it and recreate it from the image. This from the image. This from the image. This gave us such a gave us such a gave us such a boost because boost because boost because now sometimes agents now sometimes agents now sometimes agents can work for 2 can work for 2 can work for 2 hours in the background hours in the background hours in the background , doing , doing , doing quite a lot of quite a lot of quite a lot of work. And we just work. And we just work. And we just leave it there and leave it there and leave it there and ignore it until we ignore it until we ignore it until we know we know we know we need to come back need to come back need to come back to it. There is much to it. There is much to it. There is much less cognitive less cognitive less cognitive load and we load and we load and we can start to can start to can start to use the use the use the extra time. extra time. The last thing we The last thing we found we had to found we had to found we had to change was our change was our change was our architecture. architecture. architecture. Since writing Since writing Since writing our program in 2020, our program in 2020, our program in 2020, we have we have we have been using the been using the been using the onion architecture onion architecture . So, we have our API, . So, we have our API, . So, we have our API, application, core, application, core, application, core, persistence, persistence, persistence, infrastructure, and it has infrastructure, and it has infrastructure, and it has worked well for worked well for worked well for us. It turns out that us. It turns out that us. It turns out that agents hate this agents hate this agents hate this because to find because to find because to find functionality for functionality for functionality for something they have to something they have to something they have to go through all the code, go through all the code, go through all the code, which takes time and which takes time and which takes time and tokens. And when they tokens. And when they tokens. And when they create files, they create files, they create files, they don't always create don't always create don't always create them on the right layer them on the right layer . So we sometimes . So we sometimes . So we sometimes had to do had to do had to do some cleaning up of what the some cleaning up of what the some cleaning up of what the agents were creating.

agents were creating. agents were creating. It still saved It still saved It still saved us a huge us a huge us a huge amount of time, but amount of time, but amount of time, but anything you do anything you do anything you do manually afterwards, you kind of manually afterwards, you kind of manually afterwards, you kind of think about hard think about hard think about hard work, and that's not the work, and that's not the work, and that's not the reason we were doing it reason we were doing it reason we were doing it . Now we have . Now we have . Now we have moved on to moved on to moved on to vertical slices. vertical slices. vertical slices. That is, we That is, we That is, we use a use a use a project for each of project for each of project for each of our virtual, our virtual, our virtual, uh, each of our uh, each of our uh, each of our vertical slices vertical slices vertical slices within our within our within our solution. And that means solution. And that means solution. And that means that every time we that every time we that every time we create code, create code, create code, AI agents know that they AI agents know that they only need to look at this one project only need to look at this one project to find everything they to find everything they to find everything they need. Again need. Again need. Again , fewer tokens, , fewer tokens, , fewer tokens, faster, and it produces fewer faster, and it produces fewer faster, and it produces fewer false positives false positives false positives when you don't know when you don't know when you don't know why things why things why things are created where are created where are created where they are. Our they are. Our they are. Our memory tools memory tools memory tools also work also work also work much better on much better on much better on this architecture than this architecture than this architecture than on the old, ahem, on the old, ahem, on the old, ahem, onion onion onion architecture. There's architecture. There's only one only one thing left. Doing it this thing left. Doing it this thing left. Doing it this way still just made me way still just made me way still just made me bored. bored.

Getting agents is not the reason I is not the reason I is not the reason I took on this job. took on this job. took on this job. That's not why I That's not why I That's not why I program after program after program after work, and that's not work, and that's not work, and that's not why I why I why I spend time doing something that spend time doing something that spend time doing something that could be useful could be useful could be useful to the community, whether it's to the community, whether it's to the community, whether it's helping out at helping out at helping out at conferences or conferences or conferences or speaking at them. I speaking at them. I speaking at them. I do it because I do it because I do it because I like the challenges like the challenges like the challenges it provides. It wasn't it provides. It wasn't it provides. It wasn't my style. I would my style. I would my style. I would burn out if I burn out if I burn out if I kept kept kept doing this, so a doing this, so a doing this, so a colleague who was colleague who was colleague who was experimenting with it and I experimenting with it and I experimenting with it and I realized we realized we realized we needed to do needed to do needed to do something different. And we wanted to something different. And we wanted to something different. And we wanted to translate this into a translate this into a translate this into a holistic flow, not holistic flow, not holistic flow, not just an just an just an implementation phase. People implementation phase. People implementation phase. People I know in my I know in my I know in my office who have also office who have also office who have also tried AI development tried AI development had the same experience. Everyone had the same experience. Everyone got bored. Everyone was got bored. Everyone was got bored. Everyone was disappointed. We disappointed. We disappointed. We needed to needed to needed to break out of this. break out of this. So, if you look So, if you look deeper, why is deeper, why is deeper, why is programming programming programming fun? Well, complex fun? Well, complex fun? Well, complex problems. Who doesn't problems. Who doesn't problems. Who doesn't like difficult like difficult like difficult problems that you problems that you problems that you really have to really have to really have to struggle with to struggle with to struggle with to solve? The feeling of solve? The feeling of solve? The feeling of satisfaction you satisfaction you satisfaction you get at the end get at the end is one of the things is one of the things is one of the things we have. But it's not just we have. But it's not just we have. But it's not just these complex these complex these complex problems. This makes problems. This makes problems. This makes the world a better place. I the world a better place. I the world a better place. I consider myself a pretty consider myself a pretty consider myself a pretty idealistic idealistic idealistic person, and I person, and I person, and I like to think like to think like to think that my work that my work helps people helps people using the using the using the app on the other app on the other app on the other end of the journey. I want to end of the journey. I want to end of the journey. I want to make their lives more make their lives more make their lives more joyful. I want to

joyful. I want to joyful. I want to help help help society do society do society do more things with more things with more things with less. And that's still less. And that's still less. And that's still necessary necessary necessary when you're working with when you're working with when you're working with artificial intelligence. artificial intelligence. It's just needed somewhere It's just needed somewhere else. So we else. So we else. So we need to get away from need to get away from need to get away from this boring world of, this boring world of, this boring world of, “Hey, here’s a change that “Hey, here’s a change that “Hey, here’s a change that needs to be made. needs to be made. needs to be made. Make a change, Make a change, Make a change, review the change, review the change, review the change, move on to the move on to the move on to the next change.” And next change.” And next change.” And actually actually actually make sure that we are make sure that we are make sure that we are involved in involved in involved in solving the process, solving the process, solving the process, that is, the problem. And to that is, the problem. And to that is, the problem. And to help us with that help us with that , we actually , we actually , we actually use Claude use Claude use Claude and Claude early in the and Claude early in the and Claude early in the process to process to process to help us help us help us do it better. To do it better. To do it better. To make sure we're make sure we're make sure we're on track and on track and on track and try to bring back the try to bring back the try to bring back the joy in life from what joy in life from what joy in life from what we were doing. we were doing. So, this is where improvement comes into So, this is where improvement comes into play. play. play. We have something that can be We have something that can be We have something that can be coded, but how do we coded, but how do we coded, but how do we actually actually actually turn it into a turn it into a turn it into a process that process that process that brings us brings us brings us energy? As I energy? As I energy? As I said, you need to be said, you need to be said, you need to be involved from the involved from the involved from the beginning, and to keep beginning, and to keep beginning, and to keep improvement from improvement from improvement from being boring, we being boring, we being boring, we also need to also need to also need to make it as make it as interactive as possible. So we interactive as possible. So we invited Claude, but invited Claude, but invited Claude, but only as a mediator.

only as a mediator. only as a mediator. Claude was never a Claude was never a Claude was never a leader, and he wasn't leader, and he wasn't leader, and he wasn't supposed to be a supposed to be a supposed to be a leader at this leader at this leader at this point. It should point. It should point. It should provoke you provoke you provoke you , make , make , make you think, but it you think, but it you think, but it shouldn't lead you. We shouldn't lead you. We shouldn't lead you. We changed our work changed our work changed our work items from our items from our improvement process, which improvement process, which we use as we use as we use as your DevOps, uh, from an your DevOps, uh, from an your DevOps, uh, from an epic plan, which epic plan, which epic plan, which was an umbrella for a was an umbrella for a was an umbrella for a bunch of features; and bunch of features; and bunch of features; and the umbrella for a the umbrella for a the umbrella for a multitude of functions, or multitude of functions, or multitude of functions, or functions, were functions, were functions, were the umbrella for a the umbrella for a the umbrella for a multitude of stories, and multitude of stories, and multitude of stories, and stories were the stories were the stories were the umbrella for a umbrella for a umbrella for a multitude of, um, tasks. multitude of, um, tasks. We changed that. Our We changed that. Our epic plan has become a epic plan has become a epic plan has become a problem. What problem. What problem. What problem are we problem are we problem are we trying trying trying to solve here? This meant to solve here? This meant that our epic plans were that our epic plans were that our epic plans were short-lived, but short-lived, but short-lived, but they also they also they also faded away faster, so you faded away faster, so you faded away faster, so you could really see could really see could really see your progress. your progress. your progress. Once we have a Once we have a Once we have a problem, and if a problem, and if a problem, and if a client offers a client offers a client offers a solution and says " solution and says " sometimes that's how it is," we sometimes that's how it is," we sometimes that's how it is," we make sure we make sure we make sure we go back from go back from go back from that to the problem that to the problem that to the problem to really to really to really make sure we make sure we make sure we know what we're know what we're know what we're trying to do. trying to do. We have selected several We have selected several hypotheses from this. We hypotheses from this. We hypotheses from this. We brought in Claude to brought in Claude to brought in Claude to help us help us brainstorm different brainstorm different ways to solve ways to solve ways to solve the problem to the problem to the problem to make sure we were make sure we were make sure we were n't just blinded by the n't just blinded by the n't just blinded by the first first first solution we solution we solution we came up with. We chose the came up with. We chose the came up with. We chose the best. The other two best. The other two best. The other two were noted were noted were noted because if the best doesn't because if the best doesn't because if the best doesn't work, then we'll work, then we'll work, then we'll go back to it.

go back to it. And then we broke And then we broke that down into functions, which that down into functions, which that down into functions, which were atomically were atomically were atomically valuable pieces of valuable pieces of valuable pieces of functionality functionality functionality needed to needed to needed to complete the entire complete the entire complete the entire solution. The features in solution. The features in solution. The features in the stories went through the the stories went through the the stories went through the same process as same process as same process as each other. We took that each other. We took that each other. We took that atomic part, atomic part, atomic part, whether whether whether it's a it's a it's a functional functional functional atomic part atomic part atomic part or it's a user or it's a user or it's a user story that's a story that's a story that's a technical or technical or technical or functional functional functional change for us. We change for us. We change for us. We made sure we made sure we made sure we knew why it knew why it knew why it fit into the plan, and fit into the plan, and fit into the plan, and how it fit into the how it fit into the how it fit into the plan. And then we plan. And then we plan. And then we used Claude as a used Claude as a used Claude as a rubber duck to rubber duck to rubber duck to make sure we did make sure we did make sure we did n't go off the rails, and n't go off the rails, and n't go off the rails, and we have a set of questions that we have a set of questions that we have a set of questions that I'll be doing a I'll be doing a I'll be doing a blog post about after blog post about after blog post about after this session, which this session, which this session, which ensured that we ensured that we ensured that we thought through the different thought through the different thought through the different parts of the systems. To parts of the systems. To parts of the systems. To make sure this is make sure this is make sure this is n't just a, um, n't just a, um, n't just a, um, bureaucratic exercise. bureaucratic exercise.

Claude actually Claude actually figured out what figured out what figured out what questions could be questions could be questions could be answered from the conversations answered from the conversations answered from the conversations and simply let and simply let and simply let us know which ones we us know which ones we us know which ones we needed to deal with needed to deal with needed to deal with before we before we before we moved on. moved on. Again, this gave Again, this gave us a big boost, us a big boost, us a big boost, a lot of excitement, a lot of excitement, a lot of excitement, because we're really because we're really because we're really starting to think about starting to think about starting to think about solving problems, solving problems, solving problems, not about implementing not about implementing not about implementing code. The only problem was that code. The only problem was that code. The only problem was that it it it gave us a lot of gave us a lot of gave us a lot of copying and copying and copying and pasting. So, by the pasting. So, by the pasting. So, by the time you got time you got time you got to the end of what you to the end of what you to the end of what you were doing, we were copying were doing, we were copying were doing, we were copying those eight questions, we those eight questions, we those eight questions, we were copying the were copying the were copying the specifications, we were specifications, we were specifications, we were copying the request for copying the request for specification discovery that we specification discovery that we generated as a generated as a generated as a result of our result of our result of our refinement. Hmm, and refinement. Hmm, and refinement. Hmm, and our entire conversation in our entire conversation in our entire conversation in this work item in this work item in this work item in Azure DevOps. Because none Azure DevOps. Because none Azure DevOps. Because none of us like of us like of us like doing it, and it was doing it, and it was doing it, and it was really really really error-prone, you could error-prone, you could error-prone, you could make make make mistakes all the time. We have added an MCP mistakes all the time. We have added an MCP server. So, now that server. So, now that server. So, now that we're at the we're at the we're at the stage stage where we're ready to say, where we're ready to say, where we're ready to say, "Yes, this work "Yes, this work "Yes, this work item is ready," item is ready," item is ready," Claude was telling us Claude was telling us Claude was telling us what he was going to put in there what he was going to put in there , and the system was creating it for us , and the system was creating it for us , and the system was creating it for us . Again, . Again, . Again, we always we always we always monitor the situation monitor the situation , we always , we always , we always check it, we check it, we check it, we just don't have to just don't have to just don't have to do the manual do the manual do the manual work. The last thing you work. The last thing you work. The last thing you need to do need to do need to do is when you is when you is when you are improving your code, it is are improving your code, it is are improving your code, it is very useful to have very useful to have very useful to have access to your code.

access to your code. Doing this from a Doing this from a desktop application is desktop application is desktop application is not easy, and we not easy, and we not easy, and we used used Claude's desktop application for this. Sure, you could Sure, you could share your share your share your files with this files with this files with this program, but it program, but it program, but it was slow and very was slow and very was slow and very expensive in tokens. So expensive in tokens. So expensive in tokens. So we actually we actually we actually ported our ported our ported our improvement into improvement into improvement into Claude's code itself, and we Claude's code itself, and we Claude's code itself, and we did it in the CLI. We did it in the CLI. We did it in the CLI. We involve involve involve other people in this, it's not other people in this, it's not other people in this, it's not just a just a just a developer thing. developer thing. developer thing. Obviously, the whole Obviously, the whole Obviously, the whole team has to team has to team has to work on this. And work on this. And work on this. And giving direct giving direct giving direct access to all access to all access to all the code and all the the code and all the the code and all the repositories repositories repositories meant we meant we meant we could make could make could make bigger changes bigger changes bigger changes because we could because we could because we could solve the whole solve the whole solve the whole problem of looking at problem of looking at problem of looking at our different our different our different repositories and knowing repositories and knowing repositories and knowing which repositories which repositories which repositories would need to be would need to be would need to be changed to get changed to get changed to get it right from the it right from the it right from the start. This saved start. This saved start. This saved us a ton of time us a ton of time us a ton of time studying the code and studying the code and studying the code and figuring out what figuring out what figuring out what we needed to do. we needed to do. And finally, to do this, And finally, to do this, we moved our we moved our we moved our improvement, our improvement, our improvement, our refinement, to refinement, to refinement, to creating working creating working creating working elements. In the past elements. In the past , as I'm sure happens in , as I'm sure happens in , as I'm sure happens in many companies many companies many companies , , , improvements improvements improvements were made long were made long were made long before coding started, before coding started, before coding started, and you had to and you had to and you had to switch between switch between switch between contexts, you contexts, you contexts, you lost knowledge as you went along lost knowledge as you went along lost knowledge as you went along . Now we are . Now we are . Now we are starting an epic starting an epic starting an epic project, and once this project, and once this project, and once this first feature is first feature is first feature is fully converted fully converted fully converted to stories, we to stories, we to stories, we are opening a new are opening a new are opening a new agent that implements agent that implements agent that implements those stories while we those stories while we those stories while we consider consider continuing to work continuing to work on those stories in on those stories in on those stories in other features. This is not other features. This is not other features. This is not for everyone. I know for everyone. I know for everyone. I know many developers many developers who complain about this

who complain about this , that they just , that they just , that they just want to be people want to be people want to be people who write code into who write code into who write code into computers. And as much as computers. And as much as computers. And as much as I hate to I hate to I hate to say it, I say it, I say it, I think that's exactly the role that think that's exactly the role that think that's exactly the role that needs to change in needs to change in needs to change in software development software development software development with the with the AI ​​tools AI ​​tools that are emerging. That that are emerging. That that are emerging. That developer who developer who developer who sits in a basement, drinks sits in a basement, drinks sits in a basement, drinks Coca-Cola, and eats Coca-Cola, and eats Coca-Cola, and eats pizza, as they described it to me pizza, as they described it to me pizza, as they described it to me at at at university back in the 90s, university back in the 90s, university back in the 90s, I don't think that will ever I don't think that will ever I don't think that will ever exist again. I exist again. I exist again. I believe that developers believe that developers believe that developers should always be should always be should always be involved in the involved in the involved in the improvement process, but in improvement process, but in improvement process, but in AI this seems to be AI this seems to be AI this seems to be optional. As I optional. As I optional. As I said, it's said, it's said, it's exciting, it's scary exciting, it's scary , it's new for everyone, and it's a , it's new for everyone, and it's a , it's new for everyone, and it's a journey for all of us. journey for all of us. And because of the speed at And because of the speed at which AI is changing, and which AI is changing, and which AI is changing, and because of the speed at because of the speed at because of the speed at which the which the which the cost of AI will change in cost of AI will change in cost of AI will change in the future, I think the future, I think the future, I think it won't be a short it won't be a short it won't be a short journey before things journey before things journey before things calm down. We calm down. We calm down. We need to need to need to make sure we make sure we make sure we keep thinking. keep thinking. Studies have been conducted on Studies have been conducted on whether chatbots are harmful whether chatbots are harmful to our brains to our brains to our brains and whether they affect and whether they affect and whether they affect our critical our critical our critical thinking. I've seen thinking. I've seen thinking. I've seen people say, "Well, it people say, "Well, it people say, "Well, it must be true because must be true because must be true because AI said so." And that's AI said so." And that's AI said so." And that's not true. This is absolutely not true. This is absolutely not true. This is absolutely wrong. We wrong. We wrong. We need to need to need to make sure that as we make sure that as we make sure that as we improve and improve and improve and code, we are in code, we are in code, we are in control control control and using and using and using AI agents as a AI agents as a AI agents as a tool, not tool, not tool, not something we can something we can something we can just hand over without just hand over without just hand over without thinking. And thinking. And thinking. And make sure you make sure you make sure you do it in small do it in small do it in small chunks. Because the

chunks. Because the chunks. Because the larger the fragment you larger the fragment you larger the fragment you give AI, the harder it is give AI, the harder it is give AI, the harder it is to improve it. to improve it. to improve it. Who wants to look at a Who wants to look at a Who wants to look at a thousand PR files? There's thousand PR files? There's thousand PR files? There's less chance of it less chance of it less chance of it going off the rails because it's a going off the rails because it's a going off the rails because it's a small fragment. And small fragment. And small fragment. And if it does go off the if it does go off the if it does go off the rails, it will have rails, it will have rails, it will have less impact, because it's less impact, because it's less impact, because it's just that small just that small just that small fragment. So we fragment. So we fragment. So we try to get try to get try to get stories stories stories that that that take no take no take no more than a few hours to complete with agents more than a few hours to complete with agents more than a few hours to complete with agents to make sure the to make sure the to make sure the output is output is output is small enough for us to small enough for us to small enough for us to safely safely safely review. So does review. So does review. So does this mean that after this mean that after this mean that after all this and the all this and the all this and the excitement I've excitement I've excitement I've brought back into my life brought back into my life , my boredom is gone , my boredom is gone , my boredom is gone forever? No. forever? No. Absolutely not. But Absolutely not. But was development ever was development ever was development ever free from boredom? There free from boredom? There free from boredom? There are always are always are always times when you don't times when you don't times when you don't like it. The difference is like it. The difference is like it. The difference is that boredom is that boredom is that boredom is no longer what I no longer what I no longer what I experience by experience by experience by default when default when default when I use our I use our I use our AI agents. Because I AI agents. Because I AI agents. Because I always use always use always use my brain to my brain to my brain to try new things. try new things. try new things. I am always I am always I am always experimenting and experimenting and experimenting and always putting always putting always putting the user first the user first the user first . It brings . It brings . It brings a lot of energy and a lot of energy and a lot of energy and excitement back into my excitement back into my excitement back into my life. And this is also life. And this is also life. And this is also related to the process.

related to the process. related to the process. These things don't stand still These things don't stand still These things don't stand still . As I've heard from the . As I've heard from the . As I've heard from the Agile world, if you're Agile world, if you're Agile world, if you're working in an Agile style working in an Agile style working in an Agile style and doing the same thing you were doing and doing the same thing you were doing and doing the same thing you were doing 12 months ago, you're 12 months ago, you're 12 months ago, you're not Agile. AI agents are not Agile. AI agents are not Agile. AI agents are so new that this is so new that this is so new that this is constantly changing. constantly changing. constantly changing. We change our We change our We change our process probably process probably process probably at least every 2 at least every 2 at least every 2 weeks. Sometimes we weeks. Sometimes we weeks. Sometimes we change it daily change it daily change it daily as we find things to as we find things to experiment with experiment with to improve. This must to improve. This must to improve. This must continue. And I continue. And I continue. And I want to address the want to address the want to address the elephant in the room, elephant in the room, elephant in the room, the people who are shouting the people who are shouting the people who are shouting about lines of code. When about lines of code. When about lines of code. When we were doing manual we were doing manual we were doing manual development, lines of code development, lines of code development, lines of code were a poor were a poor were a poor indicator. When indicator. When AI is involved in the development, it is an even worse AI is involved in the development, it is an even worse indicator. Earlier indicator. Earlier indicator. Earlier this year, back in this year, back in this year, back in March, people were shouting March, people were shouting that they were doing 37 that they were doing 37 that they were doing 37 lines of code a day. lines of code a day. What they didn't What they didn't shout about was shout about was whether they were creating whether they were creating whether they were creating value. I know value. I know value. I know people have looked at people have looked at people have looked at some of these some of these some of these repositories but haven't repositories but haven't repositories but haven't been impressed with the code been impressed with the code been impressed with the code they've seen. So they've seen. So they've seen. So we need to we need to we need to get rid of the idea that get rid of the idea that get rid of the idea that using AI agents using AI agents has to involve has to involve creating tens of creating tens of creating tens of thousands of lines of code. This thousands of lines of code. This thousands of lines of code. This should allow us to should allow us to should allow us to use our use our use our time to solve time to solve time to solve user problems user problems , solve them better, , solve them better, , solve them better, and create more and create more and create more valuable valuable valuable software. In fact

software. In fact , the first lecture I ever , the first lecture I ever , the first lecture I ever took, took, took, back in 1994 when I back in 1994 when I back in 1994 when I started started started university, was in a university, was in a university, was in a software software software engineering course, and the lecturer engineering course, and the lecturer engineering course, and the lecturer really tried really tried really tried to get across to us as to get across to us as software developers, as software engineers, that you software engineers, that you shouldn't be shouldn't be shouldn't be typists. You have to typists. You have to typists. You have to be the be the be the problem solver. Code problem solver. Code is simply your is simply your is simply your tool for how tool for how tool for how you do it. you do it. you do it. The solution you The solution you The solution you create for create for create for the user is what the user is what the user is what you need to you need to you need to focus on. I think that's focus on. I think that's true, just as true, just as true, just as true today as it was true today as it was true today as it was in 1994. Let in 1994. Let in 1994. Let artificial intelligence artificial intelligence artificial intelligence type. You type. You type. You think for yourself, solve think for yourself, solve think for yourself, solve problems, and problems, and problems, and come up with wonderful come up with wonderful come up with wonderful things for society. things for society. things for society. Thank you. Thank you. Oh, that's great, Stacey. I Oh, that's great, Stacey. I mean, you've mean, you've mean, you've touched on all the touched on all the touched on all the major pain major pain major pain points of the vigorous points of the vigorous points of the vigorous AI development that's AI development that's AI development that's happening. Such as happening. Such as happening. Such as the importance the importance the importance of specifications, of specifications, of specifications, keeping keeping keeping agent context in line, and agent context in line, and agent context in line, and specialized specialized specialized agents. Preserving agents. Preserving agents. Preserving the joy of it so the joy of it so the joy of it so that we don't that we don't that we don't just turn just turn just turn into automatons. Uh, and into automatons. Uh, and into automatons. Uh, and then automating then automating then automating what could what could what could simply be simply be simply be automated. And those automated. And those automated. And those lines of code do not lines of code do not lines of code do not equal equal equal code quality or code quality or code quality or problem solving, which is why we are here.

problem solving, which is why we are here. That's why you get paid That's why you get paid after all. after all. Right? Does anyone Right? Does anyone get paid for a get paid for a get paid for a line of code? line of code? line of code? No. I hope not. No. I hope not. No. I hope not. I would hope not, I would hope not, I would hope not, right? I mean right? I mean , as a consultant, people , as a consultant, people , as a consultant, people can try it, can try it, can try it, but, you know, it just but, you know, it just but, you know, it just won't work. Uh, yes, won't work. Uh, yes, won't work. Uh, yes, definitely. Um, I think definitely. Um, I think everyone has encountered everyone has encountered everyone has encountered this. If you've this. If you've this. If you've tried an agent in the tried an agent in the tried an agent in the last year, you've last year, you've last year, you've gotten into this gotten into this gotten into this deadly spiral deadly spiral deadly spiral where you create a request where you create a request , it generates code, the code is , it generates code, the code is crap. Okay, another crap. Okay, another crap. Okay, another query will fix that. query will fix that. query will fix that. He's spouting even more He's spouting even more He's spouting even more crap. Oh no, it's crap. Oh no, it's crap. Oh no, it's broken. And these are broken. And these are broken. And these are just just snowballs. snowballs. Uh, what I like to say is, Uh, what I like to say is, Uh, what I like to say is, you know, I can pretty much you know, I can pretty much you know, I can pretty much screw up the screw up the screw up the codebase myself. I don't codebase myself. I don't codebase myself. I don't need it to need it to escalate so quickly, okay? escalate so quickly, okay? But so is the agent, right But so is the agent, right But so is the agent, right ? So it's the same. Like ? So it's the same. Like ? So it's the same. Like these protective fences, these protective fences, these protective fences, I like it. I like it. I like it. You hit all the You hit all the You hit all the points that points that points that people really people really people really need to know about when it comes to need to know about when it comes to AI coding. AI coding. Yes, I really Yes, I really Yes, I really hope that people hope that people hope that people can learn from can learn from can learn from our journey and our our journey and our our journey and our mistakes. It's been, um, a mistakes. It's been, um, a mistakes. It's been, um, a long journey. We long journey. We long journey. We made a lot of made a lot of made a lot of mistakes. We mistakes. We mistakes. We swore a lot. Um, swore a lot. Um, swore a lot. Um, yeah, there yeah, there yeah, there wasn't enough wasn't enough profanity in this conversation, I think.

profanity in this conversation, I think. Well, yes, I tried to Well, yes, I tried to Well, yes, I tried to keep the language keep the language keep the language safe for work. safe for work. safe for work. I was hoping for I was hoping for I was hoping for more swear words, more swear words, more swear words, but oh, I but oh, I but oh, I love love love Martin's comment. I get paid Martin's comment. I get paid Martin's comment. I get paid per line of code, so I do per line of code, so I do per line of code, so I do n't write anything. n't write anything. Um, but no, I Um, but no, I really enjoyed this really enjoyed this really enjoyed this conversation too. There are a lot of conversation too. There are a lot of conversation too. There are a lot of really useful really useful really useful pragmatic pragmatic pragmatic comments there, and they comments there, and they comments there, and they also correlate really well also correlate really well also correlate really well with what with what with what you're you're you're saying there. Sometimes I saying there. Sometimes I saying there. Sometimes I feel something like feel something like imposter syndrome imposter syndrome related to what's related to what's related to what's going on with going on with going on with agent agent agent coding, coding, coding, processes, etc. And it's processes, etc. And it's processes, etc. And it's nice to hear other nice to hear other nice to hear other people say they've people say they've people say they've solved these problems, solved these problems, solved these problems, but improving and but improving and but improving and constantly changing constantly changing constantly changing processes is really the processes is really the processes is really the key thing, in my key thing, in my key thing, in my opinion, to opinion, to opinion, to just keep just keep just keep moving forward. moving forward. And, as I said, as And, as I said, as I was going through this, it I was going through this, it struck me at some point that these are not new struck me at some point that these are not new problems. This is a new problems. This is a new problems. This is a new way of identifying way of identifying way of identifying problems, but these are the problems, but these are the problems, but these are the same problems we've same problems we've same problems we've had in had in software development since software development since I started in the 90s I started in the 90s . It's just that you can . It's just that you can . It's just that you can see it much see it much see it much faster if you just faster if you just faster if you just hand it off to an agent hand it off to an agent hand it off to an agent who can create who can create who can create 37,000 lines of code a 37,000 lines of code a 37,000 lines of code a day. Suddenly all your day. Suddenly all your day. Suddenly all your problems fall into problems fall into problems fall into place.

place. Again, as you Again, as you just said, it's an just said, it's an just said, it's an amplifier. This is an amplifier. This is an amplifier. This is an amplifier in amplifier in amplifier in many ways many ways . Okay, fantastic. . Okay, fantastic. . Okay, fantastic. Unfortunately, I think we don't have much Unfortunately, I think we don't have much Unfortunately, I think we don't have much time. I think we time. I think we time. I think we could talk a lot about this could talk a lot about this could talk a lot about this . Someone . Someone . Someone just added to the chat just added to the chat just added to the chat that they are that they are that they are looking forward to the looking forward to the looking forward to the blog post. Hmm, we're also blog post. Hmm, we're also blog post. Hmm, we're also looking forward looking forward looking forward to reading it. to reading it. to reading it. Perfectly. I will definitely Perfectly. I will definitely Perfectly. I will definitely publish this. And now publish this. And now publish this. And now I'll go to the chat... I'll go to the chat... I'll go to the chat... Great... Great... Great... and see what's there. and see what's there. and see what's there. So if anyone has any So if anyone has any So if anyone has any questions, ask questions, ask questions, ask and I'll be there in about and I'll be there in about and I'll be there in about 5 minutes. 5 minutes. 5 minutes. Fantastically. Fantastically. Fantastically. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Thank you very much. Stay with us. Stay with us. Stay with us. Thank you. Thank you for Thank you. Thank you for Thank you. Thank you for inviting me. Have a inviting me. Have a inviting me. Have a nice day and nice day and nice day and enjoy enjoy enjoy the rest of your day.