The Software Factory: From Bug Report to Production Code — Davis Palmie, Factory

Hello everyone, I'm Davis Hello everyone, I'm Davis Palmi, a technical Palmi, a technical Palmi, a technical specialist at specialist at specialist at Factory. Before that, I Factory. Before that, I Factory. Before that, I co-founded Lumetric co-founded Lumetric , an agent platform , an agent platform , an agent platform for investment for investment for investment teams that we teams that we teams that we developed through Y developed through Y developed through Y Combinator and was later Combinator and was later Combinator and was later acquired by Factory. And before that, acquired by Factory. And before that, acquired by Factory. And before that, I was a technical I was a technical I was a technical manager at the manager at the Slalom innovation lab. So , I've seen , I've seen , I've seen AI implementations both in AI implementations both in AI implementations both in large corporations large corporations large corporations and in organizations and in organizations and in organizations that were originally that were originally that were originally created around created around created around AI. With that in mind, AI. With that in mind, AI. With that in mind, today I'll tell today I'll tell today I'll tell you about the software you about the software you about the software factory and how to factory and how to factory and how to move towards an move towards an move towards an autonomous system autonomous system autonomous system that transforms input that transforms input that transforms input signals into working signals into working signals into working code. So, first of all, let code. So, first of all, let code. So, first of all, let me point out that we've me point out that we've me point out that we've gone through three gone through three gone through three different eras of AI different eras of AI engineering. Initially, engineering. Initially, engineering. Initially, there was autocomplete there was autocomplete there was autocomplete on the Tab key. on the Tab key. on the Tab key. Prediction of the Prediction of the Prediction of the next token. And next token. And next token. And the engineer still had a the engineer still had a the engineer still had a firm grip on the steering wheel firm grip on the steering wheel firm grip on the steering wheel . Then the AI . Then the AI . Then the AI learned to generate learned to generate learned to generate entire files, and the entire files, and the entire files, and the developer became a kind of developer became a kind of developer became a kind of selective selective selective copy-paster. But copy-paster. But copy-paster. But they still maintained they still maintained they still maintained strict control over the strict control over the AI's output. And then AI's output. And then agents appeared that were agents appeared that were agents appeared that were more tightly integrated more tightly integrated more tightly integrated with the code. They could with the code. They could with the code. They could independently gather independently gather independently gather context, call context, call context, call tools, and tools, and tools, and debug debug debug their own code. Autocomplete once their own code. Autocomplete once seemed like magic. And seemed like magic. And now it is, at best now it is, at best now it is, at best , a pleasant , a pleasant , a pleasant memory.

memory. memory. Note the speed with Note the speed with Note the speed with which AI engineering which AI engineering which AI engineering climbs the ladder of climbs the ladder of climbs the ladder of abstraction. From abstraction. From abstraction. From tokens to files, to tokens to files, to tokens to files, to context, and now context, and now to entire systems. to entire systems. Every such change Every such change requires adaptation. requires adaptation. But each step up the But each step up the ladder of abstraction ladder of abstraction ladder of abstraction makes engineers makes engineers makes engineers more valuable, not the more valuable, not the more valuable, not the other way around. So, other way around. So, other way around. So, we are now at a we are now at a we are now at a transformational transformational transformational point. Engineers point. Engineers point. Engineers are moving from are moving from are moving from writing code to writing code to writing code to managing agents. managing agents. And eventually, you And eventually, you organize these organize these organize these agents into a system. This agents into a system. This agents into a system. This is your software is your software is your software factory. Thus, factory. Thus, factory. Thus, the role of the engineer the role of the engineer the role of the engineer shifts to shifts to shifts to maintaining reliable maintaining reliable maintaining reliable fuses, fuses, fuses, identifying identifying identifying code deviations, and determining code deviations, and determining code deviations, and determining priorities and priorities and priorities and directions for work. directions for work. directions for work. Meanwhile, the system Meanwhile, the system Meanwhile, the system absorbs input absorbs input absorbs input signals, such as signals, such as signals, such as bug reports or bug reports or bug reports or user feedback user feedback , and produces ready-to- , and produces ready-to- , and produces ready-to- run code. So, if run code. So, if run code. So, if your AI systems your AI systems your AI systems are creating more are creating more are creating more code than humans, it code than humans, it code than humans, it raises important raises important raises important questions that questions that everyone rightly asks. First, how everyone rightly asks. First, how to prevent huge to prevent huge to prevent huge budget overspending budget overspending ? We've all seen ? We've all seen ? We've all seen the headlines about how the headlines about how Uber's CTO spent a year's Uber's CTO spent a year's budget on AI by budget on AI by budget on AI by April, or how Microsoft is April, or how Microsoft is April, or how Microsoft is revoking licenses revoking licenses revoking licenses to Claude to cut to Claude to cut to Claude to cut costs. With Factory, you are not

costs. With Factory, you are not costs. With Factory, you are not tied to a single tied to a single model provider. This means model provider. This means you get you get you get access to the full access to the full price- price- performance spectrum of performance spectrum of performance spectrum of large language large language large language models. You need models. You need models. You need maximum maximum maximum cost- cost- cost- effectiveness with the effectiveness with the effectiveness with the necessary minimum necessary minimum necessary minimum functionality for functionality for functionality for each task. As each task. As each task. As our CEO likes to say our CEO likes to say , if your child , if your child needs an needs an needs an algebra tutor, you'll definitely algebra tutor, you'll definitely algebra tutor, you'll definitely find someone find someone find someone cheaper than cheaper than cheaper than Albert Einstein. Albert Einstein. Second point: How do Second point: How do I ensure I ensure I ensure the best access to the best access to the best access to whatever model is the whatever model is the whatever model is the best right now? Well, best right now? Well, best right now? Well, we believe that we believe that we believe that having to having to having to bet on one bet on one bet on one favorite is a wrong favorite is a wrong favorite is a wrong choice. Especially with the choice. Especially with the choice. Especially with the increasing increasing increasing effectiveness of effectiveness of effectiveness of open source models, we open source models, we open source models, we strongly advocate for the strongly advocate for the strongly advocate for the ability to choose the ability to choose the ability to choose the right right right tool for a tool for a tool for a specific task specific task specific task or even for its or even for its or even for its dynamic dynamic dynamic routing. Number routing. Number routing. Number three: what will happen to three: what will happen to software engineers? What does software engineers? What does the future of the future of the future of software development look like software development look like software development look like ? Well, ? Well, ? Well, the role is clearly the role is clearly the role is clearly evolving, and the line evolving, and the line evolving, and the line between product between product between product work and work and work and engineering engineering engineering is blurring. Now, is blurring. Now, is blurring. Now, engineers engineers engineers will build and will build and will build and maintain maintain maintain the system the system the system responsible for responsible for responsible for building your building your building your working code and working code and working code and set the set the set the strategic strategic strategic direction for the direction for the direction for the software factory. And software factory. And software factory. And lastly: how will this lastly: how will this lastly: how will this change the organization?

change the organization? Well, team boundaries Well, team boundaries are also starting are also starting are also starting to blur, and to blur, and to blur, and context needs to context needs to flow seamlessly across those flow seamlessly across those boundaries. Information boundaries. Information boundaries. Information cannot be cannot be cannot be isolated. This was isolated. This was isolated. This was true for humans, true for humans, true for humans, but it is especially but it is especially but it is especially true for AI true for AI agents. Everyone on all agents. Everyone on all agents. Everyone on all your teams your teams your teams will share will share will share responsibility for the responsibility for the responsibility for the pillars of the software pillars of the software pillars of the software factory. Actually, factory. Actually, factory. Actually, coding was the coding was the coding was the easiest easiest easiest part. It was never a bottleneck part. It was never a bottleneck . Your context . Your context is expressed in natural is expressed in natural is expressed in natural language. I language. I language. I bet if bet if bet if you look at your you look at your you look at your actual engineering actual engineering actual engineering metrics, what is the metrics, what is the metrics, what is the average average average PR review time? How much PR review time? How much PR review time? How much time do you spend reproducing time do you spend reproducing user error reports or user error reports or testing and testing and testing and debugging code? debugging code? debugging code? Who updates your Who updates your Who updates your documentation with documentation with documentation with every PR and who every PR and who every PR and who keeps it keeps it keeps it current after current after current after every strategy every strategy every strategy meeting? Who supports meeting? Who supports meeting? Who supports your test suites? It your test suites? It your test suites? It often happens that often happens that often happens that testing teams, testing teams, testing teams, engineering teams, and engineering teams, and engineering teams, and product teams product teams product teams are located on are located on are located on different continents.

different continents. different continents. If you take your If you take your If you take your DORA metrics, the time from DORA metrics, the time from DORA metrics, the time from writing code to writing code to writing code to deploying to deploying to deploying to production, I'm production, I'm production, I'm sure, significantly sure, significantly sure, significantly exceeds the time exceeds the time exceeds the time spent spent spent directly on directly on directly on making engineering making engineering making engineering changes. The real changes. The real changes. The real need today is for need today is for need today is for all of these areas to come together all of these areas to come together all of these areas to come together . Now . Now . Now documentation documentation documentation is getting outdated, is getting outdated, is getting outdated, dead code is left behind dead code is left behind dead code is left behind , and there are too many , and there are too many , and there are too many layers of communication layers of communication layers of communication between your different between your different between your different teams. Again teams. Again teams. Again , this is hard for , this is hard for , this is hard for human engineers, but human engineers, but human engineers, but it it it feels even more feels even more feels even more acute for agents. A necessary acute for agents. A necessary acute for agents. A necessary task for task for task for building your building your building your factory is factory is to make sure that to make sure that to make sure that agents can agents can agents can combine all these combine all these combine all these services and understand the services and understand the services and understand the nuances within nuances within nuances within your organization. your organization. So, this brings So, this brings us to the concept of a us to the concept of a us to the concept of a software factory, software factory, software factory, which is a system of which is a system of which is a system of agents that will take agents that will take agents that will take you from you from you from input to input to input to deployment in deployment in deployment in production. Your agent is production. Your agent is production. Your agent is no longer no longer no longer just about coding. He just about coding. He just about coding. He takes on takes on takes on customer support, customer support, customer support, product, engineering, product, engineering, product, engineering, operational operational operational deployment, and deployment, and deployment, and many other roles in many other roles in many other roles in these areas. Your agent these areas. Your agent these areas. Your agent should triage should triage should triage incidents, build incidents, build incidents, build plans, create and plans, create and plans, create and update update update documentation, documentation, documentation, execute code, execute code, execute code, validate it, validate it, validate it, test it, test it, test it, deploy it, and then deploy it, and then deploy it, and then track that track that track that release. Of course, this is a release. Of course, this is a release. Of course, this is a huge amount of huge amount of huge amount of work. And you shouldn't

work. And you shouldn't work. And you shouldn't try to cover try to cover try to cover everything at once. First, everything at once. First, everything at once. First, it's too difficult a it's too difficult a it's too difficult a task, and second, task, and second, task, and second, your teams won't have your teams won't have your teams won't have time to time to time to learn to trust the learn to trust the learn to trust the system. Start with system. Start with system. Start with something narrow and something narrow and something narrow and proven, like proven, like incident triage. incident triage. Let the agent Let the agent Let the agent read read read Sentry alerts, collect Sentry alerts, collect Sentry alerts, collect traces and other traces and other traces and other context, and then context, and then context, and then publish everything to Slack. publish everything to Slack. publish everything to Slack. But leave it to But leave it to But leave it to the engineers to the engineers to the engineers to check it out and check it out and check it out and make the final make the final make the final decisions. And as the decisions. And as the agent provides agent provides agent provides increasingly accurate diagnoses, increasingly accurate diagnoses, increasingly accurate diagnoses, your team will begin your team will begin your team will begin to trust this system to trust this system to trust this system and better understand how it and better understand how it and better understand how it works. Each stage of the " works. Each stage of the " software factory software factory software factory " " " requires careful requires careful requires careful control and control and control and monitoring before monitoring before monitoring before you allow the system you allow the system you allow the system to run in a loop, to run in a loop, to run in a loop, but when you can but when you can but when you can connect the connect the connect the outputs to the outputs to the outputs to the inputs, you inputs, you inputs, you unlock the mechanism unlock the mechanism unlock the mechanism of self-improvement of self-improvement of self-improvement of the organization. We of the organization. We of the organization. We believe that your " believe that your " factory" should be factory" should be factory" should be based on based on based on several key several key several key principles. First, principles. First, principles. First, it must be model- it must be model- agnostic. Of course, agnostic. Of course, agnostic. Of course, different models are better different models are better different models are better at at at different tasks different tasks different tasks and have different and have different and have different cost- cost- cost- effectiveness. We effectiveness. We effectiveness. We found, for example, found, for example, found, for example, that it is possible to achieve that it is possible to achieve that it is possible to achieve the same the same the same efficiency in efficiency in efficiency in code review with GPT-5.2 and the code review with GPT-5.2 and the code review with GPT-5.2 and the latest latest latest Opus models, but at half the Opus models, but at half the Opus models, but at half the cost. And if cost. And if cost. And if you consider open-source

you consider open-source you consider open-source alternatives, alternatives, alternatives, the cost can be the cost can be the cost can be reduced by 10–30 reduced by 10–30 reduced by 10–30 times. So in Factory you times. So in Factory you times. So in Factory you manage the interaction manage the interaction manage the interaction of speed, of speed, execution efficiency and execution efficiency and cost. cost. Not being Not being tied to a single tied to a single tied to a single lab lab lab gives you access gives you access gives you access to the entire Pareto frontier of to the entire Pareto frontier of to the entire Pareto frontier of large language large language large language models. Second, models. Second, models. Second, your your your deployment model must be deployment model must be deployment model must be sovereign. You shouldn't have to sovereign. You shouldn't have to compromise on compromise on security architecture security architecture security architecture or policies to or policies to or policies to accommodate an accommodate an AI deployment model. In Factory, you choose your own workspace and workspace and deployment model. We deployment model. We deployment model. We support everything from support everything from support everything from fully managed fully managed fully managed solutions to custom solutions to custom solutions to custom servers and even servers and even servers and even systems in isolated systems in isolated systems in isolated networks. And third, networks. And third, networks. And third, your "factory" your "factory" your "factory" must be must be must be integrated into the full integrated into the full integrated into the full software development cycle (SDLC). software development cycle (SDLC). software development cycle (SDLC). These are no longer separate, These are no longer separate, These are no longer separate, unrelated unrelated unrelated tasks. Changes in tasks. Changes in tasks. Changes in documentation should documentation should documentation should be reflected in the code, be reflected in the code, be reflected in the code, and vice versa.

and vice versa. and vice versa. Planning goals should Planning goals should Planning goals should define define define testing criteria, testing criteria, testing criteria, regardless of the regardless of the regardless of the number of number of number of communication communication communication levels between levels between levels between strategy, strategy, strategy, product, development, product, development, product, development, and testers. and testers. Each of these pillars Each of these pillars supports the "factory," and supports the "factory," and supports the "factory," and weakness in one of weakness in one of weakness in one of them means them means them means vulnerability of the entire vulnerability of the entire vulnerability of the entire system. So, what does it system. So, what does it system. So, what does it look like in practice look like in practice look like in practice in an organization where AI is the in an organization where AI is the in an organization where AI is the foundation of work? We foundation of work? We foundation of work? We usually think of usually think of usually think of this as the unification and this as the unification and this as the unification and gradual gradual gradual automation of automation of automation of many sub-teams many sub-teams many sub-teams under the auspices of a single " under the auspices of a single " software factory software factory software factory ". ". Quite common Quite common tasks include tasks include tasks include triage and processing of triage and processing of triage and processing of incoming signals ( incoming signals ( e.g. e.g. e.g. Sentry notifications), where an agent Sentry notifications), where an agent Sentry notifications), where an agent performs performs performs diagnostics before diagnostics before diagnostics before a human has time to a human has time to a human has time to answer answer answer the call, as well as the call, as well as the call, as well as proactive proactive proactive security testing, security testing, security testing, scanning for scanning for scanning for secrets secrets secrets or vulnerabilities, or vulnerabilities, or vulnerabilities, creating and creating and creating and maintaining tests, and maintaining tests, and maintaining tests, and monitoring those that monitoring those that monitoring those that become obsolete become obsolete become obsolete or unnecessary.

or unnecessary. Also—synchronizing Also—synchronizing documentation with code documentation with code documentation with code and with various and with various and with various communication channels during communication channels during communication channels during meetings. And, of course, the meetings. And, of course, the meetings. And, of course, the classics: classics: classics: code review, commenting on code review, commenting on code review, commenting on merge requests (PRs), merge requests (PRs), merge requests (PRs), deploying and deploying and deploying and monitoring releases. monitoring releases. When the agent does When the agent does all this work, the role of the all this work, the role of the all this work, the role of the human engineer human engineer human engineer shifts to shifts to shifts to reacting, reacting, reacting, applying applying applying critical thinking, critical thinking, critical thinking, and correcting the and correcting the and correcting the AI's mistakes so that it AI's mistakes so that it AI's mistakes so that it can learn from them. can learn from them. Now all of this becomes Now all of this becomes transitive for transitive for transitive for all subcommands. all subcommands. Improving one Improving one group means group means group means improving the entire improving the entire improving the entire system. So why system. So why system. So why should your company should your company should your company care about a care about a software factory? The software factory? The larger and more larger and more larger and more dispersed you are, the dispersed you are, the dispersed you are, the more acutely you more acutely you more acutely you feel the linear feel the linear feel the linear pain of managing pain of managing pain of managing teams. In technical teams. In technical teams. In technical terms: your factory terms: your factory terms: your factory should reduce this " should reduce this " pain" from O(N) to O(1). You pain" from O(N) to O(1). You pain" from O(N) to O(1). You can define can define can define your framework, agents, your framework, agents, your framework, agents, documentation, documentation, documentation, testing policies, and— testing policies, and— testing policies, and— in fact—governance in fact—governance , just once, and , just once, and , just once, and then then then standardize it standardize it standardize it across the entire across the entire across the entire organization. This organization. This organization. This standard should standard should standard should ensure that everyone ensure that everyone ensure that everyone is held to the is held to the is held to the highest standard, not the highest standard, not the highest standard, not the lowest lowest lowest common denominator common denominator . If agents can . If agents can . If agents can move between move between move between these interfaces, these interfaces, these interfaces, they can they can they can maintain that

maintain that maintain that quality across all quality across all quality across all teams, and you don't teams, and you don't teams, and you don't have to have to have to spend your time linearly spend your time linearly spend your time linearly managing each one managing each one managing each one . With agents . With agents . With agents working across all of these working across all of these working across all of these environments, you have environments, you have environments, you have a single a single a single place to consolidate place to consolidate place to consolidate context and context and context and gain management gain management gain management visibility into the visibility into the visibility into the performance of your entire performance of your entire development lifecycle (SDLC). Just development lifecycle (SDLC). Just as as model-agnosticism model-agnosticism eliminates eliminates eliminates vendor lock-in, your vendor lock-in, your vendor lock-in, your agents should also agents should also agents should also be be be interface-agnostic. They interface-agnostic. They interface-agnostic. They should be should be should be easily accessible to easily accessible to easily accessible to your teams, your teams, your teams, regardless of regardless of regardless of what tools they what tools they what tools they use. use. use. Your teams shouldn't have to Your teams shouldn't have to compromise their compromise their processes or processes or processes or tools just to tools just to fit into fit into your AI provider's terms. your AI provider's terms. your AI provider's terms. At Factory, our agents At Factory, our agents At Factory, our agents are available everywhere: from are available everywhere: from are available everywhere: from remote machines and CLI remote machines and CLI remote machines and CLI to Slack—in any to Slack—in any to Slack—in any tool tool tool your your your teams use. And even teams use. And even teams use. And even if humans and agents if humans and agents if humans and agents use different use different use different interfaces, everything should interfaces, everything should interfaces, everything should share the same share the same share the same underlying mechanism and underlying mechanism and underlying mechanism and context. So, context. So, context. So, the process of creating a the process of creating a the process of creating a factory will reveal “ factory will reveal “ tribal knowledge,” tribal knowledge,” tribal knowledge,” manual processes, and manual processes, and manual processes, and outdated information outdated information . Eliminating them . Eliminating them . Eliminating them will help not only will help not only will help not only your agents, but also your your agents, but also your your agents, but also your engineering teams.

engineering teams. At Factory, we At Factory, we use use use the concept of the concept of the concept of agent readiness and evaluate it agent readiness and evaluate it agent readiness and evaluate it based on eight based on eight based on eight pillars. The first is pillars. The first is pillars. The first is validation. Do you have validation. Do you have validation. Do you have safeguards like safeguards like safeguards like linters and linters and linters and code formatters code formatters ? Next is your ? Next is your ? Next is your build system. Are build system. Are build system. Are your your your build commands well documented in CI? build commands well documented in CI? build commands well documented in CI? Feedback loops Feedback loops Feedback loops . Are . Are . Are your your your unit and integration unit and integration unit and integration tests set up reliably? And do you have tests set up reliably? And do you have tests set up reliably? And do you have documentation like documentation like documentation like readmes or agents.md files? readmes or agents.md files? Are your Are your development environments easily development environments easily development environments easily reproducible for reproducible for reproducible for agent use agent use ? Is your code ? Is your code ? Is your code modular, with clear modular, with clear modular, with clear boundaries and rules boundaries and rules boundaries and rules against chaos? Do against chaos? Do against chaos? Do you have means of you have means of you have means of observation? observation? observation? How long does it take you to How long does it take you to find out the cause of the find out the cause of the error? And do error? And do error? And do you perform you perform you perform proactive proactive proactive security scans? security scans? security scans? To not worry To not worry To not worry about secrets leaking about secrets leaking about secrets leaking or vulnerabilities. The or vulnerabilities. The main conclusion main conclusion is that is that is that if agents and human if agents and human engineers follow the engineers follow the engineers follow the same paths, then same paths, then same paths, then their high-quality their high-quality their high-quality arrangement is arrangement is arrangement is doubly beneficial. You doubly beneficial. You doubly beneficial. You can't manage what you can't manage what you can't manage what you can't see, this can't see, this can't see, this applies to both applies to both applies to both agents and agents and agents and engineers. You can engineers. You can engineers. You can start by assessing start by assessing agent readiness, but in any case, engineers any case, engineers any case, engineers should monitor should monitor should monitor their performance and their performance and their performance and behavior. In Factory, behavior. In Factory, behavior. In Factory, every action is subject to

every action is subject to every action is subject to audit. It works audit. It works audit. It works according to your according to your according to your standards, standards, standards, such as such as role-based access control and role-based access control and the principle of least the principle of least the principle of least privilege. Your privilege. Your privilege. Your teams building teams building teams building this software factory this software factory this software factory must first must first must first build trust and build trust and build trust and understanding. That is why it is understanding. That is why it is understanding. That is why it is worth implementing worth implementing worth implementing it gradually. it gradually. it gradually. You need to You need to You need to automate automate automate individual pillars first individual pillars first individual pillars first before converting before converting before converting the system to a loop. And the system to a loop. And the system to a loop. And again, precise again, precise again, precise execution does not save execution does not save execution does not save from bad design. from bad design. from bad design. It never It never It never worked. People are still worked. People are still worked. People are still critically critically critically needed for needed for needed for architecture, architecture, architecture, strategy, and strategy, and strategy, and direction. direction. Engineers Engineers will not be replaced. However, will not be replaced. However, will not be replaced. However, judgment and judgment and prioritization are now prioritization are now valued more than valued more than valued more than mechanical mechanical mechanical coding skills. coding skills. Agents won't be able to Agents won't be able to handle everything. It will be handle everything. It will be handle everything. It will be necessary necessary necessary to strengthen to strengthen to strengthen the fuses, the fuses, the fuses, correct the deviating code correct the deviating code correct the deviating code . People . People . People must remain the must remain the validation checkpoints in this validation checkpoints in this factory. But just as we factory. But just as we factory. But just as we no longer need to no longer need to no longer need to feed punch cards feed punch cards feed punch cards into a machine, we no longer into a machine, we no longer into a machine, we no longer need to write code need to write code need to write code by hand. Your team by hand. Your team by hand. Your team should set should set should set the strategy and the strategy and the strategy and risk parameters, risk parameters, risk parameters, then let then let then let the agents do the agents do the agents do the work. Understanding the work. Understanding the work. Understanding whether whether whether your agents are performing well your agents are performing well your agents are performing well depends on depends on depends on measuring the measuring the measuring the right right right metrics. In essence, we metrics. In essence, we metrics. In essence, we need to measure need to measure need to measure results, not results, not results, not tokens. The number of

tokens. The number of tokens. The number of lines of code and lines of code and lines of code and tokens generated tokens generated are metrics that are metrics that are metrics that can be manipulated. can be manipulated. can be manipulated. They are not really They are not really They are not really related to the end related to the end related to the end results. results. Leaderboards for token usage in large token usage in large companies only companies only companies only encourage encourage encourage waste. This is waste. This is waste. This is Goodhart's Law in Goodhart's Law in Goodhart's Law in real time. If real time. If real time. If you decide you decide you decide to measure to measure to measure token usage token usage , engineers will , engineers will , engineers will optimize optimize optimize this metric and it will this metric and it will this metric and it will no longer be no longer be no longer be useful. It's not about useful. It's not about useful. It's not about who spends who spends who spends the most tokens. It's about the most tokens. It's about who who gets the most out gets the most out gets the most out of these expenses. of these expenses. Some metrics that we Some metrics that we at Factory consider at Factory consider at Factory consider important are: important are: important are: signal-to-production time, signal-to-production time, signal-to-production time, number of number of number of human interventions, median human interventions, median human interventions, median recovery time. recovery time. recovery time. Code expiration date. And Code expiration date. And Code expiration date. And the cost of one PR. These are the cost of one PR. These are the cost of one PR. These are real, tangible real, tangible real, tangible results. Therefore, results. Therefore, results. Therefore, agent readiness is a agent readiness is a agent readiness is a prerequisite for prerequisite for prerequisite for effective engineering effective engineering , regardless of whether an , regardless of whether an , regardless of whether an agent or a human agent or a human agent or a human is working in this system.

is working in this system. Your agents Your agents need the same need the same need the same careful management careful management as your engineers: as your engineers: as your engineers: tools, tools, tools, access levels, access levels, access levels, verification systems. Don't verification systems. Don't verification systems. Don't just release them just release them just release them into your codebase into your codebase into your codebase . And don't exhaust your . And don't exhaust your . And don't exhaust your token limits. token limits. token limits. Measure what Measure what Measure what really matters. really matters. really matters. Your end Your end Your end users don't users don't users don't care how much you care how much you care how much you spend on AI. It is spend on AI. It is spend on AI. It is important for them to experience the important for them to experience the important for them to experience the significant improvements significant improvements significant improvements that your that your that your software factory will provide. software factory will provide. software factory will provide. Thank you.