I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

I believe everything I believe everything my agents tell me my agents tell me my agents tell me . And yet . And yet . And yet my agents are constantly my agents are constantly my agents are constantly wrong. This wrong. This wrong. This year I have passed year I have passed year I have passed over over over 107 billion tokens through agents. 107 billion tokens through agents. 107 billion tokens through agents. So I analyzed So I analyzed So I analyzed my agents' sessions from my agents' sessions from my agents' sessions from early 2026 to early 2026 to early 2026 to see where they see where they see where they made mistakes made mistakes . In about . In about . In about four out of ten four out of ten four out of ten sessions, the agent made a sessions, the agent made a sessions, the agent made a significant error. This is significant error. This is significant error. This is how often agents how often agents how often agents get it wrong in get it wrong in get it wrong in principle. It's not how principle. It's not how principle. It's not how often these errors often these errors often these errors made it into the made it into the made it into the final results of the final results of the final results of the work. Catching these work. Catching these work. Catching these errors before errors before errors before they reach me they reach me they reach me is the job of the agent- is the job of the agent- is the job of the agent- based validation method based validation method based validation method I've I've I've been using for been using for been using for most of this most of this most of this year, and it's that year, and it's that year, and it's that method I'm method I'm method I'm sharing with you today. This sharing with you today. This sharing with you today. This analysis revealed three analysis revealed three analysis revealed three ways in which agents ways in which agents ways in which agents continue to make continue to make continue to make mistakes. First, mistakes. First, mistakes. First, they exaggerate the they exaggerate the they exaggerate the work they have done. work they have done. work they have done. That is, they say that That is, they say that That is, they say that everything is ready or everything is ready or everything is ready or checked, even though it is checked, even though it is checked, even though it is not. Second, they are not. Second, they are not. Second, they are wrong about the facts. wrong about the facts. wrong about the facts. For example, For example, they misinterpret the source. they misinterpret the source. Or, thirdly, they are Or, thirdly, they are Or, thirdly, they are mistaken in their mistaken in their mistaken in their reasoning. So, reasoning. So, reasoning. So, the facts are correct, but the facts are correct, but the facts are correct, but the conclusion is not. The method the conclusion is not. The method the conclusion is not. The method consists of three consists of three consists of three levels of verification, one levels of verification, one levels of verification, one for each of the for each of the for each of the three reasons why three reasons why three reasons why agents consistently agents consistently agents consistently make mistakes. I make mistakes. I make mistakes. I created each level created each level created each level as a separate as a separate as a separate agent skill. Here's how agent skill. Here's how agent skill. Here's how each of them works, and then I'll each of them works, and then I'll each of them works, and then I'll show how they were show how they were show how they were applied in applied in applied in three real-world three real-world three real-world cases. The first cases. The first cases. The first level prevents level prevents agents from exaggerating the agents from exaggerating the work performed. Before work performed. Before work performed. Before the agent starts the agent starts the agent starts working, the task working, the task working, the task clearly states what clearly states what clearly states what it should

it should it should return: the file it is return: the file it is return: the file it is going to, and the command going to, and the command going to, and the command that checks it. that checks it. that checks it. It indicates the " It indicates the " paths of least paths of least paths of least resistance" that agents resistance" that agents resistance" that agents have used before have used before , so that this agent does not , so that this agent does not , so that this agent does not repeat them. The agent repeat them. The agent repeat them. The agent performing the check performing the check performing the check must complete it in must complete it in must complete it in one of two one of two one of two ways. The artifact ways. The artifact ways. The artifact is confirmed by a is confirmed by a is confirmed by a digital fingerprint of digital fingerprint of digital fingerprint of the file or files the file or files the file or files documenting the work, documenting the work, documenting the work, or the artifact or the artifact or the artifact is missing with is missing with is missing with an indication of the reason. an indication of the reason. Then the agent who Then the agent who sent the task sent the task sent the task runs the check runs the check runs the check on its own. Simply on its own. Simply on its own. Simply saying "done" is saying "done" is not enough. The second not enough. The second not enough. The second level prevents level prevents level prevents agents from making agents from making agents from making factual errors. When accuracy factual errors. When accuracy factual errors. When accuracy matters, matters, matters, first find in first find in first find in the source what the source what the source what the answer should the answer should the answer should agree with. The agree with. The agree with. The two agents then receive two agents then receive two agents then receive the same instructions the same instructions the same instructions and independently and independently and independently perform the same perform the same perform the same job. Neither of them job. Neither of them job. Neither of them sees the sees the sees the other's work. Everyone other's work. Everyone other's work. Everyone notes where notes where notes where each fact is taken from and each fact is taken from and each fact is taken from and checks their checks their checks their result for result for result for consistency with the facts consistency with the facts . Then a third agent, . Then a third agent, . Then a third agent, with the with the with the original source in front of him, original source in front of him, original source in front of him, compares the work of the two compares the work of the two compares the work of the two independent agents independent agents independent agents line by line and line by line and line by line and resolves each resolves each resolves each discrepancy by discrepancy by discrepancy by referring to the referring to the referring to the source. He never source. He never source. He never forces the results to forces the results to forces the results to match. The third match. The third match. The third level prevents level prevents level prevents agents from making errors in agents from making errors in agents from making errors in reasoning. When reasoning. When reasoning. When clarity of thought is really clarity of thought is really clarity of thought is really important, we conduct a important, we conduct a important, we conduct a competitive check competitive check competitive check in three stages. First, in three stages. First, in three stages. First, the agent conducts the agent conducts the agent conducts an audit of his own work.

an audit of his own work. an audit of his own work. It lists every It lists every It lists every claim it claim it claim it makes and labels makes and labels makes and labels each one as each one as each one as measured, measured, measured, source-verified, inferred, source-verified, inferred, source-verified, inferred, or assumed. Anything or assumed. Anything or assumed. Anything he can't he can't he can't confirm is either confirm is either confirm is either verified or verified or verified or labeled as an labeled as an labeled as an assumption. Second, assumption. Second, assumption. Second, reviewers who reviewers who reviewers who lack the context of the lack the context of the lack the context of the original agent, original agent, original agent, each acting as a each acting as a each acting as a person with a motive person with a motive person with a motive to find errors, to find errors, to find errors, read it without read it without read it without bias. Each of bias. Each of bias. Each of them plays the role of a them plays the role of a them plays the role of a real reader, real reader, real reader, receiving work and receiving work and receiving work and sources, but no sources, but no sources, but no history, drafts, or history, drafts, or history, drafts, or conclusions to conclusions to conclusions to agree with. agree with. Their conclusions Their conclusions are also checked against are also checked against are also checked against the source before the source before the source before anything is changed. anything is changed. Third, once Third, once Third, once the work has been reviewed, the work has been reviewed, the work has been reviewed, completely new completely new completely new reviewers who have reviewers who have reviewers who have never seen it before never seen it before never seen it before check the final check the final check the final version. Each level version. Each level version. Each level builds on the previous one builds on the previous one builds on the previous one as the stakes as the stakes as the stakes increase. In these increase. In these increase. In these cases, each cases, each cases, each check took check took check took minutes, not hours, minutes, not hours, minutes, not hours, and the agents do the checking and the agents do the checking and the agents do the checking , not me. With , not me. With , not me. With our scale of our scale of our scale of work, independent work, independent work, independent verification is not an verification is not an verification is not an option. So, option. So, option. So, let's look let's look let's look at each level of at each level of at each level of verification verification verification applied in three applied in three applied in three real-world cases.

real-world cases. First, a memory leak. The First, a memory leak. The open source dictation app open source dictation app I use had a I use had a I use had a memory leak. It has memory leak. It has memory leak. It has grown to 9 GB. We grown to 9 GB. We grown to 9 GB. We decided to find the decided to find the decided to find the leak, fix it, leak, fix it, leak, fix it, and get the and get the fix accepted by the project maintainer. And to project maintainer. And to fix it, we fix it, we fix it, we had to had to had to figure out what figure out what figure out what was actually was actually was actually causing it. My agents causing it. My agents causing it. My agents and I went through this, and and I went through this, and and I went through this, and three times the agent told three times the agent told three times the agent told me things that were me things that were me things that were not true. First, the not true. First, the not true. First, the subagent I subagent I subagent I sent to measure the sent to measure the sent to measure the leak with one leak with one leak with one tool tool tool used another and used another and used another and reported reported reported the job as done the job as done the job as done without saying that he did not without saying that he did not without saying that he did not use the use the use the tool as tool as tool as instructed. At first, instructed. At first, instructed. At first, no one no one no one noticed, including noticed, including noticed, including the agent who issued the the agent who issued the the agent who issued the task. This task. This task. This became clear when I became clear when I became clear when I questioned the questioned the questioned the results. The file results. The file results. The file that was supposed to create the that was supposed to create the that was supposed to create the correct correct correct tool didn't tool didn't tool didn't exist, and no one exist, and no one exist, and no one noticed. This is why the noticed. This is why the first first level of verification is needed. In level of verification is needed. In level of verification is needed. In this case, we this case, we this case, we sent a subagent sent a subagent sent a subagent to create a specially to create a specially to create a specially equipped version of the equipped version of the equipped version of the application and collect application and collect application and collect telemetry while it was telemetry while it was telemetry while it was running. However, the assembly running. However, the assembly running. However, the assembly failed. Because failed. Because failed. Because we instructed we instructed we instructed the subagent to provide the subagent to provide the subagent to provide documentation of all of documentation of all of documentation of all of his work, instead of his work, instead of his work, instead of just “done,” just “done,” just “done,” he said that the he said that the he said that the main deliverable main deliverable main deliverable was not delivered and was not delivered and was not delivered and explained why. He explained why. He explained why. He reported that an reported that an reported that an artifact was missing, a artifact was missing, a artifact was missing, a software build software build software build , and that there , and that there , and that there were 25 were 25 were 25 compiler errors, and compiler errors, and compiler errors, and correctly marked correctly marked correctly marked what he was able to provide as what he was able to provide as what he was able to provide as partial. Sending partial. Sending partial. Sending a subagent with the a subagent with the first first level verification task enabled took 19 level verification task enabled took 19 level verification task enabled took 19 minutes and about 16 minutes and about 16 minutes and about 16 million tokens.

million tokens. Two other significant Two other significant agent inaccuracies were about to agent inaccuracies were about to agent inaccuracies were about to become become become public, and the stakes public, and the stakes public, and the stakes were higher, so for were higher, so for were higher, so for those we borrowed those we borrowed those we borrowed third- third- third- tier methods—reviewers tier methods—reviewers tier methods—reviewers who have no who have no who have no agent context. We'll agent context. We'll agent context. We'll soon see soon see soon see all three stages of all three stages of all three stages of third- third- third- level validation in the example level validation in the example level validation in the example I explained I explained I explained earlier, but here's how it earlier, but here's how it earlier, but here's how it worked to worked to worked to fix the fix the fix the memory leak. In the second memory leak. In the second memory leak. In the second case, the agent case, the agent case, the agent blamed the wrong code blamed the wrong code . To find the source of . To find the source of . To find the source of the leak, he wrote a the leak, he wrote a the leak, he wrote a small test that small test that small test that used the used the used the same library as the same library as the same library as the application. The test also application. The test also application. The test also showed a leak, so the showed a leak, so the showed a leak, so the agent agent agent concluded that concluded that concluded that the library was to blame. He the library was to blame. He the library was to blame. He said, "This is said, "This is said, "This is weighty enough to serve weighty enough to serve ." The reviewers ." The reviewers ." The reviewers reproduced the test and reproduced the test and reproduced the test and found that it found that it found that it missed an important missed an important missed an important step required by step required by step required by the library. When the library. When used properly, used properly, the library had no the library had no the library had no leaks. The problem leaks. The problem leaks. The problem was how was how was how the application the application the application used this used this used this library. In the third library. In the third library. In the third major error by major error by major error by agents agents agents working on a working on a working on a memory leak, the memory leak, the memory leak, the agent described our agent described our agent described our fix to a pull fix to a pull request and said that request and said that request and said that it completely fixed the it completely fixed the it completely fixed the leak, but measured leak, but measured leak, but measured it in a way that left most of it it in a way that left most of it undetected.

undetected. The reviewers noticed The reviewers noticed The reviewers noticed this too. The fix this too. The fix this too. The fix reduced the leak but did reduced the leak but did reduced the leak but did not eliminate it not eliminate it not eliminate it completely. So a pull completely. So a pull request was submitted with request was submitted with request was submitted with honest numbers, and honest numbers, and honest numbers, and the maintainer accepted the maintainer accepted the maintainer accepted the fix and the fix and the fix and merged it. Each of merged it. Each of merged it. Each of these these these third-level checks third-level checks third-level checks took about 15 took about 15 took about 15 minutes and together minutes and together minutes and together cost approximately 6 cost approximately 6 cost approximately 6 million tokens. million tokens. million tokens. For the next For the next For the next example, we example, we example, we will consider will consider will consider the accounting of a the accounting of a the accounting of a portfolio company portfolio company . Buying and . Buying and . Buying and building a business building a business building a business can be a messy can be a messy can be a messy affair. For one of affair. For one of affair. For one of our portfolio our portfolio our portfolio companies, we companies, we companies, we needed to needed to needed to convert annual convert annual convert annual scanned reports from the scanned reports from the scanned reports from the source source source accounting accounting accounting system into a complete and system into a complete and system into a complete and accurate record of every accurate record of every accurate record of every business transaction for business transaction for business transaction for financial reporting financial reporting . The agent pulled a . The agent pulled a . The agent pulled a year's worth of data and returned it year's worth of data and returned it year's worth of data and returned it saying, "Hey, I saying, "Hey, I saying, "Hey, I recommend selectively recommend selectively recommend selectively checking a few of checking a few of checking a few of these transactions." This these transactions." This these transactions." This shifted the burden of shifted the burden of shifted the burden of verifying the verifying the verifying the agent's work back onto us, which is agent's work back onto us, which is agent's work back onto us, which is unacceptable. unacceptable. Extracting data from Extracting data from scans often goes scans often goes scans often goes wrong in ways that are wrong in ways that are wrong in ways that are tedious and tedious and tedious and difficult to difficult to difficult to detect. detect. detect. Transactions Transactions Transactions are missed are missed are missed entirely, entirely, entirely, fall into the wrong fall into the wrong fall into the wrong months, or months, or months, or are recorded with are recorded with are recorded with incorrect incorrect incorrect details. One agent details. One agent details. One agent simply reporting simply reporting simply reporting on execution on execution on execution cannot say whether cannot say whether cannot say whether any of this happened any of this happened . And that's exactly why there . And that's exactly why there . And that's exactly why there 's a second level of 's a second level of 's a second level of verification method. Here verification method. Here verification method. Here we use we use we use the balance printed the balance printed the balance printed on the reports as a on the reports as a on the reports as a control total control total control total for each month.

for each month. for each month. Two agents independently Two agents independently Two agents independently extracted data for extracted data for extracted data for each month, and each each month, and each each month, and each returned to the reports returned to the reports returned to the reports until each until each until each month's balance converged. A month's balance converged. A third agent then third agent then compared these two compared these two compared these two samples line by samples line by samples line by line and checked each line and checked each line and checked each discrepancy against the discrepancy against the discrepancy against the reports themselves. reports themselves. Finally, two more Finally, two more agents with the same agents with the same agents with the same tasks tasks tasks checked each of the checked each of the checked each of the final final final transaction records before transaction records before transaction records before accepting them. This accepting them. This accepting them. This “fan-out, fan- “fan-out, fan- in” model for agents with in” model for agents with in” model for agents with the same the same the same tasks is tasks is tasks is extremely extremely extremely effective in effective in effective in ensuring ensuring ensuring accuracy and accuracy and accuracy and reliability. When we reliability. When we reliability. When we performed this performed this performed this second-level review, we second-level review, we second-level review, we found erroneous found erroneous found erroneous transactions, transactions, transactions, incorrect operations, incorrect operations, incorrect operations, and transactions and transactions and transactions attributed to attributed to attributed to inappropriate inappropriate inappropriate months. A comparison of the months. A comparison of the months. A comparison of the two samples revealed two samples revealed two samples revealed 56 discrepancies in 56 discrepancies in 56 discrepancies in descriptions, dates, and descriptions, dates, and descriptions, dates, and notes. But, notes. But, notes. But, in the end, the whole year in the end, the whole year in the end, the whole year came down to a penny. came down to a penny. came down to a penny. This entire verification This entire verification This entire verification took about 20 took about 20 took about 20 minutes and cost 7.5 minutes and cost 7.5 minutes and cost 7.5 million tokens. million tokens. million tokens. Finally, an example with the Finally, an example with the Finally, an example with the highest stakes. highest stakes. highest stakes. Again, confusion in Again, confusion in Again, confusion in the affairs of the portfolio the affairs of the portfolio the affairs of the portfolio company. One of them company. One of them company. One of them had a had a had a billing dispute with a billing dispute with a billing dispute with a supplier, and it supplier, and it supplier, and it resulted in thousands of resulted in thousands of resulted in thousands of dollars in dollars in dollars in legal legal legal consequences. We consequences. We consequences. We needed to needed to needed to send a demand letter send a demand letter send a demand letter clearly stating our clearly stating our clearly stating our position and requirements, and position and requirements, and position and requirements, and every statement in every statement in every statement in it had to be it had to be it had to be supported by a supported by a supported by a contract and contract and contract and invoices. The agent invoices. The agent invoices. The agent drafted the letter based drafted the letter based drafted the letter based on the fact that the contract with on the fact that the contract with on the fact that the contract with the supplier the supplier the supplier had expired. Anything had expired. Anything had expired. Anything with legal or with legal or with legal or financial implications financial implications financial implications goes through a goes through a goes through a third-level review before third-level review before third-level review before being sent, and here's how

being sent, and here's how being sent, and here's how it went for it went for it went for this letter. First, this letter. First, this letter. First, self-audit. The agent self-audit. The agent self-audit. The agent checked the statements in checked the statements in checked the statements in the letter with the contract and the letter with the contract and the letter with the contract and invoices, including invoices, including invoices, including dates, and confirmed dates, and confirmed dates, and confirmed their accuracy. The their accuracy. The first stage has been completed. Second, independent Second, independent reviewers, each of reviewers, each of reviewers, each of whom had reason whom had reason whom had reason to find an error: our to find an error: our to find an error: our lawyer, a lawyer, a lawyer, a supplier representative, and an supplier representative, and an supplier representative, and an auditor who doesn't auditor who doesn't auditor who doesn't trust any of the trust any of the trust any of the numbers. They numbers. They numbers. They only received a letter, a contract, only received a letter, a contract, only received a letter, a contract, and invoices, and nothing and invoices, and nothing and invoices, and nothing more. The auditor more. The auditor more. The auditor counted the counted the counted the financial claim from financial claim from financial claim from the accounts, and it was the accounts, and it was the accounts, and it was accurate to the cent. But accurate to the cent. But accurate to the cent. But both lawyers both lawyers both lawyers independently independently independently found something found something found something hidden deep in the hidden deep in the contract's fine print. Clause on contract's fine print. Clause on automatic automatic automatic extension for three extension for three extension for three years. The contract years. The contract was automatically extended less than was automatically extended less than three weeks ago. The three weeks ago. The three weeks ago. The letter's demands letter's demands letter's demands would have forced us to terminate would have forced us to terminate would have forced us to terminate the contract that was the contract that was the contract that was in effect and owe in effect and owe in effect and owe the supplier thousands of the supplier thousands of the supplier thousands of dollars for dollars for dollars for early exit. The early exit. The two legal agents two legal agents also discovered that also discovered that also discovered that during the editing process, the during the editing process, the during the editing process, the original agent had original agent had original agent had even deleted a line even deleted a line even deleted a line where the extension was where the extension was where the extension was correctly identified. The correctly identified. The original agent's own verification original agent's own verification confirmed his own confirmed his own confirmed his own error. This is exactly what error. This is exactly what error. This is exactly what this verification method is this verification method is this verification method is designed to detect.

designed to detect. designed to detect. Our Our Our lawyer's verdict, however, was not to lawyer's verdict, however, was not to lawyer's verdict, however, was not to send the letter. This send the letter. This send the letter. This three-step three-step three-step independent verification process independent verification process independent verification process took 15 minutes and took 15 minutes and took 15 minutes and cost 13 million cost 13 million cost 13 million tokens. So, the letter tokens. So, the letter tokens. So, the letter was rewritten and was rewritten and was rewritten and passed on to the third passed on to the third passed on to the third stage of the stage of the stage of the third-level verification. third-level verification. third-level verification. Brand new Brand new Brand new reviewers who had reviewers who had reviewers who had never seen it before never seen it before reviewed the final reviewed the final reviewed the final version. The erroneous version. The erroneous version. The erroneous version was never version was never version was never sent. So, that's sent. So, that's sent. So, that's three three three audit histories and three levels of audit histories and three levels of audit histories and three levels of audit. An agent audit. An agent audit. An agent who couldn't create a who couldn't create a who couldn't create a customized version of customized version of customized version of the app reported the app reported the app reported this, instead of this, instead of this, instead of falsely falsely falsely claiming that everything was claiming that everything was claiming that everything was ready. Two groups of ready. Two groups of ready. Two groups of two agents two agents two agents extracted data and extracted data and extracted data and reduced annual business reduced annual business transactions to a transactions to a transactions to a penny. And third-party penny. And third-party penny. And third-party reviewers, who did not reviewers, who did not reviewers, who did not have the context of the have the context of the have the context of the original agent, original agent, original agent, edited the letter, edited the letter, edited the letter, which the review of the which the review of the arranging agent himself found to arranging agent himself found to be confirmed. I be confirmed. I be confirmed. I packaged these three packaged these three packaged these three levels of verification as levels of verification as levels of verification as agent skills and agent skills and agent skills and released them as released them as released them as open source open source open source software. The description software. The description software. The description can be found on the website can be found on the website can be found on the website gallon.me. Provide your gallon.me. Provide your gallon.me. Provide your agent with a link to agent with a link to agent with a link to this description and they this description and they this description and they will know how will know how will know how to verify their own to verify their own to verify their own work using this method work using this method so you can so you can so you can trust them. To trust them. To trust them. To learn more about the learn more about the learn more about the best ways to best ways to best ways to get real get real get real work done with work done with work done with AI agents, AI agents, AI agents, subscribe to my subscribe to my subscribe to my newsletter and newsletter and newsletter and join me join me join me here. So, I believe here. So, I believe here. So, I believe everything everything everything my agents tell me. Not my agents tell me. Not my agents tell me. Not because they are because they are because they are right. They are constantly right. They are constantly right. They are constantly wrong. I trust wrong. I trust wrong. I trust them because everything them because everything them because everything I rely on has been I rely on has been I rely on has been thoroughly thoroughly thoroughly vetted and vetted and vetted and confirmed beforehand. And, confirmed beforehand. And, confirmed beforehand. And, most importantly, not by me

most importantly, not by me .