How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

Thank you for joining Thank you for joining this session. This is a this session. This is a this session. This is a bit of an emotional bit of an emotional bit of an emotional moment for me. My moment for me. My moment for me. My name is Nathaniel. And I am a name is Nathaniel. And I am a non-evaluator. It's been a while non-evaluator. It's been a while non-evaluator. It's been a while since since since I I I last released an last released an last released an agent without any agent without any agent without any evaluation. And today I evaluation. And today I evaluation. And today I want to tell want to tell want to tell the story of how I the story of how I the story of how I broke this broke this broke this addiction to releasing addiction to releasing addiction to releasing agents without agents without agents without evaluation. So, I evaluation. So, I evaluation. So, I work for Air Canada. So, work for Air Canada. So, work for Air Canada. So, compared to the compared to the compared to the previous previous previous report, this is more of a report, this is more of a report, this is more of a focus on a specific focus on a specific focus on a specific business. I joined business. I joined business. I joined Air Canada in June of Air Canada in June of Air Canada in June of this year. And Air Canada this year. And Air Canada is a huge is a huge is a huge company. We work company. We work company. We work in operational, in operational, in operational, corporate, corporate, corporate, commercial commercial commercial and loyalty areas. and loyalty areas. and loyalty areas. Each of these Each of these Each of these segments has its segments has its segments has its own challenges. My own challenges. My own challenges. My team is responsible team is responsible team is responsible for implementing AI for implementing AI solutions for all of them. And solutions for all of them. And solutions for all of them. And the problem is that the problem is that the problem is that when we take on a when we take on a when we take on a task and create an task and create an task and create an agent for a very agent for a very agent for a very specific task, specific task, specific task, we don't have public we don't have public we don't have public benchmarks to benchmarks to benchmarks to rely on. And this is the rely on. And this is the rely on. And this is the main problem main problem main problem that I am trying to that I am trying to that I am trying to highlight in this highlight in this highlight in this presentation. More presentation. More presentation. More importantly, it's not importantly, it's not importantly, it's not just a matter of lack of a just a matter of lack of a just a matter of lack of a benchmark. The thing is benchmark. The thing is benchmark. The thing is , we are data , we are data scientists or AI scientists or AI engineers who can engineers who can engineers who can build agents, build agents, build agents, but we are not but we are not subject matter experts. And subject matter experts. And these experts these experts these experts are located in different are located in different are located in different business units. And business units. And business units. And they are very busy with they are very busy with they are very busy with their own their own their own tasks. And we tasks. And we tasks. And we have have have limited access to them. So limited access to them. So limited access to them. So both of these factors both of these factors both of these factors together create

together create together create a situation in which it is a situation in which it is a situation in which it is very difficult for us to very difficult for us to very difficult for us to construct any kind of construct any kind of construct any kind of assessment or any kind of assessment or any kind of assessment or any kind of benchmark for our benchmark for our benchmark for our tasks. So, before tasks. So, before tasks. So, before joining Air Canada, I joining Air Canada, I joining Air Canada, I worked for another worked for another worked for another company. This was a company. This was a company. This was a company where company where company where the task was to the task was to the task was to create a Q&A assistant create a Q&A assistant create a Q&A assistant for for for telecommunications engineers. I don't telecommunications engineers. I don't telecommunications engineers. I don't know if you are familiar with know if you are familiar with know if you are familiar with this field. this field. this field. Telecommunications is a Telecommunications is a Telecommunications is a huge industry. There are huge industry. There are huge industry. There are about 10,000 about 10,000 about 10,000 documents there. And in documents there. And in documents there. And in addition, there are many addition, there are many addition, there are many documents from documents from documents from vendors. Logic is needed to vendors. Logic is needed to vendors. Logic is needed to understand understand which document to which document to which document to find to find to find to answer the answer the answer the question. So this is a question. So this is a question. So this is a somewhat difficult somewhat difficult somewhat difficult task. And then, in task. And then, in task. And then, in 2024, everyone wanted to 2024, everyone wanted to 2024, everyone wanted to create RAG systems create RAG systems . So we started . So we started . So we started doing it. Yes, it doing it. Yes, it doing it. Yes, it sounds very easy on LinkedIn. sounds very easy on LinkedIn. sounds very easy on LinkedIn. But that's not true. So, But that's not true. So, uh... The thing is, when uh... The thing is, when uh... The thing is, when we took on this we took on this we took on this project, we had the same project, we had the same project, we had the same problem as Air Canada problem as Air Canada : we don't have access : we don't have access : we don't have access to experts. We to experts. We to experts. We had limited access had limited access had limited access to experts. And we to experts. And we to experts. And we ourselves could not evaluate ourselves could not evaluate ourselves could not evaluate the answers we received.

the answers we received. the answers we received. It is this problem that It is this problem that It is this problem that I want to I want to I want to talk about today. Oddly enough talk about today. Oddly enough talk about today. Oddly enough , we competed with , we competed with , we competed with other teams. Two other teams. Two other teams. Two external, one external, one external, one internal. It wasn't internal. It wasn't internal. It wasn't exactly parallel, but exactly parallel, but exactly parallel, but rather in segments. And rather in segments. And rather in segments. And our solution our solution our solution actually worked actually worked . By "it worked" I . By "it worked" I . By "it worked" I mean actually mean actually mean actually getting the result getting the result . People liked . People liked . People liked the solution, they actually the solution, they actually the solution, they actually used it. used it. used it. But, to be honest, I But, to be honest, I But, to be honest, I don't know why exactly we don't know why exactly we don't know why exactly we managed to do it managed to do it managed to do it better than the other team better than the other team . So, today I'm . So, today I'm . So, today I'm just going to show you what just going to show you what just going to show you what I or my team I or my team I or my team did to make it did to make it did to make it work. Everyone had the work. Everyone had the work. Everyone had the same documents, the same same documents, the same task, everything was the task, everything was the same, but same, but same, but somehow our solution somehow our solution somehow our solution worked. So, for worked. So, for worked. So, for those familiar with RAG, the those familiar with RAG, the those familiar with RAG, the principle is very simple: principle is very simple: principle is very simple: ask the ask the ask the system a question. The system system a question. The system system a question. The system found the relevant found the relevant found the relevant documents. Based on documents. Based on documents. Based on this relevant this relevant this relevant document, document, a response is formulated. This has been around for a response is formulated. This has been around for quite some time. And quite some time. And quite some time. And for those who have actually for those who have actually for those who have actually tried, if you tried, if you tried, if you find the right find the right find the right page or page or page or documents, 90% of the documents, 90% of the time you time you time you get the right get the right get the right answer. So answer. So answer. So the challenge is to find the challenge is to find the challenge is to find the right documents, the right documents, the right documents, at least in this at least in this at least in this case. We created a case. We created a case. We created a very complex very complex very complex pipeline that pipeline that pipeline that became more complex over time, because in became more complex over time, because in became more complex over time, because in the publications of that the publications of that the publications of that time everyone time everyone time everyone offered great offered great offered great methods for improving the methods for improving the methods for improving the solution. We solution. We solution. We tested them. This tested them. This tested them. This didn't actually

didn't actually didn't actually improve the solution. improve the solution. improve the solution. So on paper So on paper So on paper it worked for their publications it worked for their publications it worked for their publications , but it did , but it did , but it did n't really help us. And, n't really help us. And, n't really help us. And, more importantly, as we more importantly, as we more importantly, as we added complexity, added complexity, added complexity, we noticed that we were we noticed that we were we noticed that we were breaking something else. breaking something else. breaking something else. So, what worked So, what worked So, what worked before now before now before now stopped stopped stopped working. And this is working. And this is working. And this is actually a real actually a real actually a real problem. We can problem. We can problem. We can add more ideas add more ideas add more ideas to solve different to solve different to solve different problems, but we problems, but we problems, but we keep keep keep breaking something all the time. So, breaking something all the time. So, breaking something all the time. So, complexity doesn't always complexity doesn't always complexity doesn't always help. And the other help. And the other help. And the other problem is this. We problem is this. We problem is this. We can change can change can change the agent in a few the agent in a few the agent in a few minutes, add a certain minutes, add a certain minutes, add a certain level of level of level of query writing, or any query writing, or any query writing, or any stage of reasoning. But stage of reasoning. But stage of reasoning. But to evaluate it, we to evaluate it, we to evaluate it, we had these had these had these crazy meetings crazy meetings crazy meetings with experts once a with experts once a with experts once a week. We set up a week. We set up a week. We set up a huge screen with huge screen with huge screen with five different five different chatbots, asked chatbots, asked each of them the same question, and each of them the same question, and asked an expert to asked an expert to asked an expert to rate them. I rate them. I rate them. I prefer this, I prefer this, I prefer this, I prefer that. prefer that. prefer that. And, strangely enough, they didn't And, strangely enough, they didn't And, strangely enough, they didn't always agree always agree . It's very difficult when you . It's very difficult when you . It's very difficult when you gather experts in gather experts in gather experts in one room and you one room and you one room and you get three different get three different get three different opinions. So, that was the opinions. So, that was the opinions. So, that was the problem we problem we problem we faced. Our faced. Our faced. Our budget allowed us budget allowed us budget allowed us to test only one to test only one to test only one solution per week, and solution per week, and solution per week, and that wasn't that wasn't that wasn't enough to enough to enough to move quickly. So move quickly. So move quickly. So before before before moving on, the process moving on, the process moving on, the process was like this. We ask was like this. We ask was like this. We ask questions, the agent questions, the agent questions, the agent answers, we answers, we answers, we ask an expert ask an expert ask an expert to check. Sometimes to check. Sometimes to check. Sometimes they don't know they don't know they don't know the answer, they the answer, they the answer, they have to search, and have to search, and have to search, and that takes a lot of time that takes a lot of time . And we had a budget for

. And we had a budget for . And we had a budget for one test per one test per one test per week. So, there is nothing week. So, there is nothing week. So, there is nothing new here. new here. new here. This well-known This well-known This well-known approach is that approach is that we turn we turn we turn the problem upside down. So we the problem upside down. So we the problem upside down. So we start with the start with the start with the page. We ask page. We ask page. We ask LLM to write a question LLM to write a question LLM to write a question about this page, which is a about this page, which is a about this page, which is a fairly simple fairly simple fairly simple task. We ask the task. We ask the task. We ask the agent to find agent to find agent to find the answer and the answer and the answer and compare what the compare what the compare what the agent found with agent found with agent found with the page that was the page that was the page that was used to used to used to create the question. create the question. create the question. We can We can We can use code use code use code to evaluate this. to evaluate this. to evaluate this. No more thoughts, No more thoughts, No more thoughts, no more assumptions. We no more assumptions. We no more assumptions. We can can can automate this automate this automate this task. And this task. And this task. And this allows us to allows us to allows us to move much move much move much faster. For those faster. For those faster. For those working with RAG, there are working with RAG, there are working with RAG, there are many options for many options for many options for implementing this. I did implementing this. I did implementing this. I did n't invent this. I n't invent this. I n't invent this. I just think it's very just think it's very , very smart. And this , very smart. And this , very smart. And this was etched in my was etched in my was etched in my memory. Why memory. Why memory. Why can't this principle be can't this principle be can't this principle be applied to other applied to other applied to other tasks? The result of tasks? The result of tasks? The result of this was that this was that this was that we went from we went from we went from one assessment per one assessment per one assessment per week to dozens week to dozens week to dozens per day. And we began per day. And we began per day. And we began to see how the accuracy of to see how the accuracy of expert assessments is constantly expert assessments is constantly increasing. In fact, increasing. In fact, increasing. In fact, eliminating unnecessary eliminating unnecessary eliminating unnecessary complexity helps complexity helps . We didn't need much of it . We didn't need much of it . We didn't need much of it .

. . Focusing on Focusing on Focusing on the right the right the right embedding model, embedding model, embedding model, the right chunking, the right chunking, the right chunking, and the right alpha and the right alpha coefficient in BM25 is coefficient in BM25 is coefficient in BM25 is what needs to be done. And what needs to be done. And what needs to be done. And we saw the we saw the we saw the accuracy increase accuracy increase accuracy increase over the weeks. I don't over the weeks. I don't over the weeks. I don't know why we were know why we were know why we were successful, but I'm successful, but I'm successful, but I'm sure the other sure the other sure the other team didn't have team didn't have team didn't have any test any test any test suite. They suite. They suite. They used this used this used this long cycle with long cycle with long cycle with experts, perhaps experts, perhaps experts, perhaps even more even more even more qualified than qualified than qualified than those those those we had access to. But we we had access to. But we we had access to. But we had this test had this test had this test set. My lesson from set. My lesson from set. My lesson from this 2024 experience is this 2024 experience is this 2024 experience is this: if you this: if you this: if you can't evaluate can't evaluate can't evaluate an agent, you can't an agent, you can't an agent, you can't make them better. make them better. It has stayed with me It has stayed with me to this day, to this day, to this day, 2 years have passed. And now 2 years have passed. And now 2 years have passed. And now I want to go back to I want to go back to I want to go back to Air Canada. Where do we have another Air Canada. Where do we have another Air Canada. Where do we have another project? In 2026, project? In 2026, project? In 2026, the project consists of the project consists of the project consists of analyzing contracts with analyzing contracts with analyzing contracts with suppliers. You suppliers. You suppliers. You can imagine that Air can imagine that Air can imagine that Air Canada has many Canada has many Canada has many suppliers. We suppliers. We suppliers. We get a lot of get a lot of get a lot of these contracts. There are these contracts. There are these contracts. There are experts who experts who experts who review them one review them one review them one by one. Checking by one. Checking by one. Checking each of these points.

each of these points. each of these points. They need to They need to They need to analyze them and analyze them and analyze them and provide provide provide feedback. For example, feedback. For example, feedback. For example, this item does not this item does not this item does not meet the requirements. meet the requirements. meet the requirements. This paragraph needs to be This paragraph needs to be This paragraph needs to be rewritten. This rewritten. This rewritten. This requires a requires a requires a huge amount of huge amount of huge amount of work. This is a very work. This is a very work. This is a very time-consuming and very time-consuming and very time-consuming and very costly task because costly task because costly task because if we need the if we need the if we need the supplier to supplier to supplier to deliver deliver deliver immediately and they do immediately and they do immediately and they do n't, it n't, it n't, it causes a lot of causes a lot of causes a lot of delays in the delays in the delays in the supply chain. And if we supply chain. And if we supply chain. And if we miss a miss a miss a point, it is also very point, it is also very point, it is also very costly, costly, costly, because in the end it can because in the end it can because in the end it can turn against us. turn against us. turn against us. So this is not a simple So this is not a simple So this is not a simple problem. Um, and they problem. Um, and they problem. Um, and they turned to me, the turned to me, the data analytics team, to data analytics team, to build an agent that build an agent that build an agent that could do that for could do that for could do that for them. So can we them. So can we them. So can we automate this automate this automate this task? Of course, task? Of course, task? Of course, there are many there are many there are many documents, many documents, many documents, many rules, and many things to rules, and many things to consider consider consider when you do this. when you do this. when you do this. Experts are very Experts are very Experts are very busy. You can busy. You can busy. You can imagine the cost of an imagine the cost of an imagine the cost of an hour of lawyer's work. hour of lawyer's work. hour of lawyer's work. And there are no standards for this And there are no standards for this And there are no standards for this . We . We . We can't just can't just can't just go online and go online and go online and find documents or an find documents or an find documents or an existing standard existing standard existing standard that we can that we can that we can apply. So apply. So apply. So let's go back to the same let's go back to the same let's go back to the same problem we had in problem we had in problem we had in 2024. The cycles were very 2024. The cycles were very 2024. The cycles were very slow, and we slow, and we slow, and we knew we wanted to knew we wanted to knew we wanted to get to a state get to a state get to a state where we could act where we could act where we could act quickly, try, quickly, try, quickly, try, try, and try, and try, and try again. And I knew try again. And I knew try again. And I knew that if we studied this that if we studied this that if we studied this situation, we could situation, we could situation, we could have a better have a better have a better trajectory in terms of trajectory in terms of trajectory in terms of accuracy. And that accuracy. And that accuracy. And that was a challenge for this was a challenge for this was a challenge for this project? So project? So project? So the problem of generating a

the problem of generating a document-based response is well document-based response is well known and defined known and defined known and defined in RAG, but for this in RAG, but for this in RAG, but for this particular project particular project particular project , there's no one in the world , there's no one in the world , there's no one in the world who says, "Hey, who says, "Hey, who says, "Hey, you should you should you should use this use this use this technique." You should technique." You should technique." You should use this use this use this technique. So we're technique. So we're technique. So we're trying to trying to trying to figure out how to figure out how to figure out how to apply the apply the apply the same mechanism that same mechanism that same mechanism that I learned at RAG to I learned at RAG to I learned at RAG to this particular this particular this particular project. And at Air Canada we have project. And at Air Canada we have project. And at Air Canada we have many projects. many projects. many projects. So we So we So we use the use the use the same concept: same concept: same concept: first get first get first get the answer, and then the answer, and then the answer, and then try to train try to train try to train the agent to find it the agent to find it the agent to find it again. So, again. So, again. So, getting into getting into getting into the details, it's a little the details, it's a little the details, it's a little oversimplified, but there are oversimplified, but there are oversimplified, but there are actually actually actually four steps for an agent. We four steps for an agent. We four steps for an agent. We need to find a need to find a need to find a clause in the contract. clause in the contract. clause in the contract. We need to find the We need to find the We need to find the relevant documents relevant documents to substantiate to substantiate to substantiate the amendment. We the amendment. We the amendment. We need to write this need to write this need to write this amendment. And then, amendment. And then, amendment. And then, at the end, there is a at the end, there is a at the end, there is a classification system that classification system that classification system that suggests whether or not to suggests whether or not to suggests whether or not to keep this amendment. keep this amendment. keep this amendment. So, at the first stage, So, at the first stage, So, at the first stage, for example, for example, all all sections of the contract are classified sections of the contract are classified sections of the contract are classified into certain areas. And into certain areas. And into certain areas. And then it turns out whether there then it turns out whether there then it turns out whether there is a flaw in this section or whether is a flaw in this section or whether changes need to be made. So changes need to be made. So we had this RAG trick.

we had this RAG trick. we had this RAG trick. So, we know how So, we know how So, we know how to find documents. to find documents. to find documents. So, we've So, we've So, we've covered that. But for the rest of the covered that. But for the rest of the covered that. But for the rest of the agent, agent, agent, we had nothing. we had nothing. So I'm going to present So I'm going to present five different methods five different methods five different methods that we're that we're that we're using or using or using or trying to trying to trying to use in use in use in this project. This is this project. This is this project. This is an attempt to solve this an attempt to solve this an attempt to solve this specific problem. specific problem. My goal is not My goal is not to show off. My goal is to show off. My goal is to inspire you on to inspire you on to inspire you on how to apply these how to apply these how to apply these methods to your methods to your methods to your own tasks. So own tasks. So , the first way is , the first way is to use to use to use past solutions and past solutions and past solutions and answers. We can answers. We can answers. We can get the previous get the previous get the previous contract and compare contract and compare contract and compare it with the amended one. it with the amended one. The second way is to choose what you are to choose what you are to choose what you are looking for and write or looking for and write or looking for and write or generate a contract generate a contract generate a contract around it. The third around it. The third around it. The third way is to take a way is to take a way is to take a good signed good signed good signed contract and violate contract and violate contract and violate its terms. The fourth one its terms. The fourth one its terms. The fourth one is about things that is about things that is about things that shouldn't shouldn't shouldn't change, no matter what you change, no matter what you change, no matter what you do, you do, you do, you shouldn't get shouldn't get shouldn't get extra marks. And extra marks. And extra marks. And the fifth is actually the the fifth is actually the the fifth is actually the most powerful. This is most powerful. This is most powerful. This is very difficult. This is very difficult. This is very difficult. This is converting one converting one converting one example into many example into many example into many using templates.

using templates. So we immediately So we immediately switched to this switched to this switched to this approach. We asked approach. We asked approach. We asked the expert to provide the the expert to provide the the expert to provide the original original original contract, the one that was contract, the one that was contract, the one that was signed. We take signed. We take signed. We take it, do it, do it, do a comparison like in code, a comparison like in code, a comparison like in code, see what see what see what has changed, and the agent has changed, and the agent has changed, and the agent should provide the same should provide the same should provide the same marks. That would be marks. That would be marks. That would be ideal. However, for ideal. However, for ideal. However, for many reasons, our many reasons, our many reasons, our team does not retain team does not retain team does not retain all contracts. They all contracts. They all contracts. They don't keep don't keep don't keep the original, only the the original, only the the original, only the latest version, they latest version, they latest version, they don't have the entire history of don't have the entire history of don't have the entire history of contract changes. contract changes. contract changes. Sometimes it goes into the Sometimes it goes into the Sometimes it goes into the negotiation stage, negotiation stage, negotiation stage, there are many nuances. there are many nuances. there are many nuances. Therefore, we were unable to Therefore, we were unable to Therefore, we were unable to obtain a single pair of obtain a single pair of obtain a single pair of such contracts. such contracts. such contracts. So, this method doesn't So, this method doesn't So, this method doesn't work. Okay, the second work. Okay, the second work. Okay, the second way. We adopted a way. We adopted a way. We adopted a rule. And that rule. And that rule. And that changes the subject a bit. changes the subject a bit. changes the subject a bit. Instead of creating Instead of creating Instead of creating tags or an agent tags or an agent tags or an agent that issues them, we that issues them, we that issues them, we start with rules. start with rules. The rule, for example, is that we have a limit of over we have a limit of over we have a limit of over $1 million.

$1 million. $1 million. Every contract Every contract Every contract involving Arcadia must involving Arcadia must involving Arcadia must adhere to this adhere to this adhere to this rule, or many rule, or many rule, or many others in certain areas others in certain areas . You know the rule, . You know the rule, . You know the rule, you create a point without you create a point without you create a point without violating that violating that violating that rule, you generate a rule, you generate a rule, you generate a contract around contract around contract around it, and you have to it, and you have to it, and you have to find that rule— find that rule— the mark that the mark that the mark that violates it. It's simple. It violates it. It's simple. It violates it. It's simple. It doesn't require much doesn't require much doesn't require much work. Many things work. Many things work. Many things can be generated can be generated can be generated automatically. automatically. automatically. The problem is that you have to The problem is that you have to The problem is that you have to know the know the know the rules. So, rules. So, rules. So, fortunately, we were able to fortunately, we were able to fortunately, we were able to get some of get some of get some of this either from an this either from an this either from an expert or from the expert or from the expert or from the documentation that documentation that documentation that they had. But we they had. But we they had. But we cannot guarantee cannot guarantee cannot guarantee complete coverage of complete coverage of complete coverage of everything. So we everything. So we everything. So we also made up our also made up our also made up our own rules. And own rules. And own rules. And so on. So, so on. So, so on. So, in the end, we created in the end, we created in the end, we created 275 tests that we 275 tests that we 275 tests that we know we can know we can know we can run and know run and know run and know the answer. So, the answer. So, the answer. So, similar to the "rug similar to the "rug " method, we inverted " method, we inverted " method, we inverted the problem, threw it up, the problem, threw it up, the problem, threw it up, and tried and tried and tried to find it back. Another one that is to find it back. Another one that is to find it back. Another one that is actually very actually very actually very complicated. We took an complicated. We took an complicated. We took an example of a signed example of a signed example of a signed contract, removed a contract, removed a contract, removed a clause, and the agent has to clause, and the agent has to clause, and the agent has to say, "Hey, say, "Hey, say, "Hey, something's missing here."

something's missing here." This is a very difficult This is a very difficult task because task because task because some some some providers have different providers have different providers have different templates and different types of templates and different types of templates and different types of content. To know what content. To know what content. To know what should be in which should be in which should be in which contract, you need to contract, you need to contract, you need to think a little. And we think a little. And we think a little. And we mix them with mix them with mix them with existing contracts existing contracts existing contracts that shouldn't that shouldn't that shouldn't cause any cause any cause any markings. So, this is the markings. So, this is the markings. So, this is the most powerful approach most powerful approach we have we have we have today. today. today. Using this Using this Using this approach, we can approach, we can approach, we can create many create many create many contracts, and ultimately contracts, and ultimately contracts, and ultimately we find that our we find that our we find that our agents miss agents miss agents miss most of the removed most of the removed most of the removed clauses. So, this is a very, very clauses. So, this is a very, very strong strong strong benchmark. OK. The benchmark. OK. The benchmark. OK. The fourth technique, um fourth technique, um ...You're not changing, you're ...You're not changing, you're ...You're not changing, you're changing something that changing something that changing something that shouldn't affect the shouldn't affect the shouldn't affect the agent. This shouldn't agent. This shouldn't agent. This shouldn't change anything in the change anything in the change anything in the end. end. end. For example, For example, For example, supplier name, supplier name, supplier name, supplier address supplier address , something that shouldn't , something that shouldn't , something that shouldn't affect the affect the affect the number of marks. number of marks. You can You can rearrange rearrange rearrange the items, so a certain amount the items, so a certain amount the items, so a certain amount of randomness of randomness of randomness should also lead to the should also lead to the should also lead to the same same same results. And you results. And you results. And you can run the can run the can run the same contract over and over again same contract over and over again same contract over and over again . We . We . We check all of this to check all of this to check all of this to make sure the make sure the make sure the result is result is result is consistent across all of consistent across all of consistent across all of them. In this example, them. In this example, them. In this example, everything can also be everything can also be everything can also be tested tested tested using code. You using code. You using code. You don't need to don't need to don't need to use LLM as a use LLM as a use LLM as a judge. The previous judge. The previous judge. The previous speaker showed that this is a speaker showed that this is a speaker showed that this is a very difficult very difficult very difficult task. Therefore, we task. Therefore, we stay as far away as possible from stay as far away as possible from using LLM as a using LLM as a using LLM as a judge. The fifth, um, is

judge. The fifth, um, is judge. The fifth, um, is the concept of a template. the concept of a template. Our experts have Our experts have different types of templates different types of templates different types of templates with spaces. I don't with spaces. I don't with spaces. I don't know if know if know if you've ever had to you've ever had to you've ever had to write a contract. These write a contract. These write a contract. These gaps should be gaps should be gaps should be filled in by an expert. filled in by an expert. And instead of And instead of trying to trying to trying to fill them in, we fill them in, we fill them in, we generate an array for generate an array for generate an array for each of those each of those each of those gaps. This allows gaps. This allows gaps. This allows us to generate a us to generate a us to generate a multitude of contracts. multitude of contracts. multitude of contracts. Some datasets Some datasets Some datasets violate the rules, violate the rules, violate the rules, some don't. We some don't. We some don't. We know this. So, we know this. So, we know this. So, we know what exactly should be know what exactly should be know what exactly should be marked as a marked as a marked as a violation and what should not. violation and what should not. violation and what should not. Again, this is a Again, this is a Again, this is a code review. We code review. We code review. We don't need to involve don't need to involve don't need to involve any LLM. The good news any LLM. The good news any LLM. The good news in this case is in this case is in this case is that that that our experts are already our experts are already our experts are already using using using templates, so we templates, so we templates, so we can can can reuse them reuse them reuse them to create to create to create contracts. The cycle in contracts. The cycle in contracts. The cycle in this case was this case was this case was a little slower, a little slower, a little slower, i.e. 4 weeks to i.e. 4 weeks to i.e. 4 weeks to get feedback, but get feedback, but get feedback, but now we can now we can now we can run a script and run a script and run a script and get an estimate in 5 get an estimate in 5 get an estimate in 5 minutes. I just want to minutes. I just want to minutes. I just want to make one point, make one point, make one point, we went through all we went through all we went through all those those those assessment categories listed assessment categories listed assessment categories listed below. Exact below. Exact below. Exact match or code match or code match or code works very fast.

works very fast. works very fast. Once we Once we Once we move on to move on to move on to using LLM as a using LLM as a using LLM as a judge, it can be judge, it can be judge, it can be very effective and very effective and very effective and useful. This is very useful. This is very useful. This is very difficult. We get difficult. We get difficult. We get so much noise from the LLM so much noise from the LLM judge compared to the judge compared to the judge compared to the noise from the agent noise from the agent noise from the agent we create. we create. we create. Solving this Solving this Solving this problem is actually problem is actually problem is actually almost as difficult almost as difficult almost as difficult as solving the as solving the as solving the main problem itself. Therefore, main problem itself. Therefore, main problem itself. Therefore, finding the right finding the right finding the right prompt or the prompt or the prompt or the right right right toolkit for an LLM toolkit for an LLM judge is a very difficult judge is a very difficult judge is a very difficult task. There are certain task. There are certain task. There are certain supplement methods supplement methods supplement methods that can be that can be that can be applied. applied. applied. task. You can task. You can task. You can do do do fact checking. There are a lot of things you can do fact checking. There are a lot of things you can do fact checking. There are a lot of things you can do , but , but , but it actually requires it actually requires it actually requires so much work that so much work that so much work that we couldn't find a we couldn't find a we couldn't find a suitable option suitable option suitable option for this project. for this project. There are many other There are many other techniques. I won't techniques. I won't techniques. I won't list them list them list them all. For example, all. For example, all. For example, trajectory. We have trajectory. We have trajectory. We have another project that another project that another project that requires the agent requires the agent requires the agent to interact with certain to interact with certain to interact with certain endpoints. endpoints. endpoints. So, regarding the trajectory So, regarding the trajectory , you can you know , you can you know , you can you know what the endpoint should be what the endpoint should be what the endpoint should be in a certain in a certain in a certain order. You create a order. You create a order. You create a story about it. You story about it. You story about it. You ask the model ask the model ask the model to fill in the data, to fill in the data, to fill in the data, use the agent, and use the agent, and use the agent, and view view view the trace to the trace to the trace to see if see if see if the trajectory is the trajectory is the trajectory is correct. Text-to-SQL, you correct. Text-to-SQL, you may already know may already know may already know about it. It doesn't about it. It doesn't about it. It doesn't matter what matter what matter what agent you have. If you know agent you have. If you know agent you have. If you know how to choose an answer, how to choose an answer, how to choose an answer, if you know what you're if you know what you're if you know what you're looking for, I've shown you looking for, I've shown you looking for, I've shown you some examples.

some examples. some examples. Generate input Generate input Generate input data and then data and then data and then try to find it. try to find it. try to find it. This does not apply to This does not apply to This does not apply to everything. And sometimes we everything. And sometimes we everything. And sometimes we had to had to had to rework our rework our rework our solution to solution to solution to fit this fit this fit this paradigm, even paradigm, even paradigm, even if we deviated a bit if we deviated a bit if we deviated a bit from the from the from the original design. original design. original design. With the initial With the initial With the initial design, we were unable to design, we were unable to design, we were unable to conduct an evaluation. conduct an evaluation. conduct an evaluation. Because of this, we moved on to something we Because of this, we moved on to something we Because of this, we moved on to something we can can can actually evaluate. actually evaluate. It is worth noting that It is worth noting that this is not the end of the road. You this is not the end of the road. You this is not the end of the road. You generate a test. You generate a test. You generate a test. You get a better get a better get a better agent. You get agent. You get agent. You get real real real feedback. So, an expert feedback. So, an expert feedback. So, an expert can provide feedback, and can provide feedback, and can provide feedback, and we don't use we don't use we don't use that feedback to that feedback to that feedback to correct the agent. correct the agent. correct the agent. We use We use We use this feedback to this feedback to this feedback to create better create better create better tests. Best tips tests. Best tips . Best ideas for . Best ideas for . Best ideas for deleting items, deleting items, deleting items, etc. And one etc. And one etc. And one iteration at a time. Uh, iteration at a time. Uh, iteration at a time. Uh, you can do this you can do this you can do this with rise. There are with rise. There are with rise. There are many tools there. many tools there. many tools there. Thank you for the invitation. Thank you for the invitation. Thank you for the invitation. Um, and this is my Um, and this is my Um, and this is my final slide. final slide. final slide. Why the answer Why the answer Why the answer first? Um, I first? Um, I first? Um, I think I think I think I explained that. Um, when you explained that. Um, when you explained that. Um, when you generate these tests, generate these tests, generate these tests, it's not enough it's not enough it's not enough to know if the to know if the to know if the product is ready to be released. This is product is ready to be released. This is product is ready to be released. This is just a starting point.

just a starting point. just a starting point. This is a tool that This is a tool that This is a tool that helps the developer helps the developer helps the developer stay ahead of stay ahead of stay ahead of the curve. This is not a the curve. This is not a the curve. This is not a "fit or "fit or "fit or not" criterion for a decision. So I not" criterion for a decision. So I not" criterion for a decision. So I hope you hope you hope you enjoyed this enjoyed this enjoyed this presentation. I know presentation. I know presentation. I know that the 24-hour cycle is that the 24-hour cycle is the most complicated. If the most complicated. If the most complicated. If you want to stop you want to stop releasing an agent without releasing an agent without an evaluation, I would be happy an evaluation, I would be happy an evaluation, I would be happy to support you and to support you and to support you and get your feedback. get your feedback. get your feedback. Thank you very much.