How Epic's Lore VCS Handles Massive Binary Files | Ben Cooke & Ryn Slack (AWS) | GameDev Day 2026

So, next we have Ben and So, next we have Ben and Ren talking to Ren talking to Ren talking to us about Lore and us about Lore and us about Lore and version control from version control from version control from Epic. It is very interesting Epic. It is very interesting Epic. It is very interesting to learn more about to learn more about to learn more about this topic. Congratulations. this topic. Congratulations. this topic. Congratulations. Congratulations. Congratulations. Congratulations. How are you doing How are you doing How are you doing today? today? today? Hey, great. Perfectly. Hey, great. Perfectly. Hey, great. Perfectly. Yes, we are happy to have this Yes, we are happy to have this Yes, we are happy to have this conversation. conversation. conversation. We discovered a lot of new things, and We discovered a lot of new things, and We discovered a lot of new things, and it's really very it's really very it's really very interesting. A new interesting. A new interesting. A new version control system version control system version control system doesn't come along every day doesn't come along every day . . . That's right. That's right. That's right. That's right. Well, we Well, we can hear you well, and your slides are can hear you well, and your slides are can hear you well, and your slides are already on the screen. So, already on the screen. So, already on the screen. So, take us into the world of take us into the world of take us into the world of version control. version control. version control. Perfectly. Thank you, Michael. Perfectly. Thank you, Michael. Perfectly. Thank you, Michael. Hello everyone, thank you for Hello everyone, thank you for Hello everyone, thank you for spending the spending the spending the next 50 minutes with us. This next 50 minutes with us. This next 50 minutes with us. This conversation is about files conversation is about files conversation is about files for which most for which most version control systems version control systems were never created. were never created. Textures, meshes, audio, Textures, meshes, audio, levels—all that levels—all that levels—all that heavy binary heavy binary heavy binary content that content that content that makes up games. We will makes up games. We will makes up games. We will properly properly properly introduce ourselves when the introduce ourselves when the introduce ourselves when the first question comes up first question comes up first question comes up . But see . But see . But see that orange that orange that orange square? Remember square? Remember square? Remember it. I want to start with it. I want to start with it. I want to start with one artist and one artist and one artist and one file.

one file. Monday, 9:00 a.m., Monday, 9:00 a.m., the artist opens a the artist opens a the artist opens a two-gigabyte two-gigabyte two-gigabyte source atlas. This is a source atlas. This is a source atlas. This is a huge uncompressed huge uncompressed huge uncompressed image image image containing all the containing all the containing all the fragments. It fragments. It fragments. It combines many combines many combines many small textures small textures small textures together. The artist together. The artist together. The artist redraws one redraws one redraws one small square. small square. small square. Maybe 1% of the image, Maybe 1% of the image, Maybe 1% of the image, saves it and saves it and saves it and commits it. A normal commits it. A normal commits it. A normal routine shift. routine shift. routine shift. Now, run this Now, run this Now, run this same edit same edit same edit through three through three through three version control systems. Git (with version control systems. Git (with version control systems. Git (with or without LFS), a or without LFS), a or without LFS), a centralized centralized centralized server, and Lore: one server, and Lore: one server, and Lore: one file, one change, the file, one change, the file, one change, the same network, and three same network, and three same network, and three other sites that other sites that other sites that need it. So, need it. So, need it. So, keep an eye on keep an eye on keep an eye on the indicators. One of the indicators. One of the indicators. One of them was completed them was completed them was completed long before the long before the long before the others really others really others really started working. started working. started working. Interestingly, this has nothing Interestingly, this has nothing Interestingly, this has nothing to do with a faster to do with a faster to do with a faster server; Let me clarify: this server; Let me clarify: this server; Let me clarify: this slide is not slide is not slide is not a measurement, it is just a a measurement, it is just a a measurement, it is just a demonstration of the demonstration of the demonstration of the problem. We'll problem. We'll problem. We'll talk about talk about talk about the results in a moment, but for now, the results in a moment, but for now, the results in a moment, but for now, just just just keep your eyes on that keep your eyes on that keep your eyes on that orange square orange square .

. So, the main question is : what was actually : what was actually : what was actually transmitted? Before we transmitted? Before we transmitted? Before we answer that, answer that, answer that, let me let me let me introduce myself: I'm Ren introduce myself: I'm Ren introduce myself: I'm Ren Slack, an Slack, an Slack, an AWS Gaming Solutions Architect based in AWS Gaming Solutions Architect based in AWS Gaming Solutions Architect based in Arlington, Arlington, Arlington, Virginia. With me Virginia. With me Virginia. With me today is Ben Cook, a today is Ben Cook, a today is Ben Cook, a senior senior senior gaming solutions architect from gaming solutions architect from gaming solutions architect from Austin, Texas. Austin, Texas. Austin, Texas. Ben will explain how it Ben will explain how it Ben will explain how it works and what exactly we works and what exactly we works and what exactly we measured, and I'll measured, and I'll measured, and I'll explain how to explain how to explain how to run it and speak run it and speak run it and speak on behalf of the studio and on behalf of the studio and on behalf of the studio and anyone who has to anyone who has to anyone who has to wake up to a wake up to a wake up to a call at 3 a.m. call at 3 a.m. call at 3 a.m. Yes. We want Yes. We want you to take three things away from this report. you to take three things away from this report. First, a mental First, a mental First, a mental model of model of model of fragmented fragmented fragmented storage. Secondly, storage. Secondly, storage. Secondly, why does the architecture why does the architecture why does the architecture look like this? And look like this? And look like this? And third, a starting third, a starting third, a starting point for you to be point for you to be point for you to be able to launch this able to launch this able to launch this yourself, and how much yourself, and how much yourself, and how much it costs. And every time it costs. And every time it costs. And every time we give we give we give numbers, we numbers, we numbers, we will tell you will tell you will tell you where they came from. where they came from. where they came from. So, asking the So, asking the So, asking the obvious question: obvious question: obvious question: every studio I every studio I every studio I talk to already has a talk to already has a version control system. So why is it version control system. So why is it worth worth worth dedicating an entire dedicating an entire dedicating an entire report to this? Well, because the report to this? Well, because the report to this? Well, because the game repository game repository is not a is not a is not a source code repository. These are source code repository. These are source code repository. These are gigabytes of binary gigabytes of binary gigabytes of binary files that change files that change files that change daily and tools daily and tools daily and tools designed designed designed specifically for specifically for specifically for source code. Git source code. Git source code. Git was never was never was never designed for this kind of designed for this kind of designed for this kind of task in the first place. Here is an example of an task in the first place. Here is an example of an task in the first place. Here is an example of an Unreal-style project, Unreal-style project, Unreal-style project, where each block represents where each block represents where each block represents a size in bytes, and the a size in bytes, and the a size in bytes, and the largest block is largest block is largest block is the textures, i.e. the the textures, i.e. the the textures, i.e. the images on each images on each images on each surface in the game. Next

surface in the game. Next surface in the game. Next comes the source art: comes the source art: comes the source art: multi-layered multi-layered multi-layered image files and image files and image files and sculptures created by sculptures created by sculptures created by artists, then 3D artists, then 3D models (meshes), models (meshes), models (meshes), engine binaries, and audio. engine binaries, and audio. And this thin And this thin orange strip orange strip orange strip on the right—that's the on the right—that's the on the right—that's the source code, about source code, about source code, about 2%. So, the point is in 2%. So, the point is in 2%. So, the point is in the ratio. the ratio. the ratio. Almost all of this is binary Almost all of this is binary Almost all of this is binary data. Some individual data. Some individual data. Some individual files reach files reach files reach gigabytes. gigabytes. gigabytes. Artists Artists Artists re-save them re-save them re-save them all day. all day. all day. String comparison and String comparison and String comparison and code merging are great code merging are great code merging are great for that 2%, for that 2%, for that 2%, but the other 98% are but the other 98% are but the other 98% are opaque bytes. So opaque bytes. So opaque bytes. So you can't you can't you can't compare the texture compare the texture compare the texture line by line and you line by line and you line by line and you can't merge them. Therefore, can't merge them. Therefore, there are three ways there are three ways to control versions of to control versions of to control versions of this type. When I this type. When I this type. When I mention Git or Git LFS, I mention Git or Git LFS, I mention Git or Git LFS, I mean their mean their mean their default settings default settings default settings . In the . In the . In the first column, first column, first column, we'll talk about Git, we'll talk about Git, we'll talk about Git, specifically about LFS, which specifically about LFS, which specifically about LFS, which stands for Large stands for Large File Storage. This allows you File Storage. This allows you to remove large files to remove large files to remove large files from history, leaving from history, leaving from history, leaving only small only small only small pointers. But LFS pointers. But LFS pointers. But LFS cannot transfer cannot transfer cannot transfer part of the file. Changing part of the file. Changing part of the file. Changing one byte means one byte means one byte means loading a new loading a new loading a new complete object, which is complete object, which is complete object, which is completely downloaded completely downloaded completely downloaded at each checkout at each checkout at each checkout where it is needed. So where it is needed. So where it is needed. So Git LFS can send Git LFS can send Git LFS can send deltas, but after 512 deltas, but after 512 deltas, but after 512 megabytes it megabytes it megabytes it stops trying, and the stops trying, and the stops trying, and the standard clone standard clone standard clone contains the entire history.

contains the entire history. So, the second column is a centralized is a centralized is a centralized model. This is a model where there model. This is a model where there model. This is a model where there is one main is one main is one main server, and to server, and to server, and to scale, you scale, you scale, you add edge add edge add edge servers closer to servers closer to servers closer to your team. In this your team. In this your team. In this classic system, classic system, classic system, each site each site each site runs a full runs a full runs a full replica with a replica with a replica with a standard standard standard binary file type. binary file type. Each change is a new Each change is a new revision, and each revision, and each revision, and each replica downloads the replica downloads the replica downloads the entire file. entire file. Some centralized Some centralized servers may servers may servers may only send only send only send modified parts of modified parts of modified parts of uncompressed binary uncompressed binary uncompressed binary files, but they files, but they files, but they store each store each store each version in its entirety. This is a version in its entirety. This is a version in its entirety. This is a proven design. proven design. I'll add from an I'll add from an operator's perspective: replicas operator's perspective: replicas operator's perspective: replicas scale reads, scale reads, scale reads, and it works. But as the and it works. But as the and it works. But as the person managing them, person managing them, person managing them, you get three you get three you get three standing accounts. standing accounts. standing accounts. The first is the capacity The first is the capacity The first is the capacity at each at each at each site. It is site. It is site. It is planned in advance planned in advance , because each replica , because each replica , because each replica preserves the entire archive. preserves the entire archive. preserves the entire archive. The second is The second is The second is to set up a new to set up a new to set up a new edge node edge node edge node whose metadata whose metadata whose metadata needs to be restored from a needs to be restored from a needs to be restored from a checkpoint.

checkpoint. checkpoint. Most teams Most teams Most teams also copy the archive also copy the archive also copy the archive in advance, because the first in advance, because the first in advance, because the first sync over sync over sync over WAN is unheard of. WAN is unheard of. WAN is unheard of. The third is the morning The third is the morning The third is the morning rush hour. This is the moment when the rush hour. This is the moment when the rush hour. This is the moment when the entire studio entire studio entire studio synchronizes synchronizes synchronizes at the same time, and every at the same time, and every at the same time, and every line has to be on time. line has to be on time. line has to be on time. Perfectly. Yes. And then, Perfectly. Yes. And then, Perfectly. Yes. And then, oh. The third column is oh. The third column is oh. The third column is here. Lore is also here. Lore is also here. Lore is also centralized. But the centralized. But the storage unit itself is changing. storage unit itself is changing. Lore does not save entire Lore does not save entire Lore does not save entire files. It stores files. It stores files. It stores parts of files parts of files parts of files called called called fragments. When fragments. When fragments. When an artist makes an artist makes an artist makes a commit, Lore determines a commit, Lore determines a commit, Lore determines which pieces have changed which pieces have changed which pieces have changed and only those are pushed and only those are pushed and only those are pushed . Therefore, copies that . Therefore, copies that . Therefore, copies that already have the old already have the old already have the old version have version have version have unchanged fragments of unchanged fragments of unchanged fragments of the file, and nothing the file, and nothing the file, and nothing else else else needs to be transferred there. That's the needs to be transferred there. That's the whole whole difference. Two models difference. Two models difference. Two models version version version entire files, while one entire files, while one only versions what only versions what only versions what has changed. So, let has changed. So, let 's go back to that 's go back to that atlas I atlas I atlas I mentioned at the beginning. mentioned at the beginning. Now with real Now with real numbers: one numbers: one numbers: one small small small redrawn redrawn redrawn square is about 2% square is about 2% of the 2-gigabyte of the 2-gigabyte of the 2-gigabyte atlas, uncompressed atlas, uncompressed atlas, uncompressed and divided into tiles.

and divided into tiles. Each column is the Each column is the amount of data that amount of data that amount of data that the site transfers: the site transfers: the site transfers: uploads and uploads and uploads and downloads to downloads to any node that any node that any node that receives the change. Git LFS receives the change. Git LFS receives the change. Git LFS transfers all 2 transfers all 2 transfers all 2 gigabytes, the file goes gigabytes, the file goes gigabytes, the file goes up, and each up, and each up, and each site site site downloads those two downloads those two downloads those two gigabytes in full again. gigabytes in full again. gigabytes in full again. None of them None of them None of them send only the send only the send only the changed part. So changed part. So , you know, in a , you know, in a , you know, in a centralized centralized centralized model, by model, by model, by default for default for default for binary types, it's the binary types, it's the binary types, it's the same two gigabytes per same two gigabytes per same two gigabytes per server and then two server and then two server and then two gigabytes to each gigabytes to each gigabytes to each replica at the edge. replica at the edge. replica at the edge. The newer option to change The newer option to change The newer option to change parts would make parts would make parts would make the column smaller, but each the column smaller, but each the column smaller, but each version is still version is still version is still effectively effectively effectively stored in its entirety. stored in its entirety. stored in its entirety. But with Lore, you know, in But with Lore, you know, in But with Lore, you know, in this snippet, you'll this snippet, you'll this snippet, you'll see that Lore is see that Lore is see that Lore is only transferring only transferring only transferring about 20 megabytes here. about 20 megabytes here. about 20 megabytes here. So, it's barely more So, it's barely more So, it's barely more than 1%. You know, than 1%. You know, than 1%. You know, the fragments on each the fragments on each the fragments on each end have to be end have to be end have to be transmitted in their entirety. transmitted in their entirety. transmitted in their entirety. So in the change you So in the change you So in the change you will have like partial will have like partial will have like partial fragments. But on the fragments. But on the fragments. But on the same scale as the others. same scale as the others.

same scale as the others. This fragment is really This fragment is really This fragment is really so thin, both the sites so thin, both the sites so thin, both the sites and each column and each column and each column are multiplied by the same are multiplied by the same are multiplied by the same amount. So the gap amount. So the gap amount. So the gap here remains here remains here remains about 100 to about 100 to about 100 to one. These are the dimensions one. These are the dimensions one. These are the dimensions without compression. You know without compression. You know , every system , every system , every system compresses something, and already compresses something, and already compresses something, and already compressed assets compressed assets compressed assets only benefit only benefit only benefit a little here. This is for a little here. This is for a little here. This is for illustration purposes only, illustration purposes only, illustration purposes only, it is not a it is not a it is not a benchmark test at this time. We'll benchmark test at this time. We'll benchmark test at this time. We'll get to that, but you get to that, but you get to that, but you can imagine: can imagine: can imagine: multiply that by every multiply that by every multiply that by every site, every site, every site, every artist, every artist, every artist, every commit, every day. commit, every day. So, with the So, with the default settings default settings default settings that that most of you are using, most of you are using, one changed byte one changed byte one changed byte means a whole new means a whole new means a whole new version. You know, it version. You know, it version. You know, it 's stored in its entirety, 's stored in its entirety, 's stored in its entirety, sent in its entirety sent in its entirety sent in its entirety to every site that to every site that to every site that needs it. And that needs it. And that , in fact, is the , in fact, is the , in fact, is the whole problem in one whole problem in one whole problem in one line. This sounds like a line. This sounds like a line. This sounds like a small small small inefficiency, but inefficiency, but inefficiency, but it's not, because the it's not, because the it's not, because the studios are paying studios are paying studios are paying three times for it. You pay for three times for it. You pay for three times for it. You pay for bandwidth bandwidth bandwidth with every commit, with every commit, with every commit, you pay for you pay for you pay for storage, because storage, because storage, because every version of a every version of a every version of a large binary large binary large binary file is stored in its file is stored in its file is stored in its entirety. And for entirety. And for entirety. And for most of you, this is the most of you, this is the default setting. And default setting. And then you also then you also then you also pay the wait pay the wait pay the wait on each site. So on each site. So on each site. So every time someone every time someone every time someone syncs, the syncs, the syncs, the larger the larger the larger the files get, the higher files get, the higher files get, the higher all three all three all three costs get. This does not mean at all costs get. This does not mean at all costs get. This does not mean at all that someone that someone that someone is doing something wrong. It's just how the is doing something wrong. It's just how the is doing something wrong. It's just how the model model model was built.

was built. was built. Using a Using a source source code-based version control system for binary code-based version control system for binary code-based version control system for binary data. But, you know, data. But, you know, data. But, you know, the unit of versioning the unit of versioning the unit of versioning is actually a file. is actually a file. is actually a file. Each version is an Each version is an Each version is an entire file. So entire file. So entire file. So by changing this unit, by changing this unit, by changing this unit, you can change all you can change all you can change all the costs along with it. the costs along with it. So Lore's answer to So Lore's answer to this is to stop this is to stop this is to stop versioning files versioning files . But what is . But what is . But what is versioned then? versioned then? Well, fragments are versioned Well, fragments are versioned . So, Lore does not . So, Lore does not . So, Lore does not version these files. version these files. version these files. It version only It version only It version only parts of the files, and parts of the files, and parts of the files, and only the changed fragments are moved only the changed fragments are moved . So from a developer's perspective, it developer's perspective, it works just like works just like works just like any other any other any other version control system. The same version control system. The same version control system. The same commands commands commands you're used to. Pull, push, commit, you're used to. Pull, push, commit, you're used to. Pull, push, commit, sync, you know, but "under sync, you know, but "under sync, you know, but "under the hood" it works the hood" it works the hood" it works completely differently than the others completely differently than the others . And I think that's what . And I think that's what . And I think that's what I want to show you in I want to show you in I want to show you in this report. So, this report. So, this report. So, this bar, let's say, is this bar, let's say, is just one file, and a 2- just one file, and a 2- gigabyte texture. gigabyte texture. gigabyte texture. When you commit, When you commit, When you commit, the client on your the client on your the client on your workstation workstation workstation cuts the file into cuts the file into cuts the file into pieces pieces pieces using content- using content- aware splitting.

aware splitting. aware splitting. You know, instead of You know, instead of You know, instead of slicing through a certain slicing through a certain slicing through a certain number of bytes, the number of bytes, the number of bytes, the content itself determines where the content itself determines where the content itself determines where the cuts will be. With cuts will be. With cuts will be. With fixed- fixed- fixed- size slicing, one size slicing, one size slicing, one inserted byte would inserted byte would inserted byte would shift each shift each shift each subsequent slice, and subsequent slice, and subsequent slice, and each fragment would each fragment would each fragment would appear new. appear new. So here the sections So here the sections move with the move with the move with the content. Therefore, content. Therefore, content. Therefore, after each after each after each insertion, most insertion, most insertion, most fragments fragments fragments remain exactly the same as remain exactly the same as remain exactly the same as before. before. Each fragment Each fragment is given a name, which is a is given a name, which is a is given a name, which is a Blake3 hash of its Blake3 hash of its Blake3 hash of its bytes. It's like a bytes. It's like a bytes. It's like a fingerprint: fingerprint: fingerprint: the same bytes, the the same bytes, the same name. The names same name. The names same name. The names on this slide are on this slide are on this slide are just simulated just simulated just simulated values. But a values. But a values. But a fragment with a specific fragment with a specific fragment with a specific name is stored name is stored name is stored in the system exactly in the system exactly in the system exactly once. This applies to once. This applies to once. This applies to different files, branches, different files, branches, different files, branches, history, and as history, and as history, and as a result, a result, a result, deduplication deduplication deduplication follows from this follows from this follows from this naming system. naming system. Now imagine that the Now imagine that the artist repainted artist repainted artist repainted one area. The two one area. The two one area. The two fragments below it fragments below it fragments below it receive new names. The receive new names. The receive new names. The other 12 keep other 12 keep other 12 keep theirs because their bytes haven't theirs because their bytes haven't theirs because their bytes haven't changed. Now changed. Now changed. Now loading: loading: loading: Only Only Only fragments with new fragments with new fragments with new names are loaded. The 12 gray names are loaded. The 12 gray names are loaded. The 12 gray fragments are already in the fragments are already in the fragments are already in the repository from the first repository from the first repository from the first version, so version, so version, so only two only two only two of the fourteen are transferred here. And of the fourteen are transferred here. And of the fourteen are transferred here. And the version—it's really the version—it's really the version—it's really just that. This is a list of just that. This is a list of just that. This is a list of names. Version two is the names. Version two is the names. Version two is the version one list with version one list with version one list with two two two entries replaced.

entries replaced. entries replaced. Only two new Only two new Only two new fragments and their fragments and their fragments and their new list were recorded. So new list were recorded. So new list were recorded. So the versions in Lore are the versions in Lore are the versions in Lore are atomic. I think this is a atomic. I think this is a atomic. I think this is a great great great feature of this feature of this feature of this system as a whole. So system as a whole. So system as a whole. So what size is what size is what size is a fragment in bytes? a fragment in bytes? A small one. The A small one. The expected expected expected fragment size is fragment size is fragment size is about 64 kilobytes. And about 64 kilobytes. And about 64 kilobytes. And fragments never fragments never fragments never exceed 256 exceed 256 exceed 256 kilobytes. So that 2- kilobytes. So that 2- gigabyte gigabyte gigabyte texture atlas is tens of texture atlas is tens of texture atlas is tens of thousands of fragments, thousands of fragments, thousands of fragments, not 14. So the 14 here is not 14. So the 14 here is not 14. So the 14 here is just a simplified diagram. just a simplified diagram. just a simplified diagram. In fact, there are In fact, there are In fact, there are many more. And many more. And many more. And editing one editing one editing one section of this atlas section of this atlas section of this atlas affects only those affects only those affects only those fragments that it fragments that it fragments that it covers. That's why covers. That's why covers. That's why about 20 about 20 about 20 megabytes from a few megabytes from a few megabytes from a few slides ago, you slides ago, you slides ago, you know where that know where that know where that comes from. Now the other comes from. Now the other comes from. Now the other part I part I part I want to talk about is what want to talk about is what want to talk about is what makes it makes it makes it special from a special from a special from a network perspective, and it's network perspective, and it's network perspective, and it's incredibly fast. incredibly fast. incredibly fast. I think one of my I think one of my I think one of my favorite quotes from favorite quotes from favorite quotes from one of the one of the one of the Epic AR architects Epic AR architects Epic AR architects is that he is that he is that he could "melt" a could "melt" a could "melt" a network card, and network card, and network card, and now, after now, after now, after experimenting, I can experimenting, I can experimenting, I can see it. So, each see it. So, each see it. So, each fragment has its own fragment has its own fragment has its own query. So you query. So you query. So you see, thousands of see, thousands of see, thousands of small requests small requests small requests have to be constantly have to be constantly have to be constantly moving. Lore does moving. Lore does moving. Lore does this using the this using the this using the QUIC protocol. This is a QUIC protocol. This is a QUIC protocol. This is a modern modern modern web3 transport that works web3 transport that works web3 transport that works on top of UDP. You know, on top of UDP. You know, on top of UDP. You know, one connection carries one connection carries one connection carries eight threads. Two of eight threads. Two of eight threads. Two of them are free from the them are free from the them are free from the main data, for main data, for main data, for fast fast fast tree reading and tree reading and tree reading and synchronization needs. So, synchronization needs. So, synchronization needs. So, the client can

the client can the client can support up to 10,000 support up to 10,000 support up to 10,000 requests in a process, and requests in a process, and requests in a process, and the responses can the responses can the responses can come in come in any order, any order, any order, sorted by ID. As you sorted by ID. As you sorted by ID. As you know, this is UDP. So what's know, this is UDP. So what's know, this is UDP. So what's going on? OK. going on? OK. going on? OK. Let's say you Let's say you Let's say you lost a package. Well, okay lost a package. Well, okay lost a package. Well, okay . Only its own . Only its own . Only its own stream is waiting to be stream is waiting to be resent. The other resent. The other seven continue seven continue seven continue to work, whereas in to work, whereas in to work, whereas in TCP, one lost TCP, one lost TCP, one lost packet stops the entire packet stops the entire packet stops the entire connection. Another thing to connection. Another thing to remember: remember: remember: QUIC works in the application, QUIC works in the application, QUIC works in the application, not in the OS. So, Lore can not in the OS. So, Lore can not in the OS. So, Lore can customize itself customize itself customize itself . . . BBR congestion control, BBR congestion control, BBR congestion control, i.e. i.e. i.e. bottleneck bandwidth bottleneck bandwidth bottleneck bandwidth and distribution cycle time and distribution cycle time and distribution cycle time —when —when —when loading, it loading, it loading, it sends one sends one sends one stream at a time, so stream at a time, so stream at a time, so fragments arrive fragments arrive fragments arrive faster. Then each faster. Then each faster. Then each window is the window is the window is the bandwidth multiplied bandwidth multiplied bandwidth multiplied by the cycle time, so for by the cycle time, so for by the cycle time, so for customers it's a gigabit customers it's a gigabit customers it's a gigabit per second for 100 per second for 100 per second for 100 milliseconds, or milliseconds, or milliseconds, or about 13.4 megabytes.

about 13.4 megabytes. When a client When a client opens a single opens a single opens a single connection, by connection, by connection, by default it maxes out default it maxes out at the window value at the window value divided by the divided by the divided by the cycle time. So, at 100 cycle time. So, at 100 cycle time. So, at 100 milliseconds, that's milliseconds, that's milliseconds, that's about 134 megabytes about 134 megabytes about 134 megabytes per second. And about per second. And about per second. And about 200 at 67. This is a calculated 200 at 67. This is a calculated 200 at 67. This is a calculated value, not a value, not a value, not a measured one. The testing measured one. The testing measured one. The testing we conducted here was we conducted here was we conducted here was done entirely in done entirely in done entirely in our VPC, or our VPC, or our VPC, or virtual virtual virtual private cloud. private cloud. private cloud. Another thing to Another thing to Another thing to consider: the machines consider: the machines consider: the machines we test on, we test on, we test on, EC2 instances, are limited to EC2 instances, are limited to EC2 instances, are limited to about five about five about five gigabits per gigabits per gigabits per second, or about second, or about second, or about 625 megabytes per 625 megabytes per 625 megabytes per second. So each second. So each second. So each client number client number client number you see in you see in you see in the following sections the following sections the following sections is actually just is actually just is actually just one connection. If the one connection. If the one connection. If the remote remote remote client's channel is faster than this client's channel is faster than this client's channel is faster than this limit, increase limit, increase limit, increase the maximum the maximum the maximum number of connections number of connections number of connections to, say, 10. But to, say, 10. But to, say, 10. But you know, one way or another, the you know, one way or another, the you know, one way or another, the lower level lower level lower level of the server will help. of the server will help.

So, So, block splitting, hashing, block splitting, hashing, block splitting, hashing, compression, compression, compression, QUIC encryption—all of this QUIC encryption—all of this QUIC encryption—all of this is done on the is done on the is done on the client's processor, and client's processor, and client's processor, and that's something to consider. that's something to consider. So how does the customer So how does the customer get their file get their file get their file back? It back? It back? It only communicates with the only communicates with the only communicates with the Lore server, an Lore server, an Lore server, an edge node, edge node, edge node, or directly with the or directly with the or directly with the core tier. core tier. core tier. He doesn't have He doesn't have He doesn't have AWS credentials or AWS credentials or AWS credentials or AWS SDKs. Fragments AWS SDKs. Fragments AWS SDKs. Fragments are transmitted over a are transmitted over a are transmitted over a QUIC connection, as you QUIC connection, as you QUIC connection, as you have seen. And everything else— have seen. And everything else— branches and locking— branches and locking— is transmitted via gRPC is transmitted via gRPC is transmitted via gRPC over TCP. So, over TCP. So, over TCP. So, on the right is the on the right is the on the right is the authoritarian level. authoritarian level. S3 is an object S3 is an object storage, one object storage, one object storage, one object per shard, named per shard, named per shard, named by its hash. And DynamoDB by its hash. And DynamoDB is a managed NoSQL is a managed NoSQL is a managed NoSQL key- key- value database. It value database. It value database. It contains an index contains an index contains an index of fragments and of fragments and of fragments and metadata.

metadata. metadata. Branch pointers and Branch pointers and Branch pointers and main- main- main- level locks in the Lore.8 structure are level locks in the Lore.8 structure are level locks in the Lore.8 structure are only available for only available for only available for this level. So only this level. So only this level. So only Lore servers have Lore servers have Lore servers have access to them. They access to them. They access to them. They act as a point of act as a point of act as a point of abstraction. When abstraction. When abstraction. When a client searches for a revision, a client searches for a revision, a client searches for a revision, it requests a it requests a it requests a pointer to the branch, and pointer to the branch, and pointer to the branch, and the server reads it from the server reads it from the server reads it from DynamoDB. Then it DynamoDB. Then it DynamoDB. Then it traverses the tree, traverses the tree, traverses the tree, because a tree is because a tree is because a tree is also fragments, and also fragments, and also fragments, and the client the client the client only views the paths in its only views the paths in its only views the paths in its view. Third, this is view. Third, this is view. Third, this is data retrieval: one data retrieval: one data retrieval: one GET request per fragment, GET request per fragment, GET request per fragment, maximum 256 KB, many maximum 256 KB, many maximum 256 KB, many requests at the same time. requests at the same time. requests at the same time. So, here the node does not So, here the node does not So, here the node does not find the data. The find the data. The find the data. The peripheral cache is not peripheral cache is not peripheral cache is not working. Then it working. Then it working. Then it goes to the goes to the goes to the main tier, which main tier, which main tier, which checks its own checks its own checks its own cache before cache before cache before retrieving the data from S3. In retrieving the data from S3. In retrieving the data from S3. In both cases, you'll both cases, you'll both cases, you'll see these two levels of see these two levels of see these two levels of caching, and I'll caching, and I'll caching, and I'll go into that a little bit go into that a little bit go into that a little bit later. Finally, the later. Finally, the later. Finally, the recovery process.

recovery process. The client checks the hash of The client checks the hash of each tree, each tree, each tree, assembles the file, and writes assembles the file, and writes assembles the file, and writes it to your it to your it to your workspace. Re- workspace. Re- workspace. Re- reading does not reading does not reading does not contact contact contact the server. The client the server. The client the server. The client never knows exactly which never knows exactly which never knows exactly which level responded level responded level responded in this case. But in this case. But in this case. But here it is worth considering here it is worth considering here it is worth considering three points. Every three points. Every three points. Every byte passes through the byte passes through the byte passes through the Lore server. So for Lore server. So for Lore server. So for large large large syncs and syncs and syncs and imports, imports, imports, server bandwidth is the server bandwidth is the server bandwidth is the real limit real limit real limit we measure, and we'll we measure, and we'll we measure, and we'll look at that data in look at that data in look at that data in chapter five, but chapter five, but chapter five, but every miss that every miss that every miss that hits S3 is one hits S3 is one hits S3 is one S3 GET call per S3 GET call per S3 GET call per shard. So, shard. So, shard. So, getting that getting that getting that fragment, you know, and fragment, you know, and fragment, you know, and looking it up in DynamoDB, where looking it up in DynamoDB, where looking it up in DynamoDB, where the index the index the index and, actually, the tables are stored. and, actually, the tables are stored. So, and at the same time, the client needs a the client needs a lore token to log in. This is another lore token to log in. This is another lore token to log in. This is another point to point to point to remember: AWS permissions are remember: AWS permissions are remember: AWS permissions are invisible to this invisible to this invisible to this system. All of this is system. All of this is system. All of this is governed within the boundaries of lore.

governed within the boundaries of lore. So, with that in mind, I So, with that in mind, I want to take a step want to take a step want to take a step back. Maybe there back. Maybe there back. Maybe there were were were too many details on the last slide, but this is too many details on the last slide, but this is too many details on the last slide, but this is exactly the exactly the exactly the picture that is worth picture that is worth picture that is worth remembering. There are remembering. There are remembering. There are two levels here. On the right is two levels here. On the right is the trusted tier the trusted tier the trusted tier we just we just we just talked about: Amazon S3 and Amazon talked about: Amazon S3 and Amazon talked about: Amazon S3 and Amazon DynamoDB. So, S3, or Simple DynamoDB. So, S3, or Simple DynamoDB. So, S3, or Simple Storage Service, is a Storage Service, is a Storage Service, is a scalable cloud scalable cloud scalable cloud storage of objects, storage of objects, storage of objects, imagine files. imagine files. It provides It provides eleven nines of eleven nines of eleven nines of reliability and reliability and reliability and multi-level multi-level deletion control mechanisms, so I deletion control mechanisms, so I personally believe that personally believe that personally believe that this is one of the main this is one of the main this is one of the main reasons for the saying: " reasons for the saying: " The Internet The Internet The Internet forgets nothing." However, its forgets nothing." However, its forgets nothing." However, its partner in this partner in this partner in this system is DynamoDB. It is a system is DynamoDB. It is a system is DynamoDB. It is a serverless serverless serverless NoSQL database that NoSQL database that NoSQL database that delivers delivers single- single- millisecond performance millisecond performance millisecond performance at virtually any at virtually any at virtually any scale. Together, this scale. Together, this scale. Together, this combination is an combination is an combination is an authoritative copy of authoritative copy of authoritative copy of your repository.

your repository. Next to them Next to them is the is the is the write layer, which receives write layer, which receives write layer, which receives each commit and each commit and each commit and handles handles handles read operations. In this read operations. In this read operations. In this case, it is a container case, it is a container case, it is a container on EC2 with a local NVMe on EC2 with a local NVMe device, which is device, which is device, which is essentially a very fast SSD. essentially a very fast SSD. And this is, in fact, the first And this is, in fact, the first level of caching in the level of caching in the level of caching in the system. To the left of system. To the left of system. To the left of it are it are it are edge edge edge pods—these are regular pods—these are regular pods—these are regular EC2 instances with EC2 instances with EC2 instances with NVMe cache. You place these NVMe cache. You place these NVMe cache. You place these nodes where nodes where reads occur, reads occur, such as near your such as near your such as near your CI fleet or CI fleet or CI fleet or regional regional regional teams. They are the teams. They are the teams. They are the second level of cache in the second level of cache in the second level of cache in the system. And on system. And on system. And on the far left the far left the far left side are side are side are the clients: artists, CI the clients: artists, CI machines. Clients machines. Clients machines. Clients connect connect connect directly to the directly to the directly to the machines that machines that machines that serve them, in this serve them, in this serve them, in this case, an case, an case, an edge node edge node edge node or a central or a central or a central recording tier. But what's recording tier. But what's recording tier. But what's interesting here is the interesting here is the interesting here is the removal test. If you're removal test. If you're removal test. If you're asking what can be asking what can be asking what can be deleted on this deleted on this deleted on this system without losing system without losing system without losing any any any committed commits committed commits , the answer is right , the answer is right , the answer is right here. In general, anything on here. In general, anything on here. In general, anything on the left the left the left can fail. And, can fail. And, can fail. And, in fact, losing these in fact, losing these in fact, losing these caches costs time, caches costs time, caches costs time, but not data. You know, the but not data. You know, the but not data. You know, the recording level recording level recording level can also be replaced. It's really just a can also be replaced. It's really just a cache, essentially. So, this is the only cache, essentially. So, this is the only input for commits,

input for commits, input for commits, although as we'll although as we'll although as we'll discuss later, discuss later, discuss later, it works in an it works in an it works in an auto- auto- auto- scaling group. So scaling group. So scaling group. So it will automatically it will automatically it will automatically be replaced if be replaced if be replaced if the instance goes the instance goes the instance goes down, and the commits down, and the commits down, and the commits will just wait for it to will just wait for it to will just wait for it to come back. As I come back. As I come back. As I mentioned, S3 and DynamoDB in the mentioned, S3 and DynamoDB in the mentioned, S3 and DynamoDB in the repository have repository have repository have multi-layered multi-layered multi-layered protection against deletion, protection against deletion, protection against deletion, but anything left behind but anything left behind but anything left behind can be lost at can be lost at can be lost at any time without harm any time without harm any time without harm to the data. So, like to the data. So, like to the data. So, like other other other version control systems, Lure has the same version control systems, Lure has the same version control systems, Lure has the same commands and commands and commands and actions. The arrows here actions. The arrows here actions. The arrows here show which direction show which direction show which direction the bytes are moving. For the bytes are moving. For the bytes are moving. For a commit, the main heavy lifting a commit, the main heavy lifting a commit, the main heavy lifting is done is done is done at the at the at the artist's workstation. Let's say artist's workstation. Let's say artist's workstation. Let's say artist A breaks the artist A breaks the artist A breaks the files into fragments, files into fragments, files into fragments, hashes them, compresses them, hashes them, compresses them, hashes them, compresses them, uploads new ones, and the uploads new ones, and the uploads new ones, and the server stores server stores server stores each one only once. each one only once. These are the two orange These are the two orange blocks you see blocks you see blocks you see here. The push is actually here. The push is actually here. The push is actually very small. This is an interesting very small. This is an interesting very small. This is an interesting difference from difference from difference from other systems. The commit other systems. The commit other systems. The commit has already loaded has already loaded has already loaded the fragments, so the push the fragments, so the push the fragments, so the push only checks for their only checks for their only checks for their presence and presence and presence and moves the moves the moves the branch pointer in Dynamo DB from one branch pointer in Dynamo DB from one branch pointer in Dynamo DB from one version to another. So version to another. So version to another. So now "main" just now "main" just now "main" just points to v2. In our points to v2. In our points to v2. In our tests, this usually tests, this usually tests, this usually takes less than half a takes less than half a takes less than half a second. Again, second. Again, second. Again, this was the most this was the most this was the most surprising moment when surprising moment when surprising moment when starting work.

starting work. starting work. Now synchronization: Now synchronization: Now synchronization: let's say artist B let's say artist B let's say artist B needs to synchronize needs to synchronize , he , he , he only needs to pull up the only needs to pull up the only needs to pull up the missing pieces. In missing pieces. In missing pieces. In this case, through a this case, through a this case, through a peripheral node. peripheral node. peripheral node. This node and these caches This node and these caches This node and these caches keep a copy of keep a copy of keep a copy of the fragments for the the fragments for the the fragments for the next next next user as well. It user as well. It user as well. It gets every file from gets every file from gets every file from your working your working your working environment and nothing environment and nothing environment and nothing outside of it. Here, the outside of it. Here, the outside of it. Here, the write layer is located write layer is located write layer is located on the arrow between the node on the arrow between the node on the arrow between the node and the repository, it is and the repository, it is and the repository, it is actually built actually built actually built into that point. Imagine into that point. Imagine into that point. Imagine Lure servers Lure servers Lure servers in the middle of this in the middle of this in the middle of this scheme. Remember that scheme. Remember that scheme. Remember that moving moving moving fragments is a fragments is a fragments is a heavy process heavy process heavy process that grows in size that grows in size that grows in size with changes, and the with changes, and the pointer movement during a push is tiny. Up to is tiny. Up to is tiny. Up to this point, you know this point, you know , there are four sentences , there are four sentences , there are four sentences that really that really that really describe this whole describe this whole describe this whole model. So, everything model. So, everything model. So, everything we look at next— we look at next— size, cost, size, cost, size, cost, drawbacks—it all drawbacks—it all drawbacks—it all boils down to one of boils down to one of boils down to one of these four points. these four points.

First: content First: content is broken into is broken into is broken into small fragments and small fragments and small fragments and deduplicated. Therefore, deduplicated. Therefore, deduplicated. Therefore, editing a editing a editing a huge file huge file huge file affects only the changed affects only the changed affects only the changed fragments, and the same fragments, and the same fragments, and the same parts in different parts in different parts in different versions are saved versions are saved versions are saved only once. The following only once. The following only once. The following commits are actually the commits are actually the commits are actually the actions that load actions that load actions that load the fragments. The the fragments. The the fragments. The push operation checks for their push operation checks for their push operation checks for their presence and presence and presence and moves the moves the moves the branch pointer. One of them is branch pointer. One of them is branch pointer. One of them is heavy, and the other is quite heavy, and the other is quite heavy, and the other is quite light. And, as I light. And, as I light. And, as I said, it came as a said, it came as a said, it came as a surprise to me when surprise to me when surprise to me when I first committed I first committed . It seems like it was . It seems like it was . It seems like it was Staccobot or some other project; Staccobot or some other project; Staccobot or some other project; I recorded and I recorded and I recorded and sent, well, I don't know sent, well, I don't know , about a gigabyte of data. , about a gigabyte of data. , about a gigabyte of data. This push happened almost This push happened almost This push happened almost instantly, which instantly, which instantly, which surprised me. I guess surprised me. I guess surprised me. I guess I didn't notice that it was the I didn't notice that it was the I didn't notice that it was the commit that moved commit that moved commit that moved the fragments into the system. the fragments into the system. Now the third point. Now the third point. Syncing Syncing Syncing actually actually actually only loads your only loads your only loads your view of the data. view of the data. view of the data. Pod never Pod never Pod never keeps a single copy of keeps a single copy of keeps a single copy of anything anything anything recorded. So, a recorded. So, a recorded. So, a pod can fail, pod can fail, pod can fail, and a new one will fill up and a new one will fill up and a new one will fill up through the appropriate through the appropriate through the appropriate level when level when requests for new requests for new fragments appear in the system. And lastly: fragments appear in the system. And lastly: fragments appear in the system. And lastly: the repository the repository the repository actually only works in actually only works in actually only works in append mode.

append mode. append mode. Caches can be Caches can be Caches can be cleared, but nothing cleared, but nothing cleared, but nothing will delete data from S3. will delete data from S3. From this S3 storage. So From this S3 storage. So every fragment in every fragment in every fragment in your commit your commit your commit remains until remains until remains until someone deliberately deletes it someone deliberately deletes it someone deliberately deletes it . Lore can . Lore can . Lore can clear history, but clear history, but clear history, but not by default. not by default. not by default. So the storage volume So the storage volume So the storage volume will constantly grow. will constantly grow. You keep saying You keep saying that peripheral cache that peripheral cache that peripheral cache can be thrown away, as if can be thrown away, as if can be thrown away, as if that's some kind of advantage. that's some kind of advantage. that's some kind of advantage. Why is this important for the Why is this important for the Why is this important for the studio? Because a cache studio? Because a cache studio? Because a cache that can be discarded that can be discarded is a cache that doesn't is a cache that doesn't is a cache that doesn't need to be filled need to be filled need to be filled or or or backed up. Rene, backed up. Rene, backed up. Rene, I think you've been through this I think you've been through this I think you've been through this . . . Oh yes, and how. And that's what Oh yes, and how. And that's what Oh yes, and how. And that's what it gives the studio. The new it gives the studio. The new it gives the studio. The new location location location is synchronized is synchronized is synchronized through the edge through the edge through the edge pod instead of pod instead of pod instead of waiting for a full waiting for a full waiting for a full archive node. archive node. archive node. CI agents can also CI agents can also CI agents can also synchronize synchronize synchronize through it. through it. through it. Starting from the top Starting from the top Starting from the top panel, a complete panel, a complete panel, a complete archive node archive node archive node built in the built in the built in the classic way. classic way. classic way. For this example, we will For this example, we will For this example, we will consider a consider a consider a 500 500 500 gigabyte repository and a gigabyte repository and a gigabyte repository and a 1 gigabit 1 gigabit 1 gigabit per second communication channel. Provisioning per second communication channel. Provisioning per second communication channel. Provisioning resources and resources and resources and restoring a restoring a restoring a checkpoint checkpoint checkpoint takes about 20 takes about 20 takes about 20 minutes, and copying minutes, and copying minutes, and copying the archive over a the archive over a the archive over a communication channel takes about 67 communication channel takes about 67 communication channel takes about 67 minutes at full minutes at full minutes at full speed. The speed. The speed. The verification step adds another 10 verification step adds another 10 verification step adds another 10 minutes. And this allows us minutes. And this allows us minutes. And this allows us to make our first to make our first to make our first useful commit at useful commit at useful commit at minute 97. A caching minute 97. A caching minute 97. A caching proxy or proxy or proxy or edge node in edge node in edge node in caching mode caching mode caching mode makes this bandwidth makes this bandwidth makes this bandwidth shorter. And the pod also shorter. And the pod also shorter. And the pod also resets the checkpoint resets the checkpoint resets the checkpoint and database of the and database of the and database of the edge node.

edge node. Now it Now it takes 5 takes 5 takes 5 minutes to start the oven. The first minutes to start the oven. The first minutes to start the oven. The first sync sync sync only loads the only loads the only loads the fragment that fragment that fragment that the site is working on, which adds the site is working on, which adds the site is working on, which adds another couple of minutes. And then the another couple of minutes. And then the another couple of minutes. And then the first useful first useful first useful commit appears after commit appears after commit appears after about 8 about 8 about 8 minutes. So this minutes. So this minutes. So this arrow is the delay arrow is the delay arrow is the delay added by the full added by the full added by the full archive of the edge archive of the edge archive of the edge node, which is node, which is node, which is about 89 minutes. And the about 89 minutes. And the about 89 minutes. And the gray bar is gray bar is gray bar is the filling of the pod cache the filling of the pod cache the filling of the pod cache while people are working. while people are working. while people are working. No one is waiting for it to No one is waiting for it to No one is waiting for it to end. And end. And end. And the first sync the first sync the first sync of a file, picking it up of a file, picking it up of a file, picking it up from the right from the right from the right level, means that all level, means that all level, means that all subsequent users subsequent users subsequent users simply get simply get simply get it from the pod. And three it from the pod. And three it from the pod. And three accounts with the previously accounts with the previously accounts with the previously mentioned pods, without a mentioned pods, without a mentioned pods, without a full archive on the full archive on the full archive on the site and without site and without site and without the need for an the need for an the need for an initial initial initial download. You download. You download. You still calculate still calculate still calculate the size based on the size based on the size based on how many people how many people how many people are syncing at 9:00, are syncing at 9:00, are syncing at 9:00, but we'll but we'll but we'll get to that later. So, the third get to that later. So, the third get to that later. So, the third pattern is pattern is pattern is migration day. When you migration day. When you migration day. When you move a large move a large move a large existing tree, Ben, existing tree, Ben, existing tree, Ben, you were in charge of it. And so it you were in charge of it. And so it you were in charge of it. And so it was. We was. We was. We imported a snapshot of imported a snapshot of imported a snapshot of 105 105 105 gigabytes of media data. gigabytes of media data. Actually, it was Actually, it was Meridian from Netflix, a film Meridian from Netflix, a film Meridian from Netflix, a film under a Creative under a Creative under a Creative Commons license, but it was Commons license, but it was Commons license, but it was uncompressed uncompressed uncompressed UHD video frames and UHD video frames and UHD video frames and Dolby Atmos master copies, Dolby Atmos master copies, Dolby Atmos master copies, it's not a game project, there's it's not a game project, there's it's not a game project, there's no story here.

no story here. We sent it to the We sent it to the desired level, but desired level, but desired level, but the first time it the first time it the first time it didn't work: within a few didn't work: within a few didn't work: within a few minutes, S3 applied minutes, S3 applied minutes, S3 applied limits to this limits to this limits to this new new new account and new account and new account and new bucket, and the speed bucket, and the speed bucket, and the speed dropped by about dropped by about dropped by about nine times. She nine times. She nine times. She never recovered. never recovered. never recovered. So after about an hour or So after about an hour or So after about an hour or so, we just so, we just so, we just stopped the process. But stopped the process. But stopped the process. But through the peripheral through the peripheral through the peripheral pod, the same import pod, the same import pod, the same import completed completed completed in about 8 minutes and 45 seconds. in about 8 minutes and 45 seconds. in about 8 minutes and 45 seconds. So what happened? The pod So what happened? The pod So what happened? The pod does not use does not use does not use buffering. It buffering. It buffering. It confirms each confirms each confirms each fragment only fragment only fragment only once. once. once. S3 has the required level and never S3 has the required level and never S3 has the required level and never keeps a single copy. keeps a single copy. keeps a single copy. So, one So, one So, one run for each. This run for each. This run for each. This was using was using was using lore 0.8.6. The servers have been lore 0.8.6. The servers have been lore 0.8.6. The servers have been increased in size increased in size increased in size compared to our compared to our compared to our initial initial initial calculations. And, calculations. And, calculations. And, you know, you know, you know, the catch in this case— the catch in this case— we'll look at why we'll look at why we'll look at why this didn't work out and this didn't work out and this didn't work out and what to look out for what to look out for what to look out for when when when deploying Lore— deploying Lore— is that it is that it is that it routes large routes large routes large imports, meaning anything imports, meaning anything over 50 gigabytes over 50 gigabytes , through a pod. Um, the , through a pod. Um, the , through a pod. Um, the back pressure and back pressure and back pressure and control that's provided there is control that's provided there is much better, again much better, again , at the time of , at the time of , at the time of writing this writing this writing this , and I have to , and I have to , and I have to make a little make a little make a little introduction. So, Lore is currently in the introduction. So, Lore is currently in the pre-version 1.0 stage. So it's pre-version 1.0 stage. So it's actively working on a actively working on a actively working on a 1.0 release, but we 1.0 release, but we 1.0 release, but we wanted to share wanted to share wanted to share some of the things we some of the things we some of the things we encountered

encountered encountered while working with it while working with it while working with it and loading 105 and loading 105 and loading 105 gigabytes of data. gigabytes of data. For example, 10-gigabyte For example, 10-gigabyte repositories repositories repositories went straight through went straight through went straight through without any problems. without any problems. But 105 gigabytes But 105 gigabytes directly, with directly, with directly, with standard standard standard settings on the settings on the settings on the new account, did not new account, did not new account, did not complete the process. complete the process. Again, I'll Again, I'll explain why a little explain why a little explain why a little later and how you can later and how you can later and how you can avoid this avoid this avoid this problem. So, with that in problem. So, with that in problem. So, with that in mind, 50 is mind, 50 is mind, 50 is our threshold between them. This is our threshold between them. This is our threshold between them. This is not a measured threshold, not a measured threshold, not a measured threshold, but 50 gigabytes "out of but 50 gigabytes "out of but 50 gigabytes "out of the box" is a good the box" is a good the box" is a good number to keep number to keep number to keep in mind. in mind. So, in general, a pod So, in general, a pod never keeps a never keeps a never keeps a single copy of your single copy of your single copy of your data. Reading data. Reading data. Reading fills it as it fills it as it fills it as it runs, and writing runs, and writing runs, and writing flows through flows through flows through it to a reliable it to a reliable it to a reliable storage level. storage level. storage level. This is the architecture. This is the architecture. This is the architecture. What does it take to What does it take to What does it take to actually actually actually set this up? You set this up? You set this up? You ran these modules. ran these modules. Yes. So, once Yes. So, once your server image is already in your server image is already in your server image is already in the registry and you have a the registry and you have a the registry and you have a network path to AWS, network path to AWS, network path to AWS, deployment takes deployment takes deployment takes about half a day. And you about half a day. And you about half a day. And you only need one only need one only need one Terraform module to Terraform module to Terraform module to deploy all three deploy all three deploy all three levels. This module levels. This module levels. This module comes from the Cloud Game comes from the Cloud Game comes from the Cloud Game Development Toolkit, which is an Development Toolkit, which is an open source AWS project open source AWS project consisting of three consisting of three consisting of three parts. Assets parts. Assets are reusable are reusable are reusable components. These are your components. These are your components. These are your Packer templates, Packer templates, Packer templates, pipeline definitions, pipeline definitions, pipeline definitions, and and and deployment scripts. Modules deployment scripts. Modules are actually Terraform itself, are actually Terraform itself, are actually Terraform itself, called from called from called from your code. And

your code. And your code. And the examples are complete the examples are complete the examples are complete configurations that configurations that configurations that demonstrate how the demonstrate how the demonstrate how the modules work together. modules work together. modules work together. Modules cover Modules cover Modules cover version control, Unreal version control, Unreal version control, Unreal Horde, Unreal Cloud DDC, which Horde, Unreal Cloud DDC, which Horde, Unreal Cloud DDC, which caches derivative data caches derivative data caches derivative data like compiled like compiled like compiled shaders, shaders, shaders, build servers, virtual build servers, virtual build servers, virtual workstations, and workstations, and workstations, and Unity tools. Lore is the Unity tools. Lore is the newest of them. You newest of them. You newest of them. You can see it can see it can see it highlighted in highlighted in highlighted in orange orange orange . And you . And you . And you only take the component only take the component only take the component you need. This is you need. This is you need. This is regular Terraform. You regular Terraform. You regular Terraform. You can can can read it first. read it first. read it first. The repository is on The repository is on The repository is on the screen, we'll also the screen, we'll also the screen, we'll also share it on the share it on the share it on the last slide. last slide. This is what This is what this terraform apply command creates for you this terraform apply command creates for you this terraform apply command creates for you . This is an example from the . This is an example from the . This is an example from the default settings default settings default settings , and , and , and from left to right you from left to right you from left to right you see clients, see clients, see clients, i.e. artists and CI i.e. artists and CI agents. They agents. They agents. They connect through connect through connect through our network our network our network services as a dedicated services as a dedicated services as a dedicated line to AWS. Pods and the line to AWS. Pods and the line to AWS. Pods and the recording layer recording layer recording layer are in are in are in private subnets private subnets . They are not accessible from . They are not accessible from . They are not accessible from the internet and the internet and the internet and only accept only accept only accept traffic from office or traffic from office or traffic from office or VPN ranges that you VPN ranges that you VPN ranges that you allow. The clients allow. The clients allow. The clients then then then synchronize synchronize synchronize via peripheral via peripheral via peripheral pods. There are two of them, they are pods. There are two of them, they are pods. There are two of them, they are located in different located in different located in different places in the region and places in the region and places in the region and are designed are designed are designed for approximately 20-40 people.

for approximately 20-40 people. for approximately 20-40 people. Behind them, the Behind them, the Behind them, the write level is accepted by each write level is accepted by each write level is accepted by each commit. If a machine commit. If a machine commit. If a machine fails, fails, autoscaling groups autoscaling groups will replace it. And will replace it. And will replace it. And finally, on the right, finally, on the right, the same reliable the same reliable the same reliable layer that layer that layer that Ben talked about: S3, which Ben talked about: S3, which Ben talked about: S3, which stores the shards, stores the shards, stores the shards, and DynamoDB for indexes and DynamoDB for indexes and DynamoDB for indexes and branch pointers. and branch pointers. A few people may A few people may skip the skip the skip the pod stage when pod stage when pod stage when deploying. The deploying. The deploying. The reliability level is reliability level is reliability level is the same in both the same in both the same in both configurations. So configurations. So configurations. So adding pods adding pods adding pods later doesn't require later doesn't require later doesn't require moving data. moving data. moving data. You add pod blocks, You add pod blocks, You add pod blocks, run apply and run apply and run apply and simply direct simply direct simply direct clients to these pods. clients to these pods. clients to these pods. If the recording level If the recording level If the recording level also increases, this can be also increases, this can be also increases, this can be taken into account by simply taken into account by simply taken into account by simply changing the variable. changing the variable. changing the variable. During testing, we During testing, we During testing, we also ran our also ran our also ran our own copy of the module. There own copy of the module. There are actually only three are actually only three decisions to decisions to decisions to make on the first make on the first make on the first day of day of day of setup. Everything setup. Everything setup. Everything else can wait. else can wait. else can wait. The first is the The first is the environment container image and environment container image and allowed allowed allowed IP address ranges for login. IP address ranges for login. You can start with an You can start with an example and then example and then example and then simply fill in simply fill in simply fill in the rest of the data.

the rest of the data. the rest of the data. Instance types for the Instance types for the Instance types for the record level and each record level and each record level and each pod, the number of pods pod, the number of pods , the number of , the number of , the number of peripheral pod blocks, and the peripheral pod blocks, and the often forgotten region. It is defined in It is defined in the provider, not in the provider, not in the provider, not in the module. You can the module. You can the module. You can choose the instance type choose the instance type choose the instance type yourself. Each pod will yourself. Each pod will have around 32 vCPUs by default, have around 32 vCPUs by default, which will cost which will cost which will cost around $1,000 per around $1,000 per around $1,000 per month depending on the month depending on the month depending on the region. The region. The region. The default entry level will default entry level will default entry level will have have have an instance for around an instance for around an instance for around $250. Ben $250. Ben $250. Ben notes that smaller notes that smaller notes that smaller instances work just as well instances work just as well instances work just as well . The . The default and default and minimal examples minimal examples minimal examples are provided as are provided as are provided as development development development configurations. There configurations. There configurations. There is no authorization and no is no authorization and no is no authorization and no protection against protection against protection against deletion. For deletion. For deletion. For production systems, it is worth production systems, it is worth production systems, it is worth setting the auth mode to setting the auth mode to setting the auth mode to Cognito or another Cognito or another Cognito or another external external identity provider. identity provider. Enable deletion protection Enable deletion protection Enable deletion protection and and and disable force disable force disable force destroy. The bucket destroy. The bucket destroy. The bucket comes with versioning disabled comes with versioning disabled , so you should you should think about your backup plan yourself think about your backup plan yourself . In pods, the standard . In pods, the standard . In pods, the standard pod block also doesn't pod block also doesn't pod block also doesn't have have have login credentials, so it's login credentials, so it's login credentials, so it's worth worth worth using the using the using the patch pod block. So, the next patch pod block. So, the next patch pod block. So, the next question I question I question I always get asked is: always get asked is: always get asked is: what size? And Ben, what size? And Ben, what size? And Ben, you spent the summer you spent the summer you spent the summer measuring it.

measuring it. Yes. So, you know, the Yes. So, you know, the honest answer to honest answer to honest answer to the question of "what the question of "what the question of "what size" is something size" is something size" is something new. It used to be new. It used to be new. It used to be just "it depends on the just "it depends on the just "it depends on the circumstances." But that's exactly why circumstances." But that's exactly why circumstances." But that's exactly why we created a we created a we created a test system test system test system called lorebench to called lorebench to called lorebench to measure this. And it measure this. And it measure this. And it launches real launches real launches real lore clients, real lore clients, real lore clients, real deployments. It deployments. It deployments. It actually takes everything actually takes everything actually takes everything you saw in the you saw in the you saw in the previous diagram and previous diagram and previous diagram and combines it with combines it with combines it with a few test a few test a few test clients, allowing clients, allowing clients, allowing us to very us to very us to very quickly iterate over quickly iterate over quickly iterate over instance sizes instance sizes instance sizes and available cache at and available cache at and available cache at different levels; this different levels; this different levels; this yielded a lot of great yielded a lot of great yielded a lot of great data. As I said, data. As I said, data. As I said, all this was done all this was done all this was done on version 0.8.6. I think on version 0.8.6. I think version 0.10 has already been released today. We had to We had to apply one apply one apply one fix, do a fix, do a fix, do a cherry-pick. But, again cherry-pick. But, again cherry-pick. But, again , keep in mind that this is the , keep in mind that this is the version that was tested. We ran version that was tested. We ran this on 21 different this on 21 different this on 21 different server configurations server configurations , from one to 29 , from one to 29 , from one to 29 clients, with two clients, with two clients, with two builds: the first was builds: the first was builds: the first was stackobot, a sample project stackobot, a sample project stackobot, a sample project from Epic for Unreal Engine. That's from Epic for Unreal Engine. That's from Epic for Unreal Engine. That's about 2000 files and about 2000 files and about 2000 files and about one about one about one gigabyte. Another was UHD gigabyte. Another was UHD content: content: content: multi-gigabyte multi-gigabyte multi-gigabyte files, mostly 10 files, mostly 10 files, mostly 10 gigabytes each, and a total of

gigabytes each, and a total of gigabytes each, and a total of about 105 gigabytes about 105 gigabytes about 105 gigabytes for import and for import and for import and migration. There are a migration. There are a migration. There are a few caveats here. few caveats here. few caveats here. All All All latency metrics are obtained in the latency metrics are obtained in the latency metrics are obtained in the cloud via NVPC. We cloud via NVPC. We cloud via NVPC. We wanted to eliminate the wanted to eliminate the wanted to eliminate the impact of client impact of client impact of client connections, like my connections, like my connections, like my slow home slow home slow home internet connection internet connection internet connection or the sometimes or the sometimes or the sometimes unstable unstable unstable connection in Rennes. So this is the connection in Rennes. So this is the connection in Rennes. So this is the minimum threshold, minimum threshold, minimum threshold, take it as it is. Um take it as it is. Um ...Now, regarding ...Now, regarding size classification. When we size classification. When we say "small," " say "small," " medium," or " medium," or " large," we large," we large," we mean this. We mean this. We mean this. We mean the size of the mean the size of the mean the size of the studio. Small is studio. Small is studio. Small is perhaps one perhaps one perhaps one to five people with to five people with to five people with a project up to 20 gigabytes in size a project up to 20 gigabytes in size a project up to 20 gigabytes in size . And with . And with . And with peak synchronous peak synchronous peak synchronous load for load for load for two or three two or three two or three users. You know, the users. You know, the users. You know, the average level is average level is average level is probably 20 to 40 probably 20 to 40 probably 20 to 40 people, somewhere between 50 and 200 people, somewhere between 50 and 200 people, somewhere between 50 and 200 gigabytes, and about gigabytes, and about gigabytes, and about 30 syncs 30 syncs 30 syncs simultaneously during that simultaneously during that simultaneously during that same same same Monday Monday Monday morning "storm."

morning "storm." You know, a large You know, a large level is hundreds of level is hundreds of level is hundreds of people at many people at many people at many other venues. These are other venues. These are other venues. These are simply our simply our simply our assumptions about how assumptions about how assumptions about how we determine the we determine the we determine the scale of the problem. So scale of the problem. So scale of the problem. So we have allocated a separate pool of resources for we have allocated a separate pool of resources for we have allocated a separate pool of resources for each of these each of these each of these sizes sizes sizes . The small . The small . The small tier is three tier is three tier is three clients on nine clients on nine clients on nine different different different write tier configurations. write tier configurations. Medium—that's about Medium—that's about 30 clients across 30 clients across 30 clients across multiple nodes and also multiple nodes and also multiple nodes and also eight different eight different eight different node configurations. node configurations. On a large scale, On a large scale, we are still waiting for we are still waiting for we are still waiting for approval to approval to approval to launch something launch something launch something this this this large. But in large. But in large. But in addition to that, we addition to that, we addition to that, we developed scripts developed scripts developed scripts to guide these to guide these to guide these tests. So: tests. So: tests. So: Monday's Monday's Monday's synchronization synchronization synchronization storm; connecting a storm; connecting a storm; connecting a new site to the new site to the new site to the cold storage facility; cold storage facility; cold storage facility; project import; project import; project import; daily receipt of daily receipt of daily receipt of minor edits with a " minor edits with a " canary" test, canary" test, canary" test, where one user where one user where one user works during works during works during processes; filling processes; filling processes; filling cold cold cold recording levels; and fast recording levels; and fast recording levels; and fast small commits.

small commits. Pre- Pre- heating and complex heating and complex heating and complex fusion testing still needs to be done. It is also important fusion testing still needs to be done. It is also important fusion testing still needs to be done. It is also important to remember that every to remember that every to remember that every file that was cloned file that was cloned file that was cloned and synchronized and synchronized and synchronized was verified was verified was verified outside of the Lore system. We did outside of the Lore system. We did outside of the Lore system. We did n't just trust n't just trust n't just trust successful completion successful completion successful completion from the client, but each from the client, but each from the client, but each file was fully file was fully independently verified. So we independently verified. So we could detect any could detect any data integrity issues. The first thing that's interesting here is that The first thing that's interesting here is that we deployed the we deployed the we deployed the write layer across six write layer across six write layer across six configurations—our configurations—our C8 GD and C9 GD compute instances of C8 GD and C9 GD compute instances of varying sizes. These are varying sizes. These are varying sizes. These are compute- compute- optimized optimized optimized instances for the instances for the instances for the write level, from one to write level, from one to write level, from one to eight virtual eight virtual eight virtual processors. But what's processors. But what's processors. But what's interesting here is that if interesting here is that if interesting here is that if you look at the you look at the you look at the top, we can see the top, we can see the top, we can see the import speed— import speed— how quickly we can how quickly we can how quickly we can add data to the add data to the add data to the repository. And, here you repository. And, here you repository. And, here you will see that the will see that the will see that the import speed import speed import speed directly directly directly depends on depends on depends on the number of available the number of available the number of available virtual virtual virtual processors on the processors on the processors on the servers. But if servers. But if servers. But if you look at the you look at the you look at the bottom, you know, as an bottom, you know, as an bottom, you know, as an artist or a artist or a artist or a game developer game developer game developer working with this working with this working with this repository, it's P50

repository, it's P50 repository, it's P50 for the small edits for the small edits for the small edits that we make. You that we make. You that we make. You will find that this almost will find that this almost will find that this almost never changes. never changes. never changes. So the size of the server So the size of the server So the size of the server doesn't really affect doesn't really affect doesn't really affect the speed of the speed of the speed of commits. This commits. This commits. This really really really only matters for large only matters for large only matters for large bulk imports. bulk imports. You know, this bottom You know, this bottom line—this is exactly what line—this is exactly what line—this is exactly what your artists and your artists and your artists and game developers game developers game developers work with every day. So work with every day. So work with every day. So it's still it's still it's still incredibly fast incredibly fast incredibly fast no matter no matter no matter what size server what size server what size server is used. is used. So, why is this line So, why is this line flat? This is because it is flat? This is because it is flat? This is because it is mostly mostly mostly just exchanging just exchanging just exchanging metadata with DynamoDB. metadata with DynamoDB. So, on one machine, So, on one machine, the same commit the same commit the same commit takes, say, 4 takes, say, 4 takes, say, 4 tenths of a second tenths of a second tenths of a second within the VPC. But large within the VPC. But large within the VPC. But large asset commits are asset commits are asset commits are another matter. The another matter. The workstation workstation has to break them has to break them has to break them up, hash, and up, hash, and up, hash, and compress them, so the compress them, so the client CPU performance sets the pace, which client CPU performance sets the pace, which we've seen we've seen we've seen is around is around is around 170 to 290 170 to 290 170 to 290 megabytes per megabytes per megabytes per second for two to second for two to eight client eight client eight client vCPUs. One run for vCPUs. One run for vCPUs. One run for each match, like a each match, like a each match, like a direction, direction, direction, is not really a measure of is not really a measure of is not really a measure of speed. So, there are speed. So, there are speed. So, there are two modes here.

two modes here. Increasing the Increasing the recording level gives an advantage recording level gives an advantage recording level gives an advantage during the first import during the first import during the first import and possibly during " and possibly during " cold" hydration. cold" hydration. cold" hydration. This does not ensure This does not ensure This does not ensure faster execution of faster execution of faster execution of daily commits. I daily commits. I daily commits. I think this was a pretty think this was a pretty think this was a pretty interesting conclusion from interesting conclusion from interesting conclusion from this testing. this testing. Now regarding the edge pods, I would say that edge pods, I would say that this is a significant feature of the this is a significant feature of the this is a significant feature of the system if system if system if you ask me. This you ask me. This you ask me. This really helps really helps really helps to spread the to spread the to spread the load. They load. They load. They can set up can set up can set up hydration with hydration with hydration with the server and bring the server and bring the server and bring data closer to users. data closer to users. But, except for a few But, except for a few points, the size of points, the size of points, the size of the module is virtually the module is virtually the module is virtually imperceptible. With a imperceptible. With a imperceptible. With a steady stream of steady stream of steady stream of small edits... small edits... The slowest 1% of all The slowest 1% of all calls took calls took calls took approximately 170 approximately 170 approximately 170 milliseconds on milliseconds on milliseconds on each module with each module with each module with two or more vCPUs. two or more vCPUs. The option with one vCPU, as The option with one vCPU, as you can see, is not suitable.

you can see, is not suitable. So for both the write layer and modules, we recommend modules, we recommend using using using at least two vCPUs or at least two vCPUs or at least two vCPUs or more. We encountered more. We encountered more. We encountered issues when issues when issues when using using using single single single vCPU instances. Um, but then, vCPU instances. Um, but then, vCPU instances. Um, but then, you know, when we you know, when we you know, when we run our run our run our sync sync sync storms, we were looking storms, we were looking storms, we were looking at, you know, the at, you know, the at, you know, the shape of when, when shape of when, when shape of when, when exactly do you want to exactly do you want to exactly do you want to scale up these pods, scale up these pods, scale up these pods, you know, these you know, these you know, these peripheral pods? peripheral pods? You know, this is something I would You know, this is something I would really watch on the really watch on the really watch on the dashboard: as soon as, um, dashboard: as soon as, um, dashboard: as soon as, um, during a during a during a sync sync sync storm or a big storm or a big storm or a big import, you know, import, you know, import, you know, as soon as the outbound as soon as the outbound as soon as the outbound traffic of the pod levels traffic of the pod levels traffic of the pod levels out, you know, out, you know, out, you know, the data going, um, the data going, um, the data going, um, upstream upstream upstream becomes flat, becomes flat, becomes flat, you know, and the processor is you know, and the processor is you know, and the processor is actually at full load at that point actually at full load at that point . That's when, um, it . That's when, um, it makes sense makes sense makes sense to scale them if to scale them if to scale them if you need more you need more system bandwidth. So, this is actually system bandwidth. So, this is actually again again again a function, you know, a function, you know, a function, you know, for these different types of for these different types of for these different types of pods; this is the kind of pods; this is the kind of pods; this is the kind of performance you performance you performance you can expect from the can expect from the can expect from the system. So, system. So, system. So, what should actually what should actually what should actually be on the dashboard?

be on the dashboard? Well, I think specifically Well, I think specifically here, for this here, for this here, for this case, this is where, um, case, this is where, um, case, this is where, um, with CloudWatch logs, with CloudWatch logs, with CloudWatch logs, CloudWatch agent on CloudWatch agent on CloudWatch agent on pods or at the pods or at the pods or at the write level. Um, it write level. Um, it write level. Um, it actually, you know, actually, you know, actually, you know, collects CPU and collects CPU and collects CPU and network data every, say network data every, say , 10 seconds and just , 10 seconds and just , 10 seconds and just signals our signals our signals our ENA counter. You know, ENA counter. You know, ENA counter. You know, exceeding the exceeding the exceeding the allowed allowed allowed bandwidth bandwidth bandwidth per output. So once per output. So once per output. So once you reach you reach you reach that, you know, that's when it makes that, you know, that's when it makes that, you know, that's when it makes sense to move up sense to move up sense to move up to a larger to a larger to a larger pan size. So another pan size. So another pan size. So another point that I point that I point that I think is interesting here is, think is interesting here is, think is interesting here is, um, when we um, when we um, when we look at why, you look at why, you look at why, you know, I think know, I think know, I think most would most would most would deploy this, you deploy this, you deploy this, you really really really only need the recording layer only need the recording layer only need the recording layer to make it work, and the to make it work, and the to make it work, and the peripheral pods peripheral pods peripheral pods could could could just be considered an additional just be considered an additional just be considered an additional cost. But, cost. But, cost. But, you know, when would you you know, when would you you know, when would you want to deploy want to deploy want to deploy peripheral pods? There are peripheral pods? There are peripheral pods? There are several reasons for several reasons for several reasons for using them: you know, using them: you know, using them: you know, the first one is really the first one is really the first one is really local. And, local. And, local. And, you know, part of this you know, part of this you know, part of this lore bench setup lore bench setup lore bench setup is that I is that I is that I had peripheral pods, had peripheral pods, had peripheral pods, and, you know, in, in, um, on the and, you know, in, in, um, on the and, you know, in, in, um, on the other side of the world, other side of the world, other side of the world, actually. Let's say the actually. Let's say the actually. Let's say the record level, everything record level, everything record level, everything was hosted in the was hosted in the was hosted in the US East2 instance, and, US East2 instance, and, US East2 instance, and, you know, in a bucket, you know, in a bucket, you know, in a bucket, DynamoDB tables, but DynamoDB tables, but DynamoDB tables, but then I set up the then I set up the then I set up the edge pods, um, edge pods, um, edge pods, um, globally, um, to, well, globally, um, to, well, globally, um, to, well, test it. And that's where test it. And that's where test it. And that's where , um, you'd , um, you'd , um, you'd want to add

want to add want to add peripheral pods for peripheral pods for peripheral pods for locality, first of all locality, first of all , and the other reason is, , and the other reason is, , and the other reason is, of course, large of course, large of course, large synchronizations. Another synchronizations. Another synchronizations. Another criterion for criterion for any of the any of the any of the repository servers is that repository servers is that repository servers is that these instances have an NVMe these instances have an NVMe these instances have an NVMe instance store. They are instance store. They are instance store. They are equipped with equipped with equipped with NVMe drives. And NVMe drives. And NVMe drives. And this is quite important, this is quite important, this is quite important, because this is one of the because this is one of the because this is one of the main reasons for the main reasons for the main reasons for the high speed of high speed of high speed of this this this version control system— version control system— the use of very the use of very the use of very fast memory on fast memory on fast memory on these cache nodes. So, these cache nodes. So, these cache nodes. So, considering these two considering these two considering these two options, when directly options, when directly options, when directly transitioning to the transitioning to the transitioning to the desired level with desired level with desired level with large imports, large imports, large imports, this is where this is where this is where problems arise. Again problems arise. Again problems arise. Again , I'll , I'll , I'll share the solution soon. share the solution soon. But you'll see that But you'll see that once you start once you start once you start looking at the looking at the looking at the number of customers number of customers number of customers using a using a using a particular pod, 14 or 15 particular pod, 14 or 15 particular pod, 14 or 15 active active active users—that's the users—that's the users—that's the "magic number" "magic number" "magic number" that we're seeing.

that we're seeing. Once you Once you have that have that have that many many many users in a users in a users in a given location, given location, given location, the use of the use of the use of peripheral pods peripheral pods peripheral pods becomes really becomes really becomes really justified. And for justified. And for justified. And for one to five people, one to five people, one to five people, peripheral pods peripheral pods peripheral pods may not be a may not be a may not be a necessity. Again necessity. Again necessity. Again , they , they , they provide some provide some provide some buffering for buffering for buffering for large imports large imports large imports etc., but they etc., but they etc., but they can also be started and can also be started and can also be started and stopped as needed. stopped as needed. stopped as needed. So in reality, So in reality, So in reality, you're not buying the first pod you're not buying the first pod you're not buying the first pod for the sake of parallelism for the sake of parallelism . I tested this with . I tested this with . I tested this with about 29-30 about 29-30 about 29-30 clients. But no clients. But no clients. But no pod worked pod worked pod worked faster than a direct faster than a direct faster than a direct connection. It can connection. It can connection. It can accept large accept large accept large imports to imports to imports to place the cache next place the cache next place the cache next to the readers, to the readers, to the readers, for example for a CI for example for a CI system. But when you system. But when you system. But when you run pods, run pods, run pods, try to try to try to maintain a maintain a maintain a synchronized synchronized synchronized load of load of load of about 15 about 15 about 15 clients per clients per clients per hot pod. And here is the hot pod. And here is the hot pod. And here is the bill we bill we bill we see for one month of see for one month of see for one month of using US East 2 on using US East 2 on using US East 2 on demand. Here, as you demand. Here, as you demand. Here, as you can see, most of the can see, most of the can see, most of the costs are on costs are on costs are on computing computing computing resources. Storage costs resources. Storage costs resources. Storage costs , i.e. S3 and , i.e. S3 and , i.e. S3 and DynamoDB for a 20 DynamoDB for a 20 gigabyte gigabyte gigabyte repository, repository, repository, are only a small are only a small are only a small fraction of this. So, fraction of this. So, fraction of this. So, to get started,

to get started, to get started, for 1-5 people and 20 GB, a for 1-5 people and 20 GB, a for 1-5 people and 20 GB, a C 8 g.large instance for the C 8 g.large instance for the C 8 g.large instance for the repository tier repository tier repository tier will cost around will cost around will cost around $77 per month. $77 per month. For a slightly larger For a slightly larger volume, say with a volume, say with a volume, say with a pair of peripheral pair of peripheral pair of peripheral pods and a C 8 g.xlarge. pods and a C 8 g.xlarge. the right level, uh, you the right level, uh, you know, the cost know, the cost know, the cost will go up to about will go up to about will go up to about $312 a month $312 a month $312 a month for that. And then, you know, for that. And then, you know, for that. And then, you know, eventually, for, say, eventually, for, say, workload-intensive teams, where workload-intensive teams, where you have larger pods and you have larger pods and you have larger pods and significantly more significantly more significantly more distributed distributed distributed resources. So here's the resources. So here's the resources. So here's the bottom line—it's bottom line—it's bottom line—it's a little less than a little less than a little less than $500 a month for $500 a month for $500 a month for this. this. So is $77 the So is $77 the whole bill for a whole bill for a whole bill for a small team? small team? small team? Good question about Good question about Good question about calculations and backend. calculations and backend. calculations and backend. Yes. But you know what Yes. But you know what Yes. But you know what I didn't include here, I didn't include here, I didn't include here, just for the sake of just for the sake of just for the sake of simplicity and because of the simplicity and because of the simplicity and because of the different different different deployment options that deployment options that deployment options that everyone has, is: I didn't factor in everyone has, is: I didn't factor in everyone has, is: I didn't factor in VPC, NAT gateways, or VPC, NAT gateways, or VPC, NAT gateways, or CloudWatch costs, CloudWatch costs, CloudWatch costs, depending on depending on depending on how detailed the how detailed the how detailed the logs are that you want to have.

logs are that you want to have. So, there's also, let's say, So, there's also, let's say, about $40-50 about $40-50 about $40-50 a month, you know, for a month, you know, for a month, you know, for things like Site-to-Site things like Site-to-Site things like Site-to-Site VPN and some other VPN and some other VPN and some other related services. But related services. But related services. But again, I think the again, I think the again, I think the main point here is main point here is just to show that from a just to show that from a just to show that from a storage perspective, it's storage perspective, it's , you know, just a fraction of the , you know, just a fraction of the , you know, just a fraction of the cost of computing. And cost of computing. And cost of computing. And the calculations depend the calculations depend the calculations depend on how much on how much on how much data needs to be data needs to be data needs to be moved into moved into moved into and out of the system at a and out of the system at a and out of the system at a given point in time. given point in time. given point in time. So there is some So there is some So there is some elasticity here. Now, elasticity here. Now, elasticity here. Now, a few a few a few precautions we precautions we precautions we learned through Lore learned through Lore learned through Lore Bench and testing. Bench and testing. Again, this is pre- Again, this is pre- 1.0, and we 1.0, and we 1.0, and we discovered all of this discovered all of this discovered all of this during targeted during targeted during targeted stress testing before stress testing before stress testing before any of you could any of you could any of you could encounter it. So, encounter it. So, you know, don't you know, don't use use use single single single vCPU configurations for anything. vCPU configurations for anything. Any level or sub, Any level or sub, Any level or sub, they are slower, and there they are slower, and there they are slower, and there are some are some reliability issues. So two reliability issues. So two vCPUs or more, and vCPUs or more, and vCPUs or more, and use use use NVMe instance storage NVMe instance storage NVMe instance storage to maintain to maintain to maintain speed. Now, regarding the speed. Now, regarding the speed. Now, regarding the cold front: cold front: cold front: connect no connect no connect no more than five clients more than five clients more than five clients to our workstations at a time to our workstations at a time to our workstations at a time . Keep . Keep . Keep in mind that a cold in mind that a cold in mind that a cold pod has a completely pod has a completely pod has a completely empty cache. And you empty cache. And you empty cache. And you probably saw it in probably saw it in probably saw it in one of the scenarios in the one of the scenarios in the one of the scenarios in the original list.

original list. original list. This is part of what This is part of what This is part of what Lore Bench does to Lore Bench does to Lore Bench does to help solve the help solve the help solve the pre- pre- pre- heating issue. How can heating issue. How can heating issue. How can these caches, say, be these caches, say, be these caches, say, be pre-warmed pre-warmed pre-warmed before that before that before that Monday Monday Monday synchronization storm? Third, synchronization storm? Third, synchronization storm? Third, you know, check, you know, check, you know, check, be sure be sure be sure to check after to check after to check after each cloning each cloning each cloning and synchronization. This is and synchronization. This is and synchronization. This is part of the Lore Bench's work part of the Lore Bench's work —this very check. —this very check. —this very check. You can check the You can check the You can check the expected number of expected number of expected number of files, but, you know, it's files, but, you know, it's files, but, you know, it's worth making sure worth making sure worth making sure that there are no temporary that there are no temporary that there are no temporary files left. files left. And again, that's And again, that's why we built why we built why we built Lorebench exactly this way, to Lorebench exactly this way, to Lorebench exactly this way, to get an independent, get an independent, get an independent, separately verified separately verified separately verified answer: "yes," all answer: "yes," all answer: "yes," all syncs, clones, syncs, clones, syncs, clones, and commits are completely and commits are completely and commits are completely intact. And finally: if intact. And finally: if intact. And finally: if you are setting up you are setting up you are setting up an import of more than an import of more than an import of more than 50 gigabytes, then 50 gigabytes, then 50 gigabytes, then it works it works it works much better through a pod than much better through a pod than much better through a pod than directly. I directly. I directly. I mean, it's just a mean, it's just a mean, it's just a certain amount of stock. This is not a certain amount of stock. This is not a certain amount of stock. This is not a measured threshold, but measured threshold, but measured threshold, but as we see now, it is an as we see now, it is an as we see now, it is an approximate limit for approximate limit for approximate limit for large imports. This is large imports. This is large imports. This is probably the probably the probably the most important slide most important slide most important slide if you are new to if you are new to if you are new to using Lore on using Lore on using Lore on AWS. Here are some things AWS. Here are some things AWS. Here are some things we came across that are we came across that are we came across that are worth worth worth keeping in mind. Remember, keeping in mind. Remember,

keeping in mind. Remember, in the previous in the previous in the previous slide, the large slide, the large slide, the large import to the "right" level import to the "right" level import to the "right" level did not complete due to did not complete due to S3 speed limits? S3 processes about S3 speed limits? S3 processes about 3500 records per second 3500 records per second 3500 records per second per partition for a per partition for a per partition for a brand new brand new brand new bucket in a new bucket in a new bucket in a new account. So when we account. So when we account. So when we tried to tried to tried to download those 105 download those 105 download those 105 gigabytes, which gigabytes, which gigabytes, which consist of consist of consist of many small many small many small fragments, the fragments, the fragments, the speed limit started speed limit started speed limit started and some and some and some problems arose. What problems arose. What problems arose. What partitioning does—and S3 partitioning does—and S3 partitioning does—and S3 does it does it does it automatically at the automatically at the automatically at the system level—is the system level—is the system level—is the ability to separate ability to separate ability to separate data based on data based on data based on file name. Since these are file name. Since these are file name. Since these are hashes that are fairly hashes that are fairly hashes that are fairly random, and random, and random, and file names are essentially file names are essentially file names are essentially hexadecimal hexadecimal hexadecimal values, you can values, you can values, you can quickly send quickly send quickly send a request to support a request to support asking them to pre- asking them to pre- partition the trash by the partition the trash by the partition the trash by the first character of first character of first character of the name. This allows you the name. This allows you the name. This allows you to increase the speed to increase the speed to increase the speed from 3,500 write operations from 3,500 write operations from 3,500 write operations per second to 56,000, and per second to 56,000, and per second to 56,000, and the separation by the first the separation by the first the separation by the first character is quite character is quite character is quite sufficient for the systems, sufficient for the systems, sufficient for the systems, at least for the at least for the at least for the loads we loads we loads we tested. One more tested. One more tested. One more point: DynamoDB. A fresh point: DynamoDB. A fresh point: DynamoDB. A fresh pay-per- pay-per- pay-per- query table processes query table processes query table processes about 4,000 records per about 4,000 records per about 4,000 records per second before second before second before scaling. And with scaling. And with scaling. And with that 105GB import,

that 105GB import, that 105GB import, both both both fragment tables still fragment tables still fragment tables still hit the hit the hit the speed limit. speed limit. So, keep in mind So, keep in mind that this is 8.6 tables, and from what that this is 8.6 tables, and from what that this is 8.6 tables, and from what I heard from Epic, some I heard from Epic, some I heard from Epic, some of this will also be of this will also be of this will also be migrated to migrated to migrated to metadata in S3. But, metadata in S3. But, metadata in S3. But, you know, if you look you know, if you look you know, if you look at it as an at it as an at it as an average per minute, average per minute, average per minute, it will hide the spikes in it will hide the spikes in it will hide the spikes in load. So you load. So you load. So you can can can preheat the tables, preheat the tables, preheat the tables, if you're doing a if you're doing a if you're doing a large import, you large import, you large import, you can preheat the can preheat the can preheat the DynamoDB tables to DynamoDB tables to DynamoDB tables to maintain those maintain those maintain those per second rate limits per second rate limits per second rate limits . This can be . This can be . This can be easily done easily done easily done using the command using the command using the command line for line for preheating. Our cloud game development toolkit module cloud game development toolkit module does not yet have this does not yet have this does not yet have this feature. We are trying feature. We are trying feature. We are trying to decide exactly how to to decide exactly how to to decide exactly how to implement this. Again implement this. Again implement this. Again , if you , if you , if you 're working with StackOBot or 're working with StackOBot or 're working with StackOBot or under 50 under 50 under 50 gigabytes, you gigabytes, you gigabytes, you don't need to worry about this yet don't need to worry about this yet don't need to worry about this yet . This . This . This only applies to very only applies to very only applies to very large imports large imports large imports when you need to when you need to when you need to work work work as quickly as possible.

as quickly as possible. The last thing to The last thing to mention, and this mention, and this mention, and this brings us back to the brings us back to the brings us back to the QUIC protocol, is that QUIC protocol, is that QUIC protocol, is that using the VPC using the VPC resolver for DNS resolver for DNS resolver for DNS is limited to 1024 is limited to 1024 is limited to 1024 packets per second packets per second packets per second per network per network per network interface. So, interface. So, interface. So, the client itself, because it the client itself, because it the client itself, because it requests so many requests so many requests so many different fragments different fragments different fragments across multiple threads, across multiple threads, across multiple threads, may encounter may encounter DNS rate throttling. Therefore, caching DNS rate throttling. Therefore, caching is necessary on is necessary on is necessary on each client side. This is a each client side. This is a each client side. This is a standard feature on standard feature on standard feature on Linux and Mac, but for Windows, Linux and Mac, but for Windows, Linux and Mac, but for Windows, make sure you make sure you make sure you enable enable enable DNS caching to avoid DNS caching to avoid DNS caching to avoid problems. problems. So, if a team of So, if a team of five five five is trying is trying is trying to set this up to set this up to set this up tomorrow, what should they enter tomorrow, what should they enter tomorrow, what should they enter into the Terraform variables? There into the Terraform variables? There are only three are only three lines here that lines here that lines here that matter. The main one matter. The main one matter. The main one is the is the is the write tier variable, and a write tier variable, and a write tier variable, and a good place to start would be the good place to start would be the good place to start would be the instance type c 8 g.dl.large.

instance type c 8 g.dl.large. This provides a good This provides a good base for a small base for a small base for a small group at a price of group at a price of group at a price of around $77 per around $77 per around $77 per month. And then, I think, month. And then, I think, when we when we when we scale to 20–40 scale to 20–40 scale to 20–40 people, where the data volume people, where the data volume people, where the data volume is from 50 to 200 is from 50 to 200 is from 50 to 200 gigabytes. This is where gigabytes. This is where gigabytes. This is where you define several you define several you define several pods: two pods with 8 pods: two pods with 8 pods: two pods with 8 g.dl.large, and then with 8 g.dl.xl, g.dl.large, and then with 8 g.dl.xl, g.dl.large, and then with 8 g.dl.xl, which costs about which costs about which costs about $311 per month. Um, $311 per month. Um, $311 per month. Um, you know, and of course, you know, and of course, you know, and of course, the signals to the signals to the signals to watch for are those watch for are those watch for are those 10-second metrics 10-second metrics 10-second metrics for the pod processor. for the pod processor. If it is If it is 100% loaded and the 100% loaded and the 100% loaded and the outgoing traffic levels outgoing traffic levels outgoing traffic levels out during the out during the out during the morning morning morning sync, then as soon as sync, then as soon as sync, then as soon as that happens, that happens, that happens, you replace the pods with the you replace the pods with the you replace the pods with the XL version, one at a XL version, one at a XL version, one at a time. And both of these lines time. And both of these lines time. And both of these lines use use use Graviton dimensions. Therefore, Graviton dimensions. Therefore, Graviton dimensions. Therefore, you will need to you will need to you will need to create a create a create a server image for ARM64. And server image for ARM64. And server image for ARM64. And the module the module the module will only warn if there will only warn if there will only warn if there is a discrepancy.

is a discrepancy. is a discrepancy. Yes, these are ARM cores, you know Yes, these are ARM cores, you know , ARM64 instances that are , ARM64 instances that are , ARM64 instances that are being used. being used. And 100 people is a And 100 people is a big studio. We have big studio. We have big studio. We have n't built this n't built this n't built this option yet, as option yet, as option yet, as Ben mentioned. So this line is Ben mentioned. So this line is Ben mentioned. So this line is just a prediction just a prediction just a prediction based on our based on our based on our measurements. You should measurements. You should measurements. You should keep about 15 keep about 15 keep about 15 syncs running syncs running syncs running simultaneously per simultaneously per simultaneously per warm pod, 7 warm pod, 7 warm pod, 7 compute- compute- optimized optimized optimized XL pods per 100 people, and XL pods per 100 people, and XL pods per 100 people, and at least level 2XL at least level 2XL at least level 2XL for import and CI. That's for import and CI. That's for import and CI. That's about $1,500 about $1,500 about $1,500 per month, according to per month, according to per month, according to estimates, again estimates, again estimates, again depending on depending on depending on your region, for up your region, for up your region, for up to a terabyte. to a terabyte. to a terabyte. Each subsequent hundred Each subsequent hundred Each subsequent hundred synchronizations adds synchronizations adds synchronizations adds about seven pods, about seven pods, about seven pods, which is which is which is about another thousand about another thousand about another thousand dollars. Our concurrency test dollars. Our concurrency test stopped at 29 stopped at 29 clients, and two clients, and two clients, and two questions remain questions remain questions remain open. A open. A open. A multi-terabyte multi-terabyte multi-terabyte repository is repository is repository is outgrowing the local outgrowing the local outgrowing the local disk of write levels, and disk of write levels, and disk of write levels, and 50+ CI agents 50+ CI agents 50+ CI agents are creating a second are creating a second are creating a second morning morning morning load peak. load peak.

Multisite is Multisite is something we something we something we will work on will work on will work on later, and one later, and one later, and one module call per module call per module call per region creates region creates region creates separate repositories. separate repositories. separate repositories. None of these lines None of these lines None of these lines include your include your include your AWS link, outbound traffic, AWS link, outbound traffic, AWS link, outbound traffic, or other or other or other infrastructure. infrastructure. infrastructure. Again, around Again, around Again, around $40–50 per $40–50 per $40–50 per month. And for the month. And for the month. And for the NAT gateway disks and logs, for the NAT gateway disks and logs, for the NAT gateway disks and logs, for the first line you first line you first line you will also need a DNS will also need a DNS endpoint, that's another endpoint, that's another endpoint, that's another $200 or so. $200 or so. $200 or so. We will publish the exact We will publish the exact We will publish the exact Terraform configurations with Terraform configurations with Terraform configurations with our our our sizing information. sizing information. sizing information. If you want to If you want to If you want to compare our numbers compare our numbers compare our numbers to your own to your own to your own deployment, we deployment, we deployment, we are working on are working on are working on releasing this releasing this releasing this tool. It's tool. It's tool. It's not quite ready yet, not quite ready yet, not quite ready yet, but here's a link but here's a link but here's a link to Laura Bench in the middle. to Laura Bench in the middle. Actually, that's what we Actually, that's what we created it for—to created it for—to created it for—to really help really help really help provide provide provide stability and stability and stability and answer some of answer some of answer some of these questions. This is a very these questions. This is a very these questions. This is a very new system, and Laura Bench new system, and Laura Bench new system, and Laura Bench can help can help can help answer many of answer many of answer many of these questions. So these questions. So these questions. So use it use it use it to simulate a " to simulate a " sync storm" sync storm" sync storm" or a wave of onboarding, or a wave of onboarding, or a wave of onboarding, large imports, etc.

large imports, etc. You know, as soon as it You know, as soon as it comes out, it will be comes out, it will be comes out, it will be announced on our announced on our announced on our blog. The link there is blog. The link there is the first in this series. the first in this series. the first in this series. There will be even more There will be even more There will be even more information. Of course, information. Of course, information. Of course, there you will find there you will find there you will find links to links to links to Epic's Lore itself, as well as to the Epic's Lore itself, as well as to the cloud cloud game development toolkit. When it is released, game development toolkit. When it is released, game development toolkit. When it is released, we recommend deploying we recommend deploying we recommend deploying it to your own it to your own it to your own AWS account AWS account AWS account and comparing it to our and comparing it to our and comparing it to our tables. These modules tables. These modules tables. These modules are in this are in this cloud cloud game development toolkit. You can game development toolkit. You can game development toolkit. You can deploy it, deploy it, deploy it, report report report issues, and issues, and issues, and submit submit submit change requests to change requests to change requests to help us help us help us close gaps. close gaps. close gaps. Yes, great. I would like to Yes, great. I would like to Yes, great. I would like to add one more add one more add one more request before we request before we request before we wrap up: we would wrap up: we would wrap up: we would really like to see really like to see really like to see your your your churn data and your churn data and your churn data and your own experience of own experience of own experience of how much your how much your how much your repository repository repository changes per day. changes per day. changes per day. Who exactly is making Who exactly is making Who exactly is making the changes? These indicators of the changes? These indicators of the changes? These indicators of change and role, I would change and role, I would change and role, I would say, are the least say, are the least say, are the least valid valid valid inputs in inputs in inputs in our model. We've our model. We've our model. We've worked with many worked with many worked with many different game different game different game studios, but overall, studios, but overall, studios, but overall, getting more getting more getting more data on data on data on volume estimation would definitely volume estimation would definitely volume estimation would definitely make these models make these models make these models better. So, as you've better. So, as you've better. So, as you've heard from us heard from us heard from us several times, Lore several times, Lore several times, Lore versioned chunks, versioned chunks, versioned chunks, not files. This is a not files. This is a not files. This is a conscious conscious conscious architectural choice. So architectural choice. So architectural choice. So let's discuss let's discuss let's discuss three main takeaways three main takeaways three main takeaways from this design from this design from this design decision.

decision. The first is that each cache contains that each cache contains that each cache contains parts, not entire parts, not entire parts, not entire versions. The working node versions. The working node versions. The working node already has most of the already has most of the already has most of the current current current files, and the inactive files, and the inactive files, and the inactive node is simply node is simply node is simply filling up. So filling up. So filling up. So nothing recorded is nothing recorded is nothing recorded is lost. Second lost. Second : the new site or : the new site or : the new site or CI agent will CI agent will CI agent will only download the parts only download the parts only download the parts it needs to it needs to it needs to work. So, it is work. So, it is work. So, it is useful right useful right useful right after the first after the first after the first sync. sync. And lastly: the And lastly: the storage capacity only grows storage capacity only grows storage capacity only grows in accordance with the in accordance with the in accordance with the changes made. Each changes made. Each changes made. Each part is stored part is stored part is stored only once. Therefore, it is only once. Therefore, it is worth remembering that this worth remembering that this data can be cleared. data can be cleared. data can be cleared. But this is a completely manual But this is a completely manual But this is a completely manual operation. Storage operation. Storage operation. Storage can can can grow continuously if grow continuously if grow continuously if not cleaned up, but as you not cleaned up, but as you not cleaned up, but as you have seen, this is only have seen, this is only have seen, this is only a fraction of the total a fraction of the total a fraction of the total cost, while cost, while compute costs make up the compute costs make up the bulk. bulk. Thank you very much to everyone who Thank you very much to everyone who joined us joined us joined us today. today. We really appreciate We really appreciate your time and are willing your time and are willing your time and are willing to stay for a couple more to stay for a couple more to stay for a couple more minutes if you have any minutes if you have any minutes if you have any questions. So, Michael.

questions. So, Michael. Hello, you're just in Hello, you're just in time. We have time. We have time. We have about 5 minutes for about 5 minutes for about 5 minutes for questions, maybe questions, maybe questions, maybe a little less, but that's a little less, but that's a little less, but that's ideal. Many ideal. Many ideal. Many thanks for your thanks for your thanks for your talk and performance. talk and performance. talk and performance. So, I know Daniel is excited So, I know Daniel is excited So, I know Daniel is excited about about about it too. it too. it too. Yes, thank you. Yes, Yes, thank you. Yes, Yes, thank you. Yes, thank you very much, Rene and Ben. thank you very much, Rene and Ben. thank you very much, Rene and Ben. This was a very interesting This was a very interesting This was a very interesting session. We saw session. We saw session. We saw a lot of interest in Lore a lot of interest in Lore when it was when it was when it was first announced at first announced at first announced at Unreal Fest earlier Unreal Fest earlier Unreal Fest earlier this year, and we've already this year, and we've already this year, and we've already started started started experimenting with experimenting with experimenting with how how how integration integration integration between Lore and TeamCity could work through a between Lore and TeamCity could work through a between Lore and TeamCity could work through a custom plugin. custom plugin. So let's see, So let's see, stay tuned stay tuned stay tuned to to to see what we see what we see what we add there. But yes, we add there. But yes, we add there. But yes, we got some really got some really got some really good comments in the good comments in the good comments in the chat, and yes, kudos for chat, and yes, kudos for chat, and yes, kudos for all the simulations you provided all the simulations you provided all the simulations you provided . . Yes, great. I Yes, great. I mean, this is, again, mean, this is, again, mean, this is, again, brand new, brand new, brand new, and you know, we and you know, we and you know, we tried tried tried to take an approach to take an approach to take an approach that was honest and, I would that was honest and, I would that was honest and, I would say, transparent about how say, transparent about how say, transparent about how it works.

it works. it works. AWS and I are working with Epic to AWS and I are working with Epic to AWS and I are working with Epic to continuously continuously continuously improve Lore. I improve Lore. I improve Lore. I think there's think there's think there's a lot of great stuff ahead, but a lot of great stuff ahead, but a lot of great stuff ahead, but it's a fully modern it's a fully modern version control system that works version control system that works more like a more like a data storage software stack than data storage software stack than anything else. So, anything else. So, anything else. So, and given that this is a and given that this is a and given that this is a preliminary version, do preliminary version, do preliminary version, do you see any you see any you see any concerns about this concerns about this concerns about this from studios, or do from studios, or do from studios, or do you see them you see them you see them already using already using already using it in production? it in production? it in production? I think everyone I think everyone I think everyone is taking a is taking a is taking a wait-and-see approach wait-and-see approach , but many are already , but many are already , but many are already deploying it and deploying it and deploying it and starting starting starting to use it. And we to use it. And we to use it. And we will see. I think the will see. I think the will see. I think the first users will appear first users will appear first users will appear , I'll just , I'll just , I'll just say so. You know, say so. You know, say so. You know, there are still some parts of there are still some parts of there are still some parts of this ecosystem that this ecosystem that this ecosystem that I think I think I think need to be need to be need to be refined. Right now refined. Right now refined. Right now it's a CLI-only interface it's a CLI-only interface it's a CLI-only interface , but you can , but you can , but you can imagine how this would imagine how this would imagine how this would integrate with integrate with integrate with asset managers asset managers asset managers and things like that, and things like that, and things like that, it would help a lot. But it would help a lot. But it would help a lot. But again, I think again, I think again, I think we'll probably start to we'll probably start to we'll probably start to see some good see some good see some good implementations.

implementations. implementations. Honestly, my prediction is Honestly, my prediction is within the next within the next within the next three to six months. three to six months. three to six months. I agree. I think we're I agree. I think we're I agree. I think we're seeing a lot of seeing a lot of seeing a lot of promising numbers promising numbers . And you saw in our . And you saw in our . And you saw in our presentation that Epic is presentation that Epic is presentation that Epic is very actively very actively very actively developing and developing and developing and improving these things. improving these things. improving these things. Something Something Something has even changed since has even changed since we first collected we first collected we first collected all this data. all this data. Yes. Perfectly. Yes. Yes. Perfectly. Yes. Thank you. Thank you. Thank you. Yes. Sorry, Michael. Yes. Sorry, Michael. Yes. Sorry, Michael. We have a few questions We have a few questions We have a few questions in the chat. in the chat. in the chat. Yes, we have another Yes, we have another Yes, we have another question here. Um, question here. Um, question here. Um, one expert said one expert said one expert said we were surprised we were surprised we were surprised that Lore is that Lore is that Lore is open source technology and open source technology and open source technology and there are no there are no there are no additional additional additional licensing costs, licensing costs, licensing costs, which is a great bonus which is a great bonus which is a great bonus for independent for independent for independent teams. You mentioned a little bit at teams. You mentioned a little bit at teams. You mentioned a little bit at the end the end the end about the proposed about the proposed about the proposed configuration for configuration for configuration for teams of different teams of different teams of different sizes. What else are you sizes. What else are you sizes. What else are you thinking about the thinking about the thinking about the value of Lore for value of Lore for value of Lore for different teams as you different teams as you different teams as you continue this continue this continue this research? I would research? I would research? I would say that there was say that there was say that there was one one one thing missing that thing missing that thing missing that needed to be implemented needed to be implemented — it was actually a — it was actually a — it was actually a separate instance on a separate instance on a separate instance on a single server. It is a single server. It is a single server. It is a very modern cloud very modern cloud very modern cloud architecture that architecture that architecture that uses Amazon S3 uses Amazon S3 uses Amazon S3 for reliability, as well for reliability, as well for reliability, as well as Dynamo DB for as Dynamo DB for as Dynamo DB for scalability. But scalability. But scalability. But in general you will see in general you will see in general you will see that it can be run that it can be run that it can be run on a single instance, on a single instance, on a single instance, however I think the however I think the performance results performance results will be quite different.

will be quite different. And such a separate And such a separate instance with an instance with an instance with an NVMe drive NVMe drive NVMe drive can be run on can be run on can be run on your own equipment, your own equipment, your own equipment, at your own location. But at your own location. But at your own location. But all the reasons all the reasons all the reasons people use people use people use S3, like reliability, S3, like reliability, S3, like reliability, deletion protection, deletion protection, deletion protection, etc.—that's where it etc.—that's where it etc.—that's where it scales really scales really scales really well. So I believe well. So I believe well. So I believe this is a part of the lorebench this is a part of the lorebench this is a part of the lorebench that also needs to be that also needs to be that also needs to be released to released to released to understand what it understand what it understand what it looks like as a standalone looks like as a standalone looks like as a standalone instance. But I instance. But I instance. But I think you'll find that think you'll find that think you'll find that this architecture this architecture scales incredibly well for a scales incredibly well for a distributed team distributed team distributed team or a distributed or a distributed or a distributed studio. studio. Yes. It's a very deep Yes. It's a very deep understanding, and as understanding, and as understanding, and as you said, it's early you said, it's early you said, it's early technology, so we're technology, so we're technology, so we're constantly learning, and constantly learning, and constantly learning, and everything around us is constantly everything around us is constantly everything around us is constantly changing. So it's changing. So it's changing. So it's great to see great to see great to see people getting excited about it people getting excited about it people getting excited about it . I will say that we did not . I will say that we did not . I will say that we did not find any find any find any flaws in the flaws in the flaws in the fundamental fundamental fundamental architecture of AR. Again architecture of AR. Again architecture of AR. Again , this is essentially , this is essentially , this is essentially Blake 3 hashing and the Blake 3 hashing and the Blake 3 hashing and the QUIC protocol, which are QUIC protocol, which are QUIC protocol, which are probably some of the probably some of the probably some of the best around best around best around today, and as today, and as today, and as a result, it a result, it a result, it delivers high delivers high delivers high performance.

performance. It's amazing how It's amazing how quickly you can quickly you can quickly you can upload upload upload and and and download projects. download projects. Perfectly. We have a Perfectly. We have a few more questions in the few more questions in the few more questions in the chat, but we're chat, but we're chat, but we're out of time. So out of time. So out of time. So we are very grateful that you are we are very grateful that you are we are very grateful that you are with us today and with us today and with us today and shared all shared all shared all this valuable this valuable this valuable information. And I think information. And I think information. And I think people will want to people will want to people will want to learn more about learn more about learn more about all these technologies and all these technologies and all these technologies and how to how to how to use them and use them and use them and track their track their track their development. development. Of course. Ren and I will Of course. Ren and I will also take a look at the chat also take a look at the chat also take a look at the chat and try to and try to and try to answer answer answer questions questions questions directly. So directly. So directly. So keep writing and keep writing and keep writing and we'll do our we'll do our we'll do our best to best to best to respond in the chat. respond in the chat. respond in the chat. Yes, that would be great. Yes, that would be great. Yes, that would be great. Yes, there are quite a Yes, there are quite a Yes, there are quite a few questions there. few questions there. few questions there. Thank you for that. Thank you for that. Thank you for that. Perfectly. Perfectly. Perfectly. Class. Yes, thank you. Class. Yes, thank you. Class. Yes, thank you. Thanks again, Rene and Thanks again, Rene and Thanks again, Rene and Ben. Thank you for Ben. Thank you for Ben. Thank you for joining us.