[00:00.000 --> 00:00.320] Welcome. [00:06.180 --> 00:06.560] Hi, everybody. [00:08.180 --> 00:10.760] My name is Salman, and this is Filippo. [00:11.240 --> 00:13.480] And we're here to talk to you about... Oh, it's on now. [00:14.300 --> 00:17.560] We're here to talk to you about a project that we've been working on recently. [00:20.340 --> 00:22.060] It's called the Code Archive. [00:22.700 --> 00:27.240] And before we begin, we should mention that we both work for Cloudflare. [00:27.760 --> 00:31.000] But this is a completely personal project. [00:31.180 --> 00:32.980] We've been working on it in weekends. [00:32.980 --> 00:34.820] And our own spare time. [00:36.700 --> 00:49.260] So, before I start telling you about what we did, I want to talk a little bit about open-source software, how we store it, where we store it, and how we share it. [00:50.300 --> 01:02.540] We've built these tools as a community, like Git, that are amazing at tracking all the code and, you know, just knowing where all the code is. [01:02.540 --> 01:04.060] And histories and revisions, etc. [01:04.700 --> 01:14.000] And we have amazing services, like GitHub, that help us collaborate and communicate over this code. [01:17.140 --> 01:27.900] And it's so good that GitHub alone has over 34 million repositories and over 14 million users. [01:28.140 --> 01:29.480] These numbers might be out of date as well. [01:29.480 --> 01:38.820] Like, they're so essential to the health of the open-source community, essentially part of the ecosystem. [01:39.400 --> 01:47.420] So, it's clear that GitHub by now became a critical infrastructure for the open-source community and for everything that relies on it. [01:47.420 --> 02:09.280] So, just to take an example, the Go language made that reliance convention by using paths for import packages and allowing you to just have github.com slash user slash package to point to a dependency that will be fetched automatically by the tooling. [02:10.260 --> 02:29.680] Now, obviously, all this is great because it allows us to have a central place for discovery, for code retrieval, enabling, for example, the Go behavior, and it very much lowers the barrier to entry for a new contribution to the projects. [02:30.380 --> 02:34.740] Because, because everyone knows how GitHub works and everyone has a GitHub account. [02:35.340 --> 02:40.820] So, everything you need to start contributing is probably already in your toolkit. [02:42.820 --> 02:47.680] So, what we wanted to do, however, is take a slightly different perspective. [02:48.160 --> 02:54.300] And if you look a bit more long-term, there is something that doesn't sit that well in this picture. [02:55.360 --> 02:58.220] Put simply, repositories can disappear. [03:00.080 --> 03:01.460] Repositories get deleted. [03:01.820 --> 03:15.700] And if the author of a repository that I'll call a random name LeftPad removes it from GitHub and your software relied on it, now your software doesn't build or doesn't run. [03:18.400 --> 03:23.060] Even if the repository doesn't disappear, branches get rebased. [03:23.060 --> 03:40.460] And if you know that in April 2014, something was able to go get cleanly because all the dependencies worked, good luck finding out exactly what it was, master of the Git branch master, in April 2014. [03:41.100 --> 03:43.420] You'll say, well, but Git has history. [03:43.620 --> 03:48.560] I can just go look what commit was made before April 2014. [03:49.560 --> 03:53.220] The point is that Git history is completely malleable. [03:54.000 --> 03:59.980] History can be rewritten by the owner of the project or anyone with access to the repositories. [04:00.500 --> 04:04.880] This can be made by mistake, can be made by a specific workflow. [04:05.540 --> 04:16.900] It can even not be a mistake at all and just be the case that the timestamps on Git don't match when code was pushed to GitHub. [04:19.120 --> 04:23.540] Yeah, and occasionally also, in addition to all of that, services go down. [04:24.280 --> 04:26.500] We're all familiar with the angry unicorn. [04:27.840 --> 04:42.600] And even though Git is distributed by nature, we've come to rely on GitHub's central kind of infrastructure, so that whenever GitHub's actually down, not much work happens, apparently. [04:44.620 --> 04:49.860] And in addition to that, also, some services disappear, like Google code is discontinued. [04:50.400 --> 04:52.080] Gitorious no longer exists. [04:52.920 --> 05:04.720] With these two cases, we were lucky because there were projects and ways in which we could retrieve the code and migrated something else, mostly GitHub. [05:05.480 --> 05:07.920] But we might not be lucky in the future. [05:08.340 --> 05:16.120] I'd like you to make a quick experiment and think of your oldest Git repository you have on GitHub. [05:16.500 --> 05:21.920] What are the chances that you made that on an old machine and you don't actually have a copy of that anymore? [05:21.920 --> 05:27.000] And you're relying on GitHub to keep that code and effort you spent alive. [05:27.540 --> 05:47.080] Or if you have something with a complex dependency tree, of which most are hosted on GitHub, try to think when, in 2030, Jason Scott will be trying to run that code on some weird JavaScript emulator for x86, because x86 will be dead, but JavaScript will definitely still be alive. [05:48.220 --> 06:01.800] Now, how likely it is that both GitHub, the specific GitHub repository, will still be available and your code will still be accessible like it is today? [06:03.940 --> 06:05.020] Don't get us wrong. [06:05.320 --> 06:07.740] We don't think GitHub is going to go anywhere anytime soon. [06:07.860 --> 06:08.640] We really love it. [06:08.780 --> 06:12.200] Except that we just really need to be prepared just in case. [06:13.980 --> 06:15.580] So, we thought about this. [06:15.580 --> 06:19.700] And we thought, the web has archive.org's Wayback Machine. [06:20.480 --> 06:22.560] Why don't we have the same thing for code? [06:23.400 --> 06:26.500] I mean, code should be easier, shouldn't it? [06:27.360 --> 06:28.620] It's well structured. [06:29.320 --> 06:30.920] It compresses really well. [06:32.580 --> 06:35.220] It's meant to be consumed by machines. [06:38.620 --> 06:40.200] So, we built it. [06:41.040 --> 06:42.000] And it's running. [06:42.500 --> 06:44.580] And it's cloning repos. [06:44.580 --> 06:49.480] So far, we've got about 100,000 repositories, just under that. [06:49.740 --> 06:53.900] And just under 400,000 snapshots of these repositories. [06:54.200 --> 06:55.840] And in total, it's about a terabyte. [06:56.140 --> 06:58.020] And that's only the beginning. [07:00.660 --> 07:02.240] This is just the prototype. [07:02.240 --> 07:04.320] It's only fetching stuff from GitHub. [07:05.160 --> 07:08.220] And it's only fetching active repositories. [07:08.340 --> 07:15.180] So, it's only, like, triggered when a push happens. [07:16.180 --> 07:16.560] Sorry. [07:17.800 --> 07:23.320] When is it, like, how are you determining which repositories to pull in? [07:24.140 --> 07:26.520] This is actually exactly what they're talking about. [07:26.680 --> 07:28.760] I think we should probably keep questions at the end. [07:28.980 --> 07:29.420] Sorry. [07:29.420 --> 07:29.880] No hard feelings. [07:30.040 --> 07:30.400] No worries. [07:30.640 --> 07:35.360] So, we're trying to limit the scope of this prototype. [07:35.360 --> 07:40.160] We're fetching only the popular repositories, the ones which have 10 stars or more. [07:41.140 --> 07:45.960] And by default, we're saying reasonably sized repositories. [07:46.800 --> 07:48.280] We'll see what that means a bit later. [07:50.360 --> 07:53.060] So, this is what we did. [07:54.160 --> 07:55.000] I'm going to talk... [07:55.000 --> 07:57.100] We're going to talk for the rest of the talk about how we did it. [07:57.700 --> 08:01.560] This is vaguely, hand-wavingly, the architecture we used. [08:03.120 --> 08:04.040] They're all... [08:04.040 --> 08:05.100] It's all written in Go. [08:06.040 --> 08:07.820] Which was definitely a good idea. [08:08.040 --> 08:11.440] And they're all running in containers on Kubernetes because we're modern. [08:11.840 --> 08:12.800] Which also... [08:12.800 --> 08:14.920] Which actually was a terrible idea. [08:14.920 --> 08:20.360] Because it turns out if you have a single machine, a cluster manager might not be what you want. [08:20.580 --> 08:23.780] Because then you end up with a cluster to manage your cluster of one machine. [08:25.020 --> 08:25.720] So, briefly. [08:25.860 --> 08:28.160] We'll get into the details throughout the talk. [08:28.360 --> 08:30.440] But briefly, we have a thing we call the drinker. [08:30.560 --> 08:34.040] Which watches what we call the GitHub firehose. [08:34.660 --> 08:35.980] Which is all the events that happen on GitHub. [08:37.040 --> 08:40.160] And it cues the repos which should be cloned. [08:40.780 --> 08:42.160] We have the thing called the fetcher. [08:42.160 --> 08:45.620] Which fetches all the repos which are in the queue. [08:47.040 --> 08:48.100] And we... [08:48.100 --> 08:50.640] It stores all the data in a blob store. [08:50.880 --> 08:52.660] And we have a front end. [08:53.120 --> 08:55.940] Which is what your Git client can interact with. [08:56.140 --> 08:57.060] To clone these... [08:57.060 --> 08:58.480] And retrieve these packages... [08:58.480 --> 08:59.380] These repos again. [09:01.140 --> 09:01.420] And... [09:01.700 --> 09:03.160] To get into the details. [09:03.160 --> 09:04.780] Philippe is going to start with the drinker. [09:05.320 --> 09:05.340] Yeah. [09:06.080 --> 09:06.480] So... [09:06.480 --> 09:06.740] Good. [09:06.880 --> 09:08.260] Let's break it down... [09:08.260 --> 09:09.740] In two pieces. [09:10.400 --> 09:12.100] The drinker's job is... [09:12.100 --> 09:12.800] To be fair, the easiest. [09:13.100 --> 09:14.840] Because everything it has to do... [09:14.840 --> 09:17.080] Is to figure out what needs to be fetched. [09:17.260 --> 09:18.140] And put in a queue. [09:18.820 --> 09:19.300] Which... [09:19.300 --> 09:20.200] Since we're lazy. [09:20.320 --> 09:22.660] We just made out of a MySQL table. [09:24.580 --> 09:25.060] Anyway... [09:25.680 --> 09:28.020] What the drinker is supposed to do... [09:28.020 --> 09:29.780] Is monitor the GitHub firehose. [09:29.780 --> 09:32.860] Which is the stream of all the API events... [09:32.860 --> 09:33.600] That happen on GitHub. [09:34.280 --> 09:35.860] For pushes... [09:35.860 --> 09:38.100] Or other events that suggest... [09:38.100 --> 09:40.520] That something about the Git structure changed. [09:41.540 --> 09:42.180] Then... [09:42.180 --> 09:43.280] It puts... [09:43.280 --> 09:44.420] Those repos in a queue... [09:44.420 --> 09:46.380] Based on how many stars they have. [09:48.320 --> 09:48.960] Now... [09:48.960 --> 09:49.560] This... [09:49.560 --> 09:50.480] This would be... [09:50.480 --> 09:51.380] Very easy. [09:51.660 --> 09:52.300] But... [09:52.300 --> 09:53.520] GitHub has... [09:53.520 --> 09:54.880] Actually pretty reasonable... [09:54.880 --> 09:56.240] Rate limit of... [09:56.240 --> 09:57.960] 5,000 requests per hour. [09:58.440 --> 09:59.760] If you want to keep up with... [09:59.780 --> 10:01.400] The firehose that has... [10:01.400 --> 10:01.960] Tens of... [10:01.960 --> 10:03.580] Thousands of events... [10:03.580 --> 10:04.240] Every... [10:04.570 --> 10:05.100] Every hour. [10:05.840 --> 10:06.060] You... [10:06.060 --> 10:07.720] You might have to go above that. [10:08.100 --> 10:08.400] And... [10:08.400 --> 10:09.760] For sure you are putting stress... [10:09.760 --> 10:10.580] On the GitHub servers. [10:11.220 --> 10:12.760] Thankfully there is this project... [10:12.760 --> 10:14.140] Called the GitHub Archive. [10:14.740 --> 10:16.580] Not to be confused with the Code Archive. [10:18.060 --> 10:19.220] Which caches... [10:19.220 --> 10:20.740] Which bundles... [10:20.740 --> 10:23.520] All the output of the GitHub firehose... [10:23.520 --> 10:24.980] Into nice tarballs. [10:25.260 --> 10:26.760] Which are published with... [10:26.760 --> 10:27.760] A delay of... [10:28.280 --> 10:28.720] Barely... [10:28.720 --> 10:29.140] Five minutes. [10:29.640 --> 10:30.200] So... [10:30.200 --> 10:31.580] At five minutes on the hour... [10:31.580 --> 10:32.500] A new tarball... [10:32.500 --> 10:32.980] Becomes... [10:32.980 --> 10:34.720] Available on the GitHub Archive... [10:34.720 --> 10:35.080] Website... [10:35.440 --> 10:35.780] Compressed... [10:35.780 --> 10:36.520] And... [10:36.520 --> 10:37.180] Wrapped... [10:37.180 --> 10:37.860] And... [10:37.860 --> 10:38.880] The drinker... [10:38.880 --> 10:39.680] Just... [10:39.680 --> 10:41.080] Waits for that to happen... [10:41.080 --> 10:41.940] Fetches that... [10:41.940 --> 10:42.460] And... [10:42.460 --> 10:44.400] Starts going over the events... [10:44.400 --> 10:44.720] Into the... [10:44.720 --> 10:45.300] In that... [10:45.300 --> 10:46.780] In the tarball. [10:46.780 --> 10:48.040] So... [10:48.040 --> 10:49.800] The only thing that... [10:49.800 --> 10:52.160] That tarball doesn't tell us... [10:52.160 --> 10:52.520] Is... [10:52.520 --> 10:54.000] How many stars... [10:54.000 --> 10:55.240] Each repository has. [10:56.260 --> 10:57.020] So... [10:57.020 --> 10:59.160] Obviously the easy approach is... [10:59.160 --> 11:00.140] For every repository... [11:00.420 --> 11:01.340] That we have to queue... [11:01.340 --> 11:02.640] We ask the... [11:02.640 --> 11:03.040] GitHub API. [11:03.460 --> 11:04.120] But again... [11:04.120 --> 11:05.420] We can only do that... [11:05.420 --> 11:06.580] Five thousand times... [11:06.580 --> 11:06.980] An hour. [11:07.780 --> 11:08.240] So... [11:08.240 --> 11:08.780] The solution... [11:09.300 --> 11:10.340] For example... [11:10.340 --> 11:10.780] Here... [11:10.780 --> 11:11.640] You see the... [11:11.640 --> 11:12.900] The GitHub... [11:13.340 --> 11:14.900] Rate limit... [11:14.900 --> 11:16.040] Going down... [11:16.040 --> 11:16.560] It's the... [11:16.560 --> 11:17.480] It's the blue one. [11:18.300 --> 11:18.800] So... [11:18.800 --> 11:20.200] What we did... [11:20.200 --> 11:21.000] Instead... [11:21.000 --> 11:21.380] Is... [11:21.380 --> 11:22.340] Build a cache... [11:22.340 --> 11:23.760] That would hit GitHub... [11:23.760 --> 11:24.480] The first time... [11:24.480 --> 11:25.260] We have to figure out... [11:25.260 --> 11:26.480] How many stars... [11:26.480 --> 11:27.760] A repository has... [11:27.760 --> 11:28.640] And then uses... [11:28.640 --> 11:29.460] The firehose... [11:29.460 --> 11:30.360] To keep that number... [11:30.360 --> 11:31.100] Up to date... [11:31.100 --> 11:31.500] So that... [11:31.500 --> 11:32.660] If a new repository... [11:32.660 --> 11:33.100] Appears... [11:33.100 --> 11:33.700] And becomes... [11:33.700 --> 11:34.220] Quickly popular... [11:34.220 --> 11:35.580] It will see the star... [11:35.580 --> 11:36.540] Events... [11:36.540 --> 11:37.240] Happen... [11:37.240 --> 11:37.660] And... [11:37.660 --> 11:38.640] It will make it... [11:38.640 --> 11:39.100] Fetch... [11:39.100 --> 11:40.560] When the tenth star... [11:41.780 --> 11:42.280] Happens... [11:42.280 --> 11:43.380] Getting that... [11:43.980 --> 11:44.420] Cache bootstrapped... [11:44.920 --> 11:45.280] Has been... [11:45.860 --> 11:46.180] An... [11:46.180 --> 11:47.220] Ordeal... [11:47.220 --> 11:47.760] Because... [11:47.760 --> 11:49.040] It started by... [11:49.040 --> 11:49.920] Being able to do... [11:49.920 --> 11:51.320] Barely 10 minutes... [11:51.320 --> 11:52.480] Of firehose... [11:52.480 --> 11:53.560] Processing 10 minutes... [11:53.560 --> 11:54.100] Of firehose... [11:54.100 --> 11:55.280] With 5,000 requests... [11:55.820 --> 11:56.220] And... [11:56.220 --> 11:56.940] We felt like... [11:56.940 --> 11:58.100] We would never catch up... [11:58.100 --> 11:58.480] And... [11:58.480 --> 11:59.760] We probably would have... [11:59.760 --> 12:00.860] To find another way... [12:00.860 --> 12:01.400] Or... [12:01.400 --> 12:02.140] Bypass the... [12:02.140 --> 12:03.620] The rate limits... [12:03.620 --> 12:04.200] Instead... [12:04.200 --> 12:05.060] Slowly... [12:05.060 --> 12:05.800] But surely... [12:05.800 --> 12:06.600] By... [12:06.600 --> 12:07.320] Keeping it running... [12:07.320 --> 12:08.160] Many nights... [12:08.160 --> 12:08.700] And sometimes... [12:09.050 --> 12:09.820] Waking up... [12:09.820 --> 12:10.340] To check... [12:10.340 --> 12:10.840] If... [12:10.840 --> 12:11.580] It crashed... [12:11.580 --> 12:11.920] Or... [12:11.920 --> 12:13.200] It was still going... [12:13.200 --> 12:13.740] Eventually... [12:13.740 --> 12:14.120] It did... [12:14.120 --> 12:15.740] Populate the cache... [12:15.740 --> 12:16.020] Enough... [12:16.020 --> 12:17.360] To start catching up... [12:17.360 --> 12:17.760] And then... [12:17.760 --> 12:18.140] Actually... [12:18.140 --> 12:18.860] Go faster... [12:18.860 --> 12:19.500] Than... [12:19.500 --> 12:19.700] The... [12:19.700 --> 12:20.720] The... [12:20.720 --> 12:21.340] The... [12:21.340 --> 12:21.360] The... [12:21.360 --> 12:21.920] The rate limit... [12:27.560 --> 12:29.800] Was not exhausted... [12:29.800 --> 12:30.760] At each hour... [12:30.760 --> 12:32.260] After catching up... [12:32.260 --> 12:33.000] And... [12:33.000 --> 12:33.420] These days... [12:33.420 --> 12:35.080] You can't even notice... [12:35.080 --> 12:35.140] The... [12:35.140 --> 12:36.280] The rate limit... [12:38.160 --> 12:38.580] Dwindling... [12:39.240 --> 12:40.080] So... [12:40.080 --> 12:40.900] Now... [12:40.900 --> 12:42.220] Um... [12:42.220 --> 12:43.140] This is... [12:43.140 --> 12:43.460] How... [12:44.080 --> 12:44.500] Average... [12:44.500 --> 12:45.020] Hour... [12:45.020 --> 12:45.840] Of drinking... [12:45.840 --> 12:46.600] Looks like... [12:46.600 --> 12:46.960] It... [12:46.960 --> 12:47.780] We have... [12:47.780 --> 12:48.400] I see... [12:48.400 --> 12:50.340] 21,000... [12:50.340 --> 12:50.580] Push... [12:50.580 --> 12:51.400] Events... [12:51.400 --> 12:52.240] Of which... [12:52.240 --> 12:53.660] 20.7... [12:53.660 --> 12:54.040] We... [12:54.040 --> 12:54.860] Can figure out... [12:54.860 --> 12:55.560] From the cache... [12:55.560 --> 12:56.820] We hit the API... [12:56.820 --> 12:58.900] Just 300 times an hour... [12:58.900 --> 12:59.560] And... [12:59.560 --> 13:00.380] We queue about... [13:00.380 --> 13:01.020] 2,000... [13:01.020 --> 13:02.620] Repositories every hour... [13:02.620 --> 13:03.180] Uh... [13:03.180 --> 13:04.560] You must consider that... [13:04.560 --> 13:05.640] We don't queue... [13:05.640 --> 13:06.320] Uh... [13:06.320 --> 13:07.580] Repositories that have... [13:07.580 --> 13:08.720] Less than 10 stars... [13:08.720 --> 13:09.520] And we don't queue... [13:09.520 --> 13:10.880] Repositories that were changed... [13:10.880 --> 13:12.240] Twice in the same hour... [13:12.240 --> 13:13.080] We queue them... [13:13.080 --> 13:13.900] Only once... [13:14.660 --> 13:15.120] Uh... [13:15.120 --> 13:15.800] You can also see... [13:15.800 --> 13:16.700] From the timestamps... [13:16.700 --> 13:17.180] That... [13:17.180 --> 13:17.340] The... [13:17.340 --> 13:18.920] Tarble is published... [13:18.920 --> 13:20.240] Approximately at... [13:20.240 --> 13:22.180] 5 minutes and 50 seconds... [13:22.180 --> 13:23.220] On the hour... [13:23.220 --> 13:23.660] And... [13:23.660 --> 13:24.180] By... [13:24.180 --> 13:25.720] 7 minutes on the hour... [13:25.720 --> 13:26.560] We finished... [13:26.560 --> 13:26.700] Uh... [13:26.700 --> 13:27.780] Queuing the... [13:27.780 --> 13:28.400] Repositories... [13:31.020 --> 13:31.720] Uh... [13:31.720 --> 13:32.920] Interestingly enough... [13:32.920 --> 13:33.760] While... [13:33.760 --> 13:34.820] Developing this... [13:34.820 --> 13:35.240] Uh... [13:35.240 --> 13:35.940] This drink... [13:35.940 --> 13:37.200] We hit the... [13:37.200 --> 13:38.520] Go GitHub library... [13:38.520 --> 13:39.320] Harder than... [13:39.320 --> 13:40.220] Than probably... [13:40.220 --> 13:41.200] It was done before... [13:41.200 --> 13:43.700] And we figured out that it was not doing HTTP pipelining... [13:43.700 --> 13:44.700] So costing... [13:45.480 --> 13:46.460] A lot of... [13:46.460 --> 13:46.960] Uh... [13:46.960 --> 13:47.340] CPU... [13:47.340 --> 13:49.160] To spin up new TLS connections... [13:49.160 --> 13:49.740] Every request... [13:50.560 --> 13:50.860] And... [13:50.860 --> 13:51.780] By fixing that... [13:51.780 --> 13:52.940] We probably reduced... [13:52.940 --> 13:54.460] CPU load on GitHub... [13:54.460 --> 13:54.740] And... [13:54.740 --> 13:55.700] We made the library... [13:55.700 --> 13:57.040] Four times faster... [13:57.040 --> 13:57.440] So... [13:57.440 --> 13:58.540] I like to think that... [13:58.540 --> 14:00.340] The net result of this project... [14:00.340 --> 14:00.980] On the... [14:00.980 --> 14:01.700] GitHub CPUs... [14:01.700 --> 14:02.780] Is actually negative... [14:02.780 --> 14:03.440] Because now... [14:03.440 --> 14:05.200] Everyone that uses the library... [14:05.200 --> 14:05.780] Will... [14:05.780 --> 14:06.180] Put... [14:06.180 --> 14:07.720] Much less trains... [14:10.260 --> 14:10.780] Alright... [14:10.780 --> 14:11.060] So... [14:11.060 --> 14:11.840] This was the drinker... [14:12.240 --> 14:13.100] And this was easy... [14:13.100 --> 14:13.900] Now we have... [14:13.900 --> 14:15.820] This queue of things to fetch... [14:16.280 --> 14:17.720] Now obviously we have to fetch them... [14:18.520 --> 14:18.980] So... [14:18.980 --> 14:19.940] Enter the fetcher... [14:22.500 --> 14:23.540] Interestingly named... [14:23.540 --> 14:25.440] Obviously the fetcher is about... [14:25.440 --> 14:25.860] Fetching repositories... [14:26.340 --> 14:26.700] But... [14:26.700 --> 14:27.940] Fetching to what format... [14:27.940 --> 14:28.720] And to what... [14:28.720 --> 14:30.560] Obviously you can't just... [14:30.560 --> 14:32.760] Make one repository on the file system... [14:32.760 --> 14:34.700] For each repository on GitHub... [14:34.700 --> 14:35.780] And then run git pool... [14:36.480 --> 14:38.400] Not that we didn't try that... [14:38.400 --> 14:38.920] But... [14:38.920 --> 14:39.780] Turns out you can't... [14:41.100 --> 14:41.660] Uh... [14:41.660 --> 14:41.900] So... [14:41.900 --> 14:43.260] Instead we'll have to understand... [14:43.260 --> 14:45.220] A bit more how git works... [14:45.220 --> 14:45.740] Git... [14:45.740 --> 14:46.300] In... [14:46.300 --> 14:47.400] How everything... [14:47.400 --> 14:49.000] In git is stored... [14:49.000 --> 14:50.180] Is in a... [14:50.180 --> 14:50.780] Content addressed... [14:51.540 --> 14:52.100] Uh... [14:52.100 --> 14:53.140] Graph... [14:53.140 --> 14:53.720] Tree... [14:53.720 --> 14:54.620] Graph... [14:54.620 --> 14:55.300] Graph... [14:56.160 --> 14:56.720] Graph... [14:57.960 --> 14:58.520] Um... [14:58.520 --> 14:59.040] Content addressed... [14:59.480 --> 15:00.020] Means that... [15:00.020 --> 15:01.800] The key to retrieve something... [15:01.800 --> 15:02.360] Is... [15:02.360 --> 15:03.760] The hash of the content... [15:03.760 --> 15:04.180] So... [15:04.180 --> 15:05.640] Two objects that are equal... [15:05.780 --> 15:07.060] Have the same key... [15:07.060 --> 15:08.620] And don't have to be stored twice... [15:09.600 --> 15:10.160] Uh... [15:10.160 --> 15:10.900] When I say that... [15:10.900 --> 15:11.600] Everything... [15:11.600 --> 15:13.080] Is stored in... [15:13.080 --> 15:13.700] Uh... [15:13.700 --> 15:15.080] In this content address... [15:15.080 --> 15:15.660] Storage in git... [15:15.660 --> 15:16.600] I mean everything... [15:16.600 --> 15:16.760] So... [15:16.760 --> 15:17.760] When you make a commit... [15:17.760 --> 15:19.400] A small text file... [15:19.400 --> 15:20.520] That says... [15:20.520 --> 15:20.780] Uh... [15:20.780 --> 15:21.840] This person... [15:21.840 --> 15:22.960] At this time... [15:23.560 --> 15:23.840] Committed... [15:23.840 --> 15:24.220] Uh... [15:24.220 --> 15:25.720] This tree... [15:25.720 --> 15:26.360] Over... [15:26.360 --> 15:28.140] With this parent commit... [15:28.140 --> 15:29.500] And this message... [15:29.500 --> 15:30.620] Gets created... [15:30.620 --> 15:32.400] That small text files... [15:32.400 --> 15:33.100] Gets hashed... [15:33.100 --> 15:34.460] And is now the... [15:34.460 --> 15:35.480] Shawan commit... [15:35.480 --> 15:36.240] Uh... [15:36.240 --> 15:36.640] Id... [15:36.640 --> 15:37.400] That... [15:37.400 --> 15:38.440] You are used to see... [15:39.100 --> 15:39.720] That... [15:39.720 --> 15:41.100] That file has inside... [15:41.100 --> 15:41.920] Two hashes... [15:41.920 --> 15:43.250] One of the previous commit... [15:43.700 --> 15:44.200] So... [15:44.200 --> 15:45.320] Whichever commit is... [15:45.320 --> 15:45.900] Before... [15:45.900 --> 15:46.820] In the history... [15:46.820 --> 15:48.440] And one to a tree... [15:48.440 --> 15:49.700] A tree is another... [15:49.700 --> 15:50.880] Small text file... [15:50.880 --> 15:51.250] That says... [15:51.680 --> 15:52.310] That lists... [15:52.700 --> 15:53.420] Every folder... [15:53.420 --> 15:54.040] In the root... [15:54.040 --> 15:54.880] Of the... [15:54.880 --> 15:55.720] Repository... [15:55.720 --> 15:57.220] At the time you made the commit... [15:57.220 --> 15:57.760] So... [15:57.760 --> 15:58.460] It will say... [15:58.460 --> 15:59.000] README... [15:59.960 --> 16:00.240] And... [16:00.240 --> 16:01.220] A hash... [16:01.220 --> 16:02.160] And it will say... [16:02.160 --> 16:03.140] SRC... [16:03.140 --> 16:04.100] And a hash... [16:04.100 --> 16:05.340] The README hash... [16:05.340 --> 16:06.220] In turn... [16:06.220 --> 16:07.060] Will point... [16:07.060 --> 16:07.880] To a blob... [16:07.880 --> 16:09.380] Which is the content of the README... [16:09.380 --> 16:10.880] You can see where this is going... [16:10.880 --> 16:10.960] Right? [16:11.340 --> 16:12.540] Everything is... [16:12.540 --> 16:13.520] Is a... [16:13.520 --> 16:14.780] Content-addressed object... [16:14.780 --> 16:15.620] And... [16:15.620 --> 16:17.180] If you make a commit... [16:17.180 --> 16:18.840] That didn't change the README... [16:18.840 --> 16:20.500] The same... [16:20.500 --> 16:21.120] README... [16:21.120 --> 16:21.920] Blob... [16:21.920 --> 16:22.620] Will be used... [16:22.620 --> 16:24.400] In the next commit... [16:25.460 --> 16:26.020] So... [16:26.020 --> 16:27.460] Knowing this... [16:27.460 --> 16:28.740] We can have a look at... [16:28.740 --> 16:29.300] How the... [16:29.300 --> 16:30.360] Protocol... [16:30.360 --> 16:31.420] The git protocol... [16:31.420 --> 16:31.860] Works... [16:31.860 --> 16:32.100] So... [16:32.100 --> 16:33.640] Essentially what your client... [16:33.640 --> 16:34.960] And the github servers... [16:34.960 --> 16:35.660] Speak... [16:35.660 --> 16:36.880] When you connect... [16:36.880 --> 16:37.380] To github... [16:37.380 --> 16:38.120] To make a clone... [16:38.120 --> 16:39.160] Or a pool... [16:39.160 --> 16:41.100] As soon as you open the connection... [16:41.100 --> 16:42.100] GitHub... [16:42.100 --> 16:43.020] Starts by... [16:43.020 --> 16:43.560] Telling you... [16:43.560 --> 16:44.040] All... [16:44.040 --> 16:44.720] The... [16:44.720 --> 16:45.860] Comet hashes... [16:45.860 --> 16:46.440] Of... [16:46.440 --> 16:47.060] The... [16:47.060 --> 16:47.720] REFs... [16:47.720 --> 16:48.880] Are like... [16:48.880 --> 16:49.640] Branches... [16:49.640 --> 16:50.360] So... [16:50.360 --> 16:51.800] Everything that has a name... [16:51.800 --> 16:53.020] And it will tell you... [16:53.020 --> 16:53.860] For example... [16:53.860 --> 16:54.360] Head... [16:54.360 --> 16:55.340] Is this comet... [16:55.780 --> 16:56.040] Master... [16:56.040 --> 16:57.000] Is this comet... [16:57.000 --> 16:57.960] Version 1... [16:57.960 --> 16:59.280] Is this comet... [16:59.280 --> 17:00.900] And then the client... [17:00.900 --> 17:01.720] Replies to that... [17:01.720 --> 17:02.940] By saying... [17:02.940 --> 17:03.800] Interesting... [17:03.800 --> 17:04.140] So... [17:04.140 --> 17:05.780] I want this comet... [17:05.780 --> 17:06.640] And this comet... [17:06.640 --> 17:07.520] And this comet... [17:07.520 --> 17:09.740] And I have... [17:09.740 --> 17:10.920] This comet... [17:10.920 --> 17:11.740] And this comet... [17:11.740 --> 17:12.560] And this comet... [17:13.420 --> 17:13.940] So... [17:13.940 --> 17:15.120] What the server can do... [17:15.120 --> 17:16.500] Is do graph transversal... [17:16.500 --> 17:26.080] And follow the graph to mark everything that is below the graph of the things you say you have and say, okay, they already have this. [17:26.240 --> 17:30.060] Because if they have this commit, it means they have this version of the README. [17:30.460 --> 17:34.520] So if the README didn't change, I don't have to send them the object of the README again. [17:34.940 --> 17:44.680] And then it figures out everything that you said you want and everything that is below in the graph of that and makes a pack of that and sends you the pack. [17:45.260 --> 17:55.240] Essentially, it builds a mind image of how your graph looks, compares it with their server-side graph, and sends you the difference. [17:58.420 --> 18:02.320] Now, let's go back at how we store this, because this is what we are here to do. [18:03.440 --> 18:05.200] Storing the refs is kind of easy. [18:05.400 --> 18:08.780] We just take a snapshot every time we fetch. [18:09.000 --> 18:18.720] This way, we don't have problems with force pushes or anything, because every time we connect, we take a snapshot of a name pointing to a comet. [18:18.900 --> 18:23.140] We don't care if it's a fast-forward, if it's a rebase, if anything. [18:23.580 --> 18:26.900] And we store this in a simple database, because they're not that big. [18:28.100 --> 18:29.560] Then we make another fetch. [18:29.760 --> 18:33.820] We just store another snapshot of the same or different names. [18:36.000 --> 18:43.480] Then the question is, how do we store all the actual objects that build the graph and that are needed to reveal the repository? [18:46.340 --> 18:51.840] So, to begin in the initial version, we used KamliStore. [18:52.180 --> 18:56.000] For those who aren't familiar with KamliStore, it's a content-addressed storage system. [18:57.240 --> 19:00.100] We were like, well, git's content-addressed. [19:00.260 --> 19:01.500] KamliStore is content-addressed. [19:01.680 --> 19:07.080] We'll just throw all the git objects as KamliStore blobs, and we get everything for free. [19:07.740 --> 19:11.980] And indeed, that was really useful, because it allowed us to focus on the other parts. [19:12.800 --> 19:16.340] And I was personally familiar with KamliStore, so that wasn't a problem. [19:17.660 --> 19:28.560] But that really couldn't scale for too long, because git does a lot of clever compressions and deltas and stuff that wouldn't really map well to KamliStore. [19:28.740 --> 19:34.140] So it wasn't a good use case for KamliStore in the long term, but it was really useful for bootstrapping. [19:34.480 --> 19:35.880] You should also check out KamliStore. [19:35.940 --> 19:36.380] It's a good project. [19:38.200 --> 19:42.800] Yeah, we're actually very sad not to be able to use it, because it was so elegant. [19:42.800 --> 19:48.920] We would just take this graph and drop it in a bigger graph, and we don't have to worry about what belongs to what repository. [19:49.460 --> 19:51.040] But yeah, that didn't scale. [19:52.000 --> 19:54.480] So let's go back to the protocol. [19:54.480 --> 20:10.820] We have something we can take advantage of here, and it's that we can send GitHub whatever we have, the haves of the comets, and let GitHub figure out the hard diff-making diff part for us. [20:11.620 --> 20:24.560] And GitHub will give us a pack, which is nothing else than a lot of objects that are needed to go from point A, whatever we said we have, to point B, whatever it is that was pushed last to GitHub. [20:26.060 --> 20:44.680] So this can be made to work very well with cold storage, because obviously we will be storing a lot of data, and it would be nice to be able to store it in some cheaper storage that has maybe higher latency and more expensive retrieval. [20:45.240 --> 20:54.460] So what we are doing is we are keeping the refs in a normal database, because that's easy and that's small. [20:54.760 --> 21:01.800] And then we take the pack files that we receive from GitHub and we drop them directly in cold storage just as they are. [21:01.900 --> 21:02.640] We don't parse them. [21:02.720 --> 21:07.180] We just keep the GitHub's word for it and place them in storage. [21:08.100 --> 21:26.560] And since it's the server that will do the differentiating, we don't have to ever look at the pack files again to make the next fetches, because the next time something is pushed to the Rails repository, we know what the refs of the last commit, of the last fetch were, [21:26.780 --> 21:28.420] because we have that in the live database. [21:28.800 --> 21:38.440] We send that to GitHub, and GitHub builds the smallest pack to get from what we have to what we need and gives us that pack, binary pack. [21:38.680 --> 21:42.800] We drop that into cold storage, and now we have everything we need. [21:46.500 --> 21:52.260] Now, the only disadvantage of this is that now, obviously, you have a linear dependency. [21:52.380 --> 22:06.180] You can't use a pack alone, because that pack depends on all the other packs to form a real full repository, because it might have hashes that point to objects that you retrieved before. [22:06.500 --> 22:11.200] So you just have to keep track of the order in which you got the packs. [22:12.260 --> 22:17.580] Now, this is all nice and linear and simple, but GitHub has forks. [22:20.080 --> 22:21.300] Go fork yourself. [22:23.000 --> 22:23.800] Bad joke. [22:24.060 --> 22:24.360] Next. [22:24.960 --> 22:32.020] So, forks are essentially the same repo, because probably they don't diverge that much. [22:32.120 --> 22:35.640] They're usually made just to make a small contribution to the main repository. [22:36.200 --> 22:43.640] And indeed, we know that GitHub, they have some outstanding technical blog posts, by the way, about how they do it. [22:43.760 --> 22:51.700] And we know that they store the entire network, and I call a network the collection of parent repo, and all its forks. [22:52.140 --> 23:00.900] They store it as a single actual Git repository, so that the graph is shared, what we wanted to do globally with Canvas Store. [23:01.960 --> 23:18.700] Now, obviously, forks often sync with the parent, so storing them separately ourselves would be insane, because it would mean that we would have 16,000 copies of Rails, just because there are 16,000 forks of Rails. [23:18.700 --> 23:21.540] And who wants 16,000 copies of Rails? [23:22.440 --> 23:23.000] Sorry. [23:25.320 --> 23:32.020] So, instead, what we are doing is, again, taking advantage of the haves and wants mechanism of the Git protocol. [23:32.340 --> 23:42.720] When we go to ask for a fetch, we tell GitHub that we have all the commits of the last fetch of the main repo, and of all the forks. [23:43.240 --> 23:46.480] This way, GitHub will figure out the difference for us. [23:46.480 --> 23:59.820] And if we go to fetch a fork that was just created, we'll still send all the haves of the main repo, and GitHub will only give us the small difference of the fork from the main. [24:01.380 --> 24:10.860] The only disadvantage of this is that now, instead of a succession of dependencies, like you have to use all these packs to rebuild the repository, now we have a tree. [24:11.240 --> 24:27.140] Because it might be that a fork got this commit first, but then it got merged into master, into the main, and now, to clone the main, you need also that object from the fork, because GitHub didn't send it again, because we said, we have these commits from the fork, [24:27.220 --> 24:27.360] too. [24:28.020 --> 24:35.460] So, essentially, now, instead of a succession of dependencies, we have a tree of dependencies, which was just one more MySQL table. [24:38.120 --> 24:40.560] So, you might have noticed a pattern here. [24:40.560 --> 24:45.740] We've really struggled to make this entire thing as nice as possible to GitHub. [24:46.420 --> 24:56.300] Because, obviously, GitHub here is providing a service for free, and we are trying both not to be shut down, and obviously also not to cost them too much money. [24:56.880 --> 25:02.980] So, first, the drinker versus fetch architecture allows us to run at off-peak times. [25:03.160 --> 25:09.000] So, we can build a queue with the drinker during the day, and not lose any events. [25:09.320 --> 25:18.660] And then, at night, when bandwidth is either cheap or even free based on how you pay for the bandwidth, we start fetching. [25:20.080 --> 25:25.420] We use the raw git protocol, because we suspect that bypasses the HTTP load balancers. [25:25.640 --> 25:27.200] So, maybe that's faster. [25:27.460 --> 25:32.320] We set user agents, so that if they want to shut us down, they can do it cleanly. [25:32.720 --> 25:35.560] And, as we said, we only fetch differences. [25:35.560 --> 25:43.700] So, we don't abuse the bandwidth between us and GitHub, and only make the necessary bytes travel over traffic. [25:46.320 --> 25:56.460] We thought that we didn't have the problem of bandwidth limiting at all, because we were under the impression that we were being rate limited at 10 megabits per second. [25:56.940 --> 26:04.640] We were sort of okay with that, because we were on Kubernetes, so we can just scale it up and run multiple fetchers. [26:06.520 --> 26:10.100] Turns out, we weren't slow because of GitHub. [26:10.420 --> 26:13.540] We were slow because of our own infrastructure. [26:13.900 --> 26:19.860] And when we switched to something simpler and faster, there were still four fetchers running. [26:20.100 --> 26:24.300] And we briefly fetched at 4 gigabits per second from GitHub. [26:24.720 --> 26:27.740] Which, impressingly, just took the load. [26:27.740 --> 26:34.840] Let us download that 4 gigabits per second for a good 10 minutes, until we noticed and, in a panic, shut everything down. [26:35.840 --> 26:38.220] So, one, sorry about that. [26:38.540 --> 26:43.040] Second, if you have graphs, I would love to see if we made a dent on the global graph. [26:44.080 --> 26:45.580] But, yeah, I diverged. [26:46.140 --> 26:52.340] Anyway, now we're running only one fetcher, and one fetcher averages around 600 megabits per second. [26:52.640 --> 26:58.480] Which is something that we estimated to be probably irrelevant at off-peak times. [26:58.740 --> 27:04.540] Obviously, if anyone at GitHub wants to get in contact and tell us how to shape our traffic, we are very happy to do that. [27:05.980 --> 27:09.460] So, this is how a week of fetching looks like. [27:09.780 --> 27:13.360] The big green at the bottom is the queue size. [27:13.360 --> 27:21.820] You can see it increase every hour during the day, when we're not fetching, and then drop suddenly when we start fetching. [27:21.980 --> 27:25.640] And then you can see small bits of work every hour. [27:25.860 --> 27:27.500] Let's zoom in on a night. [27:28.220 --> 27:40.680] We do a long period of fetching as soon as we can wake up, and then every hour there's some drinking at the 5-minute mark, and then immediately it gets fetched. [27:40.680 --> 27:50.020] For example, this random hour, recent hour, did 800 fetches, of which 45 were new repositories. [27:50.220 --> 27:52.400] The others were pools, essentially. [27:53.300 --> 28:03.880] There were one repository that disappeared in the short time between when it was pushed to and recorded in the firehose and when it was fetched. [28:04.020 --> 28:07.480] And we did approximately one gigabyte of download. [28:09.700 --> 28:13.700] Obviously, this wasn't all smooth and easy. [28:14.960 --> 28:20.440] And we're trying to run this as transparently as possible because it's a public archival project. [28:20.720 --> 28:23.820] So now we're going to talk about our favorite crashes. [28:24.520 --> 28:26.080] The first one wasn't a crash. [28:26.460 --> 28:34.860] For the longest time, we stared for like a half an hour at this graph with the bandwidth high and the queue that wasn't decreasing. [28:35.300 --> 28:37.060] Wondering what the hell was going on. [28:37.220 --> 28:38.700] Probably metrics broke or something. [28:39.660 --> 28:43.400] InfluxDB and Grafana were possibly the hardest thing to set up. [28:43.580 --> 28:44.800] So that was likely. [28:45.180 --> 28:49.200] Until we noticed what repository we had just fetched. [28:49.300 --> 29:01.260] A nine gigabytes dump of a forum with daily comments, with all the user data and the avatars and the uploaded zip files. [29:03.260 --> 29:04.680] And that had stars. [29:08.680 --> 29:12.960] That did cause us to put a limit on the size of repos. [29:13.200 --> 29:13.500] Yeah. [29:14.140 --> 29:23.440] So now if you're storing more than 250 megabytes and soon to be raised in a single Git repository, we should probably have a talk. [29:25.480 --> 29:29.440] Then other problems, other time we saw a crash. [29:29.680 --> 29:35.040] And that crash was caused by a 401 return code from GitHub. [29:35.460 --> 29:36.800] And we panicked. [29:37.080 --> 29:39.600] Because 401 is unauthorized. [29:40.020 --> 29:42.680] And we're like, damn, they don't want us to do this. [29:42.800 --> 29:43.700] They blocked us. [29:43.840 --> 29:46.220] We triggered some automatic abuse system. [29:47.140 --> 29:48.480] Everything will not work. [29:48.580 --> 29:49.560] We wasted so much time. [29:49.560 --> 29:51.120] Oh, wait. [29:51.440 --> 29:54.580] Have you tried cloning it over the Git protocol and not HTTP? [29:56.040 --> 30:02.040] Turns out, 401 is what GitHub returns when a repository doesn't exist. [30:02.260 --> 30:09.780] Because otherwise you would be able to distinguish private repositories, which require authorization, and ones that actually don't exist. [30:10.320 --> 30:13.540] So in this case, it was a DMCA. [30:14.520 --> 30:14.890] So... [30:15.440 --> 30:36.680] DMCA aside, when we figured out that the repo we were getting 401s because the repo vanished, in the amount of time between it being put on the queue and the fetcher fetching it, that really validated why we wanted to do this project. [30:36.960 --> 30:37.320] Right? [30:37.520 --> 30:38.680] Repos really do disappear. [30:38.880 --> 30:42.500] Although in this case, it's a DMCA thing, so maybe not so valid. [30:45.120 --> 30:52.340] And finally, when we were preparing this very presentation, we went to Grafana to fetch some shiny graphs to show you all. [30:52.580 --> 30:55.680] And we found 50,000 repositories queue. [30:56.520 --> 30:58.960] Something didn't crash this time, something froze. [30:59.340 --> 31:02.580] So nothing worked for the last week or so. [31:02.580 --> 31:08.640] And so, obviously, we were preparing a talk, so we didn't really know what... [31:09.400 --> 31:10.220] how to... [31:10.220 --> 31:12.140] You know, we didn't have the time to fix it. [31:12.420 --> 31:16.100] So it turns out that turning things off and on again can help. [31:18.200 --> 31:21.380] So you can see it being turned off and on again there. [31:22.180 --> 31:26.440] It quickly caught up, and now in the to-do list there's alerting. [31:27.380 --> 31:27.780] Good. [31:29.980 --> 31:30.900] Oh, yeah. [31:30.900 --> 31:33.380] So, well, that was a fetcher. [31:34.580 --> 31:36.400] The next part is the back panel. [31:37.380 --> 31:39.880] And that's the fun and hacky part that I enjoy talking about. [31:41.060 --> 31:53.500] So what we mean by back panel is some way of having an admin UI to do all the manual stuff that we need to do to the data set. [31:53.500 --> 32:02.680] So we talked about large repos before, and we have a limit of 250 megabytes. [32:02.880 --> 32:07.720] But for some repos, like the Linux kernel, we want to clone it regardless. [32:08.420 --> 32:19.980] So we have a blacklist of big repos, and manually, if we think one of them is worth the exception, we'll just move it to a whitelist. [32:19.980 --> 32:23.560] And that was just... these are just like two tables on a... [32:23.560 --> 32:25.400] actually one table on a database. [32:26.320 --> 32:27.680] And we're like, we need to be UI this. [32:27.940 --> 32:30.800] We need to build a UI for this, but we really suck at building UIs. [32:31.940 --> 32:38.880] So we got an idea from Twitter to use Trello as an interface. [32:40.180 --> 32:42.360] So we used Trello as an interface. [32:43.000 --> 32:46.920] It's lazy and it's hacky, a little bit, but it really works, and it works really well. [32:48.200 --> 32:51.380] So we have a thing that works like a cron job. [32:52.940 --> 32:59.540] It uses the Trello API, and it synchronizes these two lists with the repos from our database. [32:59.920 --> 33:03.680] One for like the... it synchronizes the blacklist and the whitelist. [33:03.960 --> 33:20.460] And if we decide that we want to move one from the blacklist to the whitelist, say we see, oh, the Linux kernel is on the blacklist because it's too large, we need to be cloning that, we just drag the card to the whitelist, the cron job synchronizes it again, [33:20.620 --> 33:22.340] and we clone it in the next phase. [33:24.520 --> 33:26.700] Oh, by the way, that's public. [33:26.960 --> 33:30.760] So you can go and see the repos that we're cloning that are too large. [33:31.840 --> 33:45.760] So essentially we got authentication, comments, and all the JavaScript part for free, and we just run synchronization, and we can go there and comment on why something got on the blacklist and why something got on the whitelist. [33:45.940 --> 33:53.080] And again, everything is transparent because we literally don't have backpanel tools that are not exposed publicly and logged. [33:55.800 --> 33:56.520] All right. [33:57.000 --> 33:58.960] So, all good and nice. [33:59.060 --> 34:08.160] Now you have this huge archive of pack files, but now you need to somehow give access to people to those repositories. [34:08.160 --> 34:10.200] So this is the front-end, obviously. [34:10.440 --> 34:12.520] We wanted two interfaces for this. [34:12.800 --> 34:21.300] A git-native interface, so that you can git-clone a repository as if it was from GitHub, and a web interface. [34:21.980 --> 34:25.420] The problem with this is that, first, retrieval is expensive. [34:25.960 --> 34:32.660] This is near-line storage, so getting things out of cold storage should be done as little as possible. [34:32.880 --> 34:37.620] And outbound bandwidth is the most expensive thing in the entire system. [34:37.620 --> 34:43.200] So every time someone clones, we can't just give them everything and hope for the best. [34:44.980 --> 34:51.740] So, we figured how we solved it is using a local cache and something called git-alternates. [34:52.140 --> 34:57.320] Git has so many features that I didn't know about that I discovered through this project. [34:57.780 --> 35:02.660] This one, in particular, allowed us to make the smallest git repository ever. [35:02.660 --> 35:08.060] It's one, two, three, four files, and three directories. [35:08.520 --> 35:11.240] And that's a completely functional git repository. [35:11.500 --> 35:15.580] You can cd in there, run git log, and it will work. [35:16.100 --> 35:19.180] It has the head to say just what is checked out. [35:19.420 --> 35:20.920] The config is essentially empty. [35:21.500 --> 35:26.800] The packed refs is just a file with the refs, which are those mappings from name to commit. [35:26.940 --> 35:29.700] Master, next, v1, etc. [35:30.120 --> 35:40.600] And then it has this file, which is called alternates, that just says, to look for the actual objects in the graph, go look at this other place in the file system. [35:40.600 --> 35:50.260] So what we do is that for repositories that do get cloned, we have a cache on disk of all the pack files just thrown in a folder. [35:50.940 --> 35:53.800] And then the alternates points to that folder. [35:54.060 --> 35:57.880] And the git binary, by itself, can figure out everything else. [35:58.060 --> 35:59.440] So we didn't... [35:59.440 --> 36:11.220] We could have done, you know, actual tree transversal and figured out how to do optimized cloning and make sure that when you wanted a snapshot you didn't get all the data for all the other snapshots. [36:11.440 --> 36:34.020] And instead we were lazy and literally just grabbed everything on disk from the tree of pack file dependencies, pointed git at it, and when you want to git clone, we run git, you tell git what you want and what you have, and git computes all the hard graph transversals for us. [36:35.380 --> 36:37.860] So it will look a bit like this. [36:37.860 --> 36:47.600] You can just git clone, code archive.org, timestamp, github.com, username and repo name. [36:48.420 --> 36:51.340] Or even ask us to give you all the refs. [36:51.500 --> 36:56.560] So if you ask us for a timestamp, we'll give you how the repo looked like in that day. [36:57.000 --> 37:02.240] And otherwise we'll just give you all the refs just prefixed with the timestamp. [37:04.220 --> 37:06.660] However, again, this is expensive. [37:06.980 --> 37:10.300] So we'll have to ask you for one more step. [37:12.960 --> 37:29.040] And the idea is that you shouldn't really use this in CI, for example, to replace a repository that is gone, fetching every day or every build, because that will cost us a lot of money when instead you could just fetch it once so that you have it. [37:29.160 --> 37:33.460] And then we upload it on your GitHub account or whatever new service we are using now. [37:34.140 --> 37:46.920] So what we are doing is we are using the git side channel, which is the one that allows git to print stuff on your terminal, like counting objects and that kind of thing, to show you a link. [37:47.460 --> 37:52.000] You click on this link and you get on a page where we check that you are human. [37:52.600 --> 38:03.080] You click the box, feel that annoying captcha, and as soon as you submit the captcha, we let the git fetch proceed. [38:03.660 --> 38:16.640] This is obviously annoying, but it's something you can do to recover some repository that otherwise would be lost, but will stop you from putting it into CI and just being like, well, they will pay for it. [38:19.400 --> 38:27.600] And finally, there will be a web UI, which obviously has less bandwidth requirements, but as we said, we are terrible at building web UIs. [38:27.760 --> 38:30.560] So contributions for that are very welcome. [38:33.900 --> 38:38.000] Yeah, so that's kind of the state as it is right now. [38:38.720 --> 38:40.540] Obviously, it doesn't end there. [38:40.740 --> 38:42.220] There's some things that we want to do. [38:43.480 --> 38:48.920] GitHub's great and git is amazing, but there's also other places that people store their code at. [38:49.320 --> 38:50.740] There's Bitbucket and GitLab. [38:50.960 --> 38:58.080] There's also other SCMs, like Fossil and Mercurial and Bizarre, if anybody actually uses that. [39:00.740 --> 39:04.000] So we'd like to kind of eventually clone these as well. [39:05.380 --> 39:15.280] And another thing that we feel that we might have to do, just to be responsible, I guess, is to allow people to hide their repositories from the archive. [39:15.880 --> 39:36.580] So we're thinking of having a thing where you log in with your GitHub credentials, like via the GitHub API, OAuth, and mark the repos that you own to be hidden from the archive, because for some legit reasons, people might not want their repos archived. [39:36.580 --> 39:44.820] And also, we need to deal with the MCAs, and we're going to try to automate that somehow, just to form. [39:46.900 --> 39:48.200] Okay, so... [39:48.200 --> 39:48.460] Sorry. [39:48.660 --> 39:49.040] Sorry. [39:49.740 --> 39:50.060] Braces. [39:50.440 --> 39:51.200] You do it. [39:51.420 --> 39:51.540] Yeah. [39:52.200 --> 40:01.900] So, honestly, all of this requires a lot of storage, and this is a side project, as we said, so we couldn't just go out and pay for the storage ourselves. [40:01.900 --> 40:07.160] So we were looking for a partner, someone to sponsor the storage for us. [40:07.880 --> 40:08.160] Just... [40:08.160 --> 40:12.900] Literally just two nights ago, we finally found one in Backblaze. [40:13.960 --> 40:25.700] Backblaze launched a new product, which is B2, which is their object storage system, similar to the Google object storage or S3. [40:26.400 --> 40:33.140] And they accepted to give us a huge allowance for free for the next five years. [40:33.800 --> 40:34.320] So... [40:34.320 --> 40:41.520] And committed to letting us get the data out, if at the end of that we will not proceed with the relationship. [40:43.400 --> 40:44.440] So, interesting... [40:44.440 --> 40:48.060] Something I learned from this is that cold emailing actually works. [40:48.320 --> 40:56.800] It might take a couple of weeks, but cold emails actually work, and companies sometimes actually go out of their way to make projects possible. [40:57.640 --> 41:07.060] Nevertheless, we would be obviously happy to have multiple storage systems and redundancy, so our emails are on the landing page. [41:07.320 --> 41:08.480] So, do get in contact. [41:10.260 --> 41:11.760] And this is it. [41:12.220 --> 41:15.240] We've seen a need for something that should exist. [41:15.520 --> 41:20.380] We were incredulous that it didn't exist yet, and we went out and built it. [41:20.380 --> 41:22.160] You can go at that link. [41:22.400 --> 41:28.580] It will show you essentially nothing, because it's a landing page, because indeed the front end is not done yet. [41:28.900 --> 41:41.840] But there's one of those annoying email boxes where you can type your email, and we'll email you to tell you about when it actually launches, and then forget about your email addresses forever, I swear. [41:42.440 --> 41:46.320] The code is also on GitHub, so clone it and help us. [41:46.540 --> 41:46.660] Yeah. [41:47.040 --> 41:52.080] Oh, by the way, if you could just follow that link, go to the repository and star it. [41:52.320 --> 41:56.900] If we can pass the 10 stars threshold, we'll start archiving ourselves. [41:58.640 --> 42:01.600] It was kind of sad that we aren't doing it yet, so please do. [42:03.400 --> 42:04.380] Okay, questions? [42:07.530 --> 42:08.350] Is that Mike? [42:09.110 --> 42:09.450] Mike. [42:11.950 --> 42:24.270] First of all, I just wanted to say that I'm really into software archiving, but I do the really old stuff from the late 50s to the present, and I've got a large archive that I work on for a website called vintagecomputer.net. [42:25.470 --> 42:35.750] But aside from that, I'm also into big data, and I'm looking at this and I'm thinking, wow, you could be getting some interesting metadata from these files and creating a database with that. [42:35.750 --> 42:40.630] You might find some interesting authors and connections of Git. [42:41.070 --> 42:47.610] You might be able to create a web of where people came from, and you might be able to actually amass some interesting statistics from that. [42:47.810 --> 42:49.350] So I just thought you might want to think about that. [42:49.470 --> 42:50.090] We shouldn't be too hard. [42:50.690 --> 42:51.750] That's definitely true. [42:51.990 --> 42:58.090] And we had a lot of fun playing with the data set that Google released with BigQuery recently. [42:58.090 --> 43:04.290] They did a similar thing where they just cloned a lot of Git repos from GitHub. [43:05.110 --> 43:18.210] The main difference of that is that it's a snapshot, so it's very good for data analysis, but it's not really about historical archiving because it's a specific point in time snapshot of the state. [43:18.430 --> 43:19.270] But definitely. [43:23.250 --> 43:26.650] Did you guys discover any particularly interesting repos? [43:27.970 --> 43:37.530] Like, I don't know, but like that Trello page, just scrolling through it was really interesting for me because I was just like, oh, I've never heard of this, but this sounds really cool. [43:37.790 --> 43:39.250] Oh, that sounds really amazing too. [43:39.470 --> 43:44.490] So next thing I just end up with like 20 tabs of just cool repos that I want to look at and code amazing. [43:44.790 --> 43:55.250] To be honest, interestingly or actually unsurprisingly, the repos that disappear or that crash your Git implementation end up being the most interesting. [43:55.250 --> 43:56.670] So, yeah. [43:57.570 --> 44:02.190] We have at least seen a lot of kind of weird Git corner cases. [44:03.970 --> 44:06.650] Kind of playing on that question, I'm just curious like how often... [44:06.650 --> 44:09.970] You mentioned at the beginning like a lot of your motivation for the project and I think you... [44:09.970 --> 44:11.470] I think it's a legit motivation. [44:11.630 --> 44:23.230] I'm just curious like if you have any statistics on how common various events are that might make Git inadequate, whether like repos being deleted or like rebasing or other stuff like that. [44:23.410 --> 44:23.610] Okay. [44:23.910 --> 44:35.890] Well, obviously we don't have like a statistical sample of global critical infrastructure projects that failed because that's something that happens in the span of decades. [44:36.190 --> 44:41.950] But Gittorius definitely was such a Git host that completely went off the network. [44:41.950 --> 44:45.670] And we only have that because the archive team grabbed a copy. [44:46.650 --> 44:47.350] I might be... [44:47.350 --> 44:53.670] Don't quote me on this, but I think I remember it being the archive team and just grabbed a copy at the last minute. [44:54.130 --> 44:58.310] Obviously, we would like not to find ourselves in the same situation with GitHub. [44:58.310 --> 45:00.150] What about things like history being written? [45:00.290 --> 45:05.030] Have you kind of looked into that, gathering statistics on that or is that just later down the road? [45:05.190 --> 45:10.330] We've only got a few weeks worth of data here, but there's something that I wanted to investigate. [45:10.610 --> 45:15.650] It's just like, you know, once we've got this running for a few months, it'd be like, how often do people rewrite history? [45:15.810 --> 45:17.610] How often do people just delete their repos? [45:18.430 --> 45:22.850] Basically the same questions we wanted to ask them with the data, once we have the data for like a longer period of time. [45:23.950 --> 45:27.310] I noticed that all of the questions are about how to use the data set. [45:27.310 --> 45:31.430] We would obviously be more than happy to give people access to the data set. [45:31.870 --> 45:44.170] The obvious problem is that retrieval is costly, so we can't just give some complete access because retrieving the entire tree requires going to cold storage. [45:44.470 --> 45:55.730] So we probably can't give an automated way, but if anyone wants to email us with ideas for things to run or even funds to sponsor them, we would be more than happy to help with that. [45:55.730 --> 45:57.950] By the way, you hit 11, you are at 11 stars. [45:58.250 --> 45:58.770] Say again? [45:59.030 --> 46:00.270] You're at 11 stars. [46:00.790 --> 46:01.450] 11 stars! [46:01.470 --> 46:02.570] Oh wow, that's great, hooray! [46:02.750 --> 46:02.930] Yes! [46:03.130 --> 46:04.010] Like, end of hour. [46:04.010 --> 46:05.470] End of hour, we should probably... [46:05.470 --> 46:08.530] In 15 minutes, we will be in our own archive. [46:08.730 --> 46:11.090] Thank you, all seven of you. [46:13.470 --> 46:17.750] Did you run into any issues with Git LFS, and do you support sub-modules? [46:19.210 --> 46:19.770] Okay. [46:20.410 --> 46:31.070] Sub-modules are completely abstracted away from the Git low-level protocol, so they look like just text and a Git-modules file that says where to find the sub-module. [46:31.350 --> 46:37.990] So probably if you have a repository with some modules that are not app themselves, you will... [46:37.990 --> 46:42.990] Cloning the first time will fail, and you have to correct the URL to point it to us. [46:43.110 --> 46:47.290] We don't want to change the data we are serving, so I don't really see a way around that. [46:47.610 --> 46:52.130] But I'm pretty sure there's somewhere some feature of Git that takes care of that, because there are so many. [46:53.710 --> 46:54.490] Git LFS. [46:55.350 --> 47:02.010] Honestly, I think that's completely, like, parallel to the Git protocol, so we're not archiving that. [47:02.210 --> 47:06.850] And I suspect, obviously, being a large file system, it might be, you know, costly. [47:07.030 --> 47:10.510] But I'd be happy to investigate that, or take poor requests. [47:11.410 --> 47:11.930] Thanks. [47:12.290 --> 47:12.710] Thank you. [47:13.970 --> 47:14.410] Hi. [47:15.150 --> 47:22.010] This might be an obvious question, but you had mentioned something about bandwidth being cheap at night. [47:22.110 --> 47:27.810] So I was wondering if it mattered where the Git servers were and when night time was for them, if you had to think about that at all? [47:30.210 --> 47:35.950] So, obviously, what we are trying to optimize for is not our expenses, but GitHub expenses here. [47:35.950 --> 47:47.070] So we kind of throw a guess at what they look like by looking at another popular company graphs. [47:47.570 --> 47:51.310] And we just decided, based on that, when to run. [47:51.470 --> 47:53.390] Again, we might have gotten completely wrong. [47:53.570 --> 48:03.830] So if GitHub comes and says, actually, you're starting while we still see a lot of traffic from the West Coast, so please don't do that, because it would be cheaper if you started two hours later. [48:04.030 --> 48:07.410] We are more than happy to just shift it. [48:07.910 --> 48:12.130] What we are trying to do is to, like, if you look at bandwidth graphs, they all look like this. [48:12.330 --> 48:16.190] We are trying to feel the bottom of the GitHub bandwidth graph. [48:16.190 --> 48:17.410] That's interesting. [48:17.570 --> 48:17.870] Thank you. [48:18.030 --> 48:18.470] Thank you. [48:20.290 --> 48:31.910] You may have touched on this in the talk, but I'm just curious, like, how much, how resilient or redundant is your system in the event, like, let's say your Internet connection goes down where your server is for, like, a day or something. [48:32.010 --> 48:32.970] Do you lose data? [48:33.150 --> 48:34.230] What happens in that case? [48:36.350 --> 48:45.250] So, obviously, we would lose data if, in that span of time, someone pushes something and then rewrites history on top of that. [48:45.250 --> 48:50.610] Or deletes the repository in the span between when we go down and when we go back up. [48:50.930 --> 48:56.290] If nothing is destroyed in the meantime, we'll go back and backfield the work. [48:56.730 --> 49:06.430] But definitely, if you're quick enough at rebasing something and it's daytime, so we're not actually fetching, we will lose some detail. [49:06.990 --> 49:10.870] It's a compromise with the being nice part. [49:11.090 --> 49:11.370] Yeah. [49:11.730 --> 49:11.850] Thanks. [49:15.360 --> 49:15.960] Hi. [49:16.160 --> 49:28.160] The drinker caching and queuing seem like an interesting technical problem and I was just curious what the implementation you were using to solve that. [49:29.860 --> 49:31.080] Go for the road, okay. [49:31.260 --> 49:40.600] So, the cache is, right now, something like six million entries, which, you know, seem a lot, but turns out computers got pretty fast. [49:40.760 --> 49:41.020] Yeah. [49:41.020 --> 49:43.560] So, we just literally threw a BoltDB at it. [49:44.000 --> 49:44.080] Okay. [49:44.180 --> 49:51.300] And it's a key value store with the repo name and value the number of stars. [49:51.760 --> 49:55.040] It's clocking at, I don't know, 500 megabytes or something. [49:55.920 --> 49:59.800] It's, no, sorry, five gigabytes of cache. [50:00.320 --> 50:04.520] So, in the queuing mechanism that you're using, what are you using for that? [50:04.540 --> 50:05.940] Oh, this is so embarrassing. [50:06.260 --> 50:20.180] We literally have a MySQL table with a column that has unique key constraints, and when we add a line, we say, add a line, or if it fails, whatever, it probably is already queued. [50:20.340 --> 50:20.620] Okay. [50:20.620 --> 50:21.680] This is not a real queue. [50:21.880 --> 50:22.960] Don't go home and do that. [50:24.140 --> 50:25.520] It's functionally a queue. [50:28.040 --> 50:32.180] And just one more implementation question. [50:32.460 --> 50:38.780] When you're actually manipulating the Git repository, what are you... [50:38.780 --> 50:42.920] Are you using, like, the Git C library wrapper for something else? [50:45.440 --> 50:48.120] For the cloning part, we just wrote our own implementation. [50:48.300 --> 50:48.600] Okay. [50:48.960 --> 50:49.240] In Go. [50:49.580 --> 50:50.440] Oh, okay. [50:50.600 --> 50:54.140] So you basically simulated Git client in Go. [50:54.140 --> 50:54.400] Exactly. [50:54.720 --> 50:54.980] Thank you. [50:55.140 --> 51:02.580] Git has this very interesting, like, feature negotiation that shows how the protocol is very old. [51:02.740 --> 51:20.820] And, for example, how you signal the features your client supports is by putting a zero byte at the end of one of the lines, the first line, and then putting the capabilities after that, which I have no proof, but I suspect it's because this way old clients that don't have these, [51:20.820 --> 51:27.420] you know, features which support, would just see that zero byte, and stop parsing the line, and it would still work. [51:27.600 --> 51:27.920] Okay. [51:27.920 --> 51:30.500] I don't know if it's that, but I like to think it is. [51:30.680 --> 51:31.000] Okay. [51:31.120 --> 51:31.400] Thank you. [51:32.060 --> 51:32.380] Thank you. [51:33.120 --> 51:38.980] So would you be open to a manual request to Archive, instead of getting 10 stars? [51:40.020 --> 51:40.880] Definitely, yeah. [51:41.200 --> 51:41.380] Cool. [51:41.380 --> 51:43.820] It would be great to have, like, another field somewhere where you can just add it. [51:44.120 --> 51:44.380] Yeah. [51:44.460 --> 51:44.620] Awesome. [51:44.780 --> 51:49.140] And have you reached out to the Archive team, to partner? [51:49.460 --> 51:50.340] Because I think they may be open to it. [51:50.340 --> 51:50.900] Not quite yet. [51:50.900 --> 51:52.840] We haven't had time yet. [51:53.140 --> 51:56.760] We would be very happy to, like, offer them an API or something. [51:56.760 --> 51:57.440] I'll try to connect you guys. [51:57.720 --> 51:59.740] Or, like, a Trello login. [52:00.840 --> 52:02.200] Make an account on Trello. [52:02.260 --> 52:04.980] Go to Jason Scott's talk and talk to him. [52:05.680 --> 52:05.960] Wonderful. [52:06.180 --> 52:06.320] Yes. [52:09.780 --> 52:14.880] So, one thing came to mind when you guys said you're using MySQL for the queue. [52:15.060 --> 52:16.000] Why not use Redis? [52:16.100 --> 52:17.560] That's exactly what it's good for. [52:21.510 --> 52:22.710] That's why it exists. [52:22.710 --> 52:24.390] And that's how it... [52:24.390 --> 52:24.930] You're not wrong. [52:25.310 --> 52:29.810] We just had a MySQL running already and making a new table was easier. [52:31.470 --> 52:32.390] We had a... [52:32.390 --> 52:33.650] Feeding Redis is easier, too. [52:34.330 --> 52:36.490] There's probably a library for it. [52:37.030 --> 52:37.610] Yes. [52:38.270 --> 52:38.850] But... [52:38.850 --> 52:39.290] Kubernetes. [52:40.910 --> 52:43.970] You have to make a pod for it and figure out the ports. [52:44.510 --> 52:45.090] And... [52:45.090 --> 52:48.790] Did I say that Kubernetes was not our most brilliant idea? [52:49.990 --> 52:50.550] Indeed. [52:51.270 --> 52:52.270] ProRequest is welcome. [52:52.550 --> 52:53.630] It would definitely work better. [52:54.590 --> 52:56.130] I don't know those, though. [52:57.910 --> 53:01.690] Do you guys ever look at the Gideon, I think it is? [53:01.870 --> 53:11.150] The Gideon protocol, I guess, for looking at binaries and determining that they were compiled from a particular version? [53:11.990 --> 53:12.390] Ooh. [53:12.710 --> 53:13.450] I haven't. [53:13.650 --> 53:14.070] That would be... [53:14.070 --> 53:14.670] Sounds cute. [53:14.770 --> 53:15.450] Sounds interesting. [53:15.650 --> 53:17.410] It sounds like something worth archiving. [53:17.510 --> 53:27.370] Because I was thinking like, yeah, I could go back and look at code like 10 years ago and like probably never be able to compile it because although you've archived most of the dependencies, there's some of the things that you haven't archived. [53:27.850 --> 53:38.210] In some ways, I feel like if you could archive secure binaries, binaries who've had Gideon hashes developed on them, then that might be something that you might want to... [53:38.210 --> 53:38.510] I don't know. [53:38.670 --> 53:39.770] For really popular things. [53:39.950 --> 53:40.670] Sounds awesome. [53:40.770 --> 53:41.650] Let's talk about that. [53:41.830 --> 53:41.890] Yeah. [53:42.010 --> 53:44.370] Send us an email or catch up later. [53:45.250 --> 53:45.430] Definitely. [53:45.930 --> 53:56.850] Something we are thinking about is also the Debian sources archives, which are already well structured, and the Homebrew formulas, all the turbos they point to, which already have all the dependencies. [53:57.510 --> 53:58.670] Alright, final question. [53:59.090 --> 54:09.170] Yeah, so I guess the guy that invented the BFG tool had said that the Java library was pretty quick, and I'm curious what the... [54:09.170 --> 54:21.670] and that they had shared ideas, I guess, back and forth between the Git and the Java library, and I'm curious what your speed difference is and whether you had gained any ideas and there was any sharing there. [54:22.490 --> 54:25.090] We haven't really looked at the Java library. [54:26.170 --> 54:33.590] I mean, it would be interesting to compare at some point, but we do go all the time, so we just use what we know. [54:34.970 --> 54:35.330] Awesome. [54:35.650 --> 54:35.930] Okay. [54:36.170 --> 54:36.790] Thank you very much.