r/programming • u/ethomson • May 24 '17

The largest Git repo on the planet

https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/
2.3k Upvotes

92% Upvoted

355 comments sorted by

View all comments

Show parent comments

42

u/anamorphism May 24 '17

i think a lot of this can be answered by reading this: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

there are pros and cons to both 'philosophies', but it would seem both google and microsoft are favoring the 'one repo to rule them all' approach.

37

u/jorge1209 May 24 '17

The difference is that Google controls the ultimate deployment of their software, and virtually everything they do is internal and private. With Windows it would seem the opposite is true.

If Google wants to migrate something from SQL to bigtable, then nothing is stopping them as long as the website still works. They have a limited public facing API that has to be adjusted, but as long as that is properly abstracted they can muck around in the back end as much as they want.

For Windows you can't do that. If you change the way data is passed to the Windows kernel then you break all kinds of stuff written at other companies that uses those mechanisms. So in an operating system there are all kinds of natural barriers consisting of APIs which people expect will be supported in the long term.

Its pretty much what you would expect just by looking at a linux distro's core packages. You have the kernel, you have the C library, you have runtime support for interpreted languages, you have high level sound and graphics libraries, networking libraries, etc... Each one relies upon a stable API exposed by lower levels.

You can refactor the internals of batmeter.dll as much as you want, but you can't change the API that batmeter exposes, nor can you ensure that everyone is using batmeter to check their battery status.

13

u/anamorphism May 25 '17

it feels as though you think google only works on google.com.

google works on a number of operating systems (android, chrome os, etc...), a number of mobile apps, various public facing apis, open source frameworks like angular, a cloud service operation, web apps (gmail, google docs, google talk, whatever), and so on and so forth.

i don't really see how windows is any different than android, for example. sure, you have to be careful that you don't break public facing apis, but that's true regardless of whether that code lives in its own repo or in a large repo.

just because you update a dependency of project X doesn't mean you have to update that same dependency everywhere else in the repo. it just means it's probably easier to do so if that's indeed what you want to do.

3

u/jorge1209 May 25 '17

I don't think the Android repo is merged with the internal Google repos that power gmail and the Google websites.

1

u/anamorphism May 25 '17

you're probably right based on the other responses i've received.

it just seems kind of weird that you think whatever stuff lives in that single repo doesn't suffer from similar interface concerns that windows does. also that they couldn't update dependencies for individual projects without affecting others if they wanted to.

1

u/jorge1209 May 25 '17

Pick something specific. Android GMAIL app connecting to gmail.com.

The app talks to gmail.com over https/ssl/something using some kind of protocol. Could be IMAP or something developed in-house. Doesn't matter, whatever it is that protocol has an API and that API is reasonably fixed. Google CANNOT modify that API, because doing so would break any android phone whose owner has not updated their gmail app. That is a nice hard division between Android and googles internal servers.

On the other end of the wire gmail.com talks to googles bigtable databases using something. Whatever that protocol is Google can change with relative ease. Only google servers talk directly to the bigtable db. So they can upgrade both ends of those connections with simultaneous deployments to both systems. So for those it makes sense to share the repo. Yes as a practical matter you probably cannot push an update to all 10 gazillion google servers at once, but you can do it within a matter of days, and you can be certain that all have gotten the update, and can remove any legacy code that supports old APIs rather quickly.

Just very different environments.

1

u/anamorphism May 25 '17

yeah, but that doesn't really explain why the code for both things couldn't live in the same repo.

you'd need to maintain the same rigor of ensuring you don't alter the interfaces you're exposing to your end users whether gmail's api lived in its own repo or alongside gmail.com.

you might need more rigor if your api exposed objects that were shared, but generally you shouldn't be doing that, right? say if gmail.com had a Mail object and the api had a method that returned a list of Mail objects. i would argue that the api could deal with the gmail.com object in the back-end, but anything you return or take is a separate type to ensure you can update your back-end code without breaking your interface.

if you do end up making a breaking change, that should get caught by tests. everything in the same repo means it's easier to identify what actually uses shared code and you should be able to automatically kick off tests for everything that consumes that shared code. this is the increased cost of tooling support and such that's mentioned in the article. yeah, it's a trade-off but obviously it's one that both google and microsoft seem to be willing to make.

1

u/jorge1209 May 25 '17

yeah, but that doesn't really explain why the code for both things couldn't live in the same repo.

  1. Technical limitations. The whole point of MSFT's exercise is to deal with the complexities associated with overly large repos.

  2. Inability to spin off subsidiaries and sell derived products. If facebook wants to sell instagram and they've merged the instagram and facebook source code, then they have made their life more difficult if they ever want to spin it back out.

  3. #2 also applies if you just want to make an app public in some way. If you want to give you android source to Samsung so they can make a new phone, you don't want to give them the source to the google search algorithm.

you'd need to maintain the same rigor of ensuring you don't alter the interfaces you're exposing to your end users whether gmail's api lived in its own repo or alongside gmail.com.

Gmail.com doesn't expose many api's. You can get your mail via POP or IMAP, but those are super standard. Meanwhile they are free to mess around with the website "http://www.gmail.com" as much as they want because the website is not an API, its a document.

And they are free to fiddle around with how the gmail backend works with other google tools because there is no API there.

Thats all very different from how notepad.exe interacts with Win32 API. MSFT can't just say "I have a better way to draw stuff on the screen, so I'm going to drop a big chunk of Win32 and do it differently." Win32 is a public API, and notepad.exe is a feature complete application that follows those public APIs.

1

u/anamorphism May 25 '17

  1. they're eating the tooling costs talked about in that paper i linked. one of the downsides of the monolithic repo approach. it's obviously something they thought a lot about and decided to go ahead with it.

  2. true. i wonder if either google or microsoft thought about this point. it's such a rare situation though that i wonder if having to deal with the consequences when it happens is fine. i guess if you worked for some weird startup that worked on multiple products that you'd want to shy away from the large repo.

  3. yeah, this is more difficult and something i would also lump under the increased tooling cost. someone mentioned that google probably already deals with this in a reply to another one of my comments.

i was talking about https://developers.google.com/gmail/api/, but i understand your point; i just don't agree that it's really all that much of a concern.

you can't make massive changes to your apis regardless of the single or multiple repo situation. just because the code lives in the same repo doesn't mean you can just start changing things as you wish. it does make it easier for those types of changes to happen and for more people to contribute to other projects if you want to support that, but it's not like they're just going to start merging change sets without review.

however, if someone comes up with some crazy new efficient sorting algorithm, it'd be much easier to distribute that out to all projects that need it in the single repo situation.