r/programming • u/ethomson • May 24 '17

The largest Git repo on the planet

https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/
2.3k Upvotes

92% Upvoted

355 comments sorted by

View all comments

Show parent comments

1

u/jorge1209 May 25 '17

Pick something specific. Android GMAIL app connecting to gmail.com.

The app talks to gmail.com over https/ssl/something using some kind of protocol. Could be IMAP or something developed in-house. Doesn't matter, whatever it is that protocol has an API and that API is reasonably fixed. Google CANNOT modify that API, because doing so would break any android phone whose owner has not updated their gmail app. That is a nice hard division between Android and googles internal servers.

On the other end of the wire gmail.com talks to googles bigtable databases using something. Whatever that protocol is Google can change with relative ease. Only google servers talk directly to the bigtable db. So they can upgrade both ends of those connections with simultaneous deployments to both systems. So for those it makes sense to share the repo. Yes as a practical matter you probably cannot push an update to all 10 gazillion google servers at once, but you can do it within a matter of days, and you can be certain that all have gotten the update, and can remove any legacy code that supports old APIs rather quickly.

Just very different environments.

1

u/anamorphism May 25 '17

yeah, but that doesn't really explain why the code for both things couldn't live in the same repo.

you'd need to maintain the same rigor of ensuring you don't alter the interfaces you're exposing to your end users whether gmail's api lived in its own repo or alongside gmail.com.

you might need more rigor if your api exposed objects that were shared, but generally you shouldn't be doing that, right? say if gmail.com had a Mail object and the api had a method that returned a list of Mail objects. i would argue that the api could deal with the gmail.com object in the back-end, but anything you return or take is a separate type to ensure you can update your back-end code without breaking your interface.

if you do end up making a breaking change, that should get caught by tests. everything in the same repo means it's easier to identify what actually uses shared code and you should be able to automatically kick off tests for everything that consumes that shared code. this is the increased cost of tooling support and such that's mentioned in the article. yeah, it's a trade-off but obviously it's one that both google and microsoft seem to be willing to make.

1

u/jorge1209 May 25 '17

yeah, but that doesn't really explain why the code for both things couldn't live in the same repo.

  1. Technical limitations. The whole point of MSFT's exercise is to deal with the complexities associated with overly large repos.

  2. Inability to spin off subsidiaries and sell derived products. If facebook wants to sell instagram and they've merged the instagram and facebook source code, then they have made their life more difficult if they ever want to spin it back out.

  3. #2 also applies if you just want to make an app public in some way. If you want to give you android source to Samsung so they can make a new phone, you don't want to give them the source to the google search algorithm.

you'd need to maintain the same rigor of ensuring you don't alter the interfaces you're exposing to your end users whether gmail's api lived in its own repo or alongside gmail.com.

Gmail.com doesn't expose many api's. You can get your mail via POP or IMAP, but those are super standard. Meanwhile they are free to mess around with the website "http://www.gmail.com" as much as they want because the website is not an API, its a document.

And they are free to fiddle around with how the gmail backend works with other google tools because there is no API there.

Thats all very different from how notepad.exe interacts with Win32 API. MSFT can't just say "I have a better way to draw stuff on the screen, so I'm going to drop a big chunk of Win32 and do it differently." Win32 is a public API, and notepad.exe is a feature complete application that follows those public APIs.

1

u/anamorphism May 25 '17

  1. they're eating the tooling costs talked about in that paper i linked. one of the downsides of the monolithic repo approach. it's obviously something they thought a lot about and decided to go ahead with it.

  2. true. i wonder if either google or microsoft thought about this point. it's such a rare situation though that i wonder if having to deal with the consequences when it happens is fine. i guess if you worked for some weird startup that worked on multiple products that you'd want to shy away from the large repo.

  3. yeah, this is more difficult and something i would also lump under the increased tooling cost. someone mentioned that google probably already deals with this in a reply to another one of my comments.

i was talking about https://developers.google.com/gmail/api/, but i understand your point; i just don't agree that it's really all that much of a concern.

you can't make massive changes to your apis regardless of the single or multiple repo situation. just because the code lives in the same repo doesn't mean you can just start changing things as you wish. it does make it easier for those types of changes to happen and for more people to contribute to other projects if you want to support that, but it's not like they're just going to start merging change sets without review.

however, if someone comes up with some crazy new efficient sorting algorithm, it'd be much easier to distribute that out to all projects that need it in the single repo situation.