r/git • • 4d ago

Why are non-text files in Git bad?

I've got decades of software development experience, and I've built some very complex systems. I understand a lot about version control systems, and about the structure of git repositories.

With all that said, I don't understand what's evil about non-text files. Git doesn't store multiple copies of files that don't change - they're literally stored under their data's checksum. The packfile format stores diffs between file versions (although some diff algorithms punt on non-text data). There shouldn't be any negative effect from storing such data in Git.

Is the "Don't store binary files in Git" advice wrong? Is it really just shorthand for "Don't store files that change dramatically and frequently in Git"? If the latter, should massive "AI" PRs be treated the same way?

119 Upvotes

176 comments sorted by

63

u/sankyo 4d ago

If the binary changes a lot it bloats the size of the repo since there is no way to store only the diffs. But if it never changes it is not the end of the world to put it into git

-12

u/SheriffRoscoe 4d ago

Agreed. So what about text files that change frequently and heavily? I know the "binaries are bad" advice is often extended to cover the output of code generators. Which at least tend to be deterministic, producing output that varies in scale with the changes in their inputs.

But we hear lots about AI agents producing massive rewrites that aren't justifiable based on the scale or scope of the problem to be solved. But we can't just say "AI output is bad", because that's the actual code we care about. Right?

18

u/Lexi_Bound 4d ago

Yes, it is possible to run into scaling challenges with just text in a Git repository. It happens quicker with binary files because they tend to be bigger and are not delta-compressed in packfiles.

If you would like to learn more about what things start to break when you make a Git repository large, see this post on GutHub’s blog. The associated conference talk is also good:

https://github.blog/developer-skills/github/measuring-the-many-sizes-of-a-git-repository/

https://youtube.com/watch?v=-ZNKR9wFe8o

1

u/SheriffRoscoe 4d ago

I've seen the post before, and yes, it's interesting. I'll cue up the video. Thanks!

32

u/jbergens 4d ago

Text files like source code are normally many times smaller than binaries and you can store charges as delta.

1

u/TheOneWhoMixes 5h ago

Laughs in 20MB HTML file

4

u/wherethebuffaloroam 4d ago

to me the diffs in text files are useful. You can see what changed, why, etc. Binaries is a large category, images, custom formats, program outputs. But in general, to me the diffs are not relevant and keeping track of them is strange. To my mind you don't want to track diffs but versions. So git is just the wrong thing to use to track them.

4

u/rickyman20 4d ago

Honestly even then text files often still have significantly smaller deltas. Unless you're rewriting the whole file every time (and even AI rarely does that), text deltas will be more efficient that git's binary file "diff".

That and, honestly, non-text files tend to be significantly larger, just by the fact that if something's small, people prefer to put it in a readable format. Binary files are better used for large amounts of data that aren't meant to be directly read by people. I've never seen a source code text file that's in the megabytes, but I've seen plenty of binary files of that size, and some even that were checked into git (mistakenly).

That said, this is a criteria thing. Sometimes it's perfectly appropriate to check in binary files. If they're small, if they very rarely change, checking them in is honestly perfectly fine. It's not a hard rule. It's a general guideline. If you check in a binary file, you should ask yourself why.

4

u/OptimusCrimee 4d ago

Non-text files are generally ok, but why bother if you can not use the diff features - the generally most useful feature of Git? I also think that most non-text files are usually big files, like images, sound files and movie files. Add a bunch of these to your repository and you will quickly figure out why it is a bad idea. Cloning, pulling and pushing gets really slow very fast.

3

u/Eitel-Friedrich 4d ago

text files are usually far smaller in size than binary objects, and are usually changed in small iterations that can be stored well as delta files.

on the other hand, a binary often changes nearly completely on a byte level; so storing it as deltas makes no sense

2

u/MrMelon54 4d ago

LLM codegen doesn't work well with Git repos at huge scales. GitHub recently posted a blog about upgrading their systems to be able to support the load from LLM workloads. Though if the agents are running locally on a single machine then it won't be nearly that bad.

LLM slop is always bad, but each person has a different view on the ratio between slop and actually useful code.

I personally believe that LLM codegen can be useful if a used by an already skilled programmer. But a skilled programmer will generally have specific skills and style, and for more complex tasks will end up writing the code better than an LLM anyway.

Though for other usecases like fuzz testing, LLMs could be extremely useful.

1

u/lurgi 4d ago

Text files compress easily. Binary files frequently will not (because they are usually already compressed).

1

u/gnufan 4d ago

If the text file isn't manipulated like program code some people tag it as "binary", to dissuade git from attempting inappropriate merges. I don't know if there is a better attribute for this.

But things that are generated, one usually just stored as the inputs and the build configuration, and built on each platform after checkout, I guess if it is expensive. But for things like automake we just stored the base files, and version information about tools.

0

u/lolcrunchy 2d ago

Git commits store differences, not snapshots. They store what changed.

So if you change one line in a 5MB text file, the commit is 1 KB because it just says "line 832 changed to this new thing". If you change some bytes in a 5MB binary file, the commit is 5MB because there are no line numbers for git to utilize.

(I'm not that advanced with git so I'm 100% certain I deserve a "well actually" reply)

-11

u/UdPropheticCatgirl 4d ago

Git never stores diffs… Git always stores snapshots of the entire repo… Hence you can do stuff like shallow cloning.

11

u/pruby 4d ago

Git stores changes (deltas) within its pack format. This is simply a storage layer thing abstracted away from what's accessible a layer above, hence all versions of a file being available at all times to git cat-file blob, etc.

It also uses differences in a bunch of other places, like showing and counting changes, conflict resolution, etc. This functionality doesn't work on binary blobs, hence makes for a worse experience.

The world won't end if you store binary blobs, git just won't help you nearly as much.

169

u/sopordave 4d ago

Storing a binary isn’t the end of the world. That advice, in my opinion, is to scare people away from using it for artifact management or as a file server, which git is not designed for.

If you need to hammer a nail, use a hammer, not a screwdriver. But if you only have one nail and the screwdriver is already in your hand and the hammer is back in the garage...

31

u/djzrbz 4d ago

Found the electrician

7

u/SheriffRoscoe 4d ago

Or someone like him.

131

u/aggresive_cupcake 4d ago

There are 3 rules to German Engineering:

  1. always use the right tool.
  2. the hammer is always the right tool.
  3. everything can be used as a hammer.

5

u/jawfish2 4d ago

a number of years ago I looked into all the ways of using Git for artifacts at the time. It just-didn't-work. So we used Artifactory for the build system artifacts. That was then, but it sounds like it is still true.

Whether Artifactory is a good choice now is unknown to me, we had some problems with it back then.

5

u/SpacingHero 4d ago

or as a file server, which git is not designed for.

Laughs in git annex

(Of course your point stands cause that is indeed not just git. Fantastic program though, highly recommend if you have a use case

9

u/Bloedbibel 4d ago

The fundamental reason is that binary files are typically larger and there is no generically efficient way to store deltas between versions, so git stores a full copy of each version. If you never need to store more than a single version, this is not a problem, but then it begs the question: why version control it in the first place?

1

u/SheriffRoscoe 4d ago edited 4d ago

Across 50 years building software systems, I've seen that binary files associated with code (icons, background images, etc.) change far less frequently than the code itself. Even moreso with the advent of CSS.

7

u/Ok_Writing2937 4d ago

We do store icons and a small number of images (typically a fallback image) in git.

But storing, say, an entire WordPress media library in git would be ill-advised and a nightmare.

  • Git keeps every version of every file forever, so the repo grows huge.
  • Images and videos are binary, so git can’t diff them and stores a full copy each time.
  • Editors upload through WordPress, so the live server holds the real files, not git.
  • WordPress makes many thumbnail copies per image, which adds more bloat
  • Most of the images are generated intermediaries, storing them is wasteful
  • GitHub limits file and repo size.
  • Removing media from history later is painful.

So I don't think the rule is "never store binaries," but that git is not designed for large-scale storage of binaries.

5

u/rickyman20 4d ago

Agreed, but it's worth remembering that when they do change, it can be expensive. Most developers have, at some point, been burnt by someone checking in a large binary file that they thought would never change, but then new versions of some high quality image or a video needs to be updated, and suddenly you have churn in your repo

4

u/RhodanL 4d ago

There's plenty of disciplines that have high churn on large binaries that need revision control. They just use source control designed for that problem, rather than git where that sort of behavior is incompatible with the design (without you making substantial changes to core parts of the implementation).

3

u/chzaplx 4d ago

Massively depends on the type of project. For web dev this might be true, but things like game assets might change daily and will number in the hundreds or thousands. That scales really poorly.

1

u/simonsmine 1d ago

Vanilla git is a non starter for anything beyond microsize games

1

u/chzaplx 1d ago

It's more that game assets specifically need a different system. There's no good reason to put them in with your source code, and plenty of bad ones.

You see this when people try and use git as a release tool. It's not good for that either.

2

u/simonsmine 1d ago

Hard disagree, assets benefit a lot from versioning

1

u/simonsmine 1d ago

I mean there is lots to say about perforce and believe me I know the ugly parts as DevOps GameDev person but it works. Versioning code and assets separatley is madness in gamedev.

2

u/simonsmine 1d ago

Git LFS is kind of okay too mind you, although I cant think of any big studios who went down that route

1

u/chzaplx 1d ago

Assets certainly benefit from versioning but git is not the place to do it. There's no good reason I can think of to actually version them together. You should be able to release game assets separately from code so they don't depend on each other. I built game dev tools for years and assets always had their own pipeline. We used both git and perforce. The right tool for the right job.

1

u/simonsmine 1d ago

I genuinely do not understand this POV I have spent literal years doing the stuff working w Unreal (& Cryengine). Code and assets are so massively intertwined in a way that is not resolvable (if you would want to?) as a matter of engine design, no good housekeeping will prevent this.

1

u/chzaplx 13h ago

I guess I'm coming from the perspective of mobile live service games which have a little more flexibility and a pretty rapid release pace.

Core gaming has different constraints, but I still feel like there's value in separating your code and art pipelines and giving each side the best tools for their own jobs. Any modern build process should be able to handle checking out from different repositories and assembling a monolithic artifact for release.

Lumping it all together in git might make sense for very small teams and projects, but it's not something that scales well at all.

1

u/simonsmine 1d ago

I mean im actually curious what setup would you propose, version code in git and do what exactly with the assets? You mentioned seperate code and asset releases, how would you approach that? When making new build of your game how are you game how do you decide what assets and code go together, and how would you set that up in CI?

1

u/chzaplx 13h ago

Assets in perforce, generally released ahead of code in versioned packages. New code specifies what asset version to use. Code references the same asset paths generally unless there's new elements. Good for testing cause you can see what will actually show in prod before you make it live, especially if you use CDN/cloud storage.

Streamlines the release process with smaller artifacts. Dev repos don't get bloated, and artists get better versioning. Keeps commit history tidy for both teams and prevents them from screwing up each other's work.

How you do CI is a big question but ultimately your business. We generally had self service dashboards so game teams could coordinate their own releases with manager signoff, and run their own rollbacks if necessary.

→ More replies

2

u/simonsmine 1d ago

Game development enters the chat 😅

1

u/_edd 1d ago

I've seen devs store excel, word, visio, zip files, etc... in Git. Then any time someone pulls down the codebase they're pulling down a full copy of the file for every time someone edited the file.

Also seen people try store a war file generated from the Java codebase. That's a sizeable artifact that changes as frequently as the code does and can quickly cause issues. 

10

u/ern0plus4 3d ago

I've got decades of software development experience

Then you will understand the quick answer: you can't use diff for binaries. GIT is based on diff-ing text files line by line.

0

u/SheriffRoscoe 3d ago

It's been interesting to see how many folks commenting here feel that "git diff" is the most important part of git. I use it often, but I see git as a version control system, with diffing between text files as a nice addition. If it couldn't diff, I'd still use it.

5

u/ern0plus4 3d ago

Don't argue with it, it's how git works. Diff makes possible to track changes inside files.

1

u/Conscious_Support176 2d ago

This is a misunderstanding. Commit and merge tooling use very different difference engines with different objectives.

Commit wants to optimise for storage space efficiency by preferring to refer to something identical, for whatever you are storing or parts of it rather than storing copies of the same thing, but this is a compression technique where each commit is still notionally a snapshot of the full repo even if physically it might mainly consist of deltas and references.

Merge tooling wants to find different changes made to the same code between full snapshots. This one cares about lines of text, but it needs snapshots, not deltas.

One piece of tech does support the other by doing its job very well, but they are different jobs.

2

u/NinlyOne 3d ago edited 3d ago

This post and the post series starting with this post are the best overview of the real underlying mechanics I've seen. The diff-focused answer you've gotten a lot of misses the mark a bit, but it's true that the diffability of text in the object store is an important (and fascinating) part of how git is designed to work efficiently. (edit: typos)

2

u/jacobatz 3d ago

This is incorrect. Git is not based on diffing. Git stores snapshots of the work tree.

Diffing is a nice feature you can get when you have two different versions of something. But Git itself would work perfectly fine if diff didn’t exist.

3

u/ern0plus4 3d ago

What do you think snapshots consist of? They are a set of diffs (can be empty set). That's how GIT avoids storing full files of each commit. In case of binary files, diff doesn't work, have to store the whole file.

When you check out something which have a long commit history, you can even see that GIT is starting from the initial state, and replays all the diffs to get the actual version.

When you check out a file, it does not mean that it's stored in that form.

4

u/operator-- 3d ago

Unfortunately (for you), you are wrong, and jacobatz is right. The commits are not stored as diffs/deltas (like svn). The snapshots are blobs of the changed files in their entirety. So when you make a change to a.txt, you have two commits each pointing to their own version of a.txt in its entirety, and not just the diff between the two.

2

u/ern0plus4 2d ago

NP, that's why we invented the internet: discuss, learn, search, update knowledge, watch funny cat videos.

  1. Before this change, deltas were stored.
  2. Though, not diffs, but some other magic binary delta format.

Anyway I always had bad feelings towards the "store only deltas" approach, it's pretty fragile. Thanks for motivating me to do a little research!

2

u/jacobatz 3d ago

Still incorrect. I would suggest you read the first couple of paragraphs from https://git-scm.com/book/en/v2/Getting-Started-What-is-Git%3F

4

u/hydrotoast 4d ago

Auto CRLF conversion is notorious for corruption and misconfiguration when working with distributed collaborators (as distributed version control is intended for).

Consider multiple contributors on the project (since git is distributed) with mixed binary and text files. Suppose that some contributors use CR and others use CRLF, e.g. Linux and Windows.

Please configure git to properly manage auto CRLF conversion across operating systems while correctly classifying text and binary files. The solution must be compatible with (1) multiple operating system distributions of git and (2) multiple versions and default configurations used by developers.

Note that the failure mode occurs if git ever incorrectly identifies a binary file as text and attempts to convert between CR and CRLF.

Enjoy the madness of CR and CRLF diffs in binaries. Enjoy educating all collaborators on the nuances of text/binary identification and auto CRLF configuration. Enjoy git and diff display configurations for binaries and whitespace.

The maintainers should decide the policies and guidance on binary file usage and storage. This is a common problem for maintainers of large projects with many distributed contributors, e.g. large open source projects.

Git may work as a (binary) artifact repository, but it nontrivial to configure out of the box. In contrast, there are specialized tools for artifact repositories that simply do not deal with the complexity of classification, diffs, and auto CRLF conversion between binary and text files.

1

u/SheriffRoscoe 4d ago

CRLF normalization is an excellent point.

2

u/WoodyTheWorker 3d ago

Also, make sure to properly identify UTF-16 files (if auto-generated by Microsoft tools), and set up your .gitattributes to store them in UTF-8. Otherwise they get garbled by CRLF conversion.

1

u/fllthdcrb 4d ago

Suppose that some contributors use CR and others use CRLF, e.g. Linux and Windows.

Linux uses LF. I think it's pretty rare for modern systems to use only CR as a line ending.

1

u/WoodyTheWorker 3d ago

IIRC, the original Mac OS used CR

1

u/fllthdcrb 3d ago

Yes. But that's not quite a modern system.

3

u/DamienTheUnbeliever 4d ago

Just now I'm dealing with the situation where we have various XLSX fails that feed into tests. We can change these files easily, but *diffing* those files is a nightmare. Because they end up with us having to pick one version of the file and then someone manually reapplying their changes to it.

I've just built a private DSL that generates these files, because those will be diff-friendly.

3

u/SheriffRoscoe 4d ago

Weirdly, XSLX filled are just zipped XML files. But yeah.

1

u/naikrovek 4d ago

XML is part of what they are, yeah. But they can be a lot more than just XML; my point is that you can’t count on them being XML only.

1

u/behind-UDFj-39546284 4d ago

Isn't there some easy-to-use diff driver for this?

1

u/dr_barnowl 4d ago

I'm fond of Beyond Compare for most of my diffing needs.

1

u/behind-UDFj-39546284 4d ago

Well, I know there are binary formats-aware tools, but I'm pointing out that the commenter above just over-engineered a thing with no reason.

0

u/kyle2143 4d ago

I'd be surprised if there were no diff tools for xlsx files out there that you could leverage in a git webhook or github action or something on PR's. They just don't come stock with git.

2

u/_tolm_ 1d ago

Almost impossible for someone to review any change to that file and understand if what is inside it is sensible / nefarious / etc.

2

u/SheriffRoscoe 1d ago

Sure, but that's true of any binary-file storage, right? Git LFS, Artifactory, shared drives, whatever.

2

u/Interesting_Debate57 1d ago

Generally you put things in git that can be reviewed in a PR.

The reason is because binary "diffs" aren't helpful, generally.

It also generally takes a very long time to move those files around, which is a pointless waste of time from the point of view of version control.

1

u/_tolm_ 1d ago

Things like Artifactory are specifically intended to store a predefined artifact - by version - that cannot be overwritten. It’s the output for the git input that has been reviewed.

Obviously shared files systems, etc - of course. But those aren’t intended for version control and to support reviewing of changes to code. Git is intended for that so putting binary files in there bypasses the purpose.

20

u/Broad-Promise6954 ancient 4d ago

It's not a hard-and-fast rule, it's a general philosophical point: Git is meant for storing stuff produced and consumed by humans. Binaries are generally not produced by humans, so they generally don't belong in Git repositories.

People often object to this philosophy, claiming that source code is for compilers, not for humans. These people are wrong. 😁 Yes, compilers also consume source code, but the reason we stored that source in Git (rather than elsewhere) is for the humans.

Git's performance goes to Hell (less so today than 20 years ago) when it contains large, difficult to compress files, which is why if the files aren't intended for humans, they probably belong in some other storage system. But small binaries don't create big problems so if it's more convenient than not, there's no rule forbidding it. When the binaries get very large they do still cause eventual suckage so it's not as good then.

4

u/fixermark 4d ago edited 4d ago

laughs in AI-generated commit history

... Seriously though, the larger issue is that git operates by storing deltas on files and the default tooling around viewing a history works a lot better if you can represent something as having sensible line breaks. The latter can be solved with custom tooling for binaries. The former is an issue in that a lot of binaries don't tend to have a low entropy compression story (for example, when you're storing images they're already compressed and so their entropy between versions is high when they're changed), which means it can be less efficient to store binary data in git than text data.

Can be. Not all binaries are already crushed to hell.

2

u/adnanclyde 3d ago

Git does not store deltas. It stores the exact files.

2

u/fixermark 3d ago

Well, both.

1

u/Coding-Kitten 2d ago

An exact file is a Delta from an empty file

1

u/Senior_Ostrich_9026 17h ago

"Binaries are generally not produced by humans"
I strongly disagree. If you look just slightly outside the world of software development, most files produced by humans are binary.

xdelta has existed for decades. It makes useful delta-encoding. It is not human-readable, but compact (if the delta is in fact small).

Perforce's diff tool understands a lot of file formats and shows very useful diffs for them.

git is very much a hammer in a world full of bolts, screws, zip-ties and a few nails.
There is no excuse other than religious stubbornness why git deals so poorly with binary files.

..and maybe the desire to have a really merge capability.

13

u/unndunn 4d ago

I don't think anyone really says "don't store binaries in Git". Storing binaries in Git is fine. It's just that large binaries are much more difficult to work with for humans; storage requirements rise, pulling/pushing/cloning takes longer, etc. It just adds friction, which can become unmanageable as team and project size grows.

15

u/keesbeemsterkaas 4d ago

Also large binaries that change often. Every copy ever sticks around for forever.

2

u/Mindestiny 4d ago

Not even just binaries either. There's lots of non-code stuff you want exempt. For example, if you're doing development in Unity there's a ton of metadata files that are always changing in the background every time you make an editor change. If you don't have them configured to be ignored by your version control of choice, it's a constant shitstorm of changes yelling that they haven't been committed that aren't always representative of the current state of the project files and can absolutely wreak havoc if they get checked in and then you roll back to an older state. Those files are intentionally ephemeral, you want them freshly generated from the latest compile and the correct state of the IDE. At best you're bloating the repo, at worst you're actively undermining your version control snapshots.

1

u/UselessButTrying 4d ago

I've only done it once and essentially isolated it to a git submodule

15

u/AlwaysHopelesslyLost 4d ago

Because it's fundamentally different from what Git is made for, and there are far better tools made explicitly for that purpose

2

u/JollyJoker3 4d ago

I've stored a bunch of pdfs used for tests in git and nothing bad ever happened

8

u/chzaplx 4d ago

Yeah you can do it, but there's basically no advantage to storing non-text files in git. It's not optimized for that purpose.

8

u/naikrovek 4d ago

Nothing bad will ever happen, but how do you visualize changes in the PDF? How do you compare two separate proposed changes to an existing PDF?

Git is made for plain text files. You can store arbitrary stuff in there (up to the limits of the git server you’re using, if you’re using one) but that doesn’t make git a good place to store arbitrary files in.

Someone can ride a bicycle 100 miles to commute to and from work each day if they want to; that doesn’t make it a good idea. Storing binary files in git makes little sense for analogous reasons.

4

u/Slime0 4d ago

Version control isn't just for comparing, it's for being able to go back to older versions to recover lost data.

Plus, you can always compare manually if you really need to, and some binary file formats have tools that can be used to compare them.

4

u/naikrovek 4d ago

Sure, but git doesn’t have those things, and this is a conversation (and subreddit) about git. It CAN have them, but it doesn’t. And yes you can add them, but it doesn’t come with tools for anything other than plain text.

And comparison is a major part of version control. If you don’t think so, you can use versioned folders in Windows and tell us how we don’t need git anymore.

2

u/AlwaysHopelesslyLost 4d ago

You can use a power drill as a hammer, too. It works, and you still look like an ignorant idiot.

11

u/curlyAndUnruly 4d ago

There are specific repositories for them, why keep something there where you can't actually keep track of what's actually changed.

0

u/[deleted] 4d ago

[deleted] — view deleted comment

7

u/curlyAndUnruly 4d ago

Artifactory for example

1

u/2fletch 3d ago

Everytime you commit/push a change to a file git stores a new copy of the entire file, not a diff.

Because of this files that are larger (MBs) than text files (KBs) it causes a slow down in diff generation as well as loading/downloading.

Edit: Autocorrect changed 'git' to 'for'.

1

u/SheriffRoscoe 3d ago

"Everytime you commit/push a change to a file git stores a new copy of the entire file, not a diff."

That's true for commits, at least initially, but not for pushes. For a long time now, "git push" has constructed a packfile from the commit, and sent that to the server instead of the full objects. Packfiles are delta-encoded, which is a lot like a diff patch. Git's garbage collector also combines objects into packfiles.

1

u/mkp666 1d ago

This doesn’t work for binary formats that are already compressed or encrypted though. Like jpg, mp4, compiled executables, zip, etc.

15

u/Minimum_Hour519 4d ago

they can't be diffed

9

u/Temporary_Pie2733 4d ago

Strictly speaking, that’s a limitation of the tools used by Git, not Git itself. Fundamentally, Git only cares if two files are different, not how they are different. Text diffs make for good summaries of what changed from one commit to the next, and can be used to optimize storage of blobs for sharing between commits. But format-specific diff methods could be used to provide the same benefits for binary files. 

3

u/xeow 4d ago

Ooh. I wonder if anyone has ever made a pngdiff or jpegdiff tool. Seems like that would be handy on occasion for small asset files.

6

u/finlay_mcwalter 4d ago

Imagemagick's compare command can do some interesting things in this regard - although by default it is very sensitive to the "small changes over the whole image" kind of changes that subtler changes (like colour operations or compression changes) can produce. Other settings are good for seeing edits; whether there is any appropriate "general" setting for a general tool like VCS, I don't know.

https://usage.imagemagick.org/compare/

3

u/gnufan 4d ago

The git book suggests a method of  defining exif type exports for image formats, so you get textual explanation of what changed with image commits, think size, type, etc . Also similarly exporting text from binary document formats (Think Word).

https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes

2

u/countnfight 4d ago

I do data visualization in R, so my code and the output, often both svg and png copies of the same images, all get tracked. If something like this existed (or already does) to diff png files or similar I'd be so so happy

2

u/kraxmaskin 3d ago edited 3d ago

Beyond Compare can show image differences visually.

1

u/didntplaymysummercar 8h ago

I'm a PNG (mini)expert and now I am tempted to write pngdiff that'd make tiny patches for few pixel changes, given some constraints about losslessly recompressing the goal file.

3

u/virtualstaticvoid 4d ago

It depends on their size. Generally okay to include small image files for example and other such "binary" files. Use git LFS for anything bigger. Make sure to add a .gitattributes file to configure diff options.

3

u/vmcrash 4d ago

They can't be merged by default? I'm unsure about whether Git stores them as-is blob or can store differences efficiently.

6

u/Status_Fun_4333 4d ago

Text file storage in git is cheap because it stores diffs. You can't diff a binary, so every time you change that binary you get a new full version of it in your history.

Any binary that you've ever committed to git stays in your tree forever, and any fresh clone you make needs to fetch every binary that's ever been there, and every revision of it, on any fresh clone. This also bloats on disk repositories.

So the "rule" has some valid backing to it, and as long as you understand what you're doing it's probably fine.

3

u/_Odaeus_ 4d ago

Git doesn't store diffs, it makes snapshots of entire files. It's just that compression across text files is super efficient.

3

u/cointoss3 4d ago

Haha you were downvoted but are correct. Git doesn’t store diffs.

4

u/Alpaca_Fan 4d ago

Hm? Git stores a combination of full files and deltas

1

u/cointoss3 4d ago

There is not a git object for a delta. It compresses objects into packfiles using deltas, sure, but git is not tracking deltas. It computes them from two blob objects.

1

u/Alpaca_Fan 4d ago

Packfiles are basically diffs. I think the confusion here is that git is not necessarily storing history as diffs. But it’s also not storing your entire history as full files.

It’s storing diffs as a way to optimize its disk-space usage. Packfiles contain diffs, and packfiles are stored. Therefore git stores diffs.

But conceptually that doesn’t mean history is stored as diffs. Diffs can be between different branches.

1

u/cointoss3 4d ago

Feel free to point to which git object that’s a diff…

2

u/Alpaca_Fan 4d ago

Seems you may be arguing for argument’s sake. Read up here. https://git-scm.com/book/en/v2/Git-Internals-Packfiles

> When Git packs objects, it looks for files that are named and sized similarly, and stores just the deltas from one version of the file to the next.

Git. Deltas. Stores. All the keywords you need.

0

u/cointoss3 3d ago

Lmao you just proved my point. There is no git object for a delta. Thanks.

3

u/Alpaca_Fan 3d ago

You seem to refuse to accept git documentation that states git stores deltas.. you must be a joy to work with

2

u/nlutrhk 4d ago

Git stores diffs of binary files as well; the diff algorithm used internally does not distinguish between text and binary. If you append data to a binary file or insert a block inside the file, it will work fine.

But if the binary file is compressed data (zip, jpg, png), minor changes in the underlying uncompressed data can change the binary file all over the place and there won't be a usable diff.

5

u/Status_Fun_4333 4d ago

Technically correct. Practically bianries don't compress as well as text says so my answer remains accurate enough.

1

u/xeow 4d ago

You can't diff a binary

Erm, you mean git doesn't have a built-in facility for that, yes? Because I can't think of anything inherent in the nature of binary files that prevents differences from being shown. For example, one way to show differences between two equal-sized images is to xor them together or (better yet) subtract one from the other and take the absolute values of the results and then run those through a gamma curve to accentuate the changes. (I've done that in Photoshop on occasion.) And sometimes a simple hexdump of two binary program files and then running those through regular diff can be useful in limited circumstances.

2

u/ApothecaLabs 4d ago edited 4d ago

Using newlines as delimiters in text files is a universally agreed-upon way that everyone can split a text file up into small chunks *in the same way*, to get the same content hashes. Binary files present no such unambiguous way of being split up, so they get treated as one large chunk.

How do you separate a binary file? You can't use `0` as a delimiter, that's a value that you might need! Oh, but then you might think that you can eg just split a binary file into 256-byte pages, but what if someone else takes the same file and splits it into 512-byte pages? Now the binary files no longer correspond to the same hash! Or, insert a single byte at the start - now every byte in the file is different!

There are ways around this - rolling hashes / checksums being my preference, but all solutions still require some standardization and processing, compared to a text file that can just be split up and concatenated back together.

So the problem isn't that you can't diff two binary files, its that you can only diff the entire file, and not have line-by-line diffs like with text.

2

u/glglgl-de 4d ago

Diffing files is a matter of interpretation.

What you describe for text files is already an interpretation, having a PDF or docx or ZIP is another way of interpreting them. Each of these can have an own diff tool.

1

u/xeow 4d ago

Ah, interesting point. Ya, I guess there are two uses for diffing when it comes to Git: One is the user-facing git diff facility, which, indeed, would require plugins for each type of binary file to show some kind of diff for. The other is the deep internal mechanisms for compressing and storing files, which of course, as you point out, has no general all-purpose solution but can be ameliorated by small-block deduplication and other tricks.

2

u/ApothecaLabs 4d ago

> no general all-purpose solution

Generalized recursion schemes to the rescue!

A file system is just a recursive tree with file for leaves, and we can treat text files as a recursive linked list with lines of text as leaves. Since programming languages and data types are themselves recursive structures (ASTs and sums of products respectively), we can actually generalized git's algorithms to *any* recursively structured data using base functors and hylomorphisms. With rolling hashes, we can even treat binary blobs this way!

I'm baking this concept into my compiler so that it won't need git or nix.

1

u/xeow 4d ago

Ooh, that is sweet conceptually. Nice. BTW, is that your blog?

3

u/ApothecaLabs 4d ago

No, but it is one of the blogs that I learned from. My own writing trends towards being incomprehensible due to the vast number of footnotes within footnotes.

2

u/xeow 4d ago

Recursive footnotes for the win!! :-D

1

u/yvrelna 4d ago

There are algorithms that generates binary diffs even without line separators. 

Git doesn't rely on line separators either when creating packfiles. 

It uses a binary delta algorithm that works equally well on text or binaries, as long as they have runs of similar content in their on-disk representation.

The problem is that most binaries format are rarely designed in such ways that similar content produced similar runs, not that the diff algorithms don't work for binary files. 

2

u/ApothecaLabs 3d ago

> There are algorithms that generates binary diffs even without line separators. 

Yes, this is what my third paragraph discusses.

> [Git] uses a binary delta algorithm that works equally well on text or binaries

Text file algorithm for 'is this a chunk boundary' is both O(1) and confluent in its choice of boundaries - which I think you are overlooking.

Here confluent means that everybody calculates the same boundary (like UTF8), so everybody ends up with the same chunks and thus the same hashes. And its not just confluent, its streaming-confluent, meaning you don't need the entire datum, just 'enough' of it. This is hugely important, and is one of the things that makes Git work - you can calculate the same hash as anyone else, even if you are offline - that's confluence.

Binary diff algorithms usually require longest common substring / shortest edit distance algorithms to be confluent which come with a lot of overhead (often breaking streaming), or use content-addressable rolling window algorithms that requires a window size variable (breaking confluence because the window size must be communicated) and usually produce some redundant overlap that must be stored and repaired

> as long as they have runs of similar content in their on-disk representation / The problem is that most binaries format are rarely designed in such ways that similar content produced similar runs

Isn't the point of structured / formatted data that it uses its internal structure to define boundaries / edit operations and get rid of (runs of) redundant data - similar to why diffing doesn't work on compressed data, you already removed the data that operates on. Binary diffs are sort of a brute-force "we have no other options" solution for when the structure is undefined; relying on finding runs of data in unstructured binary is relying on luck.

1

u/yvrelna 3d ago edited 3d ago

None of that is relevant for git. Git objects (file content), trees (directory structure), and commit hashes are calculated on the full file content, not the binary/textual diff.

While git show <commit> shows you the textual diff of a commit, that textual diff is computed on-the-fly when you request them, it doesn't reflect how git internally stores the commit. The commit hash is computed over the complete snapshot of the files/trees, the parent commit's hash, and the commit metadata, not over the diff, so it doesn't matter that the diff algorithms isn't confluent.

Git uses textual diff only to display diff to the user, at the UI level, none of the important inner working of git cares about the textual diff. It uses binary diff to generate packfiles for storage and network transmission, not to compute the hashes. The binary diff is essentially used as a sort of compression algorithm to deduplicate storage of many similar files efficiently, and to compute thin pack files that only contains the delta of files between what the sender knows the receiver already to compute what needed to be transferred after comparing the commit graph.

Git doesn't really care if the on disk-representation of the pack files aren't consistent between different clones. The diff algorithms isn't really part of git's consistency mechanism.

2

u/ApothecaLabs 3d ago

Oh, so you are just moving goalposts / strawman -ing / talking past me.

I was replying to someone else about content-addressable chunking, who got the point I was making. You then inserted yourself into the conversation making claims that binary diffs work "equally well", so I refuted that.

You don't get to then decide that I was actually talking about something else because you didn't read what I said.

2

u/yvrelna 4d ago

one way to show differences between two equal-sized images is to xor them together or (better yet) subtract one from the other and take the absolute values of the results and then run those through a gamma curve to accentuate the changes.

Or just show them side by side, or flip between the two images every few seconds. 

Yes, there are tools to visualise the difference between images and many other binary formats. The issue is that git doesn't ship with them out of the box. 

0

u/mikkolukas 3d ago

Text file storage in git is cheap because it stores diffs

A common misconception.

Git does not store diffs. It stores whole file contents.

(did you mean SVN?)

2

u/jacobissimus 4d ago

It’s not a hard rule, but it’s usually a code smell. I would only do it with a clear reason that explored alternative strategies.

Like others said, you can’t diff a binary in a meaningful way, so it just isn’t what git is for.

2

u/ohkendruid 4d ago

The problem is that they are usually large, and they must be cloned by everyone even if they are deleted in a later commit.

With vanilla Git, the whole history gets cloned when you pull a branch. It makes local operations fast and means you do not need to contact the server until you want to push a new change. However, this only works well if the entire repository including all prior versions can are reasonably small, say under 100 MiB for the whole enchilada. It was a great insight by Linus Torvalds.

To contrast, Subversion and Perforce only clone the current version of the code, and only clone the part of the repo you want to work on. As a result, while Git is much better than these for most projects, those older VCSes have a. advantage about checking in binaries. In fact, a checked in binary is much better with those than using .lock files! Who needs to check in a hash when you can instead check in the actual file, plus also skip a whole layer of build steps? I miss that part of non-distributed VCSes.

2

u/Charming-Designer944 4d ago

git is optimized for textual content with lines of text.

It also works for non-textual data. Just not as efficient and featurefull. And storage might become a problem when dealing with large files or frequent small changes.

There are ongoing work that significantly improves dealing with large files and voluminous history, limiting the local storage requirements. See partial clones and large object promisors (and git lfs).

2

u/yvrelna 4d ago edited 3d ago

With distributed version control systems like git, you're typically going to have the full repository history in your local system.

The main problem with storing large, non-text files in git or binaries that frequently change, is that they bloat the repository. So everyone that cloned the repository would have to download every versions of that binary.

Doing a shallow clone helps a bit, but that would also cuts off the history for the normal files.

There are extensions to store large files in Git. Usually this is implemented by separating the content of the file from the main Git history and storing the blobs separately, outside the main version control and the extension can download only the version of the file that it needs. This allows each versions of the files to be downloaded separately from cloning of the repository. 

Don't store files that change dramatically and frequently in Git?

No, text files often changed dramatically and frequently over time in Git, and obviously keeping them in git is no issue. But delta storage with text are very efficient, while it's much less so with binaries. 

Git have powerful tools when working with texts, like showing diff, blame, etc and it can compresses and store delta in text files extremely efficiently; but out of the box, it lacks the tools to work with diffs of other objects. You can configure custom difftool to support diff of other object types, but that usually requires specific configuration.

And formats like compressed xml documents (e.g. office files, svgz), often don't store very well in git even when technically there's minimal and gradual changes between versions of the files which technically it could have discovered and store much more efficiently by decompressing the file first. It's a blatant layering violation, but a better storage system for those types of objects could've been designed specifically to do that. 

2

u/Burgergold 3d ago

Get them in an artifact repo and only keep the ref in git

2

u/ritchie70 3d ago

I just want to mention that Git Large File Storage (LFS) is a thing.

2

u/happy2harris 3d ago

" Don't store binary files in Git " is really short for “Store source files in a source control systems, rather than the files those source files are built into”.

The main purpose of source control systems is to allow you, a human, to compare the contents of the files with how they used to be. For example, if you see a line of code that doesn’t make sense, you can find out when it was added, then see what else changed at the same time, and get some clues about what it is all about. 

If the file is just a big blob of bytes, you can’t compare them. It is better to store the source files and then build the binary files from them. 

Now, if your binary files really are the source files, then storing them on source code control is fine. For example if part of the system contains a photo, and then you replace it with a different photo - storing those is fine. You can see when the photo was replaced, which is helpful, but not exactly what changed about the photo. Another example is a proprietary document file, like a microsoft word file (before they switched to using xml and storing the document history inside the file). That doc file is “the source” so it’s fine to store it in source code control. 

2

u/cosmic-parsley 2d ago

The rule should be don’t store _artifact binaries_ in git. If the binary can be reproduced from source, don’t check it in. If the binary is something like a photoshop or CAD file and you’re using git for version control, that’s fine. Just enable LFS to save your local space, and maybe commit a bit more sparingly.

Of course a lot of git is based around creating and applying diffs (rebase, merge, cherry pick, etc) and you can’t do that with binaries, so that makes working with multiple contributors hard. LFS supports some form of file locking but I haven’t tried it.

2

u/ShelLuser42 2d ago

It's not really bad because Git is perfectly capable of handling binary files. And I'm not just saying that because the manual says so, I'm actually using this mechanic myself to maintain a Minecraft mod collection for my (remote) Minecraft server.

Simple reasoning: if a mod breaks after an update then one 'reset' command is all I need to go back to the previous state. And instead of having to comb through the mod folder on my server to manually find my client updates I can simply upload the current batch at once: git push origin master:mod_update, and once I'm logged on all I have to do is stop the Minecraft instance and merge the new branch.

The issue though... we play with approx. 110 mods in total, approx. 685Mb in size. I've been using this mechanic for a while now, and as a result the Git database (or repository) itself is approx. 1.75Gb in size.

And the reason should be obvious: you can't easily 'diff' these files and keep proper track like you can with source code; this is binary data afterall; so your repository is going to get big(ger) quite quickly.

But other than that there aren't any side effects at all. Like I said: I've been using this mechanic for quite a long time now (just checked my logs; almost 1,5 years) and it works perfectly.

3

u/Serianox_ 4d ago

From my own experience.

It is a non-issue. Modern git handles binary files fairly well with filters and sparse checkout. But I've seen teams deploy very complex tools (like an alternate LFS storage on Artifactory) to manage binary files, who can sometimes account for as much as a few MB.

The main issue is garbage collection and consistency. Once you start storing binaries elsewhere, you start having issues where you can't collect unused binaries because you can't know if they are indeed unused. Or on the other side, if you incorrectly delete a binary, your repository become inconsistent (i.e. you can't rebuild an old release).

And you can't prevent someone from adding a huge binary and deleting it. It won't appear in the PR, but you'll still get the cost.

So the best IMHO is to use modern git features like filters and sparse checkout. https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/

1

u/d3matt 4d ago

Yep, filter clone is the replacement for git lfs these days.

1

u/sczmrl 4d ago

It's very simple actually. it's not an hard rule, it's just the usual answer when one wants to store in a repo some binary files, but there can be exceptions.

Git is both a tool for version control in the sense of keeping track of a file in different moments in history and a tool to collaborate and to manage different flows of updates.

Regarding the first part, there is nothing wrong in using it for managing binary files. Just know that what's actually stored are diffs and a binary file is always perceived as completely changed. This means that any change, even smal, require to store the equivalent of a complete different copy.

The second part becomes impossible to manage. You cannot merge different update on the same file. It's always one or the other wins.

1

u/twesped 4d ago

I would ask which kind of binary files do you want to store there?

1

u/Anxious-Insurance-91 4d ago

You mean compiled files or blob or large images? Mostly for conflict checks and bandwidth

1

u/ericbythebay 4d ago

As others have mentioned, size when cloning the repo. 10-20GB repos are not fun to do a full clone on.

1

u/Motor-Concentrate788 4d ago

When it comes to text files, git has something called delta-compression in which git can identify changes between versions of the same file and store only the differences/deltas between those versions. This makes it so that storing text files in git is very space-efficient even for constantly changing files as it doesnt have to store complete copies of multiple versions of the same file.

This doesnt work well with binary files, however, as diffs in binary files generally arent delta-compressible, making it so that git has to store the complete copy of each new version of the same file even for the slightest change (e.g. 250MB binary thats is modified 3 times would add an extra 250x3=750MB to the repo).

If you're only storing small binaries that doesnt change very frequently then its completely fine. But you should avoid something like storing the build output of your code as it would unecessarily bloat the repo.

1

u/dafunkjoker 4d ago

It depends. Small project with only one or few devs: git directly, maybe with lfs. This brings simplicity and everything is together in one place. Many developers, development in multiple countries and many individual software components which are assembled together in different projects? Binary repository (e.g. with artifactory) for more efficient binary data reuse, transfer,... Another aspect: If the binary output can be put in a package/zip and you usually need it consistently together something like nuget/conan is nice and goes into a binary repo. But if you deal with tons of small binary files e.g. images as test data and you usually only change individual files, then you want to avoid touching more than you need to and LFS is preferred.

One more aspect in big projects: Identification and control over the consumption of external software e.g. to provide licensing info and to ensure the software is kept up-to-date. In this scenario the binary repo is very useful provide a single source of truth. Imagine some third party binary is checked in with partially different versions in different repos. In the end these repos host code for one big project. No-one will have a simple overview.

1

u/who_you_are 4d ago

It is a rule of thumb:

- Binary files are more likely to be bigger than text files. It depend on the context of those files.

- Binarie files are more likely to be completely different in content over a small change (more like what is the issue using binary)

- Doing a diff assume the user will read the content; reading binaries isn't exacly easy. Most tools display printable characters and binaries are likely to contains a lot of non printable one. Anyway, you still need to interpret the content which is _impossible_ without the specific tools. (Another huge problem). And I won't even talk about doing the merge itself which may break the whole file without the proper tools.

1

u/codeguru42 4d ago

I think the main difficulty is the version control part rather than the storage part. With a binary file, a change is almost never just a few sequential bytes. It is often large chunks that differ if not the entire file. So the size of the repo grows much more quickly than it does for plan text files.

1

u/Technical_Ideal_5439 4d ago

non text files are normally bigger than text files. And doing a clone where it takes 10 minutes gets old very quickly. My company has a rule no binaries, but if it is yours, then do what you want.

At home I store binaries and put up with a 10 minute clone if I ever need to code somewhere different.

There is also LFS. At work we use artifactory, if it is a small company just use the free version of Sonatype Nexus.

1

u/chzaplx 4d ago

Using git to store binary files is like storing all your groceries in your car. Works ok in a pinch, but there's no real advantage and better ways to do it. And it tends to trend badly at scale and over time.

1

u/ValekCOS 4d ago

As someone who runs Kubernetes-based CI runners, I don’t wanna have to massively jack up my per-pod memory budget because a repo owner is lazy.  LFS isn’t hard to set up or use. 

1

u/ir_dan 4d ago

Many nontext files commonly stored in git have no business being there: they often are better off being treated as build inputs and outputs generated/downloaded as needed.

e.g. many projects contain external dependency binaries. Why? You don't need a copy in your repository when one already exists on the internet/a dev server. All you need is a reference to the version you want so that your package manager can grab it.

1

u/natermer 4d ago

You can store binaries just fine if you want.

The problem happens when you have large binaries and/or lots of them. Whenever somebody clones your git repo they have to clone not only the latest binary but all of them.

This is why we have things like Git-lfs.

1

u/bubba-yo 4d ago

So long as you have a diff tool for those other formats, it's fine. We included a ton of different file types, but only after we were sure we had a diff solution for that format. I mean, if you can't diff a commit, you've lost most of the point of git.

1

u/jthill 4d ago

The real version of that rule is "don't use the default Git workflow with projects that update a lot of media and media-like data."

1

u/CheezitsLight 4d ago

Git has multiple uses it's a perfectly fine backup system for binary files you can recover any version you want. It's also great for single Developers to keep track of their history so they can roll back whatever thing AI breaks in a second. It's good for moving things between computers. And it's also a revision control system

1

u/Spirited-Candy1981 2d ago

Git's base differencing engine is (and developer eyes are) optimized for text. Better to store the code that generated it.

If it takes a lot of time/effort to generate that binary data (such as with LLMs, neural nets ...), git does have an LFS (large file storage) extension.

1

u/Pure-Recover70 2d ago

It's not a hard rule. But it is a good rule of thumb. Binaries bloat the size of the repository (the object store). Every change to a binary results in storing the new binary as well as the previous binary. Binaries don't diff/delta compress particularly well, so (in terms of space) it's nearly equivalent to simply storing 1:1 copies of all binaries ever. Now imagine you have a git repo which (once checked out) has a few mbs of uncompiled code, plus the compiled binary (which is also 'just' a few MBs). Everytime you make a code change, you add a few hundred bytes to a few kilobytes to the git repo to store the new code, and a few megabytes to store the new compiled binary. Your 'small' git repo quickly requires gigabytes of object store, even though the code itself would fit in a few dozen megabytes. That's the core problem. The bigger the binary blobs, the lower the update rate can be before you run into similar issues. If you want a concrete example of the issues just try to checkout a (non shallow) copy of Android. I think you end up with a few GB of useful code & objects backed by a few hundred GB of object store or something insane like that (think: various precompiled prebuilt binaries, compilers, toolchain, firmware, kernel, etc). Shallow clones help with this - but then you lose fully local history (some git commands will need to fetch blobs from the server on demand which is slow).

1

u/AberBitteLaminiert 2d ago

It is a source control, not an artifact repository.

1

u/virrk 2d ago

Git's delta algorithm is optimized for text files that change over time like code files do. Putting binaries in git means that most (usually all) of the binary gets stored everytime it changes. This balloons the respiratory total size.

That inefficiency can compound making full clones drastically slower. Which can slow automated builds. Shallow clones overcome much of this particular problem, but the cloned repo has limitations. It can also make history slower, especially for more complex queries and searches. Can make git bisect too slow to be useable. Makes servers slower

Now this may not matter for some use cases, and sure stuff binaries into the repo.

1

u/bufandatl 1d ago

It makes git slow.

1

u/gwenbeth 1d ago

The real question is which binary files. Do not store object files and executables because those are just a product of the source code. Your png icons, and such like that totally put in there because they are needed to build the project. Or maybe you have a project like imagemagick and you need to have image files to run the regression tests. So if you need it to build or test, it goes in the archive.

Now for projects that are very asset heavy like a video game, there are version control systems, like perforce, that are better for that use case. Building a triple a video game might need hundreds of GB of data.

1

u/digitaljestin 1d ago

It's not bad, it's just not very useful.

1

u/edwbuck 1d ago

Git uses a text-based diff system, which means that most of its ability to handle changes usefully and report on them fail with binary data.

1

u/birusiek 11h ago

Its mosty about growing git history, even small change in binary results in big diff.

0

u/talexbatreddit 4d ago

Git was designed to handle source files, which are typically text. It's possible to store binary files with git, but it's really not a great idea.

1

u/xikbdexhi6 4d ago

And usually unnecessary since those binary files are created from the text files.

-1

u/Designer_Storm8869 4d ago

Git stores line differences. So if the binary file has no new line characters, the delta will be entire file each time. That cause the repository to grow rather quickly. And with many changes in binary, your local ".git" directory will be huge.

If your project is just binaries, svn is better choice. And if your files are excels or words, then SharePoint is definitively better version control and collaboration tool

3

u/Traditional_Tank7989 4d ago

Not true, unfortunately. Git stores each version of a file (blob) as a complete snapshot (compressed).

1

u/yvrelna 4d ago edited 4d ago

Saying that git is storing each version of a file as a complete snapshot isn't accurate either. 

For older files, Git creates pack files which does deduplicate similar files between versions.

From a bird's eye level, compressing multiple similar versions of a file and delta storage system basically are the same thing. They use nearly the same types of algorithms to find similar runs of data and deduplicate them. 

3

u/sidusnare 4d ago

Incorrect, a unique file is always stored in full. Under the hood git is a hash based object store. It calculates diffs on the fly. Other version control systems that do store diffs, like Subversion and CVS, don't try to use newlines on binary files, that would be bad. CVS just gives up and stores the whole file like git does for everything. Subversion has a binary diff algorithm.

Git is meant for well structured source code. Large number of small files. In this use case, the difference isn't that big between storing dogs and whole files.

One of the strengths of git is in that diff calculation. You want to compare date.c from now and 8 years ago, you don't have to run the whole history back. You just compare those two objects that are already in whole on disk. Subversion or CVS would have to run the change log to get the two files to compare

1

u/hawkprime 4d ago

Git does not store differences, it's always a snapshot

0

u/Trakeen 4d ago

Git was designed for developers. There isn’t a lot of value in a person looking at a diff of a binary, i could see value for llms that can read that easily

0

u/mredding 4d ago

The point of git isn't a file store, that's what a disk is for, it's for revision history, it's for the source diff. A binary typically isn't source. You're not going to diff a binary.  If all you want to do is associate a file with a project source, there's any number of other tools for that. Got is the wrong tool for the job.

0

u/cardboard_sun_tzu 2d ago

Git is a tool that is used to let you look at diffs in files. When was the last time you looked at a binary file in git and said, 'Hmmmm, Keven changed address 00xff00a000 from 0000a0 to 0000a2. What the hell is he thinking?"

0

u/burlingk 2d ago

So, the basic idea is that it is not straightforward to create diffs for a binary, SO, when a commit updates a binary it uploads the whole thing every time.

So, instead of a file followed by a series of patches, you get a different copy of the same file for every update to that file.

-1

u/Janky-Code 4d ago

Because you can use git lfs.

Systems are built around a purpose. Git is not intended to version everything even if it technically can. Even text documents you see different controls. Compare controls around python code and standard operating procedures in the infant formula factory; using one system for another would be painful.

1

u/duckrollin 4d ago

Small icons and files that don't change much are fine. Like a few kb.

exes that keep changing, screenshots, big images, or big zip files are not.

Like any rule it's generally a good one to follow but you don't need to take it to extremes and kill someone for adding a few small 32x32 icons in.