task-manager has come a long way. It has commit messages that explain the why, a clean history with an agreed integration policy, the right files inside and properly treated, and its secrets outside. One last problem of the module remains, far less dramatic than the previous one but increasingly annoying:

"Every git status takes three seconds. The clone of the repository weighs 900 MB and the code is four. And yesterday, cloning onto Carla's new laptop took eight minutes."

A slow repository does not break anything, but it silently degrades everything else: if git status takes three seconds, you stop running it; if the clone takes eight minutes, CI becomes expensive; if git log crawls, you stop consulting the history. Slowness erodes the habits we have built up across the whole module.

This lesson teaches you to measure before optimising, and then to act on whatever the measurement points to. It focuses on the normal-sized repository that has become slow through neglect. Genuinely enormous repositories and monorepos have techniques of their own — partial clones, shallow clones, a sparse index — and they are the content of lesson 10-04.

1. Measure before optimising
2. git count-objects -vH: the snapshot of the repository
3. Finding the largest objects in the history
4. Packing: loose objects and packfiles
5. git gc: what it really does
6. git maintenance: the modern replacement
7. Speeding up git status: fsmonitor and untrackedCache
8. feature.manyFiles and other settings
9. The cost of large binary files
10. Reference hygiene: prune and packed-refs
11. Other slow commands and their causes
12. Good habits that avoid the problem

The rule is the same as in any optimisation: your intuition about what is slow is nearly always wrong. Before touching anything, measure.

Git ships with a tracing mechanism that says exactly where the time goes:

There is the complete diagnosis: 1.36 seconds walking directories and 0.41 reading the index. The problem is not the history: it is the number of files in the working copy. That points to section 7, not to section 5.

A more structured approach, with the modern trace in readable form:

And to compare before and after a change, measure several times:

It is the first command to run. It gives you the state of the object database on one screen:

How to read it:

In the example: 743 MB of packfiles for a project whose code takes up 4 MB. There is the problem, and section 3 says where it comes from.

A comparison with the real size:

A 200-to-1 ratio between the history and the current content is an unmistakable sign that there is something in the history that should not be there.

This is the key diagnosis, and it is worth understanding piece by piece. The recipe combines two low-level commands you already know from lesson 01-04.

The diagnosis leaps out at you:

• presentation-video.mp4 appears twice, at 214 and 71 MB. They are two versions of the same file: somebody updated it and Git keeps both in full.
• mockups.psd also appears twice. The same thing.
• An 84 MB database dump that should never have been versioned.
• And a node_modules/.package-lock.json, a leftover of the contamination we cleaned up in lesson 08-03.

The two sizes the script prints are important:

• objectsize (actual): the size of the uncompressed content.
• objectsize:disk (on disk): what it really takes up in the packfile, after compression and deltas.

The fact that they are similar for the videos and the images says it all: already-compressed formats do not compress further and do not admit useful deltas. That is section 9.

This view by extension is the most useful for making decisions: it tells you which category of file is inflating the repository.

Locating the problem is half the work; the other half is section 9, and it anticipates the conclusion: deleting the file from the tip frees nothing. Getting it out of the history demands the same rewrite we saw in lesson 08-05, with the same consequences for the whole team.

To understand gc you have to understand how Git stores objects. There are two ways:

Each object is a separate file in .git/objects/, compressed with zlib, in a directory named after the first two characters of its SHA:

It is the format Git uses when creating new objects: fast to write, very inefficient to store. If you change one line of a 1 MB file and commit, a brand new, complete 1 MB blob is created.

A packfile is a single file containing many objects, with two optimisations:

1. Joint compression, more effective than compressing each object separately.
2. Deltas: instead of storing every version in full, Git stores one complete version and the rest as differences from it.

From 3 MB to a little over 1 MB. With a history of hundreds of versions of a text file, the saving is one or two orders of magnitude.

Details worth knowing:

• Git deltas against the most similar object, not necessarily against the previous version. It can chain deltas (with a configurable depth limit).
• Deltas are computed by content, not by history. Two similar files from distant commits can serve as each other's base.
• Already-compressed formats (JPG, PNG, MP4, ZIP, PSD) do not admit useful deltas. A tiny change in the image changes the whole compressed byte stream. That is why in section 3 the size on disk and the actual size matched.

When a packfile is created: on git gc (manual or automatic), on git clone or git fetch (what travels over the network is always a pack), and when the gc.auto thresholds are reached.

git gc (garbage collection) is the classic maintenance command. It does five things:

1. Packs the loose objects into packfiles.
2. Consolidates several packfiles into fewer.
3. Removes unreachable objects that have passed the grace period.
4. Packs the references into .git/packed-refs (section 10).
5. Expires old reflog entries (90 days by default for what is reachable, 30 for the rest).

The 8,432 loose objects have gone, the 7 packfiles have been consolidated into 1, and the total has come down. Note that there are still 698 MB: gc reorganises and compresses, but it cannot remove objects that are still reachable from some reference. The videos from section 3 are still there because they are in commits that are still in the history.

Git already runs gc --auto on its own after certain commands (commit, merge, rebase, receive-pack). It only acts if some thresholds are exceeded:

--aggressive discards the existing deltas and recomputes everything from scratch, with a much larger search window. It is extremely expensive and the benefit is usually marginal, because the deltas that were already there were reasonably good.

If you really want a thorough repack, this is more controllable than --aggressive:

• -a: everything in a single pack. -d: delete the old packs. -f: recompute the deltas.
• --window: how many objects to consider as a delta base (more = better and slower).
• --depth: the maximum length of the delta chain (more = smaller and slower to read).

gc can remove unreachable objects, and with them the possibility of recovering lost commits:

After those two commands, a commit you had lost with an unfortunate reset can no longer be recovered. Recovering lost commits and the role of the reflog are the content of lesson 09-04; until you have seen it, do not run --prune=now unless you know exactly what you are doing (for example, after the filter-repo of lesson 08-05, where it is precisely what you want).

Since Git 2.29 there has been git maintenance, designed to replace both manual and automatic gc. Its advantages:

• It is scheduled in the background, so it does not block your commands.
• Separate tasks with different frequencies, instead of a monolithic gc.
• Incremental: it repacks bit by bit instead of all at once.
• It does not expire the reflog by surprise.

That registers the repository and creates the system's scheduled tasks (cron, systemd, launchd or the Windows task scheduler, depending on the platform). From that moment on:

It is probably the most rewarding improvement in this lesson. The commit-graph file is an index with the structural information of the commits — parents, dates, generation numbers — which avoids having to read and decompress every commit object in order to walk the history.

The effect on a repository with a long history:

It speeds up git log --graph, git merge-base, git branch --contains, git tag --contains, git bisect (lesson 06-02) and everything that needs to compute reachability. It is free and it has no downsides.

This is Carla's problem, and it is worth understanding because the cause is not the one people assume.

git status has to answer three questions:

1. Which tracked files have changed? → compare the index with the disk.
2. What is staged? → compare the index with HEAD.
3. Which untracked files are there? → walk every directory of the working copy.

The third is the expensive one. Walking 40,000 files — node_modules included, even though it is ignored, because Git has to look in order to know that it ignores it — costs thousands of system calls. On Linux it is fast; on Windows and macOS, noticeably slower, and that is why Carla suffers more than Ana.

Instead of walking the tree, Git asks the operating system's file watching service what has changed since last time. Since Git 2.37 there is a built-in monitor, with no external tools needed:

The first run starts a background daemon:

The effect is dramatic:

It caches the result of the directory walk, using each directory's modification timestamp to know which ones have to be looked at again.

It requires the file system to update directory mtime reliably. The test command verifies that; if it fails, do not turn it on.

The last row is the one that pays off most and the one least often applied: if node_modules has 40,000 files, no Git optimisation is going to be as good as not having them. Modern dependency systems with a central store and links reduce that number a great deal.

A simple trick for measuring how much the untracked-file walk costs:

If the difference is large, your problem is the directory walk and fsmonitor is the solution.

Git groups recommended configurations into feature "macros":

It is equivalent to turning on, all at once:

It is designed for working copies with many files. It does not touch the history: it does not help with a repository that is large because of its history, only because of its current number of files.

Other settings with a good benefit-to-cost ratio:

And a setting you should not touch lightly:

It makes the repository take up considerably more in exchange for a marginal improvement. It only makes sense in very specific cases, and almost never in yours.

Let us go back to the diagnosis in section 3: 357 MB of videos and 154 MB of PSD files. This is the structural problem of task-manager, and it is worth understanding properly why it is so serious.

1. They do not compress. An MP4 or a PSD is already compressed. Git applies zlib on top and the result is practically the same size.

2. They do not admit useful deltas. Changing one frame of a video alters the whole compressed byte stream that follows. Git finds no usable similarity and stores every version in full.

3. Every version is stored whole. A 200 MB video updated five times is 1 GB in the history, for ever.

4. Everybody downloads it. git clone brings the complete history. Carla downloads all five videos even though she only needs the last one. So does CI, on every run that does not use a cache.

5. It cannot be merged. A conflict in a binary can only be resolved by choosing one whole version (lesson 08-04, the binary attribute).

A comparison with a text file:

It is exactly the same mechanism we saw with secrets in lesson 08-05: the new commit does not include the file, but the earlier commits still reference the blobs, which are still reachable and which gc therefore never touches.

To recover the space for real you have to rewrite the history:

With exactly the same consequences as in the previous lesson: every SHA changes, every clone becomes obsolete, you have to coordinate with the team, and the forks (Diego's) do not clean themselves up. Rewriting the history to save space is a serious decision, not a maintenance chore. Nearly always it is better to learn the lesson and not do it again.

For binaries you do have to version — designs, a game's graphical assets, master documents — the answer is Git LFS (Large File Storage). Remember from section 13 of lesson 08-04 that it is activated with a filter in .gitattributes:

The repository stores a 130-byte text pointer instead of the file, and the content lives in a separate store that is only downloaded when it is needed.

The full mechanism, the storage server, the costs, git lfs migrate for converting an existing history and the limitations you have to know about before adopting it are the content of lesson 10-03: Git LFS for Large Files.

References — branches, tags, tracking branches — are 41-byte files (lesson 03-01). Individually they weigh nothing; accumulated by the thousand, they do matter.

When somebody deletes a branch on the server, your local copy of that reference does not disappear on its own:

1,247 remote branches, of which perhaps 8 still exist. Every fetch processes them and every git branch -a lists them.

Picking up on lesson 03-06: fetch.prune true should be in everybody's global configuration. It is one line that avoids a silent accumulation nobody ever looks at.

And local branches that are already merged:

Beware of --merged if your integration policy is squash (lesson 08-02): a squashed branch does not appear as merged, because its commits are not in main. Check before deleting, or go by the remote branch that the platform removes on merge.

With thousands of loose references, each one is a tiny file and reading them all costs thousands of disk operations:

git pack-refs consolidates them into a single .git/packed-refs file:

git gc does it automatically, so you rarely have to run it by hand. References that change are written out again as loose files and are repacked at the next gc.

One specific case that takes people by surprise: the worktrees of lesson 06-06 leave behind references and metadata. If you create and delete worktrees often:

One specific case deserves a mention, because it links to lesson 06-05: submodules slow git status down noticeably, because Git has to go into each one and check its state.

With ui-components as a submodule, those two settings are noticeable on every git status.

Everything above is treatment. This is prevention, which is infinitely cheaper:

1. A correct .gitignore from the very first commit (lesson 08-03). Almost every enormous repository is enormous because of something that should never have gone in.

2. Never version large binaries without thinking about it. Before adding a file of more than a few MB, ask yourself: is it going to change? How many times? If the answer is "yes, many", you need Git LFS (lesson 10-03) or an external store.

3. Set a limit and check it in the pre-commit hook (lesson 06-01):

And the same check in CI (lesson 07-06), because --no-verify exists.

4. Small, atomic commits (lesson 08-02). A history of small commits compresses better and produces more efficient deltas than one of gigantic commits.

5. fetch.prune true and a sensible gc.auto in the whole team's global configuration.

6. git maintenance start in every repository you work with daily.

7. Measure from time to time. A quarterly git count-objects -vH catches the problem while it is still easy to fix.

8. Do not version tool output. Minified files, generated documentation, screenshots of failed tests. They can be regenerated.

9. Beware of database dumps. As well as being large, they usually contain personal data (lesson 08-05).

Mistake 1: optimising without measuring. Running gc --aggressive because "the repository is slow" when the problem is the untracked-file walk burns hours of CPU for nothing. Measure first with GIT_TRACE_PERFORMANCE=1.

Mistake 2: gc --aggressive as a routine. It is expensive and its benefit is marginal except after a complete rewrite of the history or an import.

Mistake 3: believing git gc will shrink a repository inflated by binaries. gc reorganises and compresses, but it cannot remove reachable objects. The binaries are still there because they are still in the history.

Mistake 4: deleting a large file and expecting the repository to slim down. The same mechanism as with secrets: git rm does not touch the history. You need filter-repo, with all its consequences.

Mistake 5: git reflog expire --expire=now --all && git gc --prune=now without knowing what you are doing. It destroys the safety net that allows lost commits to be recovered (lesson 09-04).

Mistake 6: not turning on fetch.prune. Thousands of ghost remote branches that slow down every fetch and clutter every listing.

Mistake 7: versioning large binaries "because there are only a few". Five versions of a 200 MB video are 1 GB permanently, for everybody who clones, for ever.

Mistake 8: ignoring the commit-graph. It is the optimisation with the best benefit-to-cost ratio in the whole lesson and almost nobody turns it on.

Tip 1: git maintenance start in your working repositories. It replaces manual gc, it runs in the background and it keeps the commit-graph up to date.

Tip 2: core.fsmonitor true if you have many files, especially on Windows or macOS. It is the difference between 2.8 s and 0.15 s on every git status.

Tip 3: measure the cost of untracked files with git status --untracked-files=no. If the difference is large, you know where to act.

Tip 4: keep the script from section 3. Finding the largest objects in the history is the question you ask once a year and always have to look up again.

Tip 5: set a size limit in pre-commit and in CI. It costs ten lines and it avoids the whole problem.

Tip 6: a two-minute quarterly review. The script in section 12 catches the problems while they are still cheap to fix.

1. Create a repository and commit a small app.js.
2. Add a large randomly generated binary file and commit it:

   head -c 20000000 /dev/urandom > demo/video.bin
   

3. Modify it three times (by regenerating it) and commit each version.
4. Run git count-objects -vH and note size-pack.
5. Run the script from section 3 and identify the large objects.
6. Delete the file with git rm, commit, run git gc and look at count-objects again. Explain the result.
7. Compare the size with the same experiment done on a 20 MB text file modified three times. Explain the difference.

1. In a test repository, create 20,000 small files in subdirectories:

   for i in $(seq 1 200); do
     mkdir -p "dir$i"
     for j in $(seq 1 100); do echo "content $i-$j" > "dir$i/f$j.txt"; done
   done
   

2. Commit them and time git status three times.
3. Time git status --untracked-files=no and work out what percentage of the time goes on the directory walk.
4. Turn on core.fsmonitor true, run git status twice (the first one starts the daemon) and time it again.
5. Try git config feature.manyFiles true and time it again.
6. Generate the commit-graph and compare git log --graph --all before and after.

1. Create a repository with 50 local branches, half of them merged into main.
2. Count the loose references with find .git/refs -type f | wc -l.
3. Run git pack-refs --all and count again. Examine .git/packed-refs.
4. Delete the merged branches with git branch --merged. Explain what would happen if the integration policy were squash.
5. Turn on git maintenance start and check which tasks are registered with git config --get-all maintenance.repo --global.
6. Write the review script from section 12 into a file, run it and interpret each block of the output.

Solution 1:

76 MB for a project whose code is 30 bytes. The four versions are stored whole: there is no compression possible on random data, and no usable deltas.

Four different blobs with the same path: the four versions, each one complete.

The size has not gone down by a single byte. The four blobs are still reachable from the earlier commits, which are still in main's history. gc --prune=now only removes unreachable objects, and these are not unreachable. It is exactly the same mechanism that stopped us deleting a secret in lesson 08-05: Git's objects are immutable and the old trees still point at them.

To recover the space you would need:

5.7 MB against 76 MB. With 80 MB of logical content in both cases. Two mechanisms explain the difference:

1. Compression: repetitive text compresses down to a fraction of its size; random data (like a video or an already-compressed image) does not compress at all.
2. Deltas: versions 2, 3 and 4 of the text are stored as differences of a few lines from the complete version. In the binary, each version is a completely different byte stream and Git finds no usable similarity.

This is the underlying technical reason why Git is excellent for text and terrible for large binaries, and why Git LFS exists (lesson 10-03).

Solution 2:

The directory walk costs 0.71 s out of 0.92 s: 77 % of the time. The diagnosis is clear and it points to fsmonitor.

From 0.92 s to 0.12 s: almost eight times faster. The daemon receives change notifications from the operating system, so Git no longer needs to walk anything.

The additional improvement is small when fsmonitor is already on, because the main bottleneck is already solved. Without fsmonitor, untrackedCache on its own usually gives a noticeable improvement (of the order of 40-60 % on the walk).

On a history of 300 commits the difference is small; with tens of thousands it is an order of magnitude. The reason: without the commit-graph, Git has to read and decompress every commit object to find out its parents and its date; with it, that information is in a flat binary index.

Solution 3:

51 tiny files turned into a single sorted one. With thousands of references, the difference when reading them is noticeable.

What would happen with a squash policy (lesson 08-02): git branch --merged checks whether the branch's tip commit is reachable from main. With a squash merge, the branch's original commits never enter main: a new commit is created with the same content but a different SHA and a different lineage. Therefore:

• git branch --merged main would not list any of those branches, even though their work is entirely in main.
• git branch -d would refuse to delete them ("not fully merged"), forcing you to use -D, which deletes without checking anything.

With squash, the reliable criterion is not --merged but the remote branch: the platform removes it when the PR is merged, and git fetch --prune cleans up your local copy. It is one more reason to have fetch.prune true in your global configuration.

maintenance.auto=false disables the automatic gc after each command: it is no longer needed, because the scheduled tasks take care of it in the background.

How to interpret each block:

The essentials of this lesson:

• Measure before optimising. GIT_TRACE_PERFORMANCE=1 says exactly where the time goes, and git count-objects -vH gives you the snapshot of the repository on one screen. Intuition about what is slow nearly always fails.
• git rev-list --objects --all + git cat-file --batch-check is the recipe for finding the largest objects in the history. Seeing the same file repeated several times is the unmistakable symptom of a versioned binary.
• Git stores objects loose (fast to write, inefficient) and in packfiles (jointly compressed and with deltas). git gc packs, consolidates and cleans up what is unreachable, but it cannot remove what is still in the history. --aggressive is almost never needed: only after a complete rewrite or an import.
• git maintenance start is the modern replacement for manual gc: separate tasks, scheduled and in the background. And it brings with it the commit-graph, which is the optimisation with the best benefit-to-cost ratio in the whole lesson.
• git status is slow because of the untracked-file walk, not because of the history. It is measured with --untracked-files=no and fixed with core.fsmonitor (Git 2.37+) and core.untrackedCache. feature.manyFiles groups the settings for working copies with many files.
• Large binaries are expensive because they do not compress, they do not admit deltas, they are stored whole in every version and everybody downloads them. And deleting them does not free space, for exactly the same reason that deleting a secret does not remove it: you have to rewrite the history, with all its consequences. The right solution is Git LFS, in lesson 10-03.
• Reference hygiene: fetch.prune true in everybody's global configuration, deleting merged branches (carefully if the policy is squash) and packed-refs, which gc maintains on its own.
• And above all, prevention: a .gitignore from the first commit, a size limit in the pre-commit hook and in CI, small commits, and a two-minute quarterly review.

For what goes beyond this — monorepos, partial clones with --filter=blob:none, shallow clones, a sparse index — there is lesson 10-04.

Module 7 ended by saying that task-manager was lacking habits. Now it has all of them.

Messages that explain the why and from which the version and the changelog can be derived (08-01). A history that is readable, bisectable and reversible, with a written, agreed integration policy (08-02). Only what should be inside (08-03), treated as it deserves, with Carla's line endings sorted out once and for all (08-04). Secrets outside, and a written procedure for the day it happens again (08-05). And a repository that is fast, measured and maintained (08-06).

Ana, Bruno, Carla and Diego now have the tool, the process and the habits. They have done everything right.

And that is exactly what does not happen in real life.

Because somebody is going to run git reset --hard over three days of uncommitted work. Somebody is going to commit on the wrong branch, or with the wrong user, and will not notice until after publishing. Somebody is going to pull on a branch that has diverged and will find themselves with a tangle they do not know how to undo. Somebody is going to delete a branch that did matter and will discover, in a panic, that git log can no longer find it. And one day, a corrupt .git/index is going to make Git refuse to do absolutely anything.

All of that has a solution, and nearly always a simpler one than it looks in the moment of panic. Learning how to get out of trouble — undoing changes, resolving divergence with the remote, recovering lost commits with the reflog, repairing a broken repository and diagnosing whatever does not add up — is the whole of module 9.

We start with the catalogue of the problems everybody runs into sooner or later, in lesson 09-01: Common Git Problems.