Investigating with history

Goal

Find the exact commit that introduced a bug in a few minutes, even when nobody knows when it started.

The problem

A Linkr Pro customer writes in: their Instagram campaigns stopped showing up in their analytics tool “a few weeks ago”. After some digging, the cause is clear. When someone opens linkr.app/sale?utm_source=ig, the redirect sends them to the destination without the utm_source. The query string is being dropped.

Nobody remembers touching that, and main has 180 commits since the last release known to work. Reading the current code shows what is wrong. It doesn’t show which change broke it, why that change was made, or whether undoing it breaks something else. The history has all of that, and a few commands can search it.

Searching the history

git log with filters

Start by narrowing the search to the code you suspect. Here, that’s the redirect:

# Commits that touched the redirect code in the last 6 weeks
git log --oneline --since="6 weeks ago" -- src/redirect/

# Same, with the diff of each commit
git log -p --since="6 weeks ago" -- src/redirect/

# Follows the file even if it was renamed
git log --follow --oneline -- src/redirect/build-url.ts

Pickaxe: when did this string appear or disappear?

git log --grep searches commit messages. To search the code itself, use -S:

# Commits that changed the number of occurrences of "searchParams"
git log -S 'searchParams' --oneline -- src/

# Same, with the diff
git log -S 'searchParams' -p -- src/

-S finds the commits where the string was added or removed. It ignores commits that only moved the line around. For a regular expression, or to catch any change to lines containing the text, use -G:

git log -G 'utm_[a-z]+' --oneline

It answers questions like “when did we stop calling X?” or “which commit introduced this constant?” in seconds.

The history of a single function

-L shows how one range of lines, or one function, evolved, commit by commit:

git log -L :buildRedirectUrl:src/redirect/build-url.ts

The output is a sequence of diffs that only covers that function, from the most recent change backward. For a 30-line function with a long history, it’s far more readable than git log -p on the whole file.

Blame and its limits

git blame shows, for each line, the last commit that changed it:

git blame -w -C -C -C src/redirect/build-url.ts

-w ignores whitespace changes. -C -C -C follows lines that were moved or copied from other files, so the blame points to where the code was written, not to the refactor that moved it.

Even so, blame shows the last change to a line. If someone ran Prettier on the whole project, half the file points to the commit “format code”. The fix is to list those commits in a file and tell Git to skip them:

# .git-blame-ignore-revs (one full SHA per line, with a comment)
# Prettier on the whole project
3f9c2a1e8b7d4f6a0c5e2b9d1a8f7e6c5b4a3d2e

git config blame.ignoreRevsFile .git-blame-ignore-revs

GitHub reads this file automatically in its blame view. Commit it to the repository root.

Use blame to find the commit, then read it: git show <sha> for the full diff and the message, and the PR for the discussion. What you’re after is what the author was trying to do, since a line that looks wrong often has a reason recorded only in the PR.

# Find the PR that brought a commit in (GitHub)
gh pr list --state merged --search <sha>

Bisect: binary search over the history

When you don’t know which file, which string or which function to look at, but you can test whether the bug exists, use git bisect. You give it a commit where the bug exists (bad) and one where it didn’t (good). Git checks out the commit in the middle, you test, and the search space is cut in half each time.

Commits in the range	Steps to find the culprit
100	7
1,000	10
10,000	14

For Linkr’s 180 commits, that’s 8 tests.

Manual

git bisect start
git bisect bad                 # the current commit has the bug
git bisect good v1.14.0        # this release worked

# Git checks out a commit in the middle. Test it, then:
git bisect good                # or: git bisect bad
# ...repeat until Git prints "<sha> is the first bad commit"

git bisect reset               # back to where you started

If a commit doesn’t build or can’t be tested for an unrelated reason, git bisect skip moves to a nearby one.

Automatic

If you can write a script that exits 0 when the code is good and 1 when it’s bad, Git does all the steps on its own. Keep the script outside the repository, because it has to exist in every commit bisect checks out, including old ones.

/tmp/check-utm.mts

import { pathToFileURL } from 'node:url';

// Imports from the current checkout, whatever commit bisect is on
const { buildRedirectUrl } = await import(pathToFileURL('src/redirect/build-url.ts').href);

const url = buildRedirectUrl('https://shop.example/sale', new URLSearchParams('utm_source=ig'));
process.exit(url.includes('utm_source=ig') ? 0 : 1);

git bisect start HEAD v1.14.0
git bisect run sh -c 'npm ci --silent || exit 125; npx tsx /tmp/check-utm.mts'
git bisect reset

Exit code 125 means “can’t test this one, skip it”. Here it’s used when dependencies fail to install. Any code from 1 to 127 other than 125 means bad.

A few minutes later:

a41f3c2 is the first bad commit
    refactor(redirect): build destination with the URL constructor

The refactor replaced string concatenation with new URL(destination) and forgot to copy the incoming query parameters. The tests didn’t cover links with a query string, so they passed. Now you know the cause, the reason for the change (it fixed destinations with spaces, according to the PR) and what the fix has to preserve.

If your history has many merge commits, git bisect start --first-parent only tests the commits on main and points to the merge, that is, the PR, that introduced the bug.

Pitfalls

• Flaky checks. Bisect trusts every answer. One wrong “good” sends the search to the wrong half, and the result is a commit that has nothing to do with the bug. If the check isn’t deterministic, run it a few times per step or fix it first.
• The check script inside the repository. If an old commit doesn’t have the file, the check fails for the wrong reason. Keep it outside, in /tmp or another folder.
• Giant commits. If each PR is squashed into a 3,000-line commit, bisect finds the right commit, but you still have to search inside it. Small PRs (4.1) make bisect precise.
• Forgetting git bisect reset. You’re left in detached HEAD on an old commit, and the next commit you make doesn’t land on any branch.
• Blame as an accusation. It shows who touched the line last, and that person was often just moving or formatting code. Read the commit and the PR before drawing conclusions.
• What AI tends to get wrong: asked “why did this break?”, an assistant reads the current code and suggests a plausible cause, which may not be the real one. Ask it to run git log -S or git bisect run with a check script instead. Agents are good at this, because the process is mechanical. AI is also quick to reformat whole files along with a real change. That mixes noise into the history and ruins blame. Ask for formatting in a separate commit and add it to .git-blame-ignore-revs.

Apply it to your project

1. Pick a constant or function in your project and find the commit that introduced it with git log -S. Read the PR it came from.
2. If you’ve ever run a formatter on the whole project, create .git-blame-ignore-revs with that commit and configure blame.ignoreRevsFile.
3. Next time a bug shows up with no known start date, write the check script before reading the code and run git bisect run.

Go deeper

Materials in English unless noted.

• Book
  Debugging with GitPro Git (Scott Chacon, Ben Straub)

  The chapter on blame and bisect in the official Git book. Free online.

• Docs
  git-bisectGit

  Full reference, including bisect run exit codes, --first-parent and custom terms other than good/bad.

• Docs
  git-logGit

  The -S, -G and -L options and the filters by date, author and path.

• Docs
  Ignore commits in the blame viewGitHub

  How GitHub uses .git-blame-ignore-revs.