How To Remove Secrets From The Git History Remove Secrets From The Git History

Removing secrets with BFG Repo-Cleaner

BFG Repo-Cleaner is a tool designed for removing large files and sensitive data, such as passwords and credentials, from a Git repository history.

The first thing to do before using BFG on a repository is to clone it on your local machine using the git clone command with the --mirror flag:

$ git clone --mirror <url>

Run in Warp

Where:

• <url> is the URL of the Git repository you want to clone.

This will create an exact and complete replica of the original repository, including all branches, tags, and commits. Note that it is usually recommended to make a backup of this copy in order to prevent unintentional or unexpected data loss.

Next, you can download the BFG Repo-Cleaner from the official website and run the following command to clean your repository:

$ java -jar bfg.jar --delete-files <files> <repository>

Run in Warp

Where:

• java -jar bfg.jar is used to execute the BFG tool.
• <files> are the files you want to delete from the repository's history.
• <repository> is the path to the Git repository on your local machine you want to clean up.

Finally, once the files have been removed by BFG, you can push the updated history version to the remote repository using the git push command:

$ git push

Run in Warp

Note that, although BFG will clean the repository's commits, branches, and tags, it will not physically eliminate the unwanted data. For a thorough removal of this data, please refer to the "Removing all references" section later in this article.

Removing secrets with git-filter-repo

git-filter-repo is a versatile and powerful tool designed for filtering and rewriting the history of Git repositories, modifying commit messages, combining repositories, and more. It is especially useful for removing files containing sensitive information, managing large files, and restructuring repositories for optimal efficiency and security.

Installing git-filter-repo

To install it on Windows, you can use the following pip command:

$ pip install --user --upgrade git-filter-repo

Run in Warp

To install it on macOS, you can use the following brew command:

$ brew install git-filter-repo

Run in Warp

To install it on Linux, you can download the executable and place it anywhere in your $PATH to use it. See the installation documentation for more information.

Removing sensitive files

To remove sensitive files from a Git repository history, you can use the following git filter-repo command:

$ git filter-repo --path <files> --invert-paths

Run in Warp

Where:

• <files> are the path to the files you want to remove.
• --invert-paths is used to include all files except those explicitly mentioned by --path.

Note that, by default, the git filter-repo command will automatically remove all references to the remote repository in order to avoid accidental updates before everything is ship-shape.

After you've double-checked that the repository is as you want it to be, you can manually re-add those references using the following git remote command:

$ git remote add origin <url>

Run in Warp

Where:

• <url>: is the URL of the Git repository you are fixing.

To update the remote repository and forcefully overwrite its history, you can use the git push command with the --force flag as follows:

$ git push origin --force --all

Run in Warp

Finally, to ensure that the sensitive files are also removed from your tagged releases, you can perform a force push against your Git tags as follows:

$ git push origin --force --tags

Run in Warp

You can learn more about this command by visiting the official user manual page.

Note that, both the BFG Repo-Cleaner and git-filter-repo are pretty heavy-handed as they remove references and changes to these files since the beginning of the repository's existence. However, if the change is more recent (specifically the last Git commit you made), undoing the last Git commit might be of better service to you. 

Easily retrieve this command using Warp’s AI Command Suggestions

If you’re using Warp as your terminal, you can easily retrieve the git-filter-repo command using the Warp AI Command Suggestions feature:

Entering remove secret from git in the AI Command Suggestions will prompt a git command that can then quickly be inserted into your shell by doing CMD+ENTER.

Removing all references

Although tools like BFG and git-filter-repo will clean up your repository's history from secrets, it is possible that the repository may still have references to some of these files and commits. To completely clean the repository from all references, you have to delete out-of-date references, expire and prune reflogs, force garbage collection, and encourage your collaborator to rebase their branches.

Deleting out-of-date references

In order to completely delete all old references (i.e. branches and tags) pointing to the previously cleaned up files and commits, you can use the following git command:

$ git for-each-ref --format="delete %(refname)" <namespace> | git update-ref --stdin

Run in Warp

Where:

• The git for-each-ref command is used to iterate over references in a Git repository.
• The --format="delete %(refname)" flag is used to specify the format for each reference to be deleted followed by the reference name. This is used to create a list of commands to delete references.
• The <namespace> argument is the source of the references you want to process.
• The git update-ref --stdin command is used to read and execute the commands generated by the git for-each-ref command.

For example, the following command will delete all the references in the refs/heads/feature/ namespace:

$ git for-each-ref --format="delete %(refname)" refs/heads/feature/ | git update-ref --stdin

Run in Warp

Expiring and pruning reflogs

Once all the references have been deleted, the next step consists in deleting the repository's reflogs, which are records of the various updates made to the repository over time.

To expire the reflogs, you can use the following git reflog command:

$ git reflog expire --expire=now --all

Run in Warp

Where:

• The --expire=now flag is used to expire all reflog entries immediately, which means that any reflog entry older than this instant will be considered for deletion.
• The --all flag is used to include all the reflogs of all the branches and references in the repository.

Forcing the garbage collection

To thoroughly clean up your repository and permanently erase any residual traces of sensitive data, you can perform a manual garbage collection using the following git gc command:

$ git gc --prune=now

Run in Warp

Where:

• The --prune=now flag is used to ensure that the garbage collection process also prunes unreachable objects in the repository that are not referenced by any branch or tag.

This will enhance your repository's security and cleanliness by removing unnecessary files and optimizing Git's objects.

Encouraging your collaborators to rebase

Lastly, since Git is by design a distributed system, some of your collaborators may still have bits of the old repository's history on their local machines.

This is why you should encourage them to rebase their branches instead of merging them, as it will minimize the chances of unintentionally reintroducing commits and references to the previously removed sensitive data back into the repository's history.

Preventing future accidental commits

While knowing how to clean up a repository's history is crucial to maintain its integrity and security, it is equally important to implement practices that will reduce the likelihood of sensitive data exposure.

Reviewing changes before pushing

The first thing you can do to prevent pushing sensitive data to a remote repository is to carefully review the staged files before they are committed using the git status command.

You can also use other Git tools such as lazygit, which offers a more visual representation of branches, commit history, files, and changes.

Finally, you can enhance your text editor with Git plugins, such as magit for Emacs or fugitive for Vim and Neovim, that provide support for Git commands and functionalities in a more interactive and visual way.

Adding sensitive files to .gitignore

The easiest way to avoid accidentally committing unwanted, sensitive, or large files to a repository is to list their paths into the .gitignore file.

This will tell Git not to track them and prevent them from being included in the staging area, even when staging the entire project using commands such as git add . or git add -A.

For example, you can use the echo command the following way to add a file to the .gitignore file:

$ echo "<file_name>" >> .gitignore

Run in Warp

Where

• <file_name> is the path to the file that contains your sensitive data.

Note that the .gitignore file should also be committed to the repository to ensure consistency across all the environments the project is cloned in:

$ git add .gitignore 
$ git commit -m "Update gitignore"

Run in Warp

Using Git hooks

With Git Hooks, you can write "pre-commit" hook and "pre-push" hook scripts that will scan the staged files for secrets using commands like grep or sed, and prevent the files from being committed to the history or pushed to the remote repository until the data is removed.

Note that you can also use more advanced tools like git-secrets that will automatically scan the files being committed for any matches with a list of user-defined patterns or regular expressions that are commonly associated with sensitive information.