GitHub swaps AI model to find forgotten passwords in code

GitHub has put a new AI model into use in its secret scanning. It is meant to detect when developers accidentally put passwords and other secret keys into their code. The model is called ModernBERT and has been fine-tuned for the task.

If such information ends up in a code repository, others can, in the worst case, use it to get into systems they do not have access to.

How it works

The solution has two steps. First, regular expressions, which are fixed search patterns, find text strings that look like credentials. Then the AI classifier assesses each candidate based on the file path, variable names and the surrounding code or text.

The goal is to tell real passwords apart from placeholders such as "changeme," test values and redacted values. The model is trained on examples from GitHub's products, on candidates that humans have reviewed, and on synthetic examples of difficult cases such as documentation, test data and checksums.

The model covers both push protection, which stops a secret before the change is accepted, and scanning after a push.

Cheaper and faster, says GitHub

According to GitHub, the new solution has cut the cost of running the model by more than 98 percent compared with the previous solution based on a large language model (LLM). The median time for scanning after a push has been cut by 86 percent.

In GitHub's own offline test, precision was 13 percent higher and recall 19 percent higher. Precision is about how many alerts are correct, and recall is about how many real secrets are found. In an A/B test, the weighted acceptance rate for alerts rose by 11.5 percent, according to Help Net Security. All figures come from GitHub itself and have not been independently verified. It is also unclear which LLM was used for comparison.

Who gets it?

Existing customers will have their AI-detected password alerts upgraded automatically at no extra charge, according to eSecurity Planet. It requires GitHub Team or Enterprise Cloud with paid security coverage. AI-based push protection is only in private preview, and Copilot security review is coming in private preview. The new features will use AI Credits, and billing details have not been published. GitHub Enterprise Server 3.23 gets the alerts in public preview, but not the new features.

GitHub itself says that false alerts and missed credentials can still occur. The company plans to use user feedback to improve the model and to explore solutions that find and extract secrets in a single step.