Free trick makes AI tool llama.cpp up to 42 times faster

llama.cpp is a free, open source program that makes it possible to run artificial intelligence right on your own computer instead of sending data to a cloud server. The program was created by the Bulgarian developer Georgi Gerganov and is used by millions of developers around the world.

A group of developers has now improved a technique in the program called "prompt lookup decoding." In short, the AI model keeps track of text it has already seen or written itself. If it needs to repeat something similar, such as editing code or making changes to a document, it can recognize the pattern and write several words at a time instead of one by one. This makes the process significantly faster.

With the latest optimizations, this recognition process is up to 42 times faster than before, and it also uses far less memory on the computer. If you add an extra optimization from researcher Daniel Lemire on top, the total improvement reaches around 140 times, according to tests.

But there is an important catch: The benefit only applies to tasks where the text repeats itself, such as code editing, JSON data or changes to existing text. If, on the other hand, the AI model has to write something completely new, such as answering a question freely or carrying on a normal conversation, there is no extra speed to be gained. Testers have also confirmed that the technique makes no noticeable difference in regular chat.

This means the gain is mainly of interest to companies and developers who use AI for coding or automated edits in large amounts of text, and who run their AI models themselves instead of using a paid service like ChatGPT.

The broader point of the story is still worth noting: Much of the debate about AI costs has been about building bigger and more expensive computing facilities. This case shows that there are still large savings to be had simply by making existing software smarter, without it costing a cent extra in hardware.