Proposal could cut startup of large AI model from eight minutes to one

Starting a large AI model on an ordinary computer can take many minutes. Now a contributor to the Strata project has proposed a change that, according to his own measurements, brings startup time down from 503 to 62 seconds, a drop of about 88 percent.

The figures have not been confirmed by anyone else, however, and the proposal is not part of Strata. The author, chenyb999-zhcn, withdrew it himself so he could measure again.

What is Strata?

Strata is an open inference engine, meaning a program that runs an AI model on your own computer instead of on an online service. It runs DeepSeek V4.1-Flash locally. According to the release notes for version 0.2.4 from October 24, it requires one graphics card with 12 GB and 46-64 GB of RAM.

The model is divided into so-called experts, which are smaller parts of the model. Strata keeps the most frequently used ones in RAM and loads the rest from the SSD. Version 0.2.4 ships with precomputed profiles of the most frequently used experts, so the first prompts don't start cold.

What does the contributor propose?

Pull request number 4065 from October 25 proposes that the expert profile be filled in parallel at startup instead of one item at a time. The number of threads is controlled by the environment variable STRATA_FILL_THREADS, and the default value is eight.

The proposal consists of four independent changes: parallel filling in layer-expert order, parallelization of unbuffered layers, larger read-ahead steps (from 128 KiB to 4 MiB) and direct I/O reads for dense weights and embeddings.

The measurements

The contributor measured on a system with two Tesla V100 cards. These are data center cards, not gaming cards. Profile filling on CUDA0 dropped from 329 to 19.6 seconds, and stage fill dropped from 146.8 to 16.4 seconds.

The test used a ZhiTai Ti600 NVMe drive with no DRAM in the controller. It showed periodic drops in performance that were not related to the access pattern. That suggests throttling, meaning the drive slows itself down.

The outlet Startup Fortune, which covered the proposal, writes that a clean before-and-after test had not been run when its article was published. The outlet also writes that serial loading alone took 336 of the 525 seconds of startup on the system. The difference between 525 and 503 seconds is not explained.

The proposal may therefore show that startup can be significantly faster, but how much of a difference it makes on ordinary computers is unknown.