Cheap AI model passed only half of tough work tasks

China's DeepSeek has launched the V4 Flash model, and it is cheap to use. But a new test from the company Composio shows that it barely manages half of more demanding work tasks.

Composio ran the model in eight different agent harnesses. A harness is the program that surrounds an AI model and lets it use tools and carry out tasks on its own. Among them were Claude Code, Codex, OpenCode and Hermes Agent. The tasks were 30 deliberately difficult multistep workflows in which the model had to use real services such as Gmail, Google Calendar, Google Sheets, GitHub, Slack and Airtable. Each task had a time limit of 900 seconds, and the results were checked programmatically rather than by another AI.

Big differences between the tools

Overall, 129 of 240 runs passed, or 53.8 percent. Only 6 of the 30 workflows succeeded in all harnesses.

Pi Agent did best at 66.7 percent, followed by Prime Agent at 62.5 percent. However, six runs were excluded there because of timeouts and registration errors. At the bottom was OpenCode at 46.7 percent. Claude Code, Codex and DeepAgents all landed at 53.3 percent.

Composio concludes that harnesses can matter as much as the models they run, and that no single harness is best on pass rate, speed and price all at once.

The price is low, but is it real?

According to the outlet Crypto Briefing, V4 Flash costs $0.14 per million input tokens (small units of text that AI models count in). That is about one tenth of comparable American models. The outlet judges, however, that the savings disappear once you factor in the failed runs and the time spent debugging. According to Crypto Briefing, DeepSeek offers the model as a public beta.

Hallucinations depend on mode

On the release day, July 31, 2026, a developer tested how often the model makes things up. He asked 10 difficult questions about topics that do not exist or cannot be known. In thinking mode, where the model reasons as it goes, the hallucination rate was about 0 percent. But five of the ten questions produced a completely empty answer, because the model spent its entire output budget on thinking.

In non-thinking mode there were no empty answers, but the hallucination rate was about 20 percent. It mostly involved invented works with made-up authors and plots.

The test is small and was run on only one day, so it is indicative at best. The developer compares it with a rate of 94-96 percent that circulated after a preview release in April. He believes the stable version is markedly better, and he recommends non-thinking mode for agent workflows.

For ordinary users, the takeaway is that a cheap model is not necessarily cheaper in practice, and that the choice of tool around the model can matter a great deal.