Choosing an LLM Isn’t About Chasing the Newest One
We rotated base models on MinuteShark AI a few times — GPT-5.4 mini, then GPT-5.6 Terra and Sol, then Luna — to compare speed, accuracy, and cost.
The tempting story is that newer = better. The real story was quieter: for our workload, accuracy was already good enough. What changed between models was mostly speed, price, and how much prompt work we had to do to keep behavior stable.
Bottom line: Don’t upgrade because a model is new. Upgrade because you have a real pain point — slowness, accuracy gaps, caching behavior, cost, or something users can feel.
What We Were Actually Testing
MinuteShark AI isn’t a chatbot that answers questions and stops. It’s a workspace assistant with tool calling and confirmation cards: log time, add clients and projects, start timers, generate reports.
That means “model quality” for us isn’t a vibes score. It’s whether the model:
- Picks the right tools
- Keeps multi-step flows on track
- Doesn’t invent actions it can’t do
- Feels fast enough that confirmation cards show up before the user loses interest
Once the product architecture is solid — resolve, propose, confirm — the model becomes one dial among several. Not the whole product.
What We Found
Terra & Sol: good, but overkill
Strong models. For our assistant flows, they didn’t meaningfully beat Luna on accuracy. We were paying for more capability than the job required.
Luna: same accuracy, better fit
For logging time, setting up work, and pulling reports, Luna landed in the same accuracy band — with a better speed/cost profile for high-volume product use.
GPT-5.4 mini: just as accurate, slower
Accuracy held up. We did need to adjust the prompt a lot to keep behavior consistent — but the bigger gap wasn’t intelligence. It was latency. Users feel that immediately.
The Trap of Chasing New Models
Every release cycle invents a reason to migrate: better reasoning, better tools, better caching, better price.
Some of those reasons are real. Many are not your reasons.
If your assistant already routes tools correctly and users are happy with the outcomes, jumping to the flagship tier can become an expensive habit. You spend engineering time on the swap, re-tuning prompts, re-checking edge cases — and the user experience barely moves.
A useful question: If we switch models tomorrow, what user-visible problem gets better?
If you can’t answer that in one sentence, you’re probably chasing novelty.
Choose From a Pain Point
Model selection got clearer when we stopped ranking models in the abstract and started ranking our problems.
Speed
If cards feel sluggish, latency is a product bug — not a badge of honor for using a heavier model.
Accuracy
If the model keeps picking the wrong tool, skipping steps, or needing constant clarification, you have an accuracy problem. Fix prompts and architecture first; then decide if the model is still the bottleneck.
Caching & cost
If usage is climbing and prefixes are large, caching behavior and write/read pricing matter as much as sticker price per million tokens.
Prompt fragility
If a model only works after heavy prompt rewrites, include that migration cost in the decision. “Cheaper tokens” can still be expensive engineering.
What We’re Taking Forward
We’ll keep evaluating new models. That’s part of building a production assistant.
But the bar is no longer “is this the latest?” It’s:
- Does it fix a pain we can name?
- Does accuracy stay at least as good on real MinuteShark flows?
- Is the speed/cost tradeoff better for users and for us?
- How much prompt and telemetry work does the swap actually require?
For MinuteShark AI right now, that pointed us away from overkill flagship tiers when Luna already matched the accuracy we needed.
Reflection: The best model isn’t the newest one. It’s the one that solves your current bottleneck without making the rest of the system more expensive, slower, or harder to maintain.