Engineering
How We Evaluate MinuteShark AI Without Letting It Guess Your Timesheet
Evaluating AI time tracking is not the same job as scoring a chatbot. A wrong trivia answer is embarrassing. A wrong time entry can become an invoice line, a retainer argument, or a week you cannot explain.
MinuteShark AI is a workspace copilot. It can add a client, start a timer, or log the hour you forgot after a call. Those writes only happen after you confirm a proposal card. This post is not another tour of that card. It is how we evaluate the assistant before the card is worth tapping.
The split we test: a read path answers “how much did I track today?” from stats you already have. A write path proposes a mutation — log time, rename a client, start a timer — and waits. Mixing those paths is an eval failure, not a clever shortcut.
The assistant runs on Ionic and Angular: a floating widget on web, a full chat on iOS and Android through Capacitor. Firebase Auth, Firestore, and Functions hold the commit. The model is not the database, and the database does not move until a human says so.
What contract are we actually evaluating?
Product evals fail when they chase “sounds helpful.” MinuteShark AI has a narrower contract. It manages a workspace in plain English. It is not a general Q&A bot. If you ask for the weather, it should stay a copilot and decline, not invent a forecast and a timesheet.
The utterances on our MinuteShark AI introduction are the acceptance suite we re-run when models, prompts, or tool wiring change:
- Onboard a client, then a project, then a task, then log time on that task.
- Bulk-add clients in one sentence.
- Rename, move, or delete without picking the wrong node in the tree.
- Log forgotten time on a named day (“yesterday’s standup”).
- Start a timer on a specific task — one running timer, not a second clock.
- Open Reports with filters, or answer how much was tracked today from supplied stats.
If those utterances regress, the blog post about confirmation UX will not save you. Confirmation is the last human gate. Evaluation is how we keep the card from being confidently wrong in the first place.
Why a read question must never become a write
“How much time did I track today?” is a read. The honest implementation is: look at the stats already in context, answer with those numbers, and do not create an entry to make the answer nicer.
That sounds obvious until you watch a tool-calling model try to be useful. If the day looks empty, a “helpful” agent wants to fill the gap. If the user mentioned a standup, it wants to log thirty minutes. If two tasks share a nickname, it wants to pick one and keep talking.
We treat that as a failed eval even when the invented entry would have been plausible. The read path is allowed to say “I don’t see time on that filter yet.” It is not allowed to quietly repair the dashboard.
Two paths, two passing grades
Read path. Questions about today, this week, or who worked on which customer. Context includes timezone, work-week bounds, the active timer, recent activity (about seven days), the visible catalog, and permissions. The answer must cite that context. It must not insert time.
Write path. Log, edit, or delete an entry; add or rename a client, project, or task; assign; add a note; start a timer; close or open a task. The assistant returns a proposal card. The user can edit the card. Firebase Functions commit only on Confirm. Many actions can be reverted from chat if you change your mind.
An eval case that starts as a read and ends as a write is a bug, regardless of how polite the prose was.
When is ambiguity a passing grade?
Freelancers reuse names. “Homepage” exists under two clients. “Standup” exists on three projects. “Acme” might mean the old legal entity or the new one you have not renamed yet.
A model that always resolves ambiguity looks decisive in a demo. It is dangerous in a billing product. Our passing behavior is a short clarification or a picker: which client, which task, which day. That is slower than a confident guess. It is also the difference between a copilot and a liability.
Timezone is the quiet version of the same problem. “Yesterday” at 12:10 a.m. is not a vibe. Evaluation includes work-week bounds and the user’s timezone in the context bundle, then checks that “today” and “yesterday” land on the dates a consultant would defend on an invoice.
What context is in bounds — and what is not
The assistant sees what a user can see: the permitted catalog, the running timer, recent activity, and the same client → project → task → subtask tree. No second workspace. No screenshot of your screen. MinuteShark is not an employee-monitoring product, so we do not evaluate “did the model infer work from app usage.” Time lives on a task because someone started a timer, typed an entry, or confirmed an AI proposal.
Permissions are part of the suite. A regular user should not be coached into an admin-only mutation. If the catalog is filtered, the picker must not offer a project the user cannot see. Eval cases that ignore roles are not “edge cases.” They are how a small team stops trusting chat.
Chat history is kept for up to seven days, with multiple conversations. That window is also the recent-activity window we feed the model. We do not pretend the assistant remembers last quarter unless that data is in the current context.
Where the stack shows up in the test, not the demo
People ask which model we use. We have written about that separately. Evaluation is less glamorous and more load-bearing:
- Ionic + Angular 16. The web widget has to render a card you can actually edit, not a markdown dump that looks like a form.
- Capacitor 7. Mobile is a full chat plus voice: microphone, transcript review, then send. We evaluate that the transcript checkpoint exists. Raw audio does not become a write.
- Firebase Functions + Firestore. Confirm is a server commit so web and phone cannot disagree after a flaky optimistic update.
- Algolia. Search still starts tasks when chat is the wrong interface. Eval must not require every action to go through the sparkles icon.
- PostHog. We watch whether people open a card and then abandon it. Rubber-stamping is a product bug, the same as a wrong tool pick.
We do not score the assistant on writing essays about Chart.js, Quill, or invoice PDF libraries. Those are app features. The copilot should open Reports with filters or point you at the report you asked for — not invent a chart in chat.
What a good failure looks like
Out of scope should be short. “I can help with your workspace — clients, projects, tasks, timers, notes, reports — not general questions.” That is a pass. A long apology plus a guessed time entry is a fail.
Destructive actions should look expensive on the card: delete, move, rename. Eval checks that the proposal names the actual node, not a cousin with a similar label. If the user confirms, they confirmed that move.
AI usage is metered per user per calendar month (Starter 30, trial 120, Standard 300, Enterprise 2,000). Evaluation is not an excuse to burn the quota on retries. A picker is cheaper than three wrong writes you then revert.
Why this matters if you never read an eval doc
If you invoice from time, you need an assistant that is boring in the right places: no silent edits, no padding an empty day, no treating GitHub or a calendar as a billable hour. Propose the task and duration you named, wait, then write.
You can see the same philosophy in the rest of the product — the features list is still clients, projects, tasks, reports, and invoices, not a marketplace of silent integrations. The assistant is that structure, in English.
If you want to feel the eval from the other side, open the app, tap sparkles, and ask how much you tracked today. You should get a read. Then ask it to log the last hour. You should get a card. That difference is the whole design.