A new model arrives. Someone shares an impressive demo. Your team spends an afternoon trying it, finds a few promising uses, and moves on. Next week, the cycle starts again.
Exploration can be useful. The problem is having no way to distinguish learning that improves future work from activity that merely feels productive.
David Ondrej’s “Directionally correct” argument offers a useful starting point: when precise AI forecasts are unreliable, invest in capabilities that could improve your position across several possible futures. His examples include technical fluency, research literacy, model experimentation, and building agents.
For a growth team, that principle needs a budget, a deliverable, and a stopping rule. Otherwise, almost any tool experiment can qualify as a sensible investment.
The operating system below is a proposed way to make those choices. Its central test is simple: what useful thing remains after the exploration ends?
Start with a recurring business problem
“Get better at AI” is too broad to guide a week of work.
Choose a problem the team encounters repeatedly. Perhaps customer research takes too long to turn into testable messaging. Perhaps campaign analysis requires repetitive preparation. Perhaps generated content needs so much correction that the apparent speed advantage disappears.
Describe the work before choosing a tool:
- What decision or output does the task support?
- How often does it occur, and who owns it?
- What does acceptable quality look like?
- Where does the current process consume time or introduce errors?
That description tells you what to measure for a baseline and gives you a reason to invest. If you cannot identify a likely user or the next occasion for using the output, keep the idea in exploration rather than treating it as an operating priority.
A capability is worth considering when it addresses this problem and remains useful beyond one implementation. An evaluation set can survive a vendor change. A documented review process can improve several workflows. A tool-specific shortcut may still be valuable, but its narrower usefulness should affect how much time you give it.
Create a small capability portfolio
Start with three candidates. This is a practical constraint, not an empirically established optimum: enough room to compare investments without creating an unmanageable learning backlog.
For each candidate, write a short investment record:
| Field | What to specify |
|---|---|
| Business use | The recurring task and decision it supports |
| Capability | The skill or infrastructure being developed |
| Deliverable | A concrete artifact someone can use |
| Reuse opportunity | When and where it should be used next |
| Baseline | Current quality, effort, and cost |
| Budget | Time and incremental spending allowed |
| Decision date | When to continue, revise, or stop |
For example, model evaluation could produce a labeled set of research-synthesis tasks. Workflow automation could produce a bounded process for preparing an experiment report. Human-review design could produce acceptance criteria and an escalation checklist for generated marketing drafts.
These are illustrative investments, not reported results.
Prioritize the candidate with a near-term use, a clear owner, and a plausible benefit relative to its cost. Portability matters, but it should not outweigh usefulness. A beautifully reusable framework that nobody needs is still overhead.
Evaluate models on stable tasks
If the task changes every time you try a model, comparisons tell you very little.
Create a small evaluation set from representative work you are authorized to use. Include ordinary cases and cases where a plausible error would change a decision. For research synthesis, that might include contradictory feedback, sparse evidence, and requests that require distinguishing customer statements from interpretation.
Write acceptance criteria before evaluating outputs. Keep the inputs and criteria stable when comparing candidates, and document differences in setup or assistance.
Track dimensions separately:
- Quality: Is the output accurate and useful for the intended decision?
- Reliability: Does it handle difficult cases consistently?
- Latency: How long does the user wait?
- Review effort: How much checking and correction are required?
- Cost: What does each accepted output cost, including human work?
A fast draft with a long correction cycle may offer little benefit. Evaluate the full path to an accepted output.
Refresh the evaluation set when the work changes. Keep a stable portion for comparison and a separate set for newly discovered failures. Otherwise, improving scores may reflect familiarity with the test rather than broader usefulness.
Run a bounded capability sprint
Give each sprint one deliverable and one intended reuse opportunity.
A useful brief might be: “Build a research-synthesis evaluation set that the team will use in its next model comparison.” That is specific enough to finish and inspect. “Explore research agents” leaves both completion and value open to interpretation.
At the end of the sprint, record what was built, hours invested, defects discovered, and whether someone used it in real work. Schedule a later check for reuse; a deliverable completed today may not have had a fair opportunity to prove useful.
One possible test is an eight-week comparison that alternates structured sprint weeks with unstructured exploration weeks, keeping time budgets equal. This is a proposed extension of the principle, not a study reported by Ondrej. It could provide a learning exercise, but alternating weeks would not establish causality. Skills carry over, tasks differ, and early deliverables have more time to be reused.
If you try it, give each deliverable the same follow-up window and use consistent usefulness criteria. Treat the comparison as limited evidence for your next allocation decision.
Make reuse earn its place
Reuse is a useful signal because it asks whether an artifact escaped the learning session. It is not sufficient evidence of value.
A workflow can be reused because it is mandatory. A template can spread while degrading decisions. Pair reuse with quality, correction effort, and the business purpose defined at the start.
At the decision date, choose among three actions:
- Continue: The artifact has been useful in real work, and another investment has a specific expected benefit.
- Revise: The need remains, but quality, integration, or review effort prevents useful adoption.
- Stop: No credible use has emerged, costs exceed the budget, or a simpler approach meets the need.
Before approving another sprint, name the work it would displace. Improving an AI workflow may be less valuable than resolving a known activation problem or running an overdue customer interview. Capability building needs to compete for time on those terms.
Watch for activity disguised as progress
One failure mode is rewarding production of artifacts instead of improvement in work. More prompts, agents, and evaluation documents do not automatically mean more capability.
Another failure is hiding maintenance. A workflow that saves effort during use may still require frequent repairs. Include that work in its cost.
Finally, avoid declaring a capability durable simply because it sounds fundamental. Durability is a hypothesis. Look for evidence that the skill or artifact transfers across tasks, survives implementation changes, or reduces the effort required for the next useful project.
Evidence and limitations
The starting principle comes from David Ondrej’s argument for building useful capabilities under AI uncertainty. Attribution relies on the available account of his argument; the original post and linked article were not independently checked for this article. The available account contains no causal evidence or documented business outcomes demonstrating that this approach outperforms other planning methods.
The portfolio, evaluation process, and stopping rules here are operator recommendations developed from that principle. The examples are hypothetical, and the suggested sprint comparison has not produced reported results.
Reuse, quality, effort, and cost can inform local decisions. They do not establish incremental revenue or long-term career value. Use this system to make capability investments inspectable while retaining broader strategy, scenario planning, and business priorities.
Source basis
- The supplied account of David Ondrej’s conceptual argument for developing useful AI capabilities under uncertainty.
- Critical analysis of that principle and a proposed, untested capability-sprint design.