A team can produce more ads without learning more about its customers. It can also improve reported return on ad spend without creating enough additional value to pay for the production work.
The useful question is specific: does increasing creative cadence improve customer economics enough to justify the extra effort?
In a promotional interview summary, Harry Stebbings argues for high creative output alongside holdout measurement. That pairing is worth exploring. The supplied summary, however, does not establish a creative volume target that another business should copy.
Treat faster production as a hypothesis. Give it a budget, a comparison, and a decision date.
Define the change before counting assets
“More creative” can mean more customer problems explored, more formats, more hooks, or more variations of an existing ad. Those changes answer different questions.
A new concept changes the reason someone might care. For example, one concept could explain how a product reduces setup work, while another demonstrates how it prevents a recurring mistake. Changing the opening line of the same demonstration is a variation within that concept.
Both can be useful. Counting them together makes the result difficult to interpret.
Choose the change you want to evaluate. A practical hypothesis is:
At the same media budget, introducing more distinct creative concepts will improve new-customer contribution enough to cover the additional production cost.
Record the current cadence and the proposed cadence using the same definition. Also specify whether older ads stay active, how new concepts receive delivery, and when weak concepts are retired. Otherwise, the treatment becomes a bundle of undocumented changes.
Give every concept a job
Before production, create a short record for each concept:
- The customer situation it addresses.
- The problem or motivation it leads with.
- The claim and supporting proof.
- The offer and intended landing page.
- The uncertainty the concept is meant to resolve.
An uncertainty might be whether buyers respond more strongly to avoiding mistakes or finishing a task faster. That is more useful than a brief asking for something fresh.
Keep concept identifiers connected to the resulting assets and customer outcomes. This lets the team distinguish a promising message from one unusually effective execution.
Ad performance still reflects several things at once: message, format, audience delivery, and the path after the click. Treat a winning concept as a lead for further investigation, rather than proof of a universal customer preference.
Build a comparison you can interpret
Compare the existing cadence with the proposed cadence in groups that are as comparable as the available setup allows. Random assignment is preferable when feasible. Geographic comparisons may be practical, but differences in local demand and market conditions can make their results harder to interpret.
Keep media spending, bidding approach, offers, landing pages, and eligibility comparable. If a concept requires a different page, acknowledge that the experiment evaluates the combined ad-and-page change.
Apply the same rules for stopping poor ads and allocating spend in both groups. Record exceptions. Moving budget toward the treatment halfway through because it looks promising changes the question being tested.
There are two distinct measurement questions:
- Does the higher cadence outperform the current cadence? Compare customer outcomes between the two advertising groups.
- How much demand does advertising create beyond what would happen without it? Use an appropriate no-ad holdout.
The first comparison can support a decision about the production process. It cannot, by itself, establish total advertising incrementality. If you need both answers, design the holdouts before launch and check whether the available scale can support them.
Measure the economics of the change
Choose a customer outcome your business can observe consistently. New-customer contribution over a defined window is one option: revenue from those customers minus the variable costs relevant to serving them, with refunds and cancellations handled consistently.
State whether advertising and production costs are included in that measure. Avoid subtracting them twice.
When media spending is comparable, the central decision is whether the estimated improvement in contribution exceeds the additional creative cost. Include production, editing, review, trafficking, and any added contractor or tooling expense.
Track acquisition cost, customer quality, and repeat behavior as guardrails. A faster cadence could attract more first purchases while bringing in customers who are less likely to return.
Give each group the same amount of time to mature. Comparing customers observed for a month with customers observed for a week creates a misleading difference. Early results can help identify delivery problems; they should not substitute for the customer outcome selected for the decision.
Set the decision rule before launch
Choose the smallest economic improvement that would make the change worthwhile. Then assess whether the available audience, spending, and observation window can distinguish that improvement from ordinary variation.
There is no universal test duration. The calendar should reflect purchase frequency, conversion delay, repeat behavior, and the amount of evidence available.
Write down three possible decisions:
- Expand the cadence when the evidence supports a worthwhile improvement after added costs, with acceptable guardrails.
- Retain the current cadence when the evidence indicates the added effort does not earn its cost.
- Call the result inconclusive when the uncertainty still includes materially different decisions.
An inconclusive result is useful if it prevents a weak signal from becoming a permanent staffing or spending commitment.
Watch for ways the experiment can mislead you
Too little delivery per concept. At a fixed budget, increasing the number of concepts leaves less spending available for each. Record actual exposure. Uploading an asset does not mean it received a meaningful test.
Quality changes with cadence. Rushed production may weaken proof, clarity, or execution. That is part of the practical cost of the proposed operating model, but it limits any conclusion about creative quantity in isolation.
Groups overlap. Customers may encounter both sets of ads, and geographic boundaries may not prevent exposure. Document likely contamination and reduce confidence accordingly.
Reporting shifts without customer improvement. Attribution reports can change while total customer outcomes barely move. Use consistent business records and the planned comparison to inform the decision.
A temporary benefit becomes a permanent assumption. A winning test supports the tested cadence under the tested conditions. It does not establish that doubling output again will help.
Automating briefs, variations, tagging, and reporting may reduce execution cost. Keep human review focused on claims, concept quality, and consequential budget decisions. Faster execution is valuable when it makes a useful experiment cheaper or clearer.
Evidence and limitations
This playbook draws on a supplied summary of an X post by Harry Stebbings promoting an interview. The summary advocates creative scale and holdout measurement, but provides no underlying experiment results, production economics, or complete methodology.
The comparison structure, cost accounting, and decision rules here are operator recommendations. They are not reported outcomes from the source or a claim of firsthand execution. No particular creative volume or improvement is established by the available evidence.
A credible result will depend on assignment quality, delivery, customer identification, observation windows, and measurement uncertainty. Use the experiment to decide whether one defined increase in cadence earns its cost.
Source basis
- Supplied summary of Harry Stebbings’s public X post advocating higher creative output and holdout measurement.
- Source limitations concerning missing experimental methods, production costs, and supporting results.
- Original operating recommendations for testing creative cadence, interpreting comparisons, and accounting for added costs.