Defining product success metrics: what does the dashboard actually count?

Before developing a product dashboard, agree on what each metric means. Define which event counts as success, who belongs in the measured group, the observation window and what remains unobservable. Without these definitions, a number can be calculated consistently while answering a different question from the one the business asked. In a technology product specification, defining the metric is part of the behaviour to implement and test, not merely a label on a reporting card.

What does the DeVry story illustrate about definitions?

In January 2016, the FTC announced a lawsuit against the operators of DeVry University. The agency alleged that graduate-employment advertising misled consumers, including through classifications of employment in a field and exclusions of graduates seeking work. These were allegations presented when the complaint was filed, not our statement about the outcome of the proceedings. The story does not establish that the number was correct; it illustrates why inclusion and classification deserve scrutiny.

Our proposed product lesson is to distinguish calculation from meaning. Even if two developers return exactly the same percentage, ask whether the numerator represents the intended success and the denominator includes the relevant people. An overly narrow definition can conceal failures, while an overly broad one can attribute success to a product that did not produce it. Both choices should be visible before anyone relies on the report.

Start with the decision the metric should inform

Write down which decision the evidence will change: improving a form, expanding a pilot or investing in a sales process. Then define the corresponding outcome. A button click may help assess interest but does not prove that a request was saved. A saved request does not prove that a meeting was booked. If the report is intended to inform lead-quality decisions, its source needs to represent that later stage rather than only the beginning of the journey.

Give each metric a plain-language definition that a non-developer can check. A hypothetical example is the number of unique enquiries the server confirmed as saved during a specified period, excluding identified tests. This sentence forces decisions about uniqueness, confirmation authority and test identification. It is better to expose these questions during the product specification than after the team has already presented a chart to management.

Define numerator, denominator and time window together

When reporting a success rate, name both parts of the fraction. Does the denominator contain visitors, identified users, organisations or action attempts? One person can make several attempts, so an attempt-based ratio differs from a person-based ratio. No single unit fits every purpose. Choose the unit that supports the decision and display it next to the number so readers do not have to guess how the percentage was constructed.

Align the observation window too. Dividing meetings booked this week by enquiries received this week may combine different groups if some meetings came from older enquiries. You can measure activity within a period or follow a cohort of enquiries over time, but these are different reports. When following a cohort, allow time for outcomes to emerge and avoid comparing a newly created group with one that has completed a full handling cycle.

A documented example: saved enquiries on Julius's website

During a production check of Julius's website on October 15, 2026, explicitly labelled test enquiries were submitted on both domains. Verification checked that they were saved as CrmLead records in CodeManage and that the corresponding Umami event appeared when analytics consent was present. The record distinguishes CRM persistence from browser-side measurement. This is focused evidence that the integration worked during that check, not a measurement of business success.

The goal has a limited meaning: an enquiry saved and confirmed by the CRM. It is neither a booked meeting nor a sales-qualified lead. Enquiries can also be saved without analytics consent, so the analytics event total is not necessarily the total enquiry count. That distinction changes how the data should be read. A difference between the two tools alone does not establish data loss, and a successful integration test does not establish improved conversion.

Test the metric with small, explainable examples

Before launch, prepare cases with known expected counts. Include a successful operation, a failure before persistence, a repeated submission, missing information and an internal test. For each case, explain whether it enters the numerator and denominator and why. Also test a late-arriving event and the day boundary in the chosen time zone. The purpose is to understand the rule rather than generate a large, impressive-looking number.

Define what happens when the answer is unknown. A user without a consistent identifier or a pending outcome is not necessarily a failure, but neither is it a success to invent. Preserve an unknown state where necessary and show its size. Document deduplication and excluded records without transferring unnecessary personal details into analytics. This makes it possible to explain a changed result without reconstructing afterward which definition was silently altered.

Version definitions and distinguish measurement from claims

A definition change needs a date, an owner and an explanation. If you move from counting clicks to counting confirmed saves, do not join the two series as though they were one metric. Show an overlap period or mark the change. The same principle applies to AI development: the share of requests returning an answer is not the share returning a correct answer. Our guide to choosing AI developers explains why quality and failure cases need definitions separate from the technical model connection.

When requesting a proposal from a software development company, include a short metric list with the business decision, data source, counting rules, observation window and acceptance check. Identify who approves the meaning and who maintains collection. The dashboard can then become something people can inspect and explain, rather than a collection of numbers whose names promise more than the system actually knows.