# How do you know whether a website redesign worked?

Decide what the redesign should improve before you release it. Keep a usable baseline, record which version people see, and check that the new journey works. You can then investigate whether the business outcome improved. Skip the measurement checks, and a duplicate activation event can make an ordinary release look spectacular.

Keep three questions separate: did you fix the problem, did the numbers change, and did your fix cause that change? Verify the first directly. The second needs dependable measurement; the third usually needs a credible controlled comparison. Some questions may remain open.

## Turn the redesign brief into a testable claim

“Make the site feel modern” gives the designer a direction. It gives the person evaluating the release much less to work with. What should people be able to do more easily, and why would that matter to the business?

For an illustrative project-management product, the brief might become: “New trial owners cannot tell how to start their first project. Explain the blank-project and template options on the welcome page so they can begin useful work.”

Define activation as the service saving a workspace's first project with at least one task within seven days of its owner's first welcome-page exposure. Count each eligible new trial workspace once. Template previews are diagnostic events; creation errors and unwanted duplicate projects are guardrails.

Clearer choices should help owners begin a project. If activation rises but those choices never appear, something else may be responsible. Don't let a favorable number rescue an explanation that doesn't fit what happened.

## Save the context while you still have it

Before the old page disappears, record its version, event definitions, filters, date range, and exclusions. Save successful and unsuccessful journeys, with campaign and device context. Someone outside the release meeting should be able to understand what these numbers represent.

Check these details before release:

- A conversion means the same business outcome in both versions.
- Staff, synthetic tests, and known automated traffic have consistent treatment.
- The unit counted, whether a workspace, visitor, or session, stays consistent.
- The observation window allows the outcome to occur.
- Consent and capture settings are documented.
- Releases to related pages are recorded too.

If the old measurement is unreliable, say so. You can still verify the new experience and start a clean measurement period. Forcing two incompatible dashboards into a before-and-after chart doesn't recover the evidence you're missing.

Keep the old design's relevant assets available for authorized replay. Otherwise, opening an old recording against today's styles may change the evidence you think you are reviewing.

Check who is represented, too. Backend activation counts may include workspaces whose visits were never recorded. Recordings cover the permitted, observed subset, not necessarily everyone in the outcome report. Keep private project content masked or excluded.

## Give each meaningful page state an identity

The URL stayed the same, so the comparison looks straightforward. But that address may have served an old page, a new page, an experiment variant, or personalized content. You need to know which experience produced the evidence.

Track a release or page-revision identifier alongside the route and any intentional variant. When a material change affects layout, wording, or the task, it should be possible to distinguish the visits that saw it. A deployment identifier alone may be too broad if most pages remain unchanged.

| Evidence | Keep together | Separate when it changes the interpretation |
| --- | --- | --- |
| Trial activation funnel | Same outcome and eligibility definitions | Setup flow or available templates |
| Click map | Compatible layout and element meaning | Page revision, variant, and device layout |
| Scroll behavior | Comparable page structure | Content length or section order |
| Session replay | The visitor's recorded state | Later assets or a different page version |

Do not expect changing a background screenshot to transform historical interactions into evidence about the new page. Clarity, for example, documents that changing the selected screenshot updates click and area data but does not update scroll data. Inspect the rules of the tool you use. [Clarity screenshot selection](https://learn.microsoft.com/en-us/clarity/heatmaps/heatmaps-screenshots).

## Verify the release before reading the score

Run the primary journey using synthetic information on the actual deployed page. Verify navigation, keyboard interaction, errors, accepted outcomes, and the events those actions produce. Check mobile layouts with the on-screen keyboard visible where forms are involved.

Confirm that the first saved project containing a task creates one activation per workspace. Repeated clicks, browser back navigation, and refreshing the page should not manufacture additional outcomes.

Next, confirm exposure: which version did the visitor actually see? A cached page, delayed deployment, or client-side navigation can make a release timestamp an unreliable substitute for a version recorded with the visit.

If the comparison is a randomized experiment, also check assignment and whether the observed population matches the design. Microsoft’s pre-experiment guidance describes how design and data-quality problems can undermine otherwise convincing results. [Microsoft experiment planning](https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/).

If measurement is broken, pause the interpretation. You may need to repair it and restart the comparison. Waiting for more visits won't make a broken event definition dependable.

## Separate traffic changes from page changes

The redesign goes live, the chart moves, and everyone has an explanation. Before you choose one, check what else happened between the periods. A before-and-after chart includes all those changes together.

Suppose 20 of 500 eligible trial workspaces exposed to the old welcome page activated, compared with 30 of 600 exposed to the new page. All have completed the seven-day outcome window. The observed rates are 4% and 5%, a one-percentage-point difference, or a 25% relative increase. Label both correctly.

Then you learn that a partner campaign brought in teams already familiar with the product. That's a different audience, arriving during the same period as your new page. The improvement is interesting. You can't attribute it to the redesign from these numbers alone.

Inspect meaningful groups, such as campaign source, trial plan, and device class. Show counts with rates, and avoid slicing into tiny groups until a flattering result appears. If traffic is sparse, [choose methods suited to low-traffic decisions](https://hootlens.com/learn/conversion-optimization-low-traffic/). Equivalent calendar duration alone does not make two audiences comparable.

A concurrent randomized test can provide stronger causal evidence when designed and analyzed appropriately. If that is unavailable, explain the limitations of the observational comparison and keep the decision proportionate to them.

## Connect the outcome to the behavior you changed

Review the revised journey against the original problem. Can trial owners distinguish a blank project from a template? Does the creation flow retain their choice? Do errors still interrupt the same step?

[Review recordings from the relevant versions and devices](https://hootlens.com/learn/how-to-review-session-recordings/), including visits that succeeded. If people now spend less time on the page, what happens next? They may be choosing more easily, losing interest, or encountering a failure. Follow the sequence to its outcome before deciding that faster is better.

You can also improve a metric by changing what it means. Automatically creating a project full of example tasks could increase activation without helping anyone begin real work. Check that the redesign hasn't quietly changed your outcome's meaning. Ask support staff about confusion and unwanted projects, too.

Maintain separate conclusions when the evidence differs: “The selection error is resolved in verification,” “a higher activation rate was observed,” and “the causal conversion effect remains uncertain.” Clear boundaries make a release report credible and useful.

## Write a decision, not a victory announcement

The team still needs a decision. Keep the change, revise it, roll it back, or continue observing with a clear plan. Give remaining questions an owner so “we'll keep an eye on it” doesn't become the permanent outcome.

```text
Problem: Trial owners could not distinguish project-starting options.
Change: Explain blank projects and templates beside the creation action.
Versions reviewed: welcome-v4 and welcome-v5.
Verified: The chosen option creates the intended project.
Observed outcome: Counts and rates, with audience and date range.
Guardrails: Creation errors and unwanted duplicate projects.
Limitations: Campaign mix changed during the comparison.
Decision: Keep the clarity fix; do not claim a measured causal lift.
Next check: Review activation after every included workspace's seven-day window.
```

Save the reasoning with the release. When someone proposes replacing this page six months from now, they should be able to see what the team learned, which tradeoffs it accepted, and what it never established.

## Sources

- [Microsoft Clarity: Heatmap screenshots](https://learn.microsoft.com/en-us/clarity/heatmaps/heatmaps-screenshots)
- [Microsoft Research: Pre-Experiment Stage](https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/)

## Related reading

- [How do you read a website heatmap?](https://hootlens.com/learn/how-to-read-a-website-heatmap)
- [How do you improve conversions with low traffic?](https://hootlens.com/learn/conversion-optimization-low-traffic)
- [Heatmaps, session recordings, and web analytics](https://hootlens.com/learn/heatmaps-session-recordings-web-analytics)

---

This is the Markdown copy of https://hootlens.com/learn/measure-website-redesign/