Skip to main content

A/B Testing Analytics with Vizly

After an experiment runs, you need to see whether the new variant actually wins. Vizly covers the standard A/B analysis visuals: variant comparison bars, conversion distributions with boxplots, and funnel steps.

Variant comparison​

Compare the headline metric across control and treatment variants. A bar chart with the win rate per variant is the simplest view.

import vizly as vz

vz.set_theme("corporate")

variants = [
    {"variant": "Control", "conversion": 3.2},
    {"variant": "Variant A", "conversion": 4.1},
    {"variant": "Variant B", "conversion": 3.6},
]

chart = vz.bar(variants, x="variant", y="conversion", title="Conversion rate %")
chart.render("ab_test.html")

Conversion distributions​

Boxplots show how individual users or sessions converted across variants, revealing spread and outliers, not just averages.

import pandas as pd
import vizly as vz

# One row per session with the variant label and conversion value
sessions = [
    {"variant": "Control", "conversion": 0.0},
    {"variant": "Control", "conversion": 1.2},
    {"variant": "Variant A", "conversion": 2.1},
    {"variant": "Variant A", "conversion": 3.4},
    {"variant": "Variant A", "conversion": 1.8},
]
df = pd.DataFrame(sessions)

chart = vz.boxplot(df, x="variant", y="conversion", title="Conversion by variant")

Funnel across variants​

Compare funnel conversion at each step for control versus treatment.

funnel = [
    {"stage": "Visits", "control": 1000, "variant_a": 1020},
    {"stage": "Signups", "control": 210, "variant_a": 260},
    {"stage": "Paid", "control": 61, "variant_a": 88},
]

chart = vz.line(funnel, x="stage", y=["control", "variant_a"], title="Funnel: control vs variant A")

Lift and confidence​

Plot the observed lift per segment to see where the variant wins and where it ties.

segments = [
    {"segment": "New users", "lift": 4.2},
    {"segment": "Returning", "lift": 1.1},
    {"segment": "Mobile", "lift": 6.3},
    {"segment": "Desktop", "lift": -0.4},
    {"segment": "Free tier", "lift": 2.8},
]

chart = vz.bar(segments, x="segment", y="lift", title="Lift % by segment")

Scatter for correlation​

Check whether a secondary metric moved with the primary metric, using a scatter plot.

df_scatter = pd.DataFrame({
    "conversion": [3.0, 3.4, 3.8, 4.2, 4.6, 5.0],
    "engagement": [12, 14, 15, 17, 19, 22],
})

chart = vz.scatter(df_scatter, x="conversion", y="engagement", title="Conversion vs engagement")

Full experiment dashboard​

Compose the experiment results into a single review page.

import pandas as pd
import vizly as vz

vz.set_theme("corporate")

chart = vz.page(
    charts=[
        vz.bar(variants, x="variant", y="conversion", title="Conversion rate %"),
        vz.boxplot(df, x="variant", y="conversion", title="Conversion by variant"),
        vz.line(funnel, x="stage", y=["control", "variant_a"], title="Funnel comparison"),
        vz.bar(segments, x="segment", y="lift", title="Lift % by segment"),
    ],
    title="Experiment dashboard",
)
chart.render("experiment.html")

Tips​

TipDetail
Boxplots need raw datavz.boxplot wants one row per observation, not pre-aggregated stats.
Lift charts are barsPer-segment lift reads best as a bar chart, with negative values visible below zero.
Funnels are lines hereWhen comparing two variants across stages, a multi-series line shows both at once.
Pair with SQLLoad experiment data straight from a data warehouse with from_sql. See the SQL recipe.