"AI-personalized copy converts better" - prove it or delete it. Here is the 40-line A/B harness

The variants

Half of the "we added AI personalization and conversions went up 40%" posts you read have no control group. The other half measured clicks, not revenue. If you are going to put LLM-generated copy in front of your paywall, you owe yourself a real experiment - and it turns out the harness is tiny if your metering and analytics live in the same place. This is the exact setup I use to test composed paywall copy against static copy, end to end. Static is the copy you have today. Composed is generated per user from their real usage via Vevee's compose() (@vevee/sdk):

• Deterministic hashing, not Math.random() - a user who sees the paywall five times sees the same variant five times. Random-per-view contaminates both arms.
• Fallbacks are relabeled, not hidden. If compose() returns opted_out (the user declined AI personalization) or the monthly compose budget is exhausted, the user gets static copy - so they must be counted as static. Logging them as "composed" dilutes your treatment arm with control copy.
• The exposure event fires exactly once per view, with the experiment name in properties, so you can run the next experiment without torching this one's data.

import { createClient } from "@vevee/sdk";
import { createHash } from "node:crypto";

const vevee = createClient({ apiKey: process.env.VEVEE_SECRET_KEY! });

interface PaywallCopy {
  headline: string;
  subheading: string;
  ctaText: string;
}

// deterministic assignment - same user, same bucket, every time
function bucket(userId: string): "composed" | "static" {
  const h = createHash("sha256").update(`paywall-exp-1:${userId}`).digest();
  return h[0] % 2 === 0 ? "composed" : "static";
}

async function getPaywall(userId: string) {
  let variant = bucket(userId);
  let copy: PaywallCopy = STATIC_COPY;

  if (variant === "composed") {
    const result = await vevee.compose<PaywallCopy>("personalized-paywall", userId);
    if (result.status === "generated") {
      copy = result.output;
    } else {
      variant = "static"; // opted-out or budget-capped → control, honestly labeled
    }
  }

  await vevee.capture({
    distinctId: userId,
    event: "paywall_shown",
    properties: { experiment: "paywall-exp-1", variant },
  });

  return { copy, variant };
}

The funnel

paywall_shown, paywall_clicked, checkout_started, and checkout_completed are reserved event names in the SDK's analytics taxonomy - the dashboard recognizes them and offers the funnel preset. The remaining captures look like this. Build the funnel twice, filtered by variant. The number you report is shown → checkout_completed, per variant. Clicks are a sanity check, not a result - composed copy that doubles clicks and does not move completed checkouts is copy that writes better buttons, not better arguments.

await vevee.capture({ distinctId: userId, event: "paywall_clicked",
  properties: { experiment: "paywall-exp-1", variant } });

await vevee.capture({ distinctId: userId, event: "checkout_started",
  properties: { experiment: "paywall-exp-1", variant } });

await vevee.capture({ distinctId: userId, event: "checkout_completed",
  properties: { experiment: "paywall-exp-1", variant } });

Reading it honestly

The harness is the easy part; the discipline is in how you read it. Three rules before you peek at the dashboard:

• Pre-register the metric (shown → completed) and a minimum sample before you peek. Paywall traffic is lower than you think; most "AI lifted conversions 40%" results are noise on n=80.
• Segment by usage tier after the topline. My hypothesis going in: composed copy wins big for heavy users (it has material to work with) and does nothing for day-one users (no usage data → generic output anyway). If that is your pattern, the production move is composed for active users, static for new ones - better than either arm alone.
• Cost the win. Each composed result reports usage.costMicroUsd. Lift per dollar is the actual decision number: total compose spend ÷ incremental conversions. When the per-conversion cost is a rounding error against LTV, ship it; the dashboard's monthly compose budget keeps the downside capped while you decide.