A/B Testing

An A/B test serves different ad configurations during the same period and compares their performance. In A2, you can divide an AdUnit’s users into buckets and configure each bucket’s line items and selection method. A bucket is a group containing a configuration to compare. For example, assign different line items to two buckets with a user proportion of 1:1, then compare CTR or cost per conversion.

This page covers experiments comparing line item configurations within A2. Before splitting requests between your existing system and A2, separately define routing and how results from both systems will be compared.

Before you begin

API access: Prepare the API address and access token. Reading experiments and metrics requires experiment read permission; creating, updating, and deleting requires write permission.
Target and line items: Work with your operator to prepare the AdUnit and line items to compare, and obtain their IDs. Confirm that the line items can serve on that AdUnit. See Core concepts for terminology and relationships.
User IDs: Consistently send the same user’s ID in ad requests. See the Ad Request API for request fields.
Data delivery: Check the Product Catalog, User Activity Logs, and Ad Event Tracking.

For configuration in Ad Manager, see A/B experiment operations.

Define the comparison

Agree on these points with your operator before creating an experiment.

Item	What to define
Configurations	AdUnit, line items, and selection method for each bucket.
Primary metric	The metric used for evaluation, such as CTR, CVR, or CPA.
Conversion	The goal, such as a purchase or signup, when it completes, and which ad receives credit.
Cost and sales	How costs are recorded and the units used for sales.
Period and proportions	Start and end times, timezone, and relative user proportions.

To compare CTR, record impressions and clicks under the same conditions in both buckets. To compare CPA, use the same conversion definition and confirm that costs are recorded. To compare sales, also send conversion_data.sales with conversions. The primary metric is an evaluation criterion. Do not add a separate KPI field to the experiment creation request.

Data needed for each metric

Metric	Data to verify first
CTR `ctr`	Recorded `imp` and `clk` with consistent impression tracking conditions.
CVR `cvr`	Recorded `clk` and `conv` under the same definitions. A2 calculates CVR as conversions ÷ clicks × 100.
CPA `cpa`	Recorded conversions and ad cost. Cost calculations use bucket metric `pub_revenue`.
Sales `sales`	`conversion_data.sales` sent in consistent units with conversions assigned to experiment buckets.

Compare the same AdUnit and period, keeping other conditions stable. Record configuration change times so results before and after a change can be distinguished.

User assignment

A2 determines the bucket using the experiment ID and the ad request’s user ID. With the same bucket configuration, the same user is assigned to the same bucket. Do not implement per-request random bucket selection.

Keep the same user’s ID consistent across requests.
Requests without a user ID do not receive a bucket assignment. Experiment line items can participate in the regular ad auction and serve, but these requests are not included in experiment bucket metrics.
Changing bucket proportions or configuration during an experiment can change assignments. Record what changed and when.
line_item_rotate: "random" controls line item selection within an assigned bucket. It is separate from user bucket assignment.

proportion is a relative weight. 1:1 assigns equal proportions of users to two buckets; it does not guarantee equal impression or click counts.

Configuration fields

Experiment

Field	Required	Description
`name` string	Required	Experiment name.
`target_type` string	Required	Use `ad_unit` for the AdUnit experiments in this guide.
`target_id` UUID string	Required	ID of the AdUnit to compare.
`start_date` date-time string	Required	Start time with a timezone, such as `2026-11-06T00:00:00+09:00`.
`end_date` date-time string	Required	End time, later than the start time.
`buckets` object array	Required	Configure 1–15 buckets. Use two to compare two configurations.
`status` string	Optional	`inactive`, `active`, or `completed`. New experiments are created as `inactive`. Change to `active` through the update API to start.
`ext` JSON value	Optional	Additional information stored with the experiment.

Bucket `buckets[]`

Field	Required	Description
`name` string	Required	Bucket name.
`proportion` nonnegative integer	Required	Relative user assignment weight. Use a positive value for participating buckets.
`line_items` object array	Required	Up to 10 line items to compare in the bucket. Include at least one line item to serve.
`line_item_rotate` string	Optional	`sequential`, `weighted`, `random`, or `score_aware`. Defaults to `random`.
`id` integer	Required for existing buckets on update	Omit on creation. When updating an existing bucket, retain the ID returned by A2.
`is_deleted` boolean	Optional	Send `true` to delete an existing bucket. Defaults to `false`.

Include these values for each line item.

Field	Required	Description
`line_items[].lid` UUID string	Required	Line item ID.
`line_items[].weight` nonnegative integer	Required	Relative weight for line item selection. The example uses `1` for each item.

For advanced settings such as bucket gating, gating_policy, and line item gating, see the Create Experiment API schema. Keep conditions other than those being compared consistent.

Create an experiment through the API

Prepare the request body

Save the following as experiment-create.json. Replace the AdUnit ID, two line item IDs, name, and period with your values. Omit the experiment ID and bucket IDs when creating.

Create the experiment

Call the creation API. Retain the experiment id and buckets[].id from the HTTP 201 response. The new experiment is inactive.

Review the configuration

Use the read API to verify the target, period, line items, and proportions before starting.

experiment-create.json

{
  "name": "Ad configuration comparison",
  "target_type": "ad_unit",
  "target_id": "11111111-1111-4111-8111-111111111111",
  "start_date": "2026-11-06T00:00:00+09:00",
  "end_date": "2026-11-13T00:00:00+09:00",
  "buckets": [
    {
      "name": "control",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "22222222-2222-4222-8222-222222222222", "weight": 1 }
      ]
    },
    {
      "name": "variant",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "33333333-3333-4333-8333-333333333333", "weight": 1 }
      ]
    }
  ]
}

curl -i -X POST "<A2_API_URL>/api/manager/experiments" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-create.json

Replace <experiment_id> with the experiment ID from the response.

curl "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>"

API Reference: Create · Get · List

Start and complete an experiment

States follow inactive → active → completed. An experiment applies only when it is active and the current time is within its configured period. Only one active experiment can exist for the same target: the combination of target_type and target_id. An AdUnit experiment and an experiment on its parent Placement can be active simultaneously because their targets differ. If multiple experiments apply to the same request, A2 applies only the one with the smallest experiment UUID. Check for overlapping parent Placement experiments before comparing traffic. The update API accepts the full configuration. Include the name, target, period, and all buckets along with status. The following starts the previous experiment if A2 returned bucket IDs 1 and 2. Replace these with the IDs and settings from your response.

experiment-update.json

{
  "name": "Ad configuration comparison",
  "target_type": "ad_unit",
  "target_id": "11111111-1111-4111-8111-111111111111",
  "start_date": "2026-11-06T00:00:00+09:00",
  "end_date": "2026-11-13T00:00:00+09:00",
  "status": "active",
  "buckets": [
    {
      "id": 1,
      "name": "control",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "22222222-2222-4222-8222-222222222222", "weight": 1 }
      ]
    },
    {
      "id": 2,
      "name": "variant",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "33333333-3333-4333-8333-333333333333", "weight": 1 }
      ]
    }
  ]
}

curl -i -X PUT "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-update.json

Complete: Read the latest configuration, change status to completed, and send the full configuration. Once the end time passes, the experiment no longer applies to traffic, but its status does not automatically change to completed.
Restart: A completed experiment cannot restart. Create a new experiment for a new comparison.
Update buckets: Keep existing bucket IDs. To delete a bucket, retain it in the request with is_deleted: true. Omit the ID for a new bucket. At most 15 bucket IDs can be issued over an experiment’s lifetime, including deleted IDs.
Delete experiments: Only inactive experiments can be deleted. The deletion API returns HTTP 204 on success.

curl -i -X DELETE "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>"

API Reference: Update · Delete

Verify experiment application

After saving the configuration, inspect assignment information in actual service responses that return an ad. Consistently include the same user’s ID. If no ad is returned, check the no-ad settings in Ad Serving (Advanced) first.

Response field	What to check
`ext.ab_test.experiment_id`	Matches the intended experiment’s `id`.
`ext.ab_test.bucket_id`	Matches an ID in the experiment’s `buckets[].id`.

Verify stable assignment for the same user, experiment, and bucket configuration. See the Ad Request API for the response schema. If assignment information is missing, check the user ID, status, period, and target. If a different experiment ID appears, check overlapping AdUnit and parent Placement experiments. These fields confirm the assigned experiment and bucket. They do not prove the selected ad or final impression, click, and conversion aggregates; also compare the bucket metrics below. Additional verification requests record impressions under on_ad_response, so use actual service request records.

Query results

Use the Experiment Bucket Metrics API to compare buckets over the same period. Save the following as experiment-metrics.json and replace the experiment ID and period.

experiment-metrics.json

{
  "experiment_id": "44444444-4444-4444-8444-444444444444",
  "from_datetime": "2026-11-06T00:00:00+09:00",
  "to_datetime": "2026-11-13T00:00:00+09:00",
  "interval": "daily",
  "timezone": "Asia/Seoul"
}

curl -X POST "<A2_API_URL>/api/manager/metric/experiment" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-metrics.json

Field	Required	Description
`experiment_id` UUID string	Required	Experiment to query.
`from_datetime` date-time string	Required	Inclusive start time with a timezone.
`to_datetime` date-time string	Required	Exclusive end time, later than the start time.
`interval` string	Optional	`hourly`, `daily`, `weekly`, or `monthly`. Defaults to `daily`. Maximum query spans are 7, 60, 90, and 90 days respectively.
`timezone` string	Optional	IANA timezone for grouping time intervals. Defaults to `Asia/Seoul`.

In the response, total contains each bucket’s totals for the query period. timeseries contains metrics by time interval and line item breakdowns. Match bucket_id to the bucket IDs in the experiment configuration.

Metric	Meaning
`impression`, `click`, `conversion`	Impression, click, and conversion counts.
`sales`	Sum of sales sent with conversions.
`pub_revenue`	Ad cost recorded by A2, used as the basis for the cost metrics below.
`ctr`	Clicks ÷ impressions × 100, expressed as a percentage.
`cvr`	Conversions ÷ clicks × 100, expressed as a percentage.
`cpa`	`pub_revenue` ÷ conversions.
`cpc`, `cpm`	`pub_revenue` ÷ clicks, and `pub_revenue` ÷ impressions × 1,000 respectively.
`cps`	`pub_revenue` ÷ `sales`.

Derived metrics return 0 when the denominator is 0. Do not interpret a CPA of 0 as good performance when conversions or costs have not been collected. Verify cost recording and conversion delivery before comparing CPA.

Conversions can occur later than impressions and clicks, so check conversion occurrence times alongside the query period. Do not interpret bucket metrics as lifetime outcomes for every user exposed during the experiment. If conversions occurring outside the experiment period must be measured, separately verify whether they are included in the original experiment metrics. Compare the primary metric chosen before the experiment alongside impressions, clicks, conversions, and bucket proportions for the same period. A2 does not calculate statistical significance or automatically choose a winning bucket. Use the results to decide the next configuration and rollout scope.

If the experiment is not applied

Confirm the status is active and the current time is within the period.
Match the requested AdUnit ID to the experiment’s target_id.
Confirm that ad requests include a user ID. Without one, experiment line items can serve through the regular auction, but the requests are not included in bucket metrics.
Check for overlapping experiments on the AdUnit and its parent Placement. Only the applicable experiment with the smallest UUID is applied.
Check bucket proportions, line items, and serving conditions for the AdUnit.
If metrics are empty, check the query period, timezone, and Ad Event Tracking. A successful configuration lookup does not prove the experiment was applied to traffic or that metrics were collected.

Also check data delivery in Verification and Troubleshooting.

Getting Started

Ad Integration

Recommendation Integration

Data Integration

Verification and Experiments

Before you begin

Define the comparison

Data needed for each metric

User assignment

Configuration fields

Experiment

Bucket `buckets[]`

Create an experiment through the API

Start and complete an experiment

Verify experiment application

Query results

If the experiment is not applied

​Before you begin

​Define the comparison

​Data needed for each metric

​User assignment

​Configuration fields

​Experiment

​Bucket buckets[]

​Create an experiment through the API

​Start and complete an experiment

​Verify experiment application

​Query results

​If the experiment is not applied

Before you begin

Define the comparison

Data needed for each metric

User assignment

Configuration fields

Experiment

Bucket `buckets[]`

Create an experiment through the API

Start and complete an experiment

Verify experiment application

Query results

If the experiment is not applied