> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aiderx.io/llms.txt
> Use this file to discover all available pages before exploring further.

# A/B Testing

> Assign users to ad configurations and compare impressions, clicks, conversions, and cost metrics by bucket.

An A/B test serves different ad configurations during the same period and compares their performance. In A2, you can divide an AdUnit's users into buckets and configure each bucket's line items and selection method.

A bucket is a group containing a configuration to compare. For example, assign different line items to two buckets with a user proportion of `1:1`, then compare CTR or cost per conversion.

> [!NOTE]
>
> This page covers experiments comparing line item configurations within A2. Before splitting requests between your existing system and A2, separately define routing and how results from both systems will be compared.

## Before you begin

* **API access:** Prepare the [API address and access token](https://docs.aiderx.io/development/en/api-access-and-auth.md). Reading experiments and metrics requires experiment read permission; creating, updating, and deleting requires write permission.
* **Target and line items:** Work with your operator to prepare the AdUnit and line items to compare, and obtain their IDs. Confirm that the line items can serve on that AdUnit. See [Core concepts](https://docs.aiderx.io/guides/platform-overview/en/core-concepts.md) for terminology and relationships.
* **User IDs:** Consistently send the same user's ID in ad requests. See the [Ad Request API](https://docs.aiderx.io/en/0.5.0/decision/ad-serving/serving-an-ad.md) for request fields.
* **Data delivery:** Check the [Product Catalog](https://docs.aiderx.io/development/en/catalog.md), [User Activity Logs](https://docs.aiderx.io/development/en/user-events.md), and [Ad Event Tracking](https://docs.aiderx.io/development/en/ad-events-and-logs.md).

For configuration in Ad Manager, see [A/B experiment operations](https://docs.aiderx.io/guides/about-a2/en/experiments.md).

## Define the comparison

Agree on these points with your operator before creating an experiment.

| Item | What to define |
| - | - |
| Configurations | AdUnit, line items, and selection method for each bucket. |
| Primary metric | The metric used for evaluation, such as CTR, CVR, or CPA. |
| Conversion | The goal, such as a purchase or signup, when it completes, and which ad receives credit. |
| Cost and sales | How costs are recorded and the units used for sales. |
| Period and proportions | Start and end times, timezone, and relative user proportions. |

To compare CTR, record impressions and clicks under the same conditions in both buckets. To compare CPA, use the same conversion definition and confirm that costs are recorded. To compare sales, also send `conversion_data.sales` with conversions.

The primary metric is an evaluation criterion. Do not add a separate KPI field to the experiment creation request.

### Data needed for each metric

| Metric | Data to verify first |
| - | - |
| CTR `ctr` | Recorded `imp` and `clk` with consistent impression tracking conditions. |
| CVR `cvr` | Recorded `clk` and `conv` under the same definitions. A2 calculates CVR as conversions ÷ clicks × 100. |
| CPA `cpa` | Recorded conversions and ad cost. Cost calculations use bucket metric `pub_revenue`. |
| Sales `sales` | `conversion_data.sales` sent in consistent units with conversions assigned to experiment buckets. |

Compare the same AdUnit and period, keeping other conditions stable. Record configuration change times so results before and after a change can be distinguished.

## User assignment

A2 determines the bucket using the experiment ID and the ad request's user ID. With the same bucket configuration, the same user is assigned to the same bucket. Do not implement per-request random bucket selection.

* Keep the same user's ID consistent across requests.
* Requests without a user ID do not receive a bucket assignment. Experiment line items can participate in the regular ad auction and serve, but these requests are not included in experiment bucket metrics.
* Changing bucket proportions or configuration during an experiment can change assignments. Record what changed and when.
* `line_item_rotate: "random"` controls line item selection within an assigned bucket. It is separate from user bucket assignment.

`proportion` is a relative weight. `1:1` assigns equal proportions of users to two buckets; it does not guarantee equal impression or click counts.

## Configuration fields

### Experiment

| Field | Required | Description |
| - | - | - |
| `name` string | Required | Experiment name. |
| `target_type` string | Required | Use `ad_unit` for the AdUnit experiments in this guide. |
| `target_id` UUID string | Required | ID of the AdUnit to compare. |
| `start_date` date-time string | Required | Start time with a timezone, such as `2026-11-06T00:00:00+09:00`. |
| `end_date` date-time string | Required | End time, later than the start time. |
| `buckets` object array | Required | Configure 1–15 buckets. Use two to compare two configurations. |
| `status` string | Optional | `inactive`, `active`, or `completed`. New experiments are created as `inactive`. Change to `active` through the update API to start. |
| `ext` JSON value | Optional | Additional information stored with the experiment. |

### Bucket `buckets[]`

| Field | Required | Description |
| - | - | - |
| `name` string | Required | Bucket name. |
| `proportion` nonnegative integer | Required | Relative user assignment weight. Use a positive value for participating buckets. |
| `line_items` object array | Required | Up to 10 line items to compare in the bucket. Include at least one line item to serve. |
| `line_item_rotate` string | Optional | `sequential`, `weighted`, `random`, or `score_aware`. Defaults to `random`. |
| `id` integer | Required for existing buckets on update | Omit on creation. When updating an existing bucket, retain the ID returned by A2. |
| `is_deleted` boolean | Optional | Send `true` to delete an existing bucket. Defaults to `false`. |

Include these values for each line item.

| Field | Required | Description |
| - | - | - |
| `line_items[].lid` UUID string | Required | Line item ID. |
| `line_items[].weight` nonnegative integer | Required | Relative weight for line item selection. The example uses `1` for each item. |

For advanced settings such as bucket `gating`, `gating_policy`, and line item `gating`, see the [Create Experiment API](</en/0.5.0/data management/experiment/create-an-experiment>) schema. Keep conditions other than those being compared consistent.

## Create an experiment through the API

**1. Prepare the request body**

Save the following as `experiment-create.json`. Replace the AdUnit ID, two line item IDs, name, and period with your values. Omit the experiment ID and bucket IDs when creating.

**2. Create the experiment**

Call the creation API. Retain the experiment `id` and `buckets[].id` from the HTTP `201` response. The new experiment is `inactive`.

**3. Review the configuration**

Use the read API to verify the target, period, line items, and proportions before starting.

```json experiment-create.json theme={null}
{
  "name": "Ad configuration comparison",
  "target_type": "ad_unit",
  "target_id": "11111111-1111-4111-8111-111111111111",
  "start_date": "2026-11-06T00:00:00+09:00",
  "end_date": "2026-11-13T00:00:00+09:00",
  "buckets": [
    {
      "name": "control",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "22222222-2222-4222-8222-222222222222", "weight": 1 }
      ]
    },
    {
      "name": "variant",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "33333333-3333-4333-8333-333333333333", "weight": 1 }
      ]
    }
  ]
}
```

```bash theme={null}
curl -i -X POST "<A2_API_URL>/api/manager/experiments" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-create.json
```

Replace `<experiment_id>` with the experiment ID from the response.

```bash theme={null}
curl "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>"
```

API Reference: [Create](</en/0.5.0/data management/experiment/create-an-experiment>) · [Get](</en/0.5.0/data management/experiment/get-an-experiment>) · [List](</en/0.5.0/data management/experiment/list-experiments>)

## Start and complete an experiment

States follow `inactive` → `active` → `completed`. An experiment applies only when it is `active` and the current time is within its configured period. Only one `active` experiment can exist for the same target: the combination of `target_type` and `target_id`.

An AdUnit experiment and an experiment on its parent Placement can be active simultaneously because their targets differ. If multiple experiments apply to the same request, A2 applies only the one with the smallest experiment UUID. Check for overlapping parent Placement experiments before comparing traffic.

The update API accepts the full configuration. Include the name, target, period, and all buckets along with `status`. The following starts the previous experiment if A2 returned bucket IDs `1` and `2`. Replace these with the IDs and settings from your response.

```json experiment-update.json theme={null}
{
  "name": "Ad configuration comparison",
  "target_type": "ad_unit",
  "target_id": "11111111-1111-4111-8111-111111111111",
  "start_date": "2026-11-06T00:00:00+09:00",
  "end_date": "2026-11-13T00:00:00+09:00",
  "status": "active",
  "buckets": [
    {
      "id": 1,
      "name": "control",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "22222222-2222-4222-8222-222222222222", "weight": 1 }
      ]
    },
    {
      "id": 2,
      "name": "variant",
      "proportion": 1,
      "line_item_rotate": "random",
      "line_items": [
        { "lid": "33333333-3333-4333-8333-333333333333", "weight": 1 }
      ]
    }
  ]
}
```

```bash theme={null}
curl -i -X PUT "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-update.json
```

* **Complete:** Read the latest configuration, change `status` to `completed`, and send the full configuration. Once the end time passes, the experiment no longer applies to traffic, but its status does not automatically change to `completed`.
* **Restart:** A completed experiment cannot restart. Create a new experiment for a new comparison.
* **Update buckets:** Keep existing bucket IDs. To delete a bucket, retain it in the request with `is_deleted: true`. Omit the ID for a new bucket. At most 15 bucket IDs can be issued over an experiment's lifetime, including deleted IDs.
* **Delete experiments:** Only `inactive` experiments can be deleted. The deletion API returns HTTP `204` on success.

```bash theme={null}
curl -i -X DELETE "<A2_API_URL>/api/manager/experiments/<experiment_id>" \
  -H "Authorization: Bearer <token>"
```

API Reference: [Update](</en/0.5.0/data management/experiment/update-an-experiment>) · [Delete](</en/0.5.0/data management/experiment/delete-an-experiment>)

## Verify experiment application

After saving the configuration, inspect assignment information in actual service responses that return an ad. Consistently include the same user's ID. If no ad is returned, check the no-ad settings in [Ad Serving (Advanced)](https://docs.aiderx.io/development/en/serving-ad.md#no-ad-checklist) first.

| Response field | What to check |
| - | - |
| `ext.ab_test.experiment_id` | Matches the intended experiment's `id`. |
| `ext.ab_test.bucket_id` | Matches an ID in the experiment's `buckets[].id`. |

Verify stable assignment for the same user, experiment, and bucket configuration. See the [Ad Request API](https://docs.aiderx.io/en/0.5.0/decision/ad-serving/serving-an-ad.md) for the response schema. If assignment information is missing, check the user ID, status, period, and target. If a different experiment ID appears, check overlapping AdUnit and parent Placement experiments.

These fields confirm the assigned experiment and bucket. They do not prove the selected ad or final impression, click, and conversion aggregates; also compare the bucket metrics below. Additional verification requests record impressions under `on_ad_response`, so use actual service request records.

## Query results

Use the [Experiment Bucket Metrics API](</en/0.5.0/data management/metric/get-experiment-bucket-metrics>) to compare buckets over the same period. Save the following as `experiment-metrics.json` and replace the experiment ID and period.

```json experiment-metrics.json theme={null}
{
  "experiment_id": "44444444-4444-4444-8444-444444444444",
  "from_datetime": "2026-11-06T00:00:00+09:00",
  "to_datetime": "2026-11-13T00:00:00+09:00",
  "interval": "daily",
  "timezone": "Asia/Seoul"
}
```

```bash theme={null}
curl -X POST "<A2_API_URL>/api/manager/metric/experiment" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @experiment-metrics.json
```

| Field | Required | Description |
| - | - | - |
| `experiment_id` UUID string | Required | Experiment to query. |
| `from_datetime` date-time string | Required | Inclusive start time with a timezone. |
| `to_datetime` date-time string | Required | Exclusive end time, later than the start time. |
| `interval` string | Optional | `hourly`, `daily`, `weekly`, or `monthly`. Defaults to `daily`. Maximum query spans are 7, 60, 90, and 90 days respectively. |
| `timezone` string | Optional | IANA timezone for grouping time intervals. Defaults to `Asia/Seoul`. |

In the response, `total` contains each bucket's totals for the query period. `timeseries` contains metrics by time interval and line item breakdowns. Match `bucket_id` to the bucket IDs in the experiment configuration.

| Metric | Meaning |
| - | - |
| `impression`, `click`, `conversion` | Impression, click, and conversion counts. |
| `sales` | Sum of sales sent with conversions. |
| `pub_revenue` | Ad cost recorded by A2, used as the basis for the cost metrics below. |
| `ctr` | Clicks ÷ impressions × 100, expressed as a percentage. |
| `cvr` | Conversions ÷ clicks × 100, expressed as a percentage. |
| `cpa` | `pub_revenue` ÷ conversions. |
| `cpc`, `cpm` | `pub_revenue` ÷ clicks, and `pub_revenue` ÷ impressions × 1,000 respectively. |
| `cps` | `pub_revenue` ÷ `sales`. |

> [!WARNING]
>
> Derived metrics return 0 when the denominator is 0. Do not interpret a CPA of `0` as good performance when conversions or costs have not been collected. Verify cost recording and conversion delivery before comparing CPA.

Conversions can occur later than impressions and clicks, so check conversion occurrence times alongside the query period. Do not interpret bucket metrics as lifetime outcomes for every user exposed during the experiment. If conversions occurring outside the experiment period must be measured, separately verify whether they are included in the original experiment metrics.

Compare the primary metric chosen before the experiment alongside impressions, clicks, conversions, and bucket proportions for the same period. A2 does not calculate statistical significance or automatically choose a winning bucket. Use the results to decide the next configuration and rollout scope.

## If the experiment is not applied

* Confirm the status is `active` and the current time is within the period.
* Match the requested AdUnit ID to the experiment's `target_id`.
* Confirm that ad requests include a user ID. Without one, experiment line items can serve through the regular auction, but the requests are not included in bucket metrics.
* Check for overlapping experiments on the AdUnit and its parent Placement. Only the applicable experiment with the smallest UUID is applied.
* Check bucket proportions, line items, and serving conditions for the AdUnit.
* If metrics are empty, check the query period, timezone, and [Ad Event Tracking](https://docs.aiderx.io/development/en/ad-events-and-logs.md). A successful configuration lookup does not prove the experiment was applied to traffic or that metrics were collected.

Also check data delivery in [Verification and Troubleshooting](https://docs.aiderx.io/development/en/integration-verification-and-troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.