# Data Contract Evolution

Release a breaking change as a new major version and migrate.

Source: https://learn.datacontract.com/en/evolution/

The business wants to know *how many* units of an item are bought per order.
The orders team adds a column `quantity` to `line_items`, defaulting to 1.
Harmless? Not for a pipeline that expects exactly three columns, e.g. a `SELECT *` into a fixed target table.
So you release the change as a **new major version**: `orders_v2`, in its own database schema, next to `orders_v1`.

_Scenario: the Orders data product (output ports orders_v1 and orders_v2) is consumed by the SKU Sales data product, which the purchasing team uses._

**You will learn:**
- Release a change as a new major version of a data contract
- Compare two versions with `datacontract changelog` and `datacontract breaking`
- Understand what counts as a breaking change, and where tools reach their limits
- Announce removals early with the `deprecated` flag (new in ODCS 3.2)
- Walk through the contract lifecycle: draft → active → deprecated → retired

> **Breaking vs. non-breaking changes**
>
> A change is **breaking** when a consumer that worked yesterday can fail today without changing anything. Typical breaking changes:
>
> - **removing or renaming** a field or table
> - **changing a type**, e.g. `BIGINT` → `TEXT`
> - **weakening a guarantee**, e.g. a `required` field becomes optional, or a quality rule is relaxed
>
> Adding an optional field, improving descriptions, or adding quality checks are typically **non-breaking**.
>
> Contracts use **semantic versioning**: breaking changes bump the **major** version (`1.x` → `2.0.0`), compatible additions the minor version, fixes the patch version.

## Look at the new data

### Explore the orders_v2 schema

The `orders_v2` schema holds both tables: `orders` is unchanged, `line_items` has the new `quantity` column.

macOS / Linux:

```bash
docker compose exec postgres psql -U workshop -d workshop -c '\dt orders_v2.*' -c 'SELECT * FROM orders_v2.line_items LIMIT 5;'
```

Windows (PowerShell):

```powershell
docker compose exec postgres psql -U workshop -d workshop -c '\dt orders_v2.*' -c 'SELECT * FROM orders_v2.line_items LIMIT 5;'
```

Output:

```text
             List of relations
  Schema   |    Name    | Type  |  Owner
-----------+------------+-------+----------
 orders_v2 | line_items | table | workshop
 orders_v2 | orders     | table | workshop
(2 rows)

            lines_item_id             |               order_id               |      sku      | quantity
--------------------------------------+--------------------------------------+---------------+----------
 94aa82c8-50ba-47fb-994a-9b041b4127af | a8c38fec-2acd-4b55-883b-4b48572d4a26 | D3KT74L5EV46T |        1
 d67c963f-42a4-4aa8-afff-d7869008e3a9 | 9e44da97-4f72-4bcf-821a-9d9500d06651 | E202K62FT     |        3
 270ad2c1-f651-438e-a81a-d77713c1d3a3 | 8fc4621c-66ae-4031-91f1-5313beb9f541 | 1O7RID9Y5QJ   |        1
 cc763a72-cc07-4bc4-8ddf-c88d09db5daa | 98d48daf-3532-4a59-b7c2-3777164bdc65 | 7KJ8466FI39LW |        5
 d7ea7f72-a266-469c-a9ef-60063d5ac243 | 2fd9df43-77e8-4d00-b380-ab270e8b73f8 | 7HXBABF0AOT5  |        2
(5 rows)
```

## Create v2

### Copy the contract

Copy your v1 contract and open the copy in the editor:

macOS / Linux:

```bash
cp orders_v1.odcs.yaml orders_v2.odcs.yaml
datacontract edit orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
Copy-Item orders_v1.odcs.yaml orders_v2.odcs.yaml
datacontract edit orders_v2.odcs.yaml
```

### Bump the version

In **Fundamentals**, set the new major version:

| Field | Value |
|---|---|
| ID | `orders_v2` |
| Version | `2.0.0` |
| Status | `draft` |

The ID changes with the major version: consumers switch to `orders_v2` deliberately, and `orders_v1` stays available until they have migrated.

### Point the server to orders_v2

In **Servers**, change the **Schema** to `orders_v2`.

Then replace `orders_v1.` with `orders_v2.` in all SQL **quality checks** (e.g. the `customer_email_address` check). Otherwise they keep testing the old tables.

The copy also kept your AI `context` block. Its `verifiedStatements` contain SQL answers that query `orders_v1`, so switch them to `orders_v2` as well. An AI agent would otherwise answer questions from the old version.

> **Tip**
>
> A search-and-replace `orders_v1.` → `orders_v2.` in your IDE does it in one go, for quality checks and verified statements. Then check `id:` and the server's `schema:`.

### Add the quantity column

In **Schemas → `line_items`**, add the new property:

| Property | Logical Type | Physical Type |
|---|---|---|
| `quantity` | `integer` | `BIGINT` |

Add a quality check that `quantity` is always greater than 0. Use the "count the bad rows, expect zero" pattern.

**Solution: quantity property**

```yaml
- name: quantity
  physicalType: bigint
  logicalType: integer
  quality:
    - type: sql
      description: Ensure quantity is positive
      query: SELECT COUNT(*) FROM orders_v2.line_items WHERE quantity <= 0;
      mustBe: 0
```

![line_items in orders_v2 with the new quantity property](https://learn.datacontract.com/screenshots/editor-quantity.webp)

### Test v2

Save and run the tests:

macOS / Linux:

```bash
datacontract test orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
datacontract test orders_v2.odcs.yaml
```

Output:

```text
Testing orders_v2.odcs.yaml
Server: Orders (type=postgres, host=localhost, port=5433, database=workshop, schema=orders_v2)
╭────────┬───────────────────────────────────────────────┬───────────────────────────────┬─────────╮
│ Result │ Check                                         │ Field                         │ Details │
├────────┼───────────────────────────────────────────────┼───────────────────────────────┼─────────┤
│ passed │ Ensure line_items table has data              │ line_items                    │         │
…
│ passed │ Check that field 'quantity' is present        │ line_items.quantity           │         │
│ passed │ Check that field quantity has physical type   │ line_items.quantity           │         │
│        │ bigint                                        │                               │         │
│ passed │ Ensure quantity is positive                   │ line_items.quantity           │         │
…
╰────────┴───────────────────────────────────────────────┴───────────────────────────────┴─────────╯
🟢 data contract is valid. Run 37 checks. Took 0.63683 seconds.
```

All checks should pass, including the new `quantity` check.

## Compare the versions

Before a release, you want to know what changed and whether it breaks consumers. The CLI compares two contract files.

### Show the changelog

macOS / Linux:

```bash
datacontract changelog orders_v1.odcs.yaml orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
datacontract changelog orders_v1.odcs.yaml orders_v2.odcs.yaml
```

Output:

```text
Summary
[ 1 Added ]  [ 16 Updated ]
╭─────────┬─────────────────────────────────────────────────────────────────╮
│ Change  │ Field                                                           │
├─────────┼─────────────────────────────────────────────────────────────────┤
│ Updated │ context.verifiedStatements.How many orders were placed in 2023? │
│ Updated │ context.verifiedStatements.What was the revenue per year?       │
│ Updated │ id                                                              │
│ Updated │ schema.line_items.properties.lines_item_id.quality.[1]          │
│ Added   │ schema.line_items.properties.quantity                           │
…
│ Updated │ servers.Orders                                                  │
│ Updated │ version                                                         │
╰─────────┴─────────────────────────────────────────────────────────────────╯

Details
…
```

You get a summary and a detailed list of all changes, including the switched quality queries and verified statements. A good base for release notes.

### Check for breaking changes

`breaking` classifies each change by severity:

macOS / Linux:

```bash
datacontract breaking orders_v1.odcs.yaml orders_v2.odcs.yaml
echo "exit code: $?"
```

Windows (PowerShell):

```powershell
datacontract breaking orders_v1.odcs.yaml orders_v2.odcs.yaml
echo "exit code: $LASTEXITCODE"
```

Output:

```text
Summary
[ 11 Warning ]  [ 6 Info ]
╭──────────┬─────────┬─────────────────────────────────────────────────────────────────╮
│ Severity │ Change  │ Field                                                           │
├──────────┼─────────┼─────────────────────────────────────────────────────────────────┤
│ INFO     │ Updated │ context.verifiedStatements.How many orders were placed in 2023? │
│ INFO     │ Updated │ context.verifiedStatements.What was the revenue per year?       │
│ INFO     │ Updated │ id                                                              │
│ WARNING  │ Updated │ schema.line_items.properties.lines_item_id.quality.[1]          │
│ INFO     │ Added   │ schema.line_items.properties.quantity                           │
…
│ INFO     │ Updated │ servers.Orders                                                  │
│ INFO     │ Updated │ version                                                         │
╰──────────┴─────────┴─────────────────────────────────────────────────────────────────╯
…
exit code: 0
```

Adding `quantity` is **INFO**: adding a column is schema-compatible. The changed quality queries are **WARNING**s, because the checks now run against other tables. Nothing is an ERROR, so the exit code is `0`.

Now provoke a real breaking change. Keep a copy, change the `physicalType` of `quantity` to `text` (or delete `customer_id`), and compare:

macOS / Linux:

```bash
cp orders_v2.odcs.yaml orders_v2.before.odcs.yaml
# now edit orders_v2.odcs.yaml: change quantity's physicalType to text
datacontract breaking orders_v2.before.odcs.yaml orders_v2.odcs.yaml
echo "exit code: $?"
```

Windows (PowerShell):

```powershell
Copy-Item orders_v2.odcs.yaml orders_v2.before.odcs.yaml
# now edit orders_v2.odcs.yaml: change quantity's physicalType to text
datacontract breaking orders_v2.before.odcs.yaml orders_v2.odcs.yaml
echo "exit code: $LASTEXITCODE"
```

Output:

```text
Summary
[ 1 Error ]
╭──────────┬─────────┬───────────────────────────────────────╮
│ Severity │ Change  │ Field                                 │
├──────────┼─────────┼───────────────────────────────────────┤
│ ERROR    │ Updated │ schema.line_items.properties.quantity │
╰──────────┴─────────┴───────────────────────────────────────╯

Details
╭──────────┬─────────┬────────────────────────────────────────────────────┬───────────┬───────────┬────────────────────────────────╮
│ Severity │ Change  │ Path                                               │ Old Value │ New Value │ Message                        │
├──────────┼─────────┼────────────────────────────────────────────────────┼───────────┼───────────┼────────────────────────────────┤
│ ERROR    │ Updated │ schema.line_items.properties.quantity.physicalType │ bigint    │ text      │ Changed type at                │
│          │         │                                                    │           │           │ schema.line_items.properties.… │
│          │         │                                                    │           │           │ from 'bigint' to 'text'        │
╰──────────┴─────────┴────────────────────────────────────────────────────┴───────────┴───────────┴────────────────────────────────╯
exit code: 1
```

This time you get an **ERROR** and exit code `1`. That exit code turns the check into a CI/CD gate, see [CI/CD with GitHub Actions](https://learn.datacontract.com/en/ci-cd/).

**Revert** and run the tests again:

macOS / Linux:

```bash
mv orders_v2.before.odcs.yaml orders_v2.odcs.yaml
datacontract test orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
Move-Item -Force orders_v2.before.odcs.yaml orders_v2.odcs.yaml
datacontract test orders_v2.odcs.yaml
```

Output:

```text
Testing orders_v2.odcs.yaml
…
🟢 data contract is valid. Run 37 checks. Took 0.63683 seconds.
```

> **Tools see schemas, not consumers**
>
> `datacontract breaking` says adding `quantity` is fine. Why a new major version anyway?
>
> Because "breaking" depends on how consumers use the data. A consumer that loads `SELECT *` into a strict table, or validates the exact column list, *will* break. The tool checks compatibility rules. The producer still has to know their consumers, and releasing a major version is a deliberate, conservative decision.
>
> [Consumer-Driven Contracts](https://learn.datacontract.com/en/consumer-driven/) shows how consumers make their actual dependencies explicit.

## Execute the migration

> **The contract lifecycle**
>
> A contract version goes through these states:
>
> 1. **`draft`**: work in progress, don't build on it
> 2. **`active`**: released, consumers can rely on it
> 3. **`deprecated`**: still served, consumers should migrate; add an `endOfSupport` SLA property with a date
> 4. **`retired`**: no longer served
>
> A full migration: set v2 to `active`, deprecate v1 with an end-of-support date, announce it in the support channel, and retire v1 once nobody queries it.

> **Deprecating single fields (new in ODCS 3.2)**
>
> Not every change needs a new major version right away. With `deprecated: true` on a schema object or property, you announce: "this field goes away in the next major version, stop building on it."
> The field stays documented and tested, so existing consumers keep working. Removing it later is still a breaking change and needs the next major version.

### Deprecate a field

The orders team plans to drop `customer_id` in a future v3. Announce it now. First keep a copy of the current version:

macOS / Linux:

```bash
cp orders_v2.odcs.yaml orders_v2.before.odcs.yaml
```

Windows (PowerShell):

```powershell
Copy-Item orders_v2.odcs.yaml orders_v2.before.odcs.yaml
```

In `orders_v2.odcs.yaml`, add `deprecated: true` to the `customer_id` property:

```yaml
- name: customer_id
  deprecated: true
  physicalType: text
  logicalType: string
```

Compare both versions:

macOS / Linux:

```bash
datacontract breaking orders_v2.before.odcs.yaml orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
datacontract breaking orders_v2.before.odcs.yaml orders_v2.odcs.yaml
```

Output:

```text
Summary
[ 1 Info ]
╭──────────┬────────┬──────────────────────────────────────╮
│ Severity │ Change │ Field                                │
├──────────┼────────┼──────────────────────────────────────┤
│ INFO     │ Added  │ schema.orders.properties.customer_id │
╰──────────┴────────┴──────────────────────────────────────╯
…
│ INFO     │ Added  │ schema.orders.properties.customer_id.deprecated │           │ True      │ Added contract at              │
…
```

Only INFO: deprecating is non-breaking. Delete `orders_v2.before.odcs.yaml` afterwards.

### Deprecate v1 (practice)

In `orders_v1.odcs.yaml`, set `status: deprecated` and add an `endOfSupport` SLA property:

```yaml
slaProperties:
  # ... retention, frequency
  - property: endOfSupport
    value: "2026-12-31"
    description: orders_v1 is replaced by orders_v2. Please migrate until end of 2026.
```

Check that the contract is still valid:

macOS / Linux:

```bash
datacontract lint orders_v1.odcs.yaml
```

Windows (PowerShell):

```powershell
datacontract lint orders_v1.odcs.yaml
```

Output:

```text
╭────────┬────────────────────────────────────────────┬───────┬─────────╮
│ Result │ Check                                      │ Field │ Details │
├────────┼────────────────────────────────────────────┼───────┼─────────┤
│ passed │ Data contract is valid against ODCS v3.2.0 │       │         │
╰────────┴────────────────────────────────────────────┴───────┴─────────╯
🟢 data contract is valid. Run 1 checks. Took 0.122955 seconds.
```

### Release v2 and retire v1

As a shortcut, jump to the end state:

- set `orders_v2.odcs.yaml` to `status: active`
- set `orders_v1.odcs.yaml` to `status: retired`

Run the v2 tests one last time:

macOS / Linux:

```bash
datacontract test orders_v2.odcs.yaml
```

Windows (PowerShell):

```powershell
datacontract test orders_v2.odcs.yaml
```

Output:

```text
Testing orders_v2.odcs.yaml
…
🟢 data contract is valid. Run 37 checks. Took 0.63683 seconds.
```

**Solution: orders_v2.odcs.yaml**

```yaml title=orders_v2.odcs.yaml
version: 2.0.0
kind: DataContract
apiVersion: v3.2.0
id: orders_v2
name: Orders
status: active
domain: ecommerce
description:
  purpose: Order data for analytics and reporting
  limitations: Contains PII (customer email addresses)
tags: ['orders', 'ecommerce']
context:
  instructions: >-
    Orders of the e-commerce platform and their line items.
    order_total is in cents. Timestamps are in UTC.
    Use this data for order volume and revenue analysis.
  verifiedStatements:
  - question: How many orders were placed in 2023?
    answer: SELECT COUNT(*) FROM orders_v2.orders WHERE EXTRACT(YEAR FROM order_timestamp) = 2023;
  - question: What was the revenue per year?
    answer: SELECT EXTRACT(YEAR FROM order_timestamp)::int AS year, SUM(order_total) / 100.0 AS revenue FROM orders_v2.orders GROUP BY 1 ORDER BY 1;
  constraints:
  - constraint: Never output customer_email_address or customer_id. Aggregate the data instead.
    tags: ['pii']
servers:
- server: postgres
  type: postgres
  host: localhost
  port: 5433
  database: workshop
  schema: orders_v2
schema:
- name: orders
  synonyms:
  - synonym: purchases
  - synonym: Bestellungen
    locale: de
  context:
    instructions: One row per order. Join line_items on order_id to get the purchased SKUs.
  physicalType: table
  logicalType: object
  physicalName: orders
  properties:
  - name: order_id
    physicalType: text
    logicalType: string
    required: true
    primaryKey: true
    quality:
    - type: text
      description: Must be a valid UUID
    - type: sql
      description: Ensure order_id is unique
      query: SELECT COUNT(*) FROM (SELECT order_id FROM orders_v2.orders GROUP BY order_id HAVING COUNT(*) > 1) duplicates;
      mustBe: 0
  - name: order_timestamp
    physicalType: timestamptz
    logicalType: date
    quality:
    - type: text
      description: Must not be in the future
    - type: sql
      description: Ensure order_timestamp is not in the future
      query: SELECT COUNT(*) FROM orders_v2.orders WHERE order_timestamp > CURRENT_TIMESTAMP;
      mustBe: 0
  - name: order_total
    physicalType: bigint
    logicalType: integer
    quality:
    - type: text
      description: Order total in cents, must be non-negative
    - type: sql
      description: Ensure order_total is non-negative
      query: SELECT COUNT(*) FROM orders_v2.orders WHERE order_total < 0;
      mustBe: 0
  - name: customer_id
    deprecated: true
    physicalType: text
    logicalType: string
    quality:
    - type: text
      description: Must be a non-empty alphanumeric identifier
    - type: sql
      description: Ensure customer_id is not empty
      query: SELECT COUNT(*) FROM orders_v2.orders WHERE customer_id IS NULL OR customer_id = '';
      mustBe: 0
  - name: customer_email_address
    physicalType: text
    logicalType: string
    required: true
    classification: confidential
    examples: ['test394@example.org', 'test4757@example.org']
    quality:
    - type: text
      description: Must be a valid email address
    - type: sql
      description: Ensure email addresses contain @
      query: SELECT COUNT(*) FROM orders_v2.orders WHERE customer_email_address NOT LIKE '%@%';
      mustBe: 0
  quality:
  - type: text
    description: Orders table must not be empty
  - type: sql
    description: Ensure orders table has data
    query: SELECT COUNT(*) FROM orders_v2.orders;
    mustBeGreaterThan: 0
  - type: sql
    description: Ensure every order has at least one line item
    query: SELECT COUNT(*) FROM orders_v2.orders o LEFT JOIN orders_v2.line_items li ON o.order_id = li.order_id WHERE li.order_id IS NULL;
    mustBe: 0
- name: line_items
  synonyms:
  - synonym: order lines
  - synonym: Bestellpositionen
    locale: de
  context:
    instructions: One row per purchased SKU in an order.
  physicalType: table
  logicalType: object
  physicalName: line_items
  properties:
  - name: lines_item_id
    physicalType: text
    logicalType: string
    required: true
    primaryKey: true
    quality:
    - type: text
      description: Must be a valid UUID
    - type: sql
      description: Ensure lines_item_id is unique
      query: SELECT COUNT(*) FROM (SELECT lines_item_id FROM orders_v2.line_items GROUP BY lines_item_id HAVING COUNT(*) > 1) duplicates;
      mustBe: 0
  - name: order_id
    physicalType: text
    logicalType: string
    required: true
    relationships:
    - type: foreignKey
      to: orders.order_id
  - name: sku
    synonyms:
    - synonym: article number
    - synonym: Artikelnummer
      locale: de
    physicalType: text
    logicalType: string
    quality:
    - type: text
      description: Must be a non-empty product SKU
    - type: sql
      description: Ensure sku is not empty
      query: SELECT COUNT(*) FROM orders_v2.line_items WHERE sku IS NULL OR sku = '';
      mustBe: 0
  - name: quantity
    physicalType: bigint
    logicalType: integer
    quality:
    - type: text
      description: Must be a positive integer, defaults to 1
    - type: sql
      description: Ensure quantity is positive
      query: SELECT COUNT(*) FROM orders_v2.line_items WHERE quantity <= 0;
      mustBe: 0
  quality:
  - type: text
    description: Line items table must not be empty
  - type: sql
    description: Ensure line_items table has data
    query: SELECT COUNT(*) FROM orders_v2.line_items;
    mustBeGreaterThan: 0
  - type: sql
    description: Ensure every line_items.order_id exists in orders
    query: SELECT COUNT(*) FROM orders_v2.line_items li LEFT JOIN orders_v2.orders o ON li.order_id = o.order_id WHERE o.order_id IS NULL;
    mustBe: 0
team:
  name: order_data_team
  members:
  - username: owner@example.com
    role: Owner
support:
- channel: "#order-data-help"
  url: https://example.slack.com/archives/order-data-help
  tool: slack
slaProperties:
- property: retention
  value: 10
  unit: y
  element: orders.order_timestamp
  description: Orders are deleted after 10 years
- property: frequency
  value: 1
  unit: d
  description: Data updated daily
```

**Quick check:** Which change to the active orders_v2 contract is a breaking change for consumers?

- Adding a new column
- Changing a column's physical type from BIGINT to TEXT (correct)
- Adding a description to a column
- Marking a column as deprecated: true

A type change breaks every consumer that relies on the old type, from SQL comparisons to downstream schemas. `datacontract breaking` reports it as ERROR, so it belongs in a new major version.

## Bonus

- Export both versions as HTML and compare: `datacontract export html orders_v2.odcs.yaml --output orders_v2.odcs.html`
- Predict the severity before running `datacontract breaking`: rename a field, remove `required: true`, add a new table.
- Remove the deprecated `customer_id` from a copy of v2 and run `datacontract breaking`. The flag doesn't make the removal compatible.