Organization Migration

This is the migration plan for moving Corti's Keycloak identity model from realm groups and per-client corti.* mappers onto the Keycloak Organizations feature. It records the agreed target model, the claim-by-claim contract change, the phased rollout, and the decisions (including the rejected alternatives and the reasons).

For the current state see Token Mappers. For the developer-facing claim contract see Platform Mappers Contract. For impact analysis see Token Consumers.

1. Why now

Keycloak was upgraded to 26.7.0, which is the first release where the whole model is expressible:

The business driver is per-customer SSO: a customer wants their own external identity provider, which Keycloak only supports per organization.

The model being replaced has four structural problems, all catalogued in Token Mappers §8:

• Every client carries its own copy of the same recurring claims, so they drift between clients.
• The same logical claim is delivered two different ways — hardcoded claim on client-credentials clients, User Attribute mapper on user-based clients.
• Claims sourced from group attributes are silently absent for anything without group membership (service accounts, the studio user).
• There is no single source of truth — the mapper set lives in the Console / platform-service / platform-api code paths that call the Admin API.

2. Decision history

The design went through three public iterations before it settled. Reading an older one as current is the most likely way to get this wrong.

3. Scope

In scope — every realm with tenants.<tenant>.keycloak.enabled: true on the Corti-hosted shared clusters:

The realm count is larger than it looks: several customers have their own tenant, and therefore their own realm. prod-weu alone has ten — base, bupa, dedalus, trifork, capioswe, nhsswansea, medatixx, aleris, bighand, celo. The migration is per realm.

The version floor is already met everywhere in scope — all eight clusters run keycloak-v3 1.7.0 (appVersion 26.7.0), so no chart bump is required to start.

4. The model

Organization = Keycloak organization. Group = Keycloak organization group. The naming is 1:1 through the stack; Console keeps the user-facing "Customer" label for what is now an organization.

Organization types

There are only two, and they differ by who lives in them, not by how they were created:

Groups

• Every organization always has an All group containing every end-user. platform-api adds users to it on creation and invitation; a Hardcoded Group IdP mapper routes federated logins there so no SSO user is ever group-less. All is not deletable and not renameable; organizations and all other groups are renameable.
• Group paths are relative to the organization, so siemens/sales and acme/sales are unrelated entities with no realm-level collision.
• A user belongs to exactly one organization, and may belong to several groups within it — the latter is new.
• Template access control stays at group level. Organization-level scoping is a follow-up (§10).

flowchart TD
    subgraph normal["Normal case — customer == org (~99% of realms)"]
        direction TB
        P1["Project / realm"] --> H1["Hidden org<br/>service accounts + studio users"]
        P1 --> C1["Org: GP"]
        P1 --> C2["Org: MSK"]
        C1 --> G1["/All"]
        C2 --> G2["/All"]
    end
    subgraph dept["Department model — BUPA and Aleris only"]
        direction TB
        P2["Project / realm"] --> H2["Hidden org<br/>service accounts + studio users"]
        P2 --> D["Org: Default"]
        D --> G3["/All"]
        D --> G4["/Weight-Mgmt"]
        D --> G5["/Mental-Health"]
    end

The two shapes are per realm, not mixed within one. A realm either maps its Customers to organizations (the normal case) or collapses them into groups under a single Default organization (BUPA and Aleris, whose Customers are service lines sharing one domain).

5. Migration mapping

A Console Customer becomes an organization (customer == org) for ~99% of accounts.

BUPA and Aleris are the two exceptions and use the department model: one Default organization per realm, with their existing Customers migrated to groups underneath it. Their Customers are service lines and departments rather than distinct commercial customers, and their users share a single domain.

6. Token contract

Target shape

{
  "organization": {
    "gp": {
      "id": "f8d3c4e1-2477-44d7-a85b-d3b43f6b31fa",
      "console_project_id": ["alstertext"],
      "console_customer_id": ["gp"],
      "metronome_customer_id": ["d657c398-ec1d-4e7b-8f9e-3b175712d332"],
      "is_nfr": ["false"],
      "retention_minutes": ["43200"],
      "groups": ["/All"]
    }
  },
  "corti": {
    "retention_minutes": "720",
    "auto_created": "true",
    "is_studio_client": "false"
  }
}

There is exactly one entry under organization, keyed by the org alias.

The slim corti.* namespace

corti.* is not retired. Keycloak owns the organization object and client-level values cannot be written into it, so corti.* survives — reduced to client-level attributes only:

Precedence rule for consumers:

effective = corti.<key> ?? organization.<org>.<key>[0]

A client-level value always wins. Namespace alone tells you the level a value came from.

Dropped claims

Retired after dual-emit: the top-level groups claim. Project and customer identity move to organization.<org>.console_project_id and console_customer_id.

Scope and mapper configuration

Keycloak adds organization as an optional client scope to every client. That is not sufficient here: because corti.* no longer carries project identity, a client that omits scope=organization would receive a token with no project identity at all.

• organization becomes a default scope on every client.
• The Organization Membership mapper needs add organization id and add organization attributes enabled — neither is on by default.
• The Organization Group Membership mapper must sit in the same scope as the Organization Membership mapper, or group paths are silently omitted.

7. Phases

Phase 0 — foundations and verification

• Enable Organizations on every in-scope realm. Previously organization: true existed in exactly one file — environments/development/dev-neu/shared/apps/platform-service/values.yaml (now deleted with the dev-neu cluster) — on the base and assistant tenants. The key is tenants.<tenant>.keycloak.organization, set per cluster in environments/<env>/<cluster>/shared/apps/platform-service/values.yaml. Before repeating it ~40 times, check whether platform-service supports a tenant-level default so it can be promoted to config/services/platform-service/base/values.yaml.
• Re-verify client-credentials + scope=organization and record the version.
• Verify aggregate-attribute behaviour for organization groups, to document the rejected alternative in §9 from evidence.
• Confirm no Keycloak built-in authorization policy depends on group policies.

Phase 1 — Keycloak-side scaffolding

Owned by platform-service (realm bootstrap) and platform-api (runtime provisioning).

• Make organization a default scope at client creation, and backfill existing clients across all in-scope realms.
• Add the Organization Group Membership mapper to the same scope, with the Organization Membership mapper's id and attribute options enabled.
• platform-api: create the hidden org per project; create customer orgs; create All per org; add users to All on create and invite; configure the Hardcoded Group IdP mapper → /All and Advanced Claim to Group for department routing.
• Add client-attribute mappers for the slim corti.* set.

Phase 2 — dual-emit

• Emit the full legacy corti.*, the realm groups claim, and the new organization.* simultaneously. The precedence rule is not a blanket "one namespace wins" — it depends on which kind of corti.* key is present:
  • Org-level attributes (console_project_id, console_customer_id, metronome_customer_id, is_nfr, etc.): prefer organization.<org>. The legacy corti.* copies are stale duplicates, not overrides.
  • Slim client-level override set (retention_minutes, auto_created, is_studio_client): corti.* still takes precedence per the §6 parsing contract.
• Backfill organizations from existing realm groups, per realm — customer == org everywhere except BUPA and Aleris.
• Backfill metronome_customer_id from the user-level attribute before it can be decommissioned. On the assistant client this attribute is the only source for that claim (every other client carries a hardcoded value), so the organization attribute must be populated from it, not merely alongside it. Reconcile while doing so: the value is supposed to be identical for everything in a project, so report any user whose Metronome Customer ID diverges from its project's value rather than silently picking one. A divergence is either a data bug or a billing arrangement nobody wrote down.
• Audit per realm: users in zero or more than one organization; users missing from All; organizations missing required attributes.

Phase 3 — consumer migration

See §8. Nothing in Phase 4 starts until every in-scope consumer reads organization.*.

Phase 4 — cutover and cleanup

Per realm, in this order:

1. Shrink corti.* to the client-level set only.
2. Delete the realm groups.
3. Remove the mappers for dropped claims.
4. Decommission the user-level Metronome Customer ID attribute — remove the Metronome User Attribute mapper from the assistant-dedicated scope and delete the attribute from the user entities.

Then delete scripts/keycloak/check-realms-protocol-mappers/ — its only purpose was guarding openmeter_customer_id against being empty, which the org model makes structurally impossible. Only two files are tracked there (check_keycloak_openmeter.py and requirements.txt); any local venv/ is ignored by its own generated .gitignore and needs no cleanup.

8. Consumer workstreams

Paths from Token Consumers. Sequenced so the two shared libraries land first — between them they cover most of the surface, and fixing them once avoids three services each inventing their own organization parser.

9. Decisions record

Decisions worth recording because they were contested, or because the reasoning matters more than the outcome.

Organization group attributes and why identity lives on the organization. It was asserted during review that "Keycloak won't populate organization's group attributes to the token". That is not accurate as stated — Keycloak's organization group documentation describes surfacing group attributes via a User Attribute mapper with Aggregate attributes enabled. The real constraint is narrower: the Organization Group Membership mapper emits only group paths, and aggregated attributes merge across all of a user's groups with no per-group or per-org provenance. So is_nfr and retention_minutes do belong on the organization — but because aggregation is ambiguous, not because it is impossible. This is precisely what forecloses per-group retention overrides, and why the override lives at client level instead.

stripe_customer_id is safe to drop. Token Consumers lists copilot-api and agent-api as reading it. They declare stripe_id struct fields (copilot-api/pkg/auth/auth.go, agent-api/pkg/auth/context.go) that are never read — the only other references are "test-stripe-id" fixtures. Dropping it requires no behavioural change, only deleting dead fields.

Rejected — a 1:1 project ↔ organization mapping. Proposed to fix the fact that a Console project has no single Keycloak entity behind it. Rejected because a single SSO configuration per project is insufficient for reseller use cases, which are the reason the Organizations work exists. Several customer organizations within one project cover that instead. Recorded so it is not relitigated.

Rejected — keeping the groups claim permanently. Would have avoided all consumer changes for project and customer identity, at the cost of maintaining two group hierarchies per user forever — the exact drift the migration exists to remove.

10. Open items

Not safe to build against until closed.

• Medatixx. Under customer == org, its dozens of numeric practice-ID Customers — most with one or two users — each become an organization. This was flagged during review as the noisiest migration outcome, and it is clearly Admin-API-generated rather than a real commercial structure. BUPA and Aleris are the agreed department-model exceptions; Medatixx is not. Re-raise before Phase 2.
• Default organization optionality. Whether a default org is auto-created on project creation, or the integrator must create one. Product wants the optionality rather than a hard requirement. Undecided.
• is_nfr fate. It is being phased out (already restricted to Super Admins). If it survives the migration it must be an organization attribute, which means the internal NFR case needs its own NFR-named organization. This plan assumes it survives; confirm before Phase 2.
• console_distributor_id. Reported as never used, and separately as used for analytics — the latter reads Console's database, not the JWT, so dropping it from the token is consistent with both. Confirm.
• The type of the client-level corti.retention_minutes. The shape in §6 shows it as a string ("720"), but it is emitted as a JSON int today — platform-api sets "jsonType.label": "int" on the retention_minutes mapper — and copilot-api, its only consumer, decodes it as *int and has a test asserting that a string fails. Either keep it an int (no copilot-api change on the corti half) or move to a string for consistency with the organization arrays (copilot-api needs a custom unmarshal either way, for the organization half). Decide before Phase 1 writes the client-attribute mappers.
• Organization-level template access control. Product intent, and a hard prerequisite if the All group is ever dropped.
• Admin API surface. The intended approach is to migrate under the hood without breaking the existing API, introduce the new shape as a new part of the Admin API, and deprecate the old one slowly. Tracked in api-specs#212.
• Departments hidden for non-SSO integrators. For integrators who will never use SSO, a mandatory group level adds a concept without value; the UI should keep feeling single-level for them.

11. Verification

Model (Organizations were previously enabled on dev-neu base and assistant tenants; dev-neu has since been destroyed — verify on a cluster with Organizations enabled):

1. Create a test project; assert the hidden org, a customer org, and All all exist.
2. Mint a client-credentials token and a delegated (user) token.
3. Assert organization has exactly one entry; id is a string; every other attribute is an array; groups carries organization-relative paths.
4. Set a client-level retention override and assert corti.retention_minutes takes precedence over organization.<org>.retention_minutes.
5. Assert none of the dropped claims are present.

Config changes — render the platform-service values diff with orca tools drdep before merge.

Per-realm audit — users in zero or more than one organization, users missing from All, organizations missing required attributes. The admin-API iteration pattern in scripts/keycloak/check-realms-protocol-mappers/check_keycloak_openmeter.py (TokenManager, 60-second refresh, single 401 retry) is worth reusing before that script is deleted in Phase 4.