API reference: Metadata & Lineage

This page documents the Metadata service (openapi-metadata.yaml, port 8081) and the Lineage service (openapi-lineage.yaml, port 8084). Both serve under /api/v1 and require a bearerAuth JWT. Pipeline CRUD also lives in the metadata service — see the Pipeline Engine reference page.

Connections

Connections register external systems. Stored credentials are vault-referenced; the API never returns a plaintext password.

listConnections accepts workspaceId (query, uuid). Connection.type enum: TERADATA, SNOWFLAKE, DATABRICKS, SAP_HANA, MSSQL, ORACLE, POSTGRESQL, MYSQL, KAFKA, GCS, AZURE_BLOB. testStatus: UNKNOWN | SUCCESS | FAILED. ConnectionCreate accepts encryptedPassword (vault-referenced). ConnectionTestResult fields: success, latencyMs, message, serverVersion, testedAt.

// POST /api/v1/connections  →  201 Created
// Request: ConnectionCreate
{
  "name": "teradata-prod",
  "type": "TERADATA",
  "host": "teradata.polkomtel.pl",
  "port": 1025,
  "database": "DWH",
  "username": "dataflow_svc",
  "encryptedPassword": "vault://connections/teradata-prod"
}

// POST /api/v1/connections/{id}/test  →  200 OK  (ConnectionTestResult)
{
  "success": true,
  "latencyMs": 42,
  "message": "Connection established",
  "serverVersion": "Teradata 17.20",
  "testedAt": "2025-12-15T14:30:00Z"
}

Quality

Quality rules define data-quality checks against a pipeline; results record each check execution.

listQualityRules accepts pipelineId (query, uuid). listQualityResults accepts pipelineId, ruleId (uuid), and limit (default 50). QualityRule.type enum: ROW_COUNT, NOT_NULL, UNIQUE, REFERENTIAL, CUSTOM_SQL, FRESHNESS, SCHEMA_MATCH, VALUE_RANGE, REGEX_MATCH, ACCEPTED_VALUES. severity: WARNING | ERROR | CRITICAL. onFailure: FAIL | WARN | SKIP_TARGET | QUARANTINE.

// POST /api/v1/quality/rules  →  201 Created
// Request: QualityRuleCreate
{
  "pipelineId": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "name": "customer-id-not-null",
  "type": "NOT_NULL",
  "column": "CUSTOMER_ID",
  "severity": "ERROR",
  "onFailure": "QUARANTINE"
}

// GET /api/v1/quality/results?pipelineId=...&limit=50  →  200 OK
[
  {
    "id": "qr-0001",
    "ruleId": "rule-0001",
    "pipelineId": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
    "passed": false,
    "failedRows": 17,
    "severity": "ERROR",
    "checkedAt": "2025-12-15T02:14:00Z"
  }
]

Catalog

The data catalog browses datasets discovered through connections.

listCatalogDatasets accepts search, connectionId (uuid), and tag. CatalogDataset.type: TABLE | VIEW | MATERIALIZED_VIEW | EXTERNAL. Each dataset includes columns[] (CatalogColumn — name, dataType, nullable, isPrimaryKey, classification).

// PUT /api/v1/catalog/datasets/{id}/tags  →  200 OK
// Request: { "tags": ["billing", "pii", "gold"] }
{
  "id": "ds-0001",
  "name": "DWH.CUSTOMERS",
  "type": "TABLE",
  "tags": ["billing", "pii", "gold"],
  "columns": [
    { "name": "CUSTOMER_ID", "dataType": "BIGINT", "nullable": false, "isPrimaryKey": true, "classification": "INTERNAL" }
  ]
}

Governance

Governance manages data classifications and a queryable audit log.

getAuditLog accepts category, userId (uuid), from/to (date-time), and limit (default 100). DataClassification.level: PUBLIC | INTERNAL | CONFIDENTIAL | RESTRICTED. AuditLogEntry.category: PIPELINE | CONNECTION | USER | WORKSPACE | SECURITY | SYSTEM.

// POST /api/v1/governance/classifications  →  201 Created
// Request: DataClassificationCreate
{
  "name": "Subscriber PII",
  "level": "RESTRICTED",
  "description": "Personal data of Polkomtel subscribers"
}

GDPR

GDPR endpoints handle Data Subject Access Requests (DSARs, 30-day SLA), consent records, and Article 30 data mapping.

listDsars accepts status, requestType, page (default 0), size (default 20). getDataMap accepts workspaceId (uuid). DsarRequest.requestType: ACCESS | ERASURE | RECTIFICATION; status: SUBMITTED | IN_PROGRESS | COMPLETED | REJECTED. DataMapReport includes totalColumnsScanned, piiColumnsFound, rodoRelevantColumnsFound, entries[], and classification/sensitivity breakdowns.

// POST /api/v1/gdpr/dsar  →  201 Created
// Request: DsarSubmitRequest
{
  "subjectId": "5d6e7f80-1111-2222-3333-444455556666",
  "requestType": "ERASURE",
  "subjectEmail": "subscriber@example.pl",
  "details": "Right to be forgotten request"
}

// Response: DsarSubmitResponse
{
  "trackingId": "dsar-2025-001847",
  "status": "SUBMITTED",
  "slaDueAt": "2026-01-14T14:30:00Z"
}

// GET /api/v1/gdpr/data-map?workspaceId=...  →  200 OK  (DataMapReport)
{
  "totalColumnsScanned": 4821,
  "piiColumnsFound": 312,
  "rodoRelevantColumnsFound": 287,
  "entries": [
    { "dataset": "DWH.CUSTOMERS", "column": "PESEL", "classification": "RESTRICTED", "sensitivity": "HIGH" }
  ]
}

Admin — Users

listUsers accepts workspace (uuid), role, search, active (bool), page (default 0), size (default 20). getUserActivity accepts since (date-time). The role enum, used everywhere, is ADMIN | ENGINEER | ANALYST | VIEWER. createUser returns 409 on a duplicate user.

// POST /api/v1/admin/users  →  201 Created
// Request: CreateUserRequest
{
  "email": "anna.kowalska@polkomtel.pl",
  "displayName": "Anna Kowalska",
  "role": "ENGINEER",
  "workspaceId": "9c1f0e2a-1111-2222-3333-444455556666"
}

// GET /api/v1/admin/users?page=0&size=20  →  200 OK  (UserListResponse)
{
  "items": [
    { "id": "u-1001", "email": "anna.kowalska@polkomtel.pl", "role": "ENGINEER", "active": true }
  ],
  "total": 134,
  "page": 0,
  "size": 20
}

Admin — Workspaces

// POST /api/v1/admin/workspaces/{id}/members  →  201 Created
// Request: AddMemberRequest
{
  "userId": "u-1001",
  "role": "ANALYST"
}

Admin — Audit log

queryAdminAuditLog accepts userId, workspaceId (uuid), action, resourceType, since/until (date-time), page (0), size (20). getAdminAuditStats accepts workspaceId (uuid) and days (default 30). exportAdminAuditLog accepts the same filters plus format (json | csv, default json) and returns a string body of type application/json or text/csv.

// GET /api/v1/admin/audit?page=0&size=20  →  200 OK  (AuditLogResponse)
{
  "items": [
    { "action": "PIPELINE_UPDATE", "resourceType": "PIPELINE", "userId": "u-1001", "at": "2025-12-15T14:30:00Z" }
  ],
  "total": 982,
  "page": 0,
  "size": 20
}

Admin — System settings

// PUT /api/v1/admin/settings/{key}  →  200 OK
// Request: SetSettingRequest
{ "value": "30" }

Lineage Service

The Lineage service tracks dataset and column lineage, impact analysis, and ingests OpenLineage events. Dataset IDs use the namespace.name format — for example teradata.DWH.CUSTOMERS.

Lineage graph

getColumnLineage takes a fully qualified column ID such as ns.table.column.

// GET /api/v1/lineage/datasets/teradata.DWH.CUSTOMERS  →  200 OK  (LineageGraph)
{
  "datasetId": "teradata.DWH.CUSTOMERS",
  "upstream": ["teradata.STG.CUSTOMERS_RAW"],
  "downstream": ["snowflake.DWH.DIM_CUSTOMER"],
  "jobs": ["billing-daily-load"]
}

Traversal

getUpstreamLineage and getDownstreamLineage accept depth (default 5). getSubgraph accepts depth (default 3) and direction (UPSTREAM | DOWNSTREAM | BOTH, default BOTH).

// GET /api/v1/lineage/upstream/snowflake.DWH.DIM_CUSTOMER?depth=3  →  200 OK
[
  { "datasetId": "teradata.DWH.CUSTOMERS", "depth": 1 },
  { "datasetId": "teradata.STG.CUSTOMERS_RAW", "depth": 2 }
]

Analytics

ImpactAnalysis.riskLevel: LOW | MEDIUM | HIGH | CRITICAL; impactedDatasets[].impactType: DIRECT | INDIRECT. LineageStats fields: totalDatasets, totalEdges, totalJobs, totalColumnLinks, orphanDatasets, maxDepthUpstream, maxDepthDownstream, lineageBreaks.

// GET /api/v1/lineage/impact/teradata.DWH.CUSTOMERS  →  200 OK  (ImpactAnalysis)
{
  "datasetId": "teradata.DWH.CUSTOMERS",
  "riskLevel": "HIGH",
  "impactedDatasets": [
    { "datasetId": "snowflake.DWH.DIM_CUSTOMER", "impactType": "DIRECT" },
    { "datasetId": "snowflake.MART.CHURN_SCORE", "impactType": "INDIRECT" }
  ]
}

Search

searchLineage requires q and accepts type, connection, and entity (DATASET | COLUMN, default DATASET); it returns 400 when q is missing. getHotDatasets accepts limit (default 10).

// GET /api/v1/lineage/search?q=customer&entity=DATASET  →  200 OK
[
  { "datasetId": "teradata.DWH.CUSTOMERS", "name": "CUSTOMERS", "connection": "teradata-prod" }
]

OpenLineage events

getRecentEvents accepts limit (default 100). RunEvent.eventType: START | RUNNING | COMPLETE | ABORT | FAIL | OTHER; it carries run (RunFacet), job (JobFacet), and inputs[]/outputs[] (DatasetEvent with schema, dataSource, and columnLineage facets).

// POST /api/v1/lineage/events  →  201 Created
// Request: RunEvent (OpenLineage)
{
  "eventType": "COMPLETE",
  "eventTime": "2025-12-15T02:30:00Z",
  "run": { "runId": "3fa85f64-5717-4562-b3fc-2c963f66afa6" },
  "job": { "namespace": "dataflow", "name": "billing-daily-load" },
  "inputs": [{ "namespace": "teradata", "name": "DWH.CUSTOMERS" }],
  "outputs": [{ "namespace": "snowflake", "name": "DWH.DIM_CUSTOMER" }]
}

// Response
{ "status": "RECORDED", "job": "billing-daily-load" }

// POST /api/v1/lineage/events/batch  →  201 Created
{ "status": "ACCEPTED", "accepted": 48, "total": 50 }