API reference

API reference: overview

The DataFlow AI Platform exposes a microservice REST API fronted by a single API Gateway. Every client request enters through the gateway, which authenticates the JWT, applies per-route rate limiting, attaches a correlation ID, and transparently proxies the call to the backend service that owns the route prefix.


Base URLs

All client traffic should target the gateway. Backend services are also reachable directly in local development for debugging.

EnvironmentBase URL
Production (gateway)https://api.dataflow.polkomtel.pl
Local gatewayhttp://localhost:8080
Local metadata (direct)http://localhost:8081/api/v1
Local pipeline-engine (direct)http://localhost:8082/api/v1
Local lineage (direct)http://localhost:8084/api/v1
Local monitor (direct)http://localhost:8085/api/v1

Every backend service also publishes the production server https://api.dataflow.polkomtel.pl/api/v1 — that is, the path you reach through the gateway. Every functional endpoint is prefixed with /api/v1.

Use the gateway

In production always call the gateway. Direct service ports are firewalled and intended only for local development. The gateway is what adds authentication enforcement, rate limiting, and the correlation headers documented below.


Services and ports

The platform is built from five OpenAPI-specified services plus three proxied-only services. All specs declare openapi: 3.1.0 and info.version: 1.0.0.

ServiceSpec fileLocal portResponsibility
API Gatewayopenapi-gateway.yaml8080Auth, rate limiting, routing, SSE/WS passthrough
Metadata Serviceopenapi-metadata.yaml8081Pipelines, connections, quality, catalog, governance, GDPR, admin
Pipeline Engineopenapi-pipeline-engine.yaml8082Execution, scheduling, orchestration, Git, Flink
Lineage Serviceopenapi-lineage.yaml8084Dataset/column lineage, impact analysis, OpenLineage events
Monitor Serviceopenapi-monitor.yaml8085Dashboard metrics, alerts, alert definitions/events, SSE
Copilot Serviceproxied8086AI copilot
Migration Serviceproxied8087Migration plans
Connector SDKproxied8088CDC connectors

Gateway routing

The gateway maps each route prefix to a backend service. Most gateway paths are $ref-mounted re-exports of the backend specs; the gateway also defines six first-class operations of its own (health, copilot, migration, CDC).

Route prefixBackend servicePort
/api/v1/pipelines/**metadata-service8081
/api/v1/connections/**metadata-service8081
/api/v1/quality/**metadata-service8081
/api/v1/catalog/**metadata-service8081
/api/v1/governance/**metadata-service8081
/api/v1/gdpr/**metadata-service8081
/api/v1/admin/**metadata-service8081
/api/v1/runs/**pipeline-engine8082
/api/v1/scheduler/**pipeline-engine8082
/api/v1/orchestrator/**pipeline-engine8082
/api/v1/git/**pipeline-engine8082
/api/v1/flink/**pipeline-engine8082
/api/v1/runs/*/streampipeline-engine (WebSocket)8082
/api/v1/lineage/**lineage-service8084
/api/v1/monitor/**monitor-service8085
/api/v1/monitor/sse/**monitor-service (SSE)8085
/api/v1/ai/**copilot-service8086
/api/v1/migration/**migration-service8087
/api/v1/cdc/**connector-sdk8088

Gateway-owned operations

These six operations are defined directly by the gateway rather than re-exported from a backend spec.

MethodPathOperation IDPurposeAuth
GET/healthgatewayHealthGateway health checkNone
GET/api/v1/ai/copilotcopilotEndpointAI copilot (proxied to copilot-service)JWT
GET/api/v1/migration/plansmigrationPlansMigration plans (proxied to migration-service)JWT
GET/api/v1/cdc/connectorslistCdcConnectorsList CDC connectors (proxied to connector-sdk)JWT
POST/api/v1/cdc/connectorsdeployCdcConnectorDeploy a CDC connectorJWT
GET/api/v1/cdc/healthcdcHealthCDC health summaryJWT
// GET /health  →  200 OK
{
  "status": "UP",
  "gateway": "dataflow-api-gateway"
}

Versioning

The platform uses path-based versioning. The current and only version is v1, embedded in every functional route as /api/v1/....

  • All five OpenAPI specs declare info.version: 1.0.0.
  • The gateway GET /health endpoint is unversioned — it carries no /api/v1 prefix.
  • A new major version, if introduced, would appear as a new path segment (/api/v2/...); v1 routes remain stable.

Authentication

The platform authenticates with Keycloak OIDC + JWT Bearer tokens.

  • Security schemes. The gateway declares two schemes — oidcAuth (openIdConnect) and bearerAuth (http / bearer / JWT). Backend services declare only bearerAuth.
  • OIDC discovery: https://auth.dataflow.polkomtel.pl/realms/dataflow/.well-known/openid-configuration
  • Realm: dataflow
  • Client IDs: dataflow-ui (public, Authorization Code + PKCE) and dataflow-api (confidential, Client Credentials).
  • Header: Authorization: Bearer <token>

Required JWT claims

Every authenticated request must carry a token with the following claims.

ClaimDescription
subUser ID
realm_access.rolesUser roles — one or more of ADMIN, ENGINEER, ANALYST, VIEWER
workspace_idActive workspace (custom claim)
GET /api/v1/pipelines HTTP/1.1
Host: api.dataflow.polkomtel.pl
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6...

Exceptions

  • GET /health (gateway) is unauthenticated — its OpenAPI security is [].
  • Webhook endpoints (orchestrator trigger, Git webhooks) authenticate with HMAC signature headers — X-Webhook-Signature, X-Hub-Signature-256, X-Gitlab-Token — instead of a JWT.

Heads up

A missing or invalid token yields 401 UNAUTHORIZED. A valid token whose roles or workspace_id do not permit the operation yields 403 — for example, a cross-workspace pipeline dependency.


Error format

Every service returns a uniform ErrorResponse body on failure.

{
  "error": "NOT_FOUND",
  "message": "Pipeline not found: f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "timestamp": "2025-12-15T14:30:00Z"
}
FieldTypeNotes
errorstringMachine-readable code — BAD_REQUEST, UNAUTHORIZED, NOT_FOUND, RATE_LIMITED, GATEWAY_TIMEOUT, SERVICE_UNAVAILABLE
messagestringHuman-readable description
timestampstring (date-time)Optional; present on most responses

HTTP status codes

StatusMeaning
200OK
201Created
202Accepted — async (pipeline run / trigger)
204No Content — delete
400Bad request / invalid payload
401Missing or invalid JWT
403Forbidden — e.g. cross-workspace dependency
404Resource not found
409Conflict — duplicate user/cluster, merge conflict
429Rate limited (gateway only)
502 / 503 / 504Gateway-level upstream errors

Gateway-specific errors

The gateway extends ErrorResponse for its own failure modes.

ErrorStatusExtra fields
RateLimited429Adds retryAfter (integer); also sets X-RateLimit-Limit and X-RateLimit-Reset headers
GatewayTimeout504error: "GATEWAY_TIMEOUT"
ServiceUnavailable503error: "SERVICE_UNAVAILABLE"; body adds service
// 429 Too Many Requests
{
  "error": "RATE_LIMITED",
  "message": "Rate limit exceeded for /api/v1/pipelines",
  "retryAfter": 42,
  "timestamp": "2025-12-15T14:30:00Z"
}

Rate limiting and correlation headers

The gateway applies these headers to every proxied response.

HeaderDescriptionExample
X-RateLimit-LimitMax requests per minute100
X-RateLimit-RemainingRemaining requests in the window87
X-RateLimit-ResetUnix timestamp of the window reset1734272400
X-Request-IdRequest correlation UUID for tracingd1e2f3a4-5b6c-7d8e-9f0a-1b2c3d4e5f6a

Rate limits are enforced per route. When the window is exhausted the gateway responds 429 with a retryAfter value (seconds). Always log the X-Request-Id on the client side — it is the single correlation key for support and tracing across all services.


Pagination

There is no platform-wide pagination contract. Three distinct patterns are used across the services; there is no cursor-based pagination anywhere.

1. Page/size offset pagination

Used by Admin (/admin/users, /admin/audit) and the GDPR DSAR list. Query params page (default 0) and size (default 20). The response wraps the items with total, page, and size.

// GET /api/v1/admin/users?page=0&size=20  →  200 OK
{
  "items": [ /* UserResponse[] */ ],
  "total": 134,
  "page": 0,
  "size": 20
}

2. limit-capped lists

Most "list / history / recent" endpoints accept a limit query param (defaults vary — 10, 50, 100). There is no cursor; the response is a bare array.

GET /api/v1/quality/results?pipelineId=...&limit=50

3. Time-window filters

Alert and event history endpoints filter by time rather than page number, using sinceSeconds, from, and to.

GET /api/v1/monitor/alerts/history?sinceSeconds=86400

Endpoint group index

The full API is documented across three reference pages. Use this index to find the right group.

GroupServiceReference page
Pipelines, Connections, QualityMetadataAPI reference: Pipeline Engine (pipelines CRUD) / Metadata & Lineage
Catalog, Governance, GDPR, AdminMetadataAPI reference: Metadata & Lineage
Execution, Scheduler, OrchestratorPipeline EngineAPI reference: Pipeline Engine
Git, FlinkPipeline EngineAPI reference: Pipeline Engine
Lineage graph, traversal, analytics, search, OpenLineage eventsLineageAPI reference: Metadata & Lineage
Dashboard, Alerts, Definitions, Events, SSE, ChannelsMonitorAPI reference: Monitor & AI
Copilot, Migration, CDCProxied servicesAPI reference: Monitor & AI

Endpoint counts

ServicePath operations (approx.)
Gateway (own)6 (plus many $ref re-exports)
Metadata38
Pipeline Engine38
Lineage14
Monitor35

Proxied-only services (copilot, migration, CDC, connector-sdk) are surfaced through the gateway but are not formally specified beyond the six gateway stub operations.

Previous
RBAC & permissions