Developer guide

This guide is for engineers building on the DataFlow AI Platform. It covers the monorepo layout, how to set up a local development environment, the build and test commands for each component, the coding standards the project enforces, the Docker Compose dev loop, debugging, the contributing workflow, and how to add a new backend service or frontend feature.

Architecture at a glance

DataFlow AI is a microservices platform with domain-driven service boundaries. An API gateway fronts six backend services that share a single PostgreSQL database.

                    +-------------------+
                    |    API Gateway    |
                    |   (Spring Cloud)  |
                    |     port 8080     |
                    +---------+---------+
                              |
        +-------+-------+----+----+-------+-------+
        |       |       |         |       |       |
   +----+--+ +--+---+ +-+------+ +---+--+ +--+--+ +---+----+
   |Metadata| |Engine| |Lineage | |Monitor| | AI  | |Migrate |
   |Service | |      | |Service | |Service| |Copilot| |Engine |
   | :8081  | | :8082| | :8084  | | :8085 | | :8086| | :8087 |
   +---+----+ +--+---+ +---+----+ +---+---+ +--+--+ +---+---+
       |          |         |          |        |         |
       +----------+---------+----------+--------+---------+
                           |
                    +------+------+
                    | PostgreSQL  |
                    | (shared DB) |
                    +-------------+

Service responsibilities

Communication is synchronous REST between the gateway and services, SSE for alert/metrics streaming, and WebSocket for pipeline status. Kafka for event-driven lineage ingestion is planned but not yet wired.

Monorepo layout

The repository is a polyglot monorepo. The top-level directories that matter for development are:

polcomtel/
├── backend/
│   ├── platform/            Kotlin services (Gradle multi-project build)
│   │   ├── api-gateway/
│   │   ├── metadata-service/
│   │   ├── pipeline-engine/
│   │   ├── lineage-service/
│   │   ├── monitor-service/
│   │   ├── common/          Shared model + enums
│   │   ├── connector-sdk/   Connector framework + 21 connectors
│   │   ├── pushdown-sql/    SQL dialect / push-down support
│   │   ├── integration-tests/
│   │   ├── settings.gradle.kts
│   │   └── gradlew
│   ├── ai-services/         Python services (Poetry)
│   │   ├── copilot/
│   │   └── migration-engine/
│   ├── cli/                 Go CLI (cmd/, internal/, pkg/)
│   ├── docker/
│   ├── docker-compose.yml
│   └── Makefile
├── frontend/                React + Vite + TypeScript SPA
│   └── src/                 api/, auth/, components/, pages/, hooks/, stores/
├── docs/                    Markdown reference docs + OpenAPI specs
├── docs-ts/                 This documentation site (Next.js + Markdoc)
└── scripts/

Build systems by component

The Kotlin services are a single Gradle multi-project build rooted at backend/platform/settings.gradle.kts; modules are addressed with paths such as :platform:metadata-service.

Local development setup

Prerequisites

Start local infrastructure

# From backend/ — start PostgreSQL, Keycloak, and Redis
docker-compose up -d

# Verify the stack
docker-compose ps

The Compose stack starts:

Build and test commands per component

Kotlin services (Gradle)

Run a service locally with the Spring Boot bootRun task:

# Run each service on its port
./gradlew :platform:metadata-service:bootRun    # 8081
./gradlew :platform:pipeline-engine:bootRun     # 8082
./gradlew :platform:lineage-service:bootRun     # 8084
./gradlew :platform:monitor-service:bootRun     # 8085
./gradlew :platform:api-gateway:bootRun         # 8080

Python AI services (Poetry)

# Copilot service (port 8086)
cd backend/ai-services/copilot
poetry install
poetry run uvicorn copilot.main:app --host 0.0.0.0 --port 8086 --reload

# Migration engine (port 8087)
cd backend/ai-services/migration-engine
poetry install
poetry run uvicorn migration.main:app --host 0.0.0.0 --port 8087 --reload

Go CLI

cd backend/cli
go build -o dataflow ./cmd

# Verify
./dataflow version
./dataflow login --server http://localhost:8080

Frontend

cd frontend
npm install
npm run dev          # Vite dev server at http://localhost:5173

OpenAPI spec validation

cd docs/api
npx vitest run __tests__/validate-openapi.test.ts

The local Docker Compose dev loop

A typical inner-loop session combines containerised infrastructure with locally-run services so you can hot-reload the component you are working on.

Debugging

OpenAPI specs

Full OpenAPI specifications live in docs/api/:

Coding standards

The project enforces one style per language. CI runs the corresponding linter and fails the build on a violation.

Run the linters before pushing:

./gradlew ktlintCheck            # Kotlin
poetry run ruff check .          # Python (from a service directory)
gofmt -l .                       # Go — lists files needing formatting

Contributing workflow

Branch naming

feature/<ticket-id>-<short-description>    # New features
fix/<ticket-id>-<short-description>        # Bug fixes
refactor/<description>                     # Refactoring
docs/<description>                          # Documentation only

Examples: feature/DF-1234-add-kafka-connector, fix/DF-5678-scheduler-timezone-bug.

Commit message format

The project follows Conventional Commits:

<type>(<scope>): <subject>

<body>

<footer>

feat(engine): add Kafka streaming connector support

Implements KafkaConnector extending NativeConnectorBase with
consumer group management, offset tracking, and exactly-once
semantics via idempotent producer.

Closes: DF-1234

Pull request process

1. Create a feature branch from main.
2. Implement the change together with its tests.
3. Ensure all tests pass: ./gradlew test.
4. Ensure lint passes: ./gradlew ktlintCheck (Kotlin), poetry run ruff check . (Python).
5. Push the branch and open a PR on GitHub.
6. A PR is mergeable when it has at least one approval from a code owner, all CI checks pass (build, test, lint, security scan), and there are no merge conflicts.
7. Squash-merge to main.

Adding a new backend service

A new Kotlin service joins the Gradle multi-project build. The high-level steps:

A Python AI service follows the same shape under backend/ai-services/: a Poetry project with pyproject.toml, a FastAPI app served by Uvicorn, a tests/ directory exercised by pytest, and a gateway route.

Adding a new frontend feature

The frontend is a React + Vite + TypeScript single-page app. Source is organised by responsibility under frontend/src/:

To add a feature:

Command reference