AI Copilot

The AI Copilot is the platform-wide intelligence layer of the DataFlow AI Platform. It is not a standalone page but an omnipresent overlay docked into the application shell — a collapsible right-side slide-in panel available on every screen, powered by real Claude API calls. It knows your current screen, the selected pipeline, the active node, the editor content, and your role, and it adapts its language and suggestions accordingly.

New here? What the Copilot is, in plain words

The AI Copilot is a helper you can talk to in ordinary language — English or Polish — anywhere in the platform. Think of it as a knowledgeable colleague who is always available in a side panel: you ask a question or describe what you want, and it answers, writes the code, or builds the thing for you.

It is genuinely powered by a real AI model (Anthropic's Claude), not a set of canned replies. That means you can phrase things naturally — there is no special syntax to learn.

Here is what it can do for you, with everyday examples:

• Answer questions about your data. "Which tables hold daily call records?" — it searches the real catalog and tells you.
• Write SQL for you. "Show me total revenue by region for last month" — it produces a working query you can run with one click. ("SQL" is the language used to ask databases questions; the Copilot writes it so you don't have to.)
• Build a whole pipeline from a description. "Load daily sales from SAP HANA into Snowflake, skip test rows" — it drafts the pipeline and offers to create it.
• Explain things. "What does the CHURN_SCORE column mean?" or "Explain what this pipeline does" — it reads the metadata and explains in plain language.
• Diagnose failures. When a pipeline breaks, it reads the error and tells you, in plain words, what went wrong and how to fix it.
• Suggest improvements on its own. It quietly watches your pipelines and surfaces tips — "this pipeline could run 70% cheaper" — as little insight cards.

What the AI Copilot does

The Copilot sits behind a floating sparkle trigger that appears on every page. Clicking it opens a 320px-wide sidebar that slides in from the right. The Copilot delivers a stack of capabilities through one panel:

• Conversational chat — pipeline building, debugging, and data exploration in natural language.
• NL-to-SQL — turns plain-language questions into SQL across five dialects.
• NL-to-pipeline generation — turns a prompt into a complete DataFlow YAML pipeline.
• RAG catalog search — answers grounded in your real catalog metadata via a pgvector retrieval pipeline.
• Inline suggestions — Monaco ghost-text completion for SQL, Python, and YAML.
• Proactive insights — background analysis surfaced as actionable insight cards.
• Error diagnosis — structured root-cause analysis of failed pipeline runs.
• Agentic pipeline builder — an autonomous, tool-using agent that builds pipelines end-to-end behind approval gates.
• Root-cause analysis (RCA) — an incident analysis agent that walks real monitoring and metadata tools.

How it opens: via setCopilotOpen(true) in the shell store (the floating trigger), or by navigating to /ai-copilot?q= (prefilled prompt) or /ai-copilot?insight= (jump to an insight). Entry components: src/components/ai-copilot/*.

Who uses it

The Copilot serves every persona; only the language complexity and the available quick actions change.

Screen layout

The Copilot sidebar is a fixed, full-height panel pinned to the right edge of the viewport. It slides in over 300ms and contains a header, a tab bar, a scrollable content area, a quick-actions grid, and an input field.

+----------------------------------------------------+
| HEADER                                              |
| [Sparkle] AI Copilot   [Context pill]    [X Close]   |
+----------------------------------------------------+
| TAB BAR                                              |
| [ Chat ]  [ Insights (3) ]  [ Diagnosis ]            |
+----------------------------------------------------+
|                                                      |
|  MESSAGE AREA (scrollable)                           |
|                                                      |
|  +------------------------------------------------+  |
|  | [AI]  AI Copilot                      10:32 AM |  |
|  | Hello Marek! I can see you're in Design Studio.|  |
|  +------------------------------------------------+  |
|                                                      |
|  +------------------------------------------------+  |
|  |                          [User]  Marek         |  |
|  | I need daily sales from SAP HANA               |  |
|  +------------------------------------------------+  |
|                                                      |
|  +------------------------------------------------+  |
|  | [AI]  AI Copilot                      10:33 AM |  |
|  | Here's what I suggest:                         |  |
|  |  +------------------------------------------+  |  |
|  |  | SAP HANA --> Filter --> Your Workspace   |  |  |
|  |  +------------------------------------------+  |  |
|  |  [ Yes, create ]   [ Modify first ]           |  |
|  +------------------------------------------------+  |
|                                                      |
+----------------------------------------------------+
| QUICK ACTIONS (2x2)                                  |
| [ Build Pipeline ]   [ Debug Error ]                 |
| [ Explain Column ]   [ Map Data ]                    |
+----------------------------------------------------+
| INPUT                                                |
| [ Ask AI anything...                          [>] ]  |
+----------------------------------------------------+

UI controls

Message bubbles

User messages render right-aligned in violet bubbles; AI messages render left-aligned in light bubbles. While the model is generating, the AI bubble shows a three-dot streaming animation. An AI message can embed a MiniPipelineDiagram (a lightweight SVG source→transform→target preview), a DiffViewer (for an "Apply Fix" code diff), and inline action buttons such as Yes, create or Modify first.

Quick Actions

Chat and RAG catalog search

The chat panel is the default mode. When you send a message, the CopilotService orchestrator runs a multi-step flow before any answer reaches you:

1. Intent routing — keyword detection routes SQL-style questions to the NL-to-SQL path and error reports to error diagnosis.
2. RAG enrichment — RAGService.get_context_for_query() retrieves relevant catalog metadata. This is best-effort: if pgvector is unreachable, the Copilot falls back to non-RAG chat rather than failing.
3. Prompt assembly — an enriched system prompt is built, injecting the active pipeline context and the RAG results.
4. Model call — the request goes to Claude through the pluggable provider layer.
5. Action parsing — the response is scanned for [ACTION type=... label=... payload={...}] blocks, which become structured action buttons in the chat.
6. Confidence scoring — a heuristic produces a 0–1 confidence score (0.7 base, +0.15 with RAG context, +0.05 with pipeline context).

RAG search uses pgvector cosine similarity over a catalog_embeddings table, with embeddings produced by the all-MiniLM-L6-v2 sentence-transformers model (384 dimensions). A rag_similarity_threshold of 0.72 filters weak matches. The catalog Q&A endpoint, /catalog/ask, answers questions grounded directly in metadata-service data.

NL-to-SQL

The Copilot turns plain-language questions into SQL across five dialects — Snowflake, Teradata, PostgreSQL, SAP HANA, and MSSQL. The dialect is auto-detected from your message text, or you can name it explicitly.

The NLToSQLService supports four operations:

Generated SQL carries a confidence estimate and warnings — the Copilot flags SELECT *, DELETE/UPDATE without a WHERE clause, DROP/TRUNCATE, and dialect-invalid LIMIT usage. A richer semantic-layer path (/semantic/query) enriches the question against the Polkomtel business glossary — roughly 11 metrics (churn rate, ARPU, MoU, net adds, revenue), 14 dimensions (region, tariff plan, segment, channel), and 7 filter templates (active subscribers, postpaid/prepaid, B2B) — before delegating to NL-to-SQL.

Inline suggestions

When you edit SQL, Python, or YAML in a Monaco editor (for example, Design Studio's SQL mode), the Copilot offers ghost-text completions ahead of the cursor.

Suggestions are tuned to the DataFlow domain — Teradata SQL dialects, SAP HANA schemas, and Polkomtel-specific table names such as td_dwh_mona.STG_BIURO_SPRZEDAZY_PLK.

Proactive insights

The Copilot runs background analysis over your pipelines, quality metrics, and platform usage, and surfaces actionable recommendations as insight cards. Unread insights raise the red count badge on the sparkle trigger.

The insights endpoint, /insights, is cached for 300 seconds because the underlying model call is slow (~50s) and the dashboard polls roughly every minute. If the model is unavailable, a static fallback set of insights is shown.

Error diagnosis

When you navigate to a failed pipeline run in Monitor Center or Design Studio, the Copilot automatically activates the Diagnosis tab and opens the sidebar if it was collapsed.

The diagnosis panel presents a structured root-cause analysis:

• Failure header — pipeline name, run number, failure timestamp, and the failed node.
• Root cause — an AI-generated, plain-language explanation with a confidence percentage.
• Suggested fix — a recommended remedy with a risk level (low / medium / high).
• Diff preview — a DiffViewer showing the proposed YAML/SQL change, with an Apply Fix action.
• Related incidents — past runs that hit the same error, with how each was resolved.

The /diagnose endpoint produces a structured diagnosis across eight error categories, encoding Polkomtel-specific error knowledge (Teradata TPT/PERM space errors, SAP HANA CDC issues, JDBC lock-contention timeouts).

Agentic pipeline builder

For end-to-end pipeline construction, the Copilot exposes a Level-4 agentic builder — an autonomous agent that uses Claude tool_use to discover schemas, generate a pipeline, validate it, and create it, all in one conversation. The agent loop runs up to 15 iterations.

The agent has six tools:

The two destructive tools — create_pipeline and run_pipeline — are gated. When the agent reaches one, it pauses and surfaces a PendingAction; nothing is created or run until you explicitly approve it. The agent's tool backend executes against the real platform and catalog APIs, with domain-knowledge fallbacks (for example, canned Polkomtel CDR/billing schemas) if the catalog is down.

Root-cause analysis agent

Beyond per-run error diagnosis, the Copilot includes a dedicated RCA agent for incident investigation. RcaAgent.analyze() walks four real HTTP tool calls — recent alerts and runs from the monitor service, run logs, and the schema-change timeline from the metadata service — then synthesizes the evidence into a structured RcaResponse.

An RCA response contains a summary, an event timeline, two to four ranked hypotheses (each with a confidence score, supporting evidence, and a suggested action), and the catalog assets cited. It carries real telemetry — the model used, token count, latency, and tool-call count — and uses tolerant JSON parsing so it never invents content.

The RCA agent is reached via /rca/analyze.

How to use it — click-paths

Ask a question

1. Click the floating sparkle trigger at the top-right of any screen.
2. The Copilot sidebar slides in on the Chat tab.
3. Type your question into the input — e.g. "Which tables hold daily CDR aggregates?" — and press Enter.
4. The Copilot routes the intent, runs RAG enrichment against the catalog, and streams the answer token-by-token into an AI bubble.
5. If the answer embeds an [ACTION] block, click the inline action button (for example Open table or Yes, create) to act on it.

Behind the scenes: POST /chat (or GET /chat/stream for SSE token streaming) via api/copilot.ts, with RAG context from RAGService and a confidence score returned in the X-Copilot-Confidence header.

Generate a pipeline from a prompt

1. Open the Copilot and click the Build Pipeline quick action (or type a build request directly).
2. Describe the pipeline in plain language — e.g. "Daily sales from SAP HANA, filtered to active records, into my workspace."
3. The Copilot replies with a MiniPipelineDiagram (source → transform → target) and a plain-language plan.
4. Refine it conversationally if needed — e.g. "Add aggregation by region" — and the diagram updates.
5. Click Yes, create. The pipeline is scaffolded and opens in the Design Studio, pre-populated.

Behind the scenes: POST /generate-pipeline turns the prompt into DataFlow YAML; the response's PipelineNode/PipelineEdge arrays drive the diagram. For full autonomous construction, the agentic builder (/agent/chat) discovers schemas and validates the pipeline before pausing at the create_pipeline approval gate.

Diagnose a failed run

1. From Monitor Center, open a failed run at /monitor/runs/{runId}.
2. The Copilot auto-opens with the Diagnosis tab active and the context pill showing the failed run.
3. Read the Root Cause section and its confidence percentage.
4. Review the Suggested Fix and its risk level; inspect the Diff Preview of the proposed YAML change.
5. Click Apply Fix to stage the change, or scan Related Incidents to see how the same error was resolved before.
6. For a deeper, multi-source investigation, trigger the RCA agent to walk alerts, logs, and the schema-change timeline.

Behind the scenes: POST /diagnose for the structured diagnosis, POST /rca/analyze for the multi-tool RCA agent.

Tips — how to get great answers

The Copilot is good, but it is far better when you give it good prompts. A few habits make a big difference:

• Be specific with names. "How many rows are in crm.DIM_CUSTOMER?" beats "how big is the customer table?" — name the exact schema and table.
• Name the dialect when you want SQL. "Write this query in Teradata syntax" avoids the Copilot guessing the wrong database flavour.
• Describe the goal, not the steps. For pipeline building, say what you want ("daily sales from SAP HANA, no test rows, into my workspace") and let the Copilot work out the nodes.
• Refine in conversation. You do not have to get the prompt perfect first time. Send a follow-up — "now add aggregation by region" — and the Copilot adjusts.
• Paste errors directly. When something fails, paste the whole error message into the chat. You do not even need to read it first; the Copilot will.
• Enrich the catalog for better results. The more column descriptions and glossary terms exist, the sharper the Copilot's SQL and pipeline suggestions become.
• Always review before you run. Treat AI-generated SQL and pipelines as a strong first draft. Check connector names and filter conditions, and use the SQL Explain view, before running anything on production.

Common questions

Is this a real AI, or pre-written answers? It is a real AI model (Anthropic Claude) making live calls. Answers are generated fresh from your actual catalog and pipeline context — they are not canned.

Can I write to it in Polish? Yes. The Copilot understands Polish-language input, including for natural-language-to-SQL requests.

Will it ever change my data or pipelines without asking? No. The Copilot can plan, draft, and validate freely, but any action that creates or runs something — creating a pipeline, running it — pauses at an approval gate. Nothing happens until you explicitly click to approve it.

What if the AI gets the SQL or pipeline wrong? Treat its output as a draft. Generated SQL carries a confidence score and warnings (it flags risky things like DELETE without a WHERE clause). Review it, use Explain to see what the query will do, and click Regenerate or refine your prompt if it is off.

What happens if the AI service is unavailable? The Copilot degrades honestly. If the catalog search is down it falls back to a non-RAG chat; if the model itself is unreachable it tells you so, rather than inventing an answer.

How do I open the Copilot? Click the floating sparkle button at the top-right of any screen, or press Ctrl+/ from anywhere. On a failed pipeline run it opens itself, straight to the Diagnosis tab.

What is the difference between "Insights" and "Diagnosis"? Insights are proactive — the Copilot surfaces them on its own as tips (cheaper, faster, idle clusters). Diagnosis is reactive — it appears when a specific pipeline run has failed and explains that one failure.

Does the Copilot remember our conversation? Yes, within a session — it keeps recent conversation turns so follow-up questions make sense. Each session is private to your account.

Behind the scenes — API and architecture

The Copilot is a Python 3.12 / FastAPI microservice (backend/ai-services/copilot/, port 8090). Its router set is mounted under three prefixes — /api/v1/ai/**, /api/v1/copilot/**, and /api/copilot/** — so the frontend can reach it however the gateway addresses it.

Key frontend modules: api/copilot.ts (chat + SSE streaming, suggest, generate-SQL, diagnose), api/textToSql.ts, api/catalogAsk.ts, api/aiProvider.ts, api/rca.ts. The sidebar UI lives in src/components/ai-copilot/ (around 16 components including AICopilotSidebar, ChatMessage, MiniPipelineDiagram, DiffViewer, ErrorDiagnosis, and InsightCard).

Every AI output across the Copilot — chat, SQL, generated pipelines, schema matches, RCA hypotheses — carries a 0–1 confidence score, so you always know how much the model trusts its own answer.