Pipelines — AI-Driven FHIR Data Ingestion & Mapping

Healthcare data doesn't always start in FHIR format. Hospitals, labs, and clinics often have legacy databases, CSV exports, or custom EMRs that store vital information — but in non-standard schemas.

ByteEngine's Pipelines solve this problem by using AI to automatically map legacy data to FHIR and expose it as a FHIR-compliant API — instantly interoperable with the rest of your digital health ecosystem.

What Are Pipelines?

A Pipeline is a smart data bridge that:

• Understands your legacy schema
• Uses AI to map it to the correct FHIR resources
• Exposes a FHIR API endpoint
• Handles bi-directional ingestion (import & export)

In short, pipelines turn anything — a database, CSV, HL7 message, or even a Google Sheet — into a live FHIR server endpoint.

Real-World Example

Imagine you're a hospital with an on-premise PostgreSQL database called patients_db containing patient info and lab results in custom tables:

CREATE TABLE patients (
  id SERIAL PRIMARY KEY,
  name TEXT,
  dob DATE,
  gender TEXT
);

CREATE TABLE labs (
  id SERIAL PRIMARY KEY,
  patient_id INT,
  test_name TEXT,
  result_value TEXT,
  result_date DATE
);

Instead of manually mapping these to FHIR's Patient and Observation resources, ByteEngine's AI engine analyzes your schema and auto-generates a FHIR mapping configuration.

Result: you get an instant API like this:

GET https://api.engine.boolbyte.com/api/pipeline/patient-data/Patient
GET https://api.engine.boolbyte.com/api/pipeline/patient-data/Observation

Now, external FHIR systems or AI workers can access your data as if it were native FHIR — securely, compliantly, and in real time.

Creating a Pipeline

You can create pipelines using:

• The ByteEngine Console (UI)
• The REST API
• Or YAML configuration (for automation and version control)

Using the ByteEngine Console

1. Go to Pipelines → New Pipeline
2. Choose Data Source Type:
   • Database (PostgreSQL, MySQL, MSSQL)
   • CSV/Excel Upload
   • API Endpoint
3. Provide connection details (e.g., DB URI, credentials, or file)
4. Select "AI Mapping Mode"

The AI automatically:

• Reads your schema
• Suggests FHIR mappings
• Generates a test mapping preview

Once confirmed, your pipeline is live and exposes FHIR endpoints instantly.

Using the REST API

curl -X POST https://api.engine.boolbyte.com/api/pipelines \
  -H "Authorization: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "patient-data-pipeline",
    "source": {
      "type": "postgresql",
      "uri": "postgresql://user:pass@host:5432/patients_db"
    },
    "mode": "ai_mapping",
    "fhirVersion": "R4"
  }'

Response:

{
  "success": true,
  "data": {
    "id": "pipe_78342",
    "name": "patient-data-pipeline",
    "status": "mapping",
    "message": "AI is analyzing schema and generating FHIR mappings.",
    "createdAt": "2024-01-15T10:00:00.000Z"
  }
}

Once complete, the mapping is visible in your dashboard as a YAML configuration.

Using JavaScript SDK

import { EngineClient } from '@boolbyte/engine';

const client = new EngineClient({ apiKey: 'YOUR_API_KEY' });

// Create a pipeline
const pipeline = await client.pipeline.createPipeline({
  name: 'patient-data-pipeline',
  source: {
    type: 'postgresql',
    uri: 'postgresql://user:pass@host:5432/patients_db'
  },
  mode: 'ai_mapping',
  fhirVersion: 'R4'
});

console.log('Pipeline created:', pipeline.data.id);

Example: Generated Mapping (YAML)

Here's what ByteEngine's AI might generate for your sample database:

pipeline:
  id: patient-data-pipeline
  source:
    type: postgresql
    uri: postgresql://user:pass@host:5432/patients_db

  mappings:
    - table: patients
      resource: Patient
      fields:
        id: id
        name: name
        birthDate: dob
        gender: gender

    - table: labs
      resource: Observation
      fields:
        id: id
        subject.reference: "Patient/{{patient_id}}"
        code.text: test_name
        valueString: result_value
        effectiveDateTime: result_date

You can tweak, validate, and version this mapping via the UI or API.

Using Your Pipeline Endpoint

Once deployed, your pipeline exposes live FHIR endpoints like:

GET /api/pipelines/patient-data-pipeline/Patient
GET /api/pipelines/patient-data-pipeline/Observation
POST /api/pipelines/patient-data-pipeline/Patient

FHIR Query Example:

curl -X GET "https://api.engine.boolbyte.com/api/pipelines/patient-data-pipeline/Observation?patient=Patient/123" \
  -H "Authorization: Bearer <ACCESS_TOKEN>"

Returns:

{
  "resourceType": "Bundle",
  "entry": [
    {
      "resource": {
        "resourceType": "Observation",
        "subject": { "reference": "Patient/123" },
        "code": { "text": "Blood Pressure" },
        "valueString": "120/80",
        "effectiveDateTime": "2024-01-10"
      }
    }
  ]
}

Two-Way Data Flow

ByteEngine pipelines are bi-directional:

• Ingest Mode: Convert external data → FHIR
• Emit Mode: Convert FHIR data → external format

Example:

• Ingest new Observation resources from lab systems
• Emit standardized FHIR data back to your EMR or analytics tool

Compliance & Security

• All data passing through pipelines is encrypted (TLS 1.3)
• Data residency options ensure mapping happens in-region (EU, US, Global)
• No raw data leaves your environment if using a private connector
• Schema and mapping logs are anonymized for AI processing

Common Use Cases

Scenario	Description
Legacy EMR Integration	Connect your old EMR database and auto-convert data to FHIR
Data Migration	Move healthcare data between systems or clouds in standardized FHIR
Analytics Enablement	Expose FHIR APIs to BI tools and dashboards
AI Training Data Prep	Convert clinical data into structured FHIR for model training
Interoperability Gateways	Bridge siloed systems (e.g., lab → hospital → payer) instantly

Example Architecture Diagram

                +----------------------------+
                |  Legacy System (DB, CSV)   |
                +------------+---------------+
                             |
                             | (AI Mapping)
                             v
                +----------------------------+
                |    ByteEngine Pipeline     |
                |  - FHIR Mapping            |
                |  - Validation              |
                |  - API Exposure            |
                +------------+---------------+
                             |
                             v
                +----------------------------+
                |      FHIR-Compatible App   |
                |  (EMR, AI Worker, Analytics)|
                +----------------------------+

Validation & Testing

Every pipeline comes with:

• FHIR validation reports (auto-run)
• Mapping test console
• Monitoring dashboard for request volumes and errors

Developers can run local validation via CLI:

byteengine pipelines validate patient-data-pipeline

Best Practices

Goal	Recommendation
Keep mappings consistent	Use AI-suggested templates and manually review before publishing
Secure connections	Always use SSL DB URIs or private connectors
Version control	Export mappings as YAML and commit to Git
Testing	Validate pipelines before enabling live ingestion
Governance	Assign pipeline access roles (read/write/admin)

Real-World Implementation Examples

Legacy EMR Integration

// Connect to legacy EMR database
const pipeline = await client.pipeline.createPipeline({
  name: 'legacy-emr-pipeline',
  source: {
    type: 'mysql',
    uri: 'mysql://emr_user:password@legacy-emr-db:3306/patient_records'
  },
  mode: 'ai_mapping',
  fhirVersion: 'R4'
});

// AI automatically maps tables to FHIR resources
// Result: Instant FHIR API for legacy EMR data

CSV Data Migration

// Upload CSV and create pipeline
const pipeline = await client.pipeline.createPipeline({
  name: 'lab-results-migration',
  source: {
    type: 'csv',
    file: 'lab_results_export.csv'
  },
  mode: 'ai_mapping',
  fhirVersion: 'R4'
});

// AI analyzes CSV structure and maps to FHIR Observation resources
// Result: FHIR-compliant API for lab data

Next Steps

Next Section → Workflows — Visual Healthcare Automation Builder

Learn how to visually orchestrate pipelines, AI Workers, and data flows into automated healthcare workflows.

• Pipelines API Reference - Complete API documentation
• Health Data Store Guide - Set up your FHIR infrastructure
• AI Workers Guide - Connect AI Workers to your pipeline data
• Quick Start Guide - Build your first healthcare application