Breadth Analytics

The open lakehouse you can run on your own hardware.

Lattice is an open-source data platform from Breadth Analytics. It aims to cover the everyday analytics work teams buy commercial lakehouse platforms for, using only open components, on servers you control.

Follow Lattice on GitHub → Try the lab
  • Open source
  • On-prem first
  • Lab to enterprise
  • No lock-in

A complete data platform, assembled from open parts.

Lattice is a lakehouse framework that brings storage, cataloging, ingestion, a semantic layer, BI dashboards, machine learning, AI agents, monitoring and a management console together in one stack today, with optional single sign-on through your own identity provider. Notebooks are next. Your data lives once, in versioned Delta Lake tables on S3-compatible storage you control. Spark writes it, DuckDB reads it, and Unity Catalog keeps every engine pointed at the current version by name.

Every piece is a proven open-source component. There is no proprietary runtime underneath, and anything Breadth Analytics builds on top, such as the management console, will be open-sourced too.

Open parts, one stack.

Each row is a layer, from the tools people use down to storage. Orchestration, secrets, monitoring and the edge proxy run across every layer. Solid parts run in the lab today; dashed parts are planned.

Experience
BI dashboards (Superset)Agent chat (Open WebUI)Lattice ConsoleNotebooks (planned)
Semantic
Semantic layer (Cube)Agent tool server (MCP)
Compute
Distributed processing (Spark)Embedded SQL engine (DuckDB)ML tracking & registry (MLflow)LLM gateway (LiteLLM)
Catalog
Data catalog (Unity Catalog)Open catalog API (Iceberg REST)
Table format
Open table format (Delta Lake UniForm)
Storage
Object storage (RustFS)Metadata database (PostgreSQL)
Orchestration
Workflow scheduler (Airflow)Data validation (Great Expectations)
Secrets & identity
Secrets vault (Vault)Single sign-on (any OIDC provider, optional)Sign-in gate (oauth2-proxy)
Monitoring
Metrics (Prometheus)Dashboards (Grafana)
Edge
TLS reverse proxy (Caddy)

Try the lab on one server.

The lab is a single-host Docker Compose stack. Configure your secrets and DNS once, then one command brings the stack up from an empty server and a second proves it end to end.

Recommended lab environment

Recommended Host
8+ cores, 32 GB RAM, 200 GB SSD
Software
Docker Engine with Compose v2, make, git, python3
Network
An isolated lab network, with wildcard DNS for your lab domain
lab server · as root 
git clone https://github.com/BreadthAnalytics/Lattice.git /srv/lattice/repo
cd /srv/lattice/repo/deploy
cp lab.env.example lab.env  # set LAB_DOMAIN
cp secrets.local.example /srv/lattice/secrets.local  # replace every CHANGE_ME
make bootstrap && make smoke
make sample-data  # optional: TPC-H sample data

What you get

  • vault.<lab-domain> Vault, with every service’s secrets seeded
  • airflow.<lab-domain> Airflow, running the ingestion, training and scoring DAGs on Spark
  • catalog-ui.<lab-domain> Unity Catalog, over RustFS object storage
  • superset.<lab-domain> Superset dashboards on the Cube semantic layer
  • chat.<lab-domain> A chat with the data-aware agent
  • mlflow.<lab-domain> MLflow experiments and model registry
  • console.<lab-domain> The Lattice Console: stack health, and admin for Vault, Airflow and Unity Catalog
  • grafana.<lab-domain> Grafana dashboards for every service and pipeline
  • mcp.<lab-domain> The semantic layer’s MCP server, for Claude Code and other MCP clients
  • make sample-model Once the sample data has loaded: trains, promotes and scores an example classifier

Isolated networks by default

Out of the box, Unity Catalog and MLflow have no login, so anyone who can reach the lab server can change the catalog and the model registry. Keep the lab on an isolated network, or turn on single sign-on with your own OIDC provider to put them behind a login.

Full setup guide: generating secrets, DNS, certificates and backups →

Platform recommendations

Virtualized host: LXC or VM will work well.

A Proxmox unprivileged LXC is recommended.

What’s in the stack.

Runs today

  • Versioned lakehouse storage

    Delta Lake tables with UniForm on S3-compatible storage. Every change is versioned, and the same table reads as Delta or Iceberg.

  • One catalog for every engine

    Unity Catalog tracks every table by name and hands each engine short-lived, table-scoped credentials.

  • Orchestrated ingestion

    Airflow picks up new files every five minutes, validates them with Great Expectations and merges them into Delta tables, keeping the newest row for each key. Files that fail validation are quarantined, not loaded.

  • Metrics defined once

    A Cube semantic layer defines measures as code, so Superset dashboards, the AI agent and any Postgres-compatible SQL client all get the same numbers.

  • Machine learning built in

    Train scikit-learn classifiers from Airflow, track every run in MLflow, promote a version when it’s ready, and score new and changed records every hour, for jobs like ticket routing and priority.

  • Data-aware AI agents

    Ask questions in plain language, in a browser chat, from the command line, or from Claude Code over MCP. The agent can only read governed metrics, every answer ends with its sources, and it runs on Claude or a local model.

  • Lattice Console

    One web console for the stack’s health and everyday admin: Vault secrets and policies, Airflow DAG runs, and Unity Catalog’s catalogs, schemas and tables. What each person can do follows their group, and every action lands in its own audit log.

  • Monitoring built in

    Prometheus watches Vault, Airflow, Spark, RustFS, PostgreSQL and the proxy, and Grafana dashboards show service health, failed tasks, pipeline run times and storage at a glance.

  • Single sign-on, your provider

    Optional sign-in through the OpenID Connect provider you already run, such as Authentik, Keycloak, Entra ID or Okta. Its groups become each tool’s roles, and tools without their own login sit behind a sign-in gate.

  • Secrets in one vault

    Vault holds every credential and setting, and each service reads only its own.

  • One-command lab

    Go from an empty server to a healthy stack with one command, then redeploy it automatically on every push to main.

Planned, coming next

  • Notebooks

    Standalone and scheduled notebooks with PySpark, scikit-learn and MLflow, working on the same governed tables as everything else.

What we’re building toward.

  1. 01

    Match the commercial platforms where it counts.

    Cover the large majority of everyday analytics work that teams buy commercial lakehouse platforms for, from ingestion and reporting to notebooks and ML.

  2. 02

    Private cloud first.

    Run the whole stack on your own servers by default. Swap in hosted services, such as a cloud LLM, only where you choose to.

  3. 03

    Start small, scale up.

    Begin with a single lab server brought up by one command, then grow to an enterprise deployment with a Spark cluster and full orchestration across business domains.

  4. 04

    No lock-in.

    Open table formats and open components mean your data stays readable by any engine, with or without Lattice.

  5. 05

    One pane of glass.

    Close the biggest gap with commercial platforms: a single console to monitor, administer and log in to the whole stack. The Lattice Console and single sign-on are the start, with more components joining over time.

Build it with us.

Lattice is in active development, and every design decision is written down in the open. Read the specs, follow the build, and try the lab stack on your own server. If your team wants an open lakehouse without the platform bill, we’d like to hear from you.

Follow Lattice on GitHub → Open an issue on GitHub
License
Apache-2.0
Design specs
8 written, in docs/specs
Status
Most of the stack runs in the lab today; notebooks are next