Sync

Setting up Sync takes three steps:

1. Add a source: the connection to your database.
2. Create a synced index from a definition file: which table, which SQL renders a document, which engine it lands on.
3. Run the source's setup SQL on your database, then test the source. Sync backfills the index and then streams changes.

The dashboard's Sync page adds, tests, pauses and deletes sources, and pauses, rebuilds and deletes synced indexes. Creating a synced index takes the CLI or the API.

A source is one Postgres database. Two kinds exist:

The connection URL must point at a direct Postgres endpoint. Sync refuses connection poolers: hosts containing pooler or pgbouncer, and ports 6543, 6432 and 25061, answer 422 pooler_refused. Replication needs a real session.

TLS is required. tlsMode is verify-full (the default) or require. With verify-full you can pass a CA bundle in tlsCaPem.

{
  "name": "shop",                                     // lowercase letters, digits, _ or -
  "url": "postgres://hb_sync@db.example.com:5432/shop",
  "tlsMode": "verify-full"                            // optional
}

The user name in the URL is the role Sync connects as. The setup SQL creates that role, so pick a name that does not exist on your server yet. An organization can have up to 10 sources.

A synced index is defined by one JSON document. Keep it in your repository as <name>.sync.json: the CLI and the API take the file as it is.

{
  "name": "products",
  "instance": "<instance id>",
  "source": "<source id>",
  "root": { "table": "public.products", "key": "id" },
  "render": "SELECT p.id, p.title, p.price_cents, b.name AS brand, (p.deleted_at IS NULL) AS _searchable FROM products p LEFT JOIN brands b ON b.id = p.brand_id WHERE p.id = ANY($1::bigint[])",
  "dependencies": [
    { "table": "public.products", "map": { "self": true } },
    { "table": "public.brands", "map": { "sql": "SELECT id FROM products WHERE brand_id = ANY($1::bigint[])" } }
  ],
  "engine": {
    "settings": { "filterableAttributes": ["brand"] }
  }
}

The render

• Cast $1 to your key type, such as $1::bigint[] or $1::uuid[].
• Return the root key column under its own name. Its text is the document id.
• No row for an id deletes that document. Two rows for one id is an error.
• An optional boolean _searchable column: anything but true deletes the document. It is not stored.
• Columns starting with _hb_ are reserved.

Dependency mappings

A dependency can set debounceMs (0 to 3,600,000) to hold its changes before they render. Use it for busy counter tables.

On a Visor engine, engine.settings takes Visor's index settings, such as fields, instant and text, not Meilisearch's keys: filterable and sortable fields go in fields.filterable and fields.sortable, and validate names any key Visor does not take. Sync sets the primary key from root.key and makes it filterable and sortable, since it reads the index by its key. Visor has no index aliases, so Headband routes the index name to each new build itself. A rebuild serves its new content within about 20 seconds of finishing, and the index's engineIndexUid changes with it.

Check a file before you create the index: send {"definition": <file>} to POST /v1/sync/indexes/validate. An invalid definition answers 200 with valid: false and a list of errors.

Sync never asks for your owner credentials. Instead, each source has setup SQL that you run as the owner of the tables. It runs in one transaction and is safe to run again.

• Creates the Sync role with LOGIN and REPLICATION, and grants it SELECT on the tables.
• Creates a publication and sets it to exactly the tables your synced indexes read.
• Stops, changing nothing, if a table has no usable replica identity or the role name belongs to a role Sync did not create.

psql "<owner connection URL>" -v ON_ERROR_STOP=1 -v sync_password='<password>' -f setup.sql

Fetch and run it again whenever you add a synced index that reads a new table. Then test the source with POST /v1/sync/sources/:id/test. The first passing test turns the source active and starts the backfill.

Each action is a button on the synced index's dashboard page and an API call.

Changing a definition (PATCH /v1/sync/indexes/:id with definition) saves it, and the change takes effect at the next rebuild. Send rebuild: true in the same body to start one.

A drop answers 202 while the engine still holds the index. Send the same request again to finish it.

Deleting a source (DELETE /v1/sync/sources/:id) drops Sync's replication slots and returns dropPublicationSql. Run it on your database to remove the publication.

Every /v1/sync route needs an org-wide admin key. A search key answers 403 permission_denied and a project-scoped key answers 403 org_key_required.

Each key can make 1,000 reads and 60 writes per minute.

The headband sync source and headband sync index commands need CLI 0.3.0 or later. See CLI for the full list.

headband sync source add shop --db-url "postgres://hb_sync@db.example.com:5432/shop"
headband sync index create -f products.sync.json
headband sync source setup <source id> --owner-url "<owner connection URL>"
headband sync source test <source id>
headband sync status

sync source setup fetches the setup SQL and runs it for you with the owner URL you give it.

Self-hosted sync needs no early access. Your app records each change in an outbox table, and a worker you run writes the changed documents to Headband. It uses @headband/sync 0.1.1, @headband/drizzle 0.1.1 (for Drizzle ORM on Postgres) and CLI 0.6.1 or later.

npm install @headband/drizzle @headband/sync drizzle-orm pg
npm install --save-dev @headband/cli

The CLI reads one file, headband.config.ts, in the directory you run it from. It must export the config as its default export and the adapter by name:

import { defineConfig, rel } from "@headband/sync";
import { drizzleAdapter } from "@headband/drizzle";
import { db } from "./db"; // your drizzle(...) instance
import * as schema from "./schema"; // your pgTable(...) exports

const config = defineConfig({
  url: "https://headband.dev",
  projectId: "<project id>", // without it, new indexes go to your Default project
  indexes: [
    {
      name: "products",
      source: "products",
      primaryKey: "id",
      shouldIndex: (row) => row.active === true,
      fields: { title: true, price: true, brandName: rel("brands", "name") },
      settings: { searchableAttributes: ["title", "brandName"], filterableAttributes: ["price"] },
    },
  ],
  sync: { mode: "outbox" },
});

export default config;
export const adapter = drizzleAdapter({ db, schema, config });

Then create the indexes, the outbox table, and run the worker:

npx headband login
npx headband schema generate      # writes headband/migrations/0001_migration.json
npx headband schema apply --yes   # creates each index and backfills every row
npx headband sync init --apply    # creates the outbox table in DATABASE_URL
DATABASE_URL=postgres://... npx headband sync worker --loop
npx headband sync status          # outbox depth and dead letters

In your app, write through withHeadbandSync(db, { adapter, mode: "outbox" }) from @headband/drizzle instead of db. Each insert, update and delete then also writes an outbox row for the worker.

On Typesense, a number primary key (a serial id) is stored as its string, since Typesense ids are strings. schema apply sends an index's settings before the backfill, and sends them again after it when a sortable field has no type yet. Typesense takes searchableAttributes, filterableAttributes, sortableAttributes and synonyms. If an engine refuses a setting, apply stops and says so. The index is kept, and a re-run after you change the config picks it up.

Commit headband/migrations/. The .applied.json file in it is an optional local cache: applied migrations are read from the server. In production, run the worker with HEADBAND_API_KEY set to an admin key of the project in projectId. The CLI page lists every sync and schema command.