Long-running, fault-tolerant SQL functions for teams that already keep their state in Postgres and want to stop stitching together cron jobs, workers, queues, and status tables to make background work reliable. Define the workflow in SQL, let pg_durable checkpoint each step, and resume after crashes, restarts, or failed steps.
Durable execution is now a standard industry pattern, and pg_durable brings it inside Postgres with no extra service infrastructure required. Part of our mission to bring compute close to data.
Try pg_durable now in Azure HorizonDB, Microsoft’s new PostgreSQL cloud service engineered for performance and built with pg_durable inside
Try pg_durable now in Azure HorizonDB, Microsoft’s new PostgreSQL cloud service engineered for performance and built with pg_durable inside
Is this for me?
Who it’s for
Backend and data engineers who want workflows to live next to the data they touch.
DBAs and SREs automating runbooks that must survive restarts and be auditable in SQL.
Teams building data or AI pipelines that need durable execution per row, document, or batch.
The core idea
A pg_durable function is a graph of SQL steps that PostgreSQL executes and checkpoints as it goes. If the database crashes, restarts, or a step fails, execution resumes from the last durable checkpoint instead of making you reconstruct state by hand.
Workloads this is useful for
Vector embedding pipelines: chunk, call an embedding API, and upsert into pgvector.
Ingest pipelines: stage, deduplicate, transform, and publish large batches.
Scheduled maintenance: detect bloat, notify, wait for approval, then run the next action.
Fan-out aggregation: run independent queries in parallel, then join the results.
External API workflows: enrichment, classification, and webhook-style calls from SQL.
What you’re probably doing today instead
pg_cron plus a jobs table, status columns, retry counters, and a polling worker.
An external orchestrator such as Airflow, Temporal, Step Functions, or Argo calling back into Postgres.
A queue plus workers plus a separate state table to coordinate retries and partial completion.
A plpgsql procedure that works until a crash or long-running transaction forces you to start over.
Pain points it addresses
A restart in the middle of a long job means rerunning work that already succeeded.
One failed row or one failed API call turns into manual cleanup and uncertain replay.
Long transactions hold locks, grow WAL, and make batch jobs fragile at larger scale.
Parallel work in the app tier creates more places for partial-failure bugs and drift.
The workflow logic ends up spread across SQL, workers, queues, dashboards, and status tables.
What changes in your architecture
The workflow definition moves into SQL and starts with df.start(…).
Retry state, progress tracking, and checkpointing move into Postgres instead of bespoke app code.
Some app-tier workers, queue consumers, or scheduler glue can disappear entirely.
Operational visibility comes from Postgres tables such as df.instances, using the same auth and backup model as your data.
When not to use it
The job is already a single INSERT … SELECT or one ordinary SQL statement.
You need sub-millisecond synchronous request handling rather than durable background execution.
You cannot install extensions or run a background worker in your Postgres environment.
The workflow mostly lives outside Postgres and spans many heterogeneous systems.
You need arbitrary application logic that does not map cleanly to SQL steps, branching, loops, or HTTP calls.
How it works
Define a workflow in SQL using composable operators such as ~> and |=>.
Start it with df.start() and get back an instance ID.
Let the runtime execute each step durably with checkpointing between steps.
Query status and results from PostgreSQL while the workflow runs or after it completes.
Limitations
The model is intentionally SQL-shaped. If a step needs arbitrary code, a non-HTTP SDK, or rich in-memory control flow, you may need to wrap that logic in a SQL function, expose it behind an HTTP endpoint for df.http(), or use a general-purpose orchestrator for that part of the system.
Features
Durable — Function state persists to PostgreSQL. Survives crashes, restarts, and failovers.
SQL-native — Define functions in SQL using composable operators.
Database-aware — First-class primitives for scheduling, conditions, and parallel execution.
Zero infrastructure — Runs as a PostgreSQL extension. No Redis, no Temporal, no external services.
Quick Example
– A durable function that processes data in steps SELECT df.start( ‘SELECT id FROM documents WHERE processed = false LIMIT 100’ |=> ‘batch’ ~> ‘UPDATE documents SET processed = true WHERE id = ANY($batch)’ );
Packages
Tagged releases publish Debian packages for PostgreSQL 17 and 18 on amd64 from the GitHub release assets. Packages are named pg-durable-postgresql-<PG major>_<pg_durable version>-1_<arch>.deb and install the extension library, control file, and SQL upgrade files into the matching PostgreSQL installation directories.
After installing a package, add pg_durable to shared_preload_libraries, restart PostgreSQL, and create the extension in the configured pg_durable database:
CREATE EXTENSION pg_durable;
The default pg_durable database is postgres; see User Guide for background worker configuration and privilege setup.
Release assets also include source archives for building from source.
Development Installation
Prerequisites
PostgreSQL 17 or 18
Rust (nightly)
cargo-pgrx 0.16.1
GitHub Codespace
The main branch prebuild installs PostgreSQL 17, builds pg_durable, and prepares a local cluster under ~/.pgrx with the extension ready. PostgreSQL is not left running, so start it when you begin working.
# Start PostgreSQL ./scripts/pg-start.sh
# Connect ~/.pgrx/17.*/pgrx-install/bin/psql -h localhost -p 28817 -d postgres
On a branch without a ready prebuild, run pg-start.sh — it will build and install the extension on first run (expect a few minutes):
./scripts/pg-start.sh
Other environments
Local and Dev Container
A VS Code Dev Container (.devcontainer/) provides Rust, cargo-pgrx, and PostgreSQL 17 pre-installed. For a bare local machine, install the toolchain first by following the steps in .devcontainer/onCreateCommand.sh.
# Build, initialize PostgreSQL, and install the extension # This takes a while - go do something else ./scripts/pg-start.sh
# Connect to the local pgrx PostgreSQL instance ~/.pgrx/17.*/pgrx-install/bin/psql -h localhost -p 28817 -d postgres
pg-start.sh bootstraps new local data directories with a postgres superuser and also creates a matching superuser role for the current OS user, so default local psql usage continues to work. Use -U postgres if you want to force the canonical bootstrap role explicitly.
Docker
# Build and test ./scripts/test-e2e-docker.sh –rebuild
# Optional: Deploy to ACR (for custom PG17 image with pg_durable baked-in) ./scripts/deploy-acr.sh
Multi-User Setup
CREATE EXTENSION pg_durable does not grant any privileges to PUBLIC. After installing the extension, the admin must explicitly grant access to application roles. Row-level security (RLS) ensures each user can only see and manage their own durable function instances and nodes.
Grant privileges to an application role:
– Grant to specific roles after CREATE EXTENSION SELECT df.grant_usage(‘app_role’);
Alternatively, create an indirection role and grant membership to application roles:
– Create a shared role for pg_durable access CREATE ROLE pg_durable_user NOLOGIN; SELECT df.grant_usage(‘pg_durable_user’);
– Grant membership to application roles GRANT pg_durable_user TO app_backend, etl_service;
See the User Guide — Privilege Grants section for the full list of individual grants, revoking access, and hardening upgraded installs.
See the User Guide — Privilege Grants section for the full list of individual grants, revoking access, and hardening upgraded installs.
Note: GRANT EXECUTE ON ALL FUNCTIONS only applies to functions that exist when the grant runs. After upgrading pg_durable with ALTER EXTENSION pg_durable UPDATE, re-run df.grant_usage(‘role’) (or re-issue the manual grants) so new functions are accessible.
Note: GRANT EXECUTE ON ALL FUNCTIONS only applies to functions that exist when the grant runs. After upgrading pg_durable with ALTER EXTENSION pg_durable UPDATE, re-run df.grant_usage(‘role’) (or re-issue the manual grants) so new functions are accessible.
Key points:
The background worker role (pg_durable.worker_role GUC, default: azuresu) must be a superuser — it bypasses RLS to manage all users’ instances
Users get SELECT + INSERT on df.instances / df.nodes, column-level UPDATE (status, updated_at) on instances for df.cancel()
Identity column (submitted_by) cannot be modified by users
df.vars uses per-user scoping — each user has their own variable namespace via an owner column and RLS. Superusers bypass RLS but DSL functions still scope to the calling user via explicit filters. Avoid storing secrets in plain text
Continuous Integration
All pull requests must pass the following checks before merging:
Format Check — cargo fmt –check
Clippy & Tests — cargo clippy, unit tests (cargo pgrx test pg17), pg_regress tests, and E2E tests
The CI workflow is defined in .github/workflows/ci.yml. It uses pgrx to download and manage PostgreSQL.