Backups & Restore

GeoLens ships an automated backup container that runs on a configurable cron schedule, retains daily and weekly snapshots locally, and optionally replicates to S3-compatible storage. There is no /admin/backups UI page. Backups are operated entirely from the Docker Compose CLI and via environment variables.

Replace https://geolens.example.com with your GeoLens instance’s URL in every example below.

Automated backups are on by default

The backup container starts with the default Docker Compose stack. A plain docker compose up -d runs scheduled pg_dump backups, keeps the newest backup from each of the last 7 days and each of the last 4 ISO weeks, and optionally uploads to S3. The service uses the same database credentials as the API, so no additional configuration is required for the local-only path. Prebuilt installs pull the published multi-arch ghcr.io/geolens-io/geolens-backup image at the pinned release version; source installs build the same stage from the root Dockerfile.

To verify the backup container is running:

docker compose ps backup

The healthcheck measures freshness, not liveness: the container is healthy only while the last fully successful backup cycle (dump, verification, staging archive, and S3 upload when enabled) is younger than BACKUP_MAX_AGE_MINUTES (default 1560 minutes, i.e. 26 hours: the daily schedule plus slack). A container whose backups quietly stop succeeding turns unhealthy roughly one missed cycle later. The schedule fires once per day, so the default 1560 covers it; raise BACKUP_MAX_AGE_MINUTES only if you have moved backups to a slower external scheduler. On a fresh install the service shows starting until the initial on-startup backup succeeds.

Once a cycle has uploaded offsite, the healthcheck also requires /backups/.last-offsite-success to be younger than BACKUP_MAX_AGE_MINUTES, even if BACKUP_S3_ENABLED is later turned off. An override or .env edit that drops the offsite settings then shows as an unhealthy backup service instead of quietly going local-only, and the container logs a warning at start when it finds the marker with offsite upload off. If you turned offsite upload off on purpose, delete the marker (the service stays unhealthy until you do, and nothing else waits on it):

docker compose exec backup rm /backups/.last-offsite-success

The backup container restarts automatically; if it exits non-zero (e.g., disk full, connectivity issue), Docker restarts it and the next scheduled run resumes normally.

Configuration

Variable	Default	Purpose
BACKUP_SCHEDULE	0 2 * * *	Daily run time, written as M H * * * — a literal minute and hour with the remaining three fields *. Step (*/6), range, and list syntax, and any non-* day-of-month/month/day-of-week, are rejected: the container logs the error and exits at startup
BACKUP_RETENTION_DAILY	7	Days of daily backups to keep. The newest dump of each of this many most recent days that have a backup is kept, locally and in the S3 backups/daily/ prefix when BACKUP_S3_ENABLED=true. Before 1.25.0 this counted dumps
BACKUP_RETENTION_WEEKLY	4	ISO weeks of weekly (Sunday) backups to keep, one per week. Same scope as the daily value. Before 1.25.0 this counted dumps
BACKUP_S3_ENABLED	false	Upload backups to S3 in addition to local storage
S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_REGION	(none)	S3 credentials (shared with the storage provider)
S3_ADDRESSING_STYLE	auto	S3 addressing style (auto, path, virtual); use path for MinIO
BACKUP_MAX_AGE_MINUTES	1560	Healthcheck freshness threshold: the container reports unhealthy when the last fully successful backup cycle is older than this many minutes
BACKUP_MEM_LIMIT	512m	Container memory cap for the backup service (the nightly pg_dump + tar peaks around 300 MB). Applies to the prebuilt docker-compose.prod.yml stack only — the source-build docker-compose.yml sets no mem_limit on the backup service, so the variable is inert there and the container is uncapped

Set these in .env before starting the stack. Changes require a container restart:

docker compose restart backup

The restart takes a backup straight away, and that dump replaces any earlier dump from the same day (see Retention policy). Don’t restart the service while the database is damaged or being restored; follow Restoring from a backup instead.

The backup service reads the same S3_* credentials as the storage provider. There are no dedicated BACKUP_S3_* bucket/credential overrides. BACKUP_S3_ENABLED is the only backup-specific S3 knob; off-site upload reuses S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, and S3_REGION. To back up to a separate bucket, point those shared S3_* values at it. The uploader follows whatever scheme S3_ENDPOINT carries, so use an http:// endpoint for a plain-HTTP MinIO; S3_ALLOW_HTTP only affects the app’s own object-storage client, not the backup uploader. See Configuration Reference.

Tip

The schedule is compared against the container’s own wall clock, and the container inherits UTC because neither compose file sets TZ — so the default 0 2 * * * runs at 02:00 UTC daily. Adding TZ to the backup service’s environment: block does shift the run time, but converting to UTC explicitly is the safer choice: it keeps the schedule stable across DST transitions.

Backup destinations

Backups are written to:

• /backups/daily/<db>_<timestamp>.dump: every scheduled run
• /backups/weekly/<db>_<timestamp>.dump: Sundays only
• /backups/{daily,weekly}/staging-<timestamp>.tar.gz: the local upload_staging volume (rasters/COGs/source uploads), tarred alongside each dump with the same timestamp so a restore can recover a working instance rather than an empty-object catalog
• /backups/{daily,weekly}/globals-<timestamp>.sql: the cluster globals (pg_dumpall --globals-only) — roles, their passwords, and cluster-wide grants — written next to the dump with the same timestamp
• s3://<bucket>/backups/{daily,weekly}/...: the dump, the staging archive, and the globals dump, when BACKUP_S3_ENABLED=true

The <db> segment is the database name (default geolens). The <timestamp> segment is YYYYMMDD_HHMMSS in UTC. Dumps are PostgreSQL custom-format (pg_dump -Fc --no-owner --no-acl). Restore with pg_restore rather than psql.

Copy all three paired artifacts together

The three files sharing a timestamp are one backup set. Copying the .dump alone is the common mistake, and it is only survivable while the cluster you restore onto still has the roles.

A database-scoped pg_dump cannot recreate roles, so restoring onto a fresh cluster without replaying globals-<timestamp>.sql first fails with role ... does not exist. Retention treats the set as a unit: the staging-*.tar.gz and globals-*.sql files are pruned when the dump they belong with ages out, so a companion never outlives its dump.

globals-*.sql contains role password verifiers. It is written mode 0600, and every copy of it must keep that protection — use umask 077 when copying it out of the volume, and treat it with the same care as the dump itself.

Upgrading from before 1.7.0?

The globals-*.sql artifact arrived in 1.7.0. The backup daemon reads its entrypoint at container start, so an already-running daemon keeps the old cycle until it restarts: run docker compose restart backup (or any full stack restart) once after upgrading, then confirm a globals-*.sql file appears. The restart runs a cycle at once, and its dump replaces that day’s earlier one.

Note

The staging-*.tar.gz archive captures only the local upload_staging volume. Deployments that offload objects to external S3/MinIO must back those buckets up separately (via the bucket’s own lifecycle), since the backup service mounts upload_staging read-only and never reaches the external store.

Local backups live inside the backup_data Docker volume. Inspect them by execing into the backup container:

docker compose exec backup ls -lh /backups/daily/
docker compose exec backup ls -lh /backups/weekly/

To copy a specific backup off the host, take the whole set — all three files share one timestamp:

# umask 077: the globals file holds role password verifiers, so the copies
# must not be readable by other users on the host either.
(umask 077
 docker compose cp backup:/backups/daily/geolens_20260101_020000.dump ./
 docker compose cp backup:/backups/daily/staging-20260101_020000.tar.gz ./
 docker compose cp backup:/backups/daily/globals-20260101_020000.sql ./)

The staging archive is absent on a fresh install with no uploaded datasets; the globals dump is not optional, and a cycle that fails to produce it is reported as a failed cycle.

S3 signature compatibility

The backup container uploads with the bundled AWS CLI using AWS Signature Version 4, so modern AWS S3 buckets, MinIO, and Cloudflare R2 all work out of the box. For MinIO or R2 deployments that need path-style addressing, set S3_ADDRESSING_STYLE=path; for plain-HTTP local endpoints (e.g. a MinIO container on a private network), give S3_ENDPOINT an http:// scheme (S3_ALLOW_HTTP has no effect on the backup uploader).

Retention policy

The backup container enforces retention on every run. After producing a new backup:

1. Daily backups outside the newest BACKUP_RETENTION_DAILY days that have a backup are deleted from /backups/daily/. The window counts days, not dumps, so extra cycles on one day do not push older days out. A day keeps only its newest dump, though. Every start of the backup container runs a cycle, and its dump replaces any earlier dump from that day, whatever state the database is in at the time. The weekly directory behaves the same way within an ISO week.

2. Weekly backups outside the newest BACKUP_RETENTION_WEEKLY ISO weeks are deleted from /backups/weekly/. A weekly entry is a hard link to the daily dump, so it takes no extra space; a copy is made when the filesystem refuses links. New backup files are written with mode 0600.

   One dump per directory is exempt from those windows: the newest complete set — a dump with its paired globals-*.sql — is held back in addition to the retention count, so a directory can hold BACKUP_RETENTION_* + 1 dumps. This keeps a run of failed pg_dumpall cycles (each still producing a valid dump) from evicting the last set that restores onto a fresh cluster.

3. S3 retention follows the same day and week counts for the backups/daily/ and backups/weekly/ prefixes, and GeoLens deletes pruned dumps there itself. For longer retention, raise BACKUP_RETENTION_DAILY and BACKUP_RETENTION_WEEKLY, or copy complete sets (each dump with its globals-*.sql and, when staged objects are backed up, its staging-*.tar.gz) to a bucket or prefix GeoLens does not manage. Bucket lifecycle rules can move objects to colder storage, such as a 30-day Glacier transition, but they cannot keep a dump GeoLens prunes.

Manual deletion is safe: the container scans the directory on the next run and updates retention based on what’s present. Out-of-band file deletion does not corrupt the backup state.

Both retention values must be plain integers of at least 1. The container refuses to start otherwise (the log shows must be >= 1), because a retention of 0 would delete each backup the moment it is written.

Manual pg_dump (alternate path)

For one-off backups outside the scheduled cron, run pg_dump directly against the database container:

# Create a custom-format dump (preferred for pg_restore). Mirror the
# automated backup's flags so it restores cleanly via scripts/restore.sh
docker compose exec db pg_dump -U geolens -d geolens -Fc --no-owner --no-acl -f /tmp/geolens_backup.dump
docker compose cp db:/tmp/geolens_backup.dump ./geolens_backup.dump

For a plain SQL backup (useful for grep/inspection):

docker compose exec db pg_dump -U geolens -d geolens > geolens_backup.sql

For volume-level snapshots (database files plus WAL):

# Stop services first to ensure a consistent snapshot
docker compose down

# Backup the pgdata volume. Replace <project> with your Compose project
# name (see `docker volume ls`); the installer's default is geolens.
docker run --rm -v <project>_pgdata:/data -v $(pwd):/backup alpine \
  tar czf /backup/pgdata_backup.tar.gz -C /data .

# Restart
docker compose up -d

Volume snapshots are faster to restore than pg_restore for full-instance recovery but cannot be restored to a different PostgreSQL version. Use the pg_dump format for long-term archival and cross-version portability.

Restoring from a backup

The full disaster-recovery procedures are in the runbook

This page covers the routine case: restoring into the running bundled db container, on a cluster that still has its roles. The operator runbook that ships in the repository, RUNBOOK.md, carries the procedures that go beyond it:

• Section 2 — Restore, bundled Postgres mode: the fresh-cluster path, including replaying globals-*.sql before pg_restore and rebuilding ownership and grants afterward.
• Section 3 — Restore, managed / external Postgres mode: RDS, Cloud SQL, and friends, and why point-in-time recovery is a different mechanism rather than a finer setting — logical dumps cannot participate in WAL replay.
• Section 5 — Incident response, data loss: picking a dump, validating it before restoring, and the recovery point objective the default schedule gives you (up to 24 hours).
• Section 7 — Schema rollback: for when the problem is a migration rather than the data.

Stop the backup service before you restore

A backup-container start takes a dump at once, and a day keeps only its newest dump. Restart the service, or run a plain docker compose up -d, while the database is damaged or half restored, and that day’s good dump is replaced by a dump of the bad state. Before any restore or incident work:

1. Copy that day’s backup set out of the volume, as shown under Backup destinations, and the set you plan to restore if it is older.
2. Stop the service: docker compose stop backup.

restore.sh stops and restarts only api and worker, so the backup service stays stopped through it. If you bring the stack back up before the restored data is verified, use docker compose up -d --scale backup=0. Once restore validation passes, start the service again with docker compose start backup, or docker compose up -d backup if you scaled it to 0.

The canonical restore path is scripts/restore.sh. It validates the dump with pg_restore -f /dev/null before touching the database (reading every data block; a merely truncated dump still passes --list), reports any sibling globals-<timestamp>.sql and names the roles it defines that the target cluster is missing, pre-creates the required extensions and schemas, stops api and worker to prevent write conflicts, runs the restore, re-grants the geolens_reader role afterward (ownership and ACLs have to be reconciled after --clean, not before) and verifies the grant took. It restarts api and worker only when the restore and the grant check both succeed. After any failure it leaves them stopped, so their startup migrations don’t run against a half-restored database:

docker compose stop backup
# Restores into the running db container from a custom-format dump
./scripts/restore.sh /path/to/geolens_20260101_020000.dump

Internally it runs:

pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner < "$BACKUP_FILE"

The --clean --if-exists flags drop existing objects first and tolerate objects that are absent on a fresh database. The script treats any nonzero pg_restore exit as a failed restore. --no-owner skips ownership commands so the dump restores cleanly under the geolens role regardless of who created it.

To restore the DB by hand (e.g. without the wrapper), match those flags and run the same grant reconciliation before anything starts again. The commands below run pg_restore and psql through sh -c so $POSTGRES_USER and $POSTGRES_DB come from the db container’s environment:

# Stop services to prevent writes and a backup cycle mid-restore
docker compose stop api worker backup

# Copy the backup into the database container
docker compose cp ./backup.dump db:/tmp/backup.dump

# Restore (mirror restore.sh's flags); stop here if this exits nonzero
docker compose exec -T db sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  --clean --if-exists --no-owner /tmp/backup.dump'

# Re-grant geolens_reader (and the runtime role, when enabled) after --clean
docker compose exec -T db /usr/local/bin/configure-runtime-db-role

# Must print t
docker compose exec -T db sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tA' <<'SQL'
SELECT has_schema_privilege('geolens_reader', 'data', 'USAGE');
SQL

If you set GEOLENS_RUNTIME_DB_ROLE, also confirm that role came back without elevated attributes before you start api and worker. This query must print t:

docker compose exec -T db sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tA \
  -v role="$GEOLENS_RUNTIME_DB_ROLE"' <<'SQL'
SELECT NOT rolsuper AND NOT rolbypassrls AND NOT rolcreaterole
       AND NOT rolcreatedb AND NOT rolreplication
FROM pg_roles WHERE rolname = :'role';
SQL

If pg_restore fails or either check prints anything else, leave api and worker stopped, because their startup migrations would run against a half-restored database. Otherwise start them, and start backup only once the restored data is verified:

docker compose start api worker

scripts/restore.sh runs all of these steps and checks for you, so prefer it unless you can’t run it.

To restore a single table, pass the schema and the table separately. pg_restore doesn’t match a schema-qualified --table value, so --table catalog.datasets restores nothing and still exits 0:

docker compose exec -T db sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  --clean --if-exists --no-owner -n catalog -t datasets /tmp/backup.dump'

Caution

With --clean, a single-table restore starts with a plain DROP TABLE, which fails when other tables have foreign keys to it. catalog.datasets is referenced by many tables (catalog.api_keys among them), so for it the drop fails, the existing table stays, and pg_restore then tries to load the dump’s rows on top of it. For a widely referenced table, restore the whole dump into a scratch database, copy back only the rows you need, then drop the scratch database:

docker compose exec -T db sh -c 'createdb -U "$POSTGRES_USER" geolens_scratch && \
  pg_restore -U "$POSTGRES_USER" -d geolens_scratch --no-owner /tmp/backup.dump'
# ...inspect or copy rows, then:
docker compose exec -T db sh -c 'dropdb -U "$POSTGRES_USER" geolens_scratch'

Note

restore.sh restores the database only. If a matching staging-<timestamp>.tar.gz object-storage archive sits next to the dump, the script does not auto-extract it. It prints the exact docker run ... tar xzf command to restore the staged objects into the upload_staging volume as a manual step (RUNBOOK.md, full restore). Replaying globals-<timestamp>.sql is likewise manual, and it comes before pg_restore rather than after. Naming the missing roles is a warning, not a gate: the script reconciles the GeoLens-managed roles and grants itself, but it will not recreate the cluster’s own role definitions and passwords for you. RUNBOOK.md section 2 has the sequence, and the fallback for when no globals dump exists.

To restore a volume-level snapshot:

docker compose down
# Replace <project> with your Compose project name (see `docker volume ls`).
docker run --rm -v <project>_pgdata:/data -v $(pwd):/backup alpine \
  sh -c "rm -rf /data/* && tar xzf /backup/pgdata_backup.tar.gz -C /data"
docker compose up -d --scale backup=0
# After the restored data checks out:
docker compose up -d backup

Caution

Volume restores destroy the destination volume. The rm -rf /data/* step is intentional: tar xzf does not overwrite existing files cleanly when ownership differs. Always restore to a fresh or known-empty volume in production.

Restore validation

restore.sh ends with a row-count sanity check (catalog.records and catalog.datasets), but beyond that restore validation is operator-driven: there is no automated post-restore checksum or integrity verification. Recommended practice:

1. Restore the most recent daily backup into a staging instance monthly.

2. Run curl https://staging.geolens.example.com/api/health and confirm database.status: ok (see Infrastructure & Monitoring for the health probe). Use /api/health, not a bare /health: through the bundled Nginx the latter returns the SPA’s HTML with status 200 and no provider JSON.

3. Spot-check 2 to 3 dataset rows from the catalog API:

   curl https://staging.geolens.example.com/api/datasets/ \
     -H "Authorization: Bearer $TOKEN" | jq '.datasets[:3]'

4. Open a representative dataset in the staging UI and confirm map preview, exports, and metadata render correctly.

If staging restore reveals corruption, the most recent daily backup may be unrecoverable. Promote the most recent weekly backup to production-restore status and investigate the failed daily as a separate incident.

See also

• Settings reference: Storage tab covers the S3_* credentials the backup service reuses when BACKUP_S3_ENABLED=true
• Self-hosted provider notes: backups and S3 off-site replication with managed providers
• Configuration reference: full BACKUP_* env-var details
• RUNBOOK.md: the operator runbook in the repository — fresh-cluster and managed-Postgres restore, data-loss incident response, schema rollback, and PostgreSQL major-version upgrades
• Upgrade guide: version-specific operator notes, including the 1.7.0 release that introduced the globals-*.sql artifact