Skip to main content

DLE 2.5: Better data extraction for logical mode and configuration improvements

ยท 4 min read

DLE 2.5: Better data extraction for logical mode and configuration improvements


Action required to migrate from a previous version. See Migration notes.

The Database Lab Engine (DLE) is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs.

DLE 2.5 significantly expands the capabilities of automatic preparation of snapshots directly from managed database services, as well as from logical dumps, namely:

  • restoring of multiple databases
  • various pg_dump output formats and file compression formats

Since version 2.5, it becomes possible to reset the clone's database state to a specific snapshot if multiple snapshots are available. See DLE CLI reference. There is also a new option for the reset command, --latest, that allows resetting to the latest available state not knowing the snapshot name. This can be very useful in situations when a clone lives long, occupying a specific port, and some applications (e.g., analytical tools) are configured to work with it โ€“ users can periodically switch to the freshest database state without a need to reconfigure their applications.

All new restore features are also already available in the Terraform module (currently works with AWS only).

Additionally, this release has a lot of improvements and fixes. Read the full changelog.

Migration notesโ€‹

The Database Lab ecosystem is growing with new products and services. Most of them have configuration files and metadata, but their structure and naming are not uniform.

To simplify work with the configuration, we decided to redesign the configuration structure of all Database Lab services and standardize the naming of configuration files and directories.

Prepare configuration directories on a host machineโ€‹

  • The main directory of Database Lab configuration is ~/.dblab. There are directories for each service inside this main directory:
    • cli - configs of Database Lab command-line interface
    • engine - configs and metadata of Database Lab Engine
    • joe - configs and metadata of Joe Bot assistant
    • ci_checker - configs of DB Migration Checker
  • Rename the configuration file of Database Lab Engine to server.yml and store it to ~/.dblab/engine/configs
  • In the running command docker run ... replace the mounting flag
    --volume ~/.dblab/server.yml:/home/dblab/configs/config.yml
    --volume ~/.dblab/engine/configs:/home/dblab/configs \
    --volume ~/.dblab/engine/meta:/home/dblab/meta

Check the example of configuration structure on a host machine:

- cli.yml

- server.yml
- state.json

- joe.yml
- sessions.json

- ci_checker.yml

(optional) Compact the configuration file using common YAML-sections and anchorsโ€‹

Database Lab Engine starts a number of Docker containers with a PostgreSQL instance inside. A powerful configuration system allows controlling various properties of running containers (PostgreSQL version, Docker image, container parameters, extensions loaded, PostgreSQL configuration parameters).

However, such flexibility may be inconvenient in the case of a number of the configuration section. On the other hand, YAML anchors and aliases can be used, help you conveniently manage your configuration sections

YAML allows defining a "global" binding with & where you can set properties for all PostgreSQL containers and then refer to it using an alias denoted by *.

โš  Note, the "local" configuration of each section would still be supported, overriding particular parts.

There is an example using an anchor:

# Configure database containers
databaseContainer: &db_container
dockerImage: 'postgresai/extended-postgres:13'
'shm-size': 1gb

- logicalDump
- logicalRestore
- logicalSnapshot

<<: *db_container
dumpLocation: '/var/lib/dblab/dblab_pool/dump'

See more examples of configuration files in the Database Lab repository

If you have problems or questions, please contact our communities for help:

Request for feedback and contributionsโ€‹

Feedback and contributions would be greatly appreciated:

Share this blog post:

Nikolay Samokhvalov
Nikolay Samokhvalov

CEO & Founder of

Working on tools to balance Dev with Ops in DevOps

Database Lab
Database Lab by

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.