DBLab Engine version 3.4, an open-source tool for PostgreSQL thin cloning and database branching, has been released with numerous improvements.

Rapid, cost-effective cloning and branching are extremely valuable when you need to enhance the development process. DBLab Engine can handle numerous independent clones of your database on a single machine, so each engineer or automated process can work with their own database created within seconds without additional expenses. This enables testing of any changes and optimization concepts, whether manually or in CI/CD pipelines, as well as validating all the concepts suggested by ChatGPT or another LLM. This effectively addresses the issue of LLM hallucinations.

The Postgres.ai team is happy to announce the release of version 3.2 of Database Lab Engine (DLE), an open-source tool that provides blazing-fast database cloning and branching for any PostgreSQL database to build powerful development, test, QA, and staging environments. DLE can run dozens of independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs. Data size and the number of snapshots/branches are virtually unlimited.

This release focuses on improving the admin experience during DLE configuration and troubleshooting. The full list of changes you can find in Release Notes.

Improved configuration and troubleshooting​

DLE UI now has two new tabs: "Logs" and "Configuration". They help DLE admins troubleshoot issues with database refresh, adjust configuration, and make another attempt to get data without the need to use SSH.

For logical mode (dump/restore), it is now possible to observe the current activity on both source and target sides:

Postgres clone errors are now stored in a special "diagnostic" directory (see new configuration section – diagnostic) that allows analyzing them even if the corresponding Postgres clone was already deleted or lost.

More flexibility for schema and data patching​

The configuration subsection queryPreprocessing is now present in the section logicalRestore, in addition to sections logicalSnapshot and physicalSnapshot. This can be helpful, for example, to mitigate errors related to CREATE POLICY and missing users – the queries provided in logicalRestore / queryPreprocessing are executed right before schema and data restoration attempt, so one can create "helper" database objects before the main body of the schema is created. For convenience, it is now also possible to provide inline SQL in queryPreprocessing in any section, instead of a path to SQL queries, for example:

retrieval:
    jobs:
        - logicalRestore
        - logicalSnapshot
    spec:
        logicalRestore:
            options:
                queryPreprocessing:
                    queryPath: ""
                    maxParallelWorkers: 2
                    inline: |
                      CREATE ROLE tony;
                      CREATE ROLE mary;
        logicalSnapshot:
            options:
                dataPatching:
                    queryPreprocessing:
                        inline: |
                            TRUNCATE audit_log;

Recently released PostgreSQL 15 is now fully supported as well as all previous versions starting with 9.6. The full list of supported PostgreSQL versions and extensions available in the default Docker images can be found in the docs).

Demo DLE​

You can see new UI working with Demo DLE: https://demo.aws.postgres.ai (token: demo-token).

DLE in AWS Marketplace is GA​

The AWS Marketplace version of Database Lab is now GA, and it's already updated to version 3.2. This offering is equipped with Standard license that includes guaranteed support from Postgres.ai, simplified setup, embedded monitoring system (Netdata), optional SSL. Paying less than for an RDS clone, you can have dozens of full-size clones running on a single machine with DLE. This short video demonstrates the steps of installation and configuration of DLE for a 100 GiB RDS database:

You can try out installing DLE from AWS Marketplace for your database here: https://aws.amazon.com/marketplace/pp/prodview-wlmm2satykuec.

Community news:

• 🌠 DLE repository on GitHub reached 1.4k stars; many thanks to everyone who supports the project in any way
• 📈 The Twitter account has reached 800 followers – please follow @Database_Lab
• 🎉 DLE now has 19 contributors. More contributions are welcome! See "good first issues"
• 🥇 Please consider various ways to contribute – read CONTRIBUTING.md

Further reading​

• DLE 3.2 release notes
• Database Lab documentation
• Tutorial for any database

Request for feedback and contributions​

Feedback and contributions would be greatly appreciated:

• Database Lab Community Slack
• DLE & DB Migration Checker issue tracker
• Issue tracker of the Terraform module for Database Lab

Like Database Lab? Give us a GitHub star: https://github.com/postgres-ai/database-lab.

Nikolay Samokhvalov

CEO & Founder of Postgres.ai

Working on tools to balance Dev with Ops in DevOps

Database Lab by Postgres.ai

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.

I'm very pleased to announce the very first preview version of Database Lab Engine (DLE) for AWS Marketplace. If you're using AWS, this is the fastest way to have powerful database branching for any database, including RDS and RDS Aurora. But not only RDS: any Postgres and Postgres-compatible database is supported as a source for DLE.

Now, for a fixed price (paying just for one EC2 instance and an EBS volume), you can have dozens of DB clones being provisioned in seconds and delivering independent databases for your Git branches, CI/CD pipelines, as well as manual optimization and testing activities.

Achieving the lowest entry barrier for the new DLE users remains to be one of our primary goals. In addition to the tutorials (we have several, including one for the RDS users) and Terraform module template for DLE, we now offer a way to install DLE on AWS only using a web browser.

What's included:

• DLE (same as Community Edition; the latest minor versions of DLE are available)
• Automated data provisioning at the logical level from live Postgres database (can be any Postgres, version 9.6 or newer)
• Automated refresh using two or more snapshots, using one EBS volume (of a bigger size), with configurable full refresh schedule (read about DLE's automated full refresh on schedule in the docs; note in AWS Marketplace version, we use a single ZFS pool but 2 (or more, if you request it) datasets to enable work with multiple full snapshots of the database – therefore, during a full refresh, users can still work with multiple clones created for the snapshot that is not being refreshed at the moment.)
• Full refresh schedule can be defined when DLE instance is created
• UI, CLI, and API to work with any number of thin clones – as usual – Optional generation of certificates and access to UI and API via HTTPS (via Envoy proxy), as well as access to the Postgres clones created by DLE users
• Guaranteed support via one of the available channels

What's not yet included in this "preview" version:

• Physical mode (for those who manage Postgres themselves)
• Many advanced DLE configuration options are not available in AWS Marketplace / CloudFormation interface; however, they can still be adjusted once the instance is created (most of them can be changed without DLE restart – see the docs)

To start, please read the documentation: "How to install DLE from the AWS Marketplace". Below you can watch a 3-minute video demonstrating the setup process.

If you have any questions, use this page to contact us: https://postgres.ai/contact.

Nikolay Samokhvalov

CEO & Founder of Postgres.ai

Working on tools to balance Dev with Ops in DevOps

Database Lab by Postgres.ai

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.

The Postgres.ai team is happy to announce the release of version 3.1 of Database Lab Engine (DLE), the most advanced open-source software ever released that empowers development, testing, and troubleshooting environments for fast-growing projects. The use of Database Lab Engine 3.1 provides a competitive advantage to companies via implementing the "Shift-left testing" approach in software development.

Database Lab Engine is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs.

In this release, the development team has also focused on the Database Lab Engine community, making it easier to get help or contribute. The team greets all new contributors: @Nikolay Devxx, @asotolongo, @Tanya301, @denis-boost, @pietervincken, @ane4ka

In DLE 3.1:

• Native support for pgBackRest as a tool to restore data from archives (physical mode, including continuously updated state), in addition to the existing support of WAL-G
• Allow configuring timezone in DLE CLI configuration to improve the experience of using DLE in CI/CD pipelines
• Improved README.md, translated to four languages, added CONTRIBUTING.md, SECURITY.md, and CODE_OF_CONDUCT.md
• Many improvements in the engine and UI to improve work both in logical and physical modes

Community news:

• 🌠 DLE repository on GitHub now has 1,100+ stars; many thanks to everyone who supports the project in any way
• 💥 Pieter Vincken has published a blog post describing their experience of using DLE: "Testing with production data made easy"
• 📈 The Twitter account has reached 400 followers – please follow @Database_Lab
• 🎉 DLE now has 15 contributors. More contributions are welcome! See "good first issues"
• 🥇 Please consider various ways to contribute – read CONTRIBUTING.md

Further reading​

• DLE 3.1 release notes
• Database Lab Documentation
• Tutorial for any database
• Tutorial for Amazon RDS
• Interactive tutorial (Katacoda)

Request for feedback and contributions​

Feedback and contributions would be greatly appreciated:

• Database Lab Community Slack
• DLE & DB Migration Checker issue tracker
• Issue tracker of the Terraform module for Database Lab

Like Database Lab? Give us a GitHub star: https://github.com/postgres-ai/database-lab.

Nikolay Samokhvalov

CEO & Founder of Postgres.ai

Working on tools to balance Dev with Ops in DevOps

Database Lab by Postgres.ai

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.

What's new in DLE 3.0?

The Postgres.ai team is happy to announce the release of version 3.0 of Database Lab Engine (DLE), the most advanced open-source software ever released that empowers development, testing, and troubleshooting environments for fast-growing projects. The use of Database Lab Engine 3.0 provides a competitive advantage to companies via implementing the "Shift-left testing" approach in software development.

Database Lab Engine is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their very own database provisioned in seconds without extra costs.

Among major changes in DLE 3.0:

• UI included to the core, it allows working with a single DLE instance,
• persistent clones: clones now survive DLE (or VM) restart,
• for the "logical" data provisioning mode: the ability to switch reset clone's state using a snapshot from different pool/dataset,
• better logging and configuration simplicity,
• improvements for the cases when multiple DLEs are running on a single machine,
• PostgreSQL 14 support.

Starting with version 3.0.0, DLE collects non-personally identifiable telemetry data. This feature is enabled by default but can be switched off. Read more in the DLE documentation. Keeping telemetry enabled can be considered your contribution to the DLE development because it helps make decisions down the road of the open-source product development.

Further, we discuss the most requested changes that were implemented in DLE 3.0 – all of them were created based on real-life user experience and invaluable feedback from the growing community of users and contributors.

The Database Lab Engine (DLE) is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs.

DLE 2.5 significantly expands the capabilities of automatic preparation of snapshots directly from managed database services, as well as from logical dumps, namely:

• restoring of multiple databases
• various pg_dump output formats and file compression formats

Since version 2.5, it becomes possible to reset the clone's database state to a specific snapshot if multiple snapshots are available. See DLE CLI reference. There is also a new option for the reset command, --latest, that allows resetting to the latest available state not knowing the snapshot name. This can be very useful in situations when a clone lives long, occupying a specific port, and some applications (e.g., analytical tools) are configured to work with it – users can periodically switch to the freshest database state without a need to reconfigure their applications.

All new restore features are also already available in the Terraform module (currently works with AWS only).

Additionally, this release has a lot of improvements and fixes. Read the full changelog.

Database Lab Engine 2.4 is out​

The Database Lab Engine (DLE) is an open-source technology to enable thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of an independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs.

DLE 2.4 brings two major capabilities to those who are interested in working with PostgreSQL thin clones:

• a new component, DB Migration Checker, that automates DB migration testing in CI/CD pipelines
• a Terraform module to deploy DLE in AWS using the "logical" provisioning mode

Additionally, this release has a lot of improvements and fixes.

About Database Lab Engine​

The Database Lab Engine (DLE) is an open-source experimentation platform for PostgreSQL databases. The DLE instantly creates full-size thin clones of your production database which you can use to:

1. Test database migrations
2. Optimize SQL queries
3. Deploy full-size staging applications

The Database Lab Engine can generate thin clones for any size database, eliminating the hours (or days!) required to create “thick” database copies using conventional methods. Thin clones are independent, fully writable, and will behave identically to production: they will have the same data and will generate the same query plans.

Learn more about the Database Lab Engine and sign up for an account at https://postgres.ai/.

Database Lab Engine 2.2.0​

Database Lab Engine (DLE) 2.2.0 further improves support for both types of PostgreSQL data directory initialization and synchronization: “physical” and “logical”. Particularly, for the “logical” type (which is useful for managed cloud PostgreSQL such as Amazon RDS users), it is now possible to setup multiple disks or disk arrays and automate data retrieval on a schedule. This gracefully cleans up the oldest versions of data, without downtime or interruptions in the lifecycle of clones.

Database Lab Engine 2.1 for PostgreSQL released​

We are happy to announce version 2.1.0 of Database Lab Engine (DLE), an open-source tool for building powerful development and testing environments based on thin cloning of PostgreSQL databases. Using Database Lab API or CLI (and if you are using Database Lab SaaS, GUI), on a single machine with, say, a 1 TiB disk, you can easily create and destroy dozens of database copies of size 1 TiB each. All these copies are independently modifiable and created/destroyed in just a few seconds. This can become a game-changer in your development and testing workflow, improving time-to-market, and reducing costs of your non-production infrastructure.

In 2.1, the main new features are:

• Better data protection and security:
  • robust configuration defining how data is patched when snapshots are automatically created (both shell and SQL scripts are now supported),
  • an option specifying whether or not passwords for the existing DB users need to be preserved.
• [experimental] DLE API and the CLI tool are extended to have a new feature: "CI Observer" helping control DB schema changes (DB migrations) — here is the reference on how to use it https://postgres.ai/docs/reference-guides/dblab-client-cli-reference#subcommand-start-observation. This is a small step towards the big goal: have 100% coverage for testing DB migrations in CI using full-sized thin clones. Watch the demo (turn captions on):

Database Lab Engine 2.0 for PostgreSQL released​

The Postgres.ai team is proud to announce version 2.0 of Database Lab Engine (DLE) for PostgreSQL, a modern database tool for building powerful development and testing environments based on thin cloning. Using Database Lab API or CLI (and if you are using Database Lab SaaS, GUI), on a single machine with, say, a 1 TiB disk, you can easily create and destroy dozens of database copies of size 1 TiB each. All these copies are independently modifiable and created/destroyed in just a few seconds. This can become a game-changer in your development and testing workflow, improving time-to-market, and reducing costs of your non-production infrastructure.

This release continues our strategy to automate all routine tasks such as initialization of the PostgreSQL data directory, data transformation, and snapshot management. In DLE 2.0, all these tasks can be flexibly configured in a single configuration file. As a result, building dev&test environments for projects with many databases (such as those that adopted microservice architecture) becomes much easier.

The previous versions of the Database Lab introduced the core technology: thin clone provisioning, based on either ZFS (default) or LVM. It was already possible to provision full-sized multi-terabyte database clones in just a few seconds and use them for a broad spectrum of tasks such as database schema changes verification, SQL query analysis, or general application testing.

Version 2.0 speeds up and empowers the initialization of DLE itself. Instead of using custom scripts for initial and continuous data retrieval, it is now possible to configure everything in a declarative manner to get the data and be up and running.