Skip to main content

Database Lab Engine

Tutorials and explanations​

References​

User Guides​

Overview​

Database Lab Engine – an open-source technology that is a core component of the Database Lab Platform. It is used to build powerful, state-of-the-art development and testing environments, based on a simple idea: with modern thin cloning technologies, it becomes possible to iterate 100x faster in development and testing. It is extremely helpful for larger or small but very agile teams that want to achieve high development velocity and the most competitive "time to market" characteristics and save budgets on non-production infrastructure.

Database Lab Platform is a paid SaaS offering developed and maintained by Postgres.ai. It provides GUI, user management, permissions control, token management, more.

Database Lab Engine​

Database Lab Engine is distributed under AGPL v3 license. Repositories:

Database Lab Engine is hosted and developed on GitLab.com. Why? GitLab Inc. is our (Postgres.ai) long-term client and early adopter (see GitLab Development Docs). GitLab has an open-source version. Last but not least: GitLab uses PostgreSQL.

However, nowadays, not many open-source projects are hosted at GitLab.com unfortunately.
⭐️ Please support the project giving a star on GitLab! It's on the main page of the Database Lab Engine repository, at the upper right corner:

Add a GitLab star

However, nowadays, not many open-source projects are hosted at GitLab.com unfortunately. Please support the project giving a GitLab star! It's on the main page, at the upper right corner:

Database Lab Engine provides the server with API, and client CLI, with basic single-user authentication.

As an example, cloning of 10 TiB PostgreSQL database takes less than 2 seconds when a single user is using the Database Lab Engine instance, and up to 30 seconds when 15 users are working with it at the same time. Moreover, such cloning (called "thin cloning") does not increase budgets: on a single mid-size machine with a single physical copy of the database, it is possible to run dozens of thin clones simultaneously.

Thin cloning is possible thanks to copy-on-write capabilities provided by either ZFS filesystem or LVM2 (other options such as hardware-based support of thin cloning are possible to develop thanks to modular and open architecture of Database Lab Engine).

Some problems that can be solved by using Database Lab:

  • help build independent development and testing environments involving full-size database without extra time and money spending,
  • provide temporary full-size database clones for SQL query optimization (see also: Joe bot, which works on top of Database Lab),
  • help verify database migrations (DB schema changes) and massive data operations.

Features​

  • Works well both on premise and in clouds.
  • Thin provisioning in seconds thanks to copy-on-write (CoW) provided by ZFS and special methodology of preparing PostgreSQL database snapshots. There is also an option to use LVM instead of ZFS.
  • Unlimited size of databases (Postgres database size is unlimited, ZFS volume can be up to 21^28 bytes, or 256 trillion yobibytes).
  • Supports PostgreSQL 9.6, 10, 11, 12, 13 and 14.
  • Thin cloning takes only a few seconds, regardless of the database size.
  • REST API.
  • Client CLI.
  • Automated deletion of clones after specified amount of minutes of inactivity (configurable).
  • Protection from deletion, to disable automated and accidental deletions.
  • Continuously updated original copy of data is supported.
  • Multiple snapshots to allow provisioning of various versions of the database.
  • Custom PostgreSQL Docker images to work with extended PostgreSQL setups (extensions, additional tools, or even modified PostgreSQL engine).

More​