3 posts tagged with "database lab"

View All Tags

Database Lab Engine 0.3, supports both ZFS and LVM

Database Lab 0.3: users can choose which "thin clone manager" to use, ZFS or LVM#

Update: see the discussion on Hacker News!

Version 0.3 of Database Lab Engine (with a minor update to 0.3.1) adds support of LVM as an alternative to ZFS to enable thin cloning of large databases. This was one of the most requested features after the initial launch of the public Database Lab version a month ago.

Database Lab Engine is an open source technology that helps you clone non-production databases in seconds.

LVM can be chosen as a "thin-clone manager" instead of ZFS for those who do not want to use ZFS and prefer staying on more popular file systems (ext4, xfs) in non-production environments. It is worth noting that ZFS remains the default and recommended option. Postgres.ai team is very satisfied with the experience of using it running Database Labs for multi-terabyte, heavily loaded databases.

Compared to ZFS, the LVM module has a certain restriction: it is not possible to support multiple snapshots and allow choosing the snapshot when requesting a new clone. With LVM, the new clones always are based on the latest state of the database.

We invite everyone to test both modules, join our Community Slack, and provide your feedback! See the links below.

Links:

Database Lab Engine 0.2

Database Lab Engine updated to 0.2: everything in containers, better API and CLI#

Update: see the discussion on Hacker News!

We have released version 0.2 of Database Lab Engine, an open source technology that helps you clone non-production databases in seconds.

Now all its components run in containers, so installation and use is much easier. Additionally, various improvements were made, including those in API and client CLI.

Work on documentation continues: we reworked Tutorial, and added new texts. One of them is What is Database Lab. This picture should help to compare Database Lab to traditional methods of development and testing involing large databases:

Comparison Matrix

Links:

The first public release of Database Lab Engine

Postgres.ai team is proud to announce the very first public release of Database Lab Engine#

Update: see the discussion on Hacker News!

Database Lab Engine helps you build non-production environments for projects that use multi-terabyte Postgres databases. Initially obtained using standard "thick" copying (such as pg_basebackup, restoration from an archive, or dump/restore), Postgres data directory then gets cloned on request. Such cloning takes just a couple of seconds. Developers, DBAs, and QA engineers can quickly get fully independent copies, perform testing, and idea verification obtaining reliable (close to production) results. As a result, development speed and quality significantly increase.

Database Lab Engine is open source, you can find the code, ongoing work, and the Issue tracker here: https://gitlab.com/postgres-ai/database-lab.

Here is the list of some tasks that Database Lab Engine can help solve:

  1. Troubleshoot an SQL query (run EXPLAIN, EXPLAIN (BUFFERS, ANALYZE)): with query planner settings matching production, one can check any query, including UPDATE, DELETE, INSERT, TRUNCATE, not putting production master into any risks. See also: Joe bot.
  2. Verify an index idea: it is easy to create an index and check if it helps optimize your queries.
  3. Check database migrations (DB schema changes) or massive data modifications and highlight potentially dangerous steps, to avoid performance degradation and downtime on production.

A single Database Lab instance can provide multiple thin Postgres clones (full-size and fully independent) simultaneously. It becomes possible thanks to copy-on-write (CoW) technology. The only option supported in version 0.1 is ZFS; however, there are plans to support other technologies in the future.

Database Lab can be installed either on a physical machine or a VM. Both on-premise or cloud setups are possible. Users communicate with Database Lab using either REST API or client CLI. The first version of Database Lab has certain limitations:

  • it works on Ubuntu 18.04 only,
  • only Postgres versions 9.6, 10, 11, and 12 are supported,
  • in addition to ZFS, the installation of Postgres and Golang is required (it is planned to get rid of this requirement in version 0.2, fully switching to containers).

Links: