5 posts tagged with "Database Lab"

Today, developers who need to work with full scale data have three options:

1. Develop scripts to backup & restore their production database
2. Use expensive cloud provided database cloning features
3. Deploy and test on production 😱

In other words, they can pay with time, money, risk, or some combination of all three.

But! There is a better, faster, cheaper, and completely safe way for developers to get the data they need to do their work properly. With the Database Lab Engine, developers can instantly provision as many full scale copies of their database as needed.

Check out this table which makes a direct comparison of the most common methods (We omitted testing on production. Don't do that.):

The price comparison here makes the following assumptions:

1. Once provisioned, the clone remains continuosly available for development and CI testing
2. The clones all run on an r5.xlarge instance
3. The database size is 100 GiB

As a result, running your development or staging environments with the Database Lab Engine (how it works) is more than 10x cheaper and 10x faster than the best available alternative.

Try it now in the AWS Marketplace. Setup docs are here and contact us if you have questions!

Dante Cassanego

Postgres.ai

Database Lab by Postgres.ai

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.

Over its relatively short history, the discipline of Software Engineering has made rapid advances in the sophistication of its development processes and tools. In the past 15 years alone, the popularization of CI/CD tools has drastically improved software quality and reliability.

However, a large gap remains on the landscape of software tooling. For many engineers, it's a gap they are so accustomed to, they can no longer even see it.

The Elements of Application Behavior

Database Lab 0.3: users can choose which "thin clone manager" to use, ZFS or LVM​

Update: see the discussion on Hacker News!

Version 0.3 of Database Lab Engine (with a minor update to 0.3.1) adds support of LVM as an alternative to ZFS to enable thin cloning of large databases. This was one of the most requested features after the initial launch of the public Database Lab version a month ago.

Database Lab Engine is an open source technology that helps you clone non-production databases in seconds.

LVM can be chosen as a "thin-clone manager" instead of ZFS for those who do not want to use ZFS and prefer staying on more popular file systems (ext4, xfs) in non-production environments. It is worth noting that ZFS remains the default and recommended option. Postgres.ai team is very satisfied with the experience of using it running Database Labs for multi-terabyte, heavily loaded databases.

Compared to ZFS, the LVM module has a certain restriction: it is not possible to support multiple snapshots and allow choosing the snapshot when requesting a new clone. With LVM, the new clones always are based on the latest state of the database.

Database Lab Engine updated to 0.2: everything in containers, better API and CLI​

Update: see the discussion on Hacker News!

We have released version 0.2 of Database Lab Engine, an open source technology that helps you clone non-production databases in seconds.

Now all its components run in containers, so installation and use is much easier. Additionally, various improvements were made, including those in API and client CLI.

Work on documentation continues: we reworked Tutorial, and added new texts. One of them is What is Database Lab. This picture should help to compare Database Lab to traditional methods of development and testing involing large databases:

Postgres.ai team is proud to announce the very first public release of Database Lab Engine​

Update: see the discussion on Hacker News!

Database Lab Engine helps you build non-production environments for projects that use multi-terabyte Postgres databases. Initially obtained using standard "thick" copying (such as pg_basebackup, restoration from an archive, or dump/restore), Postgres data directory then gets cloned on request. Such cloning takes just a couple of seconds. Developers, DBAs, and QA engineers can quickly get fully independent copies, perform testing, and idea verification obtaining reliable (close to production) results. As a result, development speed and quality significantly increase.

Database Lab Engine is open source, you can find the code, ongoing work, and the Issue tracker here: https://gitlab.com/postgres-ai/database-lab.

Here is the list of some tasks that Database Lab Engine can help solve:

1. Troubleshoot an SQL query (run EXPLAIN, EXPLAIN (BUFFERS, ANALYZE)): with query planner settings matching production, one can check any query, including UPDATE, DELETE, INSERT, TRUNCATE, not putting production master into any risks. See also: Joe bot.
2. Verify an index idea: it is easy to create an index and check if it helps optimize your queries.
3. Check database migrations (DB schema changes) or massive data modifications and highlight potentially dangerous steps, to avoid performance degradation and downtime on production.

A single Database Lab instance can provide multiple thin Postgres clones (full-size and fully independent) simultaneously. It becomes possible thanks to copy-on-write (CoW) technology. The only option supported in version 0.1 is ZFS; however, there are plans to support other technologies in the future.

Database Lab can be installed either on a physical machine or a VM. Both on-premise or cloud setups are possible. Users communicate with Database Lab using either REST API or client CLI. The first version of Database Lab has certain limitations:

• it works on Ubuntu 18.04 only,
• only Postgres versions 9.6, 10, 11, and 12 are supported,
• in addition to ZFS, the installation of Postgres and Golang is required (it is planned to get rid of this requirement in version 0.2, fully switching to containers).