Skip to main content

5 posts tagged with "Database Lab"

View All Tags

· 3 min read

Blue elephants in skies

Today, developers who need to work with full scale data have three options:

  1. Develop scripts to backup & restore their production database
  2. Use expensive cloud provided database cloning features
  3. Deploy and test on production 😱

In other words, they can pay with time, money, risk, or some combination of all three.

But! There is a better, faster, cheaper, and completely safe way for developers to get the data they need to do their work properly. With the Database Lab Engine, developers can instantly provision as many full scale copies of their database as needed.

Check out this table which makes a direct comparison of the most common methods (We omitted testing on production. Don't do that.):

Provisioning TimeMonthly Compute CostsMonthly Storage CostsTotal Monthly Costs for 10 Clones
Traditional Thick Cloning (EC2 + EBS) Hours $185 $100 per clone $1185
EC2 + EBS restored from snaphot ~10 mins plus an hour for warmup (lazy load) $185 $100 per clone $1185
RDS Clones ~10 mins plus an hour for warmup (lazy load) $365 per clone $100 per clone $4650
Aurora thin clones ~5 mins $417 per clone $100 $4270 (+ IO costs)
Database Lab Engine thin clones ~5 seconds $345 $100 $445

The price comparison here makes the following assumptions:

  1. Once provisioned, the clone remains continuosly available for development and CI testing
  2. The clones all run on an r5.xlarge instance
  3. The database size is 100 GiB

As a result, running your development or staging environments with the Database Lab Engine (how it works) is more than 10x cheaper and 10x faster than the best available alternative.

Try it now in the AWS Marketplace. Setup docs are here and contact us if you have questions!

Share this blog post:

Dante Cassanego
Dante Cassanego

Database Lab
Database Lab by

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.

· 2 min read

Over its relatively short history, the discipline of Software Engineering has made rapid advances in the sophistication of its development processes and tools. In the past 15 years alone, the popularization of CI/CD tools has drastically improved software quality and reliability.

However, a large gap remains on the landscape of software tooling. For many engineers, it's a gap they are so accustomed to, they can no longer even see it.

The Elements of Application Behavior Code AND Data

· 2 min read

Database Lab 0.3: users can choose which "thin clone manager" to use, ZFS or LVM

Update: see the discussion on Hacker News!

Version 0.3 of Database Lab Engine (with a minor update to 0.3.1) adds support of LVM as an alternative to ZFS to enable thin cloning of large databases. This was one of the most requested features after the initial launch of the public Database Lab version a month ago.

Database Lab Engine is an open source technology that helps you clone non-production databases in seconds.

LVM can be chosen as a "thin-clone manager" instead of ZFS for those who do not want to use ZFS and prefer staying on more popular file systems (ext4, xfs) in non-production environments. It is worth noting that ZFS remains the default and recommended option. team is very satisfied with the experience of using it running Database Labs for multi-terabyte, heavily loaded databases.

Compared to ZFS, the LVM module has a certain restriction: it is not possible to support multiple snapshots and allow choosing the snapshot when requesting a new clone. With LVM, the new clones always are based on the latest state of the database.

· One min read

Database Lab Engine updated to 0.2: everything in containers, better API and CLI

Update: see the discussion on Hacker News!

We have released version 0.2 of Database Lab Engine, an open source technology that helps you clone non-production databases in seconds.

Now all its components run in containers, so installation and use is much easier. Additionally, various improvements were made, including those in API and client CLI.

Work on documentation continues: we reworked Tutorial, and added new texts. One of them is What is Database Lab. This picture should help to compare Database Lab to traditional methods of development and testing involing large databases:

Comparison Matrix

· 2 min read team is proud to announce the very first public release of Database Lab Engine

Update: see the discussion on Hacker News!

Database Lab Engine helps you build non-production environments for projects that use multi-terabyte Postgres databases. Initially obtained using standard "thick" copying (such as pg_basebackup, restoration from an archive, or dump/restore), Postgres data directory then gets cloned on request. Such cloning takes just a couple of seconds. Developers, DBAs, and QA engineers can quickly get fully independent copies, perform testing, and idea verification obtaining reliable (close to production) results. As a result, development speed and quality significantly increase.

Database Lab Engine is open source, you can find the code, ongoing work, and the Issue tracker here:

Here is the list of some tasks that Database Lab Engine can help solve:

  1. Troubleshoot an SQL query (run EXPLAIN, EXPLAIN (BUFFERS, ANALYZE)): with query planner settings matching production, one can check any query, including UPDATE, DELETE, INSERT, TRUNCATE, not putting production master into any risks. See also: Joe bot.
  2. Verify an index idea: it is easy to create an index and check if it helps optimize your queries.
  3. Check database migrations (DB schema changes) or massive data modifications and highlight potentially dangerous steps, to avoid performance degradation and downtime on production.

A single Database Lab instance can provide multiple thin Postgres clones (full-size and fully independent) simultaneously. It becomes possible thanks to copy-on-write (CoW) technology. The only option supported in version 0.1 is ZFS; however, there are plans to support other technologies in the future.

Database Lab can be installed either on a physical machine or a VM. Both on-premise or cloud setups are possible. Users communicate with Database Lab using either REST API or client CLI. The first version of Database Lab has certain limitations:

  • it works on Ubuntu 18.04 only,
  • only Postgres versions 9.6, 10, 11, and 12 are supported,
  • in addition to ZFS, the installation of Postgres and Golang is required (it is planned to get rid of this requirement in version 0.2, fully switching to containers).