How to refresh data when working in the "logical" mode

For the "logical" provisioning mode, the "sync" instance is not yet supported (although, it is possible to implement based on logical replication) and the only option to get fresh data on Database Lab Engine is to refresh it fully. Follow these instructions to automate this process. Note, that it is designed for ZFS; if you have a different setup, adjust the snippets accordingly.

caution

Note, that the process described here requires a maintenance window (brief period of downtime) for the Database Lab Engine. Also, the existing clones are deleted and completely lost during the process. It means that the proper planning of the maintenance windows is needed.

If you are using the "physical" provisioning mode, read how to configure the "sync" instance instead.

Refresh data from source#

1. Cleanup#

Stop and remove the existing containers, then clean up the data directory and destroy the pool:

sudo docker ps -aq | xargs --no-run-if-empty sudo docker rm -f
sudo rm -rf /var/lib/dblab/dblab_pool/data/
sudo umount /var/lib/dblab/dblab_pool/dump
sudo rm -rf /var/lib/dblab/dblab_pool/dump
sudo zpool destroy dblab_pool

2. Set $DBLAB_DISK#

Further, we will need $DBLAB_DISK environment variable. It must contain the device name corresponding to the disk where all the Database Lab Engine data will be stored.

To understand what needs to be specified in $DBLAB_DISK in your particular case, check the output of lsblk:

sudo lsblk

Some examples:

  • AWS local ephemeral NVMe disks; EBS volumes for instances built on the Nitro system:

    $ sudo lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    ...
    xvda 202:0 0 8G 0 disk
    └─xvda1 202:1 0 8G 0 part /
    nvme0n1 259:0 0 777G 0 disk
    $ export DBLAB_DISK="/dev/nvme0n1"
  • AWS EBS volumes for older (pre-Nitro) EC2 instances:

    $ sudo lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    ...
    xvda 202:0 0 8G 0 disk
    └─xvda1 202:1 0 8G 0 part /
    xvdb 202:16 0 777G 0 disk
    $ export DBLAB_DISK="/dev/xvdb"

3. Recreate ZFS pool#

sudo zpool create -f \
-O compression=on \
-O atime=off \
-O recordsize=128k \
-O logbias=throughput \
-m /var/lib/dblab/dblab_pool \
dblab_pool \
"${DBLAB_DISK}"

4. Launch Database Lab Engine#

sudo docker run ... # the same as it was done initially

5. Checking the result manually (optional step)#

Optionally, if you do it manually, check the logs:

sudo docker logs dblab_server -f

And check the instance status using the client CLI:

dblab instance status

Automatic full refresh data from a source#

1. Prepare two disks#

Database Lab can support more than two disks for an automatic full refresh, but in this guide we will need two environment variables($DBLAB_DISK and $DBLAB_DISK2). They must contain the device names corresponding to the disks where all the Database Lab Engine data will be stored.

To understand what needs to be specified in $DBLAB_DISK and $DBLAB_DISK2 in your particular case, check the output of lsblk:

sudo lsblk

Some examples:

  • AWS local ephemeral NVMe disks; EBS volumes for instances built on the Nitro system:

    $ sudo lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    ...
    xvda 202:0 0 8G 0 disk
    └─xvda1 202:1 0 8G 0 part /
    nvme0n1 259:0 0 777G 0 disk
    $ export DBLAB_DISK="/dev/nvme0n1"
  • AWS EBS volumes for older (pre-Nitro) EC2 instances:

    $ sudo lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    ...
    xvda 202:0 0 8G 0 disk
    └─xvda1 202:1 0 8G 0 part /
    xvdb 202:16 0 777G 0 disk
    $ export DBLAB_DISK="/dev/xvdb"

2. Create two ZFS pools#

sudo zpool create -f \
-O compression=on \
-O atime=off \
-O recordsize=128k \
-O logbias=throughput \
-m /var/lib/dblab/dblab_pool \
dblab_pool \
"${DBLAB_DISK}"
sudo zpool create -f \
-O compression=on \
-O atime=off \
-O recordsize=128k \
-O logbias=throughput \
-m /var/lib/dblab/dblab_pool_02 \
dblab_pool_02 \
"${DBLAB_DISK2}"

3. Define a refresh timetable#

Set up a desirable timetable in the retrieval section of your configuration to perform a full refresh automatically

retrieval:
refresh:
timetable: "0 0 * * 1"
...

4. Launch Database Lab Engine#

sudo docker run ... # the same as it was done initially

5. Checking the result manually (optional step)#

Optionally, if you do it manually, check the logs:

sudo docker logs dblab_server -f