Skip to main content

Database Lab development roadmap


We work hard to develop new features for Database Lab SaaS and its open-source components, Database Lab Engine (DLE) and SQL Optimization Chatbot (Joe). Below you can find the main ideas we are working on now or planning to work soon.

Updated: 2021-09-07

[DLE] Physical provisioning#

Physical provisioning: native support of provisioning from archives created by a specific backup solution or based on an existing Postgres database

  • Support various sources
    • Generic (anything: pg_basebackup, rsync, backup tools)
    • Native support
      • WAL-E/WAL-G backups
      • pgBackRest backups
      • Barman backups
      • pg_probackup backups
      • Nutanix Era
  • Continuously updated state (physical replication based on WAL shipping)
  • Snapshot management (schedule, retention policy)
  • faster WAL replay (pg_prefaulter)

[DLE] Logical provisioning#

Logical provisioning: native support of provisioning for managed PostgreSQL databases

  • Support various sources
    • Simple dump/restore
    • Amazon RDS
    • Heroku Postgres
    • Azure PostgreSQL
    • Google Cloud SQL for Postgres
    • Digital Ocean Postgres
    • Any PostgreSQL DB via dump/restore
  • Continuously updated state (logical replication)
  • Restore from backups stored on AWS S3
    • uncompressed
    • compressed (gzip, bzip2)
  • Multiple pools, rotation on schedule
  • Partial data retrieval
    • specific tables
    • arbitrary filtering (columns, rows)

[DLE] Engine features#

  • Persist clones when the engine restarts 🔥
  • Point-in-time recovery (PITR) (Can be used for ultra-fast recovery of accidentally deleted data)
  • Duplicate DLE (create a new DLE based on existing one)
  • Clone analytics
  • Advanced audit
  • "Temporary" system- and Postgres-level monitoring for clones/sessions
  • Utilization of DLE instance and alerts
  • Usage and estimated savings reports
  • SSH port forwarding for API and Postgres connections
  • Tags
  • Framework for macro database experiments (work with thick/regular clones)
  • Auto-register DLE in Platform
  • Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)
  • User quotas
  • Disk quotas (zfs set quota=xx)
  • GUI with key features
  • Fast connection to clone's DB via CLI

[SaaS] Automated verification of database schema and complex data changes a.k.a. DB migrations#

  • [a] History and logging for clones/sessions
  • [a] Automated detection of locking issues
  • [a] Setting custom statement_timeout
  • [a] PostgreSQL logs for the migration
  • [a] Report in CI and Platform
  • Integration with CI tools – advanced integration
    • GitHub Actions
    • Bitbucket CI/CD
    • CitcleCI
    • Jenkins
    • GitLab CI/CD
    • Bamboo
    • TravisCI
  • Support vairous database migration tools + demo
    • Sqitch
    • Flyway
    • Liquibase
    • Ruby on Rails Active Record
    • Django migrations
  • [e] "Production timing" estimator
  • More artifacts to support decisions: pgstat*, system usage, WAL, checkpoints, etc.

[DLE] Cloning (CoW technology)#

  • ZFS
  • LVM
  • PureStorage
  • Remote clones – Amazon Aurora, Zenith

[SaaS] Automation, clouds, Kubernetes#

  • Clouds, automation of installation in clients' accounts
    • Basic Terraform templates
    • One-click setup on AWS. AWS Marketplace
    • One-click setup on GCP. GCP Marketplace
    • One-click setup on Azure. GCP Marketplace
    • One-click setup on Alibaba
  • SaaS: cloud offering (fully managed Database Lab)
    • AWS
    • GCP
    • Azure
    • Ali
  • AWS Spot instances for further savings
  • Kubernetes support

[Joe] SQL optimization chatbot#

  • Web UI version
  • Slack chatbot
  • Telegram chatbot
  • History with Search and Share options
  • Visualizations
    • explain.depesz
    • explain.dalibo (PEV2)
    • pgMustard (WebUI/SaaS only)
    • FlameGraphs
  • Better optimization recommendations
  • Macroanalysis insights (suggestions based on postgres-checkup / pgss)
  • Hypothetical indexes
  • Hypothetical partitioning
  • Index advisor
  • Utilization control
  • Restore user sessions after Joe container restarts
  • Better chatbot security
    • Do not use DB superuser
    • Quotas (rate limits)
    • Alert admins when a quota is hit

[SaaS] Data masking and anonymization#

  • Basic support for masking and obfuscation
  • Hybrid setup: raw and obfuscaned/masked clones on the same DLE instance
  • Dump/restore with runtime anonimization, parallelized, via GitOps
  • Simplified setup for anonymization - GUI
  • Automated masking / anonymization

[Docs] Better documentation and demo#

  • Tutorials
    • Basic
    • RDS
    • SQL optimization using Joe bot
    • DB migration testing in CI/CD pipelines
    • Katacoda
  • User Guides
    • DLE setup and administration
    • Cloning
    • SQL optimization with Joe bot
    • Building non-production environments
  • References
    • DLE API
    • CLI
    • DLE configuration
    • Joe configuration
  • Explanations and concepts
    • SQL optimization workflow with Database Lab
    • Configuration details
    • Why and how SQL optimization is possible on thin clones
    • Security aspects
    • Secure and robust test/staging environments
  • Interactive demo
    • Basic DLE features
    • SaaS features
    • SQL optimization using Joe bot (WebUI)
    • DB migration testing (GitHub Actions)
    • Advanced examples, use cases
  • Video demonstrations