Skip to main content

Cloud Deployment Configuration

Configures the cloud resources for each engine when deploying pipelines to DataSQRL Cloud. Deployment settings are configured under the deployment field within each engine's configuration.

{
  "engines": {
    "flink": {
      "deployment": {
        "jobmanager-size": "small",
        "taskmanager-size": "large.mem",
        "taskmanager-count": 1
      },
      "config": {
        "execution.checkpointing.interval": "180s"
      }
    },
    "postgres": {
      "deployment": {
        "instance-size": "medium",
        "replica-count": 1
      }
    },
    "vertx": {
      "deployment": {
        "instance-size": "small",
        "instance-count": 1
      }
    }
  }
}

Apache Flink deployments consist of 1 job manager and a configurable number of identically sized task managers.

{
  "engines": {
    "flink": {
      "deployment": {
        "jobmanager-size": "small",    // Job manager instance size (see table below)
        "taskmanager-size": "medium",  // Task manager instance size (see table below)
        "taskmanager-count": 2,        // Number of task managers (positive integer)
        "taskmanager-disk-size-gb": 400 // NVMe space per task manager (see "Task Manager Disk" below)
      }
    }
  }
}

Task Manager Sizes​

NameCPUTask SlotsMemory (GiB)NVMe SpaceMax CPU Burst
dev0.51220GB2
small11455GB1
medium228110GB1
large4416220GB1
xlarge8832440GB1

The dev size is intended for development and testing with small amounts of data.

Task Manager Disk​

Each task manager gets local NVMe space for RocksDB state, batch spill files and any other Flink-local data. It defaults to the "NVMe Space" column of the selected size, and taskmanager-disk-size-gb overrides that default independently of the size:

{
  "engines": {
    "flink": {
      "deployment": {
        "taskmanager-size": "small",     // 55GB of NVMe by default
        "taskmanager-disk-size-gb": 400  // ...raised to 400GB
      }
    }
  }
}

Raise it when a job needs more local disk than its CPU/memory size implies — batch jobs in particular spill shuffle and sort data to local disk far beyond their memory footprint, and a task manager that exceeds its allocation is evicted mid-job.

The value is in GiB, must be positive, and is capped at 4000. It is a hard scheduling requirement: a task manager asking for more disk than any available node offers stays Pending instead of falling back to a smaller node.

Size Qualifiers​

Task manager sizes support qualifiers for specialized workloads. Qualifiers are grouped into memory-oriented and CPU-oriented variants:

  • .mem-Nx scales the pod memory by N and gives Flink proportionally more memory (Flink heap+managed grows with N). Use for state-heavy jobs.
  • .mem-headroom-Nx scales the pod memory by N but keeps Flink's allocation at the baseline memory; the extra memory is reserved for sidecar / native consumers (e.g., DuckDB, JNI libs, page cache).
  • .cpu doubles CPU with the same memory.
QualifierPod memoryFlink heap+managedTypical use
.cpubasebase × 0.80CPU-intensive jobs
.mem / .mem-2xbase × 2base × 1.6State-heavy jobs
.mem-4xbase × 4base × 3.2Large state
.mem-8xbase × 8base × 6.4Very large state
.mem-headroom-2xbase × 2base × 1Sidecars / native memory consumers
.mem-headroom-4xbase × 4base × 1Larger sidecar headroom
.mem-headroom-8xbase × 8base × 1Maximum sidecar headroom (e.g. DuckDB)

Examples:

  • medium.mem-4x → pod 32 GB / Flink heap+managed ≈ 25.6 GB.
  • xlarge.mem-headroom-8x → pod 256 GB / Flink heap+managed = 32 GB (baseline) / 224 GB headroom.

.mem is an alias for .mem-2x. The legacy .mem-headroom qualifier (triple memory, Flink stays at baseline) is deprecated — use .mem-headroom-Nx instead.

Size qualifiers do not apply to the dev instance.

Job Manager Sizes​

NameSubTasksCPUMemory (GiB)
dev<1000.51
small100-8000.52
medium800-200014
large>200028

Choose the job manager size based on the number of subtasks in your Flink job.

PostgreSQL (engines.postgres.deployment)​

PostgreSQL deployments consist of one primary instance and a configurable number of read replicas, all using the same instance size.

{
  "engines": {
    "postgres": {
      "deployment": {
        "instance-size": "medium",      // Instance size (see table below)
        "replica-count": 1,             // Number of read replicas (0 or larger)
        "disk-size-gb": 256,            // Disk size in GB (1 or larger)
        "auto-expand-percentage": 0.2,  // Auto-expand threshold (0 to disable, must be < 1)
        "create-indexes": true,         // Whether to create table indexes (see "Create Indexes" below)
        "data-checksums": true,         // Whether data-page checksums are enabled (see "Data Checksums" below)
        "parameters": {}                // Extra postgresql.parameters (see "Parameters" below)
      }
    }
  }
}

Instance Sizes​

NameCPUMemory (GiB)Default DiskMax CPU BurstMax Connections
dev0.5410GB1.5100
small18128GB1100
medium216256GB1200
large416512GB1300
xlarge8321TB1600

The dev size is intended for development and testing with small amounts of data.

Vert.x (engines.vertx.deployment)​

Vert.x API server deployments consist of a configurable number of identically sized server instances.

{
  "engines": {
    "vertx": {
      "deployment": {
        "instance-size": "small",  // Instance size (see table below)
        "instance-count": 2        // Number of server instances (positive integer)
      }
    }
  }
}

Instance Sizes​

NameCPUMemory (GiB)NVMe SpaceMax CPU BurstPg Pool Size
dev0.52-1.255
small1455GB15
medium28110GB110
large416220GB115

The dev size is intended for development and testing with small amounts of data. The .disk qualifier enables NVMe storage for instances that require local disk access.

Dedicated Nodes (*-dedicated-nodes)​

Pins a component's pods onto dedicated nodes. Each engine's deployment accepts a list of dedicated-node names. Each name is a hard requirement: if no matching node is available, the pod stays Pending — it never falls back to a shared node.

EngineField(s)
Flinktaskmanager-dedicated-nodes, jobmanager-dedicated-nodes
PostgreSQLdedicated-nodes
Vert.xdedicated-nodes
{
  "engines": {
    "flink": {
      "deployment": {
        "taskmanager-size": "medium.mem",
        "taskmanager-count": 6,
        "taskmanager-dedicated-nodes": [ "nvme" ]   // TaskManagers MUST run on the "nvme" dedicated nodes
      }
    }
  }
}

For each name N in the list, the pod is given:

  • a node selector requiring the node label N=true (the pin), and
  • a toleration for taint key N (so it is allowed onto the dedicated, tainted nodes).

The cluster side is an infrastructure concern: the dedicated nodes for N must be labeled N=true and tainted N=<value>:NoSchedule. The taint keeps everything that does not request N off those nodes; the matching label + toleration place the requesting pods on them. This gives both pinning (the workload runs there) and isolation (nothing else does).

Multiple names are combined with AND — the pod requires a node carrying all of them.

Provisioning the dedicated nodes (cluster side)​

Dedicated nodes are provisioned by the cluster operator, not by the pipeline. The contract is simple: the nodes for N must carry the label N=true and the taint N=true:NoSchedule. The label provides the pin (pods requesting N land here); the taint provides the isolation (everything that does not request N is kept off).

On a Karpenter-managed cluster (e.g. Amazon EKS), create a NodePool. For a group named nvme backed by local-NVMe instances:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: nvme
spec:
  template:
    metadata:
      labels:
        nvme: "true"            # selected by pods requesting dedicated nodes "nvme"
    spec:
      taints:
        - key: nvme             # repels everything that does not tolerate "nvme"
          value: "true"
          effect: NoSchedule
      requirements:
        - { key: node.kubernetes.io/instance-type, operator: In, values: [ i7i.2xlarge ] }
        - { key: karpenter.sh/capacity-type, operator: In, values: [ on-demand ] }
      # nodeClassRef, arch/os, disruption budgets and limits as appropriate for your cluster

The label name and taint key must both equal the dedicated-nodes name. These nodes must also carry any standard scheduling labels your platform applies to workload nodes, so the deployment's base node selection still resolves.

Do Not Disrupt (do-not-disrupt)​

Protects a component's pods from voluntary autoscaler disruption (node consolidation / scale-down). When true, the pods are annotated so the cluster autoscaler will not evict or consolidate them. Use it for long-running, stateful, or hard-to-reschedule workloads — for example a Flink catch-up that reprocesses the whole backlog, or the PostgreSQL primary during bootstrap.

EngineFieldDefault
Flinkdo-not-disruptfalse
PostgreSQLdo-not-disrupttrue
Vert.xdo-not-disruptfalse
{
  "engines": {
    "flink": {
      "deployment": {
        "do-not-disrupt": true   // keep TaskManagers/JobManager from being consolidated mid-run
      }
    }
  }
}

Create Indexes (create-indexes)​

Controls whether the PostgreSQL table indexes are created for the deployment. Defaults to true. PostgreSQL only.

EngineFieldDefault
PostgreSQLcreate-indexestrue

Set it to false to bootstrap the database tables-only, skipping all index creation. This is intended for a catch-up profile that reprocesses a large backlog: writing to un-indexed tables drains the backlog faster. The indexes are then built when the deployment is upgraded back to a steady-state profile (where create-indexes returns to its true default), so a catch-up deployment must be followed by such an upgrade before it serves production query traffic.

{
  "engines": {
    "postgres": {
      "deployment": {
        "create-indexes": false   // tables-only bootstrap for fast backlog draining; build indexes on the steady-state upgrade
      }
    }
  }
}

Data Checksums (data-checksums)​

Controls whether PostgreSQL data-page checksums are enabled for the database. Defaults to true (the PostgreSQL 18 default). PostgreSQL only.

EngineFieldDefault
PostgreSQLdata-checksumstrue

This is an initdb-time setting applied when the database is first created — it is immutable and cannot be changed on later deployments or upgrades. Set it to false only when the write-throughput cost of checksums matters more than corruption detection, and only for a database that will keep that setting for its lifetime.

{
  "engines": {
    "postgres": {
      "deployment": {
        "data-checksums": false   // disable data-page checksums; applied at initial bootstrap only, immutable afterwards
      }
    }
  }
}

Parameters (parameters)​

Extra PostgreSQL server parameters, merged into the database's postgresql.parameters. Any key here overrides the built-in default for that parameter. PostgreSQL only; empty by default.

EngineFieldDefault
PostgreSQLparameters{}

Typically used in a catch-up profile that trades durability for ingest throughput while reprocessing a large backlog — for example a larger shared_buffers/max_wal_size together with synchronous_commit: off. Set these only in the catch-up profile: on the steady-state upgrade any parameter not listed here reverts to its default.

{
  "engines": {
    "postgres": {
      "deployment": {
        "parameters": {
          "shared_buffers": "8GB",        // override the default (256MB)
          "max_wal_size": "32GB",
          "synchronous_commit": "off"     // faster backfill, durability trade-off
        }
      }
    }
  }
}