Configure storage

Storage defaults

SUSE Observability doesn’t specify a specific storage class on its PVC’s (persistent volume claims) by default, for cloud providers like EKS and AKS this means the default storage class will be used.

The defaults for those storage classes are typically to delete the PV (persistent volume) when the PVC is deleted. However, even when running helm delete to remove a stackstate release the PVC’s will remain present within the namespace and will be reused if a helm install is run in the same namespace with the same release name.

To remove the PVC’s either remove them manually with kubectl delete pvc or delete the entire namespace.

SUSE Observability requires the underlying storage to be based on flash memory (SSD) or similar in performance.

For production environments, NFS is not recommended and supported for storage provisioning in SUSE Observability due to the potential risk of data corruption.

When in doubt, it is possible to run the performance script and compare that to the reference performance.

Customize storage

You can customize the storageClass and size settings for different volumes in the Helm chart. These example values files show how to change the storage class or the volume size. These can be merged to change both at the same time. For the size we provide the sample for both HA and NonHa depending on the sizing profile chosen during the installation process.

  • Changing storage class

  • Changing volume size for HA

  • Changing volume size Non-Ha

global:
  # The storage class for all of the persistent volumes
  storageClass: "standard"
clickhouse:
  persistence:
    # Size of persistent volume for each clickhouse pod
    size: 50Gi
elasticsearch:
  volumeClaimTemplate:
    resources:
      requests:
        # size of volume for each Elasticsearch pod
        storage: 250Gi

hbase:
  hdfs:
    datanode:
      persistence:
        # size of volume for HDFS data nodes
        size: 250Gi

    namenode:
      persistence:
        # size of volume for HDFS name nodes
        size: 20Gi


kafka:
  persistence:
    # size of persistent volume for each Kafka pod
    size: 100Gi


zookeeper:
  persistence:
    # size of persistent volume for each Zookeeper pod
    size: 8Gi

victoria-metrics-0:
  server:
    persistentVolume:
      size: 250Gi
victoria-metrics-1:
  server:
    persistentVolume:
      size: 250Gi

stackstate:
  components:
    checks:
      tmpToPVC:
        volumeSize: 2Gi
    healthSync:
      tmpToPVC:
        volumeSize: 2Gi
    state:
      tmpToPVC:
        volumeSize: 2Gi
    sync:
      tmpToPVC:
        volumeSize: 2Gi
    vmagent:
      persistence:
        size: 10Gi
  features:
    storeTransactionLogsToPVC:
      volumeSize: 600Mi
  stackpacks:
    pvc:
      size: 1Gi

backup:
  configuration:
    scheduled:
      pvc:
        size: 1Gi
minio:
  persistence:
    size: 500Gi
clickhouse:
  persistence:
    # Size of persistent volume for each clickhouse pod
    size: 50Gi

elasticsearch:
  volumeClaimTemplate:
    resources:
      requests:
        # size of volume for each Elasticsearch pod
        storage: 250Gi

hbase:
  stackgraph:
    persistence:
      # Size of persistent volume for the single stackgraph hbase pod
      size: 100Gi

kafka:
  persistence:
    # size of persistent volume for each Kafka pod
    size: 100Gi


zookeeper:
  persistence:
    # size of persistent volume for each Zookeeper pod
    size: 8Gi

victoria-metrics-0:
  server:
    persistentVolume:
      size: 50Gi

stackstate:
  components:
    checks:
      tmpToPVC:
        volumeSize: 2Gi
    healthSync:
      tmpToPVC:
        volumeSize: 2Gi
    state:
      tmpToPVC:
        volumeSize: 2Gi
    sync:
      tmpToPVC:
        volumeSize: 2Gi
    vmagent:
      persistence:
        size: 10Gi
  features:
    storeTransactionLogsToPVC:
      volumeSize: 600Mi
  stackpacks:
    localpvc:
      size: 1Gi
    pvc:
      size: 1Gi

backup:
  configuration:
    scheduled:
      pvc:
        size: 1Gi
minio:
  persistence:
    size: 500Gi

The NonHa example belongs to the biggest NonHa instance meant to observe 100 nodes and retain data for 2 weeks.

Avoiding double replication on components that already replicate

Some HA sizing-profile components run with both application-level replication (2-3 copies of their own data across pods) and storage-level replication (your StorageClass keeping another 2-3 copies of each volume). Combined, this multiplies storage usage far beyond what’s needed for redundancy — for example, 3 application copies on a StorageClass that also keeps 3 copies results in 9 copies of the same data.

For components where the application already replicates, you can use a StorageClass that keeps only one copy per volume, relying entirely on the application’s own replication for redundancy instead. This applies to any storage backend that replicates by default — consult your backend’s own documentation for the equivalent setting. Longhorn is documented in detail below since it’s the backend we’ve tested this against.

Longhorn

Longhorn’s own best practices recommend strict-local data locality with a single replica for exactly this case ("applications that support data replication, e.g. a distributed database"), or reducing numberOfReplicas to 2 as a smaller middle-ground step.

Create a StorageClass mirroring Longhorn’s default one, except for numberOfReplicas, dataLocality, and volumeBindingMode:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-database-sc
provisioner: driver.longhorn.io
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "1"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "strict-local"
  disableRevisionCounter: "true"
  dataEngine: "v1"

Set this as the global storage class to cover Kafka and Elasticsearch. Override the HDFS metadata nodes, which do not replicate their data, back to your normal storage class:

global:
  storageClass: "longhorn-database-sc"

hbase:
  hdfs:
    namenode:
      persistence:
        storageClass: "longhorn"
    secondarynamenode:
      persistence:
        storageClass: "longhorn"

global.storageClass also applies to other PVCs, including backup, cache, VictoriaMetrics, vmagent, and workload-observer PVCs. Render your chosen profile with helm template and review the resulting storageClassName values before applying this configuration.

Do not assign this StorageClass to the HDFS NameNode or SecondaryNameNode (hbase.hdfs.namenode/hbase.hdfs.secondarynamenode). Unlike DataNodes, they hold a single, non-replicated copy of the HDFS filesystem metadata regardless of sizing profile. Losing their only copy doesn’t just lose data — it can leave HDFS (and anything reading from it, like HBase) unable to start at all. Keep them on your normal, fully-redundant storage class.

This optimization is for HA profiles only. ClickHouse only replicates (clickhouse.replicaCount > 1) in the 4000-ha profile; in other profiles, override it to use your normal storage class:

clickhouse:
  persistence:
    storageClass: "longhorn"

dataLocality: strict-local is incompatible with ReadWriteMany volumes. StorageClass selection applies only when a PVC is created; changing Helm values does not migrate existing PVCs. Check the rendered manifests and existing PVCs before an upgrade.

With one storage-level copy, replace or drain one node at a time and wait for application-level replication to converge before starting the next node. For Longhorn, review node failure handling, graceful node removal, and the Node Drain Policy before maintenance.