Server metrics

CloudCasa server exposes metrics related to the execution of backup and restore jobs. These can be queried with a monitoring tool such as Prometheus, and used for basic reporting and alerting.

These metrics can be retrieved manually using the following command:

kubectl -n cloudcasa-server exec -it <gandalf-controller-pod-name> -- curl -v http://localhost:8080/metrics

Note

Another way is to use casactl command.

Server metrics provide insights into both backup and restore operations. The names of all metrics are prefixed with “cloudcasa” for easy lookup in Grafana/Prometheus. All metrics can be filtered using the listed labels for more granularity.

The following count metrics are exposed:

Total Jobs - The cumulative count of all jobs (RUNNING and COMPLETED): cloudcasa_jobs_count
- Available Labels:
  - job_type
  - cluster
  - job_definition
Completed Jobs - The total count of completed jobs: cloudcasa_jobs_completed_count
- Available Labels:
  - job_type
  - cluster
  - job_definition
  - state
Running Jobs - The total count of running jobs: cloudcasa_jobs_running_count
- Available Labels:
  - job_type
  - cluster
  - job_definition
Job Duration - The cumulative duration of completed jobs, in seconds: cloudcasa_jobs_duration_total
- Available Labels:
  - job_type
  - cluster
  - job_definition
  - state
- Example: PromQL cloudcasa_jobs_completed_count{cluster="my-test-cluster", Type="K8s_SNAP"} returns a list of matching completed_jobs:
```
{Cluster="my-test-cluster", State="SKIPPED", Type="K8S_SNAP", __name__="cloudcasa_completed_jobs", instance="gandalf-controller-manager-metrics-service.cloudcasa-server:8443", job="cloudcasa-server-metrics"}
```

Scraping Metrics with Prometheus

Update your Prometheus configuration on the Kubernetes cluster where the CloudCasa server runs to add a new job for scraping the CloudCasa metrics.

Example configuration:

scrape_configs:
- job_name: 'cloudcasa-server-metrics'
  static_configs:
  - targets: ['gandalf-controller-manager-metrics-service.cloudcasa-server:8443']
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true

Note

If running multiple CloudCasa server installs on the same cluster, add a target for each installation, and label each one so they can be distinguished in queries.

static_configs:
- targets: ['gandalf-controller-manager-metrics-service.cloudcasa-server-one:8443']
  labels:
    cloudcasa_instance: server-one
- targets: ['gandalf-controller-manager-metrics-service.cloudcasa-server-two:8443']
  labels:
    cloudcasa_instance: server-two

Ensure the Prometheus ServiceAccount has permissions to GET the metrics endpoint:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

After configuring Prometheus for metric scraping, you can verify its functionality by accessing the Prometheus web UI. Use the web interface to view and query the collected metrics.

Grafana Dashboard

A template Grafana dashboard is available for download on the CloudCasa support portal at https://support.cloudcasa.io (under Resources). This dashboard includes Time series views for completed jobs by cluster, status, and type, along with a table view for exploring detailed job metrics.

To use the dashboard, download the “CloudCasa_Metrics_Dashboard.json” file from Support Portal and Import into Grafana with your Prometheus data source.

Sending Metrics to Splunk

CloudCasa metrics are exposed in Prometheus format, which Splunk cannot scrape or ingest natively. A collector is required to scrape the endpoint and forward the data to Splunk’s HTTP Event Collector (HEC). This section uses the Splunk Distribution of OpenTelemetry Collector, Splunk’s current recommended tool, configured with a static scrape target.

Prepare Splunk to receive metrics:

Create a metrics index: Settings -> Indexes -> New Index, using index type “Metrics”. This example uses an index named cloudcasa-metrics.
Enable the HTTP Event Collector: Settings -> Data Inputs -> HTTP Event Collector -> Global Settings. Turn on “All Tokens” and note the HEC port (default 8088).
Create a new HEC token and set its default index to the “cloudcasa-metrics” index created above. Enable the new token and copy the generated token value (It is used in the collector configuration below).

Deploy the Splunk OpenTelemetry Collector to scrape CloudCasa metrics and forward them to Splunk. Add the Helm repo:

helm repo add splunk-otel-collector https://signalfx.github.io/splunk-otel-collector-chart

Create a values file that points at your Splunk HEC endpoint and adds a scrape config for the CloudCasa metrics endpoint:

clusterName: <your-cluster-name>

splunkPlatform:
  endpoint: https://<your-splunk-hec-host>:8088
  token: <HEC-token-from-previous-step>
  metricsEnabled: true
  metricsIndex: cloudcasa-metrics

agent:
  config:
    receivers:
      prometheus/cloudcasa-server:
        config:
          scrape_configs:
            - job_name: cloudcasa-server
              scheme: https
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              tls_config:
                insecure_skip_verify: true
              static_configs:
                - targets:
                    - gandalf-controller-manager-metrics-service.cloudcasa-server:8443

    service:
      pipelines:
        metrics:
          receivers:
            - prometheus/cloudcasa-server

Note

If running multiple CloudCasa server installs on the same cluster, add a target per installation and label each one, same as the note above for Prometheus. This replaces the static_configs list under agent.config.receivers.prometheus/cloudcasa-server.config shown above:

scrape_configs:
  - job_name: cloudcasa-server
    scheme: https
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets: ['gandalf-controller-manager-metrics-service.cloudcasa-server-one:8443']
        labels:
          cloudcasa_instance: server-one
      - targets: ['gandalf-controller-manager-metrics-service.cloudcasa-server-two:8443']
        labels:
          cloudcasa_instance: server-two

Install the collector:

helm install splunk-otel splunk-otel-collector/splunk-otel-collector -n splunk -f splunk-otel-values.yaml

Ensure the collector agent’s ServiceAccount has permissions to GET the metrics endpoint:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: splunk-otel-collector
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

Bind this ClusterRole to the collector’s ServiceAccount (named <release-name>-splunk-otel-collector by default).

To verify, open the Search & Reporting app in Splunk. List all collected metric names:

| mcatalog values(metric_name) WHERE index=cloudcasa-metrics

Query a specific metric, e.g. total job count:

| mstats count WHERE index=cloudcasa-metrics metric_name="cloudcasa_jobs_count"

You should see the cloudcasa_* metrics listed and returning data.

List the available dimensions (labels) for a metric:

| mcatalog values(_dims) WHERE index=cloudcasa-metrics metric_name="cloudcasa_jobs_count"

Completed jobs by status:

| mstats sum(cloudcasa_jobs_completed_count) WHERE index=cloudcasa-metrics BY state