Installation

Prerequisites

This checklist provides all the requirements that must be met before self-hosted CloudCasa can be installed. Please check and make sure that you meet all the requirements before proceeding with the installation.

  • [ ] Kubernetes cluster (required)

    • Version 1.27 or later.

      OpenShift 4.14 or later though 4.13/4.12 can also be used as long as CronJobTimeZone feature gate is enabled.

    • A dedicated cluster is highly recommended, though not mandatory.

    Note

    Do not try to backup the cluster, where CloudCasa server is being installed, with CloudCasa itself. CloudCasa catalog can be protected using catalog backups (see Catalog backup/restore)

  • [ ] CLI tools (required)

    • Helm 3 (if using Helm installation method)

    • kubectl access to the cluster.

  • [ ] Storage class (required)

    • Decide on a storage class for CloudCasa PVCs.

    • Confirm it can provision PVs successfully.

  • [ ] Network configuration (required)

    • Decide how to expose thee CloudCasa service and install any prerequisites required (such as an ingress controller).

    • See Networking for more details. In particular, check the Networking requirements.

  • [ ] Pod Security Admission (if applicable)

    • If the cloudcasa-server and/or cloudcasa-mongo namespaces enforce the “Restricted” Pod Security Standard, note that the mongo and minio components require additional Helm values to run in such namespaces. All other components are “Restricted”-compliant by default.

    • See Security Context for the required values.

  • [ ] Authentication provider (required)

    • Decide on which provider to use: LDAP, Azure, Google, any OIDC provider, or local (not recommended for production).

    • Confirm that the CloudCasa server can communicate with the selected authentication provider.

    • Decide which users can log in (CloudCasa does not allow everyone in the configured auth provider to login automatically). You also need to decide which users are going to have site-admin permissions. See Authentication for more details.

    Note

    For POCs, we recommend using “local” auth provider (see Local provider). But we can also provide a dev LDAP deployment with a pre-configured set of users and groups if needed.

  • [ ] Certificates (required)

    • Decide whether to use cert-manager (recommended) or manually create TLS secrets.

    • If using cert-manager, install it with the command below.

    • See Certificate Configuration for more details.

    To install “cert-manager”:

    helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace \
 --version v1.12.4 --set installCRDs=true

  • [ ] License & Credentials (required)

    • Confirm you have access to CloudCasa support portal at https://support.cloudcasa.io. Login with your email to receive a temporary access code.

    • Download a valid license file. The license is tied to the cluster where CloudCasa is installed. You will need to provide the UID of kube-system namespace on the cluster so that a permanent license file can be generated. (Note: the UID is not needed for POCs.)

    • Download Helm repo credentials.

    • Download Azure container registry (ACR) credentials.

  • [ ] Database (MongoDB) setup (optional)

    • Decide whether to use MongoDB installed as part of the server deployment (default) or an external MongoDB instance.

    • If using an external MongoDB:

      • Ensure it is not in the same namespace as the CloudCasa server.

      • Have the URI and authentication information (username and password) available.

      • Confirm that the CloudCasa server can communicate with the external MongoDB instance.

    • See MongoDB Setup for details.

  • [ ] Email server (optional, but recommended)

    • Prepare SMTP details for an email server that can be accessed using SMTP.

    • Make sure CloudCasa server can communicate with the SMTP server.

    • This is needed by CloudCasa to send notifications (e.g. to report job errors) and to send user invitations to join organizations.

  • [ ] Cloud account information (optional)

    • CloudCasa supports adding cloud accounts from AWS, Azure, and GCP, though this support requires several additional steps.

    • See Cloud Accounts for details.

  • [ ] Air-gapped environment (if applicable)

    • Download the Helm Chart .tgz file from support.cloudcasa.io.

    • Mirror all container images required by CloudCasa server and CloudCasa agent into your internal registry, maintaining image paths.

    • Ensure that the CloudCasa server cluster and client clusters can pull images from the selected internal registry.

  • [ ] Object storage for backups

    • Prepare S3 or Azure Storage Account details for object storage that will be used to store backups. If you want to use NFS for backups, prepare the NFS shares accordingly.

    • Ensure client clusters can communicate with the storage.

  • [ ] Logs storage (optional)

    • By default, CloudCasa server installs a managed Minio deployment to store container logs and job logs. If you would rather use your own S3 buckets for logs storage, prepare the buckets and credentials ahead of time.

    • See Logs for details on logs storage and retention configuration.

Preparing Helm Values

We highly recommend creating a Helm values file to override default values instead of using --set or directly editing the packaged values.yaml. This makes the update process straightforward and allows administrators to easily roll out changes.

For example, create a yaml file with ONLY values that you wish to modify. Undeclared values will use defaults:

storageClassName: my-custom-storageclass
licenseKey: <CONTENTS OF LICENSE FILE>

Then specify your file in the helm install command or ArgoCD config:

$ helm install --values my_values_file.yaml

The following values should be set before installation:

  • storageClassName (required) - Set it to the storage class name that should be used to provision PVs.

  • licenseKey (required) - A valid license is required to use CloudCasa. To set the license key from the license key file: --set licenseKey=$(cat <licenseKeyFilePath>).

  • serviceType (Optional) - By default, the CloudCasa service that needs to be exposed outside of the cluster is created with type “NodePort” so it will not be accessible externally. Set this parameter to any other allowed type, such as “LoadBalancer”. For more details, see Networking.

  • Certificate parameters (required) - Default settings use self-signed issuer (created above) with cert-manager. To change issuer or supply your own TLS secrets, see Certificate Configuration.

  • Auth parameters (required) - Especially auth.allowedUsers, auth.allowedUserGroups, auth.siteadminUsers, auth.siteadminUserGroups. For details, see Authentication.

  • SMTP parameters (optional) - These are required for CloudCasa to send notifications, user invite emails, etc. For details, see Email Configuration. Without this config no emails can be sent.

  • MongoDB connection parameters (optional) - For details, see MongoDB Setup.

  • On OpenShift 4.14+ Set “scc.enabled” to “true” to create “securityContextConstraint” resources. You can also set “scc.priority”. It is set to 15 by default.

  • Miscellaneous

    • Set “ui.enable_cloudaccounts” to “false” to hide cloud accounts functionality (including “Databases” tab).

Installing CloudCasa Server using Helm

  • Create the “cloudcasa-server” namespace (kubectl create ns cloudcasa-server). If your organization requires any labels or policies to be set on a new namespace, you can do so at this time.

  • (Optional) Create the “cloudcasa-mongo” namespace (kubectl create ns cloudcasa-mongo). You may want to create the namespace explicitly if you need to set some labels or policies on it before proceeding with the installation. If you do create the namespace, make sure you set the helm parameter mongo.createNamespace to false.

  • Set up a certificate issuer or create your own TLS secrets (See Certificate Configuration). Here is one way of creating a “self signed” issuer with cert-manager:

    kubectl -n cloudcasa-server apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

EOF

    Note: If you are using self-signed certificate, see Notes about cluster agent installation.

  • Create an image pull secret called “myregistrykey” in the cloudcasa-server namespace (using cloudcasa ACR credentials). Here is one way of doing it:

    kubectl create secret docker-registry myregistrykey \
 --docker-server=https://cloudcasa.azurecr.io \
 --docker-username=<username> \
 --docker-password=<password> \
 --namespace=cloudcasa-server

  • Add CloudCasa helm repo:

    helm repo add cloudcasa https://helm.cloudcasa.io --username <USERNAME> --password <PASSWORD>

  • Install the CloudCasa server by running “helm install” command.:

    $ helm install cloudcasa-server cloudcasa/cloudcasa-server --create-namespace \
    --namespace cloudcasa-server --wait --timeout 15m0s

    In case of air-gapped installation, you need to use CloudCasa helm chart file directly, like so:

    $ helm install cloudcasa-server <CLOUDCASA-CHART-TGZ> --create-namespace \
    --namespace cloudcasa-server --wait --timeout 15m0s

  • Once all CloudCasa deployments are running, you need to make sure that the service “amds-envoy-grpcapi” can be accessed from outside the cluster, either by IP or by DNS name. Once that is done, set the value “amdsEnvoyUrl” to the external address and run helm upgrade, for example:

    echo "amdsEnvoyUrl: <CLOUDCASA-ENVOY-EXTERNAL-IP>" >> my_values_file.yaml

helm upgrade cloudcasa-server cloudcasa/cloudcasa-server --wait --create-namespace \
    --namespace cloudcasa-server \
    --values my_values_file.yaml

  • If you are using cert-manager: You need to refresh TLS secrets to update them with DNS names. This can be done by deleting the TLS secret and restarting the amds-envoy deployment, like so:

    kubectl -n cloudcasa-server delete secret amds-envoy.tls && \
kubectl -n cloudcasa-server rollout restart deployments/amds-envoy

    When the cert-manager sees that the secret is gone, it will generate a new secret from the Issuer and the amds-envoy container should start. To check the status of the certificate, run:

    kubectl -n cloudcasa-server describe cert amds-envoy

That’s it! You can now access the web UI using the IP or DNS name of the “amds-envoy-grpcapi” service. The URL will be https://<CLOUDCASA-ENVOY-EXTERNAL-IP>. Note that if you used the self-signed issuer above, you will see certificate warnings twice.

Please see Post-installation tasks for details on adding clusters and defining backups.

Installing CloudCasa Server using ArgoCD

The Helm chart can be installed using ArgoCD to manage the lifecycle of cloudcasa-server resources. We recommend reading through the Helm installation steps as the ArgoCD installation is very similar, except helm upgrade/install is replaced by Argo “sync”.

  • Add cloudcasa-server helm chart to ArgoCD (2 options):

    • Download the tgz file and upload to ArgoCD

    • Add the Helm repo directly with the supplied credentials:

      • Go to Settings → Repositories → Connect Repo

        Connection Method: HTTP/HTTPS
Type: Helm
URL: https://helm.cloudcasa.io
Name: cloudcasa-server
Username/Password: Supplied by CC Team
Check “Skip server verification” to skip TLS validation

  • Create a Git repo to store all relevant yaml files (helm values, static manifests). Add the git repo to ArgoCD. Example repo structure:

      cloudcasa-server-argo-app/
├── cc_custom_values.yaml
└── cc_static_manifests.yaml

  • Define any resources that must be created before install as static yaml manifests. Any resources which must be created before install should have the annotation argocd.argoproj.io/sync-wave: “-1”.

    For example, the cloudcasa-server namespace and docker registry key can be defined here.

    apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: cloudcasa-server
---
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: cloudcasa-mongo
---
apiVersion: v1
data:
  .dockerconfigjson: <PLACEHOLDER>
kind: Secret
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: myregistrykey
  namespace: cloudcasa-server
type: kubernetes.io/dockerconfigjson
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  name: selfsigned-issuer
  namespace: cloudcasa-server
spec:
  selfSigned: {}

  • Update your helm values file with the desired configuration. There is one value which must be set for ArgoCD sync to work. Ensure the value mongo.createNamespace is false. (For ArgoCD to work, the mongo namespace should be defined in static yamls, otherwise it will be recreated by Argo on each sync).

    mongo:
  createNamespace: false

  • Create the Argo Application. At this point, static_yamls and values file should be available in Git.

    apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cloudcasa-server-app
  namespace: argocd
spec:
  project: default
  sources:
    # Source 1: Static bootstrap manifests (deployed first)
    - repoURL: git@github.com:example/cloudcasa-server-argo-app.git
      targetRevision: master
      path: cloudcasa-server-argo-app
    # Source 2: Helm chart
    - repoURL: https://helm.cloudcasa.io
      chart: cloudcasa-server
      targetRevision: 1.6.2
      helm:
        valueFiles:
          - $bootstrap/cloudcasa-server-argo-app/cc_values.yaml
    # Source 3: Git repo for values reference only
    - repoURL: git@github.com:example/cloudcasa-server-argo-app.git
      targetRevision: master
      ref: bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: cloudcasa-server

  • Deploy the ArgoCD Application with kubectl or Argo UI:

    kubectl -n argocd apply -f cloudcasa-server-argo-app.yaml

  • Sync the CloudCasa Server Application

    • At this point the Application should be visible in ArgoCD UI. With the example config above, the app will not Sync or deploy any changes automatically. To install or upgrade, manually trigger a Sync.

    • Trigger a sync and resources will be created, starting with static manifests. The sync will move on to the main Helm chart and perform upgrade with the given values. When all is done, amds-install-helper pods will run to complete setup tasks.

    • ArgoCD has no concept of install vs upgrade, just sync. So the post-upgrade amds-install-helper may run on first sync.

Upgrading from an Earlier Version

If you are upgrading to 1.6.3 or later from pre-1.6.3 versions, see Upgrade to 1.6.3+.

Updating Values

Releases may add or change Helm values, which is why we recommend using a custom values file instead of editing values.yaml. If a new value is added, the default can be read from the packaged values.yaml

If you don’t already have a custom values file and want to copy your existing install’s values, you can use helm get values to display them:

helm -n cloudcasa-server get values cloudcasa-server

# USER-SUPPLIED VALUES:
# storageClassName: my-custom-storageclass
# ...

Updating Containers

Cloudcasa-server also uses Helm values for container image tags. The values.yaml file for each release includes the latest image tags - so as long as you are not overriding the images.<component>Tag values, containers will be updated automatically during helm upgrade.

If you are using a custom container registry you will need to download the latest images for the release and add them to your registry. The list of images for every release is always uploaded to the CloudCasa support portal. For more details see Custom Container Registry Setup.

Updating the Helm Chart

New Helm chart releases are available on the CloudCasa support portal or via the Helm repo at helm.cloudcasa.io. To get the latest helm chart from the repo:

helm repo update cloudcasa

You can also check the latest release by listing versions:

helm search repo cloudcasa --versions

After updating the repo and optionally updating your values file, run the helm upgrade command to apply the changes:

helm upgrade cloudcasa-server cloudcasa/cloudcasa-server --wait \
      --namespace cloudcasa-server \
      --values my_values_file.yaml

If all goes well, you should see the new version displayed under the logo in CloudCasa Dashboard.

Also, if the new version includes any agent changes, agents on clusters that haven’t disabled automatic updates will be updated during next update cycle (by default, every 24 hours). If you need to update the agent right away, go to cluster’s “Edit” page and click on “Re-deploy agent” button.

Updating in ArgoCD

Similar to Helm upgrade, you can pull the latest Helm chart directly from the Helm repo with credentials, or upload the desired Helm chart tgz file to your Argo project. Argo “sync” takes the place of Helm upgrade.

Certificate Configuration

The CloudCasa server requires TLS for internal and external services. There are a few ways to set up TLS:

  • Using cert-manager (recommended)

  • Using an Issuer resource with a private CA

  • Manually creating certificates and TLS secrets

Using cert-manager

Install cert-manager and create an Issuer. Set the following Helm parameters accordingly:

  • certificates.issuerName - Name of the existing Issuer resource. Default value: “selfsigned-issuer”.

  • certificates.issuerType - Issuer resource type. Possible values: “Issuer” or “ClusterIssuer”. Default value: “Issuer”.

Using a private CA

Create an issuer resource for your private CA and set the following Helm parameters accordingly:

  • certificates.issuerName - Name of the issuer resource

  • certificates.issuerType - Issuer resource type. Possible values: “Issuer” or “ClusterIssuer”. Default value: “Issuer”.

  • certificates.issuerGroup - issuer group, ex “cert-manager.io”

Manually creating TLS Secrets

Custom certificates can be used without an Issuer or cert-manager by creating TLS secrets in the cloudcasa-server namespace. For information about creating TLS secrets, see Kubernetes docs: TLS Secrets.

After the secrets have been created in cloudcasa-server, set the following Helm parameters in order to use them. Note that all secret name parameters are required.

  • certificates.createTlsSecrets - Set to false to use your own TLS secrets

  • certificates.envoyTlsSecretName - Secret name for amds-envoy deployment. This is the only external-facing service, so the certificate’s dnsName or caName should match amdsEnvoyUrl

  • certificates.frontendTlsSecretName - Secret name for amds-frontend deployment. Internal-facing service.

  • certificates.dexTlsSecretName - Secret name for amds-dex deployment. Internal-facing service.

Example: Cert-manager with a custom domain

In this example, we want to host CloudCasa at a custom domain using AWS route53 and using “letsencrypt” to verify ownership of the domain and automatically create TLS secrets.

First create a “letsencrypt” Issuer. We will automatically validate domain ownership using cert-manager’s dns01 solver to authenticate with route53:

kubectl -n cloudcasa-server apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: example@example.com
    preferredChain: ""
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyIDSecretRef:
            name: cert-manager-keys
            key: access-key-id
          secretAccessKeySecretRef:
            name: cert-manager-keys
            key: secret-access-key
EOF

Now follow the basic setup instructions for the CloudCasa server. When it’s time to run helm install…, add the certificate options below. Set amdsEnvoyUrl to the desired domain name. In this case the AWS role we supplied in the Issuer has permissions to manage this domain in AWS. In this example we will also set serviceType` to provision a load balancer.

--set certificates.issuerName=letsencrypt-prod \
--set certificates.issuerType=Issuer \
--set certificates.issuerGroup=cert-manager.io \
--set amdsEnvoyUrl=cloudcasa.example.com \
--set serviceType=LoadBalancer

helm install will create the cert resources and cert-manager will attempt to validate domain ownership. Once that is done, the TLS secrets will be created and the amds-envoy deployment will start.

To complete the process, we need to create an “A” record in route53 for this domain. To do this, get the load balancer IP by checking the external IP of the service:

kubectl -n cloudcasa-server get svc amds-envoy-grpcapi

Create a DNS “A” record for cloudcasa.example.com pointing to the external IP. After DNS propagation, you should see the CloudCasa UI at this address with a valid certificate.

MongoDB Setup

By default, CloudCasa installs MongoDB in the namespace “cloudcasa-mongo” and uses it to store catalog data. If this is acceptable, no other database configuration is required.

If you want to use your own MongoDB instance, please use the following Helm parameters. If you are planning to install MongoDB yourself, make sure it is not installed in the cloudcasa-server namespace.

Helm Parameters:

  • mongo.createStatefulset - Set to false to use your own Mongo instance. Default value is “true”.

  • mongo.createNamespace - Set to false to manually manage the mongo namespace without Helm. Default “true”.

  • mongo.statefulsetNamespace - Namespace to create/manage for mongo statefulset. Default “cloudcasa-mongo”

  • mongo.pv.size - Size of Database PVC to create for mongo statefulset

  • mongo.url - URL used by the CloudCasa application to connect to MongoDB. See MongoDB Connection Strings for more details.

  • mongo.authSecretName - Name of secret in cloudcasa-server which contains username and password data fields for connecting to the provided MongoDB URL. This must be created manually if using an external MongoDB instance. Not required for default setup. Default “mongo-secrets”

  • mongo.tls.enabled - Enable to use TLS when connecting to MongoDB. Requires certificateKeyFile to be set. Not required for default setup.

  • (optional) mongo.tls.certificateKeyFile - Path to the client certificate file to present to the server for TLS. See MongoDB TLS Configuration.

Default Installation

By default, Helm will create a MongoDB StatefulSet and Persistent Volume in its own namespace: cloudcasa-mongo. Database admin credentials are generated during the install, and can be found in the mongo-credentials secret in this namespace. The container has no external port so it can only be reached from within the cluster. It is highly recommended to limit access to the namespace cloudcasa-mongo` using Kubernetes RBAC.

Changing admin credentials is not recommended, but can be done by editing the mongo-credentials secret, running helm upgrade, and restarting the mongo pod. Note that Helm does not save the password value so removing this secret will trigger a password reset on the next Helm upgrade.

Using External MongoDB (optional)

It is also possible to use an external MongoDB instance as long as CloudCasa is able to reach the instance. This includes cloud mongo services (e.g. Atlas) or other instances reachable on your network. To do this you will need a connection URL and username/password pair to authenticate.

  • Set mongo.url to the base connection string for your mongo instance, excluding credentials or TLS options. Ex: “mongodb://localhost:27017”

  • Create a secret in cloudcasa-server with data entries for the mongo username and password. User should have permissions to create and manage a new database. Example:

    kubectl -n cloudcasa-server create secret generic mongo-secrets --from-literal=username=mongo_user \
    --from-literal=password=mongo_user_password

  • Ensure mongo.authSecretName is set to the secret from the previous step.

  • If TLS is required, set mongo.tls.enabled. Download the key bundle from the provider and set mongo.tls.certificateKeyFile to its filepath.

RBAC

This section provides configuration options to setup role-based access control (RBAC) for users. Roles can be created and assigned to users to control user permissions. User groups can be created to assign roles to multiple users. See CloudCasa documentation for Roles and User Groups for details.

CloudCasa server can obtain groups for a user from the configured authentication provider. To assign a role to an external group, a corresponding user group needs to be created in CloudCasa with the same name. Follow these steps to associate an external group to a CloudCasa user group:

  1. From the CloudCasa UI, navigate to Configuration -> Roles and click Add Role. Add permissions for the role and click Create.

  2. Navigate to Configuration -> User Groups and click Add user group. The name should be the same as the name of the external group. Assign role(s) to the user group and click Save.

Note that selecting users for the CloudCasa user group is not required as the users will automatically be assigned the roles from the user group once they log in.

Email Configuration

CloudCasa needs to send emails in some cases, such as when a user is invited to join an organization or to notify users of job failures. CloudCasa supports two methods of sending email: SMTP and Azure Communication Service.

For the SMTP method, configure the following parameters:

  • emails.enabled. Set it to true.

  • emails.method. Set it to smtp.

  • emails.smtp.startTLS. Controls whether TLS is used for SMTP connection. By default, it is set to true.

  • emails.smtp.password

  • emails.smtp.sender_email

  • emails.smtp.host

  • emails.smtp.port. E.g. “587”

For the Azure Communication Service method, configure the following parameters:

  • emails.enabled. Set it to true.

  • emails.method. Set it to azureacs.

  • emails.azureacs.username

  • emails.azureacs.password

  • emails.azureacs.sender_email

Container Registries

CloudCasa Images

By default, cloudcasa-server uses two container registries:

images:
  registry: cloudcasa.azurecr.io/catalogicsoftware
  agentRegistry: cloudcasaAgent.azurecr.io/catalogicsoftware

  • images.registry: cloudcasa.azurecr.io/catalogicsoftware

    Registry for CloudCasa server images. Authentication required.

    To create a pull secret for use with the default registry:

    kubectl create secret docker-registry myregistrykey \
 --docker-server=https://cloudcasa.azurecr.io \
 --docker-username=<username> \
 --docker-password=<password> \
 --namespace=cloudcasa-server

  • images.agentRegistry: cloudcasaAgent.azurecr.io/catalogicsoftware

    Registry for CloudCasa Agent images. This repository allows anonymous pulls so that all images required for agent installation are publicly available. This means that as long as a cluster is connected to the internet (and CloudCasa), it can install the CloudCasa client.

Other Images

CloudCasa also uses some publicly available images from Dockerhub:

docker.io/mongo

Air-gapped installation

Accessing the Helm Chart

If installing from a machine that does not have access to the helm repository (helm.cloudcasa.io), you must instead download the helm chart from the CloudCasa Support Portal at https://support.cloudcasa.io.

To download the helm chart:

  1. Log into the CloudCasa support portal using your email (you will receive a temporary access code)

  2. Navigate to the relevant release

  3. Find “Files” section (under “Change log” and “Images”)

  4. Click on cloudcasa-server-<version>.tgz to download the chart

Now use the .tgz file directly in helm install/upgrade commands, replacing cloudcasa/cloudcasa-server with the path to the downloaded .tgz file.

helm install cloudcasa-server cloudcasa-server-<version>.tgz --namespace cloudcasa-server ...

Custom Container Registry Setup

In order to install CloudCasa in an air-gapped environment, all the images used by CloudCasa need to be copied to a container registry of your choice (the full list of images can be found in CloudCasa support portal). Note that wherever the images are copied, all images should be in the same folder in the registry (configurable with helm value “images.registry” - default is “catalogicsoftware”).

For example, to use a custom registry “myregistry.io”, all images should be copied to “myregistry.io/catalogicsoftware/<image>”. The images in your repository should look something like this:

myregistry.io/catalogicsoftware/amds-apiserver-selfhosted
myregistry.io/catalogicsoftware/amds-dex-selfhosted
myregistry.io/catalogicsoftware/fluentd-kubernetes-daemonset
myregistry.io/catalogicsoftware/mongo

Custom Container Registry Usage

To use a custom registry some values must be set/overridden:

  • images.userContainerRegistry: The URL of the container registry, including the root directory. Ex: “myregistry.io”

  • images.registry: Registry path used for all cloudcasa-server images.

  • images.agentRegistry: Optional registry path used for CloudCasa agent images (publicly available to all clusters) To use a custom registry set images.agentRegistry to null, otherwise the default value will pull from the public CloudCasa repository

Below is an example of values for a custom registry. These values match the example above, images would be pulled from “myregistry.io/catalogicsoftware/<image>”:

images:
  userContainerRegistry: myregistry.io
  imagePullPolicy: IfNotPresent
  registry: catalogicsoftware
  agentRegistry:

Notes

  • Normally, the cluster agent is installed by running a kubectl command such as the following:

    $ kubectl apply -f <AGENT-YAML>

    The correct command is displayed in the UI when a cluster resource is created. If you used a self-signed certificate issuer while installing cloudcasa server, you need to use slightly different command:

    $ curl -k <AGENT-YAML> | kubectl apply -f -

  • CloudCasa supports backup and restore of AWS RDS databases. (See Databases for details). However, if you are not planning to use this functionality, “Databases” tab can be hidden from the menu by setting the helm parameter ui.enable_databases to false.

Upgrade to 1.6.3+

If you are upgrading to 1.6.3 or later from pre-1.6.3 versions, you will need to do catalog restore after installing 1.6.3 (or later). This is because, in release 1.6.3, CloudCasa switched to a recent version of Mongo so old data cannot be seamlessly migrated to this version. Please follow the instructions below to properly upgrade Mongo:

1. Ensure that you have configured Helm values for catalog backup. If not, follow the catalog backup instructions to set up a backup location for MongoDB in S3 or Azure (see Catalog backup/restore).

  1. Check that no CloudCasa jobs are running. Make a local copy of any “amdsevents” resources present.:

    kubectl -n cloudcasa-server get amdsevents -o yaml >> amdsevents-backup.yaml

  2. Trigger a catalog backup job.:

    kubectl -n cloudcasa-server create job --from=cronjob/catalogbackup before-mongo-upgrade

    Check the logs of the new “before-mongo-upgrade” pod and confirm it succeeds. If there are any issues connecting to your backup target, an error will be printed in the pod logs.

4. Exec into an amds-apiserver pod and use casactl to find the ID of the snapshot created in the previous step.

Get the name of amds-apiserver pod and exec:

kubectl -n cloudcasa-server get pods -l amds.component=apiserver
kubectl -n cloudcasa-server exec -it <POD NAME> -- bash

Inside the apiserver container, run the appropriate casactl command using your mongo.backupLocation Helm values. Note “repo-prefix” must match the Helm value exactly, including trailing “/” if applicable.

For S3:

casactl [--insecure] catalog list-from-s3 --repo-password <password> --bucket <bucket> \
--access-key <key>  --secret-key <secret> --repo-prefix <prefix> \
[--region <region>]  [--endpoint <endpoint>]

For Azure:

casactl catalog list-from-azure --repo-password <password> --client-id <id> \
--client-secret <secret> --tenant-id <tenant> --region <region> \
--resgroup <resgroup> --storage-account <storage> --repo-prefix <prefix>

While this command is running a pod will be created named “catalog-list-snapshots…”. You can view logs of this pod to see any error messages.

Once complete, a list of snapshot IDs with their timestamps will be displayed. Save the ID of the most recent snapshot. This will be used during the restore.

  1. Create a manual backup as a failsafe using “mongodump”.

    Exec into the mongo pod:

    kubectl -n cloudcasa-mongo exec -it mongo-0 -- bash

    Inside the mongo container, run “mongodump” to save the catalog locally:

    mongodump -u $MONGO_ADMIN_USERNAME -p $MONGO_ADMIN_PASSWORD --db="amds-resources" \
--gzip --out="/tmp/cc_mongodump_manual" --excludeCollection="sites" \
--excludeCollection="licenses" --authenticationDatabase="admin"

    Exit the mongo container and copy the archive locally:

    kubectl -n cloudcasa-mongo cp mongo-0:/tmp/cc_mongodump_manual cc_mongodump_manual

    Inside the cc_mongodump_manual folder there should be a folder amds-resources containing bson.gz files for each collection. Keep these as a backup in case of any upgrade issues.

  2. Scale down all Cloudcasa deployments except amds-apiserver.:

    kubectl -n cloudcasa-server scale deployment -l "amds.component!=apiserver" --replicas=0

7. At this point there should be 2 backups of the Mongo catalog (S3/Azure and local). Now delete the Mongo namespace, and ensure the PV is deleted (not just PVC!). This will delete all CloudCasa server data in the cluster so confirm the backups are complete!

kubectl delete ns --force cloudcasa-mongo

  1. Run Helm Upgrade with cloudcasa-server 1.6.3+ to install the new version of Mongo.

    Note

    The new version of Mongo will not work with file storage such as NFS/SMB/Azure Files. It requires a PVC backed by a block storage driver. To be clear, PVC’s “volumeMode” should still be “FileSystem”. If you need to create a new PVC at this point with a different storage class, you can create it manually before running Helm upgrade.

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    amds.component: mongo
  name: mongo-vol-mongo-0
  namespace: cloudcasa-mongo
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 16Gi
  storageClassName: my-block-storageclass

    The Helm upgrade command will fail, but we only need to let it run until the “cloudcasa-mongo” namespace, pods, and PVC are recreated. You can watch the mongo namespace in another terminal during upgrade (kubectl -n cloudcasa-mongo get pods -w) and Ctrl+C to cancel the upgrade once the pods are created.

    If you have overridden default values, make sure “images.mongoTag” is set to 8.2.4+.

    If mongo pods are in InitError state, its possible the old pod or PVC was not fully deleted. (Mongo 8 is not compatible with volumes created by Mongo 3). You can retry this process by scaling down deployments, deleting the namespace, and running helm upgrade again.

    Mongo pods should pull the new image and eventually reach Running state.

  2. Once mongo pods have been reinitialized, scale down all deployments except apiserver again.

    kubectl -n cloudcasa-server scale deployment -l "amds.component!=apiserver" --replicas=0

  3. Exec into the newest amds-apiserver pod and use casactl to restore the Mongo snapshot.

    Get the name of amds-apiserver pod and exec:

    kubectl -n cloudcasa-server get pods -l amds.component=apiserver
kubectl -n cloudcasa-server exec -it <POD NAME> -- bash

    Inside the apiserver container, run the appropriate casactl command using the snapshot ID from step 4.

    For S3:

    casactl [--insecure] catalog restore-from-s3 --repo-password <password> --bucket <bucket> \
--access-key <key>  --secret-key <secret> --repo-prefix <prefix> \
[--region <region>]  [--endpoint <endpoint>] <SNAPSHOT ID>

    For Azure:

    casactl catalog restore-from-azure --repo-password <password> --client-id <id> \
--client-secret <secret> --tenant-id <tenant> --region <region> \
--resgroup <resgroup> --storage-account <storage> --repo-prefix <prefix> <SNAPSHOT ID>

    You will be prompted to confirm the DB overwrite. Then a “catalog-restore…” pod will be created where you can view logs. On success, there will be a message in both the casactl command output and catalog-restore pod.

11. After catalog-restore succeeds, run Helm upgrade one last time to scale deployments back up and allow amds-install-helper pods to complete.

  1. Verify that cluster connections are back and that policy configuration is restored properly, by running the following commands in apiserver pod:

    $ casactl server cluster-state
$ casactl policy check