Cluster Maintenance Guides on Cozystack

Upgrading Cozystack and Post-upgrade Checks

Mon, 01 Jan 0001 00:00:00 +0000

About Cozystack Versions

Cozystack uses a staged release process to ensure stability and flexibility during development.

There are three types of releases:

Alpha, Beta, and Release Candidates (RC) – Preview versions (such as v0.42.0-alpha.1 or v0.42.0-rc.1) used for final testing and validation.
Stable Releases – Regular versions (e.g., v0.42.0) that are feature-complete and thoroughly tested. Such versions usually introduce new features, update dependencies, and may have API changes.
Patch Releases – Bugfix-only updates (e.g., v0.42.1) made after a stable release, based on a dedicated release branch.

It’s highly recommended to install only stable and patch releases in production environments.

Upgrading from v1.6 to v1.7

Mon, 01 Jan 0001 00:00:00 +0000

Cozystack v1.7 changes the behaviour of running clusters in several places. This page lists those changes for a cluster that runs v1.6.x, and the action each one needs. The upgrade itself follows the usual procedure in Upgrading Cozystack. Read this page first: several actions have to be taken before the upgrade starts, because once it runs it is too late to act on them.

Platform migrations do not run on the default variant

On every other variant the v1.7 platform release runs platform migrations 54 to 59 as a pre-upgrade hook, before any package moves to its new chart. The default variant runs none, so on such a cluster the steps those migrations would take for SeaweedFS, VM volume hotplug and HTTPCache are yours, before the upgrade.

Before you upgrade

Every node: add drbd_transport_tcp to the Talos kernel modules before the node moves to Talos v1.14. See Talos v1.14.
isp-full and isp-full-generic: check that networking.podCIDR has a prefix of /20 or shorter. See Cilium address block.
Tenant Kubernetes clusters: remove controlPlane.apiServer.extraArgs entries that name a flag the tenant control plane sets itself. See Kamaji and tenant control planes.
Tenant Kubernetes clusters: plan for the Talos workers of every pool to be replaced once. See Worker pools become KubernetesNodes resources.
Multus: decide whether the reference CNI plugins may be written to /opt/cni/bin on every node. See Multus stages CNI plugins.
FoundationDB: upgrade every cluster that runs 7.1.x to 7.3. See FoundationDB 7.1.
PostgreSQL with external: true: find clients that disable TLS. See PostgreSQL.
OpenSearch with external: true or Dashboards: see OpenSearch.
Gateway API: check the certificate served on api.<apex> and the hostnames of routes in tenant namespaces. See Gateway API.
VM instances: fix resources values that cannot size a VM. See Smaller changes.
Kafka: plan a maintenance window for every instance. See Kafka moves to KRaft.
default variant only: pin the SeaweedFS volume size limit and look for volumes hotplugged on a VM instance only. See SeaweedFS volume size limit and Declarative volume hotplug.
While the platform upgrade runs, do not edit Kubernetes resources. See Worker pools become KubernetesNodes resources.

Platform migrations

The platform migration version moves from 54, where v1.6 left it, to 60. On every variant except default, the migrations below run once each, in order, as a pre-upgrade hook of the platform release:

Cluster Scaling: Adding and Removing Nodes

Mon, 01 Jan 0001 00:00:00 +0000

How to add a node to a Cozystack cluster

Adding a node is done in a way similar to regular Cozystack installation.

Install Talos on the node, using the Cozystack’s custom-built Talos image.

Generate the configuration for the new node, using the Talm or talosctl guide.

For example, configuring a control plane node:

talm template -e 192.168.123.20 -n 192.168.123.20 -t templates/controlplane.yaml -i > nodes/nodeN.yaml

and for a worker node:

talm template -e 192.168.123.20 -n 192.168.123.20 -t templates/worker.yaml -i > nodes/nodeN.yaml

Apply the generated configuration to the node, using the Talm or talosctl guide. For example:

How to Rotate Certificate Authority

Mon, 01 Jan 0001 00:00:00 +0000

Talos sets up root certificate authorities with a lifetime of 10 years, and all Talos and Kubernetes API certificates are issued by these root CAs. In general, you almost never need to rotate the root CA certificate and key for the Talos API and Kubernetes API.

Rotation of the root CA is only needed:

when you suspect that the private key has been compromised;
when you want to revoke access to the cluster for a leaked talosconfig or kubeconfig;
once in 10 years.

Rotate CA for Talos API

To rotate the Talos CA for the management cluster, use the following command: