<!--- Generated file, DO NOT EDIT. Source: build/templates/README.md --->
# CockroachDB Helm Chart
[CockroachDB](https://github.com/cockroachdb/cockroach) - the open source, cloud-native distributed SQL database.
## Documentation
Below is a brief overview of operating the CockroachDB Helm Chart and some specific implementation details. For additional information on deploying CockroachDB, please see:
Note that the documentation requires Helm 3.0 or higher.
## Prerequisites Details
* Kubernetes 1.8
* PV support on the underlying infrastructure (only if using `storage.persistentVolume`). [Docker for windows hostpath provisioner is not supported](https://github.com/cockroachdb/docs/issues/3184).
* If you want to secure your cluster to use TLS certificates for all network communication, [Helm must be installed with RBAC privileges](https://helm.sh/docs/topics/rbac/) or else you will get an "attempt to grant extra privileges" error.
To install the chart with the release name `my-release`:
```shell
helm install my-release cockroachdb/cockroachdb
```
Note that for a production cluster, you will likely want to override the following parameters in [`values.yaml`](values.yaml) with your own values.
-`statefulset.resources.requests.memory` and `statefulset.resources.limits.memory` allocate memory resources to CockroachDB pods in your cluster.
-`conf.cache` and `conf.max-sql-memory` are memory limits that we recommend setting to 1/4 of the above resource allocation. When running CockroachDB, you must set these limits explicitly to avoid running out of memory.
-`storage.persistentVolume.size` defaults to `100Gi` of disk space per pod, which you may increase or decrease for your use case.
-`storage.persistentVolume.storageClass` uses the default storage class for your environment. We strongly recommend that you specify a storage class which uses an SSD.
-`tls.enabled` must be set to `yes`/`true` to deploy in secure mode.
For more information on overriding the `values.yaml` parameters, please see:
If you wish to supply the certificates to the nodes yourself set `tls.certs.provided` to `yes`/`true`. You may want to use this if you want to use a different certificate authority from the one being used by Kubernetes or if your Kubernetes cluster doesn't fully support certificate-signing requests. To use this, first set up your certificates and load them into your Kubernetes cluster as Secrets using the commands below:
> Note: The subject alternative names are based on a release called `my-release` in the `my-namespace` namespace. Make sure they match the services created with the release during `helm install`
If your certificates are stored in tls secrets such as secrets generated by cert-manager, the secret will contain files named:
*`ca.crt`
*`tls.crt`
*`tls.key`
Cockroachdb, however, expects the files to be named like this:
*`ca.crt`
*`node.crt`
*`node.key`
*`client.root.crt`
*`client.root.key`
By enabling `tls.certs.tlsSecret` the tls secrets are projected on to the correct filenames, when they are mounted to the cockroachdb pods.
#### Cert-manager
If you wish to supply certificates with [cert-manager][3], set
*`tls.certs.certManager` to `yes`/`true`
*`tls.certs.certManagerIssuer` to an IssuerRef (as they appear in certificate resources) pointing to a clusterIssuer or issuer, you have set up in the cluster
Example issuer:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: cockroachdb-ca
namespace: cockroachdb
data:
tls.crt: [BASE64 Encoded ca.crt]
tls.key: [BASE64 Encoded ca.key]
type: kubernetes.io/tls
---
apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
name: cockroachdb-cert-issuer
namespace: cockroachdb
spec:
ca:
secretName: cockroachdb-ca
```
## Upgrading the cluster
### Chart version 3.0.0 and after
Launch a temporary interactive pod and start the built-in SQL client:
> If you are running in secure mode, you will have to provide a client certificate to the cluster in order to authenticate, so the above command will not work. See [here](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/client-secure.yaml) for an example of how to set up an interactive SQL shell against a secure cluster or [here](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/example-app-secure.yaml) for an example application connecting to a secure cluster.
Set `cluster.preserve_downgrade_option`, where `$current_version` is the CockroachDB version currently running (e.g., `19.2`):
```sql
> SET CLUSTER SETTING cluster.preserve_downgrade_option = '$current_version';
```
Exit the shell and delete the temporary pod:
```sql
> \q
```
Kick off the upgrade process by changing the new Docker image, where `$new_version` is the CockroachDB version to which you are upgrading:
```shell
helm upgrade my-release cockroachdb/cockroachdb \
--set image.tag=$new_version \
--reuse-values
```
Kubernetes will carry out a safe [rolling upgrade](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) of your CockroachDB nodes one-by-one. Monitor the cluster's pods until all have been successfully restarted:
Resume normal operations. Once you are comfortable that the stability and performance of the cluster is what you'd expect post-upgrade, finalize the upgrade:
Due to a change in the label format in version 3.0.0 of this chart, upgrading requires that you delete the StatefulSet. Luckily there is a way to do it without actually deleting all the resources managed by the StatefulSet. Use the workaround below to upgrade from charts versions previous to 3.0.0:
Get the new labels from the specs rendered by Helm:
Verify that no pod is deleted and then upgrade as normal. A new StatefulSet will be created, taking over the management of the existing pods and upgrading them if needed.
### See also
For more information about upgrading a cluster to the latest major release of CockroachDB, see [Upgrade to CockroachDB v21.1](https://www.cockroachlabs.com/docs/stable/upgrade-cockroach-version.html).
Note that there are some backward-incompatible changes to SQL features between versions 20.2 and 21.1. For details, see the [CockroachDB v22.2.3 release notes](https://www.cockroachlabs.com/docs/releases/v22.2.3.html#backward-incompatible-changes).
| `storage.persistentVolume.storageClass` | PersistentVolume class | `""` |
| `storage.persistentVolume.labels` | Additional labels of PersistentVolumeClaim | `{}` |
| `storage.persistentVolume.annotations` | Additional annotations of PersistentVolumeClaim | `{}` |
| `init.labels` | Additional labels of init Job and its Pod | `{"app.kubernetes.io/component": "init"}` |
| `init.jobAnnotations` | Additional annotations of the init Job itself | `{}` |
| `init.annotations` | Additional annotations of the Pod of init Job | `{}` |
| `init.affinity` | [Affinity rules][2] of init Job Pod | `{}` |
| `init.nodeSelector` | Node labels for init Job Pod assignment | `{}` |
| `init.tolerations` | Node taints to tolerate by init Job Pod | `[]` |
| `init.resources` | Resource requests and limits for the Pod of init Job | `{}` |
| `tls.enabled` | Whether to run securely using TLS certificates | `no` |
| `tls.serviceAccount.create` | Whether to create a new RBAC service account | `yes` |
| `tls.serviceAccount.name` | Name of RBAC service account to use | `""` |
| `tls.copyCerts.image` | Image used in copy certs init container | `busybox` |
| `tls.certs.provided` | Bring your own certs scenario, i.e certificates are provided | `no` |
| `tls.certs.clientRootSecret` | If certs are provided, secret name for client root cert | `cockroachdb-root` |
| `tls.certs.nodeSecret` | If certs are provided, secret name for node cert | `cockroachdb-node` |
| `tls.certs.tlsSecret` | Own certs are stored in TLS secret | `no` |
| `tls.certs.selfSigner.enabled` | Whether cockroachdb should generate its own self-signed certs | `true` |
| `tls.certs.selfSigner.caProvided` | Bring your own CA scenario. This CA will be used to generate node and client cert | `false` |
| `tls.certs.selfSigner.caSecret` | If CA is provided, secret name for CA cert | `""` |
| `tls.certs.selfSigner.minimumCertDuration` | Minimum cert duration for all the certs, all certs duration will be validated against this duration | `624h` |
| `tls.certs.selfSigner.caCertDuration` | Duration of CA cert in hour | `43824h` |
| `tls.certs.selfSigner.caCertExpiryWindow` | Expiry window of CA cert means a window before actual expiry in which CA cert should be rotated | `648h` |
| `tls.certs.selfSigner.clientCertDuration` | Duration of client cert in hour | `672h |
| `tls.certs.selfSigner.clientCertExpiryWindow` | Expiry window of client cert means a window before actual expiry in which client cert should be rotated | `48h` |
| `tls.certs.selfSigner.nodeCertDuration` | Duration of node cert in hour | `8760h` |
| `tls.certs.selfSigner.nodeCertExpiryWindow` | Expiry window of node cert means a window before actual expiry in which node certs should be rotated | `168h` |
| `tls.certs.selfSigner.rotateCerts` | Whether to rotate the certs generate by cockroachdb | `true` |
| `tls.certs.selfSigner.readinessWait` | Wait time for each cockroachdb replica to become ready once it comes in running state. Only considered when rotateCerts is set to true | `30s` |
| `tls.certs.selfSigner.podUpdateTimeout` | Wait time for each cockroachdb replica to get to running state. Only considered when rotateCerts is set to true | `2m` |
| `tls.certs.certManager` | Provision certificates with cert-manager | `false` |
| `tls.certs.certManagerIssuer.group` | IssuerRef group to use when generating certificates | `cert-manager.io` |
| `tls.certs.certManagerIssuer.kind` | IssuerRef kind to use when generating certificates | `Issuer` |
| `tls.certs.certManagerIssuer.name` | IssuerRef name to use when generating certificates | `cockroachdb` |
| `tls.certs.certManagerIssuer.clientCertDuration` | Duration of client cert in hours | `672h` |
| `tls.certs.certManagerIssuer.clientCertExpiryWindow` | Expiry window of client cert means a window before actual expiry in which client cert should be rotated | `48h` |
| `tls.certs.certManagerIssuer.nodeCertDuration` | Duration of node cert in hours | `8760h` |
| `tls.certs.certManagerIssuer.nodeCertExpiryWindow` | Expiry window of node certificates means a window before actual expiry in which node certs should be rotated. | `168h` |
| `tls.selfSigner.image.repository` | Image to use for self signing TLS certificates | `cockroachlabs-helm-charts/cockroach-self-signer-cert`|
| `tls.selfSigner.image.tag` | Image tag to use for self signing TLS certificates | `0.1` |
> **Tip**: You can use the default [values.yaml](values.yaml)
## Deep dive
### Connecting to the CockroachDB cluster
Once you've created the cluster, you can start talking to it by connecting to its `-public` Service. CockroachDB is PostgreSQL wire protocol compatible, so there's a [wide variety of supported clients](https://www.cockroachlabs.com/docs/install-client-drivers.html). As an example, we'll open up a SQL shell using CockroachDB's built-in shell and play around with it a bit, like this (likely needing to replace `my-release-cockroachdb-public` with the name of the `-public` Service that was created with your installed chart):
root@my-release-cockroachdb-public:26257> CREATE TABLE bank.accounts (id INT
PRIMARY KEY, balance DECIMAL);
CREATE TABLE
root@my-release-cockroachdb-public:26257> INSERT INTO bank.accounts VALUES
(1234, 10000.50);
INSERT 1
root@my-release-cockroachdb-public:26257> SELECT * FROM bank.accounts;
+------+---------+
| id | balance |
+------+---------+
| 1234 | 10000.5 |
+------+---------+
(1 row)
root@my-release-cockroachdb-public:26257> \q
Waiting for pod default/cockroach-client to terminate, status is Running
pod "cockroach-client" deleted
```
> If you are running in secure mode, you will have to provide a client certificate to the cluster in order to authenticate, so the above command will not work. See [here](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/client-secure.yaml) for an example of how to set up an interactive SQL shell against a secure cluster or [here](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/example-app-secure.yaml) for an example application connecting to a secure cluster.
### Cluster health
Because our pod spec includes regular health checks of the CockroachDB processes, simply running `kubectl get pods` and looking at the `STATUS` column is sufficient to determine the health of each instance in the cluster.
If you want more detailed information about the cluster, the best place to look is the Admin UI.
### Accessing the Admin UI
If you want to see information about how the cluster is doing, you can try pulling up the CockroachDB Admin UI by port-forwarding from your local machine to one of the pods (replacing `my-release-cockroachdb-0` with the name of one of your pods:
You should then be able to access the Admin UI by visiting <http://localhost:8080/> in your web browser.
### Failover
If any CockroachDB member fails, it is restarted or recreated automatically by the Kubernetes infrastructure, and will re-join the cluster automatically when it comes back up. You can test this scenario by killing any of the CockroachDB pods:
```shell
kubectl delete pod my-release-cockroachdb-1
```
```shell
kubectl get pods -l "app.kubernetes.io/instance=my-release,app.kubernetes.io/component=cockroachdb"
```
```
NAME READY STATUS RESTARTS AGE
my-release-cockroachdb-0 1/1 Running 0 5m
my-release-cockroachdb-2 1/1 Running 0 5m
```
After a while:
```shell
kubectl get pods -l "app.kubernetes.io/instance=my-release,app.kubernetes.io/component=cockroachdb"
```
```
NAME READY STATUS RESTARTS AGE
my-release-cockroachdb-0 1/1 Running 0 5m
my-release-cockroachdb-1 1/1 Running 0 20s
my-release-cockroachdb-2 1/1 Running 0 5m
```
You can check the state of re-joining from the new pod's logs:
```shell
kubectl logs my-release-cockroachdb-1
```
```
[...]
I161028 19:32:09.754026 1 server/node.go:586 [n1] node connected via gossip and
verified as part of cluster {"35ecbc27-3f67-4e7d-9b8f-27c31aae17d6"}
To enable NetworkPolicy for CockroachDB, install [a networking plugin that implements the Kubernetes NetworkPolicy spec](https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy#before-you-begin), and set `networkPolicy.enabled` to `yes`/`true`.
For Kubernetes v1.5 & v1.6, you must also turn on NetworkPolicy by setting the `DefaultDeny` Namespace annotation. Note: this will enforce policy for _all_ pods in the Namespace:
For more precise policy, set `networkPolicy.ingress.grpc` and `networkPolicy.ingress.http` rules. This will only allow pods that match the provided rules to connect to CockroachDB.
### Scaling
Scaling should be managed via the `helm upgrade` command. After resizing your cluster on your cloud environment (e.g., GKE or EKS), run the following command to add a pod. This assumes you scaled from 3 to 4 nodes:
```shell
helm upgrade \
my-release \
cockroachdb/cockroachdb \
--set statefulset.replicas=4 \
--reuse-values
```
Note, that if you are running in secure mode (`tls.enabled` is `yes`/`true`) and increase the size of your cluster, you will also have to approve the CSR (certificate-signing request) of each new node (using `kubectl get csr` and `kubectl certificate approve`).
