andata
/
rancher-partner-charts
mirror of https://git.rancher.io/partner-charts
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Base copy of rancher/charts 

Copies over the contents of rancher/charts@dev-v2.5-source following files / directories:
- .github/
- CNAME
- README.md
- scripts/
pull/39/head
Arvind Iyengar 2020-09-22 16:46:05 -07:00
parent b10d2ec665
commit eeb2a22c65
26 changed files with 755 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.github/workflows/pull-request.yaml vendored
View File

@ -2,7 +2,8 @@ name: CI-pullrequest
on: on:
pull_request: pull_request:
branches: [ main ] branches:
- dev-v2.5-source
jobs: jobs:
build: build:

11
.github/workflows/push.yml vendored
View File

@ -2,7 +2,8 @@ name: CI-push
on: on:
push: push:
branches: [ main ] branches:
- dev-v2.5-source
jobs: jobs:
build: build:
@ -12,8 +13,8 @@ jobs:
- name: Run CI scripts - name: Run CI scripts
run: |- run: |-
sudo make ci ./scripts/ci-wrapper dev-v2.5
- uses: stefanzweifel/git-auto-commit-action@v4.1.1 - name: Push charts and assets to the dev-v2.5 branch
with: run: |-
commit_message: Generated changes ./scripts/push-to-assets-branch origin dev-v2.5 && git push origin dev-v2.5

3
.gitignore vendored
View File

@ -1,2 +1,5 @@
/packages/**/charts-original /packages/**/charts-original
/packages/*/charts/charts/*.tgz
/.idea /.idea
.DS_Store
requirements.lock

1
CNAME
View File

@ -1 +0,0 @@
partner-charts.rancher.io

93
README.md
View File

@ -1,6 +1,6 @@
# partner-charts ## charts
This is an experimental project to automate system charts and adding overlays files on top of it. This is charts repo to automate system charts and adding overlays files on top of it.
Charts will have two categories: Charts will have two categories:
@ -22,6 +22,18 @@ packages/${CHART_NAME}/
values.yaml values.yaml
``` ```
For **Rancher original** charts with a crds directory, it should have the following tree structure
```text
packages/${CHART_NAME}/
package.yaml # metadata manifest to enable or disable crds generation
charts/ # regular helm chart directory
templates/
crds/
Chart.yaml
values.yaml
```
For **Rancher modified** charts, it should have the following tree structure For **Rancher modified** charts, it should have the following tree structure
```text ```text
@ -34,11 +46,13 @@ packages/${CHART_NAME}/
A regular `package.yaml` will have the following content: A regular `package.yaml` will have the following content:
```yaml ```yaml
url: https://charts.bitnami.com/bitnami/external-dns-2.20.10.tgz # url to fetch upstream chart url: https://charts.bitnami.com/bitnami/external-dns-2.20.10.tgz # url to fetch upstream chart, omit this field for a rancher original chart
packageVersion: 00 # packageVersion of modified charts, producing a $version-$packageVersion chart. For example, if istio 1.4.7 is modified with changes, rancher produces a 1.4.700 chart version that includes the modification rancher made on top of upstream charts. packageVersion: 00 # packageVersion of modified charts, producing a $version-$packageVersion chart. For example, if istio 1.4.7 is modified with changes, rancher produces a 1.4.700 chart version that includes the modification rancher made on top of upstream charts.
generateCRDChart:
enabled: true
``` ```
Here is an example of upstream chart based on git repository Here is an **example** of upstream chart based on git repository
```yaml ```yaml
url: https://github.com/open-policy-agent/gatekeeper.git # Url to fetch upstream chart from git url: https://github.com/open-policy-agent/gatekeeper.git # Url to fetch upstream chart from git
@ -46,11 +60,16 @@ subdirectory: chart/gatekeeper-operator # Sub directory for helm charts in git r
type: git # optinal, indicate that upstream chart is from git type: git # optinal, indicate that upstream chart is from git
commit: v3.1.0-beta.8 # the revision of git repo commit: v3.1.0-beta.8 # the revision of git repo
packageVersion: 00 # package version packageVersion: 00 # package version
generateCRDChart:
enabled: true
``` ```
### Example Here is an **example** of local chart with a crds directory
Todo: Add example ```yaml
generateCRDChart:
enabled: true
```
### Workflow ### Workflow
@ -74,21 +93,75 @@ The step includes:
5. Run `make CHART={CHART_NAME} clean` 5. Run `make CHART={CHART_NAME} clean`
This will clean up the `charts` directory so that it won't committed. This will clean up the `charts` directory so that it won't be committed.
This repo provides a [workflow](./.github/workflows) that automatically uploads patch files and tarball of charts. Commit will only need to update `package/${chart-name}/charts` and make sure patches are This repo provides a [workflow](./.github/workflows) that automatically uploads patch files and tarball of charts. Commit will only need to update `package/${chart-name}/charts` and make sure patches are
up-to-date with the latest chart. It also automatically build github pages to serve `index.yaml` and artifacts of charts. up-to-date with the latest chart. It also automatically build github pages to serve `index.yaml` and artifacts of charts.
### Experimental: Splitting CRDs from an upstream package into a separate package
**Note to Contributers:** This flag can only be used to create separate CRD charts if your chart meets the following three requirements:
- The chart defines one or more CRDs
- The chart is based on an upstream chart and includes a `package.yaml` (if this is not the case, you will need to manually create a separate CRD chart)
- The chart tries to install some default CRs based on the CRDs that it defines (if this is not the case, you should place the CRDs directly within the `templates/` directory of the chart; using a CRD chart is only necessary since rendering the chart will fail since the `kind` of the default CRs cannot be found in the cluster as the CRD is not installed yet).
There are cases in which upstream charts import CRDs into a cluster using the Helm 3 `crd/` directory, which allows a user to first install the CRDs before rendering the templates created by the chart. However, using this approach has certain caveats [as documented by Helm](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/), such as an inability to upgrade / delete those CRDs and or use the `--dry-run` flag on the package before installing the CRDs. As a result, it may be advised to move those CRDs into a separate chart under the `templates/` directory so that the lifecycle of those CRDs can continue to be managed by Helm.
However, in the current `rancher/charts` model, this would require deleting the CRDs from the upstream chart (which introduces significant changes to the package's patch) and maintaining a separate CRD chart that also needs to be kept up to date with the upstream chart. This poses several challenges, including but not limited to:
- Keeping the version of the Rancher chart and the CRD chart consistent
- Keeping the annotations added to the CRD chart in line with those added to the Rancher chart
- Adding a validation YAML to the Rancher chart to direct the user to install the CRD chart before installing the Rancher chart if CRDs do not currently exist on the server
- Viewing the patch between the CRDs introduced by the upstream chart and the CRDs within the CRD chart
To resolve this, `rancher/charts` has a flag that can be added to the `package.yaml` of any Rancher chart that allows you to specify `generateCRDChart.enabled=true`. When this mode is enabled, the following changes are applied during each step of the `rancher/charts` developer workflow:
1. On running `make CHART={CHART_NAME} prepare:
After running the default prepare script to pull the chart from upstream and apply the patch, a new directory called `charts-crd` is also created alongside `charts`. This will represent your new CRD chart. Any CRDs located within the Rancher chart in `charts/crd/` will be relocated to `charts-crd/templates/` and a new `charts-crd/Chart.yaml` (with chart names `{CHART_NAME}-crd`) and `charts-crd/README.md` will be generated. The `charts-crd/Chart.yaml` and `charts/Chart.yaml` will also be updated with the relevant annotations used by Rancher to auto-install the CRD chart from Dashboard.
If you are using the `generateCRDChart.assumeOwnershipOfCRDs` flag, the CRDs will instead be located in `charts-crd/crd-manifest/*` and some additional resources (ConfigMap, Jobs, and RBAC resources) will be created in the `charts-crd/templates` directory instead. This option should only be enabled if the chart is expected to be deployed in a setting where all or a subset of your CRDs may or may not already exist, in which case your CRD chart may want to assume ownership of the CRDs to prevent a failure on deploy.
In addition, a new file `charts/templates/validate-install-${CHART_NAME}-crd.yaml` will be added to your Rancher chart that is automatically configured to validate whether the CRDs that have been moved to the CRD chart are installed onto your cluster before trying to render the Rancher chart. For example, here is an error you might encounter if you try to install the Rancher chart first:
```
Error: execution error at ({CHART_NAME}/templates/validate-install-{CHART_NAME}-crd.yaml:15:5): Required CRDs are missing. Please install the {CHART_NAME}-crd chart before installing this chart.
```
See `scripts/prepare-crds` for more information on the default templates used for generating these files.
2. On making modification to either chart or running `make CHART={CHART_NAME} patch`
The experience of modifying values within the `charts` directory and making a new patch is unchanged. The same workflow also applies to the `charts-crd` directory with two caveats:
- Changes to `charts/templates/validate-install-${CHART_NAME}-crd.yaml`, `charts-crd/Chart.yaml`, and `charts-crd/README.md` will be ignored / not be shown in the patch as they are not expected to be updated
- Any changes to `charts-crd/templates/*` (`charts-crd/crd-manifest/*` if you are using the `generateCRDChart.assumeOwnershipOfCRDs` flag) will show up in the patch as if you had changed the relevant file within `charts/crd/*`.
Files added to the `overlay` directory will only overlay onto the Rancher chart, not the CRD chart.
3. On running `make CHART={CHART_NAME} clean`
This will clean up both the `charts` directory and the `charts-crd` directory so that either directory won't be committed.
4. On running `make CHART={CHART_NAME} charts`
A tarball for both the original chart and the CRD chart will be generated.
Some more considerations when migrating to using this flag:
- After adding this flag to a chart, you will have to look through the upstream chart and manually remove any CRD build specific code from the upstream chart (i.e. removing `helm.sh/hook: crd-install` from the CRD files, removing any cleanup Jobs introduced by the upstream chart to automatically delete CRDs on uninstall, etc.)
- The CRDs moved to their own chart must not contain any code that was pulled from helper templates located within the main chart. If it is found that this is necessary for any chart, please submit a feature request.
See `packages/rancher-monitoring` for an example of a chart that currently uses this flag.
### Override existing Chart ### Override existing Chart
By defauly CI script doesn't allow changes to be made against existing chart. In order to make changes you have to bump chart version. There is a backdoor method to make changes to your existing chart without having to bump version. You can delete the tar.gz file you want to override and commit the change. Here is an example of [commit](https://github.com/rancher/dev-charts/commit/8be888076487e23a24121a532d25b9bf9ea936f3). By default CI script doesn't allow changes to be made against existing chart. In order to make changes you have to bump chart version. There is a backdoor method to make changes to your existing chart without having to bump version. You can delete the tar.gz file you want to override and commit the change. Here is an example of [commit](https://github.com/rancher/charts/commit/3ec3d344c7e20eda6d2c6e0e9d33a4e00a33edfc#diff-db2aa3c5b9630208bd8568672c84f408).
### Helm repo index ### Helm repo index
To add this repo as a helm repo, run To add this repo as a helm repo, run
```text ```text
helm repo add ${repo_name} https://partner-charts.rancher.io helm repo add ${repo_name} https://charts.rancher.io
``` ```
To use a forked version of this chart repo, you can try either of these: To use a forked version of this chart repo, you can try either of these:
@ -97,7 +170,7 @@ To use a forked version of this chart repo, you can try either of these:
2. You can also setup github page to serve your tar.gz files on your forked repo. Github pages usually requires you to have this set up on [specific branches](https://help.github.com/en/github/working-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#choosing-a-publishing-source). 2. You can also setup github page to serve your tar.gz files on your forked repo. Github pages usually requires you to have this set up on [specific branches](https://help.github.com/en/github/working-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#choosing-a-publishing-source).
3. You can directly add `https://github.com/rancher/partner-charts` into rancher catalog. In order to show all the charts you have to run `make CHART=${chart_name} prepare` and make sure there is `chart-original` folder on each chart folder if your chart relies on a upstream chart. 3. You can directly add `https://github.com/rancher/charts` into rancher catalog. In order to show all the charts you have to run `make CHART=${chart_name} prepare` and make sure there is `chart-original` folder on each chart folder if your chart relies on a upstream chart.
### Makefile ### Makefile

7
scripts/chart-templates/crd-assume-ownership/templates/_helpers.tpl Normal file
View File

@ -0,0 +1,7 @@
# Rancher
{{- define "system_default_registry" -}}
{{- if .Values.global.cattle.systemDefaultRegistry -}}
{{- printf "%s/" .Values.global.cattle.systemDefaultRegistry -}}
{{- end -}}
{{- end -}}

92
scripts/chart-templates/crd-assume-ownership/templates/jobs.yaml Normal file
View File

@ -0,0 +1,92 @@
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Chart.Name }}-create
namespace: {{ .Release.Namespace }}
labels:
app: {{ .Chart.Name }}
annotations:
"helm.sh/hook": post-install, post-upgrade, post-rollback
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
metadata:
name: {{ .Chart.Name }}-create
labels:
app: {{ .Chart.Name }}
spec:
serviceAccountName: {{ .Chart.Name }}-manager
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: create-crds
image: {{ template "system_default_registry" . }}{{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: IfNotPresent
command:
- /bin/kubectl
- apply
- -f
- /etc/config/crd-manifest.yaml
volumeMounts:
- name: crd-manifest
readOnly: true
mountPath: /etc/config
restartPolicy: OnFailure
volumes:
- name: crd-manifest
configMap:
name: {{ .Chart.Name }}-manifest
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Chart.Name }}-delete
namespace: {{ .Release.Namespace }}
labels:
app: {{ .Chart.Name }}
annotations:
"helm.sh/hook": pre-delete
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
metadata:
name: {{ .Chart.Name }}-delete
labels:
app: {{ .Chart.Name }}
spec:
serviceAccountName: {{ .Chart.Name }}-manager
securityContext:
runAsNonRoot: true
runAsUser: 1000
initContainers:
- name: remove-finalizers
image: {{ template "system_default_registry" . }}{{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: IfNotPresent
command:
- /bin/kubectl
- apply
- -f
- /etc/config/crd-manifest.yaml
volumeMounts:
- name: crd-manifest
readOnly: true
mountPath: /etc/config
containers:
- name: delete-crds
image: {{ template "system_default_registry" . }}{{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: IfNotPresent
command:
- /bin/kubectl
- delete
- -f
- /etc/config/crd-manifest.yaml
volumeMounts:
- name: crd-manifest
readOnly: true
mountPath: /etc/config
restartPolicy: OnFailure
volumes:
- name: crd-manifest
configMap:
name: {{ .Chart.Name }}-manifest

14
scripts/chart-templates/crd-assume-ownership/templates/manifest.yaml Normal file
View File

@ -0,0 +1,14 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Chart.Name }}-manifest
namespace: {{ .Release.Namespace }}
data:
crd-manifest.yaml: |
{{- \$currentScope := . -}}
{{- \$crds := (.Files.Glob "crd-manifest/**.yaml") -}}
{{- range \$path, \$_ := \$crds -}}
{{- with \$currentScope -}}
{{ .Files.Get \$path | nindent 4 }}
---
{{- end -}}{{- end -}}

35
scripts/chart-templates/crd-assume-ownership/templates/rbac.yaml Normal file
View File

@ -0,0 +1,35 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ .Chart.Name }}-manager
labels:
app: {{ .Chart.Name }}-manager
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs: ['create', 'get', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ .Chart.Name }}-manager
labels:
app: {{ .Chart.Name }}-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ .Chart.Name }}-manager
subjects:
- kind: ServiceAccount
name: {{ .Chart.Name }}-manager
namespace: {{ .Release.Namespace }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Chart.Name }}-manager
namespace: {{ .Release.Namespace }}
labels:
app: {{ .Chart.Name }}-manager

11
scripts/chart-templates/crd-assume-ownership/values.yaml Normal file
View File

@ -0,0 +1,11 @@
# Default values for ${name}-crd.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
global:
cattle:
systemDefaultRegistry: ""
image:
repository: rancher/kubectl
tag: v1.18.6

7
scripts/chart-templates/crd-base/Chart.yaml Normal file
View File

@ -0,0 +1,7 @@
apiVersion: ${api_version}
version: ${chart_version}
description: Installs the CRDs for ${name}.
name: ${name}-crd
type: application
annotations:
catalog.cattle.io/hidden: "true"

2
scripts/chart-templates/crd-base/README.md Normal file
View File

@ -0,0 +1,2 @@
# ${name}-crd
A Rancher chart that installs the CRDs used by [${name}](https://github.com/rancher/dev-charts/tree/master/packages/${name}).

14
scripts/chart-templates/crd-base/overlay-upstream/templates/validate-install-crd.yaml Normal file
View File

@ -0,0 +1,14 @@
#{{- if gt (len (lookup "rbac.authorization.k8s.io/v1" "ClusterRole" "" "")) 0 -}}
# {{- \$found := dict -}}
${set_found_crd}
# {{- range .Capabilities.APIVersions -}}
# {{- if hasKey \$found (toString .) -}}
# {{- set \$found (toString .) true -}}
# {{- end -}}
# {{- end -}}
# {{- range \$_, \$exists := \$found -}}
# {{- if (eq \$exists false) -}}
# {{- required "Required CRDs are missing. Please install the ${name}-crd chart before installing this chart." "" -}}
# {{- end -}}
# {{- end -}}
#{{- end -}}

5
scripts/ci
View File

@ -1,5 +1,10 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -e set -e
set -x
if [ -n ${1} ]; then
export CHART=${1}
fi
./scripts/prepare ./scripts/prepare
./scripts/generate-charts ./scripts/generate-charts

16
scripts/ci-wrapper Executable file
View File

@ -0,0 +1,16 @@
#!/usr/bin/env bash
set -e
set -x
sudo ./scripts/bootstrap
git fetch origin ${1}
# checkout index.yaml and sha256sum folder from asset branch in order to compare sha and merge index
git checkout origin/${1} ./assets/index.yaml || true
git checkout origin/${1} ./sha256sum || true
charts=$(./scripts/produce-sha256)
if [[ -n ${charts} ]]; then
echo $charts | xargs ./scripts/ci
fi

25
scripts/clean
View File

@ -1,9 +1,24 @@
#!/usr/bin/env bash #!/usr/bin/env bash
for f in packages/*; do for f in packages/*; do
if [[ -f ${f}/package.yaml ]]; then if [[ -f ${f}/package.yaml ]]; then
if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then
rm -rf ${f}/charts split_crds=$(yq r ${f}/package.yaml generateCRDChart.enabled)
fi if [[ "${split_crds}" == "true" && -d ${f}/charts-crd ]]; then
fi ./scripts/clean-crds ${f}
fi
url=$(yq r ${f}/package.yaml url)
if [[ -n $url ]]; then
rm -rf ${f}/charts
fi
fi
fi
done
for f in packages/*; do
if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then
if [ -d "${f}/charts/charts" ]; then
rm -f ${f}/charts/charts/*.tgz
fi
fi
done done

37
scripts/clean-crds Executable file
View File

@ -0,0 +1,37 @@
#!/usr/bin/env bash
set -e
# Reverts changes done to a chart from the prepare-crd script to go back to using one chart
if [[ -z $1 ]]; then
echo "No directory provided to revert charts-crd changes within"
exit 1
fi
f=$1
if ! [[ -d ${f}/charts-crd ]]; then
echo "Chart does not have a charts-crd/ directory to revert changes from"
exit 1
fi
# Move CRDs back into ${f}/charts/crd/ and remove ${f}/charts-crd
mkdir -p ${f}/charts/crds/
if [[ -d ${f}/charts-crd/crd-manifest ]]; then
mv ${f}/charts-crd/crd-manifest/* ${f}/charts/crds/
else
mv ${f}/charts-crd/templates/* ${f}/charts/crds/
fi
rm -rf ${f}/charts-crd
if ! [[ -d ${f}/charts ]]; then
# Main chart has already been deleted; no need to modify contents
exit 0
fi
# Remove the validate-install-${name}-crd.yaml
name=$(cat ${f}/charts/Chart.yaml | yq r - 'name')
rm ${f}/charts/templates/validate-install-crd.yaml
# Remove additional annotations added to original chart if added
if [[ "$(yq r ${f}/charts/Chart.yaml 'annotations[catalog.cattle.io/auto-install]')" == "${name}-crd=match" ]]; then
yq d -i ${f}/charts/Chart.yaml 'annotations[catalog.cattle.io/auto-install]'
fi

25
scripts/generate-charts
View File

@ -1,24 +1,37 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -e set -e
add=0
for f in packages/*; do for f in packages/*; do
if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then
mkdir -p assets/$(basename -- ${f}) mkdir -p assets/$(basename -- ${f})
if [[ -d ${f}/overlay ]]; then if [[ -d ${f}/overlay ]]; then
cp -R ${f}/overlay/* ${f}/charts cp -R ${f}/overlay/* ${f}/charts
fi fi
version=""
if [[ -f ${f}/package.yaml ]]; then if [[ -f ${f}/package.yaml ]]; then
version=$(yq r ${f}/charts/Chart.yaml version) version=$(yq r ${f}/charts/Chart.yaml version)
packageVersion=$(yq r ${f}/package.yaml packageVersion) packageVersion=$(yq r ${f}/package.yaml packageVersion)
yq w -i ${f}/charts/Chart.yaml 'version' "${version}${packageVersion}" yq w -i ${f}/charts/Chart.yaml 'version' "${version}${packageVersion}"
fi fi
helm package ${f}/charts --destination assets/$(basename -- ${f}) helm package ${f}/charts --destination assets/$(basename -- ${f})
git checkout assets/$(basename -- ${f})/"$(basename -- ${f})-$(yq r ${f}/charts/Chart.yaml version).tgz" 2>/dev/null || true
# copy over contents to charts/ folder for browsing purpose
mkdir -p charts/$(basename -- ${f})
cp -R ${f}/charts/* charts/$(basename -- ${f})/
if [[ -d ${f}/charts-crd ]]; then
packageVersion=$(yq r ${f}/package.yaml packageVersion)
yq w -i ${f}/charts-crd/Chart.yaml 'version' "${version}${packageVersion}"
helm package ${f}/charts-crd --destination assets/$(basename -- ${f})
# copy over contents to charts/ folder for browsing purpose
mkdir -p charts/$(basename -- ${f})/charts-crd
cp -R ${f}/charts-crd/* charts/$(basename -- ${f})/charts-crd
fi
helm repo index --merge ./assets/index.yaml --url assets assets
fi fi
done done
if [[ -n $(git ls-files --other | grep "tgz") ]]; then cp ./assets/index.yaml .
helm repo index --merge ./assets/index.yaml --url assets assets
cp ./assets/index.yaml .
fi

50
scripts/generate-patch
View File

@ -11,13 +11,17 @@ function remove_timestamp_from_diff {
time="[[:digit:]]\{2\}:[[:digit:]]\{2\}:[[:digit:]]\{2\}.[[:digit:]]\{9\}" time="[[:digit:]]\{2\}:[[:digit:]]\{2\}:[[:digit:]]\{2\}.[[:digit:]]\{9\}"
timezone="[-\+][[:digit:]]\{4\}" timezone="[-\+][[:digit:]]\{4\}"
timestamp_format="${date}[[:space:]]${time}[[:space:]]${timezone}" timestamp_format="${date}[[:space:]]${time}[[:space:]]${timezone}"
sed -i '' "s/\(${prefix} ${filename_format}\)[[:space:]]${timestamp_format}/\1/g" ${f}/$(basename -- ${f}).patch sed -i.bak "s/\(${prefix} ${filename_format}\)[[:space:]]${timestamp_format}/\1/g" ${f}/$(basename -- ${f}).patch
rm ${f}/$(basename -- ${f}).patch.bak
} }
for f in packages/*; do for f in packages/*; do
if [[ -f ${f}/package.yaml ]]; then if [[ -f ${f}/package.yaml ]]; then
if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then
url=$(cat ${f}/package.yaml | yq r - url) url=$(cat ${f}/package.yaml | yq r - url)
if [[ -z ${url} ]]; then
continue
fi
subdirectory=$(cat ${f}/package.yaml | yq r - subdirectory) subdirectory=$(cat ${f}/package.yaml | yq r - subdirectory)
type=$(cat ${f}/package.yaml | yq r - type) type=$(cat ${f}/package.yaml | yq r - type)
commit=$(cat ${f}/package.yaml | yq r - commit) commit=$(cat ${f}/package.yaml | yq r - commit)
@ -39,9 +43,47 @@ for f in packages/*; do
mkdir -p ${f}/charts-original mkdir -p ${f}/charts-original
curl -sLf ${url} | tar xvzf - -C ${f}/charts-original --strip ${fields} ${subdirectory} > /dev/null 2>&1 curl -sLf ${url} | tar xvzf - -C ${f}/charts-original --strip ${fields} ${subdirectory} > /dev/null 2>&1
fi fi
if [[ -f ${f}/charts/requirements.yaml ]]; then
i=0
while ! [[ -z $(yq r ${f}/charts/requirements.yaml "dependencies[${i}]") ]]; do
subchart=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].name" | head -n 1)
version=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].version" | head -n 1)
helm_repo=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].repository" | head -n 1)
# Remove trailing slash
[[ ${helm_repo} == */ ]] && helm_repo=${helm_repo%/}
if [[ ${helm_repo} == file://* ]] || [[ ! ${version} =~ ^[0-9\.]+$ ]]; then
# Subchart is tracked locally; patches will not be tracked
rm -rf ${f}/charts-original/charts/${subchart}
((i=i+1))
continue
fi
# Subchart is tracked in remote Helm repo with a fixed version number; we grab the download URL and get the local chart archive
download_url=$(curl -sLf ${helm_repo}/index.yaml | cat | yq r - "entries.${subchart}.(version == ${version}).urls[0]")
if [[ -z ${download_url} ]]; then
echo "Failed to get download URL for ${subchart} at version ${version} from index.yaml at ${helm_repo}"
exit 1
fi
echo "Comparing current changes to ${subchart}-${version} against upstream found at ${download_url}"
curl -sLf ${download_url} | tar xvzf - -C ${f}/charts-original/charts > /dev/null 2>&1
((i=i+1))
done
fi
if [[ -d ${f}/charts ]]; then if [[ -d ${f}/charts ]]; then
split_crds=$(cat ${f}/package.yaml | yq r - generateCRDChart.enabled)
if [[ "${split_crds}" == "true" ]]; then
./scripts/clean-crds ${f}
fi
diff -x *.tgz -x *.lock -uNr ${f}/charts-original ${f}/charts > ${f}/$(basename -- ${f}).patch || true diff -x *.tgz -x *.lock -uNr ${f}/charts-original ${f}/charts > ${f}/$(basename -- ${f}).patch || true
remove_timestamp_from_diff if ! [[ -s ${f}/$(basename -- ${f}).patch ]]; then
# If no changes exist in patch then remove it
rm ${f}/$(basename -- ${f}).patch
fi
if [[ -f ${f}/$(basename -- ${f}).patch ]]; then
remove_timestamp_from_diff
fi
if [[ "${split_crds}" == "true" ]]; then
./scripts/prepare-crds ${f}
fi
fi fi
rm -rf ${f}/charts-original rm -rf ${f}/charts-original
fi fi

43
scripts/prepare
View File

@ -1,13 +1,14 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -e set -e
# rebase the patch with upsteam charts defined in package.yaml. Make sure patches still apply # Pull in the upstream charts
# for the latest upsteam charts. If there is conflits commiter should solve it.
for f in packages/*; do for f in packages/*; do
if [[ -f ${f}/package.yaml ]]; then if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then
if [[ -z $CHART || $CHART == $(basename -- ${f}) ]]; then unset url
if [[ -f ${f}/package.yaml ]]; then
url=$(cat ${f}/package.yaml | yq r - url) url=$(cat ${f}/package.yaml | yq r - url)
fi
if [[ -n ${url} ]]; then
subdirectory=$(cat ${f}/package.yaml | yq r - subdirectory) subdirectory=$(cat ${f}/package.yaml | yq r - subdirectory)
type=$(cat ${f}/package.yaml | yq r - type) type=$(cat ${f}/package.yaml | yq r - type)
fields=$(echo ${subdirectory} | awk -F'/' '{print NF}') fields=$(echo ${subdirectory} | awk -F'/' '{print NF}')
@ -31,31 +32,21 @@ for f in packages/*; do
else else
curl -sLf ${url} | tar xvzf - -C ${f}/charts --strip ${fields} ${subdirectory} > /dev/null 2>&1 curl -sLf ${url} | tar xvzf - -C ${f}/charts --strip ${fields} ${subdirectory} > /dev/null 2>&1
fi fi
fi
# Loop through packages again and copy dependencies over
# If sub-charts existed as package.yaml they will now be vendored into parent
./scripts/prepare-subcharts ${f}
# Apply the patch and split the CRD package
if [[ -f ${f}/package.yaml ]]; then
for file in $(find ./${f} -type f -name "*.patch"); do for file in $(find ./${f} -type f -name "*.patch"); do
basename=$(basename -- ${file}) basename=$(basename -- ${file})
patch -E -p3 -d ${f}/charts < ${f}/$basename patch -E -p3 -d ${f}/charts < ${f}/$basename
done done
copied_dependencies=() split_crds=$(cat ${f}/package.yaml | yq r - generateCRDChart.enabled)
if [[ -f ${f}/charts/requirements.yaml ]]; then if [[ "${split_crds}" == "true" ]]; then
repos=$(cat ${f}/charts/requirements.yaml | yq r - "dependencies.*.repository") ./scripts/prepare-crds ${f}
while IFS= read -r repo; do
if [[ ${repo} == file://* ]]; then
charts_path=${repo/file:\/\//${f}/charts/}
overlay_path=${repo/file:\/\//${f}/overlay/}
if [[ ! -d $charts_path ]] && [[ -d $overlay_path ]]; then
cp -R $overlay_path $charts_path
copied_dependencies+=("${charts_path}")
fi
fi
done < <(echo "${repos}")
fi fi
pwd=$(pwd)
cd ${f}/charts
helm dependency update
cd $pwd
for dep in "${copied_dependencies[@]}"; do
rm -rf $dep
done
fi fi
fi fi
done done

110
scripts/prepare-crds Executable file
View File

@ -0,0 +1,110 @@
#!/usr/bin/env bash
set -e
# Split the provided package into a charts and charts-crd package
# This script is split up into the following sections
# - Initial checks to make sure that the current state of the directory is suitable for running this script
# - Collecting information from the upstream chart that will be used to fill out a set of templates
# - Copy templates stored in ./scripts/chart-templates into ${f}/charts-crd based on flags
# - Apply templates based on information collected from the upstream chart
# - Move anything from ${f}/charts-crd/overlay-upstream to the upstream chart
# - Move any CRDs from the upstream chart into the appropriate directory for CRDs based on flags
# - Apply any additional necessary annotation logic on the Chart.yaml of both the upstream and CRD charts
if [[ -z $1 ]]; then
echo "No directory provided to initialize charts-crd within"
exit 1
fi
f=$1
if ! [[ -f ${f}/package.yaml ]]; then
echo "Could not find ${f}/package.yaml"
exit 1
fi
if [[ -d ${f}/charts-crd ]]; then
rm -rf ${f}/charts-crd
fi
if ! [[ -d ${f}/charts ]]; then
echo "Could not find ${f}/charts"
exit 1
fi
if ! [[ -f ${f}/charts/Chart.yaml ]]; then
echo "Could not find ${f}/charts/Chart.yaml"
exit 1
fi
if ! [[ -d ${f}/charts/crds ]] || [[ $(ls ${f}/charts/crds | wc -l) -eq 0 ]]; then
echo "Chart does not have any crds within a crd/ directory"
exit 1
fi
# Collect information on chart
name=$(cat ${f}/charts/Chart.yaml | yq r - 'name')
api_version=$(cat ${f}/charts/Chart.yaml | yq r - 'apiVersion')
chart_version=$(cat ${f}/charts/Chart.yaml | yq r - 'version')
# Collect information on CRDs
crd_apis=()
for crd_yaml in ${f}/charts/crds/*; do
crd_group=$(yq r ${crd_yaml} 'spec.group')
crd_kind=$(yq r ${crd_yaml} 'spec.names.kind')
crd_version=$(yq r ${crd_yaml} 'spec.version')
if [[ -z "$crd_version" ]]; then
crd_version=$(yq r ${crd_yaml} 'spec.versions[0].name')
fi
crd_apis+=("${crd_group}/${crd_version}/${crd_kind}")
done
# Set environment variable to be used in validate-install-crd.yaml template
set_found_crd=$(
for crd in ${crd_apis[@]}; do
echo "# {{- set \$found \"${crd}\" false -}}"
done
)
# Copy base template and apply variables to the template
mkdir -p ${f}/charts-crd
cp -R ./scripts/chart-templates/crd-base/* ${f}/charts-crd
assume_ownership=$(cat ${f}/package.yaml | yq r - generateCRDChart.assumeOwnershipOfCRDs)
if [[ ${assume_ownership} == "true" ]]; then
cp -fR ./scripts/chart-templates/crd-assume-ownership/* ${f}/charts-crd
fi
for template_file in $(find ${f}/charts-crd -type f); do
# Applies any environment variables currently set onto your template file
eval "echo \"$(sed 's/"/\\"/g' ${template_file})\"" > ${template_file}
done
# Move anything from ${f}/charts-crd/overlay-upstream to the main chart
cp -R ${f}/charts-crd/overlay-upstream/* ${f}/charts
rm -rf ${f}/charts-crd/overlay-upstream
# Move CRDs from main chart to CRD chart
if [[ ${assume_ownership} == "true" ]]; then
mkdir -p ${f}/charts-crd/crd-manifest
mv ${f}/charts/crds/* ${f}/charts-crd/crd-manifest
else
mkdir -p ${f}/charts-crd/templates
mv ${f}/charts/crds/* ${f}/charts-crd/templates
fi
rm -rf ${f}/charts/crds
# Make the primary chart auto-install the CRD chart
if [[ -z "$(yq r ${f}/charts/Chart.yaml 'annotations[catalog.cattle.io/auto-install]')" ]]; then
yq w -i ${f}/charts/Chart.yaml "annotations[catalog.cattle.io/auto-install]" "${name}-crd=match"
fi
# Add annotations to ${f}/charts-crd/Chart.yaml
copyAnnotations=(catalog.cattle.io/release-name catalog.cattle.io/certified catalog.cattle.io/experimental catalog.cattle.io/namespace)
for a in ${copyAnnotations[@]}; do
v=$(yq r ${f}/charts/Chart.yaml "annotations[${a}]")
if ! [[ -z ${v} ]]; then
if [[ ${a} == "catalog.cattle.io/release-name" ]]; then
v="${v}-crd"
fi
yq w -i ${f}/charts-crd/Chart.yaml "annotations[${a}]" "${v}"
fi
done

138
scripts/prepare-subcharts Executable file
View File

@ -0,0 +1,138 @@
#!/usr/bin/env bash
set -e
# Pull in the subcharts required by a chart
#
# At the end of this process, all the required subcharts will exist in the charts/charts directory.
# Charts will either exist as chart archives (i.e. a .tgz file) if they are not eligible for patches or
# as a local chart archive (i.e. a directory) if they are eligible for patches (i.e. remote subchart with fixed version number)
#
# In order to prepare sub-charts so that we can patch them:
# - We apply any patches on the requirements.yaml to track any new subcharts added via patch or updates to version numbers of the subcharts
# - We pull in any subchart that is a feature chart by running prepare on it
# - We copy in any local subcharts that are located in the overlay/ directory
# - We call `helm dependency update` to generate chart archives (.tgz files) for local and remote charts
# - We clean up any copied charts from the overlay/ directory
# - We clean up any feature charts that were recursively prepared
# - We unarchive the chart archive if and only if its a remote subchart with a fixed version number (which means it can be patched)
# - We revert the initial patch we applied on the requirements.yaml
if [[ -z $1 ]]; then
echo "No directory provided to pull in subcharts"
exit 1
fi
f=$1
if ! [[ -f ${f}/charts/requirements.yaml ]]; then
echo "Chart ${CHART} has no dependencies; skipping step to pull in dependencies"
exit 0
fi
# Apply the patch on the requirements.yaml
for file in $(find ./${f} -type f -name "*.patch"); do
# This awk statement extracts the contents between the first line that matches the regex `diff.*requirements.yaml`
# and the first line that matches the regex `diff` after it. This corresponds to the contents of any patches on the requirements.yaml file.
awk '/diff.*requirements.yaml/{flag=1;next}/diff/{flag=0}flag' $file >> ${f}/requirements.patch
done
if [[ -f ${f}/requirements.patch ]] && [[ -s ${f}/requirements.patch ]]; then
patch -E -p3 -d ${f}/charts < ${f}/requirements.patch
fi
# Pull in any dependencies that are stored as package.yamls or in the overlay/ folder
overlaid_dependencies=()
mirrored_dependencies=()
i=0
while ! [[ -z $(yq r ${f}/charts/requirements.yaml "dependencies[${i}]") ]]; do
repo=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].repository")
if [[ ${repo} == file://* ]]; then
charts_path=${repo/file:\/\//${f}/charts/}
if [[ -d ${charts_path} ]]; then
# Chart already exists
((i=i+1))
continue
fi
if [[ -f ${charts_path%/*}/package.yaml ]]; then
# Prepare the subchart and pull it in
subchart=$(basename -- ${charts_path%/*})
mirrored_dependencies+=("${subchart}")
prepare_subchart () {
local CURRENT_CHART=${CHART}
local CURRENT_INDEX=${i}
CHART=${subchart} ./scripts/prepare
CHART=${CURRENT_CHART}
i=${CURRENT_INDEX}
}
prepare_subchart
fi
overlay_path=${repo/file:\/\//${f}/overlay/}
if [[ -d $overlay_path ]]; then
# Pull in the subchart from the overlay/ directory
cp -R $overlay_path $charts_path
overlaid_dependencies+=("${charts_path}")
fi
fi
((i=i+1))
done
# Update the current requirements.lock and requirements.yaml to keep track of the overlay/ dependencies
pwd=$(pwd)
cd ${f}/charts
helm dependency update
cd $pwd
# Remove any dependencies that came from the overlay/ directory, we don't track patches for overlay files
for dep in "${overlaid_dependencies[@]}"; do
rm -rf $dep
done
# Cleanup any dependencies that came from another local chart, any changes should be made there.
for subchart in "${mirrored_dependencies[@]}"; do
cleanup_subchart () {
local CURRENT_CHART=${CHART}
CHART=${subchart} ./scripts/clean
CHART=${CURRENT_CHART}
}
cleanup_subchart
done
if ! [[ -d ${f}/charts/charts ]]; then
exit 0
fi
# If it is possible to patch a subchart, create a local chart archive for it
# Otherwise, it should only be stored as a tgz file to prevent users from accidentally patching it
i=0
while ! [[ -z $(yq r ${f}/charts/requirements.yaml "dependencies[${i}]") ]]; do
name=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].name")
version=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].version")
repo=$(yq r ${f}/charts/requirements.yaml "dependencies[${i}].repository")
if [[ ${repo} == file://* ]]; then
# Subcharts that are tracked locally should be directly edited, not via a patch
rm -rf ${f}/charts/charts/${name}
((i=i+1))
continue
fi
if [[ ! ${version} =~ ^[0-9\.]+$ ]]; then
# Subcharts that are tracked remotely must have a fixed version number
rm -rf ${f}/charts/charts/${name}
((i=i+1))
continue
fi
# Subchart could be patched
subchart_tgz=${name}-${version}.tgz
if [[ -f ${f}/charts/charts/${subchart_tgz} ]]; then
tar xvzf ${f}/charts/charts/${subchart_tgz} -C ${f}/charts/charts > /dev/null 2>&1
rm ${f}/charts/charts/${subchart_tgz}
fi
((i=i+1))
done
# Undo the patch on the requirements.yaml
if [[ -f ${f}/requirements.patch ]]; then
if [[ -s ${f}/requirements.patch ]]; then
# Should only apply a patch if the requirements.patch has something in it
patch -R -E -p3 -d ${f}/charts < ${f}/requirements.patch
fi
rm ${f}/requirements.patch
fi

33
scripts/produce-sha256 Executable file
View File

@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -e
# This script generates the sha256 sum of each file in the packages/ and stores them in the sha256/ directory.
# If the sha256/ directory exists, it only prints out the names of the charts whose sha256 sums were changed
# If the sha256/ directory does not exist, it prints out the names of all charts
if ! which sha256sum > /dev/null; then
echo "Could not find required program to compute SHA256 sums: sha256sum"
exit 1
fi
for f in packages/*; do
rm -rf ./sha256sum-new/$(basename -- ${f})
mkdir -p ./sha256sum-new/$(basename -- ${f})
find ${f} -type f | xargs sha256sum > ./sha256sum-new/$(basename -- ${f})/$(basename -- ${f}).sum
done
if [ ! -d sha256sum ]; then
cp -R sha256sum-new sha256sum
for f in packages/*; do
echo $(basename -- ${f})
done
fi
for f in sha256sum/*; do
if ! cmp -s ./sha256sum-new/$(basename -- ${f})/$(basename -- ${f}).sum ${f}/$(basename -- ${f}).sum; then
echo $(basename -- ${f})
fi
done
rm -rf ./sha256sum 2>/dev/null
mv ./sha256sum-new ./sha256sum

32
scripts/push-to-assets-branch Executable file
View File

@ -0,0 +1,32 @@
#!/usr/bin/env bash
set -e
set -x
if [[ -z ${1} || -z ${2} ]]; then
echo "Usage: ./scripts/push-to-assets-branch <remote> <assets-branch>"
exit 1
fi
remote=${1}
branch=${2}
rm -rf /tmp/tmp-assets
mkdir -p /tmp/tmp-assets
for i in assets sha256sum charts; do
if [[ -d $i ]]; then
cp -R $i /tmp/tmp-assets
fi
done
for i in index.yaml index.html; do
if [[ -f $i ]]; then
cp $i /tmp/tmp-assets
fi
done
git checkout .
git clean -fd
git fetch $remote $branch
git checkout $branch
cp -R /tmp/tmp-assets/* ./
git add .
git -c user.name="actions" -c user.email="actions@github.com" commit -m "Generated changes" || true

12
scripts/validate
View File

@ -3,14 +3,8 @@ set -e
cd $(dirname $0)/.. cd $(dirname $0)/..
./scripts/prepare CHART=$CHART ./scripts/prepare
source ./scripts/version CHART=$CHART ./scripts/generate-charts
if [ -n "$DIRTY" ]; then CHART=$CHART ./scripts/clean
echo Git is dirty
git status
exit 1
fi
./scripts/generate-charts

0
scripts/version Executable file → Normal file
View File