1ce16f5243 | ||
---|---|---|
.. | ||
crds | ||
templates | ||
Chart.yaml | ||
README.md | ||
app-readme.md | ||
values.yaml |
README.md
NetScaler CPX with NetScaler Ingress Controller running as sidecar.
In a Kubernetes or OpenShift cluster, you can deploy NetScaler CPX with NetScaler ingress controller as a sidecar. The NetScaler CPX instance is used for load balancing the North-South traffic to the microservices in your cluster. And, the sidecar NetScaler ingress controller configures the NetScaler CPX.
TL;DR;
For Kubernetes
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes
For OpenShift
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,openshift=true
Important:
The "license.accept" is a mandatory argument and should be set to "yes" to accept the terms of the NetScaler license.
NOTE:
The CRDs supported by NetScaler will be installed automatically with the installation of the Helm Charts if CRDs are not already available in the cluster.
Introduction
This Helm chart deploys a NetScaler CPX with NetScaler ingress controller as a sidecar in the Kubernetes or in the Openshift cluster using the Helm package manager.
Prerequisites
-
The Kubernetes version should be 1.16 and above if using Kubernetes environment.
-
The Openshift version 4.8 or later if using OpenShift platform.
-
The Helm version 3.x or later. You can follow instruction given here to install the same.
-
You have installed Prometheus Operator, if you want to view the metrics of the NetScaler CPX collected by the metrics exporter.
-
Registration of NetScaler CPX in ADM: You may want to register your CPX in ADM for licensing or to obtain servicegraph. For this you will have to create a Kubernetes secret using ADM credentials and provide it while install the chart. Create a Kubernetes secret for the user name and password using the following command:
kubectl create secret generic admlogin --from-literal=username=<adm-username> --from-literal=password=<adm-password>
Installing the Chart
Add the NetScaler Ingress Controller helm chart repository using command:
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
For Kubernetes:
1. NetScaler CPX with NetScaler Ingress Controller running as side car.
To install the chart with the release name my-release
:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,ingressClass[0]=<ingressClassName>
Note:
By default the chart installs the recommended RBAC roles and role bindings.
The command deploys NetScaler CPX with NetScaler ingress controller as a sidecar on the Kubernetes cluster with the default configuration. The configuration section lists the mandatory and optional parameters that you can configure during installation.
2. NetScaler CPX with NetScaler Ingress Controller and Exporter running as side car.
Metrics exporter can be deployed as sidecar to the NetScaler CPX and collects metrics from the NetScaler CPX instance. You can then visualize these metrics using Prometheus Operator and Grafana.
Note:
Ensure that you have installed Prometheus Operator.
Use the following command for this:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,ingressClass[0]=<ingressClassName>,exporter.required=true
For OpenShift:
Add the name of the service account created when the chart is deployed to the privileged Security Context Constraints of OpenShift:
oc adm policy add-scc-to-user privileged system:serviceaccount:<namespace>:<service-account-name>
1. NetScaler CPX with NetScaler Ingress Controller running as side car.
To install the chart with the release name, my-release
, use the following command:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,openshift=true
2. NetScaler CPX with NetScaler Ingress Controller and Exporter running as side car.
Metrics exporter can be deployed as sidecar to the NetScaler CPX and collects metrics from the NetScaler CPX instance. You can then visualize these metrics using Prometheus Operator and Grafana.
Note:
Ensure that you have installed Prometheus Operator.
Use the following command for this:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,openshift=true,exporter.required=true
Installed components
The following components are installed:
- NetScaler CPX
- NetScaler ingress controller (if enabled)
- Exporter (if enabled)
NetScaler CPX Service Annotations:
The parameter serviceAnnotations
can be used to annotate CPX service while installing NetScaler CPX using this helm chart.
For example, if CPX is getting deployed in Azure and an Azure Internal Load Balancer is required before CPX then the annotation service.beta.kubernetes.io/azure-load-balancer-internal:True
can be set in CPX service using Helm command:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,serviceAnnotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal=True
or the same can be provided in values.yaml:
license:
accept: yes
serviceAnnotations:
service.beta.kubernetes.io/azure-load-balancer-internal: True
which can be used to install NetScaler CPX using Helm command:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller -f values.yaml
To know more about service annotations supported by Kubernetes on various platforms please see this.
NetScaler CPX Service Ports:
By default, port 80 and 443 of CPX service will exposed when CPX is installed using this helm chart. If it is required to expose any other ports in CPX service then the parameter servicePorts
can be used for it.
For example, if port 9999 is required to be exposed then below helm command can be used for installing NetScaler CPX:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,servicePorts[0].port=9999,servicePorts[0].protocol=TCP,servicePorts[0].name=https
or the same can be provided in values.yaml:
license:
accept: yes
servicePorts:
- port: 9090
protocol: TCP
name: https
which can be used to install NetScaler using Helm command:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller -f values.yaml
Note: If
servicePorts
parameters is used, only ports provided in this parameter will be exposed in CPX service. If you want to expose default ports 80 or 443, then you will need to explicity mention these also in this parameter.
Configuration for ServiceGraph:
If NetScaler CPX need to send data to the NetScaler ADM to bring up the servicegraph, then the below steps can be followed to install NetScaler CPX with ingress controller. NetScaler ingress controller configures NetScaler CPX with the configuration required for servicegraph.
- Create secret using NetScaler Agent credentials, which will be used by NetScaler CPX to communicate with NetScaler ADM Agent:
kubectl create secret generic admlogin --from-literal=username=<adm-agent-username> --from-literal=password=<adm-agent-password>
- Deploy NetScaler CPX with NetScaler ingress controller using helm command:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,analyticsConfig.required=true,analyticsConfig.timeseries.metrics.enable=true,analyticsConfig.distributedTracing.enable=true,analyticsConfig.endpoint.metrics.service=<ADM-Agent-IP>,ADMSettings.ADMIP=<ADM-Agent-IP>,ADMSettings.loginSecret=<Secret-for-ADM-Agent-credentials>
Note: If container agent is being used here for NetScaler ADM, please provide
svcIP
of container agent in theanalyticsConfig.endpoint.metrics.service
parameter.
NetScaler CPX DaemonSet with NetScaler Ingress Controller as sidecar for BGP Advertisement
The previous section of deploying CPX as a Deployment requires a Tier-1 Loadbalancer such as NetScaler VPX or cloud loadbalancers to route the traffic to CPX instances running in Kubernetes cluster, but you can also leverage BGP network fabric in your on-prem environemnt to route the traffic to CPX instances in a Kubernetes or Openshift cluster. you need to deploy CPX with NetScaler Ingress Controller as Daemonset to advertise the ExternalIPs of the K8s services of type LoadBalancer to your BGP Fabric. NetScaler CPX establishes a BGP peering session with your network routers, and uses that peering session to advertise the IP addresses of external cluster services. If your routers have ECMP capability, the traffic is load-balanced to multiple CPX instances by the upstream router, which in turn load-balances to actual application pods. When you deploy the NetScaler CPX with this mode, NetScaler CPX adds iptables rules for each service of type LoadBalancer on Kubernetes nodes. The traffic destined to the external IP address is routed to NetScaler CPX pods. You can also set the 'ingressIP' variable to an IP Address to advertise the External IP address for Ingress resources. Refer documentation for complete details about BGP advertisement with CPX.
Download the chart
You can download the chart usimg helm pull
command.
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm pull netscaler/netscaler-cpx-with-ingress-controller
tar -zxvf netscaler-cpx-with-ingress-controller-x.y.z.tgz
Edit the BGP configuration in values.yaml
BGP configurations enables CPX to peer with neighbor routers for advertisting the routes for Service of Type LoadBalancer. NetScaler Ingress Controllers uses static IPs given in Service YAML or using an IPAM controller to allocate an External IP address, and same is advertisted to the neighbour router with the Gateway as Node IP. An example BGP configurations is given below.
# BGP configurations: local AS, remote AS and remote address is mandatory to provide.
bgpSettings:
required: true
bgpConfig:
- bgpRouter:
# Local AS number for BGP advertisement
localAS:
neighbor:
# Address of the nighbor router for BGP advertisement
- address: xx.xx.xx.xx
# Remote AS number
remoteAS:
advertisementInterval: 10
ASOriginationInterval: 10
If the cluster spawns across multiple networks, you can also specify the NodeSelector to give different neighbors for different Cluster Nodes as shown below.
bgpSettings:
required: true
bgpConfig:
- nodeSelector: datacenter=ds1
bgpRouter:
localAS:
neighbor:
- address: xx.xx.xx.xx
remoteAS:
advertisementInterval: 10
ASOriginationInterval: 10
- nodeSelector: datacenter=ds2
bgpRouter:
localAS:
neighbor:
- address: yy.yy.yy.yy
remoteAS:
advertisementInterval: 10
ASOriginationInterval: 10
Deploy the chart
For Kubernetes:
1. NetScaler CPX DaemonSet with NetScaler Ingress Controller running as side car for BGP Advertisement.
To install the chart with the release name my-release
:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true
If you are running NetScaler IPAM for auto allocation of IPs for Service of type LoadBalancer, you must enable the IPAM configurations in NetScaler Ingress Controller as show below:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ipam=true
If you are using ingress resources, you must set the ingressIP
to a valid IP Address which will enable the BGP route advertisement for this IP when ingress resource is deployed.
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ingressIP=<Ingress External IP Address>
Note:
By default the chart installs the recommended RBAC roles and role bindings.
The command deploys NetScaler CPX Daemonset with NetScaler ingress controller as a sidecar on the Kubernetes cluster with the default configuration. The configuration section lists the mandatory and optional parameters that you can configure during installation.
2. NetScaler CPX with NetScaler Ingress Controller and Exporter running as side car for BGP Advertisement.
Metrics exporter can be deployed as sidecar to the NetScaler CPX and collects metrics from the NetScaler CPX instance. You can then visualize these metrics using Prometheus Operator and Grafana.
Note:
Ensure that you have installed Prometheus Operator.
Use the following command for this:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,exporter.required=true
If you are running NetScaler IPAM controller for auto allocation of IPs for Service of type LoadBalancer, you must enable the IPAM configurations in NetScaler Ingress Controller as show below:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ipam=true,exporter.required=true
If you are using ingress resources, you must set the ingressIP
to a valid IP Address which will enable the BGP route advertisement for this IP when ingress resource is deployed.
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ingressIP=<Ingress external IP>, exporter.required=true
For OpenShift:
Add the name of the service account created when the chart is deployed to the privileged Security Context Constraints of OpenShift:
oc adm policy add-scc-to-user privileged system:serviceaccount:<namespace>:<service-account-name>
1. NetScaler CPX DaemonSet with NetScaler Ingress Controller running as side car for BGP Advertisement.
To install the chart with the release name, my-release
, use the following command:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,openshift=true
If you are running NetScaler IPAM controller for auto allocation of IPs for Service of type LoadBalancer, you must enable the IPAM configurations in NetScaler Ingress Controller as show below:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ipam=true,openshift=true
If you are using ingress or Route resources, you must set the ingressIP
to a valid IP Address which will enable the BGP route advertisement for this IP when ingress resource is deployed.
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ingressIP=<Ingress External IP Address>,openshift=true
2. NetScaler CPX with NetScaler Ingress Controller and Exporter running as side car for BGP Advertisement.
Metrics exporter can be deployed as sidecar to the NetScaler CPX and collects metrics from the NetScaler CPX instance. You can then visualize these metrics using Prometheus Operator and Grafana.
Note:
Ensure that you have installed Prometheus Operator.
Use the following command for this:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,openshift=true,openshift=true,exporter.required=true
If you are running NetScaler IPAM controller for auto allocation of IPs for Service of type LoadBalancer, you must enable the IPAM configurations in NetScaler Ingress Controller as show below:
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ipam=true,openshift=true,exporter.required=true
If you are using ingress or Route resources, you must set the ingressIP
to a valid IP Address which will enable the BGP route advertisement for this IP when ingress resource is deployed.
helm install my-release ./netscaler-cpx-with-ingress-controller --set license.accept=yes,cpxBgpRouter=true,ingressIP=<Ingress External IP Address>,openshift=true,exporter.required=true
CRDs configuration
CRDs will be installed when we install NetScaler ingress controller via Helm automatically if CRDs are not installed in cluster already. If you wish to skip the CRD installation step, you can pass the --skip-crds flag. For more information about this option in Helm please see this.
There are a few examples of how to use these CRDs, which are placed in the folder: Example-CRDs. Refer to them and install as needed, using the following command:
kubectl create -f <crd-example.yaml>
Details of the supported CRDs:
authpolicies CRD:
Authentication policies are used to enforce access restrictions to resources hosted by an application or an API server.
NetScaler provides a Kubernetes CustomResourceDefinitions (CRDs) called the Auth CRD that you can use with the NetScaler ingress controller to define authentication policies on the ingress NetScaler.
Example file: auth_example.yaml
continuousdeployments CRD for canary:
Canary release is a technique to reduce the risk of introducing a new software version in production by first rolling out the change to a small subset of users. After user validation, the application is rolled out to the larger set of users. NetScaler-Integrated Canary Deployment solution stitches together all components of continuous delivery (CD) and makes canary deployment easier for the application developers.
httproutes and listeners CRDs for contentrouting:
Content Routing (CR) is the execution of defined rules that determine the placement and configuration of network traffic between users and web applications, based on the content being sent. For example, a pattern in the URL or header fields of the request.
Example files: HTTPRoute_crd.yaml, Listener_crd.yaml
ratelimits CRD:
In a Kubernetes deployment, you can rate limit the requests to the resources on the back end server or services using rate limiting feature provided by the ingress NetScaler.
Example files: ratelimit-example1.yaml, ratelimit-example2.yaml
vips CRD:
NetScaler provides a CustomResourceDefinitions (CRD) called VIP for asynchronous communication between the IPAM controller and NetScaler ingress controller.
The IPAM controller is provided by NetScaler for IP address management. It allocates IP address to the service from a defined IP address range. The NetScaler ingress controller configures the IP address allocated to the service as virtual IP (VIP) in NetScaler VPX. And, the service is exposed using the IP address.
When a new service is created, the NetScaler ingress controller creates a CRD object for the service with an empty IP address field. The IPAM Controller listens to addition, deletion, or modification of the CRD and updates it with an IP address to the CRD. Once the CRD object is updated, the NetScaler ingress controller automatically configures NetScaler-specfic configuration in the tier-1 NetScaler VPX.
rewritepolicies CRD:
In kubernetes environment, to deploy specific layer 7 policies to handle scenarios such as, redirecting HTTP traffic to a specific URL, blocking a set of IP addresses to mitigate DDoS attacks, imposing HTTP to HTTPS and so on, requires you to add appropriate libraries within the microservices and manually configure the policies. Instead, you can use the Rewrite and Responder features provided by the Ingress NetScaler device to deploy these policies.
Example files: target-url-rewrite.yaml
wafs CRD:
WAF CRD can be used to configure the web application firewall policies with the NetScaler ingress controller on the NetScaler VPX, MPX, SDX, and CPX. The WAF CRD enables communication between the NetScaler ingress controller and NetScaler for enforcing web application firewall policies.
In a Kubernetes deployment, you can enforce a web application firewall policy to protect the server using the WAF CRD. For more information about web application firewall, see Web application security.
Example files: wafhtmlxsssql.yaml
CORS CRD:
CORS CRD Cross-origin resource sharing (CORS) is a mechanism allows a web application running under one domain to securely access resources in another domain. You can configure CORS policies on NetScaler using NetScaler ingress controller to allow one domain (the origin domain) to call APIs in another domain. For more information, see the cross-origin resource sharing CRD documentation.
Example files: cors-crd.yaml
APPQOE CRD:
APPQOE CRD When a NetScaler appliance receives an HTTP request and forwards it to a back-end server, sometimes there may be connection failures with the back-end server. You can configure the request-retry feature on NetScaler to forward the request to the next available server, instead of sending the reset to the client. Hence, the client saves round trip time when NetScaler initiates the same request to the next available service. For more information, see the AppQoE support documentation. Appqoe resource sharing CRD documentation.
Example files: appqoe-crd.yaml
WILDCARDDNS CRD:
WILDCARDDNS CRD Wildcard DNS domains are used to handle requests for nonexistent domains and subdomains. In a zone, use wildcard domains to redirect queries for all nonexistent domains or subdomains to a particular server, instead of creating a separate Resource Record (RR) for each domain. The most common use of a wildcard DNS domain is to create a zone that can be used to forward mail from the internet to some other mail system. For more information, see the Wild card DNS domains support documentation. Wildcard DNS Entry CRD documentation.
Example files: wildcarddns-crd.yaml
NetScaler CPX servicetype LoadBalancer
NetScaler CPX can be installed with service having servicetype LoadBalancer. Following arguments can be used in the helm install
command for the same:
helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,serviceType.loadBalancer.enabled=True
NetScaler CPX servicetype NodePort
NetScaler CPX can be installed with service having servicetype Nodeport. Following arguments can be used in the helm install
command for the same:
helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,serviceType.nodePort.enabled=True
Additionally, serviceType.nodePort.httpPort
and serviceType.nodePort.httpsPort
arguments can be used to select the nodePort for the CPX service for HTTP and HTTPS ports.
Tolerations
Taints are applied on cluster nodes whereas tolerations are applied on pods. Tolerations enable pods to be scheduled on node with matching taints. For more information see Taints and Tolerations in Kubernetes.
Toleration can be applied to pod running NetScaler CPX and ingress controller containers using tolerations
argument while deploying CPX+NSIC using helm chart. This argument takes list of tolerations that user need to apply on the CPX+NSIC pods.
For example, following command can be used to apply toleration on the CPX+NSIC pod:
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,tolerations[0].key=<toleration-key>,tolerations[0].value=<toleration-value>,tolerations[0].operator=<toleration-operator>,tolerations[0].effect=<toleration-effect>
Here tolerations[0].key, tolerations[0].value and tolerations[0].effect are the key, value and effect that was used while tainting the node.
Effect represents what should happen to the pod if the pod don't have any matching toleration. It can have values NoSchedule
, NoExecute
and PreferNoSchedule
.
Operator represents the operation to be used for key and value comparison between taint and tolerations. It can have values Exists
and Equal
. The default value for operator is Equal
.
Resource Quotas
There are various use-cases when resource quotas are configured on the Kubernetes cluster. If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. The resource quotas for the NSIC and CPX containers can be provided explicitly in the helm chart.
To set requests and limits for the NSIC container, use the variables nsic.resources.requests
and nsic.resources.limits
respectively.
Similarly, to set requests and limits for the CPX container, use the variable resources.requests
and resources.limits
respectively.
Below is an example of the helm command that configures
A) For NSIC container:
CPU request for 500milli CPUs
CPU limit at 1000m
Memory request for 512M
Memory limit at 1000M
B) For CPX container:
CPU request for 250milli CPUs
CPU limit at 500m
Memory request for 256M
Memory limit at 512M
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes --set nsic.resources.requests.cpu=500m,nsic.resources.requests.memory=512Mi,nsic.resources.limits.cpu=1000m,nsic.resources.limits.memory=1000Mi --set resources.limits.cpu=500m,resources.limits.memory=512Mi,resources.requests.cpu=250m,resources.requests.memory=256Mi
Analytics Configuration
Analytics Configuration required for ADM
If NetScaler CPX needs to send data to the ADM for analytics purpose, then the below steps can be followed to install NetScaler CPX with ingress controller. NSIC configures the NetScaler CPX with the configuration required for analytics.
- Create secret using ADM Agent credentials, which will be used by NetScaler CPX to communicate with ADM Agent:
kubectl create secret generic admlogin --from-literal=username=<adm-agent-username> --from-literal=password=<adm-agent-password>
|Note: If you have installed container based adm-agent
using this helm chart, above step is not required, you just need to tag the namespace where the CPX is being deployed with citrix-cpx=enabled
.
- Deploy NetScaler CPX with NSIC using helm command:
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,analyticsConfig.required=true,analyticsConfig.distributedTracing.enable=true,analyticsConfig.endpoint.transactions.service=<Namespace/ADM_ServiceName-logstream>,ADMSettings.ADMIP=<ADM-Agent-IP_OR_FQDN>,ADMSettings.loginSecret=<Secret-for-ADM-Agent-credentials>,analyticsConfig.transactions.enable=true,analyticsConfig.transactions.port=5557
|Note: For container based ADM agent, please provide the logstream service FQDN in analyticsConfig.endpoint.transactions.service
. The logstream
service will be running on port 5557
.
Analytics Configuration required for NSOE
If NetScaler CPX needs to send data to the NSOE for observability, then the below steps can be followed to install NetScaler CPX with ingress controller. NSIC configures NetScaler CPX with the configuration required.
Deploy NetScaler CPX with NSIC using helm command:
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,analyticsConfig.required=true,analyticsConfig.timeseries.metrics.enable=true,analyticsConfig.timeseries.port=5563,analyticsConfig.timeseries.metrics.mode=prometheus,analyticsConfig.transactions.enable=true,analyticsConfig.transactions.port=5557,analyticsConfig.distributedTracing.enable=true,analyticsConfig.endpoint.metrics.service=<NSOE_SERVICE_IP>,analyticsConfig.endpoint.transactions.service=<Namespace/NSOE_SERVICE_NAME>
Analytics Configuration required for export of metrics to Prometheus
If NetScaler CPX needs to send data to Prometheus directly without an exporter resource in between, then the below steps can be followed to install NetScaler CPX with ingress controller. NSIC configures NetScaler CPX with the configuration required.
- Create secret to enable read-only access for a user, which will be required by NetScaler CPX to export metrics to Prometheus.
kubectl create secret generic prom-user --from-literal=username=<prometheus-username> --from-literal=password=<prometheus-password>
- Deploy NetScaler CPX with NSIC using helm command:
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes,nsic.prometheusCredentialSecret=<Secret-for-read-only-user-creation>,analyticsConfig.required=true,analyticsConfig.timeseries.metrics.enable=true,analyticsConfig.timeseries.port=5563,analyticsConfig.timeseries.metrics.mode=prometheus,analyticsConfig.timeseries.metrics.enableNativeScrape=true
- To setup Prometheus in order to scrape natively from NetScaler CPX pod, a new scrape job is required to be added under scrape_configs in the Prometheus configuration. For more details, check kubernetes_sd_config here. A sample of the Prometheus job is given below -
- job_name: 'kubernetes-cpx'
scheme: http
metrics_path: /nitro/v1/config/systemfile
params:
args: ['filename:metrics_prom_ns_analytics_time_series_profile.log,filelocation:/var/nslog']
format: ['prometheus']
basic_auth:
username: # Prometheus username set in nsic.prometheusCredentialSecret
password: # Prometheus password set in nsic.prometheusCredentialSecret
scrape_interval: 30s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_netscaler_prometheus_scrape]
action: keep
regex: true
- source_labels: [__address__, __meta_kubernetes_pod_annotation_netscaler_prometheus_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
Note:
For more details on Prometheus integration, please refer to this
NetScaler CPX License Provisioning
Bandwidth based licensing
By default, CPX runs with 20 Mbps bandwidth called as CPX Express. However, for better performance and production deployments, customer needs licensed CPX instances. NetScaler ADM is used to check out licenses for NetScaler CPX. For more detail on CPX licensing please refer this.
For provisioning licensing on NetScaler CPX, it is mandatory to provide License Server information to CPX. This can be done by setting ADMSettings.licenseServerIP as License Server IP. In addition to this, ADMSettings.bandWidthLicense needs to be set true and desired bandwidth capacity in Mbps should be set ADMSettings.bandWidth. For example, to set 2Gbps as bandwidth capacity, below command can be used.
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes --set ADMSettings.licenseServerIP=<LICENSESERVER_IP_OR_FQDN>,ADMSettings.bandWidthLicense=True --set ADMSettings.bandWidth=2000,ADMSettings.licenseEdition="ENTERPRISE"
vCPU based licensing
For vCPU based licensing on NetScaler CPX, set ADMSettings.vCPULicense
as True and ADMSettings.cpxCores
with the number of cores that can be allocated for the CPX.
helm repo add netscaler https://netscaler.github.io/netscaler-helm-charts/
helm install my-release netscaler/netscaler-cpx-with-ingress-controller --set license.accept=yes --set ADMSettings.licenseServerIP=<LICENSESERVER_IP_OR_FQDN>,ADMSettings.vCPULicense=True --set ADMSettings.cpxCores=4,ADMSettings.licenseEdition="ENTERPRISE"
Bootup Configuration for NetScaler CPX
To add bootup config on NetScaler CPX, add commands below cpxCommands
and cpxShellCommands
in the values.yaml file. The commands will be executed in order.
For e.g. to add X-FORWARDED-PROTO
header in all request packets processed by the CPX, add below commands under cpxCommands
in the values.yaml
file.
cpxCommands: |
add rewrite action rw_act_x_forwarded_proto insert_http_header X-Forwarded-Proto "\"https\""
add rewrite policy rw_pol_x_forwarded_proto CLIENT.SSL.IS_SSL rw_act_x_forwarded_proto
bind rewrite global rw_pol_x_forwarded_proto 10 -type REQ_OVERRIDE
Commands that needs to be executed in shell of CPX should be kept under cpxShellCommands
in the values.yaml
file.
cpxShellCommands: |
touch /etc/a.txt
echo "this is a" > /etc/a.txt
echo "this is the file" >> /etc/a.txt
ls >> /etc/a.txt
Configuration
The following table lists the configurable parameters of the NetScaler CPX with NetScaler ingress controller as side car chart and their default values.
Parameters | Mandatory or Optional | Default value | Description |
---|---|---|---|
license.accept | Mandatory | no | Set yes to accept the NetScaler ingress controller end user license agreement. |
imageRegistry | Mandatory | quay.io |
The NetScaler CPX image registry |
imageRepository | Mandatory | netscaler/netscaler-cpx |
The NetScaler CPX image repository |
imageTag | Mandatory | 14.1-25.111 |
The NetScaler CPX image tag |
pullPolicy | Mandatory | IfNotPresent | The NetScaler CPX image pull policy. |
daemonSet | Optional | False | Set this to true if NetScaler CPX needs to be deployed as DaemonSet. |
hostName | Optional | N/A | This entity will be used to set Hostname of the CPX |
nsic.imageRegistry | Mandatory | quay.io |
The NetScaler ingress controller image registry |
nsic.imageRepository | Mandatory | netscaler/netscaler-k8s-ingress-controller |
The NetScaler ingress controller image repository |
nsic.imageTag | Mandatory | 2.1.4 |
The NetScaler ingress controller image tag |
nsic.pullPolicy | Mandatory | IfNotPresent | The NetScaler ingress controller image pull policy. |
nsic.required | Mandatory | true | NSIC to be run as sidecar with NetScaler CPX |
nsic.enableLivenessProbe | Optional | True | Enable liveness probe settings for NetScaler Ingress Controller |
nsic.enableReadinessProbe | Optional | True | Enable Readineess probe settings for NetScaler Ingress Controller |
nsic.livenessProbe | Optional | N/A | Set livenessProbe settings for NSIC |
nsic.readinessProbe | Optional | N/A | Set readinessProbe settings |
nsic.resources | Optional | {} | CPU/Memory resource requests/limits for NetScaler Ingress Controller container |
nsic.rbacRole | Optional | false | To deploy NSIC with RBAC Role set rbacRole=true; by default NSIC gets installed with RBAC ClusterRole(rbacRole=false)) |
nsic.prometheusCredentialSecret | Optional | N/A | The secret key required to create read only user for native export of metrics using Prometheus. |
imagePullSecrets | Optional | N/A | Provide list of Kubernetes secrets to be used for pulling the images from a private Docker registry or repository. For more information on how to create this secret please see Pull an Image from a Private Registry. |
nameOverride | Optional | N/A | String to partially override deployment fullname template with a string (will prepend the release name) |
fullNameOverride | Optional | N/A | String to fully override deployment fullname template with a string |
resources | Optional | {} | CPU/Memory resource requests/limits for NetScaler CPX container |
nitroReadTimeout | Optional | 20 | The nitro Read timeout in seconds, defaults to 20 |
logLevel | Optional | INFO | The loglevel to control the logs generated by NSIC. The supported loglevels are: CRITICAL, ERROR, WARNING, INFO, DEBUG, TRACE and NONE. For more information, see Logging. |
jsonLog | Optional | false | Set this argument to true if log messages are required in JSON format |
nsConfigDnsRec | Optional | false | To enable/disable DNS address Record addition in NetScaler through Ingress |
nsSvcLbDnsRec | Optional | false | To enable/disable DNS address Record addition in NetScaler through Type Load Balancer Service |
nsDnsNameserver | Optional | N/A | To add DNS Nameservers in NetScaler |
optimizeEndpointBinding | Optional | false | To enable/disable binding of backend endpoints to servicegroup in a single API-call. Recommended when endpoints(pods) per application are large in number. Applicable only for NetScaler Version >=13.0-45.7 |
defaultSSLCertSecret | Optional | N/A | Provide Kubernetes secret name that needs to be used as a default non-SNI certificate in NetScaler. |
defaultSSLSNICertSecret | Optional | N/A | Provide Kubernetes secret name that needs to be used as a default SNI certificate in NetScaler. |
nsHTTP2ServerSide | Optional | OFF | Set this argument to ON for enabling HTTP2 for NetScaler service group configurations. |
cpxLicenseAggregator | Optional | N/A | IP/FQDN of the CPX License Aggregator if it is being used to license the CPX. |
nsCookieVersion | Optional | 0 | Specify the persistence cookie version (0 or 1). |
profileSslFrontend | Optional | N/A | Specify the frontend SSL profile. For Details see Configuration using FRONTEND_SSL_PROFILE |
profileTcpFrontend | Optional | N/A | Specify the frontend TCP profile. For Details see Configuration using FRONTEND_TCP_PROFILE |
profileHttpFrontend | Optional | N/A | Specify the frontend HTTP profile. For Details see Configuration using FRONTEND_HTTP_PROFILE |
logProxy | Optional | N/A | Provide Elasticsearch or Kafka or Zipkin endpoint for NetScaler observability exporter. |
nsProtocol | Optional | http | Protocol http or https used for the communication between NetScaler Ingress Controller and CPX |
nsEnableLabel | Optional | True | Set to true for plotting Servicegraph. Ensure analyticsConfig are set. |
cpxBgpRouter | Optional | false | If set to true, this CPX is deployed as daemonset in BGP controller mode wherein BGP advertisements are done for attracting external traffic to Kubernetes clusters |
replicaCount | Optional | 1 | Number of CPX-NSIC pods to be deployed. With cpxBgpRouter : true , replicaCount is 1 since CPX will be deployed as DaemonSet |
nsIP | Optional | 192.168.1.2 | NSIP used by CPX for internal communication when run in Host mode, i.e when cpxBgpRouter is set to true. A /24 internal network is created in this IP range which is used for internal communications withing the network namespace. |
nsGateway | Optional | 192.168.1.1 | Gateway used by CPX for internal communication when run in Host mode, i.e when cpxBgpRouter is set to true. If not specified, first IP in the nsIP network is used as gateway. It must be in same network as nsIP |
bgpPort | Optional | 179 | BGP port used by CPX for BGP advertisement if cpxBgpRouter is set to true |
ingressIP | Optional | N/A | External IP address to be used by ingress resources if not overriden by ingress.com/frontend-ip annotation in Ingress resources. This is also advertised to external routers when pxBgpRouter is set to true |
entityPrefix | Optional | k8s | The prefix for the resources on the NetScaler CPX. |
ingressClass | Optional | N/A | If multiple ingress load balancers are used to load balance different ingress resources. You can use this parameter to specify NetScaler ingress controller to configure NetScaler associated with specific ingress class. For more information on Ingress class, see Ingress class support. For Kubernetes version >= 1.19, this will create an IngressClass object with the name specified here |
setAsDefaultIngressClass | Optional | False | Set the IngressClass object as default. New Ingresses without an "ingressClassName" field specified will be assigned the class specified in ingressClass. Applicable only for kubernetes versions >= 1.19 |
updateIngressStatus | Optional | False | Set this argument if you want to update ingress status of the ingress resources exposed via CPX. This is only applicable if servicetype of CPX service is LoadBalancer. |
disableAPIServerCertVerify | Optional | False | Set this parameter to True for disabling API Server certificate verification. |
openshift | Optional | false | Set this argument if OpenShift environment is being used. |
disableOpenshiftRoutes | false | By default Openshift routes are processed in openshift environment, this variable can be used to disable Ingress controller processing the openshift routes. | |
routeLabels | Optional | proxy in () | You can use this parameter to provide the route labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster. |
namespaceLabels | Optional | N/A | You can use this parameter to provide the namespace labels selectors to be used by NetScaler Ingress Controller for routeSharding in OpenShift cluster. |
sslCertManagedByAWS | Optional | False | Set this argument if SSL certs used is managed by AWS while deploying NetScaler CPX in AWS. |
nodeSelector.key | Optional | N/A | Node label key to be used for nodeSelector option for CPX-NSIC deployment. |
nodeSelector.value | Optional | N/A | Node label value to be used for nodeSelector option in CPX-NSIC deployment. |
podAnnotations | Optional | N/A | Map of annotations to add to the pods. |
affinity | Optional | N/A | Affinity labels for pod assignment. |
tolerations | Optional | N/A | Specify the tolerations for the CPX-NSIC deployment. |
serviceType.loadBalancer.enabled | Optional | False | Set this argument if you want servicetype of CPX service to be LoadBalancer. |
serviceType.nodePort.enabled | Optional | False | Set this argument if you want servicetype of CPX service to be NodePort. |
serviceType.nodePort.httpPort | Optional | N/A | Specify the HTTP nodeport to be used for NodePort CPX service. |
serviceType.nodePort.httpsPort | Optional | N/A | Specify the HTTPS nodeport to be used for NodePort CPX service. |
serviceAnnotations | Optional | N/A | Dictionary of annotations to be used in CPX service. Key in this dictionary is the name of the annotation and Value is the required value of that annotation. For example, see this. |
serviceSpec.externalTrafficPolicy | Optional | Cluster | Use this parameter to provide externalTrafficPolicy for CPX service of type LoadBalancer or NodePort. serviceType.loadBalancer.enabled or serviceType.nodePort.enabled should be set to true according to your use case for using this parameter. |
serviceSpec.loadBalancerIP | Optional | N/A | Use this parameter to provide LoadBalancer IP to CPX service of type LoadBalancer. serviceType.loadBalancer.enabled should be set to true for using this parameter. |
serviceSpec.loadBalancerSourceRanges | Optional | N/A | Provide the list of IP Address or range which should be allowed to access the Network Load Balancer. serviceType.loadBalancer.enabled should be set to true for using this parameter. For details, see Network Load Balancer support on AWS. |
servicePorts | Optional | N/A | List of port. Each element in this list is a dictionary that contains information about the port. For example, see this. |
ADMSettings.licenseServerIP | Optional | N/A | Provide the NetScaler Application Delivery Management (ADM) IP address to license NetScaler CPX. For more information, see Licensing. |
ADMSettings.licenseServerPort | Optional | 27000 | NetScaler ADM port if non-default port is used. |
ADMSettings.ADMIP | Optional | N/A | NetScaler Application Delivery Management (ADM) IP address. |
ADMSettings.loginSecret | Optional | N/A | The secret key to login to the ADM. For information on how to create the secret keys, see Prerequisites. |
ADMSettings.bandWidthLicense | Optional | False | Set to true if you want to use bandwidth based licensing for NetScaler CPX. |
ADMSettings.bandWidth | Optional | 1000 | Desired bandwidth capacity to be set for NetScaler CPX in Mbps. |
ADMSettings.vCPULicense | Optional | N/A | Set to true if you want to use vCPU based licensing for NetScaler CPX. |
ADMSettings.licenseEdition | Optional | PLATINUM | License edition that can be Standard, Platinum and Enterprise . By default, Platinum is selected. |
ADMSettings.cpxCores | Optional | 1 | Desired number of vCPU to be set for NetScaler CPX. |
exporter.required | Optional | false | Use the argument if you want to run the Exporter for NetScaler Stats along with NetScaler ingress controller to pull metrics for the NetScaler CPX |
exporter.imageRegistry | Optional | quay.io |
The Exporter for NetScaler Stats image registry |
exporter.imageRepository | Optional | netscaler/netscaler-adc-metrics-exporter |
The Exporter for NetScaler Stats image repository |
exporter.imageTag | Optional | 1.4.9 |
The Exporter for NetScaler Stats image tag |
exporter.pullPolicy | Optional | IfNotPresent | The Exporter for NetScaler Stats image pull policy. |
exporter.resources | Optional | {} | CPU/Memory resource requests/limits for Metrics exporter container |
exporter.ports.containerPort | Optional | 8888 | The Exporter for NetScaler Stats container port. |
exporter.serviceMonitorExtraLabels | Optional | Extra labels for service monitor whem NetScaler-adc-metrics-exporter is enabled. | |
analyticsConfig.required | Mandatory | false | Set this to true if you want to configure NetScaler to send metrics and transaction records to analytics service. |
analyticsConfig.distributedTracing.enable | Optional | false | Set this value to true to enable OpenTracing in NetScaler. |
analyticsConfig.distributedTracing.samplingrate | Optional | 100 | Specifies the OpenTracing sampling rate in percentage. |
analyticsConfig.endpoint.metrics.service | Optional | N/A | Set this value as the IP address or DNS address of the analytics server. Format: servicename.namespace, servicename.namespace.svc.cluster.local, namespace/servicename *** This value replaces the analyticsConfig.endpoint.server value used earlier. *** |
analyticsConfig.endpoint.transactions.service | Optional | N/A | Set this value as the IP address or service name with namespace of the analytics service deployed in k8s environment. Format: namespace/servicename *** This value replaces the analyticsConfig.endpoint.service value used earlier. *** |
analyticsConfig.timeseries.port | Optional | 5563 | Specify the port used to expose analytics service for timeseries endpoint. |
analyticsConfig.timeseries.metrics.enable | Optional | Set this value to true to enable sending metrics from NetScaler. | |
analyticsConfig.timeseries.metrics.mode | Optional | avro | Specifies the mode of metric endpoint. |
analyticsConfig.timeseries.metrics.exportFrequency | Optional | 30 | Specifies the time interval for exporting time-series data. Possible values range from 30 to 300 seconds. |
analyticsConfig.timeseries.metrics.schemaFile | Optional | schema.json | Specifies the name of a schema file with the required Netscaler counters to be added and configured for metricscollector to export. A reference schema file reference_schema.json with all the supported counters is also available under the path /var/metrics_conf/. This schema file can be used as a reference to build a custom list of counters. |
analyticsConfig.timeseries.metrics.enableNativeScrape | Optional | false | Set this value to true for native export of metrics. |
analyticsConfig.timeseries.auditlogs.enable | Optional | false | Set this value to true to export audit log data from NetScaler. |
analyticsConfig.timeseries.events.enable | Optional | false | Set this value to true to export events from the NetScaler. |
analyticsConfig.transactions.enable | Optional | false | Set this value to true to export transactions from NetScaler. |
analyticsConfig.transactions.port | Optional | 5557 | Specify the port used to expose analytics service for transaction endpoint. |
bgpSettings.required | Optional | false | Set this argument if you want to enable BGP configurations for exposing service of Type Loadbalancer through BGP fabric |
bgpSettings.bgpConfig | Optional | N/A | This represents BGP configurations in YAML format. For the description about individual fields, please refer the documentation |
nsLbHashAlgo.required | Optional | false | Set this value to set the LB consistent hashing Algorithm |
nsLbHashAlgo.hashFingers | Optional | 256 | Specifies the number of fingers to be used for hashing algorithm. Possible values are from 1 to 1024, Default value is 256 |
nsLbHashAlgo.hashAlgorithm | Optional | 'default' | Specifies the supported algorithm. Supported algorithms are "default", "jarh", "prac", Default value is 'default' |
cpxCommands | Optional | N/A | This argument accepts user-provided NetScaler bootup config that is applied as soon as the CPX is instantiated. Please note that this is not a dynamic config, and any subsequent changes to the configmap don't reflect in the CPX config unless the pod is restarted. For more info, please refer the documentation. |
cpxShellCommands | Optional | N/A | This argument accepts user-provided bootup config that is applied as soon as the CPX is instantiated. Please note that this is not a dynamic config, and any subsequent changes to the configmap don't reflect in the CPX config unless the pod is restarted. For more info, please refer the documentation. |
enableStartupProbe | Optional | True | Enable startupProbe settings for CPX |
enableLivenessProbe | Optional | True | Enable livenessProbe settings for CPX |
startupProbe | Optional | N/A | Set startupProbe settings for CPX |
livenessProbe | Optional | N/A | Set livenessProbe settings for CPX |
Note:
If NetScaler ADM related information is not provided during installation, NetScaler CPX will come up with the default license.
Alternatively, you can define a YAML file with the values for the parameters and pass the values while installing the chart.
For example:
helm install netscaler-cpx-with-ingress-controller netscaler/netscaler-cpx-with-ingress-controller -f values.yaml
Tip:
The values.yaml contains the default values of the parameters.
Uninstalling the Chart
To uninstall/delete the my-release
deployment:
helm delete my-release