View on GitHub

k8s-jcasc-management

Jenkins management for bash for Kubernetes, Jenkins Configuration as Code (JCasC) and Jenkins JobDSL.

Deprecated and new tool

This tool is now available as Go-Version under https://github.com/Ragin-LundF/k8s-jcasc-management-go with enhanced features.

Table of content

Kubernetes Jenkins as Code management

This project offers a template for managing Jenkins instances on Kubernetes with a JobDSL and Jenkins Configuration as Code (JcasC).

To simplify the installation and the project settings, it has a small helper tool k8s-jcasc.sh, which can be used in wizard mode or via arguments to

If you want to use existing persistent volume claims, then you have to create a persistent volume before you install the application.

The password for the preconfigured secrets file is admin. There is no valid data inside this file! Please change it for your own project!

As default the system uses encrypted passwords instead of using the password from the jenkins_helm_values.yaml. The default users and passwords are:

This can be changed on the jcasc_config.yaml file under the jenkins.securityRealm section.

Prerequisites

To use this tool, you need to have the following tools installed:

Optional the tool can use whiptail (newt) or dialog for a better workflow.

Installation whiptail/newt

Installation dialog

Basic concept

alt text

Advantages

By having all things stored in VCS repositories, which are normally backed up, it is possible to recreate every instance in no-time. It is impossible to misconfigure a Jenkins instance, because the configuration can be reloaded from this remote repository and all configurations are completely versioned.

Also, every develops maybe can have admin access to play around with the Jenkins, because they can not destroy the system permanently with the beloved “I have nothing done…” statement.

If the K8S cluster or server crashes, it is possible to redeploy everything as it was in minutes, because also the job definition is stored in a VCS repository.

Build slaves

The pre-defined slave-containers will not work directly. Every build slave container needs to set up the jenkins home work directory and jenkins user/group with uid/gid 1000.

Also, the build slaves did not need to have any jenkins agent or something else. Only the user/group and the workdir is needed.

To resolve the problem, that build containers directly shut down, simply add an entrypoint with a tail -f /dev/null.

You can also create a Jenkins build slave base container and build your own build tools container on top of it.

Example of a jenkins-build-slave-base-container:

FROM alpine:3.10

ARG VERSION=1.0.0
LABEL Description="Jenkins Build Slave Base Container" Vendor="K8S_MGMT" Version="${VERSION}"

###### GLIBC for alpine image
# GLIBC-ENVIROMENT
ENV GLIBC_LANG=en_US
ENV GLIBC_VERSION=2.28-r0
ENV LANG=${GLIBC_LANG}.UTF-8
ENV LANGUAGE=${GLIBC_LANG}.UTF-8

# install base packages, that will be used in most containers
RUN apk update && apk -U upgrade -a && \
    apk add --no-cache xz tar zip unzip sudo curl wget bash git git-lfs procps ca-certificates

# GET GLIBC FROM SGERRAND: https://github.com/sgerrand/alpine-pkg-glibc
RUN wget -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-i18n-${GLIBC_VERSION}.apk && \
    apk add --no-cache glibc-${GLIBC_VERSION}.apk glibc-bin-${GLIBC_VERSION}.apk glibc-i18n-${GLIBC_VERSION}.apk && \
    rm -f /etc/apk/keys/sgerrand.* && \
    echo "export GLIBC_LANG=${LANG}" > /etc/profile.d/locale.sh && \
    echo "LANG=${LANG}" >> /etc/environment && \
    /usr/glibc-compat/bin/localedef -i ${GLIBC_LANG} -f UTF-8 ${GLIBC_LANG}.UTF-8 && \
    rm *.apk && \
    echo "Installing additional packages... done"

###### Jenkins setup
# Required Jenkins user/group/gid/uid/workdir
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG AGENT_WORKDIR=/home/${user}/agent

# create jenkins user
RUN addgroup -g ${gid} ${group} && adduser -h /home/${user} -u ${uid} -G ${group} -D ${user}

# create directories and permissions
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}

VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}

WORKDIR /home/${user}

# let the container tail /dev/null, that Kubernetes will not shut down the container directly after startup.
ENTRYPOINT ["tail", "-f", "/dev/null"]

A build-slave container for docker can look then like this:

FROM jenkins-slave-base
ARG VERSION=1.0.0
LABEL Description="Docker container with Docker for executing docker build and docker push" Vendor="K8S_MGMT" Version="${VERSION}"

# Installing docker
RUN apk update && apk -U upgrade -a && \
    apk add --no-cache docker

# adding jenkins user to docker group
RUN addgroup -S ${user} docker

Configuration

The system has a basic configuration file to pre-configure some global settings. This file is located under config/k8s_jcasc_mgmt.cnf.

It is recommended to change the PROJECTS_BASE_DIRECTORY to a directory outside of this project. The createproject command will create new projects as subfolders of this directory. All files and directories under the `PROJECTS_BASE_DIRECTORY’ should be passed to a git repository which is backed up.

Then your existing Jenkins projects can be fully recovered from this repository.

Configure alternative configuration with overlays

To use this repository “as-it-is”, it is possible to create a config/k8s_jcasc_custom.cnf file.

This file can contain the following configuration:

# Define path to alternative configuration file
K8S_MGMT_ALTERNATIVE_CONFIG_FILE=/my/path/to/my.config

# Is the alternative configuration only a overlay or a full configuration?
# (true = both config files will be used, false = only the alternative config will be used)
K8S_MGMT_WORK_AS_OVERLAY=true

The script checks, if this file exists. If this is the case, it loads this configuration and checks the argument for the path of the alternative config file.

This means, that the K8S_MGMT_ALTERNATIVE_CONFIG_FILE key can define, where the alternative of the k8s_jcasc_mgmt.cnf is located. In the .gitignore file, this file is set to ignore, to prevent a commit.

With the K8S_MGMT_WORK_AS_OVERLAY key it is possible to tell the system to first load the original configuration (k8s_jcasc_mgmt.cnf) and then the alternative config, which was defined with the K8S_MGMT_ALTERNATIVE_CONFIG_FILE key. In the new configuration, it is only required to overwrite the options, that has to be changed instead of writing a new file with the complete configuration, which can result in update problems.

How to use

The simplest way is to call the script without arguments. Everything else will be asked by the script.

Hint: Before you install the Jenkins, you have to commit the files of your project directory and to ensure, that the jasc_config.yaml file is readable for Jenkins (public)

./k8s-jcasc.sh

For less selection and more control you can also give some arguments and the command to the script:

./k8s-jcasc.sh <arguments> <command>

The order of the arguments and commands are irrelevant.

k8s-jcasc.sh arguments

It is possible to use multiple arguments. The following arguments are supported:

Argument Description Example
-p= or --projectdir= Defines the project directory (or project name) of the Jenkins configuration. This directory is a subdirectory of the configured PROJECTS_BASE_DIRECTORY -p=myproject or --projectdir=myproject
-n= or --namespace= Defines the target namespace in Kubernetes. It is not used for encrypting or decrypting secrets. -n=jenkins-namespace or --namespace=jenkins-namespace
-d= or --deploymentname= Defines the deployment name, which is relevant only for install and uninstall. This can also be configured globally for all projects as JENKINS_MASTER_DEPLOYMENT_NAME in the config file. -d=jenkins-master or --deploymentname=jenkins-master
-o= or --output-dry-run-file Executes helm commands with --dry-run --debug option and writes the output into the given file -o=./debug.log or --output-dry-run-file=./debug.log
--nodialog If dialog is installed, but you dont want to use it n/a

k8s-jcasc.sh commands

Only one command can be used. Multiple commands are NOT supported. The following commands are supported:

Command Description
install Install Jenkins to a Kubernetes namespace (helm install).
uninstall Uninstall the Jenkins instance of a Kubernetes namespace (helm uninstall).
upgrade Upgrade the Jenkins instance of a Kubernetes namespace (helm upgrade).
encryptsecrets Encrypt the secrets (global secrets or project secrets, depending on configuration).
decryptsecrets Decrypt the secrets (global secrets or project secrets, depending on configuration).
applysecrets Apply the secrets to the Kubernetes namespace (global secrets or project secrets, depending on configuration).
applySecretsToAll Apply the secrets to all known namespaces in Kubernetes (global secrets only).
createproject Create a new Jenkins project for the configuration and deployment values from the templates. It uses a wizard to ask for relevant data.
createdeploymentonlyproject Create a new project without Jenkins for deployments. This can be used to reserve/manage the IP inside the tool and to install a loadbalancer and ingress controller to this namespace.
createJenkinsUserPassword Create a new bcryted Jenkins user password with htpasswd. You can also use this online site to create a password: https://www.devglan.com/online-tools/bcrypt-hash-generator

Templates

At the templates directory contains the following:

Deployment-only Namespaces

jenkins_helm_values.yaml offers the possibility to add other namespaces for a Jenkins instance, that should deploy. The default for this section is empty:

[...]
k8smanagement:
  rbac:
    # list of additional namespaces that should be deployable (adds RBAC roles to those namespaces)
    additionalNamespaces: {}

To let the Jenkins install applications into other namespaces, these namespaces can be added here. It is not required to add the namespace in which the Jenkins instance is running. The Helm Chart will then add additional Roles and RoleBindings to these namespaces for this instance.

This is currently a manual process after creating a new project!

Example: In this example we add the namespaces myapplication-qa, myapplication-preview and myapplication-production to this Jenkins instance. After this was deployed, Jenkins can now deploy the application into them.

[...]
k8smanagement:
  rbac:
    # list of additional namespaces that should be deployable (adds RBAC roles to those namespaces)
    additionalNamespaces:
      - myapplication-qa
      - myapplication-preview
      - myapplication-production

Sub-Templates (cloud-templates)

k8s-jcasc-management supports additional sub-templates to create projects with dynamic container configuration. These templates are located in the ./templates/cloud-templates/ directory. If this directory does not exist, the create project wizard will not ask for other sub-templates.

All files stored there can be selected with the process/menu create project and will added to the jcasc_config.yaml.

The file jcasc_config.yaml should now have a ##K8S_MGMT_JENKINS_CLOUD_TEMPLATES## placeholder:

  clouds:
    - kubernetes:
        name: "jenkins-build-slaves"
        serverUrl: ""
        serverCertificate: ##KUBERNETES_SERVER_CERTIFICATE##
        directConnection: false
        skipTlsVerify: true
        namespace: "##NAMESPACE##"
        jenkinsUrl: "http://##JENKINS_MASTER_DEPLOYMENT_NAME##:8080"
        maxRequestsPerHostStr: 64
        retentionTimeout: 5
        connectTimeout: 10
        readTimeout: 20
        templates:
##K8S_MGMT_JENKINS_CLOUD_TEMPLATES##

It is important, that the placeholder is at the beginning of the line.

If files are inside of this directory, the user can then select which (none or multiple) sub-templates should be added to the main template.

These sub-templates must also start on the beginning of the line. For an example have a look here: templates/cloud-templates/node.yaml

Execution of Scripts

It is also possible to create shell scripts for namespaces/directories. This can be helpful, if you want to install other tools besides Jenkins. The k8s-jcasc.sh tool first tries to install the secrets, PVC and Jenkins. After this was done it checks, if a directory called scripts is inside of the project directory and if it contains *.sh files.

These files have to follow these rules and naming conventions:

The deinstallation scripts can only be executed if the project directory matches the namespace name. This is necessary because normally only the namespace and no directory selection is required for deinstallation.

IP Management

For greater installations and also after a recovery case, it is helpful to know which Jenkins instance is running behind which loadbalancer IP on which namespace.

To provide a simple solution, the system stores these information (namespace and IP) into a configuration file, which can also be backed up. For every deployment of Jenkins, the system looks into this file and configures the loadbalancer with the IP. This also allows static DNS records.

If you create a new project via the wizard, the system also checks, if a IP address already exists to avoid IP conflicts.

Additional tools

k8sfullconfigexport

You can use this tool to export the complete Kubernetes configuration to a local k8s-manifests directory. This can help to figure out differences between clusters.

Screenshots

Standard k8s-jcasc-management

alt text

Whiptail based k8s-jcasc-management

alt text Screenshot: main menu with whiptail support

alt text Screenshot: enter a directory with whiptail support

Dialog based k8s-jcasc-management

alt text Screenshot: main menu with dialog support

alt text Screenshot: enter a directory with dialog support

Helpful links