GitOps CI chain on OpenShift with ArgoCD and Tekton

The GitOps model has become a must in recent years, and many products around this paradigm have emerged to meet these new needs. RedHat, through its OpenShift Container Platform, has extended its distribution with integrations of open source products such as ArgoCD (OpenShift GitOps) and Tekton (OpenShift Pipeline). In this blogpost, we will see the benefits of these new tools and the associated uses through a practical case. We will detail the essential functionalities of each solution and their complementarities in the context of a CI/CD GitOps integration chain.

This blogpost will be broken down into two parts:

Introducing the fundamental concepts of OpenShift Pipelines and OpenShift GitOps tools,
Setting up a GitOps CI/CD pipeline on the OpenShift platform.

OpenShift Pipelines

Overview

OpenShift Pipelines is a Cloud Native integration and continuous delivery solution for building CI/CD pipelines. It is based on the open-source Tekton project. This native Kubernetes CI/CD framework allows to automate deployments on several platforms (OpenShift, Kubernetes, Virtual Machine, serverless …).

Cloud-Native CI/CD

A Cloud Native CI/CD pipeline is built on the following 3 pillars:

Les 3 piliers d’un pipeline CI/CD Cloud-Native

Containers: Designed for containerized applications and running on Kubernetes,
Serverless: Runs without a CI/CD engine to manage and maintain (example: GitLab CI, Jenkins Server, Bitbucket…)
DevOps: Designed with microservices and distributed teams in mind.

The characteristics of Tekton

Here are the main features:

Cloud-Native CI/CD pipeline 3 pillars

Customizable: Tekton entities are fully customizable which allows DevOps engineers to create a varied catalog and make it available to developers,
Reusable: the resources are portable and can be distributed to different teams within the organization. Once defined, they are ready to be used by all teams,
Extensible: A catalog and a Hub (Tekton Hub) are available. They represent community repositories that allow you to reuse code snippets and overload them with your own developments,
Standardized: Tekton is installed and run as an extension of your Kubernetes cluster using an operator (CustomResourceDefinition and CustomResources) allowing the management of Tekton objects like any other Kubernetes object. Tekton workloads run in containers,
Scaling: To increase the resources related to your Cloud Native CI/CD workloads, simply add nodes to your cluster without having to change resource allocations or modify pipelines.

The Tekton Project

The Tekton project consists of two entities Tekton Pipelines and Tekton Triggers.

Tekton Pipelines is a component defining a set of Kubernetes CustomResource (CR) to build a cloud-native CI/CD pipeline.
Tekton Triggers is another component to detect and extract event information from various sources but also to instantiate/execute pipelines.

These two components are complementary for the implementation of a complete CI/CD integration chain.

Tekton Pipelines

The Tekton Pipelines project is composed of several types of CustomResource Kubernetes:

Task
- Step
Pipeline
TaskRun
PipelineRun

Here is the hierarchical tree structure of OpenShift Pipelines CustomResource :

Arborescence des objets Tekton

Task

Tasks are the smallest elements that make up the Tekton Pipelines project. They represent the definition/declaration of a unit of work to be executed. Examples of Tasks are the packaging of a Java project, the build of a container or the vulnerability scan of a container image.

Tasks have Input and Output parameters in order to be dynamic. It is also possible to configure Workspaces to share data between different Tasks. Tasks can be executed independently from a pipeline. It is also possible to use predefined Tasks, developed by the community.

To perform Tasks, we need to define a list of Steps to execute sequentially.

Task description

Step

Steps allow to execute actions using commands or scripts in containers (pods). They have the advantage of taking the characteristics of the Kubernetes parameters that we already know:

Image
Environment variable
Volume
ConfigMap
Secret
…

An example of a Step in a Task to package a Java project is the definition of the project version, or the creation of the jar.

1
2
3
4
- name: build
  image: maven:3.6.0-jdk-8-slim
  command: ["mvn"]
  args: ["install"]

Pipeline

The Pipeline defines the execution order of the Tasks. Like a Task, it represents the definition/declaration of a sequence of Tasks. Its role is to orchestrate the execution of Tasks using conditions, to restart Tasks and to define Inputs, Outputs and Workspaces allowing data sharing between Tasks.

Here is a diagram illustrating the orchestration of Tasks when setting up a pipeline:

Description d’un Pipeline

TaskRun

The TaskRun represents the instantiation/execution of a Task. It is executed as pods, on the Kubernetes/OpenShift cluster. It references a specific Task or allows the declaration of a Task directly in the TaskRun.

The TaskRun provides data to Tasks:

Parameter
Resource
Service Account
Workspace

Interaction between different Kubernetes CustomResource during the execution of a Task:

Description d’une TaskRun

PipelineRun

The PipelineRun allows the instantiation/execution of a Pipeline. It references a specific Pipeline or allows the declaration of a Pipeline directly in the PipelineRun.

Like a TaskRun, the PipelineRun provides data to the Pipeline:

Parameter
Resource
Service Account
Workspace

Interaction between different Kubernetes entities when running a Pipeline:

PipelineRun intercation description example

Example of PipelineRun instantiation from pipelines at different times:

PipelineRun scheduled at multiple hours example

Tekton Triggers

The Tekton Triggers project is composed of several types of CustomResource Kubernetes:

Event Listener
Trigger
- Interceptor
TriggerBinding
TriggerTemplate

Pipeline Triggered

Event Listener

The EventListener is a resource deployed in the Kubernetes/OpenShift cluster to listen for events from a third-party application (webhook). This resource allows to detect and forward the event to one or more referenced Triggers.

Trigger

The Trigger allows you to specify on which type of event the pipeline will be triggered. For example, the push of a new commit to the repository manager, the approval of a merge request… A Trigger specifies a TriggerTemplate CR, a TriggerBinding and generally an Interceptor parameter.

Interceptor

The Trigger specifies an interceptor allowing to filter the useful data, to secure the exchanges with a secret (token), to transform the meta data as well as to define and test the triggering conditions. Here is the list of interceptors:

Webhook
GitHub
GitLab
BitBucket
CEL
…

TriggerBinding

The TriggerBinding resource allows to recover the data intercepted and transformed by the Trigger and to transmit them to the TriggerTemplate.

TriggerTemplate

The TriggerTemplate specifies a template for the TaskRun or PipelineRun resources to instantiate/execute when the EventListener detects an event. It allows to expose the parameters dynamically retrieved during the event and to use them to execute the PipelineRun/TaskRun.

Here is a diagram of the execution of a classic Continuous Integration workflow:

Workflow classique d’une Intégration Continue

OpenShift GitOps

Overview

OpenShift GitOps is a declarative model-based continuous delivery (CD) solution for Kubernetes/OpenShift based on the open source ArgoCD project. It enables infrastructure configuration management as well as application updates in a Git repository manager.

The 4 fundamental pillars

The GitOps model is based on 4 fundamental pillars:

The declarative approach: the entire system is described declaratively,
The Git repository as the only source of truth: declarative changes allow you to consider changes as transactions allowing versioning, traceability or access control,
A Kubernetes operator: approved changes to the desired state are automatically applied to the system thanks to the operator scanning the Git repository in real time,
Continuous observability: software agents ensure accuracy and alert in case of discrepancies.

GitOps 4 pillars

Push vs Pull Method

When we talk about GitOps there are two types of approaches. The classic Push approach and the Pull approach more associated with GitOps.

To have a detailed description of the two types of approaches and their specificities, you can refer to an article previously written on the SoKube blog the GitOps and the Millefeuille dilemma.

The ArgoCD project

The ArgoCD project is composed of several types of resources:

AppProject
Application
ApplicationSet

In the context of this blogpost we will mainly use the AppProject and Application resources.

AppProject

The AppProject represents a logical grouping of applications. This Kubernetes CustomResource allows you to define from which Git repository manifests can be retrieved, in which cluster and namespace applications should be deployed, an RBAC section to establish access control to Kubernetes resources and finally which type of Kubernetes object can be created.

Application

An Application in ArgoCD represents an application deployed in an environment within the Kubernetes cluster. It allows you to specify the Git configuration with the repository URL, the branch, the folder or the environment in order to retrieve and deploy the Kubernetes manifests. It is also possible to configure other parameters such as the deployment destination (cluster and namespace), the synchronization policy or the type of deployment (Kubernetes native manifests, Helm or Kustomize).

Cloud-Native GitOps Pipeline

After having seen and explained the concepts related to OpenShift Pipelines and OpenShift GitOps in the first part of the article, we will in this second section, highlight the interaction between these different tools. The objective will be to build a GitOps Cloud Native CI/CD pipeline for the deployment of an application called Fruitz. This one is composed of two microservices:

A backend developed in Java Quarkus: Fruit Quarkus,

A frontend based on Angular technology.

For this demonstration, the Cloud-Native GitOps pipeline will be built only around the Java fruitz-quarkus microservice. It will be composed of a standard CI (Continuous Integration) orchestrated by OpenShift Pipelines (Tekton) and a CD (Continuous Deployment) managed by OpenShift GitOps (ArgoCD). For the purpose of this tutorial, we have intentionally introduced an application limitation (bug) on the Java fruitz-quarkus backend. The objective is to correct this limitation using the GitOps Cloud-Native CI/CD pipeline and to demonstrate the benefits.

To respect the GitOps pattern, the application code and the deployment code (GitOps) will be hosted in two distinct repositories on the The DevSecOps Platform platform. The Fruit-Deploy GitOps repository will use the Helm manager package to deploy the application within the OpenShift cluster.

Workflow GitOps CI/CD pipeline:

Workflow d’un pipeline CI/CD GitOps

Installation of tools

Installation of OpenShift Pipelines and OpenShift GitOps operators

This demo will be orchestrated around the Red hat OpenShift platform. You will need to have a working OpenShift cluster with cluster-admin permissions. If you want to deploy an OpenShift cluster on AWS you can refer to the article How to deploy an Openshift cluster in AWS on the SoKube blog.

Connect to the OpenShift console and install the RedHat OpenShift GitOps and RedHat OpenShift Pipelines operators:

operateurs installation des

StorageClass installation

In order to be able to provision volumes (PersitentVolume) dynamically during the execution of Tekton pipelines, but also to deploy applications, you can, for testing purposes, deploy and use the Kubernetes NFS Subdir External Provisioner project which will allow to create a StorageClass based on NFS. In our case we will use the StorageClass called nfs-client.

StorageClass nfs-client:

1
2
3
4
5
6
7
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  archiveOnDelete: "false"

OpenShift GitOps

After installing OpenShift Pipelines through the console, we will now deploy the AppProject and the ArgoCD Application. Create the following manifests:

AppProjet fruitz-deployment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: fruitz-deployment
  namespace: openshift-gitops
  labels:
    owner: sokube
    scope: fruitz
spec:
  description: ArgoCD Projet for the Fruitz Deploylent applications
  # Allow manifests to deploy only from this repositories 
  sourceRepos:
    -  git@gitlab.com:sokube-io/sample-apps/fruitz/fruitz-deploy.git
  # Only permit to deploy applications in the following clusters & namespaces
  destinations:
    - namespace: fruitz
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: '*'
    kind: Namespace
  # Enables namespace orphaned resource monitoring.
  orphanedResources:
    warn: false

Application fruitz-helm :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fruitz-helm
  labels:
    owner: sokube
    scope: fruitz
  namespace: openshift-gitops
spec:
  # Link the ArgoCD application to the fruitz-deployment AppProject 
  project: fruitz-deployment
  # Configure the synchronization of the ArgoCD application
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    
    syncOptions:
      - CreateNamespace=true
  # The specifications of the GitOps source code to be deployed in the cluster 
  source:
    path: helm
    targetRevision: redhat-opentour-geneva
    repoURL: 'git@gitlab.com:sokube-io/sample-apps/fruitz/fruitz-deploy.git'
    helm:
      valueFiles:
        - values.yaml
  # The target kubernetes cluster in which to deploy manifests  
  destination:
    server: https://kubernetes.default.svc
    namespace: fruitz

To allow OpenShift GitOps to access the repository where the code is hosted, if it does not have public visibility, then it is necessary to create a Kubernetes Secret with the private key allowing ArgoCD to access it:

Secret private-repo :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: openshift-gitops
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@gitlab.com:sokube-io/sample-apps/fruitz/fruitz-deploy.git
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

ℹ️ Remember to put your SSH private key.

From the OpenShift console, click on the small menu, then select Cluster ArgoCD to get to the ArgoCD application:

OpenShift GitOps logo

OpenShift GitOps allows SSO integration using the one already built into OpenShift. On the OpenShift GitOps authentication page select LOG IN VIA OPENSHIFT to use the OpenShift SSO and authenticate:

OpenShift GitOps login

Then on the page of the applications ArgoCD, we can note that the application fruitz-helm is present as well as in state Synced:

OpenShift GitOps synced app

Now, you can click on the application to get an overview of all deployed objects:

OpenShift GitOps global app view

On the fruitz-front-ingress object at the bottom of the page, you can click to open the frontend web page of the application:

You are now on the frontend page of the application. You can test that it works correctly with the addition of the fruit Pear. However, adding all the fruits does not work. Indeed, we can also try to add the fruit Pineapple, and see that the fruit is not added. This behavior/bug is of course expected. It is this one in particular that highlights the benefit of combining the ArgocCD and Tekton tools with the implementation of a GitOps Cloud Native CI/CD pipeline correcting the intentionally introduced bug/behavior.

We add the Pear fruit that now appears on the web interface. We then try to add the Pineapple fruit which does not work and does not appear on the web interface:

Fruitz application

When analyzing the stack trace by querying the backend pod logs, we see the following error ERROR: value too long for type character varying(6). The size of the string is too long:

Fruitz application logs

Indeed, the size of the string is limited to 6. This corresponds to the yellow highlighted code:

Fruitz application source code

We explicitly specified the constraint that the string for the fruit name could not exceed 6 characters.

OpenShift Pipelines

After demonstrating that our application was functional but intentionally limited, in this part we will show how to have a complete integration chain allowing the developer to accelerate the development of his application by creating a Continuous Integration (CI) pipeline directly from the commit of a new feature thanks to Tekton and deploy the new image in the OpenShift cluster thanks to ArgoCD.

Tekton Trigger

Create the OpenShift cicd project (namespace):

1
oc new-project cicd

You will need to create the following manifests:

An EventListener: this resource allows you to create an entry point on our OpenShift cluster in order to trigger the pipelines:
A TriggerBinding: this resource is used to bind the retrieved metadatas to the pipelines
A Trigger: this resource allows to define the type of interceptor (e.g. GitLab, GitHub …) but also on which type of event to trigger the pipelines but also to extract and transform on the recovered metadatas
A Secret used by the Trigger containing the token which will be associated to the webhook allowing the authentication
A TriggerTemplate allowing to variabilize the pipeline that will be triggered.

EventListener gitlab-listener-interceptor:

1
2
3
4
5
6
7
8
9
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: gitlab-listener-interceptor
  namespace: cicd
spec:
  serviceAccountName: pipeline
  triggers:
    - triggerRef: gitlab-listener

Creating this resource results in the creation of an OpenShift Route. You will need the host linked to this route when creating a webhook. To retrieve the host linked to the route, enter the following command:

1
oc -n cicd get route -ojsonpath='{.items[*].status.ingress[*].host}'

The result is as follows:

1
el-gitlab-listener-interceptor-cicd.apps.ocp-dev.infrasokube.io

TriggerBinding gitlab-triggerbinding:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
  name: gitlab-triggerbinding
  namespace: cicd
spec:
  params:
  - name: buildRevision
    value: $(body.checkout_sha)
  - name: gitrepositoryurl
    value: $(body.repository.git_ssh_url)
  - name: buildRevisionShort
    value: $(extensions.short_sha)    
  - name: buildRevisionBranch
    value: $(extensions.branch_name)

Trigger gitlab-listener:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: triggers.tekton.dev/v1beta1
kind: Trigger
metadata:
  name: gitlab-listener
  namespace: cicd
spec:
  serviceAccountName: pipeline
  interceptors:
    - name: gitlab 
      ref:
        name: "gitlab"
      params:
        - name: "secretRef"
          value:
            secretName: gitlab-trigger-secret
            secretKey: secretToken
        - name: "eventTypes"
          value: ["Push Hook"]
    - name: custom-parameters
      ref:
        name: "cel"
      params:
        - name: "overlays"
          value:
          - key: short_sha
            expression: "body.checkout_sha.truncate(8)"
          - key: branch_name
            expression: "body.ref.split('/')[2]"
  bindings:
    - ref: gitlab-triggerbinding
  template:
    ref: gitlab-triggertemplate

The associated secret gitlab-trigger-secret:

1
2
3
4
5
6
7
8
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-trigger-secret
  namespace: cicd
type: Opaque
stringData:
  secretToken: "abcdefghijklmnopqrstuvwxyz0123456789ABCDEF"

⚠️ This secret should also be used when creating the webhook on GitLab.

TriggerTemplate gitlab-triggertemplate :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: gitlab-triggertemplate
  namespace: cicd
spec:
  params:
  - name: buildRevision
    description: The Git commit revision
  - name: buildRevisionShort
    description: The Git commit revision
  - name: gitrepositoryurl
    description: The git repository url
  - name: buildRevisionBranch
    description: The git branch

  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: fruitz-quarkus-pipelinerun- 
      spec:
        serviceAccountName: pipeline
        pipelineRef:
          name: fruitz-quarkus
        params:
          - name: buildRevision
            value: $(tt.params.buildRevision)
          - name: buildRevisionShort
            value: $(tt.params.buildRevisionShort)
          - name: buildRevisionBranch
            value: $(tt.params.buildRevisionBranch)
        workspaces:
          - name: shared-workspace
            volumeClaimTemplate:
              spec:
                storageClassName: "nfs-client"
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 1Gi
        resources:
          - name: git-source-fruitz-application
            resourceSpec:
              type: git
              params:
                - name: revision
                  value: $(tt.params.buildRevisionBranch)
                - name: url
                  value: $(tt.params.gitrepositoryurl)
          - name: git-source-fruitz-deployment
            resourceSpec:
              type: git
              params:
                - name: revision
                  value: redhat-opentour-geneva
                - name: url
                  value: git@gitlab.com:sokube-io/sample-apps/fruitz/fruitz-deploy.git

        taskRunSpecs:
          - pipelineTaskName: maven-package
            taskServiceAccountName: pipeline
            taskPodTemplate:
              volumes:
                - name: config-volume
                  configMap:
                    name: mvn-settings

After creating the different Kubernetes resources, we need to create a webhook on the Java fruitz-quarkus application repository that will allow to take into account the changes according to the type of event and to transmit them to Tekton. In our case, when a new commit is pushed to the fruitz-quarkus repository, a Continuous Integration (CI) pipeline will be automatically triggered.

As a reminder, we use the GitLab SCM. In the project, please go to the sidebar and click on Settings ► Webhook:

Then enter the following information:

URL: https://el-gitlab-listener-interceptor-cicd.apps.ocp-dev.infrasokube.io
Secret Token: abcdefghijklmnopqrstuvwxyz0123456789ABCDEF
Trigger:
- Push events
  - All branches
SSL verification
- Enable SSL verification

The URL field represents the URL of the OpenShift Route that is created by the Tekton EventListener object. This is the URL on which Tekton Trigger listens and receives the webhooks sent by GitLab. It allows to trigger pipelines but also the transmission of information and meta data to Tekton.

Le champ Secret Token permet l’authentification lors du webhook envoyé,
Le champ Trigger permet de définir sur quel type d’évènement les webhooks seront déclenchés,
Le champs SSL, quant à lui, permet de vérifier la terminaison TLS de l’URL.

GitLab create webhooks

Tekton Pipelines

Configuration

Tasks

We now need to create the tasks that will make up the application CI pipeline. Our pipeline will be simple and composed of classical tasks:

The packaging of the Quarkus Java project and the creation of the Java artifact (.jar)
The build of the Docker image with the Java Quarkus package and the push of the built image in a container registry
A vulnerability scan of the Docker image
The update of the Helm deployment repository which will allow to redeploy the application with the correction of the application bug. You will need to create the following manifests:

Create the following manifests:

Task maven-package :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: maven-package
  namespace: cicd
spec:
  description: >-
    Maven packaging Task with multiple steps

  resources:
    inputs:
      - name: git-source-fruitz-application
        type: git

  workspaces:
    - name: shared-workspace

  params:
    - name: mavenSettingsPath
      type: string
      description: The location path of the maven settings file
      default: "/tmp"
    - name: mavenSettingsFileName
      type: string
      description: The name of the maven settings file
      default: "mvn-settings.xml"
    - name: mavenContainerImage
      type: string
      description: The name of maven container image
      default: maven:3.6.3-jdk-11

  results:
    - name: mavenProjectVersion
      description: The Maven project version number

  steps:
    - name: get-maven-project-version
      image: $(params.mavenContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
      env:
        - name: "MAVEN_OPTS"
          value: "-Dmaven.repo.local=$(workspaces.shared-workspace.path)"
      script: |
        #!/usr/bin/env sh
        mvn -s $(params.mavenSettingsPath)/$(params.mavenSettingsFileName) help:evaluate \
          -Dexpression=project.version \
          -q -DforceStdout > $(results.mavenProjectVersion.path)
      volumeMounts:
        - name: config-volume
          mountPath: "$(params.mavenSettingsPath)"
  
    - name: maven-package
      image: $(params.mavenContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
      env:
        - name: "MAVEN_OPTS"
          value: "-Dmaven.repo.local=$(workspaces.shared-workspace.path)"
      script: |
        #!/usr/bin/env sh
        mvn -s $(params.mavenSettingsPath)/$(params.mavenSettingsFileName) clean package -DskipTests
      volumeMounts:
        - name: config-volume
          mountPath: "$(params.mavenSettingsPath)"
      securityContext:
        privileged: true

    - name: copy-target-artifacts-folder
      image: $(params.mavenContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
      script: |
        #!/usr/bin/env sh
        cp -R $(resources.inputs.git-source-fruitz-application.path)/target \
          $(workspaces.shared-workspace.path)

Task container-image-build-push :

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: container-image-build-push
  namespace: cicd
spec:
  description: >-
    Container image build and push task with multiple steps
  
  resources:
    inputs:
      - name: git-source-fruitz-application
        type: git

  workspaces:
    - name: shared-workspace
  
  params:
    - name: dockerRegistryName
      type: string
      description: Name of Docker registry
      default: "registry.gitlab.com"
    - name: gitlabUsernameAccountName
      type: string
      description: Name of gitlab username
      default: "sokube-io"
    - name: gitlabGroupSampleAppsName
      type: string
      description: Name of gitlab group (1st level)
      default: "sample-apps"
    - name: gitlabSubGroupFruitzName
      type: string
      description: Name of gitlab subgroup (2nd level)
      default: "fruitz"
    - name: gitlabProjectFruitzQuarkusName
      type: string
      description: Name of quarkus project
      default: "fruitz-quarkus"
    - name: mavenProjectVersion
      type: string
      description: The Maven project version number
    - name: buildRevision
      type: string
      description: The Git commit revision
    - name: buildRevisionShort
      type: string
      description: The short Git commit revision
    - name: buildRevisionBranch
      type: string
    - name: mavenContainerImage
      type: string
      description: The name of maven container image
      default: maven:3.6.3-jdk-11
    - name: buildahContainerImage
      description: The location of the buildah builder image
      default: registry.redhat.io/rhel8/buildah:latest
    - name: buildahStorageDriver
      type: string
      description: Set buildah storage driver
      default: vfs
    - name: dockerfilePath
      type: string
      description: Path to the Dockerfile to build
      default: ./Dockerfile
    - name: buildContext
      type: string
      description: Path to the directory to use as context
      default: .
    - name: registryTlsVerify
      type: string
      description: Verify the TLS on the registry endpoint (for push/pull to a non-TLS registry)
      default: "true"
    - name: containerBuiltFormat
      type: string
      description: The format of the built container, oci or docker
      default: docker
    - name: buildahBuildExtraArgs
      type: string
      description: Extra parameters passed for the build command when building images.
      default: ""
    - name: buildahPushExtraArgs
      type: string
      description: Extra parameters passed for the push command when pushing images.
      default: ""
  
  results:
    - name: dockerImageFullName
      description: Full name of docker image builded
    - name: dockerImageFullTag
      description: Full name of docker image builded
    - name: IMAGE_DIGEST
      description: Digest of the image just built.

  steps:
    - name: show-informations
      image: $(params.mavenContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
      script: |
        #!/usr/bin/env sh
        echo "------------------------------"
        echo "Project version: $(params.mavenProjectVersion)"
        echo "------------------------------"
        echo " "
        echo "------------------------------"
        echo "The commit ID: $(params.buildRevision)"
        echo "------------------------------"
        echo " "
        echo "------------------------------"
        echo "The short commit ID: $(params.buildRevisionShort)"
        echo "------------------------------"

    - name: retrieve-target-artifacts-folder
      image: $(params.mavenContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
      script: |
        #!/usr/bin/env sh
        cp -R $(workspaces.shared-workspace.path)/target \
          $(resources.inputs.git-source-fruitz-application.path)
    
    - name: build-image
      image: $(params.buildahContainerImage)
      script: |
        # Build Image
        buildah --storage-driver=$(params.buildahStorageDriver) bud \
          $(params.buildahBuildExtraArgs) \
          --format=$(params.containerBuiltFormat) \
          --build-arg PROJECT_VERSION=$(params.mavenProjectVersion)-$(params.buildRevisionShort) \
          --build-arg BUILD_GIT_COMMIT=$(params.buildRevisionShort)  \
          --build-arg BUILD_BRANCH_NAME=$(params.buildRevisionBranch) \
          --tls-verify=$(params.registryTlsVerify) --no-cache \
          -f $(params.dockerfilePath) \
          -t $(params.dockerRegistryName)/$(params.gitlabUsernameAccountName)/$(params.gitlabGroupSampleAppsName)/$(params.gitlabSubGroupFruitzName)/$(params.gitlabProjectFruitzQuarkusName)/$(params.gitlabProjectFruitzQuarkusName):$(params.mavenProjectVersion)-$(params.buildRevisionShort) \
          $(params.buildContext)

        # Save image name to Tekton result 
        echo $(params.dockerRegistryName)/$(params.gitlabUsernameAccountName)/$(params.gitlabGroupSampleAppsName)/$(params.gitlabSubGroupFruitzName)/$(params.gitlabProjectFruitzQuarkusName)/$(params.gitlabProjectFruitzQuarkusName):$(params.mavenProjectVersion)-$(params.buildRevisionShort) \
          > $(results.dockerImageFullName.path)

        # Save project version to Tekton result 
        echo $(params.mavenProjectVersion)-$(params.buildRevisionShort) \
          > $(results.dockerImageFullTag.path)
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"

    - name: push-image
      image: $(params.buildahContainerImage)
      script: |
        buildah --storage-driver=$(params.buildahStorageDriver) push \
          $(params.buildahPushExtraArgs) \
          --tls-verify=$(params.registryTlsVerify) \
          --digestfile $(resources.inputs.git-source-fruitz-application.path)/image-digest \
          $(params.dockerRegistryName)/$(params.gitlabUsernameAccountName)/$(params.gitlabGroupSampleAppsName)/$(params.gitlabSubGroupFruitzName)/$(params.gitlabProjectFruitzQuarkusName)/$(params.gitlabProjectFruitzQuarkusName):$(params.mavenProjectVersion)-$(params.buildRevisionShort) \
          docker://$(params.dockerRegistryName)/$(params.gitlabUsernameAccountName)/$(params.gitlabGroupSampleAppsName)/$(params.gitlabSubGroupFruitzName)/$(params.gitlabProjectFruitzQuarkusName)/$(params.gitlabProjectFruitzQuarkusName):$(params.mavenProjectVersion)-$(params.buildRevisionShort)
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers
      workingDir: "$(resources.inputs.git-source-fruitz-application.path)"
  volumes:
    - emptyDir: {}
      name: varlibcontainers

Task scan-trivy :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: scan-trivy
  namespace: cicd
spec:
  description: >-
    Security scan with Trivy tool

  workspaces:
    - name: shared-workspace

  params:
    - name: dockerRegistryName
      type: string
      description: Name of Docker registry
      default: "registry.gitlab.com"
    - name: dockerImageFullName
      type: string
      description: Name of docker image to scan
    - name: trivyContainerImage
      type: string
      description: The name of Snyk container image
      default: aquasec/trivy:0.31.3

  steps:
    - name: trivy-scan
      image: $(params.trivyContainerImage)
      workingDir: "$(workspaces.shared-workspace.path)"
      env:
        - name: "TRIVY_AUTH_URL"
          value: "$(params.dockerRegistryName)"
        - name: "TRIVY_USERNAME"
          valueFrom:
            secretKeyRef:
              name: trivy-registry-gitlab-com-credentials
              key: gitlab-username-account
        - name: "TRIVY_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: trivy-registry-gitlab-com-credentials
              key: gitlab-registry-token

      script: |
        #!/usr/bin/env sh

        ## Create trivy folder
        mkdir -p $(workspaces.shared-workspace.path)/trivy/scan_result 

        ## Show Trivy version
        echo "Trivy version:"
        trivy --version
        echo ""

        ## Show image to scan
        echo "Image to scan:"
        echo $(params.dockerImageFullName)
        
        ## Scan 
        trivy image --exit-code 0 \
          --cache-dir $(workspaces.shared-workspace.path)/trivy/.trivycache/ \
          --format table \
          $(params.dockerImageFullName)

Task update-helm-deployment-repository :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: update-helm-deployment-repository
  namespace: cicd
spec:
  description: >-
    The Helm deployment repository update 

  resources:
    inputs:
      - name: git-source-fruitz-deployment
        type: git

  workspaces:
    - name: shared-workspace

  params:
    - name: buildRevision
      type: string
      description: The Git commit revision    
    - name: buildRevisionShort
      type: string
      description: The short Git commit revision
    - name: dockerImageFullTag
      type: string
    - name: toolboxContainerImage
      type: string
      description: The name of Toolbox container image
      default: samiamoura/ci-toolbox:1.1.0-fd40008e
    - name: helmDeploymentRepositoryBranch
      type: string
      description: The name of the Helm Fruitz deployment repository
      default: redhat-opentour-geneva

  steps:
    - name: update-helm-deployment-repository
      image: $(params.toolboxContainerImage)
      workingDir: "$(resources.inputs.git-source-fruitz-deployment.path)"
      script: |
        #!/usr/bin/env sh
        
        ## SSH configuration
        ls -la ~/.ssh/
        eval $(ssh-agent)
        ssh-add ~/.ssh/id_*
        
        ## Git commands
        git branch
        git checkout $(params.helmDeploymentRepositoryBranch)

        echo " "
        echo "------------------------------"
        echo "Container Image tag: $(params.dockerImageFullTag)"
        echo "------------------------------"
        echo " "
  
        ## Update the values.yaml file
        yq w -i helm/values.yaml backend.image.tag --style=double $(params.dockerImageFullTag)
        
        ## Git commands
        git --no-pager diff
        git config user.email "sami.amoura@sokube.ch"
        git config user.name "Sami Amoura"

        git add helm/values.yaml
        git commit -m "Automatic update $(params.dockerImageFullTag)"
        git push origin $(params.helmDeploymentRepositoryBranch)

You must also create the associated secrets:

The Secret gitlab-com-ssh-key that will allow Tekton to operate your private GitLab repository :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
kind: Secret
metadata:
  namespace: cicd
  name: gitlab-com-ssh-key
  annotations:
    tekton.dev/git-0: gitlab.com
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

ℹ️ Remember to put your SSH private key.

Le Secret gitlab-com-docker-registry-token qui permettra à Tekton d’opérer votre registre de conteneurs GitLab :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-com-docker-registry-token
  namespace: cicd
  annotations:
    tekton.dev/docker-0: https://registry.gitlab.com
type: kubernetes.io/basic-auth
stringData:
  username: samiamoura
  password: MY_GITLAB_SECRET_TOKEN

ℹ️ Remember to fill in the stringData.password field when generating a token on GitLab.

The Secret trivy-registry-gitlab-com-credentials which will allow the Trivy tool to directly analyze the images hosted in your GitLab image registry:

1
2
3
4
5
6
7
8
apiVersion: v1
kind: Secret
metadata:
  name: trivy-registry-gitlab-com-credentials
  namespace: cicd
stringData:
  gitlab-registry-token: GITLAB_SECRET_TOKEN
  gitlab-username-account: sokube-io

ℹ️ Remember to fill in the stringData.gitlab-registry-token field when generating a token on GitLab.

When creating a Project (namespace) on OpenShift, a pipeline ServiceAccount is created by default with all the permissions needed to execute the different Tasks. To associate the secrets created previously with the Tekton ServiceAccount and allow the use of secrets during the pipeline, it is necessary to patch the ServiceAccount using the following commands:

1
2
3
4
5
6
7
oc -n cicd patch sa pipeline \
  --type='json' \
  -p='[{"op": "add", "path": "/secrets/0/name", "value":"gitlab-com-docker-registry-token"}]'
  
oc -n cicd patch sa pipeline \
  --type='json' \
  -p='[{"op": "add", "path": "/secrets/1/name", "value":"gitlab-com-ssh-key"}]'

You can now go to the OpenShift console and click on Pipeline ► Tasks to see the Tasks created:

Tekton all tasks

Pipeline

After having created the Tasks resources, we are now going to create the pipeline resource which will allow us to orchestrate the execution of the Tasks with, among other things, the execution order of the tasks or the conditionality of the Tasks execution.

Pipeline fruitz-quarkus :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: fruitz-quarkus
  namespace: cicd
spec:

  workspaces:
    - name: shared-workspace

  resources:
    - name: git-source-fruitz-application
      type: git
    - name: git-source-fruitz-deployment
      type: git
  
  params:
    - name: buildRevision
    - name: buildRevisionShort
    - name: buildRevisionBranch

  tasks:
    - name: maven-package
      taskRef:
        name: maven-package
      resources:
        inputs:
          - name: git-source-fruitz-application
            resource: git-source-fruitz-application
      workspaces:
        - name: shared-workspace
          workspace: shared-workspace

    - name: container-image-build-push
      taskRef:
        name: container-image-build-push
      params:
        - name: buildRevision
          value: $(params.buildRevision)
        - name: buildRevisionShort
          value: $(params.buildRevisionShort)
        - name: buildRevisionBranch
          value: $(params.buildRevisionBranch)
        - name: mavenProjectVersion
          value: $(tasks.maven-package.results.mavenProjectVersion)
      resources:
        inputs:
          - name: git-source-fruitz-application
            resource: git-source-fruitz-application
      workspaces:
        - name: shared-workspace
          workspace: shared-workspace
      runAfter:
        - maven-package

    - name: scan-trivy
      taskRef:
        name: scan-trivy
      params:
        - name: dockerImageFullName
          value: $(tasks.container-image-build-push.results.dockerImageFullName)
      workspaces:
        - name: shared-workspace
          workspace: shared-workspace
      runAfter:
        - container-image-build-push

    - name: update-helm-deployment-repository
      taskRef:
        name: update-helm-deployment-repository
      params:
        - name: dockerImageFullTag
          value: $(tasks.container-image-build-push.results.dockerImageFullTag)
        - name: buildRevision
          value: $(params.buildRevision)
        - name: buildRevisionShort
          value: $(params.buildRevisionShort)
      resources:
        inputs:
          - name: git-source-fruitz-deployment
            resource: git-source-fruitz-deployment    
      workspaces:
        - name: shared-workspace
          workspace: shared-workspace
      runAfter:
        - scan-trivy

Now you can go to the OpenShift console and click on Pipelines ► Pipelines ► Pipelines tab to see the created pipeline:

Tekton pipeline view

You can also click directly on the fruitz-quarkus Pipeline to have more information about it like the number of linked Tasks, Workspaces, associated Labels…

Tekton pipeline detailled view

✅ The Tekton configuration is now complete.

Pipeline execution

We will now fix the application bug limiting the addition of fruits with a maximum number of characters of 6. We will extend this limit to 9 characters in order to add the fruit Pineapple.

On the fruitz-quarkus application repository (backend), edit the file src/main/java/org/sokube/hibernate/orm/Fruit.java to change the maximum number of characters to 9:

GitLab fruitz-quarkus update

ℹ️ We can see that the commit ID is 157dbedb

After committing and pushing the changes to the fruitz-quarkus application repository, we can see that the Tekton Pipeline has been successfully triggered and executed:

Tekton pipeline triggered

Tekton pipeline success

It is possible to have more details by clicking on the Pipeline:

Tekton pipeline success

We can also have a detailed view of the different Tasks with the execution of the Steps:

Tekton pipeline success

We can see that the image has been pushed to the GitLab container registry with the correct tag (commit ID 157dbedb):

GitLab container registry

On ArgoCD, we see that fruitz-backend has been redeployed with the new image and the correct tag (commit ID 157dbedb):

ArgoCD new image

ArgoCD new deployed image

This was made possible because the Tekton Pipeline and in particular the update-helm-deployment-repository Task updated the GitOps deployment repository synchronized with ArgoCD.

GitLab GitOps source code updated

Finally we can test the new functionality on the application. We can see that it is now possible to add the fruit Pineapple:

GitLab GitOps source code updated

In this blog post we got to see how to set up a GitOps Cloud Native pipeline on the OpenShift platform. This article introduces the tools, how to nest them together as well as how to quickly get the hang of them. Of course, this is a demonstration for a quick experimentation, many additional things have to be taken into account in an enterprise context such as secrets management, the RBAC part around OpenShift GitOps or the steps composing the Tekton Pipeline (adding tests, defining vulnerability scans for application dependencies, pushing the image in a dedicated registry only if the vulnerability scan is negative…).

In a future blogpost we will show how to set up the ArgoCD Image Updater plugin. This tool which integrates perfectly with ArgoCD, allows to have a GitOps Pipeline even more Cloud-Native. Indeed, it allows to get rid of the Tekton task update-helm-deployment-repository which commits in the deployment repository. This part will be entirely managed by the ArgoCD Image Updater tool.

To go further, discover our ArgoCD and GitOps training and our DevSecOps consulting.