GitOps Lab

Duration: 25 minutes

Objectives

By the end of this lab, you will:

• Install and configure Flux CD
• Bootstrap a Git repository for GitOps
• Deploy applications using Kustomization
• Deploy Helm charts via HelmRelease
• Implement automated image updates
• Configure notifications

Prerequisites

• Kubernetes cluster (Docker Desktop, kind, minikube, or cloud provider)
• kubectl configured
• Git repository (GitHub, GitLab, or Bitbucket)
• Personal access token for your Git provider

Task 1: Install Flux CLI

Workshop Container (already installed)

flux version

Manual Installation

Install the Flux command-line tool:

# macOS/Linux
curl -s https://fluxcd.io/install.sh | sudo bash

# Or using Homebrew
brew install fluxcd/tap/flux

# Windows (using Chocolatey)
choco install flux

# Verify installation
flux --version

Check if your cluster is ready for Flux:

flux check --pre

Solution

flux check --pre

# Expected output:
# ✔ Kubernetes 1.28.x >=1.26.0-0
# ✔ prerequisites checks passed

Task 2: Bootstrap Flux

Bootstrap Flux to your cluster and connect it to your Git repository:

# Set variables
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=fleet-infra

# Bootstrap Flux
flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=$GITHUB_REPO \
  --branch=main \
  --path=./clusters/dev-cluster \
  --personal

This command:

• Creates the repository if it doesn't exist
• Adds Flux components to the cluster
• Configures Flux to sync from the repository
• Commits manifests to the repo

Verify Flux installation:

# Check Flux pods
kubectl get pods -n flux-system

# Should see:
# - source-controller
# - kustomize-controller
# - helm-controller
# - notification-controller

# Check GitRepository source
flux get sources git

# Check Kustomizations
flux get kustomizations

Solution

flux check

# Expected output:
# ✔ Kubernetes 1.28.x >=1.26.0-0
# ✔ Flux 2.2.x installed
# ✔ source-controller: deployment ready
# ✔ kustomize-controller: deployment ready
# ✔ helm-controller: deployment ready
# ✔ notification-controller: deployment ready

Issue: Bootstrap fails with authentication error

# Solution: Ensure GITHUB_TOKEN has correct permissions
# Required scopes: repo, admin:repo_hook

# Regenerate token and retry
export GITHUB_TOKEN=<new-token>
flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=$GITHUB_REPO \
  --branch=main \
  --path=./clusters/dev-cluster \
  --personal

Task 3: Deploy Application with Kustomization

Create a simple application using Kustomize:

1. Clone your repository:

git clone https://github.com/$GITHUB_USER/$GITHUB_REPO
cd $GITHUB_REPO

2. Create application structure:

mkdir -p apps/base

3. Create apps/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
      - name: podinfo
        image: ghcr.io/stefanprodan/podinfo:6.5.3
        ports:
        - containerPort: 9898
          name: http
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

4. Create apps/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: podinfo
  namespace: default
spec:
  selector:
    app: podinfo
  ports:
  - port: 80
    targetPort: 9898

5. Create apps/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

6. Create Flux Kustomization in clusters/dev-cluster/apps.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./apps/base
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: podinfo
    namespace: default
  timeout: 2m

7. Commit and push:

git add -A
git commit -m "Add podinfo application"
git push

8. Watch Flux reconcile:

# Watch reconciliation
flux reconcile kustomization flux-system --with-source

# Check application
kubectl get pods -l app=podinfo

# Check service
kubectl get svc podinfo

# Test the application
kubectl port-forward svc/podinfo 9898:80
# Visit http://localhost:9898

Solution

kubectl get deployment podinfo
# NAME      READY   UP-TO-DATE   AVAILABLE   AGE
# podinfo   2/2     2            2           2m

kubectl get pods -l app=podinfo
# NAME                       READY   STATUS    RESTARTS   AGE
# podinfo-xxxxxxxxxx-xxxxx   1/1     Running   0          2m
# podinfo-xxxxxxxxxx-xxxxx   1/1     Running   0          2m

curl localhost:9898
# {
#   "hostname": "podinfo-xxxxxxxxxx-xxxxx",
#   "version": "6.5.3",
#   "runtime": "go1.21.3",
#   "color": "#34577c",
#   "message": "greetings from podinfo"
# }

Issue: Kustomization not reconciling

flux get kustomizations apps
kubectl describe kustomization apps -n flux-system
flux reconcile kustomization apps --with-source
flux logs --kind=Kustomization --name=apps

Task 4: Deploy Helm Chart with HelmRelease

1. Create HelmRepository source clusters/dev-cluster/sources.yaml:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 30m
  url: https://charts.bitnami.com/bitnami

2. Create HelmRelease apps/base/redis-release.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: redis
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: redis
      version: '18.x.x'
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    auth:
      enabled: false
    master:
      persistence:
        enabled: false
    replica:
      replicaCount: 1
      persistence:
        enabled: false

3. Update apps/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - redis-release.yaml

4. Commit and observe:

git add -A
git commit -m "Add Redis via Helm"
git push

# Watch Helm releases
flux get helmreleases

# Check Redis pods
kubectl get pods -l app.kubernetes.io/name=redis

Solution

flux get helmreleases
# NAME     REVISION     SUSPENDED   READY   MESSAGE
# redis    18.x.x       False       True    Release reconciliation succeeded

kubectl get pods -l app.kubernetes.io/name=redis
# NAME              READY   STATUS    RESTARTS   AGE
# redis-master-0    1/1     Running   0          3m

# Test Redis connection
kubectl run redis-client --rm -it --image redis -- redis-cli -h redis-master ping
# PONG

Issue: HelmRelease stuck in "InstallFailed"

flux get helmreleases
kubectl describe helmrelease redis -n default
flux reconcile helmrelease redis

Task 5: Image Automation

Configure Flux to automatically update container images:

1. Create ImageRepository clusters/dev-cluster/image-automation.yaml:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  image: ghcr.io/stefanprodan/podinfo
  interval: 1m0s
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    semver:
      range: 6.x.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: |
        Automated image update

        Automation name: {{ .AutomationObject }}

        Files:
        {{ range $filename, $_ := .Updated.Files -}}
        - {{ $filename }}
        {{ end -}}

        Objects:
        {{ range $resource, $_ := .Updated.Objects -}}
        - {{ $resource.Kind }} {{ $resource.Name }}
        {{ end -}}
    push:
      branch: main
  update:
    path: ./apps/base
    strategy: Setters

2. Add image policy marker in apps/base/deployment.yaml:

spec:
  template:
    spec:
      containers:
      - name: podinfo
        image: ghcr.io/stefanprodan/podinfo:6.5.3 # {"$imagepolicy": "flux-system:podinfo"}

3. Commit and push:

git add -A
git commit -m "Add image automation"
git push

# Watch image updates
flux get image repository podinfo
flux get image policy podinfo

# Monitor for automated commits
watch flux get image update

Flux will now automatically update the image tag when new versions are available!

Solution

flux get image repository podinfo
# NAME      LAST SCAN                   SUSPENDED   READY   MESSAGE
# podinfo   2024-01-01T10:00:00Z        False       True    successful scan: found X tags

flux get image policy podinfo
# NAME      LATEST IMAGE                               READY   MESSAGE
# podinfo   ghcr.io/stefanprodan/podinfo:6.5.4        True    Latest image tag for '6.x.x'

# Check for automated commit in repository
git log --oneline -n 5
# Shows automated commits from fluxcdbot when images update

Issue: Image automation not creating commits

flux get image repository podinfo
flux get image policy podinfo
flux get image update
flux logs --kind=ImageUpdateAutomation

Task 6: Suspend and Resume

Learn to control reconciliation:

# Suspend the apps Kustomization
flux suspend kustomization apps

# Make manual changes to cluster
kubectl scale deployment podinfo --replicas=5

# Resume reconciliation (will revert to Git state)
flux resume kustomization apps

# Verify reverted to 2 replicas
kubectl get deployment podinfo

Solution

# After suspend
flux get kustomizations
# NAME            REVISION        SUSPENDED   READY   MESSAGE
# apps            main@sha1:xxx   True        True    kustomization is suspended...

# After manual scale
kubectl get deployment podinfo
# NAME      READY   UP-TO-DATE   AVAILABLE   AGE
# podinfo   5/5     5            5           10m

# After resume
flux resume kustomization apps
flux get kustomizations
# NAME            REVISION        SUSPENDED   READY   MESSAGE
# apps            main@sha1:xxx   False       True    Applied revision: main@sha1:xxx

kubectl get deployment podinfo
# NAME      READY   UP-TO-DATE   AVAILABLE   AGE
# podinfo   2/2     2            2           11m
# (back to 2 replicas as defined in Git)

Task 7: Notifications

Set up Slack notifications (or use a generic webhook):

1. Create notification provider clusters/dev-cluster/notifications.yaml:

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: kubernetes-alerts
  secretRef:
    name: slack-webhook
---
apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
  name: on-deploy
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: apps
  - kind: HelmRelease
    name: '*'

2. Create secret with webhook URL:

kubectl create secret generic slack-webhook \
  -n flux-system \
  --from-literal=address=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

3. Commit and push:

git add clusters/dev-cluster/notifications.yaml
git commit -m "Add Slack notifications"
git push

You'll now receive notifications in Slack for deployments!

Solution

After setup, any deployment will trigger a Slack notification like:

{
  "text": "Kustomization apps reconciliation succeeded",
  "attachments": [
    {
      "color": "good",
      "title": "apps/default",
      "fields": [
        {"title": "Revision", "value": "main@sha1:abc123", "short": true},
        {"title": "Status", "value": "Applied", "short": true}
      ]
    }
  ]
}

Bonus Challenge 1: Multi-Environment Setup

Create separate overlays for staging and production:

mkdir -p apps/staging apps/production

# apps/staging/kustomization.yaml
cat > apps/staging/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: staging
resources:
  - ../base
patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 1
  target:
    kind: Deployment
    name: podinfo
EOF

# apps/production/kustomization.yaml
cat > apps/production/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../base
patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 3
  target:
    kind: Deployment
    name: podinfo
EOF

Create separate Flux Kustomizations for each environment.

Solution

# clusters/dev-cluster/infrastructure.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
# clusters/dev-cluster/apps.yaml (updated)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
  - name: infrastructure  # Wait for infrastructure first
  interval: 5m
  path: ./apps/base
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Final repository structure:

fleet-infra/
├── .git/
├── README.md
├── clusters/
│   └── dev-cluster/
│       ├── flux-system/
│       │   ├── gotk-components.yaml
│       │   ├── gotk-sync.yaml
│       │   └── kustomization.yaml
│       ├── apps.yaml
│       ├── sources.yaml
│       ├── image-automation.yaml
│       └── notifications.yaml
└── apps/
    ├── base/
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   ├── redis-release.yaml
    │   └── kustomization.yaml
    ├── staging/
    │   └── kustomization.yaml
    └── production/
        └── kustomization.yaml

Bonus Challenge 2: Sealed Secrets Integration

Encrypt secrets before committing to Git:

# Install sealed-secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Create and seal a secret
kubectl create secret generic mysecret \
  --from-literal=password=supersecret \
  --dry-run=client \
  -o yaml | \
  kubeseal -o yaml > apps/base/sealed-secret.yaml

# Add to kustomization and commit
git add apps/base/sealed-secret.yaml
git commit -m "Add sealed secret"
git push

Bonus Challenge 3: Flux Health Checks

Add detailed health checks to your Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  # ... existing spec ...
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: podinfo
    namespace: default
  - apiVersion: v1
    kind: Service
    name: podinfo
    namespace: default
  - apiVersion: helm.toolkit.fluxcd.io/v2beta1
    kind: HelmRelease
    name: redis
    namespace: default
  timeout: 5m
  wait: true

Bonus Challenge 4: Dependency Ordering

Ensure infrastructure is deployed before applications:

# clusters/dev-cluster/infrastructure.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
---
# clusters/dev-cluster/apps.yaml (updated)
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
  - name: infrastructure
  interval: 5m
  path: ./apps/base
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Verification

Ensure everything is working:

# Check all Flux components
flux check

# View all sources
flux get sources all

# View all Kustomizations
flux get kustomizations

# View all Helm releases
flux get helmreleases

# View all image policies
flux get image policy

# Check for errors
flux logs --all-namespaces

Cleanup

# Suspend all reconciliation
flux suspend kustomization flux-system

# Delete applications
kubectl delete -f apps/base/

# Uninstall Flux
flux uninstall --silent

# Delete repository (optional)
# Delete from GitHub/GitLab manually or via API

Key takeaways

1. GitOps provides declarative, version-controlled infrastructure
2. Flux automates synchronization between Git and cluster
3. Kustomization enables environment-specific configurations
4. HelmRelease manages Helm charts declaratively
5. Image automation keeps containers up-to-date automatically
6. Notifications provide visibility into deployments
7. Sealed Secrets enable safe secret storage in Git

Complete Workflow

1. Developer pushes code → New container image built
2. ImageRepository scans → Detects new version matching policy
3. ImagePolicy evaluates → Determines latest version
4. ImageUpdateAutomation commits → Updates manifests in Git
5. GitRepository syncs → Detects changes in repo
6. Kustomization reconciles → Applies updated manifests
7. Deployment rolls out → New pods created
8. Notification sent → Team alerted via Slack

Next section

Once you've reviewed the content and completed the lab, proceed to the next section.