Kubernetes

Setup node

Create a butane config file:

variant: fcos
version: 1.4.0
passwd:
  users:
  - name: core
    ssh_authorized_keys:
    - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO+YIUNEga6Q4KuVKO5wYoXAg4VtZqU/GLtPaA3tVgkJ sfe@SFE-X1N
boot_device:
  luks:
    tpm2: true
storage:
  disks:
    - device: /dev/nvme1n1
      wipe_table: true
      partitions:
        - number: 1
          size_mib: 204800
        - number: 2
  files:
  - path: /etc/selinux/config
    overwrite: true
    contents:
      inline: |
        SELINUX=permissive
        SELINUXTYPE=targeted
  - path: /etc/sysctl.d/90-kubelet.conf
    contents:
      inline: |
        vm.panic_on_oom=0
        vm.overcommit_memory=1
        kernel.panic=10
        kernel.panic_on_oops=1
        kernel.keys.root_maxbytes=25000000
  - path: /etc/zincati/config.d/55-updates-strategy.toml
    contents:
      inline: |
        [updates]
        strategy = "fleet_lock"
        [updates.fleet_lock]
        base_url = "http://10.43.0.11/"

Setup K3s

Create config gile:

tls-san:
- api.farm.tegridy.io
- 10.10.10.100
- 10.10.10.101
- 10.10.10.102
- 10.10.10.103
- 127.0.0.1
flannel-backend: none
disable:
- coredns
- servicelb
- traefik
- local-storage
- metrics-server
disable-cloud-controller: true
disable-kube-proxy: true
etcd-snapshot-schedule-cron: "0 5 * * *"
etcd-snapshot-retention: 30
etcd-s3: true
etcd-s3-region: eu-central-1
etcd-s3-bucket: tegridy-barn-kitten-etcd-snapshot
etcd-s3-access-key: <REDACTED>
etcd-s3-secret-key: <REDACTED>
kube-apiserver-arg:
- --enable-admission-plugins=NodeRestriction,PodSecurity,ServiceAccount
- --admission-control-config-file=/etc/rancher/k3s/podsecurity.yaml
- --oidc-issuer-url=https://issuer.zitadel.ch
- --oidc-client-id=181462379698187675@tegridy-farm
- --oidc-username-claim=email

Install on first master

curl -sfL https://get.k3s.io | sh -s - server --cluster-init

Install on other masters

curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server

OAuth

barn-kitten-api

Settings: * PKCE * User roles inside ID Token * User Info inside ID Token

issuer url: https://issuer.zitadel.ch client id: 181462379698187675@tegridy-farm

k oidc-login get-token --oidc-issuer-url=https://issuer.zitadel.ch --oidc-client-id=181462379698187675@tegridy-farm --oidc-extra-scope=email,profile

{
  iss https://issuer.zitadel.ch
  aud
     181462379698187675@tegridy-farm
     178194089269848884
  azp 181462379698187675@tegridy-farm
  nonce 4mHtJ-A1areFW88VFJPNaZmJOy7_SRCrqBsw_4-5ftU
  at_hash 5w45l3dljf-x9Nfxtgu1Cg
  c_hash H2TXPZ4pBB6d064eDmdX6Q
  amr
     password
     mfa
     otp
  exp 1664702179
  iat 1664658979
  auth_time 1664655900
  email do@tegridy.io
  email_verified true
  family_name Orakel
  gender male
  given_name Debakel
  locale en
  name Debakel Orakel
  preferred_username do@tegridy.zitadel.ch
  sub 163761476539114158
  updated_at 1664142457
  urn:zitadel:iam:org:project:roles
    barn-kitten-api
      163761476539048622 tegridy.zitadel.ch
}

Troubleshoot

#!/bin/bash

sgdisk --zap-all /dev/sda
dd if=/dev/zero of=/dev/sda bs=1M count=100 oflag=direct,dsync
blkdiscard /dev/sda

sgdisk --zap-all /dev/sdb
dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync
blkdiscard /dev/sdb

sgdisk --zap-all /dev/nvme1n1
dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=100 oflag=direct,dsync
blkdiscard /dev/nvme1n1

StorageClass Performance

TopoLVM nvme

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 351k/250k. BW: 1378MiB/s / 2197MiB/s
Average Latency (usec) Read/Write: 58.45/17.75
Sequential Read/Write: 3007MiB/s / 2581MiB/s
Mixed Random Read/Write IOPS: 236k/78.6k