how-to-declaratively-run-helm-charts-using-helmfile
1. Helm
kubernetes의 package manager
기존 kubectl 명령어로는 여러개의 yaml 파일 생성 관리해야 하는데 helm이 Charts로 묶어서 관리하게 함
1.1. Helm Install
helm 설치
] brew install helm
1.2. Chart Repository 가져오기
] helm repo add my-repo https://charts.bitnami.com/bitnami
] kubectl create ns metircs # namespace 생성
1.3. Repository에서 Chart 가져오기
kube-state-metrics: Kubernetes API server and objects의 상태값 메트릭으로 생성
] helm install kube-state-metrics my-repo/kube-state-metrics -n metrics # helm chart 추가
] helm ls -n metrics
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kube-state-metrics metrics 1 2023-02-09 11:53:54.2417933 +0100 CET deployed kube-state-metrics-3.2.8 2.7.0
] kubectl get po -o wide
] kubectl get all -n metrics
NAME READY STATUS RESTARTS AGE
pod/kube-state-metrics-647765674c-2gl2n 1/1 Running 0 179m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-state-metrics ClusterIP 10.111.131.223 <none> 8080/TCP 179m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-state-metrics 1/1 1 1 179m
NAME DESIRED CURRENT READY AGE
replicaset.apps/kube-state-metrics-647765674c 1 1 1 179m
] kubectl logs pod/kube-state-metrics-647765674c-2gl2n -n metrics
I0209 10:53:55.686419 1 wrapper.go:78] Starting kube-state-metrics
I0209 10:53:55.686869 1 server.go:138] "Used resources" resources="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,volumeattachments"
I0209 10:53:55.686904 1 types.go:184] "Using all namespaces"
I0209 10:53:55.686924 1 server.go:166] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "
W0209 10:53:55.686986 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0209 10:53:55.687305 1 server.go:311] "Tested communication with server"
I0209 10:53:55.696886 1 server.go:316] "Run with Kubernetes cluster version" major="1" minor="24" gitVersion="v1.24.0" gitTreeState="clean" gitCommit="4ce5a8954017644c5420bae81d72b09b735c21f0" platform="linux/amd64"
I0209 10:53:55.696909 1 server.go:317] "Communication with server successful"
I0209 10:53:55.697032 1 server.go:263] "Started metrics server" metricsServerAddress="[::]:8080"
I0209 10:53:55.697104 1 metrics_handler.go:97] "Autosharding disabled"
I0209 10:53:55.697132 1 server.go:252] "Started kube-state-metrics self metrics server" telemetryAddress="[::]:8081"
I0209 10:53:55.697252 1 server.go:69] levelinfomsgListening onaddress[::]:8080
I0209 10:53:55.697284 1 server.go:69] levelinfomsgTLS is disabled.http2falseaddress[::]:8080
I0209 10:53:55.697410 1 server.go:69] levelinfomsgListening onaddress[::]:8081
I0209 10:53:55.697440 1 server.go:69] levelinfomsgTLS is disabled.http2falseaddress[::]:8081
I0209 10:53:55.699392 1 builder.go:257] "Active resources" activeStoreNames="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,volumeattachments"
] kubectl port-forward svc/kube-state-metrics 8080:8080 -n metrics # WebUI로 해당 k8s metrics 확인 가능
2. Helm Custumizing
2.1. Chart 열어보기
기본적인 chart 정보
] helm show chart my-repo/kube-state-metrics
annotations:
category: Analytics
licenses: Apache-2.0
apiVersion: v2
appVersion: 2.7.0
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 2.x.x
description: kube-state-metrics is a simple service that listens to the Kubernetes
API server and generates metrics about the state of the objects.
home: https://github.com/bitnami/charts/tree/main/bitnami/kube-state-metrics
icon: https://bitnami.com/assets/stacks/kube-state-metrics/img/kube-state-metrics-stack-220x234.png
keywords:
- prometheus
- kube-state-metrics
- monitoring
maintainers:
- name: Bitnami
url: https://github.com/bitnami/charts
name: kube-state-metrics
sources:
- https://github.com/bitnami/containers/tree/main/bitnami/kube-state-metrics
- https://github.com/kubernetes/kube-state-metrics
version: 3.2.8
필요한 value정보들
변수삽입 문법을 사용하여 values.yaml 파일안의 값들을 templates 폴더안의 각종 리소스의 변수로 지정
] helm show values my-repo/kube-state-metric > values.yaml
image:
registry: docker.io
repository: bitnami/kube-state-metrics
tag: 2.7.0-debian-11-r21
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
kubeResources:
configmaps: true
cronjobs: true
daemonsets: true
deployments: true
endpoints: true
ingresses: true
jobs: true
namespaces: true
nodes: true
persistentvolumeclaims: true
persistentvolumes: true
pods: true
replicasets: true
replicationcontrollers: true
secrets: true
services: true
statefulsets: true
storageclasses: true
service:
type: ClusterIP
ports:
http: 8080
nodePorts:
http: ""
clusterIP: ""
...
resources:
limits: {}
requests: {}
replicaCount: 1
podAnnotations: {}
nodeSelector: {}
...
livenessProbe:
enabled: true
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
...
install 시 set으로 지정가능
f, set 옵션을 사용하여 values 의 변수 우선순위를 지정 가능
마지막 명시한 걸로 덮어씀
] helm install --set foo=bar --set foo=newbar myredis ./redis
] helm install -f myvalues.yaml -f override.yaml myredis ./redis
–set(key=value) -> -f (yaml file) -> values.yaml 우측으로 갈수록 우선순위가 낮음
] helm install kube-state-metrics my-repo/kube-state-metrics -n metrics \
--set configMapData.log.level=debug \
--set kubeResources.cronjobs=false \
-f values_dev.yaml
Clean up
] helm delete kube-state-metrics
] kubectl delete ns metrics
2.2. Chart 생성
기본적인 chart 모델로 boilerplate 생성
] helm create first-chart
templetes 폴더 내 configmap.yaml 추가 후 install
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: first-chart-configmap
data:
port: "8080"
] helm install first-chart .
] kubectl get cm
NAME DATA AGE
first-chart-configmap 1 8s
kube-root-ca.crt 1 7h9m
] kubectl describe configmap first-chart-configmap
Name: first-chart-configmap
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: first-chart
meta.helm.sh/release-namespace: default
Data
====
port:
----
8080
BinaryData
2.3. Chart 수정
templetes 폴더 내 configmap.yaml 수정 후 upgrade
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: first-chart-configmap
data:
port: "8080"
allowTesting: "true"
] helm template first-chart . # 변경 내용 확인
] helm upgrade first-chart .
Release "first-chart" has been upgraded. Happy Helming!
NAME: first-chart
LAST DEPLOYED: Sun Feb 12 12:06:14 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
] kubectl describe configmap first-chart-configmap # allowTesting 값 확인
Name: first-chart-configmap
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: first-chart
meta.helm.sh/release-namespace: default
Data
====
allowTesting:
----
true
port:
----
8080
BinaryData
====
Events: <none>
templetes 폴더 내 secret.yaml 추가 후 upgrade
먼저 admin 정보의 base64 encoding값 확인
] echo -n 'admin' | base64
YWRtaW4=
] echo -n '4w572$9sns1&!' | base64
NHc1NzIkOXNuczEmIQ==
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: first-secret
type: Opaque
data:
username: YWRtaW4= # 'admin' encoding한 값
password: NHc1NzIkOXNuczEmIQ== # '4w572$9sns1&!' encoding한 값
] helm template first-chart . # secret 추가 내용 확인
] helm upgrade first-chart .
Release "first-chart" has been upgraded. Happy Helming!
NAME: first-chart
LAST DEPLOYED: Sun Feb 12 12:17:33 2023
NAMESPACE: default
STATUS: deployed
REVISION: 3
TEST SUITE: None
] kubectl get secrets
NAME TYPE DATA AGE
first-secret Opaque 2 2m1s
sh.helm.release.v1.first-chart.v1 helm.sh/release.v1 1 2d18h
sh.helm.release.v1.first-chart.v2 helm.sh/release.v1 1 13m
sh.helm.release.v1.first-chart.v3 helm.sh/release.v1 1 2m1s
] kubectl describe secret first-secret # secrets 값 확인
Name: first-secret
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: first-chart
meta.helm.sh/release-namespace: default
Type: Opaque
Data
====
password: 13 bytes
username: 5 bytes
2.4. Rollback a Helm release
] helm history first-chart
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Feb 9 17:46:16 2023 superseded first-chart-0.1.0 1.16.0 Install complete
2 Sun Feb 12 12:06:14 2023 superseded first-chart-0.1.0 1.16.0 Upgrade complete
3 Sun Feb 12 12:17:33 2023 deployed first-chart-0.1.0 1.16.0 Upgrade complete
] helm rollback first-chart 2
Rollback was a success! Happy Helming!
] helm history first-chart
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Feb 9 17:46:16 2023 superseded first-chart-0.1.0 1.16.0 Install complete
2 Sun Feb 12 12:06:14 2023 superseded first-chart-0.1.0 1.16.0 Upgrade complete
3 Sun Feb 12 12:17:33 2023 superseded first-chart-0.1.0 1.16.0 Upgrade complete
4 Sun Feb 12 12:23:39 2023 superseded first-chart-0.1.0 1.16.0 Rollback to 2
5 Sun Feb 12 12:24:26 2023 deployed first-chart-0.1.0 1.16.0 Rollback to 2
3. dynamically ConfigMap 관리하기
3.1. mapping Chart value
templetes 폴더 내 configmap.yaml 수정 후 upgrade
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: first-chart-configmap-
data:
port: "8080"
] kubectl get cm
NAME DATA AGE
first-chart-configmap-0.1.0 2 2d19h
] helm upgrade first-chart . # 업그레이드
] kubectl get cm # Chart.yaml에 version값 변경 후 dynamically 변경 된 내용 체크
NAME DATA AGE
first-chart-configmap-0.1.1 1 2s
3.3. templetes functions
range
반복문
# values.yaml
list:
- a
- b
- c
# configmap.yaml
range:
-
# 결과
range:
- a
- b
- c
range 에 개별편의를 위해 자체적인 지역변수 $index, $key, $value를 제공
# values.yaml
list:
- a
- b
- c
map:
env: dev
log: info
# configmap.yaml
range:
index:
:
map:
:
# 결과
range:
index:
0: a
1: b
2: c
map:
env: "dev"
log: "info"
# configmap.yaml
sizes: |-
-
# 결과
sizes: |-
- small
- medium
- large
$ 키워드
$와 :=으로 변수 할당
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: -configmap
data:
myvalue: "Hello World"release:
with
부모객체를 가져와 지역변수처럼 사용하는 개념으로 변수명이 복잡하거나 객체의 depth 가 깊을경우 가독성처리를 위해 사용
# configmap.yaml
env:
log:
$ 키워드는 지역변수 scope 에 영향을 받지 않기 때문에 내부에서 밖에 정의해둔 변수를 참조 가능
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: -configmap
data:
myvalue: "Hello World"
drink:
food:
release:
# 결과
apiVersion: v1
kind: ConfigMap
metadata:
name: viable-badger-configmap
data:
myvalue: "Hello World"
drink: "coffee"
food: "PIZZA"
release: viable-badger
pipeline, quote, upper 함수
] values.yaml
func:
enabled: true
객체 사용 및 # 결과
# "true"
# "true"
# "TRUE"
# _helpers.tpl
labels:
generator: helm
date:
required
helm 명령을 통해 랜더링할 때 객체에 대한 정보가 없을경우 랜더링을 멈추고 에러를 반환시키고 싶다면 required 함수를 사용
value:
toyaml
# values.yaml
data:
- a
- b
- c
# configmap.yaml
my:
data:
# 결과 - 에러
my:
data:
- a
- b
- c
# configmap.yaml
my:
data:
앞의 - 가 공백 뿐 아니라 개행까지 없애주기 때문에 구문이 어디에 위치해 있던지 상관X
tpl
템플릿 구문을 해석하는 함수, 템플릿 구문 문자열을 스크립트로 실행한 결과를 반환
# values.yaml
template: ""
name: "Tom"
# configmap.yaml
t1: # "" # 그대로 출력
t2: # Tom # template함수를 사용하면 템플릿 구문 문자열을 실행한 결과를 반환
values.yaml 에는 템플릿 구문을 사용하지 못함
3.3. named_templates
사용자 정의 객체 _helpers.tpl
_helpers.tpl 파일에서 define 으로 시작해서 end 로 정의
define
# _helpers.tpl
labels:
generator: helm
date:
chart:
version:
첫 부분에 - 넣는 이유는 공백, 개행 삭제
include
정의한 템플릿을 가져오기위해 사용하는 함수
첫번째 인자에는 가져오고싶은 템플릿 이름 두번째 인자에는 해당 템플릿에 적용할 객체를 넣음(만약 적용할 객체가 없다면 . 사용)
# _helper.tplv1:
v2:
객체 사용
include1:
# 결과
include1:
v1: "value1"
v2: "value1"
template
Template:
Include:
# 결과
Template: " hello"
Include: "hello"
4. Helmfile로 관리
이제껏 imperative하게 관리하던 것을 declarative하게 관리 가능