毎度、ググっても出てこない小ネタを取り扱っております。
本記事は個人的な見解であり、筆者の所属するいかなる団体にも関係ございません。
0. はじめに
今年(2021年)、私はJetsonでK3Sな一年でした。
締めくくりとして、JetsonでK3Sを動かしてMinIOを動かすというのをやってみたいと思います。
(2021年中に終わらないかも) [終わりませんでしたw]
Standaloneで動かします。
その為、インストールしたホストが失われるとデータが消失します。
ご注意ください。
1. Jetsonのセットアップ
JetsonはSDカードにインストールするだけなのでスキップします。
利用したJetpackは、4.6です
2. K3Sをインストール
K3Sをインストールするには、k3supを使います。
なので、まずはk3supをインストールします。
$ curl -sLS https://get.k3sup.dev | sh $ sudo install k3sup-arm64 /usr/local/bin/k3sup 公式には
sudo install k3sup /usr/local/bin/k3sup
となっていますが、armではk3sup-arm64がダウンロードされるので変更しています。
3. k3sをインストール
3-1. インストール
k3supをつかって、k3sをインストールします。
$ k3sup install --local --user=$USER --local-path ./kubeconfig インストールログは以下の通りです。
k3supでk3sのインストールログ
k3sup install --local Running: k3sup install 2021/12/31 18:33:48 127.0.0.1 Executing: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='server --tls-san 127.0.0.1' INSTALL_K3S_CHANNEL='stable' sh - [INFO] Finding release for channel stable [INFO] Using v1.22.5+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/sha256sum-arm64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/k3s-arm64 [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Skipping /usr/local/bin/kubectl symlink to k3s, already exists [INFO] Skipping /usr/local/bin/crictl symlink to k3s, already exists [INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service. [INFO] systemd: Starting k3s stderr: "Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.\n"stdout: "[INFO] Finding release for channel stable\n[INFO] Using v1.22.5+k3s1 as release\n[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/sha256sum-arm64.txt\n[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/k3s-arm64\n[INFO] Verifying binary download\n[INFO] Installing k3s to /usr/local/bin/k3s\n[INFO] Skipping installation of SELinux RPM\n[INFO] Skipping /usr/local/bin/kubectl symlink to k3s, already exists\n[INFO] Skipping /usr/local/bin/crictl symlink to k3s, already exists\n[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr\n[INFO] Creating killall script /usr/local/bin/k3s-killall.sh\n[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh\n[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env\n[INFO] systemd: Creating service file /etc/systemd/system/k3s.service\n[INFO] systemd: Enabling k3s unit\n[INFO] systemd: Starting k3s\n"Saving file to: /home/hoge/kubeconfig # Test your cluster with: export KUBECONFIG=/home/hoge/kubeconfig kubectl config set-context default kubectl get node -o wide 3-2. テストする
インストールできているはずなので、接続テストしてみます。
export KUBECONFIG=/home/hoge/kubeconfig kubectl config set-context default kubectl get node -o wide 実行結果
$ export KUBECONFIG=/home/hoge/kubeconfig $ kubectl config set-context default Context "default" modified. $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME desktop Ready control-plane,master 14d v1.22.5+k3s1 192.168.99.123 <none> Ubuntu 18.04.6 LTS 4.9.253-tegra containerd://1.5.8-k3s1 問題ないので先に進みます。
4. Kubernetesの各種アプリをインストールします。
kubectlとhelmコマンドをインストールしますが、バージョンアップを楽にするためにsnapでインストールします。
4-1. Kubectlのインストール
sudo snap install kubectl --classic 4-2. helmのインストール
sudo snap install helm --classic 5. MinIOをインストール
5-1. MinIOリポジトリをhelmで追加
helm repo add minio https://charts.min.io/ 追加されたか確認
$ helm repo list NAME URL miniohttps://charts.min.io/ helm search repo minio -l helm search repo 実行結果
$ helm search repo minio -l NAME CHART VERSION APP VERSION DESCRIPTION minio/minio 3.4.4 RELEASE.2021-12-29T06-49-06Z Multi-Cloud Object Storage minio/minio 3.4.3 RELEASE.2021-12-20T22-07-16Z Multi-Cloud Object Storage minio/minio 3.4.2 RELEASE.2021-12-20T22-07-16Z Multi-Cloud Object Storage minio/minio 3.4.1 RELEASE.2021-12-10T23-03-39Z Multi-Cloud Object Storage minio/minio 3.4.0 RELEASE.2021-12-18T04-42-33Z Multi-Cloud Object Storage minio/minio 3.3.4 RELEASE.2021-12-10T23-03-39Z Multi-Cloud Object Storage minio/minio 3.3.3 RELEASE.2021-12-10T23-03-39Z Multi-Cloud Object Storage minio/minio 3.3.2 RELEASE.2021-12-10T23-03-39Z Multi-Cloud Object Storage minio/minio 3.3.1 RELEASE.2021-11-24T23-19-33Z Multi-Cloud Object Storage minio/minio 3.3.0 RELEASE.2021-11-24T23-19-33Z Multi-Cloud Object Storage minio/minio 3.2.0 RELEASE.2021-10-13T00-23-17Z Multi-Cloud Object Storage minio/minio 3.1.9 RELEASE.2021-10-10T16-53-30Z Multi-Cloud Object Storage minio/minio 3.1.8 RELEASE.2021-10-06T23-36-31Z Multi-Cloud Object Storage minio/minio 3.1.7 RELEASE.2021-10-02T16-31-05Z Multi-Cloud Object Storage minio/minio 3.1.6 RELEASE.2021-09-18T18-09-59Z Multi-Cloud Object Storage minio/minio 3.1.5 RELEASE.2021-09-18T18-09-59Z Multi-Cloud Object Storage minio/minio 3.1.4 RELEASE.2021-09-18T18-09-59Z Multi-Cloud Object Storage minio/minio 3.1.3 RELEASE.2021-09-09T21-37-07Z Multi-Cloud Object Storage minio/minio 3.1.2 RELEASE.2021-09-09T21-37-07Z Multi-Cloud Object Storage minio/minio 3.1.1 RELEASE.2021-09-09T21-37-07Z Multi-Cloud Object Storage minio/minio 3.1.0 RELEASE.2021-09-09T21-37-07Z Multi-Cloud Object Storage minio/minio 3.0.2 RELEASE.2021-09-03T03-56-13Z Multi-Cloud Object Storage minio/minio 3.0.1 RELEASE.2021-08-31T05-46-54Z Multi-Cloud Object Storage minio/minio 3.0.0 RELEASE.2021-08-31T05-46-54Z Multi-Cloud Object Storage minio/minio 2.0.1 RELEASE.2021-08-31T05-46-54Z Multi-Cloud Object Storage minio/minio 2.0.0 RELEASE.2021-08-25T00-41-18Z Multi-Cloud Object Storage minio/minio 1.0.5 RELEASE.2021-08-25T00-41-18Z Multi-Cloud Object Storage minio/minio 1.0.4 RELEASE.2021-08-25T00-41-18Z Multi-Cloud Object Storage minio/minio 1.0.3 RELEASE.2021-08-20T18-32-01Z Multi-Cloud Object Storage minio/minio 1.0.2 RELEASE.2021-08-20T18-32-01Z High Performance, Kubernetes Native Object Storage minio/minio 1.0.1 RELEASE.2021-08-20T18-32-01Z High Performance, Kubernetes Native Object Storage minio/minio 1.0.0 RELEASE.2021-08-17T20-53-08Z High Performance, Kubernetes Native Object Storage 現在、Helmでデプロイされているものを確認
$ export KUBECONFIG=/home/hoge/kubeconfig $ helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION traefik kube-system 2 2021-12-31 09:34:39.667360093 +0000 UTC deployed traefik-10.3.001 2.5.0 traefik-crd kube-system 3 2021-12-31 09:34:36.982613767 +0000 UTC deployed traefik-crd-10.3.001 K3Sでは、デフォルトでtraefikがHelmインストールされています。
このtraefikは、K3Sで使われるIngressです
5-3. helmのvaluesを確認
helmで指定できる値を確認
helm show values minio/minio helm のManifest
$ helm show values minio/minio ## Provide a name in place of minio for `app:` labels ## nameOverride: "" ## Provide a name to substitute for the full names of resources ## fullnameOverride: "" ## set kubernetes cluster domain where minio is running ## clusterDomain: cluster.local ## Set default image, imageTag, and imagePullPolicy. mode is used to indicate the ## image: repository: quay.io/minio/minio tag: RELEASE.2021-12-29T06-49-06Z pullPolicy: IfNotPresent imagePullSecrets: [] # - name: "image-pull-secret" ## Set default image, imageTag, and imagePullPolicy for the `mc` (the minio ## client used to create a default bucket). ## mcImage: repository: quay.io/minio/mc tag: RELEASE.2021-12-29T06-52-55Z pullPolicy: IfNotPresent ## minio mode, i.e. standalone or distributed or gateway. mode: distributed ## other supported values are "standalone", "gateway" ## Additional labels to include with deployment or statefulset additionalLabels: [] ## Additional annotations to include with deployment or statefulset additionalAnnotations: [] ## Additional arguments to pass to minio binary extraArgs: [] ## Port number for MinIO S3 API Access minioAPIPort: "9000" ## Port number for MinIO Browser COnsole Access minioConsolePort: "9001" ## Update strategy for Deployments DeploymentUpdate: type: RollingUpdate maxUnavailable: 0 maxSurge: 100% ## Update strategy for StatefulSets StatefulSetUpdate: updateStrategy: RollingUpdate ## Pod priority settings ## ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ ## priorityClassName: "" ## Set default rootUser, rootPassword ## AccessKey and secretKey is generated when not set ## Distributed MinIO ref: https://docs.minio.io/docs/distributed-minio-quickstart-guide ## rootUser: "" rootPassword: "" ## Use existing Secret that store following variables: ## ## | Chart var | .data.<key> in Secret | ## |:----------------------|:-------------------------| ## | rootUser | rootUser | ## | rootPassword | rootPassword | ## ## All mentioned variables will be ignored in values file. ## .data.rootUser and .data.rootPassword are mandatory, ## others depend on enabled status of corresponding sections. existingSecret: "" ## Directory on the MinIO pof certsPath: "/etc/minio/certs/" configPathmc: "/etc/minio/mc/" ## Path where PV would be mounted on the MinIO Pod mountPath: "/export" ## Override the root directory which the minio server should serve from. ## If left empty, it defaults to the value of {{ .Values.mountPath }} ## If defined, it must be a sub-directory of the path specified in {{ .Values.mountPath }} ## bucketRoot: "" # Number of drives attached to a node drivesPerNode: 1 # Number of MinIO containers running replicas: 16 # Number of expanded MinIO clusters pools: 1 # Deploy if 'mode == gateway' - 4 replicas. gateway: type: "nas" # currently only "nas,gcs" are supported. replicas: 4 gcs: serviceAccountFile: "" # credential JSON file of service account key (not required if using existing secret) projectId: "" # Google cloud project id required ## TLS Settings for MinIO tls: enabled: false ## Create a secret with private.key and public.crt files and pass that here. Ref: https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret certSecret: "" publicCrt: public.crt privateKey: private.key ## Trusted Certificates Settings for MinIO. Ref: https://docs.minio.io/docs/how-to-secure-access-to-minio-server-with-tls#install-certificates-from-third-party-cas ## Bundle multiple trusted certificates into one secret and pass that here. Ref: https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-create-kubernetes-secret ## When using self-signed certificates, remember to include MinIO's own certificate in the bundle with key public.crt. ## If certSecret is left empty and tls is enabled, this chart installs the public certificate from .Values.tls.certSecret. trustedCertsSecret: "" ## Enable persistence using Persistent Volume Claims ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/ ## persistence: enabled: true annotations: {} ## A manually managed Persistent Volume and Claim ## Requires persistence.enabled: true ## If defined, PVC must be created manually before volume will be bound existingClaim: "" ## minio data Persistent Volume Storage Class ## If defined, storageClassName: <storageClass> ## If set to "-", storageClassName: "", which disables dynamic provisioning ## If undefined (the default) or set to null, no storageClassName spec is ## set, choosing the default provisioner. (gp2 on AWS, standard on ## GKE, AWS & OpenStack) ## ## Storage class of PV to bind. By default it looks for standard storage class. ## If the PV uses a different storage class, specify that here. storageClass: "" VolumeName: "" accessMode: ReadWriteOnce size: 500Gi ## If subPath is set mount a sub folder of a volume instead of the root of the volume. ## This is especially handy for volume plugins that don't natively support sub mounting (like glusterfs). ## subPath: "" ## Expose the MinIO service to be accessed from outside the cluster (LoadBalancer service). ## or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it. ## ref: http://kubernetes.io/docs/user-guide/services/ ## service: type: ClusterIP clusterIP: ~ ## Make sure to match it to minioAPIPort port: "9000" nodePort: 32000 ## Configure Ingress based on the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/ ## ingress: enabled: false # ingressClassName: "" labels: {} # node-role.kubernetes.io/ingress: platform annotations: {} # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" # kubernetes.io/ingress.allow-http: "false" # kubernetes.io/ingress.global-static-ip-name: "" # nginx.ingress.kubernetes.io/secure-backends: "true" # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" # nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0 path: / hosts: - minio-example.local tls: [] # - secretName: chart-example-tls # hosts: # - chart-example.local consoleService: type: ClusterIP clusterIP: ~ ## Make sure to match it to minioConsolePort port: "9001" nodePort: 32001 consoleIngress: enabled: false # ingressClassName: "" labels: {} # node-role.kubernetes.io/ingress: platform annotations: {} # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" # kubernetes.io/ingress.allow-http: "false" # kubernetes.io/ingress.global-static-ip-name: "" # nginx.ingress.kubernetes.io/secure-backends: "true" # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" # nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0 path: / hosts: - console.minio-example.local tls: [] # - secretName: chart-example-tls # hosts: # - chart-example.local ## Node labels for pod assignment ## Ref: https://kubernetes.io/docs/user-guide/node-selection/ ## nodeSelector: {} tolerations: [] affinity: {} ## Add stateful containers to have security context, if enabled MinIO will run as this ## user and group NOTE: securityContext is only enabled if persistence.enabled=true securityContext: enabled: true runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 # Additational pod annotations podAnnotations: {} # Additional pod labels podLabels: {} ## Configure resource requests and limits ## ref: http://kubernetes.io/docs/user-guide/compute-resources/ ## resources: requests: memory: 16Gi ## List of users to be created after minio install ## users: ## Username, password and policy to be assigned to the user ## Default policies are [readonly|readwrite|writeonly|consoleAdmin|diagnostics] ## Add new policies as explained here https://docs.min.io/docs/minio-multi-user-quickstart-guide.html ## NOTE: this will fail if LDAP is enabled in your MinIO deployment ## make sure to disable this if you are using LDAP. - accessKey: console secretKey: console123 policy: consoleAdmin ## Additional Annotations for the Kubernetes Job makeUserJob makeUserJob: podAnnotations: annotations: securityContext: enabled: false runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 resources: requests: memory: 128Mi nodeSelector: {} tolerations: [] affinity: {} ## List of buckets to be created after minio install ## buckets: # # Name of the bucket # - name: bucket1 # # Policy to be set on the # # bucket [none|download|upload|public] # policy: none # # Purge if bucket exists already # purge: false # # set versioning for # # bucket [true|false] # versioning: false # - name: bucket2 # policy: none # purge: false # versioning: true ## Additional Annotations for the Kubernetes Job makeBucketJob makeBucketJob: podAnnotations: annotations: securityContext: enabled: false runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 resources: requests: memory: 128Mi nodeSelector: {} tolerations: [] affinity: {} ## Use this field to add environment variables relevant to MinIO server. These fields will be passed on to MinIO container(s) ## when Chart is deployed environment: ## Please refer for comprehensive list https://docs.min.io/minio/baremetal/reference/minio-server/minio-server.html ## MINIO_SUBNET_LICENSE: "License key obtained from https://subnet.min.io" ## MINIO_BROWSER: "off" networkPolicy: enabled: false allowExternal: true ## PodDisruptionBudget settings ## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ ## podDisruptionBudget: enabled: false maxUnavailable: 1 ## Specify the service account to use for the MinIO pods. If 'create' is set to 'false' ## and 'name' is left unspecified, the account 'default' will be used. serviceAccount: create: true ## The name of the service account to use. If 'create' is 'true', a service account with that name ## will be created. name: "minio-sa" metrics: serviceMonitor: enabled: false public: true additionalLabels: {} relabelConfigs: {} # namespace: monitoring # interval: 30s # scrapeTimeout: 10s ## ETCD settings: https://github.com/minio/minio/blob/master/docs/sts/etcd.md ## Define endpoints to enable this section. etcd: endpoints: [] pathPrefix: "" corednsPathPrefix: "" clientCert: "" clientCertKey: "" 5-2. MinIOをインストール
以下のコマンドでminioをインストールします。
注意点は以下の通りです。
- MinIOを1台で動かすスタンドアロンモードなので、
mode=standaloneにします。 -
rootUserとrootPasswordでrootユーザーのパスワードを指定します。 - Ingressを使うので
service.typeをClusterIPに設定します。 -
persistence.sizeはMinIOの上限データ容量です。 -
persistence.storageClassは、K3Sのデフォルトストレージクラスのlocal-pathにします。 -
resources.requests.memoryは、MinioのPodのメモリー容量です。 - サービスアクセスにIngressを使うので
ingress.enabledをtrueにします。 - サービスアクセスのIngressのhost名を
ingress.hosts[0]で指定します。 - コンソールアクセスにもIngressを使うので
consoleIngress.enabledをtrueにします。 - コンソールアクセスのIngressのHost名を
consoleIngress.hosts[0]で指定します。
StandaloneモードでDisk数が1個なので以下の機能を使うことができません。
- バージョニング
- オブジェクトロック
- バケットクオータ
パスワードには十分複雑な文字列を利用してください。
helm install minio \ --namespace minio --create-namespace \ --set rootUser=minio \ --set rootPassword=minio#123! \ --set mode=standalone \ --set service.type=ClusterIP \ --set persistence.size=50Gi \ --set persistence.storageClass=local-path \ --set resources.requests.memory=1Gi \ --set ingress.enabled=true \ --set ingress.hosts[0]=minio.192.168.99.123.sslip.io \ --set consoleIngress.enabled=true \ --set consoleIngress.hosts[0]=minio-console.192.168.99.123.sslip.io \ minio/minio 6. MinIOへ接続
6-1. MinIOコンソールへアクセス
コンソールへアクセスするには、https://minio-console.192.168.99.123.sslip.io/ へアクセスします。IPアドレスは、K3Sを動かしているJetsonのIPアドレスに変更してください。
UsernameとPasswordにrootUserとrootPasswordで指定した値を入力してログインします。
6-1-1. テスト用バケットの作成
テスト用のバケット(bucket-1)を作っておきます
「Buckets>「Create a Bucket」をクリックし、「Bucket Name」にbucket-1を指定して「Create Bucket」をクリックします。
7. MinioをCLIから使う
7-1. Minio Clientをインストール
CLIコンソールでMinIOのバケットへファイルを転送します。
その為に、Minio Client(mc)をダウンロードします。
JetsonはARMなのでArm64版のmcをダウンロードします。
wget https://dl.min.io/client/mc/release/linux-arm64/mc chmod +x ./mc その他のバイナリーについては以下からダウンロードしてください。
https://dl.min.io/client/mc/release/
- バージョン確認
$ ./mc --version mc version RELEASE.2021-12-29T06-52-55Z 7-2. Minio Clientの設定
設定方法は2つありますが、.mc/config.jsonを利用します。
./mc alias set k3s-minio https://minio.192.168.99.123.sslip.io minio minio123 --insecure SSL/TLSが自己証明書なので、--insecureを付けておきます。
mcコマンドがあるディレクトリで実行します。
設定された確認する
./mc ls k3s-minio --insecure 確認結果
./mc ls k3s-minio --insecure [2022-01-01 17:40:39 JST] 0B bucket-1/ 7-4. ファイルをアップロード
適当なファイルを生成して、ファイルをアップロードする
$ touch test.txt $ ./mc cp ./test.txt k3s-minio/bucket-1/ --insecure $ ./mc ls k3s-minio/bucket-1/ --insecure [2022-01-02 19:42:53 JST] 0B test.txt 8. まとめ
Jetson NanoにK3Sをインストールして、MinIOをインストールし、Ingressでサーバーを公開してみました。


