Kubeflow 설치
Kubeflow v1.10.2 기준
Kubeflow package download
필요한 kubeflow manifest 및 image를 다운로드 합니다.
$ curl -sfL https://cube.k3.acornsoft.io/download-package | ACCOUNT={{ ID }} PASSWD={{ PW }} PACKAGE=kubeflow VERSION=1.10.2 sh -
[INFO] starting download file kubeflow-manifests-1.10.2.tar.gz
######################################################################## 100.0%
[INFO] decompressing file kubeflow-manifests-*.tar.gz
[INFO] starting download file kubectl-*-linux-amd64
######################################################################## 100.0%
[INFO] starting download file kubeflow-images-1.10.2.tar.gz
##### 8.3%
...Cluster node들의 값을 변경 및 추가한 후 ssh 재접속 합니다.
$ sudo vi /etc/security/limits.conf
# 추가
* soft nofile 65535
* hard nofile 65535
$ sudo vi /etc/sysctl.d/99-sysctl.conf
# 추가
fs.file-max = 2097152
fs.inotify.max_user_instances=1280
fs.inotify.max_user_watches=655360
sudo sysctl -p다운로드 받은 kubeflow 이미지를 archive/push/registry 경로로 압축을 해제 후 원본 압축 파일은 옮겨놓습니다.
$ tar -zxvf ./archive/push/registry/kubeflow-images-1.10.2.tar.gz -C ./archive/push/registry
docker.io_bitnami_kubectl:1.30.4.tar
docker.io_filebrowser_filebrowser:v2.25.0.tar
docker.io_grafana_grafana:5.3.4.tar
docker.io_kong_httpbin:latest.tar
...
cubectl-v1.x.x/
├── archive
│ └── push
│ ├── localrepo
│ └── registry
│ └── kubeflow-images-1.10.2.tar.gz(해당 파일을 압축 해제 후 위치 변경)
├── bin
...
├── LICENSE.txt
├── kubeflow
│ ├── bash
│ │ └── ...
│ ├── bin
│ │ └── ...
│ └── manifest
│ └── ...
└── logsPush kubeflow image
Push registry 명령어를 이용하여 kubeflow image를 업로드해야합니다.
한 번에 모든 이미지를 올릴 수 없기 때문에 2~3번으로 나눠서 업로드하는 것을 권장합니다.
docker.io,gcr.io,ghcr.io등 태그에 따라서 나눠주는 것을 권장합니다.
$ ./bin/cubectl-test push registry -u test -p ./config/sshkey
┌─────────────────┬───────────┬────────────┬────────────┬──────────────────────────────────────────────────────────────────────────────────────────────┐
│ KIND │ NODE │ IP │ PRIVATE IP │ PUSH IMAGES │
├─────────────────┼───────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ Push Repository │ node-regi │ 10.10.0.49 │ 10.10.0.49 │ docker.io_bitnami_kubectl:1.30.4.tar │
│ │ │ │ │ docker.io_filebrowser_filebrowser:v2.25.0.tar │
│ │ │ │ │ gcr.io_knative-releases_knative.dev_eventing_cmd_in|memory_channel|controller:v1.16.4.tar │
│ │ │ │ │ gcr.io_knative-releases_knative.dev_eventing_cmd_in|memory_channel|dispatcher:v1.16.4.tar │
...
│ │ │ │ │ quay.io_jetstack_cert-manager-controller:v1.16.1.tar │
│ │ │ │ │ quay.io_jetstack_cert-manager-webhook:v1.16.1.tar │
│ │ │ │ │ quay.io_kuberay_operator:v1.3.2.tar │
│ │ │ │ │ quay.io_oauth2-proxy_oauth2-proxy:latest.tar │
│ │ │ │ │ quay.io_oauth2-proxy_oauth2-proxy:v7.7.1.tar │
└─────────────────┴───────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
Is this ok [Y/n]:Kubeflow install
Csi-nfs-driver 설치가 필수입니다.
CUBE_HOME 환경 변수를 설정합니다.
# 예시
test@kubeflow-cp-1:~/cubectl-v1.3.3$ export CUBE_HOME=$(pwd)
test@kubeflow-cp-1:~/cubectl-v1.3.3$ echo $CUBE_HOME
/home/test/cubectl-v1.3.3Kubeflow online install
Online install
./kubeflow/bin/install base -c {cubectl -c 옵션과 동일}Kubeflow airgap install
Airgap으로 설치 시 파일 내용을 변경합니다.
10.10.0.49는 예시 registry ip 입니다.
$ vi ${CUBE_HOME}/kubeflow/manifests/applications/pipeline/upstream/base/installs/generic/pipeline-install-config.yaml
$ vi ${CUBE_HOME}/kubeflow/manifests/applications/pipeline/upstream/base/installs/generic/postgres/pipeline-install-config.yaml
# registry.k8s.io/busybox -> docker.io/library/busybox$ vi ${CUBE_HOME}/kubeflow/manifests/applications/pipeline/upstream/base/installs/multi-user/pipelines-profile-controller/sync.py
# ghcr.io/kubeflow/kfp-frontend -> 10.10.0.49/ghcr.io/kubeflow/kfp-frontend
# ghcr.io/kubeflow/kfp-visualization-server -> 10.10.0.49/ghcr.io/kubeflow/kfp-visualization-server$ vi ${CUBE_HOME}/kubeflow/manifests/common/istio/profile.yaml
$ vi ${CUBE_HOME}/kubeflow/manifests/common/istio/istio-install/overlays/insecure/configmap-patch.yaml
# hub: gcr.io/istio-release -> hub: 10.10.0.49/gcr.io/istio-release로 변경$ vi ${CUBE_HOME}/kubeflow/manifests/common/cert-manager/base/upstream/cert-manager.yaml
# --acme-http01-solver-image=quay.io/jetstack/cert-manager-acmesolver:v1.16.1 ->
# --acme-http01-solver-image=10.10.0.49/quay.io/jetstack/cert-manager-acmesolver:v1.16.1로 변경$ vi ${CUBE_HOME}/kubeflow/manifests/applications/pipeline/upstream/third-party/argo/base/workflow-controller-deployment-patch.yaml
# quay.io/argoproj/argoexec:v3.5.14 -> 10.10.0.49/quay.io/argoproj/argoexec:v3.5.14$ vi ${CUBE_HOME}/kubeflow/manifests/applications/pipeline/upstream/base/pipeline/ml-pipeline-apiserver-deployment.yaml
# ghcr.io/kubeflow/kfp-driver:2.5.0 -> 10.10.0.49/ghcr.io/kubeflow/kfp-driver:2.5.0
# ghcr.io/kubeflow/kfp-launcher:2.5.0 -> 10.10.0.49/ghcr.io/kubeflow/kfp-launcher:2.5.0$ vi ${CUBE_HOME}/kubeflow/manifests/common/knative/knative-serving/base/upstream/serving-core.yaml
# queue-sidecar-image: ... -> queue-sidecar-image: 10.10.0.49/gcr.io/knative-releases/knative.dev/serving/cmd/queue:v1.16.2Airgap install
$ ./kubeflow/bin/install -r {registry ip} -c {cubectl -c 옵션과 동일} overlays/airgapped
...
validatingwebhookconfiguration.admissionregistration.k8s.io/spark-operator-webhook serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/validator.training-operator.kubeflow.org serverside-appliedLast updated