SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（静态） - Java

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

Kubernetes PersistentVolumes 持久化存储方案中，提供两种 API 资源方式: PersistentVolume(简称 PV) 和 PersistentVolumeClaim(简称 PVC)。PV 可理解为集群资源，PVC 可理解为对集群资源的请求，Kubernetes 支持很多种持久化卷存储类型。Ceph 是一个开源的分布式存储系统，支持对象存储、块设备、文件系统，具有可靠性高、管理方便、伸缩性强等特点。在日常工作中，我们会遇到使用 k8s 时后端存储需要持久化，这样不管 Pod 调度到哪个节点，都能挂载同一个卷，从而很容易读取或存储持久化数据，我们可以使用 Kubernetes 结合 Ceph 完成。

图1 Pod 存储卷、PVC 及存储设备的调用关系

实验环境搭建 - 静态供给

图2 实验环境架构

1、所有节点安装

# zypper -n in ceph-common

复制 ceph.conf 到 worker 节点上

# scp admin:/etc/ceph/ceph.conf /etc/ceph/

2、创建池

# ceph osd pool create caasp4

3、创建 key ，并存储到 /etc/ceph/ 目录中

# ceph auth get-or-create client.caasp4 mon 'allow r' \

    osd 'allow rwx pool=caasp4' -o /etc/ceph/caasp4.keyring

4、创建 RBD 镜像，2G

# rbd create caasp4/ceph-image-test -s

5、查看 ceph 集群 key 信息，并生成基于 base64 编码的key

# ceph auth list

.......

client.admin

        key: AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA==

        caps: [mds] allow *

        caps: [mgr] allow *

        caps: [mon] allow *

        caps: [osd] allow *

.......

client.caasp4

        key: AQD1VJddM6QIJBAAlDbIWRT/eiGhG+aD8SB+5A==

        caps: [mon] allow r

        caps: [osd] allow rwx pool=caasp4

client.caasp4 密钥以 base64 编码

# echo AQD1VJddM6QIJBAAlDbIWRT/eiGhG+aD8SB+5A== | base64

QVFEMVZKZGRNNlFJSkJBQWxEYklXUlQvZWlHaEcrYUQ4U0IrNUE9PQo=

6、Master 节点上，创建 secret 资源，插入 base64 key

# vi ceph-secret-test.yaml

apiVersion: v1

kind: Secret

metadata:

  name: ceph-secret-test

data:

  key: QVFEMVZKZGRNNlFJSkJBQWxEYklXUlQvZWlHaEcrYUQ4U0IrNUE9PQo=

# kubectl create -f ceph-secret-test.yaml

secret/ceph-secret-test created

# kubectl get secrets ceph-secret-test

NAME               TYPE     DATA   AGE

ceph-secret-test   Opaque         14s

7、Master 节点上，创建 Persistent Volume

# vim ceph-pv-test.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

  name: ceph-pv-test

spec:

  capacity:

    storage: 2Gi

  accessModes:

    - ReadWriteOnce

  rbd:

    monitors:

      - 192.168.2.40:6789

      - 192.168.2.41:6789

      - 192.168.2.42:6789

    pool: caasp4

    image: ceph-image-test

    user: caasp4

    secretRef:

      name: ceph-secret-test

    fsType: ext4

    readOnly: false

  persistentVolumeReclaimPolicy: Retain

# kubectl create -f ceph-pv-test.yaml

# kubectl get pv

NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE

ceph-pv-test   2Gi        RWO            Retain           Available                                   29s

8、创建 Persistent Volume Claim （PVC）

# vim ceph-claim-test.yaml

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

  name: ceph-claim-test

spec:

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 2Gi

# kubectl create -f ceph-claim-test.yaml

# kubectl get pvc

NAME              STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE

ceph-claim-test   Bound    ceph-pv-test   2Gi        RWO                           14s

9、创建 Pod

# vim ceph-pod-test.yaml

apiVersion: v1

kind: Pod

metadata:

  name: ceph-pod-test

spec:

  containers:

  - name: ceph-busybox

    image: busybox

    command: ["sleep", ""]

    volumeMounts:

    - name: ceph-vol-test

      mountPath: /usr/share/busybox

      readOnly: false

  volumes:

  - name: ceph-vol-test

    persistentVolumeClaim:

      claimName: ceph-claim-test

# kubectl create -f ceph-pod-test.yaml

# kubectl get pods

NAME            READY   STATUS              RESTARTS   AGE

ceph-pod-test   /     ContainerCreating             6s

# kubectl get pods -o wide

NAME            READY   STATUS              RESTARTS   AGE   IP       NODE

ceph-pod-test   /     ContainerCreating             32s   <none>   worker02

10、worker02 节点显示 RBD 映射

# rbd showmapped

id pool   namespace image           snap device

  caasp4           ceph-image-test -    /dev/rbd0

# df -Th | grep ceph

/dev/rbd0               ext4      2.0G  6.0M  1.9G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/caasp4-image-ceph-image-test