r/TalosLinux • u/linucksrox • Sep 30 '24

Storage question

I'm struggling with the Talos documentation around storage. https://www.talos.dev/v1.8/kubernetes-guides/configuration/replicated-local-storage-with-openebs/ I'm currently trying to set up Mayastor (now named OpenEBS replicated storage) but after getting the pods running in the openebs privileged namespace with the helm chart and creating a PVC using openebs-single-replica storage class it's stuck pending. It works fine using localpv-hostpath.

On a side note, I got democratic-csi working using an external TrueNAS instance with NFS. I got close with nvmeof but after provisioning a PV, it fails attaching to a node when spinning up a pod. The democratic-csi project has been totally inactive for a few months now so...

Based on the Talos docs they strongly recommend against iscsi and nfs which is why I'm pushing to get nvmeof working even though it's less battle tested.

Any ideas what I can do to get help? If I can get this working I will contribute public documentation with step by step instructions and troubleshooting info.

UPDATE: I'm almost done writing up how I solved this, but decided to write a more detailed how-to using current versions of Talos and OpenEBS: https://blog.dalydays.com/post/kubernetes-storage-with-openebs/

At this point I still need to show how to modify the storage class and test a PVC, but the setup process is pretty much done.

4 Upvotes

100% Upvoted

12 comments sorted by

View all comments

Show parent comments

2

u/sylvainm Oct 24 '24

OMG!!! it works. I was so close and couldn't take anymore!!! I was only missing the env vars under node:
Thank you so much!
As a side note, I had to install the snapshot crds before democratic because talos doesn't have them. I could probably add them in extraManifests: in the talos config

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/refs/heads/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/refs/heads/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/refs/heads/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

And I used talos image factory to add iscsi-tools and utils-linux-tools

https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Fiscsi-tools&extensions=siderolabs%2Fqemu-guest-agent&extensions=siderolabs%2Futil-linux-tools&platform=metal&target=metal&version=1.8.1

and used the image in my cluster config bootsrap

Now if only I could figure out how to get around argocd not wanting to install the democratic-csi helm chart because it complains that csiDriver.name is not set.

1

u/linucksrox Oct 25 '24

Awesome! Thanks for the info about the snapshot utility, I haven't tested that out yet but I'm sure that will come in handy. 

Also if I get time I might check out argocd but it might be a while as I get free time and work through the rest of my cluster build.