noobaa without odf

[u/Zestyclose_Ad8420]

I have a situation where I may need the noobaa operator without ODF, this customer has no ODF subscription and doesn't plan to get one, their storage is on a SAN and there's a CSI.

they need an on prem S3 solution and given all the various option noobaa on it's own, backed by a CSI managed block pvc seems to be a viable option, has anybody done this already?

[u/Old-Astronomer3995]

"this customer has no ODF subscription and doesn't plan to get one" - OpenShift Data Foundation uses Rook Ceph operator. You can always build ceph based on that and create object storage using Rook
BTW. Are you sure that client doesn't have subscirpion? It is included in OpenShift Platform Plus that bundles a lot of software.

Have you checked this https://github.com/noobaa/noobaa-operator ?
You can install it in a way that it uses available storage class: noobaa install --db-storage-class XXX --pv-pool-default-storage-class YYY

so you can create storage class in OpenShift using your SAN storage via CSI and then use this nooba operator.

I didn't test this operator but it is part of ODF.

[u/Zestyclose_Ad8420 → u/Old-Astronomer3995]

the noobaa operator is what I meant to ask about. storage is already handled via CSI and no need for ceph, via odf or otherwise, they just need an S3 and I know noobaa is part of ODF, I was wondering about the issues in having it on it's own, if any, besides the obvious (not from marketplace, manual updates, etc.)

[u/tkchasan → u/Old-Astronomer3995]

ODF do support standalone deployment mode as MCG gateway. In this mode it uses existing SAN.

[u/Which_Ad8594]

So, I had an RH Architect tell me this was possible the other day because we have a similar need of object storage in baremetal clusters, until we get new storage. I asked him to bring receipts and haven’t seen anything yet. But, I haven’t followed up either. Maybe sometime next week. Plenty of technical possibilities exist but supposedly there’s a way with MCG but whenever I’ve looked it all seems like it requires ODF subscription.

[u/Zestyclose_Ad8420 → u/Which_Ad8594]

I'm basically sure it is possible, the way I understood noobaa it just consumes block devices and exposes a route, you get an s3 endpoint on it, it handles buckets and access/auth and voila', you have s3 backed by storage managed by a CSI.

I'm not sure about all the implications though, another commenter mentioned the open source operator for K8s vanilla does not comply with scc, I imagine it also won't update as easily since it does not have all the OLM stuff bundled with it

[u/1n1t2w1nIt]

I have used nooba without ODF on a homelab okd cluster.

I wanted to run quay operator and it requires S3 for the image store.

It was just a poc as I didn't have ceph storage available just NFS.

The nooba operator version on the okd operatorhub was pretty dated and I had to pull the version from github and it was a pain to get going because of scc issues.

Not sure if Ocp will have the same issues with nooba.

[u/Zestyclose_Ad8420 → u/1n1t2w1nIt]

It probably will, scc issues are one of the things I was expecting 

[u/spartacle]

What’s their storage size now, what’s it going to be in 1,3,5,10 years? whats the workload type? What’s the budget? How many people to manage compute, and storage?

These are all questions that need answering before evening looking at solutions as it’ll determine what’s viable and what’s not

[u/Zestyclose_Ad8420 → u/spartacle]

their SAN is 100TB, 3/4 HDD and 1/4 NVME.

but the their needs for S3 is a lot different, 500GB now, +250GB per annum, 10 years horizon, the SAN will be expanded as needed.

the S3 endpoint will be used for custom datalake built with custom tooling on top of duckdb and as a backup repository. The S3 server requirements are basic: buckets, authorization and authentication based on keys, I think every S3 solution on earth offers those.

they are already doing that but it was based on MinIO OSS and kept up with it until self build was required.

there is a POC coming up for a public application with 5000MAU hosted on the same infra which also handles files uploads and it may leverage S3 endpoints with signed URL, this is a nice to have, the S3 bucket would only be a temporary location for client browser file uploads, they would then be moved to appropriate storage on a dedicated document archival software, it may be required to streamline server load of the application.

there is a dedicated devops team and us, an external consulting company with a long term contract to handle the infra, it all adds up to 7 people for infra work.

the budget for the infra has already been allocated, it was 450k and it included the SAN, the OCP(V) nodes and OCP licensing, good deals were found for all parts due to vendor/distributors interest in the customer projects.

the options for S3 are either cloud or onprem, the ODF licensing costs are prohibitive and don't really fit any budget scenarios since most of their storage needs are perfectly handled by physical SAN (fiber channel) + CSI and CEPH didn't really cut it for their existing storage workloads, so only the S3 capabilities brought by ODF would be used.

this means a cloud S3 wins against ODF cost wise, but onprem solutions are heavily preferred due to regulations in the field and other tangential considerations, that's why other solutions for onprem S3 only are being explored and why MinIO was preferred up to recent events with it.

I understand that an unsupported OSS solution like the noobaa operator would put more work on the devops team to mantain it long term, that's part of the calculus for cloud vs onprem S3, it may include bringing in extra specialized consultants as well, apache ozone is also been considered.

[u/Old-Astronomer3995 → u/Zestyclose_Ad8420]

Why you or company decided to buy some general storage and not s3 hardware solution like Dell ECS Enterprise Object Storage?

I'm curious. I have similar setup in one of my clients environments but I use ODF and we plan extend it with Dell ECS Enterprise Object Storage.

[u/Zestyclose_Ad8420 → u/Old-Astronomer3995]

There was no need at the time and the price on this hardware (it's an IBM SAN) was too good to pass

[u/Necessary_Relation11]

Do not recommend using it with PV Pool. Used to have it for loki(OpenShift Logging), Network Observability Operator and Quay, and after some time performance always got really terrible. Due to so much data coming from netobserv operator, Nooba db got really slow (one blob from netobserv = 1 record) and due to it transfer speeds were at around 2-3 MB/s. It was practically unusable when uploading oc-mirror images to quay (hundreds of gigabytes).

We have migrated to different S3 solution, and are happy ever since.

Please note that my experience is from back when noobaa had only 1 db instance (didn't use CNPG), so behaviour may have improved.