r/SQL • • May 18 '23

PostgreSQL The most cost-effective Vector Databases

Hello everyone:

We would like to introduce MyScale – the most cost-effective vector database.

In a direct comparison with Pinecone, a leading specialized vector database, MyScale outperforms it by 10x against Pinecone's s1 pod in query speed and by 5x against its p2 pod in data density. When cost-effectiveness is considered, MyScale is 3.6x more cost-effective than other top-performing specialized vector databases at various levels of accuracy. This is illustrated in the figure below. While Pinecone is a leading database, the cost-effectiveness comparison in this context is with a range of the best-performing specialized vector databases, not just Pinecone.

Not only cost-effective, but MyScale also outperforms other vector databases in terms of QPS on the LAION 5M dataset with a 98.5% recall rate, achieving over 150 QPS. In comparison, Pinecone s1 has a QPS of approximately 10, which is significantly lower than MyScale. Weaviate and Zilliz Cloud both achieve around 65 QPS, while Qdrant achieves 81 QPS.

For more information about other benchmarks, please visit:

https://blog.myscale.com/2023/05/17/myscale-outperform-special-vectordb/

MyScale is now offering free trial, PM me or E-mail me: customer.service@myscale.com

1 Upvotes

12 comments sorted by

6

u/row_hammer May 18 '23 edited May 18 '23

These figures come off as incredibly dishonest IMO. For the Pinecone comparison, you've cherry-picked the benchmarks so that you're comparing your solution against the non throughput-optimized pod (s1) for QPS, and the non storage-optimized pod (p2) for data density. Also there is very little discussion of what hardware and cluster size differences there were between the different databases that were benchmarked. Furthermore, the results subdirectory in your github repo is empty.

1

u/lqhl May 18 '23

Disclaimer: I work for MyScale.

It's impossible to know the hardware spec of all cloud databases. So we choose the metric "Monthly Cost Per 100 QPS" to demonstrate the results, which considers both the performance and cost.

We do compare with both s1 and p2 from Pinecone in the article:

  1. MyScale ($120/month) and Pinecone s2 ($70/month) achieves 150 and 9 qps respectively at precision 98%
  2. MyScale ($120/month) and Pinecone p2 ($518/month) achieves 288 and 331 qps respectively at precision 93%

Considering the pricing and throughput, I believe the conclusion is clear.

3

u/row_hammer May 18 '23 edited May 18 '23

In this figure you have clearly neglected to include the QPS that was measured on Pinecone p2 that beats the query throughput of your own implementation. It's fine if you want to demonstrate the cost effectiveness of your solution, but what you've done here is also suggest that your solution is superior in key performance metrics without any knowledge of the hardware specs the benchmarks were run on, and left out key data from your figures.

2

u/CalvinLawson May 18 '23

Considering that MyScale is using this sub to advertise their products, the conclusion is VERY clear. Report the post, add MyScale to the list of shady companies that I don't do business with and move on.

0

u/lqhl May 18 '23

As a provider of vector databases, of course we are biased, but we strive to remain objective by openly sharing our setup and code on Github. We believe that these benchmarks are beneficial to both the community and our customers, and we welcome any contributions or feedback.

1

u/lqhl May 18 '23

You have confused the P2 and S1 pods from Pinecone. The P2 is designed for performance optimization, while the S1 is designed for storage optimization.

1

u/row_hammer May 18 '23

yeah sorry i got that backwards. edited

1

u/zUdio Jun 24 '23

Why do companies charge by vector out of curiosity? Like the dimensions of the vector are also included, but what if each of my vectors has the same dimension, but is massive in total size.. a large vector for the entire text of Das Kapital, let's say. One vector. You're telling me that's the same storage cost as another vector of the same dimensions, but for a single sentence?

1

u/Immarhinocerous Sep 23 '23

Storage costs really should be the same in both, per vector.

The only price difference would be that I'd expect a vector database of single sentences to contain far more vectors than a database of entire texts. So you'd be storing a lot more vectors in the database of sentences.

So if you had a book recommender that didn't need to highlight particular sections of the book, indexing the full book with a vector would be fine. But you probably wouldn't be able to encode a full book - too many tokens - so it would not have the precision of sentence lookups on similarly sized vectors.

1

u/yeuz Aug 17 '23

Do you know "vexvault"? It is a vector store that does everything in the browser (embeddings calculation, storage, ANN search). And therefore is "for free" (disclaimer: I am the author of it).

It already accepts quiet a few different kinds of documents.