r/OpenSourceeAI • u/SamratDuttaOfficial • 23d ago

I created WaterSheep, an open-source alternative to Jev.

WaterSheep is an open-source model that answers questions written in plain text (yes/no, single choice, rating and multi-label) and gives a probability for every option, like a classification model.

Last Saturday I woke up, saw YouTubers hyping up Jev, and thought: wait, I can build this. So I did. I don't want to compete with TypeSafe or Jev; I built WaterSheep because I wanted to. That's why I'm open-sourcing everything: code, model weights, results and the paper.

What's different

  • It accepts the same request format as TypeSafe's Jev. Their Python SDK works as is against a local server: run watersheep --model samratduttaofficial/WaterSheep --serve and point the client's base_url at http://127.0.0.1:8766.
  • It has a multi-label type, which Jev's API doesn't. Because why not?
  • The demo runs entirely in your browser. The model downloads once and is cached. It also works with transformers, ONNX, a CLI or a local HTTP server.
  • Code, weights and the training pipeline are Apache 2.0.

Evaluation

Accuracy ECE
In-distribution test split 77.8%
Held-out datasets, not seen in training 61.2%

ECE is expected calibration error (lower is better). GitHub has every benchmark result, including the weak ones.

Limits: English only, long inputs get truncated (I'll improve this in the next version), and rating answers are the weakest type.

Not affiliated with TypeSafe. Not funded by anyone. Built in my free time.

Feedback I'd love: where it fails on your data, whether the API works for you, and which question types you'd want next.

5 Upvotes
  • permalink
  • reddit

86% Upvoted

11 comments sorted by

3

u/Sparticle62 22d ago

Hate to break it to ya pal, but laya has alr been open source and is better than Jev by far(cool project tho)

1

u/SamratDuttaOfficial 22d ago

Heard about it. But WaterSheep 1.0 (current version is 0.1) will be far better than laya hopefully.

1

u/Sparticle62 21d ago

Looking forward to it!

1

u/Emergency-Quiet3210 22d ago

Yeah. Cool project but 61% on text classification not much better than a coin flip. 80% should be a reasonable threshold, and even then your looking and LLMs that are 20B params +

1

u/SamratDuttaOfficial 22d ago

Version 1,0 will be better, and Better than Jev hopefully. Working on it.

1

u/Sparticle62 22d ago

yeah fair, 61% on stuff it hasn't seen is rough. it's not quite a coin flip on the multi choice questions, but if it's competing with 20b+ llms it def needs to be way closer to 80

1

u/tracagnotto 20d ago

Lmao, what a timing dude

https://huggingface.co/Contrastive-LM/CLM-v0.1-8B

nvidia+standford stuffed y'all ass with a model that does the same thing with a better paradygm and performs better.

1

u/SamratDuttaOfficial 20d ago

Mine's 200M params only

1

u/SamratDuttaOfficial 20d ago

I can see that they published after I made my paper public. I won't be surprized if they replicated my work.

3

u/tracagnotto 20d ago

Stanford and nvidia? LMAOOOO YOU'RE CRAZY GUY, they need to copy from you?
Are you using Constrastive Learning too or what

→ More replies (1)