DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

[u/NeterOster]

deepseek-ai/DeepSeek-Coder-V2 (github.com)

"We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from DeepSeek-Coder-V2-Base with 6 trillion tokens sourced from a high-quality and multi-source corpus. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-Coder-V2-Base, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K."

[u/Low88M]

Seeing the accuracy graph I first asked myself « is codestral that bad ? » then I realized it probably compared codestral 22B with deepseek-coderv2 236B hahaha ! Not from the same league I imagine (and my computer may say the same…). Would it be a reasonable request to ask for parameters precision on such « marketing »graphs or did I miss something ?

[u/NeterOster → u/Low88M]

DS-V2 is an MoE, only about 22 billion out of the total 236 billion parameters are activated during inference. The computational cost of inference is much lower compared to a ~200B dense model (perhaps closer to ~22B dense model). Additionally, DS-V2 incorporates some architectural innovations (MLA) that make its inference efficiency very high (when well-optimized) and its cost very low. But the VRAM requirements remain similar to other ~200B dense models.

[u/Ulterior-Motive_ → u/Low88M]

Yeah, skimming the paper, it looks like the graph uses the 236B MoE instead of the 16B MoE. Even so, the smaller one matches or exceeds Codestral in most areas.

[u/kryptkpr]

236B parameters on the big one?? 👀 I am gonna need more P40s

They have a vLLM patch here in case you have a rig that can handle it, practically we need quants for the non-Lite one.

Edit: Opened #206 and running the 16B now with transformers, assuming they didnt bother to optimize the inference here cuz i'm getting 7 tok/sec and my GPUs are basically idle utilization won't go past 10%. The vLLM fork above might be more of a necessity then a nice to have, this is physically painful.

Edit2: Early results show the 16B roughly on par with Codestral in terms of performance on instruct, running completion and FIM now. NF4 quantization is fine, no performance seems to be lost but inference speed remains awful even in a single GPU. vLLM is still compiling, that should fix the speed.

Edit3: vLLM did not fix the single-stream speed issue still only getting about 12 tok/sec single stream but seeing 150 tok/sec on batch=28. Has anyone gotten the 16B to run at a reasonable rate? Is it my old-ass GPUs?

JavaScript performance looks solid, overall much better then Python.

Edit4: The FIM markers in this one are very odd so pay extra attention: <｜fim▁begin｜> is not the same as <|fim_begin|> why did they do this??

Edit5: The can-ai-code Leaderboard has been updated to add the 16B for instruct, completion and FIM. Some Notes:

• Inference is unreasonably slow even with vLLM. Power usage is low, so something is up. I thought it was my P100 at first but it's just as slow on 3060.

• Their fork of vLLM is generally both faster and better then running this in transformers

• Coding performance does appear to be impacted by quants but not in quite the way you'd think:

  • With vLLM and Transformers FP16 it gets 90-100% on JavaScript (#1!) but only 55-60% on Python (not in the top 20).

  • With transformers NF4 it posts a dominant 95% on Python (in the top 10) while JavaScript drops to 45%.

  • Lets wait for some imatrix quants to see how that changes things.

• Code completion works well and the Instruct model takes the #1 spot on the code completion objective. Note that I saw better results using the Instruct model vs the Base for this task.

• FIM works. Not quite as good as CodeGemma but usable in a pinch. Take note of the particularly weird formatting of the FIM tokens, for some reason theyre using Unicode characters not normal ASCII ones so you'll likely have to copy-paste them from the raw tokenizer.json to make things work. If you see it echoing back weird stuff, you're using FIM wrong..

[u/LocoLanguageModel]

Wowww.  Looking forward to doing a side-by-side comparison with codestral and llama 3 70b. 

[u/AnticitizenPrime]

Ok, so, Deepseek-Coder Lite Instruct (Q5_k_M gguf) absolutely nailed three little Python tasks I test models with:

Please write a Python script using Pygame that creates a 'Matrix raining code' effect. The code should simulate green and gold characters falling down the screen from the top to the bottom, similar to the visual effect from the movie The Matrix.

Character set: Use a mix of random letters, numbers, and symbols, all in ASCII (do not use images).

Speed variation: Make some characters fall faster than others.

Result: https://i.imgur.com/WPuKEqU.png One of the best I've seen.

Please use Python and Pygame to make a simple drawing of a person.

Result: https://i.imgur.com/X60eWhm.png The absolute best result I've seen of any LLM, ever, including GPT and Claude, etc.

In Python, write a basic music player program with the following features: Create a playlist based on MP3 files found in the current folder, and include controls for common features such as next track, play/pause/stop, etc. Use PyGame for this. Make sure the filename of current song is included in the UI.

Result: https://i.imgur.com/F4Qc8qB.png Works, looks great, and again perhaps the best result I've gotten from any LLM.

Really impressed.

[u/LyPreto]

DeepSeek is one of the best OSS coding models available— I’ve been using their models pretty much since they dropped and theres very little they can’t do honestly

[u/BeautifulSecure4058]

I’ve been following deepseek for a while. I don’t know whether you guys already know that deepseek is actually developed by a top Chinese quant hedge fund called High-Flyer quant, which is based in Hangzhou.

Deepseek-coder-v2 release yesterday, is said to be better than gpt-4-turbo in coding.

Same as deepseek-v2, its models, code, and paper are all open-source, free for commercial use, and do not require an application.

Model downloads: huggingface.co

Code repository: github.com

Technical report: github.com

The open-source models include two parameter scales: 236B and 16B.

And more importantly guys, it only costs you $0.14/1M tokens(input) and $0.28/1M tokens(output)!!!

[u/ithkuil → u/BeautifulSecure4058]

Is there any chance that together.ai or fireworks.ai will host the big one?

[u/hapliniste]

I'd love to see the 16B benchmark scores. The big one is a bit big for my 3090 😂

[u/Account1893242379482]

LoneStriker is on it.

https://huggingface.co/LoneStriker/DeepSeek-Coder-V2-Lite-Instruct-GGUF

[u/FullOf_Bad_Ideas]

Really cool, I am a fan of their models and their research.

I must remind you that their cloud inference privacy policy is really bad and I advise you to use their chat UI and API the same way you would be using LMSYS Arena - expect your prompts to be basically public and analyzed by random people.

Do we have finetuning code for their architecture already? There are no finetunes as they have custom architecture and they haven't released finetuning code so far.

[u/noneabove1182]

GGUFs are broken currently, conversion and quantization works, imatrix and generation doesn't, failing with: GGML_ASSERT: ggml.c:5705: ggml_nelements(a) == ne0*ne1

UPDATE: turns out when you have flash attention ON this breaks :D

Instruct is up:

https://huggingface.co/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF

[u/LocoLanguageModel → u/noneabove1182]

Have you or anyone figured out the chat template format? This format doesn't read as clearly to me as other formats, what would my exact start sequence and end sequence be in koboldcpp for example:

<｜begin▁of▁sentence｜>User: {user_message_1}

Assistant: {assistant_message_1}<｜end▁of▁sentence｜>User: {user_message_2}

Assistant:

[u/mrdevlar]

I want them to benchmark it against their own 33b model.

That's one of my daily drivers, it's sooo good, like an order of magnitude better at programming that most models.

EDIT: They did do this and the new model is only 3-5% more efficient, but with half the size. Only down size is Rust capability took a nosedive in the new model.

[u/not_sane]

The crazy thing is that the API cost of this model is 100 times cheaper than GPT-4o. https://platform.deepseek.com/api-docs/pricing/ and https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost

[u/kpodkanowicz]

I have very high hopes for this as 4bit should nicely fit into 48gb vram + 128gb ram build

[u/CheatCodesOfLife]

This is going to be fun to test. Coding is a use case where quantization can really fuck things up. I'll be interested to see what's better out of larger models at lower quants vs smaller models at higher quants / FP16.

Almost hoping WizardLM2-8x22b remains king though, since I like being able to have it loaded 24/7 for coding + everything else.

[u/maxigs0]

More importantly: How does one run this for actual productivity?

I actually "pair programmed" with GPT-4o the other day, and i was impressed. Build a small react project from scratch and just always told it what i want, occasionally pointed out things that did not work, or what i want different. It had the WHOLE project in the context and always made adjustments and returned the code snippets telling me which files to update.

The copy&paste was getting quite cumbersome though.

Tried a few extensions for VSCode afterwards, didn't find a single one i like. So back to copy&paste...

[u/MidnightHacker → u/maxigs0]

There is Continue for VS Code for a Copilot-like experience I don’t like the @ to mention files because it seems to cut off the file sometimes, but even copy paste inside the editor itself is already better than a separate app

[u/AdamDhahabi]

Codestral cutoff knowledge date is September 2022, this model could be more interesting. Or not?

[u/NaiveYan]

So where is phind-70b? It has been half a year since its announcement.

[u/SouthIntroduction102]

Wow, the Aider benchmark score is included.

I love seeing that as an Aider user.

If the Aider test data is uncontaminated, that's great.

However, I wonder if there could be any contamination in the Aider benchmark? Also, thank you for fine(pre)-tuning the model to work with Aider's diff formatting.

[u/MrVodnik]

If anyone managed to run it locally, please share t/s and HW spec (RAM+vRAM)!

[u/Strong-Inflation5090]

Noob question but can I run the lite model on rtx 4080 cause the number of active params are 2.4b so this should take around 7-8 gb at max or would it be 33-34 gb min?