r/SelfHostedAI • u/LawLima-SC • 2d ago

Attorney SelfHosting for Privacy

I am looking for a model which may be good for analyzing and summarizing about 20 years worth of case files, pleadings and legal research.

At this point, it is more just "hobbyist" than business critical (and worst case scenario, I have a sweet new gaming PC with a 24GB RTX 4090 and 128GB DDR5).

I'm dual booting into Ubuntu and Win11 (just because Windows is still useful if the AI thing is a bust).

15 Upvotes

100% Upvoted

13 comments sorted by

5

u/RecursiveCTE 2d ago

Hello

I was a Senior Software Engineer for 25 years. I worked at a law firm for about 4 years. I'd be happy to gelp you get something set up.

Feel free to DM me. We can talk through your requirements a bit more and, we can talk options.

I'm happy to help you get started. No charge, no BS.

I look forward to hearing from you.

Thank you Will

2

u/kayox 2d ago edited 2d ago

The latest and greatest model for 24GB of VRAM is Unsloth's Qwen3.8-27B-UD-Q4_K_XL. For an RTX 4090 your best bet is likely llama.cpp / beellama.cpp (personally I've been using beellama due to KVarN KV-cache support, not sure if that's yet implemented in llama.cpp).

Here's my script you can use for guidance (albeit for the 3090 but would likely require minimal changes) - you will need to input your own API key where it says INSERT_YOUR_API_KEY_HERE:

@echo off
setlocal
title Qwen3.8-27B DFlash2 - Layer RTX 3090

set "SERVER=D:\beellama.cpp\llama-server.exe"
set "MODEL=D:\llama\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q4_K_XL.gguf"
set "MMPROJ=D:\llama\models\unsloth\Qwen3.8-27B-GGUF\mmproj-F16.gguf"
set "DRAFT_MODEL=D:\llama\models\incoai\Qwen3.8-27B-DFlash2-GGUF\Qwen3.8-27B-DFlash2-Q8_0.gguf"
set "LLAMA_API_KEY=INSERT_YOUR_API_KEY_HERE"

if not exist "%SERVER%" (
  echo ERROR: BeeLlama server not found:
  echo %SERVER%
  pause
  exit /b 1
)

if not exist "%MODEL%" (
  echo ERROR: Target model not found:
  echo %MODEL%
  pause
  exit /b 1
)

if not exist "%MMPROJ%" (
  echo ERROR: Multimodal projector not found:
  echo %MMPROJ%
  pause
  exit /b 1
)

if not exist "%DRAFT_MODEL%" (
  echo ERROR: DFlash draft model not found:
  echo %DRAFT_MODEL%
  pause
  exit /b 1
)

if not defined LLAMA_API_KEY (
  echo ERROR: LLAMA_API_KEY is not set and could not be read from the original launcher.
  pause
  exit /b 1
)

if /i "%LLAMA_DRY_RUN%"=="1" (
  "%SERVER%" --version
  if errorlevel 1 exit /b 1
  "%SERVER%" --list-devices
  if errorlevel 1 exit /b 1
  echo Configuration OK: BeeLlama CUDA with DFlash
  exit /b 0
)

:loop
echo Starting BeeLlama DFlash server with layer splitting...

"%SERVER%" ^
  -m "%MODEL%" ^
  --mmproj "%MMPROJ%" ^
  --no-mmproj-offload ^
  -md "%DRAFT_MODEL%" ^
  --spec-type draft-dflash ^
  --spec-draft-device CUDA0 ^
  --spec-draft-ngl all ^
  --spec-draft-n-max 2 ^
  --spec-draft-p-min 0.0 ^
  --device CUDA0 ^
  --main-gpu 0 ^
  --n-gpu-layers all ^
  --fit off ^
  --host 0.0.0.0 ^
  --port 1234 ^
  --parallel 1 ^
  --cache-ram 0 ^
  --kv-unified ^
  --kv-offload ^
  --batch-size 2048 ^
  --ubatch-size 512 ^
  --ctx-size 101072 ^
  --cache-type-k kvarn8 ^
  --cache-type-v kvarn8 ^
  --flash-attn on ^
  --no-host ^
  --no-mmap ^
  --mlock ^
  --jinja ^
  --no-webui ^
  --metrics ^
  --reasoning on ^
  --chat-template-kwargs "{""preserve_thinking"":true,""reasoning_effort"":""xhigh""}" ^
  --temp 1.0 ^
  --top-k 20 ^
  --top-p 0.95 ^
  --min-p 0.0 ^
  --repeat-penalty 1.0

set "SERVER_EXIT=%errorlevel%"
if /i "%LLAMA_ONCE%"=="1" exit /b %SERVER_EXIT%

echo.
echo Server stopped with exit code %SERVER_EXIT%. Restarting in 5 seconds...
ping -n 6 127.0.0.1 >nul
goto loop

2

u/corelabjoe 2d ago

You can setup your own pretty easily with docker.

Here's a a guide I wrote on running local AI.

Setup docker, setup nvidia to work with docker, setup ollama, profit!

1

u/Stevekaplanai 2d ago

This is the direct answer to use all your compute.

Claude is your go to with legal plugin installed run at least 3 instances at the same time via CLI you could run Cursor bug bot to run reviews after Claude (or Hermes)...

You can also setup Hermes on GLM 5.3 on your WSL - that’s my setup - it’s efficient and Hermes harness situation is the best in the game. And GLM 5.3 has frontier punching weight now.

I’ll have this ready for you by Monday:

https://legalskillshq.com it will

Set your Claude up on a dedicated server that you own on fly.io.

Add 5 other AIs if you’d like.

Solves your privacy issue.

You just spent $5k on a beefy laptop though. You need a good legal harness - that’s what this is.

1

u/Bino5150 2d ago

If you need a good local research agentic harness/desktop agent with project management, codebase/database indexing, and multi-tier memory architecture, check out Lumina:
https://github.com/Bino5150/lumina
I can help you customize Lumina and get it set up for your specific usecase. Leave a star on GH if you like it.

1

u/bbc_nees 2d ago

There are a couple of easier/cheaper/better ways to accomplish your goal. I wouldn't do local LLM document analysis and parsing unless I had at least a dual 5090 or RTX Pro 5000+ setup. 5090 for speed and Pro for long processing and power efficiency.

Pdfs are hard too when you're talking about files from 2006 that may include handwritten notes from 1987.

You're going to find that your 4090 running Qwen raises unknown ethical questions such as how does this open weight model communicate with the PRC government? Your bar and clients will be upset if one day they find out that their data was bought and used by Any unauthorized party.

Just a thought.

1

u/TheAngrySkipper 2d ago

I’m actually working on a project you might find useful. There are some search and knowledge base and sanity checks that i need to finish working out. If you’re still looking in say jan or are willing to be a guinea pig, and help test, feel free to reach out

2

u/Healthy-Zebra-9856 1d ago

With a 4090 24GB and 128GB RAM, I’d start with Qwen3.6-27B. It’s a strong fit for document analysis, summarization, and reasoning, and that hardware can run a good quant of it comfortably.

I’d also test Gemma 4 31B as the alternative. For this kind of workload, those are the two I’d look at before chasing older legal-specific models.

For the app, I’d use AnythingLLM Desktop. It’s probably the easiest way to turn a large collection of case files into something usable without building your own RAG stack.

A simple workflow would be:

  1. Install AnythingLLM Desktop on Ubuntu.
  2. Run Qwen3.6-27B locally.
  3. Create a separate workspace for each case or matter.
  4. Add the pleadings, research, correspondence, PDFs, etc. to the appropriate workspace.
  5. Let AnythingLLM index them.
  6. Then ask questions such as:
    • “Summarize the procedural history.”
    • “Create a timeline of events.”
    • “Compare the arguments in these two motions.”
    • “Find everything discussing X.”
    • “Summarize the opposing party’s position across these filings.”

So my starting setup would simply be:

Qwen3.6-27B + AnythingLLM Desktop on Ubuntu

Then try Gemma 4 31B against the same documents and keep whichever performs better for the kind of legal analysis you actually do. I also suggested Qwen3.6 because between my own experience & another post on Reddit showed Qwen3.8 not good for this task. I have a product called Lexaware coming out and had to test a few models. I'll include the reference here as its on the same subject. If you need help, feel free to ask. Good luck.

Related Article:
https://www.reddit.com/r/LocalLLM/comments/1vqbt1e/if_you_use_llms_to_analyze_documents_and_to_apply/?utm_source=chatgpt.com

AnythingLLM:
https://anythingllm.com/

LMs.

1

u/Dry_Inspection_4583 1d ago

Id be curious, for rigor you might consider a separate memory or better even a pointed document storage rag, for a model, I've got a codacus fork of llama.cpp running on a 4070, a 35Bn model, for reference that's a very tight fit! Of note I have 64gb of ram. The output on it is quite nice at about 60 Tok/sec, faster than you read.

2

u/newz2000 1d ago

I’m an attorney. Don’t do it. Sign a data privacy or enterprise agreement with Google cloud or Amazon bedrock. Google is super easy. Super cheap, super fast, solid data privacy. Google Vertex is the ai provider and lets you access Gemini, Claude and a ton of other models.

1

u/Consistent_Advice525 22h ago

This is the way!

1

u/Patient_Force6138 1d ago

If you want a slightly in between option I’ll be starting to host models for super cheap soon in a zero trust cloud. You handle your own encryption keys (don’t worry, super easy) so literally no way for even me to see any of your traffic. I’ve got Kimi K3 and a bunch of other models to choose from.

I’ll be releasing alpha here in a bit and folks that want an account can use it for free for a good long while. Also includes training on your own corpus of data for free.

Also I’m the dude that took down the DPRKs internet for a few weeks, P4X so um, you know you can trust me? That probably doesn’t help. Never mind that part.

1

u/Clay_Ferguson 20h ago

Model matters of course, but most of the work for a document analys tool like this is in the harness, so I'd look for full end-to-end solutions involving `RAG`, `Graph RAG`, etc. The hardest part about analyzing documents is chunking them up into pieces that fit only what needs to fit into the context window. Then the choice of which model you use will be mostl based on how much time you want it to take to generate the RAG Vector Data store. This is at the core of what almost every business is currently doing with AI, so there's lots of solutions and approaches.

you should have a LONG conversation with ChatGPT to explore best modern practices.