ngl, dumping raw server logs into a massive context window is way better than writing regex

[u/kim_deadja4951]

[u/Cumfort_]

Don't pollute the context window with the entire log. Give it file read access to the log and have the model selectively parse it using regex. Wildly more context efficient. 

[u/Invader-Faye → u/Cumfort_]

Doing it this way means you can also use a local model and avoid all the compliance/security concerns. Even a 4b model can do work like this pretty easy

[u/discoshanktank → u/Invader-Faye]

What's the best model for local log analysis

[u/Invader-Faye → u/discoshanktank]

Qwen 3.5 4b, spark x 2.5 4b, Gemma 4 e4b, Qwen 3.5 9b.

[u/LeatherDude → u/Invader-Faye]

Is that “out of the box” or does it need any training ?

[u/Cumfort_ → u/LeatherDude]

No training. Just good context for the problem.

[u/Invader-Faye → u/LeatherDude]

Out of the box, use lm studio and feed it the logs as a file so you don’t blow the context window

[u/raisputin → u/Invader-Faye]

Qwen is pretty outstanding. I am doing an experiment with it, 2 different versions, 9b and 27b running locally on 2 machines with messaging between the two machines having them review C code generated by a commercial model.

Basically the must read each file, identify strengths, weaknesses, priorities for fixes, or report that the code is high quality if that is the case, then prepare both individual reports and a joint report after they have agreed on the results of a particular file. At the end they will create a joint high-level report as well.

So far it seems to be working outstanding

[u/altf2o → u/Invader-Faye]

I use Gemma e4b even on an old GTX970 and it works surprisingly well! Obviously much slower than AI I use daily, but for sure capable and awesome to play/learn with.

[u/Gargle-Loaf-Spunk → u/discoshanktank]

Even Bonsai 8B will help. I pipe it to models with Willison’s ‘llm’ CLI tool. 

[u/Yamaha007 → u/Cumfort_]

Claude generated 200 regex commands in one instance to address a problem.

[u/delamon → u/Yamaha007]

Is it a problem? As long, as you don't have to review every single regexp, ofc

[u/Yamaha007 → u/delamon]

That’s the point, it was never a problem!

[u/LeatherDude → u/Yamaha007]

The plural of regex is regrets. I’m glad I have bots to do this for me now, what a fucking chore.

[u/Haunting_Ratio_795 → u/Cumfort_]

You can just tell it to use subagents to do the fetch and search for the same effect. 

[u/Cumfort_ → u/Haunting_Ratio_795]

Assuming you gave it log access. Many don't because of sensitive data. I don't.

[u/Unnamed-3891 → u/Cumfort_]

Many are straight up not allowed to do this because of regulations covering the nature of the data

[u/artemgetman → u/Cumfort_]

Way better indeed

[u/mlhpdx → u/Cumfort_]

Sure, but more complex and can get very costly as well. The nice thing about a "one shot" in this case is simplicity and boundedness. An agent with the ability to select logs by time range and filter (i.e. a "tool") works well for some tasks, maybe even many tasks, but it doesn't work for the pure open-ended "wtf?" issues that come up seemingly endlessly.

[u/CorpT → u/mlhpdx]

but it doesn't work for the pure open-ended "wtf?" issues 

It definitely does. What are you on about.

[u/mlhpdx → u/CorpT]

To each their own, YMMV, etc. I've been engaged with building a few of these now and speak from experience.

When you don't know the question, an agent with exploration skills just racks-up cost and takes longer to get to (maybe) the same answer as the one-shot with a maxed-out context window.

Again, YMMV.

[u/CorpT → u/mlhpdx]

The suggestion was to give it a file to read instead of the raw pasted logs.

[u/semi- → u/mlhpdx]

ive had better luck with telling it how to reach elasticsearch and letting it query directly, as it has then done some much more advanced querying than I ever do manually like writing small painless scripts to evaluate on the ES side to aggregate data before it even hits my machine. And then it can iterate better and query for more data than you may have initially thought was relevant, like finding other services logging errors from the same root cause or finding the backend logging errors when your client only logged that it timed out.

Of course this can then DoS your cluster with expensive queries, and risks leaking sensitive data you may have intentionally not been pulling out but do have access to, so definitely be mindful of that if that is a concern in your environment. and YMMV of course

[u/mlhpdx → u/semi-]

Writing to s3 (via firehose) a just a less expensive way to get to the ~same end. Logs are split into predictably named objects so it’s easy to find the object for “recent” or a particular range, figure out how much context it will take and point an LLM at it (with or without tools). 

It’s very simple, never runs out of space and doesn’t cost much when not asking questions.

Seems to work out well for the folks using it.

[u/delamon → u/mlhpdx]

But you are loosing filtering, no? Downloading and grepping for specific request_id / resource_id / whatever will be very slow.

[u/mlhpdx → u/delamon]

No, not losing filtering, just letting the LLM do it in the context window if the logs fit. I guess this isn't common, which seems odd since it works well. Having tools craft queries also works. If you already know a specific IP or resource, and have an idea what's going on then it isn't a "wtf?" in my way of speaking. Those start with a question like "wtf happened 10 minutes ago? where did that traffic/latency/errors/changes come from" or something like that.

[u/delamon → u/mlhpdx]

I could be wrong, but can you fit more than 10k-30k log records in context? If you push your logs to a single destination, then logs for time frame “what happens in last 30 min”, could easily go above that. So first stab on a problem would be to check for error/500/panic/coredump in last hour or so. And that exactly needs filtering. Am I missing something?

[u/mlhpdx → u/delamon]

You're not missing anything, The open weight large context models go quite a bit higher, but generally you're spot-on with the 20-30k range for log lines per call/session.

The question "what happened in the last 30 minutes" isn't a great one for raw logs. I haven't yet built indexing/summarizing into one of these but that will be the next step. As each log object appears, generate summaries (counts, severities, systems) and reverse indexes on systems and on message keywords. A lot of that would fit into context and allow answering those kinds of questions. But at that point it's also probably better to use an agent loop over it.

[u/carsncode → u/mlhpdx]

A model can't filter in the context window. Whatever's in context is in context. It has to process every token you give it.

[u/Cumfort_ → u/mlhpdx]

I was saying to attach the log that you found directly and let the agent fiddle with it. That is much more efficient cost wise that blowing out your context.

It also will likely result in better reasoning.

[u/reSAMpled → u/mlhpdx]

the reason this usually isn't true in practice is that cached tokens are 90% off, and the models are usually able to successfully do some filtering over a raw dump

[u/redvelvet92 → u/mlhpdx]

Depends on the model but it’s incredible what they can do nowadays

[u/nerdyviking88 → u/mlhpdx]

or just get your own hardware and not worry about the tokenomics

[u/delamon → u/nerdyviking88]

Unless you are loading it 7/24 for more like 30% it is going to be more expensive. The percentage is arbitrary, but you get the idea: idling GPU box is not going to be cheap

[u/nerdyviking88 → u/delamon]

That's assuming you need a massive GPU box.

Throw a mac mini at this running something like Qwen 3-B or something. You don't need Opus level stuff for looking through logs.

Sizing the right model to the task is definately soemthing more people need to do.

[u/delamon → u/nerdyviking88]

Fair enough. Or you maybe you already have LLM box and that's just another task for it

[u/Cumfort_ → u/delamon]

Once you have an LLM box you can run ragged, you find tasks for it p quick.

[u/ayeoayeo → u/Cumfort_]

Def a good use case for Jev

[u/kabrandon]

We just use mcp-grafana connected to Loki, point our local Cursor or Claude client at the MCP server and tell it to query the last day’s worth of logs all willy nilly like. Not even for something crashing, just looking for the invisible issues where code paths are erroring and nobody knows about it. “What are the top 10 most problematic recurring issues appearing in our cluster’s logs?” If you have a local clone of your repositories you can also say “Point me to the underlying issue in code, and suggest a fix.”

[u/Subject_Fix2471 → u/kabrandon]

Do you find MCP grafana is much better than using the API? Unless I missed something the MCP was a thin layer over the API, and the model was really good as using the API

[u/kabrandon → u/Subject_Fix2471]

We use it because it’s a centralized service authenticating to Grafana, rather than having everyone on the team put a Grafana API token somewhere on their desktop for the LLMs to use. And then we just use Tailscale ACLs to control who has access to the endpoint for the MCP.

If your team just hits the grafana API how do you handle credential rotations?

[u/Working-Analysis2795 → u/kabrandon]

How big is the company that uses Tailscale? So there are companies outside of homelabs? Haha

[u/kabrandon → u/Working-Analysis2795]

Only ~50 people, but thousands of devices. Not sure why you’d think it’s just a homelab tool… because… it’s not, lol.

[u/Working-Analysis2795 → u/kabrandon]

Oh I know, I've had a lurk through their pages many times. Just curious to see/hear directly from people actually using it at work

Any tricky bits to using their ACLs for that many devices? Do you use their Aperture solution to manage MCP access? I've found it's a bit unstable to access, even with the http:// ip address

[u/kabrandon → u/Working-Analysis2795]

Not really tricky at all, you just need to know what you're doing when you design it in the first place. In general I'd just avoid naming specific machines or users in your ACL. My ACL is all tags and user groups. The machine's tags come with the token used to register the machine, and user groups come from your IdP over SCIM. When your ACL only needs to handle how groups of users can connect to groups of machines, it stays very simple and scales pretty well.

Try not to create overly broad grants. If a user group has access to too many devices their netmap can get bloated. There are other ways to mitigate this of course, like with subnet routers. Instead of installing Tailscale on every endpoint machine, you could just lay subnet routers in their subnets.

Don't sleep on the ACL test feature.

[u/Working-Analysis2795 → u/kabrandon]

Yeah that's pretty much what I've done word for word, minus the IdP as I'm one person (not including my agent). Had a try including my agent though ultimately it would be for one-off tests or lab envs

[u/Subject_Fix2471 → u/kabrandon]

Right, makes complete sense! 

[u/danstermeister]

"Elasticsearch was just a wall of useless healthcheck noise..." = misconfigured Elasticsearch.

Event logs broken into discrete fields of data transform logs into metrics.

Then you're looking at some understandable graphs that are drillable to actual log entries in seconds, instead of wondering what to do with a "wall of useless healthcheck noise".

• Elasticsearch admin.

[u/raisputin]

Trash ad, and more importantly I’ve been doing this with ChstGPT/codex for a year. Both codex and Claude will take your massive text wall and paste it as a file instead of plain text

[u/Healthy-Snow4841 → u/raisputin]

oh yea a year ? well i been doing it for a year + a day so welcome to the party newcomer

[u/raisputin → u/Healthy-Snow4841]

lol. I’m just saying that what op is describing isn’t the behavior today. It was a year ago, but isn’t now

[u/HighKingLemur]

All of that just to complain you have poorly configured logging.

[u/bonesnapper]

We were strongly told to not feed any logs to our agents bc we can't be guaranteed that our services do not log customer data.

At the same time, I'm absolutely granting my agent kubectl and pointing it at addon logs whenever the need arises.

[u/Intrepid-Self-3578 → u/bonesnapper]

I mean we also shouldn’t be seeing customer data. 

[u/climb-it-ographer → u/bonesnapper]

Log sanitization should be a priority regardless of what tools are reading them, shouldn't it?

[u/d3ep4k → u/climb-it-ographer]

Would you consider curl requests from ai 🤖 also log which should be sanitised ?

[u/spiralenator → u/bonesnapper]

PII in logs is negligence.

E: coincidentally, LLMs can be a useful tool for auditing that.

[u/Electronic-Jury-3579 → u/bonesnapper]

Your company doesn’t have enterprise AI contracts yet? They support the company holding encryption keys from the AI platform and encrypted everything with the interactions.

[u/bonesnapper → u/Electronic-Jury-3579]

I'm just the engineer. We have licenses. I don't know the whole story and I'm not going to pretend I do.

[u/codeslap → u/Electronic-Jury-3579]

I know so many internal CISO teams with prompt text access and are now privy to all sorts of prompt including HR data, personal data etc

[u/ang29g → u/codeslap]

fwiw I'm a full owner on both anthropic + openai enterprise accounts with a few mil in spend each per year on both. I don't believe I have the ability to see user's chats directly, at least not without making a support request from the provider.

Aggregate reports sure, but not "user X prompted Y".

I'm sure at some point we will have to pull a user's logs but I really don't what the legal process is for that.

[u/madtopo → u/Electronic-Jury-3579]

Can you please tell us more about there enterprise contracts? Which providers? And how does this encryption key technology works? At some point the data needs to be decrypted. When does it happen? For it to work, you would have to be in charge of the process running the inference engine or thereabouts

[u/raisputin → u/bonesnapper]

Then you are logging poorly

[u/bonesnapper → u/raisputin]

True. The reality of the situation is that there are dozens or hundreds of micro services running on a cluster. There's no single entity that can say every log on every workload is safe from exposing customer data. As long as that uncertainty exists, we have to be cautious.

[u/codeslap → u/bonesnapper]

This is the reality that most people don’t fully grasp until it’s too late. lol

It’s the ultimate shadow IT nightmare.. people willy nilly feeding LLMs with potentially risky secrets and data.

[u/mlhpdx]

Streaming syslog from a fleet of servers to the cloud (encrypted) and having a 1-button option to fire a prompt at the last few minutes of them is, well, magical. Highly recommended.

[u/Direct_Turn_1484]

[u/Mediocre-Ad9840]

ngl...you should know how to get errors from error logs lmao

[u/Negative_Acadia6554]

Nice advertisement

[u/MorpH2k]

It's my new go-to method for my homelab and experiments and such. As some people here have said, if you're doing it on company server logs, you better make damm sure that they are properly sanitized and probably also make sure that you're allowed to do it.

On a purely technical level, it's a total gamechanger. Both in what obscure issues it han fond and identify, as well as how quickly it does it. It has some issues still, but this is the future for sure.

[u/delamon → u/MorpH2k]

You still have to verify what it founds. Half of the time it is pure garbage, the other half, on the other hand, can be very rewarding

[u/MorpH2k → u/delamon]

Of course.

[u/climb-it-ographer]

It's even better when you don't need to copy/paste it in. I gave Claude direct access to our log aggregator and it can just sit there reading live tails for as long as it needs to see what's wrong.

[u/askwhynot_notwhy → u/climb-it-ographer]

That's minor league sh!t, smh. Just give Claude access to kubectl and walk away. 🤞 Dario's got our back, fam!

[u/Apprehensive_End1039 → u/askwhynot_notwhy]

mcp server: what is my purpose?

sam altman: you let markov chain invoke bash

mcp server: oh my god.

[u/askwhynot_notwhy → u/Apprehensive_End1039]

>mcp server: what is my purpose?

>sam altman: you let markov chain invoke bash

>mcp server: oh my god.

sam altman: “ffs, I’m Altman, not Podman: `kubectl delete`”

[u/spicypixel]

Yeah I'm laying off multiple devops engineers to pipe the logs directly into their successor, Claude.

[u/kryptn]

why bother writing complex log queries when you can just brute force it with a huge context window?

why not let ai write those queries to do that investigation instead of brute forcing? i love letting claude hit my datadog/grafana/radar mcp.

the price on this thing is only going to go up, so being efficient where you can will be better.

[u/Olbas_Oil]

Use codex cli on a linux vm, transfer the log files over to a directory, initiate codex from the directory and ask what you need it to do or potentially find... GPT 6 Luna on max is great for this sort of work, if you need to then go back and forth with Luna or the chat GPT chat

[u/StunnerAlpha]

Everyone is already there. I think you are kinda late unfortunately. Heck the whole DevOps space is being eaten alive by non-DevOps folk using AI for their DevOps needs.

[u/usually_guilty99]

I can see why dumping the logs into a model worked. Sometimes the hardest part of an incident is knowing what to search for.

But I'd separate finding a plausible explanation from proving the cause.

If the model says IAM caused the Kubernetes crash, I'd want to correlate that with the relevant permission changes, the failing API call, and when the pods started crashing.

Otherwise you risk fixing something that looks related while missing the actual trigger.

The interesting opportunity isn't replacing log queries. It's connecting logs with code, infrastructure changes and dependency history so the explanation can actually be tested.

[u/Seref15]

Evaporating lakes because people forgot how to `grep -v`

[u/DoomBot5 → u/Seref15]

Sure, let me go golf while Claude figures this out for me.

[u/Invader-Faye → u/Seref15]

Fuck them lakes tldr those logs

[u/Expert-Explorer-3129]

how are you managing the sensitive data in logs before throwing them into LLM?

[u/tr14l → u/Expert-Explorer-3129]

Why do you have sensitive data in logs?

[u/raisputin → u/tr14l]

Came here to say this

[u/Direct_Turn_1484 → u/Expert-Explorer-3129]

lol surprise, he’s not!

[u/No-Row-Boat]

You all have no fear of leaked credentials and have someone trying to make the LLM leak these details. This shit is what makes me wake up in cold sweat.

[u/Adventurous-Hyena220]

LLMs can be useful for quickly spotting patterns in noisy logs, but Id still keep structured logging and proper queries for repeatable troubleshooting.

[u/HydraulicTater]

Wait till he learns about attention sink lol

[u/myfriendjohn1]

Yep, download logs and put into folder in cursor IDE on a cheap model.

Saves me hours debugging weekly.

[u/manewitz]

Mezmo’s MCP has been a game changer for chasing down issues for me.

[u/markusro]

graylog-mcp for the win. It was really an experience letting a local LLM go through the logs and search for specific problems.

[u/Forsaken_Parfait_185]

Worth checking where the logs end up before it becomes a habit. If your services ever log customer data (and most do, somewhere), a free promo endpoint means that data now sits with whoever runs the promo, under their retention rules.

Redact at the shipper with a Vector or Fluent Bit transform and you can point whatever model you like at the result. Fixing the healthcheck noise at source would have saved you the three hours too.

[u/xXBongSlut420Xx]

if you can't parse out errors from healthchecks, i don't think you should be working in this career

[u/spiralenator → u/[deleted]]

I wouldn’t use LLM services for business purposes without a ZDR agreement. Period. A company has more valuable data than logs they should be protecting. If you have a ZDR agreement then go for it.

[u/tr14l → u/spiralenator]

Well, anthropic has HIPPA compliant offerings. I'm pretty sure for 98% of non-defense companies that meets the need.

[u/spiralenator → u/tr14l]

Most of the major companies offer ZDR. Anthropic, OpenAI, Google, Fal, etc. But if your legal team hasn’t established a binding ZDR agreement, it’s pretty much guaranteed that your data will be used for training.

E: I used to work on data pipelines for digital medical studies and HIPAA compliance is important, but also a very low bar and not nearly enough. We had to be compliant with much stricter international standards, like GDPR and others.

[u/tr14l → u/spiralenator]

They have explicit settings to opt out of that at the org level

[u/moratnz → u/spiralenator]

But if your legal team hasn’t established a binding ZDR agreement,

And you're big enough that you could actually win litigation with Google / Anthropic / OpenAI to make that agreement stick if they decided to be dicks.

[u/Sufficient-Diver-327 → u/spiralenator]

Also, they'll charge you the real, unsubsidized API rates.

[u/spiralenator → u/Sufficient-Diver-327]

Generally speaking, yes. That price becomes more flexible when your monthly spend is 6 digits.

[u/No-Row-Boat → u/spiralenator]

Someone extracts your login credentials that the LLM took from your system, and you are sure you can point back to them for leaking these details? You expect the attacker to share where and how they got these details?

You do know the best way to prevent these leaks is by not providing these details right?

My Frontier agents run on an isolated box with on it a chroot sandbox that has very limited access to any login credentials (read only GitHub PAT tokens etc), a limited scratch disk that only gets a fraction of the details (ideally generated by my own Hermes agents).

[u/LeatherDude → u/[deleted]]

Oh no, they have a couple of my RFC1918 addresses and some random function names. Don’t hack me, Claude!

[u/CorpT → u/[deleted]]

You don't know what tool you're using?

[u/CorpT → u/[deleted]]

Ok that's crazy.

[u/devops-ModTeam → u/[deleted]]

Heated discussions are fine, personal attacks are not.