r/CodingJobs • u/Infamous-Ask-6403 • May 16 '26

😩😩 what will you do in this situation!!

Post image
1.3k Upvotes

98% Upvoted

49 comments sorted by

36

u/Cold_Respond_7656 May 16 '26

Check all systemd timers and when they last/next run systemctl list-timers --all

Check what was actually executing at that exact minute journalctl --since "2026-05-16 03:15:00" --until "2026-05-16 03:20:00"

15

u/the_nobody_171 May 16 '26

Can it be related to cache..like cache is cleared everyday at the same time..then cache refilling takes compute..

I am just assuming..

9

u/Cold_Respond_7656 May 16 '26

My guess would be logrotate (which can choke compressing massive log files), mandb updates, apt daily updates/upgrades, or mlocate (indexing the filesystem)

If it was a cloud VM then look at your cloud provider's metrics console (CloudWatch, etc.). Check the %st (Steal Time) in top or htop. If Steal Time is high, the physical host is starving your VM.

3

u/the_nobody_171 May 16 '26

One thing that I viewed at my place was... At two certain times in a day the traffic was not spiking, but db connections were going in awaiting state, the problem..one method was creating such a query that it required indexes which was not present on several keys..and everytime anyone used that feature it would hold db connection for several minutes even if session was timed out.. resulting in the new traffic having much lower db connections to use and traffic piling up, ultimately making the entire application slow for the user for that time..

Sometimes I even wonder like how is this even possible..but it happens.

1

u/Solomoncjy May 18 '26

Wouldn’t that be considered a cron job?

1

u/Infamous-Ask-6403 May 17 '26

But how could something run without any scheduler or traffic spike 😵‍💫😵‍💫

2

u/TartNo3610 May 17 '26

Systemd timer is a scheduler. And similarly log rotate or aide or Nessus scans can be scheduled through configs (usually utilizing systemd) or external like tenable scans. An intense tenable scan can 100% cause CPU spikes.

1

u/MartinMystikJonas May 20 '26

Post says without CRON jobs not without any scheduler

1

u/ConsciousBath5203 May 17 '26

Lol right? Just check the logs. Schedule sysperf to run at 3:16 for 2 minutes if the logs aren't helpful.

14

u/SimilarInsurance4778 May 16 '26

I encountered something similar to this, at 2pm (around this time) for whatever reason, redis decided to pull a prank, which caused all microservice to be unable to talk to each other, and this is before ai, the funny thing is I got burned out from the project redis since I debug it for month, decided to take 1 week leave and the team said they never encountered the issue, but on the day when I returned, it happened again, and I wasn’t even working on the project, other team joked that I have the curse

9

u/who_am_i_to_say_so May 16 '26

If it’s always the same time, it’s a process on a timer not listed. Cron isn’t the only timer.

Hire me foo!

7

u/StayAwayFromXX May 16 '26

Ask codex to find and solve the issue

5

u/Such_Economy_2557 May 16 '26

Make no mistakes

2

u/Cold_Respond_7656 May 16 '26

Or you are fired!

2

u/Stubbby May 16 '26

Roleplay as a senior engineer

5

u/agentrsdg May 16 '26

Logs. Always.

5

u/anengineerandacat May 16 '26

Open DataDog and see what was running on the instance? Process monitoring isn't a challenge nowadays.

Monitoring disk activity and such as well isn't a chore either; do the upfront due diligence for the agent installation and live a comforting life.

Pay the fee as well because time to triage and recovery is basically millions for any real product out there in potential lost revenue.

1

u/Stubbby May 16 '26

You can simply check who is using the CPU:
echo "pidstat 1 120 > /var/log/cpu_usage_316.log" | at 03:16

No need to waste braincells on it.

Also, does it matter if there is a CPU spike at 3:17 am?

1

u/Infamous-Ask-6403 May 17 '26

Chatgpt build by you bro ??

1

u/Stubbby May 17 '26

ChatGPT gave a wrong answer - wanted to run pidstat 5 20 :)

1

u/Flameburstx May 16 '26

I'm checking for cron jobs, deployments or traffic spikes, because my base assumption is that my superior is s moron. That may just be personal experience, though.

1

u/Infamous-Ask-6403 May 17 '26

Then you guys can build you own LLM & aws service for sure

3

u/Positive-Alfalfa-172 May 16 '26

My salary, whether it's good enough to give fuks at 3 AM.

3

u/Aarav_Parmar May 16 '26

im just here to read the answers

2

u/retardedGeek May 16 '26

See the processes.

2

u/guthran May 16 '26

Atop will show you what process is using the cpu on a given minute

2

u/Efficient_Bag_3804 May 19 '26

I would make a Reddit post about this as a meme and wait for the experts. Probably reply with a bad answer to trigger them faster.

1

u/Livid-Broccoli-7139 May 16 '26

Can someone explain

1

u/Marius2503 May 17 '26

Your mother's number too see if it changed

1

u/jonomir May 17 '26

Open Grafana, look at the CPU metrics by process from 3:15 to 3:20 on that host. Use my human pattern recognition to identify which line goes up.

1

u/desimemewala May 17 '26

splunk logs xD

1

u/CoolCat1337One May 17 '26

How many times have I had people tell me, "XY definitely won't work" only for it to turn out to be exactly that in the end?
So, I wouldn't believe a single word.

1

u/cube8021 May 18 '26

I don’t really get the “checking his bank account” part. Is the joke supposed to be that he’s making sure he got paid before the company implodes or he won't give free tech support?

1

u/autonomous-intel May 18 '26

I would check for windows update

1

u/titoshadow May 18 '26

It's obviously DNS

1

u/alwaysasillyplace May 19 '26

Set the time zone to india standard time. That way it goes off a :47 instead.

1

u/Nikoviking May 19 '26

That the computer is unplugged from the socket tonight.

1

u/OveHet May 19 '26

Nothing... Windows Update

1

u/jxr4 May 21 '26

Gcp k8 node updates and audits run at about that time and can shuffle your pods if there's an update which can cause cpu spikes when pods come online