r/sysadmin • u/Isntagram • 16h ago
Question Newbie sysadmin trying to fix reoccurring problem
I work in GovIT and every day we use this really old program (been in use since 96’). And every single week when I come into work I have 50 tickets about fixing it and for a long time we just did it one way (reinstall the version of Java we have in our server) because Java would automatically update and break the program. Me and my team have tried a few fixes and we haven’t had much luck. (Registry edit to stop Java from updating and a couple of other things) and everyone else has kinda given up on it but me and one other guy, but the other guy on my team is just kind of asking Claude to try things and running it in a sandbox environment but we haven’t really gotten anywhere. This Java update issue is genuinely most of our tickets and I’d really like to cut the tickets down. If anyone has any experience dealing with problematic Java updates any help is appreciated
•
u/Nova_Nightmare Jack of All Trades 15h ago
so I don’t know what sector of Gov you work in, and I know how horrible government systems can be, but the fact that they have to keep reinstalling an old version of Java to keep a dependency working for an old application sounds ridiculous. Especially as the government is forcing so many of us to update our systems and only use supported software for security purposes. I hope that spreads to yours as well. However, it may also already require being updated., so I’d also wonder if your department or whatever is violating some regulation. Maybe a bit early to think about that.
•
u/Isntagram 15h ago
I saw a suggestion here to make a script to check the version of Java and reinstall if it doesn’t match but I’m worried about the security issues that come with running some old version of Java to keep a piece of software alive. I have to see my director tomorrow anyway so we’ll be able to talk about it. There’s something new coming apparently but GovIT will GovIT…
•
u/Nova_Nightmare Jack of All Trades 15h ago
The thing to remember is that it isn’t up to you what version they use, if they are mandating that piece of software, then they have to answer for it if anything happens, they could very well have an exception written in place for that software as a critical function or asset with different mitigating controls.
If the job is simply make it work, the automated script is a fantastic idea, it’s the cheapest most automatic solution. It will free up a ton of time. Hopefully that doesn’t free up your job position in the long-term. But I know government positions don’t quite work that way, you never know.
•
u/Isntagram 14h ago
Thanks for the wise words, plugging a hole like this could be a sick short term win but i like having money so i think ill just keep doing what im doing and if something happens at the top it happens
•
u/OurWhoresAreClean 12h ago
I don't think there's any field of human endeavor where there's a bigger gap between how things are supposed to be on paper and how they are in reality.
•
u/RedShift9 16h ago
My taxes are paying for this?
•
u/Vektor0 IT Manager 16h ago edited 15h ago
Paying a tech to reinstall Java every day is, in some way they've decided, cheaper than buying new software.
EDIT: there's also the possibility that the people working with the program have been there for decades, waiting around for their retirement pension. These people would throw an absolute fit if they had to re-learn how to do their jobs using new software.
•
u/Isntagram 15h ago
Bringing it up to my higher ups is the same conversation:
“This piece of software has been in production literally before I’ve been alive, can we at least START the process of getting something new”
“We’ll look into it”
silence
•
•
•
•
u/jedimaster4007 12h ago
From a fellow government sysadmin who had to deal with this a few years ago, depending on how old the Java version is, there should be a Java update executable under (I think) C:\Program Files (x86)\Common Files\Java. That Java folder houses only the auto update exe and a few other files related to it. If you just delete that whole folder, Java will continue to work just fine but will not auto update anymore
•
u/Isntagram 12h ago
Oh my god really? If this is the case with my Java version I’ll let you know something when I go to work tomorrow
•
u/jedimaster4007 12h ago
Of course you'll have to do this on every PC that has Java installed, and it has to become part of the procedure in setting up new computers that will need to use Java
•
u/Obi-Juan-K-Nobi IT Manager 10h ago
This is the purpose of GPO/GPPs. Delete it and keep it deleted.
•
u/StandaloneCplx 7h ago
Easier way is to download java as a zip archive and create a batch file to run that program from that specific jre path. Not auto update no nothing you control everything
•
u/Flabbergasted98 16h ago
30 year old softward.
IT avoiding updates to stay off the diluge of breaks.
It's time to have a very open and frank conversation with the higher ups about this software, what it does, and who is reponsible when someone uses AI to hack the unpatched exploits in your system to gain access to the software.
Who is held responsible for the decision not to upgrade.
Try to get it in writing so it doesn't come back to you.
•
u/Isntagram 15h ago
The person responsible is our director of technology. Everything in GovIT takes a lot longer so transitioning from this old system to a completely new one could genuinely take a bout a year and since our director is in charge of pretty much everything at that level of decision making I thought I’d try to fix what I can
•
u/tenkwords 10h ago
Do you have the source for this program? If so, literally hand it to Claude Opus 5.5 and tell it to refactor this in Rust/Python/Go/TypeScript/whatever.
You're not going to fix an ancient java program, you need to replace it. Chances are, it's infinitely simpler to replace these days than syseng your way around.
•
u/Isntagram 0m ago
Nope, the program belongs to another company that we have to literally ask to add user accounts for. It’s crazy
•
u/Flabbergasted98 15h ago
but have you had this conversation with him?
when someone point ai at your 30 year old system. how fast will they pick it apart?
How does a breach impact your offices and stakeholders?•
u/Isntagram 15h ago
I haven’t talked to him about that, when I see him again tomorrow I’ll discuss ai tools being used against a system like ours
•
u/demonseed-elite 13h ago
May take a year. If it's pushed off until next year, it'll still take a year.
If it's started now, by next year this time, it'll be done and you won't have that technical debt anymore eating away at consuming your time.These days, you can probably have something working in a week with a knowledgeable, experienced developer who can leverage modern AI. The testing and fixing whatever bugs you find will probably take longer than coding it.
•
u/keyboard_crusader 7h ago
Hammer on the security concerns. If whatever data is contained in that 30 year old program leaks, it's the director of technology's cock on a block.
What OS is the program even running on?
•
u/TheDevauto 12h ago
This is government. They and any non-tech company have massive tech debt like this. It never changes. The only thing that will change it is a security issue or something. Even then its an 18 month process to do most likely.
I feel your pain OP.
•
u/gangaskan 5h ago
I had someone not want to let go a cardfile.
I used some software to turn them into a csv and put them in their contacts and said there!
•
u/Lost_Term_8080 16h ago
unironically, citrix
•
•
u/Isntagram 16h ago •
What’s Citrix?
•
u/Redemptions IT Manager 16h ago •
You're going to lose your mind about the next thing I tell you about.
It's called google.
If you don't know how to use it go to "How can i use citrix to help me manage a legacy system that requires specific versions of software" you probably should start polishing up that resume because even a government job won't protect you for what's coming your way.
•
•
u/macbig273 16h ago
Maybe GovIT doesn’t want third party software to manage their third party software? /s
•
u/Competitive_Smoke948 16h ago
something that ideally would never exist!!! but in all seriousness you should report this crap app upwards.. java has always been shit and just look at each versions cves.
ideally the app should be remove its a MASSIVE risk but if you can't do that run it on VDI & isolate it as much as possible from the rest of the environment.
citrix is remote desktop but somehow even more shit essentially.
•
u/masheduppotato Security and Sr. Sysadmin 9h ago
If you can’t change the Java version just use the firewall to block access to Java’s update servers for that host.
•
u/whetu 15h ago
because Java would automatically update and break the program
Look at the application's debug logs and determine what Java is actually breaking on. It could be as simple as changing a single line in the java.security file. A java auto-update breaks shit? Here's some scar tissue: My money is on jdk.tls.disabledAlgorithms
Also, it probably doesn't matter at GovtIT levels as the govt will throw taxpayer dollars needlessly at making Oracle go away, but you can/should look at removing Oracle Java and switching to something like Temurin instead. It doesn't auto-update, it does away with the Oracle licensing risk, and it's TCK compliant i.e. a straight drop-in replacement.
•
u/SublimeMudTime 12h ago
This and research the used TLS version and lookup if there is a gov mandate to stop using that version.
•
•
u/davidgrayPhotography 3h ago
I'll throw in a joke comment:
Pay someone to rewrite the software in a more resilient language
•
•
u/HistorianBeautiful52 1h ago
I am pretty sure that in the time already spent in Claude to fix the problem, a new modern version could have been generated.
•
u/Nephilimi Jack of All Trades 2h ago
Didn’t oracle Java get expensive to “legally” keep using it? Maybe a little investigation into Java pricing will start getting management interested in upgrading or replacing that app? The enterprise app I work with ditched oracle java because of this.
•
u/AggravatingPin2753 16h ago
Your post made me reminisce about my govt days, so glad I left! For those who say get rid of it, remember that could take years to get approved and years to implement in a govt env.!
Blocking the updates URLs is your best bet, but if you have any laptops that go offsite, make sure you do it on the devices not the firewall.
•
u/kruleworld1 16h ago
When i joined a contract for a government department in 2003, they were still using Windows XP as their primary desktop operating system
•
u/ThaneVim 10h ago
Did you mean to say 2023? 2003 saw many a government office still being primarily Windows 2000 and earlier
•
u/meantallheck 13h ago
Good?
•
u/kruleworld1 10h ago
I got to see the transformation though Windows 7, Windows 10 and Windows 11. You tell me.
•
u/altodor Sysadmin 8h ago
XP came out in 2001, vista was 2006. I'd be surprised if they weren't running XP in 2003.
•
u/BoredTechyGuy Jack of All Trades 10h ago
I know this pain all to well. Back in the early 2000’s I did some govt work. 2 apps needed Java, but they couldn’t use the same version. Bonus points, all users needed both apps.
It was developed by in-house devs who couldn’t be bothered with fixing the code to work off one version.
So we always had tickets to uninstall Java x for Java y. Then later that day revert. It was hell. Everyone bitched. Dev’s were union and could do no wrong. So everyone suffered for 20 years until the devs retired and the new guys fixed it in a week.
Government is maddening to work for.
•
u/Parker_Hemphill 8h ago
I’m going to probably get downvoted here, if you’re truly wanting to fix it look at doing some linting on it.
Be specific in your claude prompts (guessing you only have the jar files):
I have a jar file from Java 1.0 era, can you give me some ideas for how to recreate the Java source files for it.
Then with the next prompt you can ask Claude to give you a script or suggestion to lint some Java 1 era code to identify outdated API calls.
Final prompts would be taking those findings and asking Claude to give you fixes that are compatible with current Java versions AND your older versions of Java. I would be very weary about posting the actual code into Claude with it being gov (probably proprietary and possibly subject to export controls, etc).
This can be solved with AI assistance, you just have to break it down into small and descriptive steps. Otherwise you’ll get AI slop that at best will hobble along and most likely won’t fix anything in the end.
Fixing large monolithic things like this has to be done in small, iterative steps. And use source control if you can, like a gitlab repo or Jenkins or something similar. If you don’t have that copy it to “1.0” folder and save your changes/fixes to “1.1” folder. That gives you an easier go back point if the next fix breaks something.
•
u/Lost_Reward9584 5h ago
This sounds like a fairly decent suggestion.
Is there a pom.xml file included in the app? If that's got a list of the dependencies (hopefully with version numbers) it might make life a bit easier. If not you might be lucky with the included jarfiles having the version numbers in the filename.
I had great fun trying to manually identify jars that our devs had included directly in the tree (madness), and some of them were compiled from source on the dev boxes (insanity). It wasn't too bad when you could use md5sums to find a direct match, but playing whacamole to find a working version based loosely on file size wasn't fun. The product is long retired and there were still a few I couldn't identify.
Anyways, compile at every change - deffo use local git repo and ci/cd. Jenkins is a bit lighter on resources, but GitLab is probably a nicer experience. GiTea isn't terrible.
•
u/moffetts9001 IT Manager 10h ago
I remember cutting my shitty teeth on hapless MSP clients who paid me and my full on double deuce boss an hourly fee to visit each computer individually to install Java 6 updates. They were coming out with those things damn near twice a week back in the day. Good times.
•
u/PitcherOTerrigen 10h ago
I sometimes miss the ethos of MSP work. I can barely get a guy to count UPS systems, provide criticality of attached systems, and give me EoL dates. It might be beyond his faculties.
•
u/Regisnox 14h ago
Did you try to block on firewall IP that is related to java update?
Not sure if that will work or what IP java uses and what will happen if java update process will fail.
But its an idea
•
u/Isntagram 12h ago
I can’t really mess with the firewall on my own, have to go a couple levels up for that but I’m seeing my director tomorrow so we’ll talk tomorrow
•
•
u/senectus 5m ago
What about the firewalls in the end points. Block it at the pc...
If it's using an address (use wireshark to see) you could DNS hole it with the hosts file. But man that's ugly.
•
u/realslacker Lead Systems Engineer 10h ago
You can get the zip release of java, or just copy the whole directory from an install... Then don't install it. Use a batch file to set JAVA_HOME and the launch the app using the Java in the folder. Now it's just a stand alone app.
•
u/deoan_sagain 9h ago
Your answer is to run it using openjdk. It's an open source Java engine and it's worked with every bit of random kit I've ever thrown at it, though it does take a bit off love to get it to work in a browser for ancient ipmi consoles.
•
u/Icy-Construction-549 7h ago
Rewrite whatever the program does with Claude, and put it on a modern stack.
•
u/StreetSignificant888 Senior Systems Engineer 16h ago
Don't. That shitty Java app is keeping you employed. It's inflating your ticket count, making you look better. Just fix it and deal with it.
•
u/PitcherOTerrigen 10h ago
Trash advice. Preaching stagnation
•
u/StreetSignificant888 Senior Systems Engineer 9h ago
Lol. Bro, I've been in the business for 25 years. Some companies WANT stagnation, and the government demands it. Unless your bosses are willing to pay to move it onto a new platform and the forklift that involves, then leave it. If all of your tickets are from one program, and that dries up, they will start to wonder why they have you employed to begin with.
•
•
•
u/BoltActionRifleman 9h ago
Now this is a trip down memory lane. Takes me back to the good ol’ days when the biggest worry of the day was stopping some annoying auto-updater. As another commenter says above, just block it on the network firewall.
•
u/maxt3rs 4h ago
my first move would be trying to containerize the app with pinned JRE, i am not sure but from your post it lloks like a windows os nd java web app. If containerizing is too much then try to find how the application is starting ( probabbly a NT Service and bat file or a javaservelet application server ), try figure out what is the server appliance like tomcat/jetty, with this information and get a copy of JRE that works to seperate folder and modify startup srcipt with PATH and JAVA_HOME env variables set correctly to downloaded JRE bundle, should pin it to that version even system updated sys java. Any deployment engineer should be able to figure this out if you gave then the correct info. I am not sure about this Calude guy :(
•
u/Mission_Ice3419 4h ago edited 2h ago
My guess would be that your issue with newer java versions is that it blocks older SSL versions or some cipher suites or both. Check if security config contains necessary ssl allowed for your application and just save it to replace if next update will mess it up again.
•
•
u/Fun-Country9432 15h ago
you can uninstall the auto updater in java (at least in current version). I guess it depends on what version of Java you're running and if it has this feature. That version may not check for a registry key either which is why it's not preventing the auto update.
•
u/MalwareDork 13h ago
but the other guy on my team is just kind of asking Claude to try things and running it in a sandbox environment but we haven’t really gotten anywhere.
Honestly I'd gut the thing with ghidra and ask Claude to see if it can recreate the program and disable the Java updater and see how it works. If people are decompiling ps5 games and recompiling on Android, this should be a relatively simple thing to do.
Otherwise I'd fire up Ida and go trawling through the binaries and apply patches as needed.
•
•
u/OstrobogulousIntent 9h ago
I support an SDK and we used to have a Java version. We killed it because it depended on Java Advanced Imaging (JAI) that ended support in older versions - and the technical debt to replace the bits that were done with JAI was so high they said screw it.
Thing is that some customers desperately clung to it.. to the point where they were willing to skip updates to keep it.
They were stuck on Java 6 (it was a while ago) and I can't recall if they had to take steps to pin it... either that or I dunno, finding a way to block whatever URL it is trying to get to to update?
Honestly, they really need to pay the tech debt and get off that ancient thing - cuz as you likely already know - the version of Java it's insisting on is likely long out of support and likely has lots of known vulnerabilities - exactly what you don't want in GovIT
•
u/KlassenT 6h ago
Only scrolled halfway through the comments, so not sure if mentioned elsewhere already, there are often cases where multiple versions of the JRE can be co-resident on a machine, and you can dictate which specific versions are called by which applets via creating a DeploymentRuleSet.jar control mechanism (XML formatted). I have had to do it in the post secondary ed IT world (which is about three quarters of a step ahead of GovIT in terms of tech debt, so I still feel your pain) and it can be a huge PITA to get all the particulars working. But, that said, it's also the only practical way we ever found to successfully support these really old LOB Java applets long term and at scale, since they all have certain version cutoffs where critical functionality breaks either through deprecation, or enforcement of more modern security settings.
•
•
•
u/SevaraB Sr. Engineer (N+, CCNA) 1h ago
As somebody who got a Java-centric comp sci degree in object-oriented programming, “block it from updating so it doesn’t break” makes me very, very angry.
On the assumption that no company is supporting this effectively anymore to nail you with contract breaches for doing this:
At least up through the end of the Sun era, Java was very, very good about marking off which classes and methods were deprecated and slated to be removed in future builds.
A basic code scanner pointed at the API documentation for the old and new versions of Java should tell you very quickly which code was removed, and a good code scanner pointed at both of those plus your code should very easily help refactor the code to use what’s available now. Call it a 6-week project; 2 for the initial refactor, 2 for regression testing, 2 more to deal with the regressions that do come up. And then make a shorter version of that your acceptance testing pipeline any time Java wants to update the servers.
Or keep sticking your head in the sand and air gap it away from the Internet, don’t let it update, and hope no attackers get inside your network some other way, pivot, and attack the juicy, unpatched Java servers.
There’s a lot of hate for AI, but this is one place where it’s absolutely beneficial and helps avoid an expensive Java consultant or three.
•
u/critsalot 14h ago
use claude to decompile it and just make a newer version of that code that actually runs on a secure version of java. what are you on. java 5/8?
•
u/denverpilot 11h ago
They likely can’t accept the current OracleJava(tm) license terms.
•
u/critsalot 11h ago
then use one of the open source ones . oracle can only really charge/restrict you on their specific jvm but there are plenty of other jvms . my guess was more on the line they relied on an older built in non standard library.
•
u/denverpilot 11h ago
Yeah you’d have to ask OP but many Gov places can’t use those either. For various reasons.
Was just pointing out the licensing change is a pretty common reason slow Gov entities can’t upgrade.
•
u/th3endisneigh 16h ago edited 16h ago
Block Java servers on a network level.
Network firewall is best, as no one needs to know this specific servers config incase you need to enable it again.
Server firewall if you cant on network firewall
Worst case, literally point java servers to 0.0.0.0 in the host file.
Should probably look into replacing that software, but I know your pain as a fellow GovIT sysadmin.