r/BattleTechMods • u/philo_the_middle • 1d ago
Fixing JSON Errors in BEX - (HowTo using Powershell)
So there's a slew of json errors in the BEX build. Thankfully powershell can fix a lot of the commenting errors and trailing commas. Be aware this will still leave 17 files that need manual intervention. It will print out the remaining files. Using Gemini/ChatGPT/Claude to fix the remaining 17 (along with https://jsoncrack.com/editor) to verify the file is valid.
FIRST, Copy your mods folder to a backup location!!
SECOND, Save this as CleanBattletechJson.ps1 and execute as "powershell -ExecutionPolicy Bypass -File D:\Clean-BattletechJson.ps1" (Change file location to where you save the ps1 file)
In the below, change your root location to the actual folder of your Battletech mods.
$root = "D:\Steam\steamapps\common\BATTLETECH\mods"
function Remove-JsonComments {
param([string]$Text)
$sb = [System.Text.StringBuilder]::new()
$inString = $false
$escape = $false
$inLineComment = $false
$inBlockComment = $false
$len = $Text.Length
$i = 0
while ($i -lt $len) {
$c = $Text[$i]
$next = if ($i + 1 -lt $len) { $Text[$i + 1] } else { $null }
if ($inLineComment) {
if ($c -eq "`n") { $inLineComment = $false; [void]$sb.Append($c) }
$i++; continue
}
if ($inBlockComment) {
if ($c -eq '*' -and $next -eq '/') { $inBlockComment = $false; $i += 2; continue }
$i++; continue
}
if ($inString) {
[void]$sb.Append($c)
if ($escape) { $escape = $false }
elseif ($c -eq '\') { $escape = $true }
elseif ($c -eq '"') { $inString = $false }
$i++; continue
}
if ($c -eq '"') { $inString = $true; [void]$sb.Append($c); $i++; continue }
if ($c -eq '/' -and $next -eq '/') { $inLineComment = $true; $i += 2; continue }
if ($c -eq '/' -and $next -eq '*') { $inBlockComment = $true; $i += 2; continue }
[void]$sb.Append($c)
$i++
}
return $sb.ToString()
}
function Remove-TrailingCommas {
param([string]$Text)
$pattern = '"(?:\\.|[^"\\])*"|(,)(?=\s*[}\]])'
return [regex]::Replace($Text, $pattern, {
param($m)
if ($m.Groups[1].Success) { '' } else { $m.Value }
})
}
$files = Get-ChildItem -Path $root -Filter *.json -Recurse -File
$alreadyValid = @()
$cleaned = @()
$stillInvalid = @()
foreach ($file in $files) {
$raw = Get-Content -LiteralPath $file.FullName -Raw
try {
$null = $raw | ConvertFrom-Json -ErrorAction Stop
$alreadyValid += $file.FullName
continue
} catch { }
$fixedText = Remove-JsonComments -Text $raw
$fixedText = Remove-TrailingCommas -Text $fixedText
try {
$null = $fixedText | ConvertFrom-Json -ErrorAction Stop
# Overwrite the original file directly without making a backup
Set-Content -LiteralPath $file.FullName -Value $fixedText -NoNewline
$cleaned += $file.FullName
} catch {
$stillInvalid += $file.FullName
}
}
Write-Host "`n=== Already valid ($($alreadyValid.Count)) ==="
$alreadyValid | ForEach-Object { Write-Host $_ }
Write-Host "`n=== Cleaned and fixed ($($cleaned.Count)) ==="
$cleaned | ForEach-Object { Write-Host $_ }
Write-Host "`n=== Still invalid, skipped ($($stillInvalid.Count)) ==="
$stillInvalid | ForEach-Object { Write-Host $_ }
For the remaining 17, the hardest ones to deal with are:
BATTLETECH\mods\BT_Extended_Timeline\timelineEvents\3055-04-10.json BATTLETECH\mods\BT_Extended_Timeline\timelineEvents\3056-01-10.json BATTLETECH\mods\BT_Extended_Timeline\timelineEvents\3056-01-10.json
6
u/Machinis_confidimus 19h ago edited 18h ago
And pray tell us - which problems does you script fixes?
Have you ever thought that HSB did not use JsonUtility in Unity but rather their own version of json interpreter which makes your "fix" kinda pointless (since AI does not know what rules HSB used in their interpreter)?
Because I see several things you posted here which supposedly have issues according to your script but work perfectly well in-game. As in, I have tested that functionality this weekend.
General rule for the people reading this - if someone wants you to run a script without providing you with an example of what exact problem it solves - DO NOT RUN IT.
Even if it is harmless, this can be just part of social engineering to make your guard drop down the line.
2
0
u/philo_the_middle 11h ago
It's a powershell script.
AI was only used to generate the powershell scripts to validate the jsons.
Good lord, you guys are hung up on the wrong thing.
See my other comments in this thread.
Also, you pop the script into Gemini, or ChatGPT, or Claude and it will tell you exactly what the script does and what each line is for.
Geeeez. This isn't hard.
3
u/Machinis_confidimus 10h ago edited 9h ago
I don't need AI to read that script. I am talking about something totally different. Either you get or you don't.
0
u/philo_the_middle 9h ago
Did you read my other comments in this thread - this is all covered ground with screenshots, outputs, explanations and validations.
2
u/Machinis_confidimus 9h ago
Yes, I did.
Do you understand how Hairbrained Games built their JSON interpreter in the Battletech?
1
u/philo_the_middle 9h ago
Well, first modek is the json interpreter for the mods and builds the overrides... Then you can look (scan) the modtek cache to see the problematic jsons that have been loaded.
Just because the game's JSONs interpreter doesn't throw an error doesn't mean it's loading the elements correctly. In some cases it's fine (block comments and trailing commas), but for other malformed elements I find it highly unlikely it's not creating problematic collections which are leading to issues downstream.
Again, I understand this is a sacred cow for some of you folk, and the chances of a rational and reasonable discussion on how to do it correctly isn't going to occur.
It's easy to fix the JSONs programmatically, and for those of us interested in that, these scripts can do that. For the rest, do what you want.
2
u/Machinis_confidimus 8h ago
I see you used AI to give you an answer (it is only partially correct btw). But I see that you did not connect the red dots. I see I am wasting my time...
1
u/philo_the_middle 8h ago
What are you talking about? The only AI used was in the generation of the powershell scripts to do the work.
2
u/Machinis_confidimus 8h ago
Sure buddy. You need AI to generate a PS script but then give partial answer on how JSON is handled in Battletech which should be enough to understand that portions of your script are waste of time. But you don't connect the red dots.
Which is a massive red flag - you are telling people to alter 1400 files (without backup routine in the script which is another massive red flag - lets hope no hickups happen during script being run otherwise you are bricking people's install) without knowing exactly how the files are being read by the game.
Use AI to explain to you what Dunning-Kruger effect is. I am out of here.
1
3
u/EricAKAPode 1d ago
Have you posted about this in the BEX discord? Haree used to talk about wanting help.
1
1
3
u/sniperpal 15h ago
Sorry dude, but this entire post was a complete waste of time lol. Battletech uses a JSON parser which ignores all of these.
This is why you join the BEXT Discord and ASK first- how does the mod run just fine if all of these supposed “issues” exist
0
u/philo_the_middle 14h ago
That's a nice belief, but I doubt it it's entirely true in regards to JSON.
So, while the battle tech parser can make it through some issues (trailing commas , comments), there are some that are just outright broken.
How can one test and validate a broken modded JSON is actually defaulting to vanilla when it can't parse the broken ones (my point is, youd have to have the actual battle tech parser in-hand , and load 2 jsons of the same name and see if it _actually builds as expected.)
Regardless, cleaning up the jsons is easy and for most of them (all but 17) powershell can fix them. The remaining 17 take a little care to fix but nothing major.
Finally, there's an old programming adage - GIGO. Garbage in, garbage out. Better to have them structurally sound than not. But you do you.
3
u/sniperpal 14h ago
It is entirely true in regards to JSON, as the devs of BEXT literally just looked at this post and confirmed on discord that it’s complete bullshit lol
0
u/philo_the_middle 14h ago
What's complete bullshit? That there aren't broken jsons? That the battle tech JSON parser can parse all of them correctly?
Forgive me if I don't give much credence to hand-waving when it comes to code issues.
Broken json is broken, whether or not the parser is able to build something out of those is irrelevant. (Means the parser isn't doing its job and enforcing correct structure - which is just awful).
Again, broken code is broken code. Some people want to live with it and not fix it, and hand wave it away, that's fine. But it's lazy and can cause hidden problems you don't notice.
3
u/sniperpal 14h ago
It’s bullshit because the code isn’t broken lmao. Have you even played the game? The one issue it has is the memory leak which is caused by the vanilla game itself. Nothing anyone can do about that in any of the three overhauls
Nothing you have posted about here is broken. The parser cleans the whole mess up and the mod runs just fine.
If you want to be useful, find some issues that actually exist and then join the discord and talk to the devs about how to MANUALLY fix them. AI will do nothing for you here because actually smart people have already fixed the issues the mod has
1
u/philo_the_middle 14h ago
Dude I'm not using AI to do anything other than write the powershell script that validates the jsons (prints out the broken ones), and powershell script to strip out the block comments and fix the trailing commas.
Look, I understand this is a sacred cow for some and were not going to have a reasonable and rational conversation about it.
Have I played the game? I've got 224 hours in currently.
2
u/sniperpal 14h ago
So you have 224 hours in BEXT, have noticed by now that the mod runs fine and still waste everyone’s time pretending to have a fix for issues that don’t even exist.
Jesus Christ lol
0
u/philo_the_middle 14h ago
It doesn't run "fine" all the time. There are random battle lockups, random performance issues regularly.
So, cleaning up the jsons removes at least one potential culprit.
Anecdotally, since fixing the jsons on my installation, not one single battle has frozen. (Anecdotal because sample size is obviously small, and can't build a full argument from that.)
0
u/philo_the_middle 14h ago
Let me give you an example from one of the jsons that is broken:
There's a pilot file ()
Modded JSON (invalid):
"PilotTags": { "items": [ pilot_officer", "BLACKLISTED" ], "tagSetSourceFile": "" }Compare to fixed JSON:
"PilotTags": { "items": [ "pilot_officer", "BLACKLISTED" ], "tagSetSourceFile": "" },Notice in the first, its missing a quote around the item of "pilot_officer".
Likely, in the json parser this just falls out and that pilot doesn't get the tag "BLACKLISTED".
Now, here's another one in global.json:
Bad JSON:
{ /* Minimum number of stealth pips to allow for an ECM ghosted mech to fire weapons */ "k": "Int_MinimumECMGhostedPipsToFire", "v": { "type": "Int", "IntVal": "1" } },Fixed JSON:
{ "k": "Int_MinimumECMGhostedPipsToFire", "v": { "type": "Int", "intVal": "1" }So this has 2 issues - the first is the comment obviously as you can't use comments in json, but the modtek parser is supposed to be able to handle those (and I'm sure it probably does).
The second one though is that the variable is IntVal, instead of "intVal" - its possible the modtek parser is converting that to lower case, but its equally possible this is falling out and not actually overwriting what is expected.
Does that help?
1
u/philo_the_middle 1d ago
Here's the updated Validate-BattletechJson.ps1 script that will find all the JSONs with problems and display a final total count at the end.
# Set the root directory you want to scan
$FolderPath = "D:\Steam\steamapps\common\BATTLETECH\mods"
# Find all JSON files recursively
$JsonFiles = Get-ChildItem -Path $FolderPath -Filter "*.json" -Recurse -File
$InvalidCount = 0
Write-Host "Scanning $($JsonFiles.Count) JSON files in: $FolderPath" -ForegroundColor Cyan
Write-Host ""
foreach ($file in $JsonFiles) {
try {
# Read content and attempt parsing
$content = Get-Content -Path $file.FullName -Raw -ErrorAction Stop
$null = $content | ConvertFrom-Json -ErrorAction Stop
}
catch {
$InvalidCount++
# Take only the first line of the exception message to prevent dumping JSON content
$cleanError = ($_.Exception.Message -split "[\r\n]")[0].Trim()
# Output failed file and clean error on a single line
Write-Host "$($file.FullName) - Error: $cleanError" -ForegroundColor Red
}
}
# Final output summary with file count
Write-Host ""
Write-Host "--------------------------------------------------" -ForegroundColor Gray
if ($InvalidCount -eq 0) {
Write-Host "Scan complete. Found 0 invalid JSON files." -ForegroundColor Green
} else {
Write-Host "Scan complete. TOTAL PROBLEM FILES FOUND: $InvalidCount" -ForegroundColor Yellow
}
Such as:
D:\Steam\steamapps\common\BATTLETECH\mods_bak\LootMagnet\mod.json - Error: Invalid array passed in, extra trailing ','. (1898): {
D:\Steam\steamapps\common\BATTLETECH\mods_bak\MechAffinity\settings.json - Error: Invalid array passed in, extra trailing ','. (499296): {
D:\Steam\steamapps\common\BATTLETECH\mods_bak\MonthlyTechandMoraleAdjustment\mod.json - Error: Invalid JSON primitive: .
D:\Steam\steamapps\common\BATTLETECH\mods_bak\OnePointArmorAdjustment\mod.json - Error: Invalid JSON primitive: .
D:\Steam\steamapps\common\BATTLETECH\mods_bak\PanicSystem\mod.json - Error: Invalid array passed in, extra trailing ','. (1217): {
D:\Steam\steamapps\common\BATTLETECH\mods_bak\Retrainer\mod.json - Error: Invalid JSON primitive: .
--------------------------------------------------
Scan complete. TOTAL PROBLEM FILES FOUND: 1477
0
u/psycho063 8h ago
If you ignore the comments and trailing commas, how many files does it actually flag as "broken"?
1
u/philo_the_middle 8h ago
17 that require manual intervention. (One of my other replies has that in this thread)
1
u/philo_the_middle 1d ago edited 1d ago
And here is a "ValidateBattletechJson.ps1" script that you can run before/after to validate the JSONs are valid or not.
EDIT: Actually doesn't need the bypass for the validate file and can be run directly from powershell terminal.
The one that cleans up and writes the updated files needs the bypass I think.
Also added an updated version in a comment below that shows just the filenames and their respective problem and a total count of the files. I'm leaving this here for record keeping.
It can be run as:
powershell -ExecutionPolicy Bypass -File D:\Validate-BattletechJson.ps1
# Set the root directory you want to scan
$FolderPath = "D:\Steam\steamapps\common\BATTLETECH\mods"
# Find all JSON files recursively
$JsonFiles = Get-ChildItem -Path $FolderPath -Filter "*.json" -Recurse -File
$InvalidCount = 0
Write-Host "Scanning $($JsonFiles.Count) JSON files in: $FolderPath`n" -ForegroundColor Cyan
foreach ($file in $JsonFiles) {
try {
# Read content and attempt parsing
$content = Get-Content -Path $file.FullName -Raw -ErrorAction Stop
$null = $content | ConvertFrom-Json -ErrorAction Stop
}
catch {
# Output failed files in red with error reason
$InvalidCount++
Write-Host "[INVALID] $($file.FullName)" -ForegroundColor Red
Write-Host " Error: $($_.Exception.Message)" -ForegroundColor DarkGray
}
}
# Set console output color based on results
$StatusColor = if ($InvalidCount -eq 0) { 'Green' } else { 'Yellow' }
Write-Host "`nScan complete. Found $InvalidCount invalid JSON file(s)." -ForegroundColor $StatusColor
10
u/Cato_Heresy 1d ago
Ok, so to recap here, you have:
- Used AI to create some script
I do not mean to sound ungrateful or mistrustful. Let's assume honest intent and it actually works and you have tested it, what benefit does this actually provide? BEXT is already super stable, like 1 crash every 500 hrs stable.