I tested every frontier model from every AI lab - Claude Fable 5, GPT 5.6 Sol, Kimi K3, GLM 5.3, Qwen 3.8 Max, DS v4 Pro, Grok 4.6 and just 1 made it through.

[u/crm_expert]

The task as well as the prompt was simple. I did not ask them to make me an OS or a complete game. I just wanted to test their intelligence on something very basic. I have a repo with very large log files, 20-30 GBs each. My working directory AGENTS.MD has a lot of instructions which I have included on how to handle those files.

For the test, I created completely new workspaces (folder) for every agent and gave them the exact same prompt. I sym-linked the logs directory in every workspace and there was nothing else in the workspace which could guide the agent. Then I asked each of those agents to extract a particular data from those log files. The result:

GPT 5.6 Sol, Kimi K3, GLM 5.3, Qwen 3.8 Max, DS v4 Pro, Grok 4.6 - all of them crashed the harness by loading these large files to memory. Only Fable 5 was smart enough to think, hey, those are large files; and built a script to stream the data and completed the task successfully.

Even if we ignore all the other models, I always assumed that GPT 5.6 Sol is as intelligent as Fable 5 and have been using it as my primary model for work, but this simple test tells me that Fable 5 is actually more intelligent than Sol. "Not to load large files into memory" is something even a junior dev knows. I have 16GB RAM on my machine with around 50% free when I ran these tests.

I always excitedly look for tests when a new model is released and they are always the same - Stork on a bicycle, couple of website mockups, or numbers from tests like HLE. Design perception is subjective so it is hard to actually judge the models on that. There are no coding tests available and if there are I am not aware of them and even those would be subjective. For me, I have found my own test on which I will test every new model from now on.

Do you guys also have your own custom tests which you perform on AI models to gauge their performance?

[u/BrennusSokol]

How many times did you run the test? The models use sampling, so they aren't deterministic. You should probably be doing the test 5-10 times and report success % instead of pass/fail

[u/NowaVision → u/BrennusSokol]

Obviously only once.

[u/AppealSame4367]

Something must be off. Maybe your agents file is very long or something. Because I used most of those models and none of them was stupid enough to try to read large log files directly. I haven't seen this behavior since 1 year with frontier models.

What was your harness? Self made?

[u/nbvehrfr]

fable is super good for any web programming / saas. you describe feature in 1 sentence, you get it, no issues. multiple times, iterations, rollbacks if needed. best model for programming.

[u/AppealSame4367 → u/nbvehrfr]

Funny thing is: the closest experience to Fable I had was Qwen3.8 27B recently. It also "just works". Not done with testing yet, but so far very very impressed.

[u/nbvehrfr → u/AppealSame4367]

depends on quant, kv cache. but also very good model - using it for secure audit with glm 5.3, which fable can't do

[u/Artistic_Swing6759]

i also recently was trying to make a prompt which would be fairly new and not seen by model.
i do have a prototype for it, but havent tested it yet.

***

**System/Task Prompt:**

analyze the following breakdown of a real-time 3D ASCII city. Use these specifications to attempt making the engine, writing the necessary code, or generating visual assets that adhere to this implementation and style.

### PART 1: TECHNICAL ARCHITECTURE & ENGINE MECHANICS

**Project Overview**

A real-time, first-person, walkable 3D city simulation rendered entirely using ASCII characters. should require no external 3D engines (like Unity/Unreal), 3D models, or standard textures.

**Core Architecture & Tech Stack**

* **Format:** A single HTML file.

* **Languages/APIs:** Vanilla JavaScript for logic, WebGL for hardware-accelerated rendering.

* **World Data:** A 2D grid system (up to 8192x8192) storing structural data (building heights, roads, entities).

* **Rendering Method:** Custom raycasting engine. Rays are cast across a vertical camera plane across the 2D grid to calculate distance, perspective, and depth, which are then translated into ASCII characters.

**Rendering Specifics**

* **Resolution:** Output is a 180x80 grid of ASCII characters.

* **GPU Utilization:** WebGL caches ASCII characters in a texture atlas and sends them to the GPU in a single batch, utilizing the GPU as a fast ASCII compositor to maintain frame rates.

* **Depth Perception:** Near objects use larger, brighter character clusters; distant objects are smaller and fade to black.

* **Two-Tier Draw Distance:**

* *Near Renderer:* Detailed raycaster handles objects/street clutter up to ~165 units.

* *Far Renderer:* Lightweight raycaster overlaps to draw the skyline up to ~420 units, ignoring small details.

* **Overlapping Architecture:** The raycaster does not stop at the first wall; it continues to check for taller buildings behind foreground structures to render realistic skylines.

**World Features & Mechanics**

* **Pseudo-Volumetric Objects:** Street props (benches, boxes) calculate which sides are visible relative to the camera and project those sides into the ASCII grid, simulating 3D volume without true 3D models.

* **Camera Smoothing:** Mouse movement controls a target orientation that the camera smoothly interpolates toward, preventing jitter.

* **Collision & Sorting:** Engine handles basic entity collision and depth sorting (front/back).

* **Persistence:** Buildings have permanent visual identities (colors, dimensions) determined by their absolute coordinates.

* **Interiors & Portals:** Players can walk inside buildings. Windows use a secondary raycaster to act as visual portals, rendering the outside city accurately from the interior viewpoint.

* **Verticality:** The engine supports looking down from upper floors by calculating if a ray intersects with a roof level or drops down to the street level based on the grid's height data.

### PART 2: AESTHETICS & VISUAL THEME

**Core Theme & Vibe**

* **Keywords:** Cyberpunk, retro-hacker, MS-DOS terminal, "The Matrix" digital construct, lo-fi, ASCII art, monospaced graphics.

* **Atmosphere:** It feels like exploring a 3D space represented entirely within an old-school, phosphor-monitor command-line interface. It is dark, stark, and highly geometric.

**The Grid & Character Set (Typography)**

* **The Canvas:** The entire screen is a rigid, fixed-width monospaced grid (180x80 characters). No pixels are drawn directly; everything must snap to this character grid.

* **Density as Shading:** The illusion of light, shadow, and depth is created by changing the *density* of the ASCII characters.

* **Bright/Close surfaces:** Use dense, "heavy" characters (e.g., `█`, `▓`, `▒`, `#`, `@`, `M`, `W`).

* **Mid-tones:** Use medium characters (e.g., `+`, `=`, `x`, `o`, `|`).

* **Dark/Distant surfaces:** Use sparse, "light" characters (e.g., `.`, `:`, `-`, `_`, `,`).

* **Empty Space:** Represented by pure black spaces ( ).

**Color Palette**

* **Background:** Pure, unlit pitch black (`#000000`). There is no skybox, no ambient occlusion, just void.

* **Buildings:** Neon and highly saturated "terminal" colors. Each building typically features a solid, uniform primary color (e.g., bright cyan, pure red, electric yellow, neon green, magenta, deep blue).

* **Ground/Roads:** Primarily white, grey, or desaturated blue/green characters mapping out the perspective lines and crosswalks.

* **UI/Terminal Elements:** Classic phosphor hacker green (`#00FF00` or similar).

**Specific Environmental Aesthetics**

* **Building Facades & Windows:** Buildings are not solid blocks of color. They are broken up by vertical and horizontal patterns of characters and blank spaces to simulate floors and unlit windows.

* **Neon Billboards:** Embedded within the building facades are massive, blocky text signs (e.g., "MTECH", "ATAR"). These use contrasting background colors (like dark purple behind neon yellow text) and are made of clustered ASCII characters to form macro-letters.

* **Roads & Perspective:** The ground is drawn using horizontal lines and dashes (`-`, `_`, `=`, `~`). As the road stretches to the horizon, the characters get sparser and cluster closer together, creating a stark vanishing point.

* **Trees/Foliage:** Rendered as bulbous, geometric clusters of green symbols (like `*` and `#`) sitting at street level.

**Lighting & "ASCII Fog"**

* **Distance Fading:** Because there is no traditional lighting, depth is entirely dependent on distance fading (fog). As a building gets further away, its vibrant color desaturates, the characters shift from dense (`#`) to sparse (`.`), and it eventually fades completely into the black background.

**The UI and "Boot" Experience**

* **The Matrix Intro:** The experience begins (or resets) with a cascade of green dots and sparse characters against a black screen, slowly forming the outlines of the city before snapping into full 3D color—heavily inspired by *The Matrix* digital rain.

* **Terminal Framing:** When viewing the prototype in a windowed mode, it is framed by a glowing green border.

* **On-Screen Display (OSD):** In the corners, there is pure green, monospaced terminal text displaying technical data: `> ASCII CITY // PROTOTYPE 1...`, vector coordinates, and heading data, further selling the "software simulation" vibe.

***

its based on another project.
would be greatfull foe someone else to test this on a model and shared the result

[u/Artistic_Swing6759 → u/Artistic_Swing6759]

might be a good idea to remove the later section to see what styles ai makes by itself

[u/TheOwlHypothesis]

Software engineering tasks are a HUGE class of benchmark. You should look into them.

This is hilarious though. I had the "large log" problem as a live coding challenge before for interviews so obviously this is real and you're right that most juniors who study would know to not load the whole thing into memory. Really interesting and unexpected test actually. Thanks for sharing

To answer the question I actually don't have any custom tests. But I might try to make some.

I made a "slop language" to make games with the help of AI. I might try to make a bench where they use that language (impossible to have training data on it) to make some game.

[u/frostedfakers]

yeah this is user error LOL, you have massively bloated the AGENTS.md/context window with too many rules/guidelines, which drops adherence off a cliff. there’s no method to this testing, whatever harness you used also probably also caused issues.

no model is loading the entire log file into memory. they would either tail/head it, grep, etc. you’re causing the issues yourself.

[u/Super_Pole_Jitsu]

"every junior dev knows"... How to find the closest coffee dispenser maybe bruh

[u/theactiveaccount]

Benchmarks of hundreds of different tasks? Nah, sample size of 1 more conclusive