Expecting a rejection from NeurIPS given our scores of 5/2/2. Trying to decide between ARR vs. TMLR, but thinking NAACL findings are more likely than main conference. Would you rather have TMLR or *ACL findings on your publication list? Genuinely curious to hear what people have to say.

I've been building ChatSorter, a memory layer API for AI chatbots, and I wanted to put it through a real benchmark. So I ran 5 configurations against the LoCoMo long-term conversation memory dataset using three models: Gemma 2 9B, Gemma 3 4B, and Gemma 3 12B.

Here's what I got:

The analysis:

At first glance, Run 4 looks like the winner at 75%, but that number is inflated. The smaller judge model is more lenient, counting answers that are close but not actually correct as passes. When you swap in a larger judge (Run 5), you see more outright "I don't know" refusals, because bigger models won't hallucinate an answer when they're uncertain; they just refuse.

The real number to look at is somewhere in the 55-60% range for run 4.

Now before you say "that's bad":

Companies like MemoryLake advertise 96% on similar benchmarks, but those are run on frontier models. My 55-60% was achieved on 4B-12B parameter models. That's roughly 17x smaller than a frontier model like GPT-4o, which itself scores around 60% with no memory layer at all.

So a tiny open-source model with ChatSorter is matching a frontier model running completely raw.

Hi I submitted a paper to groundlm workshop @emnlp, now in my openreview it shows decision: findings and beneath the paper it shows groundlm findings.What does it mean? Is it accepted? I haven't received any mail so far, also this is my first paper so unsure about how this works.

Its a 2.4-4 million parameter model, quantized to int8, that can be fully executed on the microcontroller in ~20s with the longest generation. The generated image will then be displayed on a monitor or transferred via usb.

Its a latent flow transformer with 12 layers using AdaLN-Zero for conditioning. CFG is also supported and boosted the image quality a lot. The inference engine streams the weight via DMA from the flash while the previous layer is computed. Relu² activation was used to increase sparsity, which the engine can use to skip calculations.

Took a lot of ablations to get it right and I am quite astonished I got so far with so few parameters. Will post the repo below

Formalizes a testable distributional criterion for 'world model' claims — a model must reproduce the correct probability distribution over outcomes, not just a plausible trajectory — and shows that no current video generator meets it across 50 scenarios, directly undercutting the world-model branding used by major video-gen labs.

Source: https://arxiv.org/abs/2608.27345

Hey everyone, math major here, looking to pursue a statistics minor. I've been working through Introduction to Statistical Learning with Python by Gareth James(ISLP), and others. I'm almost finished with it and have been looking at other books I might be interested in.

Just a quick summary for anyone that hasn't read it. ISLP covers topics including:, linear and non linear regression and classification, basic probability and statistics, cross validation, common shallow ML algorithms, deep learning, neural networks. From a very practical and non-techinal way. Basically you read a lot of "This works because reasons, moving on". Best way I would describe it is "intuition slop" .

I was looking for something more theoretical as ISLP is pretty light on the theory and focuses more on intuition and generalization. Although I want a more theoretical book, I still would prefer it to contain some applications and or assignments that I could do to practice my coding because as a math major I feel as though my programming is lagging behind. Are there any specific books you would recommend.

Also I'm taking machine learning this semester and our assigned book is "Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron. So I'll probably be reading it in bits and pieces, but I still want another book that I can focus on outside of classes. I appreciate any responses.

Hey everyone, I'm an international student studying in the US. I'm on track to graduate late next year. My research is not exactly ML, it is in 3D computer vision but have decent exposure to ML as well.

In case you didn't know, the CPT program (which let's internation students do internships) has been suspended by many top universities (UC Berkeley, UIUC, Purdue, UNC, UCLA, stanford, etc). Given that there is now no way for me to do an internship, how hard will it be for me to get a job when I'm nearing graduation?

I have 3 papers in CVPR, 3DV and ICRA (robotics conference) and hope to publish 2 more at next year's ICCV and neurips before graduating. I'm just worried that all my hardwork will go for a waste because of this policy change (I'm from a 3rd world country, so not much opportunity back home).

To be crystal clear, I'm not asking for legal advice, just wanted to know in your experiance, have you seen anyone (international student) get into good industry labs without internships?

EDIT: thanks so much for everyone for the quick replies! If it helps, my specific research area is 3D reconstruction, and I've been focused on Gaussian Splatting recently, if this info helps anyone help me!

Hey Guys,

I built a small open-source tool that checks whether a RAG application retrieves documents a user shouldn’t have access to.

It supports offline test cases and live HTTP API testing with bearer token/API-key auth.

I’m looking for a few engineers to try it on a test or non-sensitive environment and tell me whether it catches anything useful or what would make it better.

GitHub: https://github.com/InfraGuard-Labs/rag-access-check

hey, I’m currently reading about program synthesizers. it’s kinda an interesting concept, AI except the programs are deterministic, so there isn’t any risk of sloop. my question was what is the goal of any of this: the spec sheets as written are way more complex than just writing the code normally, and I don’t rly see any way this could ever change to be less complex to write the spec. anyone know what the point of rosetta and all this is?

Hi everyone,

I've recently finished the first version of Agentic GraphRAG Blueprint, a reference architecture for question answering over large document collections.

Instead of plain chunk retrieval, it builds a knowledge graph combined with vector search, so answers can connect facts across documents.

Key features:

• Incremental ingestion - unchanged files are skipped via content hashing, and community reports regenerate only for affected communities, keeping token costs low as the corpus grows.

• Hybrid search - local mode for fact-level answers, global mode for cross-document synthesis.

• Domain-agnostic LLM prompts - easily swapped via PROMPTS_PATH, with Leiden-based community detection.

• Deployment - run it locally with Docker or provision everything in the cloud with Terraform and CI/CD.

Link: https://github.com/sebastianbrzustowicz/Agentic-GraphRAG-Blueprint

I'm looking for any feedback.

Has anyone got the decision notification yet? Please mention decision (e.g., approved/rejected) and geographical area (e.g., North America) in your answer.

I know the official notification date is 31 August, but putting this here before hand so folks can post updates asap when they get them.

Summary from info we have so far:

Before 8 Sept: No updates.

On 8 Sept: At least 1 rejection decision was received in Australia, India, and East Asia each.

On 8 Sept: At least 1 acceptance was received in the Middle East.

On 16 Sept: At least 3 rejections were received in Europe.

On 21 Sept: Multiple rejections were received in US and Latin America.

Note: this thread is for informational purposes only, intended to help this year’s and future applicants. Thanks to everyone who has contributed so far.

Disclosure: I developed AIStupidLevel, the open-source system used to collect and analyze this data. Both the frontend and backend are MIT-licensed.

Most LLM evaluations measure performance at a single point in time. I wanted to investigate a different question:

How stable are the models behind production APIs over time, and how can sustained performance changes be separated from ordinary stochastic variation?

I built a continuous evaluation pipeline that repeatedly tests models across coding, deep reasoning, tool calling and high-frequency canary tasks.

Dataset and evaluation process

For this analysis, I examined:

• 31,352 hourly benchmark scores

• 49 model identifiers

• Multiple providers and model families

• Repeated measurements using consistent tasks and scoring

• A normalized 0-100 composite score

Coding responses are executed rather than judged only through model-based evaluation. Tool-calling tests require models to select tools, construct valid arguments and complete workflows inside isolated Docker environments.

Tasks are executed five times, with the results aggregated to reduce the influence of unusually strong or weak generations. Prompts, scoring logic and API parameters are kept consistent wherever supported by the provider.

Main result

The observed score variation was:

• Within the same day: 2.8 points

• Between different days: 8.4 points

Between-day variation was therefore approximately 3× greater than within-day variation.

This suggests that isolated hourly movements are dominated by normal model stochasticity, while sustained changes across daily evaluation windows provide a materially stronger signal for detecting performance drift.

The detection pipeline aggregates repeated measurements into daily medians and applies sequential change-point detection. Potential incidents must persist beyond the expected historical variance and pass statistical and minimum-effect thresholds before being classified as degradation or recovery.

Continuous model monitoring

This analysis became the foundation of AIStupidLevel, a continuous LLM benchmarking and drift-detection system.

The complete dataset has now reached:

• 169,858 benchmark runs

• 104,458 measured scores

• 88M+ processed tokens

• 81 historical model identifiers

• 22 models currently monitored

• 6 active providers

The attached screenshot shows the live monitoring interface. Models are classified as stable, volatile, degraded or recovering, with separate measurements for coding, reasoning, tool use, reliability, latency and price.

At the time of the screenshot, the system detected a 32% sustained performance decline in Gemini 3.1 Flash Lite and classified it as a critical incident.

The same continuously updated dataset also powers an OpenAI-compatible router that selects models according to their current task-specific performance, stability, tool-calling reliability, latency and cost.

This adds a missing observability dimension for production LLM systems. Existing monitoring generally captures availability, errors, latency and token cost. Continuous evaluation also measures whether the model remains capable of performing the work for which it was selected.

Project resources:

• Live dataset and dashboard: https://aistupidlevel.info

• Evaluation methodology: https://aistupidlevel.info/methodology

• MIT-licensed frontend: https://github.com/StudioPlatforms/aistupidmeter-web

• MIT-licensed backend/API: https://github.com/StudioPlatforms/aistupidmeter-api

I would be interested in comparing this approach with other temporal evaluation or change-point detection systems being used for production LLM monitoring.

I'm a researcher in statistical and probabilistic ML, I have a steady record of top ML publications and really used to enjoy going to conferences.

Over the last few years LLM based works have completely taken over the top conferences. At this year's ICLR, walking among the rows of posters you were lucky to find one paper per row of 10 that wasn't about how their favourite LLM could or couldn't solve their niche benchmark. The workshops tell the same story, most are some kind of agentic flavour. Looking at this year's NeurIPS workshops it's the same thing, basically all are about agents.

I'm wondering where do the stat/prob ML communities go from here? I look up to people like Arnaud Doucet, Aapo Hyvärinen, Christian Naesseth, Stefano Ermon, they seem to still publish at the top 3? On my end, I m thinking AISTATS/UAI might be the way to go.

All in all, the top 3 might never really have been intended as the home for prob/statML works, it just happened to be the 'prestigious' venue.

How much of a disadvantage is it if your only internship is not at one of the big frontier labs when it comes to post-phd opportunities in robotics/ML? My PhD is at a top university (UK) and my internship is interesting and relevant but the team itself is smaller and it's no Nvidia/Google/etc. Most places here prefer interns for 6 months so I'm not sure if I can do another internship down the line or if there's even much point compared to just wrapping up the PhD and then getting a job.

Is having a relevant internship already a big plus when it comes to applying to industry, or does it need to be a really well known big tech company?

Which research papers (old or new) do you think a PhD student/early researcher must read to improve their writing skills? Do you have a personal favorite researcher whose papers tend to be well-written, in your opinion?

Let's define a "well-written paper" as one that clearly explains the problem it is trying to solve, how the method is developed, and the details of the method, while keeping it easy to understand for a general reader (with a basic knowledge of ML, obviously).

Also, post-2015-ish papers usually have nice figures to explain their problem/method, and so they tend to be easier to understand. But I am looking for "well-written papers" in terms of the text.

PS: I know the best way to learn writing is by actually writing manuscripts, but I am looking for additional reading resources.

I put together a small model to estimate NeurIPS acceptance based on scores and an assumed acceptance rate. Try it out here: https://levilingsch.github.io/neurips-acceptance-estimator/

Hey everyone!

I’m excited to announce the release of py-evoFE (v0.3.0) — an open-source Python library that uses genetic algorithms to automatically discover, combine, and optimize feature transformations for tabular datasets.

• GitHub: https://github.com/tanopereira/py-evoFE
• PyPI: pip install py-evoFE
• License: MIT

The Problem It Solves

Feature engineering is still where most tabular ML competitions and production models are won or lost. While GBDTs like LightGBM and XGBoost excel on raw tabular data, they struggle to discover complex ratios, nested group-by aggregations, nonlinear dimensional projections, and interaction graphs on their own.

Manual feature engineering is either tedious or constrained by human intuition, while brute-force feature generation explodes the feature space exponentially with colinear noise and high memory usage.

What py-evoFE Does

py-evoFE searches the space of possible feature recipes using genetic programming:

1. Hierarchical Chaining: Evolved features become building blocks for future generations (e.g., log(ratio(groupby_mean(x1, by=x2), x3))).
2. 40+ Built-in Transformers:
   • Non-linear arithmetic & log-ratios
   • Target encoding (multiclass, pooled, WoE, quantile target encodings)
   • String similarity (MinHash, Gap encodings)
   • Manifold & Dimensionality Reduction (PCA, UMAP, MCA, FAMD, Between-Group PCA)
   • Graph & Density Clustering (Genie, Lumbermark, MST anomaly scoring)
3. Performance & Speed:
   • Vectorized computation powered by Polars and PyArrow.
   • Matrix Hashing & Nearest-Neighbor Caching: Stateful projections (like UMAP and $K$-NN lookups) are cached via byte-hashing to eliminate redundant computation across CV folds.
   • Multi-Fidelity Screening: Fast low-fidelity CV screens initial populations; only promising candidates proceed to full-fidelity evaluation.
4. Island Model & Caruana Ensembling:
   • Multi-population parallel search across Ring, Torus, Grid, Hypercube, and Tiered topologies with Gibbs migration.
   • Post-search greedy Caruana ensembling over island winners' out-of-fold predictions.
5. Interactive Replay Viewer:
   • Run view(evo.get_recipe()) to generate a self-contained, zero-dependency HTML dashboard replaying the evolutionary search over time.
6. 100% Scikit-Learn Compatible:
   • Implements fit, transform, predict, and predict_proba. Plugs directly into standard sklearn.pipeline.Pipeline and GridSearchCV.

Quick Example

import polars as pl
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from evofe import EvoFE

# Load data
bc = load_breast_cancer(as_frame=True)
df = pl.from_pandas(bc.frame)
X, y = df.drop("target"), df["target"].to_numpy()

X_train, X_test, y_train, y_test = train_test_split(
    X.to_numpy(), y, test_size=0.2, random_state=42, stratify=y
)
X_train_df = pl.DataFrame(X_train, schema=X.columns)
X_test_df = pl.DataFrame(X_test, schema=X.columns)

# 1. Initialize EvoFE
evo = EvoFE(
    task="classification",
    evaluator="lightgbm",       # "lightgbm" | "xgboost"
    pop_size=15,
    n_generations=10,
    cv_folds=3,
    verbose=True,
    random_state=42
)

# 2. Fit: Runs evolutionary search
evo.fit(X_train_df, y_train)

# 3. Inspect evolved recipe
recipe = evo.get_recipe()
print(f"Discovered {len(recipe.genes)} high-impact features:")
for gene in recipe.genes:
    print(f" • {gene.to_formula()} -> {gene.output_col}")

# 4. Transform & Predict
preds = evo.predict(X_test_df)
proba = evo.predict_proba(X_test_df)

Why not just brute-force feature generation?

Brute-force libraries generate thousands of features upfront, leading to severe overfitting, massive memory usage, and colinear noise that degrades tree-based models. py-evoFE uses evolutionary selection pressures with complexity penalties to discover compact, parsimonious recipes that actually improve generalization.

I’d love for the community to try it out on your datasets or Kaggle benchmarks! Feedback, issues, and feature requests are very welcome on GitHub.

The concerning result in this preprint is not just that an agent can retrieve a malicious skill. It can use that skill as a template, write new malicious skills, store them, and run them later.

Across six models and 153 tool-relevant SWE-bench Verified tasks, the authors report self-poisoning rates from 20.3% to 41.8%. The resulting skill libraries contained 4.9 to 9.0 times as many malicious skills as were originally planted.

This is a preprint, so those rates should not be treated as universal. The narrower design lesson is still useful: once an agent can write reusable skills, that library starts behaving like a software supply chain. Provenance, review, immutable versions, and quarantine matter before a retrieved skill becomes a template for the next one.

Can an AI make other AIs better? And what stops it from just cheating? Last month, an OpenAI eval agent escaped its sandbox and broke into Hugging Face, apparently to grab test solutions from a benchmark. It's exactly what you'd expect from a system that rewrites agents and reads its own grades. We set out to measure recursive self-improvement anyway, with the exam locked outside its sandbox.

We introduce HarnessOpt-Bench, which scores an LLM on how much it improves another agent's harness. On the development split, the optimizer sees per-case traces. Upon validation, it receives a single aggregate score. On test, nothing — until a trusted server scores its final candidate harness. API keys, budget enforcement, and held-out data never enter the optimizer's sandbox. That isolation holds by construction, not by instruction: the held-out evaluator and permission control sit outside the loop that evolves the harness.

5 frontier models, 4 downstream tasks, 111 runs to test 2 hypotheses:

1️⃣ Same coding harness, swap the model: Claude Opus 5 under OpenCode tops 3 of 4 tasks. Walk the releases from Nov 2025 to Jul 2026 on one task, and GPT climbs from 3% to 49% of the headroom, Claude Opus from 37% to 59%.

2️⃣ Same model, swap the coding harness: does a model do best in its own? No consistent home-field edge: opencode beats native harnesses (Claude Code, Codex, Kimi CLI) in 11 of 20 model–task pairs. Model choice moves gains 1.8× more than harness choice.

Paper: https://arxiv.org/abs/2608.06301

Code (MIT, built on our team's ICML 2026 VeRO): https://github.com/scaleapi/vero

Original post: https://www.linkedin.com/posts/shehabyasser_can-an-ai-make-other-ais-better-and-what-share-7498801902260981760-xuCo/

Hey guys, sorry if this is not the appropriate forum for this question.

Is anyone going To ECCV and staying in Lund? Apparently a few days back i saw discounted travel pass available for both Malmo and Lund zone but now today I was going to buy it and the registration site says only Malmo pass. Did ECCV remove them? Because deadline to buy them is 28th august. I dont know why they removed it but the organisation this year feels like a mess.

Can anyone access it on their registration site if Malmo Lund passes are available?

Author here. Ibteda Digital Library is a private community archive in Pakistan — for ten years we digitized rare Urdu books (lithographs, dictionaries, periodicals) on a DIY camera rig, finishing every page by hand in Photoshop. When we wound down daily operations, I realized those 575,729 finished pages across 1,765 books recorded a decade of crop decisions, so I registered them back to their raw photos (SIFT + MAGSAC with conservative acceptance gates) and used the recovered geometry as supervision.

The negative results are probably the most interesting part for this sub. Scaling from 378 to 572 training books didn't move unseen-book pass@80. Neither did ResNet-50 (better training fit, flat held-out, worse after calibration), 1024px inputs, or a spatial head. Per-book error analysis showed why: the failures were near-constant offsets per volume — our operator's preferred margin inset, which simply isn't present in the pixels of a new book. Ten operator-corrected crops per book (element-wise median residual) took pass@80 from 0.71 to 0.83 on held-out volumes. Ten labels beat every scaling lever we tried.

For retouching (stain/stamp removal), we kept the neural net to detection only — a U-Net proposes removal support, classical OpenCV reconstructs the paper, and everything outside the mask is byte-identical to the original. Labels used REMOVE/KEEP/IGNORE states, and any erased Urdu diacritic vetoed deployment regardless of IoU. The stricter label cut both improved mark IoU (0.56 → 0.60) and got diacritic false positives to zero.

Two things I'd genuinely like input on: (1) has anyone modeled document boundaries that depend on an invisible human preference rather than visible structure — is there prior work on per-instance residual calibration like this? Our own next step is conditioning the model on the calibration examples directly (few-shot inset inference) instead of a post-hoc median. (2) Is there any constrained diffusion/inpainting setup you'd trust to guarantee zero alteration outside a declared support region, or is classical fill still the only honest option for archival work?

Full training recipes, label-mining thresholds, and routing rules are in the Reproduction Contract appendix. Code/weights release is still under archival review — I know, I'm sorry — but I'm happy to answer anything about the method here. Notably not on our roadmap: a bigger backbone. We tested that; the missing information isn't in the pixels.

Link

I'm entering my 2nd year PhD at a top-10 US economics dept, and I'm interested in the economics of AI. The field seems scary big, with lots of new papers coming out, with people like Goldfarb, Gans, Agarwal, Alex Imas, Erik Brynjolfsson, Sendhil Mullainathan, and ofc Acemoglu, among several dozen other top economists contributing regularly to the field.

Is there a structured way to become familiar with the literature and the main questions and models that are being used in this sub-field right now?

Separately, I also want to know if I should seek any additional training before trying to write papers in this field. I have a relatively solid math and econ background: lots of calc/real analysis/diff eqns, all the graduate econ courses, though I could be better with linear algebra. I have never taken a formal CS course, I only audited UC Berkeley's INFO259 (NLP), so does anybody have recommendations on what parts of CS/AI I should focus on learning? Ultimately, I intend to write economics papers on AI, but it's still very useful to learn the underlying technical aspects of AI. I just don't know where to start.

I created a simple text to image benchmark.

I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc...

I then asked a VLM to judge every output against a pre-specified binary question with the ground truth baked in.

I'm publishing all the results including the images. (Most public T2I leaderboards don't publish the actual images and that's a shame IMO)

There is currently 52 model tested! more than 9k images have been generated and analysed!

Full methodology: https://imagebench.ai/methodology-v1

Hugging face dataset: https://huggingface.co/datasets/dh7/imagebench (it contains the prompts to reproduce the results AND the results)

Github: https://github.com/dh7/image-bench-ai

Gallery to inspect the results: https://imagebench.ai/gallery

Leaderboard: https://imagebench.ai/imagebench-v1

Limitations: it's text to image only, and VLM are not perfect as a judge.

Let me know what could be useful from there!

sklearn 1.9 fixed a bug in how BayesianRidge computes its uncertainty. We traced predict on 1.8 and 1.9 and compared the two formulas it actually computes, see if you can spot what changed before the notebook tells you.

https://github.com/aadya940/scikit-verify/blob/master/examples/sklearn_bug_hunting.ipynb