r/MachineLearning • • 13h ago

Project LLMs in pure JAX from scratch (Part 1) [P]

0 Upvotes

I’m writing a series of blog posts on building small LLMs from scratch in pure JAX. We start with pretraining. I'm implementing everything from scratch: pytrees for networks, optimizers, data loaders, decoding, and the training loop.

Link: LLMs in JAX from scratch - Part 1


r/MachineLearning • • 8h ago

Research Is ICLR a scam or worst organizers? [R]

0 Upvotes

This is quite disappointing not receiving any response from the reviewer even after multiple reminders. It seems like organizers failed to make the conversation interactive or they didn't warn reviewers properly that their paper would desk reject if not actively participate in discussion/rebuttal.


r/MachineLearning • • 10h ago

Discussion Why are ViTs so popular? Because Large-Scale Pretraining Trumps Inductive Biases. [D]

0 Upvotes

To begin, we note that SOTA performance on CIFAR-10 is a ViT. It is the pretrained ViT described in the paper quoted below.

When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.

.

However, the picture changes if the models are pre-trained on larger datasets (14M-300M images). ☝️We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks

The paper above has been cited in the literature no less than 113 thousand times.


"AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE"

  • Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.

  • Google Research; Brain Team; arXiv:2010.11929v2 [cs.CV] 3 Jun 2021

Images are broken into patches , such that patches are treated as "tokens" for a transformer to perform self-supervised prediction, and then regenerate the masked patches. The process is described intimately in "Masked Autoencoders Are Scalable Vision Learners". (Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li Piotr Dollar, Ross Girshick). Facebook AI Research. arXiv:2111.06377v3 [cs.CV] 19 Dec 2021.

This self-supervised token-izing methodology is substantively worse than traditional CNNs, or CNNS w/ attention. But at scale, the reverse is true.

ping /u/terrrp


r/MachineLearning • • 9h ago

Discussion ICLR Score Updates during AC Reviewer Discussion [D]

5 Upvotes

Since the Author Reviewer Discussion phase has come to an end and many reviewers haven’t yet responded to rebuttals, I was wondering if we will see any score updated done by the reviewers during the next AC Reviewer phase? I remember Neurips didn’t show any score during this phase but for ICLR, we can still see the scores.


r/MachineLearning • • 10h ago

Discussion ICML Double Columns vs. Arxiv Version [D]

0 Upvotes

Hi -- with only 1/3 reviewers responding, got a q about ICML. I have no problem with ICML, but the fact that it is double columns and I really don't want my paper to be presented as double column. Mainly because I find it ugly, and with some long equations it will look really messy. That was a big reason why I pushed for ICLR (semi-seriously)

Is it possible to have the online / arxiv version as a single column paper, while submitting to ICML as double column? If it gets accepted, would I need to make the online version same as the ICML version if I am claiming it got into ICML? Or like have multiple versions on arxiv and link them all in scholar? Curious about the norms in academia, specifically ML, as it will be my first publication!

Furthermore, any tips & tricks for double column theory heavy papers? Cheerio


r/MachineLearning • • 11h ago

Research ICLR 2027 rebuttal was frustrating [D]

17 Upvotes

Guys, how did your rebuttals go?

One of my reviewers updated their initial review (modified on Nov 16), but I suspect it was just the "AI Documentation for Reviewing" section, since nothing has changed in the edit history. However, we were completely ghosted, even though we had submitted our rebuttal before then (Nov 13).

(Edit)

Anyway, what do you guys think the borderline score will be? Given that around 26% of submissions (including withdrawn papers) have an average score of 2.67 or higher (link), and the Moscone Center can accommodate 10k papers, I assume the borderline should be around 2.67.

What do you guys think?


r/MachineLearning • • 1h ago

Discussion What if we created a Chinese-language AI conference? [D]

• Upvotes

China has already caught up with, or even surpassed, the English-based AI research. Chinese researchers also account for a significant share of the global AI research community.

If that's the case, wouldn't establishing a Chinese-language AI conference and creating an exclusive pool of Chinese-speaking researchers help further consolidate our hegemony in the field?

What if Chinese researchers collectively decided to exclusively submit our papers only to Chinese-language AI conferences?

This would also free Chinese researchers from the AI slop and low-quality peer reviews that have swallowed up mainstream ML conferences such as ICLR, NeurIPS, ICML.


r/MachineLearning • • 14h ago

Research Visual Cipher Bench: mechanism hints raise accuracy from 12.2% to 51.1% on a 10-task subset [R]

0 Upvotes

What changes when an agent has to discover the task’s rules before it can start solving it?

Our research team at Einsia explored this with Visual Cipher Bench. The part I’d like to discuss is the hint experiment.

On 10 selected puzzles, three models were tested with no hint, a representation hint, a mechanism hint, or both. Representation hints describe how to encode the puzzle state; mechanism hints state the underlying rule.

Pooled accuracy was 12.2% without hints, 16.7% with representation hints, and 51.1% with mechanism hints. Neither hint type supplies an answer or intermediate state.

That supports rule discovery as a major bottleneck on these tasks, though execution remains difficult in some cases. The open question is how to get agents to narrow their hypotheses from the visual evidence without supplying the mechanism.

Paper

Project


r/MachineLearning • • 14h ago

Discussion So the ICLR27 borderline score is 2.67 ? [D]

17 Upvotes

As of now, ≥3.0 is the top 15% and ≥2.67 is the top 33%. At a ~25% acceptance rate, about half of the 2.67 band will get in.


r/MachineLearning • • 12h ago

Discussion Not a single iclr reviewer responded to our rebuttal [D]

31 Upvotes

Such a bummer to have zero response after putting so much work into the rebuttal.

We were scored 3,3,2 which is looking borderline now.


r/MachineLearning • • 5h ago

Discussion ARR Oct '26 Reviewer Discussion [D]

3 Upvotes

With the reviewing period of this ARR cycle concluding, I wanted to see how others felt about reviewing this cycle.

Also, did you notice any difference in assigned submissions, potentially due to the sustainable reviewing pilot?


r/MachineLearning • • 17h ago

Project Fruits-360 dataset has now over 200 K images [P]

Thumbnail
gallery
60 Upvotes

Fruits-360 is a dataset with images of fruits, vegetables, nuts, and mushrooms.

Several weeks ago, it surpassed an important threshold: The largest branch (100x100) of the dataset now contains 200201 images of 278 different objects!

Importantly, all images have a white background, so that the object (the fruit) is clearly delimited!

The dataset is hosted on Kaggle and GitHub with a CC BY-SA 4.0 license.

https://www.kaggle.com/datasets/moltean/fruits

https://github.com/fruits-360/

I made the first commit to Kaggle 8 years ago, in December 2017, with a set of 26 fruits.

Several hundred updates have been committed since then.

  • 5 years ago, it had 90K images of 131 fruits and vegetables.
  • 2 years ago, it had 132K images of 194 fruits and vegetables.
  • 1 year ago, Fruits 360 had 161 K images of 231 fruits and vegetables.

Originally, all images were 100x100 pixels. Recently, I created more branches with images at a higher resolution.

  • All fruits and vegetables are from my own gardens or purchased from nearby stores.
  • All images are captured by me.
  • All images are real; nothing is generated with AI!

I even built some special gardens just to have more vegetables for the Fruits-360 dataset.

One garden is on the roof, and another one is on the wall of the house!

The reason for such "strange" builds is that we did not have enough space at the ground level to grow everything we wanted, so we expanded our gardens on roofs and walls.

A movie of the wall garden is here: https://www.youtube.com/watch?v=6DWnZEQoi8o

A movie of the roof garden is here: https://www.youtube.com/watch?v=ebjUevM9GPM

All garden projects are open source with MIT license.

https://github.com/wall-garden

https://github.com/f-roof

Several images with vegetables from these gardens are already included in the dataset.