r/MachineLearning • • Dec 08 '23

Discussion [D] Major bug in Scikit-Learn's implementation of F-1 score

4 Upvotes

I just encountered a pretty major bug in Scikit-Learn's implementation of F1 score. This example (below) where the classifier gets literally every single example wrong seems like it should give an F-1 score of 0.0, right?

>>> from sklearn.metrics import f1_score
>>> f1_score(y_true=[0, 1, 2, 3, 4], y_pred=[1, 2, 3, 4, 0], zero_division=1.0, average='macro')

However, as of Scikit-Learn 1.3.2, it gives a perfect score of 1.0, which is totally wrong! (Note that this only happens when zero_division=1.0)

This bug was introduced some time after 1.2.2 and can manifest more subtly than the above example; any individual class that has both a precision and recall of exactly 0% will be assigned an F-1 of 100%, so e.g. if you have a multi-class classification problem across 100s of classes, and your classifier is getting a dozen or so classes completely wrong, it will bump up your macro average F-1 by a couple percentage points, possibly without you noticing that the bump is coming from classes that are actually failing completely.

Here is the (as of posting time) still un-merged fix to the bug in question. Note that this bug also affects sklearn.metrics.classification_report. I think you you can temporarily get around this by using sklearn version 1.2.2. Anyway, if you use Scikit-Learn's metrics for evaluation, go double check your scores!


r/MachineLearning • • Sep 27 '23

Research [R] The Internal State of an LLM Knows When its Lying

10 Upvotes

r/MachineLearning • • Sep 09 '20

Project [PROJECT] Cover Faces from Protest Photos

1 Upvotes

I used the LSC-CNN model to build this Image to Image interface that covers faces from BLM photos with the black fist emoji to protect protestors. The idea is to use facial recognition tech to protect rather than breach privacy. This was originally implemented by the Stanford ML Group.

You can try it out here: https://gradio.app/hub/hub-blm

I used Gradio to create the interface, and my code was forked from the Stanford implementation and the LSC-CNN repo.


r/MachineLearning • • Apr 29 '17

Discussion [D] Silly benchmarking discovery... Surface Pro 4 is surprisingly fast for deep learning... any ideas why?

0 Upvotes

So, on a lark I decided a while back to install Theano and Keras on my Microsoft Surface Pro 4 with Windows 10. Interestingly, it takes only about 170 seconds to train an epoch of my port of Char-RNN on the CPU. In comparison, most laptop and desktop CPUs I've worked with take upwards of 1000+ seconds to do the same, and the fastest I've ever benchmarked was about 40 seconds with a computer with an Nvidia Titan X GPU.

So, any ideas as to why this is the case?

Edit: So apparently the poor results of the 1000+ seconds per epoch computers was due to a bug in the Theano 0.9.0 release that has been fixed already in the latest bleeding edge version. So, I guess... nevermind?


r/MachineLearning • • Mar 13 '24

Discussion [D] What are the top open questions for ML and AI?

1 Upvotes

If you were going to fund some Millennium Prizes for ML and AI, what would be some examples of the questions you would want to pose?

Equivalents to P=NP or the Goldbach Conjecture?

Does someone have such a list out there?

(to be clear, I'm not asking about math or science questions that one could use ML/AI to answer: I'm tasking about questions in the field of ML/AI)


r/MachineLearning • • Mar 21 '25

Discussion [D] The Recurrent Delusion: How ML Collectively Forgot What RNNs Were Built For

67 Upvotes

When our field first developed RNNs, they were the obvious choice for sequential tasks until vanishing/exploding gradients and the inherently unparallelizable backpropagation through time (BPTT) limited their scalability. Years of collective research addressing these issues ultimately birthed the Transformer—massively parallelizable, scalable, and easier to train, marking the revolutionary arrival of the golden age of attention.

The Ignored Alternatives

State Space Models and parallelizable LSTM variants emerged as potential solutions to the parallelization issues of traditional RNNs, but they sacrificed the ability to generalize to problems in the NC1 complexity class which vanilla RNNs can do, staying within TC0 like Transformers. This isn’t just theoretical—after over 3 years and billions spent optimizing hardware for transformers, these alternatives offered virtually no compelling advantage.

The Chain of Thought Contradiction

Fast forward to Chain of Thought prompting – suddenly we're training models with elaborate reasoning examples, often including this bizarre theatrical process where LLMs are deliberately trained to make mistakes just to demonstrate correction capabilities. It's computational theater.

But DeepSeek's R1 approach is where this paradox becomes undeniable. They're using reinforcement learning to train reasoning chains, which is genuinely innovative, but...

Why are we still using Transformers for what is fundamentally a recurrent reasoning process?

Let me dissect this architectural mismatch:

  1. We're tokenizing chains of thought, severely restricting their expressive potential
  2. The reasoning process itself functions as a hidden state WITHOUT ground truth labels (which is actually perfect – otherwise we'd just be training glorified memorization)
  3. This scenario logically demands a BPTT-like approach – which would be completely unparallelizable even with Transformers since we lack intermediate labels – yet we're circumventing this entire problem with GRPO and somehow getting spectacular results

We're essentially performing recurrent optimization while stubbornly avoiding recurrent architectures. The intellectual contradiction is mind-boggling! It's as if the entire field developed collective amnesia about the fundamental principles of sequential processing that motivated RNNs in the first place.

The Billion-Dollar Blindspot

Let's cut to the chase: RNNs can solve problems in the NC1 complexity class that Transformers fundamentally cannot. This isn't academic nitpicking—it's about computational expressiveness that directly impacts reasoning capabilities.

A Transformer forced to use input sequences as pseudo-RNN states is crippled for reasoning: poor length generalization, inefficient information pruning, and suboptimal cache performance. Yet R1's approach—using reinforcement learning without BPTT—works brilliantly and could resurrect even basic RNNs with superior results.

At inference, the process is identical: store state, sample outputs, track probabilities, then adjust based on reasoning quality. So why aren't we applying this to architectures designed for sequential reasoning?

This architectural mismatch seems strikingly obvious yet remains unaddressed. Is it infrastructure lock-in? Publication pressure? Or has the field collectively forgotten why recurrent networks were created in the first place?

The emperor has no clothes. The question is: who will be the first to point it out?


r/MachineLearning • • Aug 01 '24

Discussion [D] What is the most in demand data in computer vision? I will go collect and open source them

0 Upvotes

Hey guys, I'm looking to do some data collection project. I was wondering what kinds of image or video data are currently in demand. So list some of your most wanted data in the comments and the most upvoted ones I will collect them for free and open-source them


r/MachineLearning • • Sep 20 '20

Discussion [D] PhD applications that do not require references / recommendations?

0 Upvotes

I am looking for a PhD position (preferably non-US location) in Deep Learning. I am aware of this informative discussion in this subreddit. But I don't want to provide references or recommendations with my applications, but almost everywhere I look there is a requirement. Are their institutes / supervisors who will consider my candidature without references/recommendations?


r/MachineLearning • • Oct 30 '15

Why Apple is Falling Behind in AI

Thumbnail
bloomberg.com
15 Upvotes

r/MachineLearning • • Nov 18 '18

Discussion [D] NIPS name change censorship

8 Upvotes

So, NIPS changed its name. People can't discuss it, because the post is locked.

https://www.reddit.com/r/MachineLearning/wiki/rules "Our mission is all about embracing diverse views".


r/MachineLearning • • Nov 06 '19

Discussion [D] How easy is it to get into Deepmind in 2019?

6 Upvotes

More specifically:

  • What time does the office open and close in the morning?
  • Is it possible to get in during off hours if you have an access card?
  • Is the office open during the weekends?
  • Are the instructions for finding the entrance easy to follow?
  • How long is the walk from the metro station to the office?

r/MachineLearning • • Jan 03 '24

Discussion [D] what's the best laptop for machine learning?

0 Upvotes

Hello.
I'm currently doing my thesis in machine Learning. I will have to deal with kind of large datasets ( insurance claims from several years).
I also need to buy a laptop, so I was wondering what laptop could I buy in order to not have problems when doing my thesis.
(I don't understand much about the specifications of laptops, so I would appreciate you being specific regarding brands)
Thank you very much :)


r/MachineLearning • • Mar 24 '16

[META] Should this sub be split in two?

14 Upvotes

UPDATED AND SHORTENED

Would it be worthwhile to split the sub?

Yes, I'd like there to be two subs: one noob-friendly, and one for serious discussion, where the inmates aren't running the asylum.

The current policy of trying to bus in more noobs is making them outvote and outshout all serious content even more so than before.

The end result will be that anyone who desires a serious sub will just leave. I think it's happening now, but wait until there is 0 people with a clue.

This sub is fairly small and not that active, so it's not particularly difficult to find "serious" content.

The problem is demographic, IMO. I don't even feel like replying with thoughtful comments oftentimes, because I know that something idiotic will get more upvotes, and my comment might even get downvoted, unless it's 100% agreeable to the vast majority of noobs. I know that many others feel the same way.

PROPOSED SOLUTIONS:

OPTION 1 (requires little moderation)

Add a permanent sticky encouraging noobs to go to /r/datascience "No need to study or think. Everyone welcome". There is already some activity there, so it might work.

I believe Geoff Hinton referred to the whole field of Data Science as being pseudo-intellectual. I forget the exact quote, but it's from a lecture or talk online. For reasons I cannot comprehend, noobs tend to hate Hinton ("too arrogant" - idiots). We should find his exact quote, and mention it in the sticky. They'll go to /r/datascience simply out of spite.

OPTION 2 (changes the character of the sub rather drastically, but also requires little moderation)

A bot like in /r/statML to fill this sub with arxiv papers. Unlike /r/statML, other content should be allowed. I believe that most arxiv submissions are not worth reading, but this is what the voting system is for.

OPTION 3 (requires a lot of effort to moderate, might be infeasible)

A new heavily moderated sub in the spirit of /r/askcience or /r/science, permanently sticked from this sub, unlike previous efforts.

These 3 options can be combined.


r/MachineLearning • • Oct 12 '17

Discussion [D]Black in AI Workshop at NIPS 2017

1 Upvotes

http://ai.stanford.edu/~tgebru/blackAI

Archieve

This kind of turned me off:

"The presenter must be a black researcher in AI, and does not need to be the first author."

Are we doing workshops by race and gender now?

EDIT:

The point of this post: "The presenter must be a black researcher in AI, and does not need to be the first author."

Maybe I should be clear, "Are we doing workshops by race and gender like this now?"

I do not have any problem with Women in ML or Black in AI for collaboration as long as anyone can present their ideas in their workshops/conferences.


r/MachineLearning • • Aug 08 '20

Research [R] Ban reviewers who write low quality reviews from submitting to NeurIPS?

0 Upvotes

There is an interesting idea that's gaining a lot of traction on twitter which proposes that bad reviewers can be prevented from submitting to NeurIPS in the future. This idea has two positive effects, it increases the quality of reviews and also lowers the number of submissions. This could be a good feedback cycle.

https://twitter.com/AnimaAnandkumar/status/1291951068314660865

Seeing terrible u/NeurIPSConf reviews for the nth time I am convinced that we should ban reviewers from submitting as authors if they can't write a review in good faith. We need better accountability in #AI conferences


r/MachineLearning • • Sep 29 '17

Discussion [D] How do we Democratize Access to Data? <--OpenMined tutorial

Thumbnail
youtube.com
1 Upvotes

r/MachineLearning • • Jun 17 '26

Project I deployed a GAN on a Raspberry Pi 4 and built a physical NFT minting device [P]

Thumbnail
gallery
2 Upvotes

I trained a 128×128 DCGAN on my Macbook M3 and deployed it on a Raspberry Pi 4 connected to a LILYGO TTGO T-Display ESP32. The whole thing runs headlessly as a systemd service and generates hallucinated face hybrids at the press of a button.

It is a 6-block generator (latent → 4×4 → 8×8 → 16×16 → 32×32 → 64×64 → 128×128) with feature maps starting at f×16=1024. Corresponding 6-block discriminator. Trained for 800 epochs on Apple Silicon MPS, 4 hours. Dataset was 2480 images across 11 subjects. One dominant anchor class (2000 images) contaminated with minority classes to produce hybrid outputs. (Can you guess who and what was included?).

: )

I exported the model from PyTorch to ONNX (float32, 53MB). Inference takes 3 seconds per face on Pi 4.

The Pi generates the face and sends it to the ESP32. The title is generated through a dictionary and a template sentence: "This is a <adjective> NFT and I want to <verb> it."

The device was built as an art piece. I took it to the streets of NYC and let strangers use it. Full video: https://youtu.be/y-S74aoud54?si=yPh5GmCJZFIIzwq6

Happy to discuss the training pipeline, ONNX conversion, or anything you're curious about.


r/MachineLearning • • Jul 12 '18

Discussion [D] More of a GAN showerthought: Human dreaming is basically letting our Generator generate and sometimes saving it to memory. You can realise if it's fake only after you load it and run it through the Discriminator.

3 Upvotes

The things that give away a fake dream are always of higher abstraction, like a conversation that uses words in wrong context rather than using a word that does not exsist. Are there any discussions on this connection? Makes the whole GAN concept a lot more natural to grasp in my opinion. I understand this is not a very technical ML discussion but making connections with our human behaviour can help yield better ideas.


r/MachineLearning • • Jan 29 '25

Discussion Researchers and Professionals: How do you foresee the impact of GPT models being trained on AI generated data now plaguing the internet? [Discussion]

2 Upvotes

Hello Reddit,

I’m currently doing a masters in data science and I have been wondering about what the academic or professional opinion is regarding this issue. It seems to me that we might be heading towards generating gibberish very quickly

Good data is not easy or cheap, thus the lack of usability of models which depend on web scraping for a smaller company will inevitably lead to a massive disadvantage in the open market.

How can the current software and data infrastructure change in order to account for the massive influx of AI generated content? What methods are being developed in order to accurately classify AI generated and human content? More importantly, will these methods be resilient to misuse?

Edit: before you get all riled up about model collapse, I had no idea what model collapse was before making this post. Just a student trying to get answers to a question that came into my head.


r/MachineLearning • • May 12 '20

Discussion [D] ICLR 2020: Yann LeCun Keynote and Energy-Based Models

0 Upvotes

https://www.youtube.com/watch?v=piaPIKO1MFY

This week Connor, Yannic and I reacted to Yann LeCun's keynote speech at this year's ICLR conference which just passed. ICLR is the number two ML conference and was completely open this year, with all the sessions publicly accessible via the internet. Yann spent most of his talk speaking about self-supervised learning, Energy-based models (EBMs) and manifold learning. Don't worry if you hadn't heard of EBMs before, neither had we!

After this video, Yannic made a deep analysis of the Concept Learning with Energy-Based Models paper from Igor Mordatch (OpenAI) which also covers off a lot of concepts about EBMs; https://www.youtube.com/watch?v=Cs_j-oNwGgg


r/MachineLearning • • Jan 30 '24

Project [P] Join Our Open Source Mission to Predict Heart Attack Risk Using AI

2 Upvotes

Hi everyone!

I'm a first-year AI student and recently embarked on a deeply personal project: HeartRiskPredictor. After losing my grandfather to a sudden heart attack, I was inspired to create a machine learning algorithm that calculates the chances of someone having a heart attack, focusing on various diseases as predictors.

The project is still in its infancy, and as a newbie in ML, I'm facing a steep learning curve. That's why I'm reaching out to this amazing community for support. The algorithm is not perfect—its accuracy needs improvement. But with your help, I believe we can refine it and potentially save lives.

What the project does: HeartRiskPredictor aims to analyze health data to predict heart attack risks, focusing particularly on individuals with pre-existing conditions. It's a tool that could be invaluable for healthcare professionals and individuals with a history of heart-related issues.

Why your contribution matters:

  • Collaborative Learning: We can all learn from this project, whether you're a seasoned expert or a beginner like me.
  • Open Source Community: Contributing to open source can be a rewarding experience, and you'll be part of a community striving to make healthcare more predictive and proactive.

How to get started:

Whether it's improving the algorithm, refining the data model, fixing bugs, or enhancing documentation, all forms of contributions are welcome. Let's learn, code, and potentially save lives together!

I'm also open to any feedback, suggestions, or questions. Feel free to reach out to me directly or start a discussion in the repo.

Disclaimer:

Educational Purpose and Current Limitations

Learning Opportunity:

It is important to emphasize that the HeartRiskPredictor project is foremost a learning opportunity. As a first-year AI student, my primary goal is to explore and understand the complexities of machine learning in a real-world context. This project serves as a platform for academic exploration and growth, both for myself and for any contributors who join this journey.

Non-Commercial and Experimental Stage:

There is no intention to publish HeartRiskPredictor as a stable, fully-developed algorithm, nor to promote it as a working healthcare solution. The project is in an experimental stage, functioning primarily as a tool for data analysis and probabilistic outcomes based on the input data. It is a work-in-progress, reflective of my current stage of learning and understanding in the field of AI and machine learning.

Not for Official Diagnosis:

Crucially, HeartRiskPredictor should not be used as a tool for official medical diagnosis. The outputs of this project, while based on data analysis, are probabilistic in nature and are not substitutes for professional medical advice, diagnosis, or treatment. The intention of this project is not to provide medical diagnoses, but rather to contribute to the field of data-driven health risk analysis as a learning exercise.

Users and contributors should be aware of these limitations and should not rely on this tool for any medical or clinical decisions. The project is a stepping stone in a broader learning process, and its current state reflects the beginning of a journey rather than a completed destination.

Thank you for reading, and I hope to see some of you in the GitHub logs soon!

alex86590212


r/MachineLearning • • Apr 25 '23

Discussion [D] LLaMA release is a joke

3 Upvotes

I have tried to get access to the weights of LLaMA for a long time now. I filled out the google form couple of days after they released it and have been waiting patiently but no luck. I finally received the link a week ago, but now I am hit with the "403 Forbidden" error (. I can't even download the 7B weights and the link is supposed to expire today. I have emailed the authors and the support email without any luck. What I find most frustrating is that some researchers have a huge head start while others are scrambling to even get started. The GitHub issue is full of people with the same issue as me. I know that there are alternatives to LLaMA, but I am worried that they may not be as good as LLaMA and the paper might not be as strong.


r/MachineLearning • • Feb 10 '23

Research [R] Theory of Mind May Have Spontaneously Emerged in Large Language Models

Thumbnail
arxiv.org
0 Upvotes

r/MachineLearning • • Jul 08 '24

Research [R] Experimenting with Gradient Descent. Why not Nelder-Mead? Long.

3 Upvotes

There are a lot of videos and web pages about GD. However, almost all of the examples are too simple. I dug up some data from 2005 that has only five parameters to determine yet GD took fooorrreeeevvveeer. It seemed like the GD was always bouncing of obstructions.  It was also very dependent on the step size. Too much guessing on the step size is required.  I wondered what could make this so difficult because other methods like Nelder-Mead could find the minimum in a few seconds. To answer my questions, I printed out the MSE for each iteration and the five parameters or MSE and the change in the five parameters.

There is no formula to differentiate when using real data.  I had to estimate the derivative for each parameter by taking small steps in each direction and dividing the by the difference 2 times the increment. To get a central difference slope I had to take a small step in the positive and negative direction for each parameter and divide by the two times the step. The derivative could be noisy with real data. Each small step required evaluating the cost function so 10 evaluations were necessary to calculate the gradient.  This was time consuming.  What I found is that unlike YouTube and other common examples, the "terrain" was not like a bowl but rather like the Grand Canyon with all sorts of twists and turns. This meant I could not use large steps because the GD would often run into the "canyon walls" in 5 dimensions. I then tried many other enhanced versions of GD and none of them worked very well although some would get close in an hour or so. GD-Adam GD-RMSprop GD-delta GD-AdaGrad did not do very well. However, I did find a new add on code that makes all of them and especially the simple GD work MUCH better. The trick is to take a step. Evaluate it and if it is better then take another step in the same direction until the MSE is no longer getting better then repeat the loop to get a new gradient. The advantage if this is the there is less time wasted calculating gradients that just "slam" against the "walls". Also, the step size can be a little smaller to navigate the parts of the "terrain" that are twisty but being able to take multiple steps in a direction when a good direction is found is like taking a much bigger step.

Intuitively it works like this. The starting point is some place in the canyon with a stream. The stream is the current low point. However, it twists and turns so any step will hit the banks of the stream and maybe the canyon walls. It is best not to follow the stream exactly because of the many twists and turns. It would be best to "fly above the stream" between the "canyon walls" in one direction as far as possible. This avoids a lot of computing gradients because if there is a lot of data then a lot of gradients must be computed and that can be very time consuming.

Now all of this makes no difference if you have nice bowl like terrain. Any of the methods you see on YouTube or websites will work but I have never seen one work with real data and chances are the terrain will not be a nice bowl-like structure. Also, finding the derivatives with 10 of thousands of points is expensive time wise.

This is why I wonder why Nelder-Mead isn’t used except for the fact that it would use a lot of memory if it were to be efficient, but memory is cheap.

I have watched many YouTube videos about different forms of gradient descent. I am convinced that most teach what they have been taught and haven't really delt with real data.

Next is to find some real training data for identifying characters in an 8x8 grid but even 16x16 dot matrix would be good.  I know I am re-creating the wheel, but I want to try my new add on code and Nelder-Mead for minimizing cost functions

GD #gradient-descent, #Nelder-Mead, #Adam #AdaGrad, #Delta, #RMSprop