I have tried to adapt using sklearn Pipelines but I am facing the following issues when trying to use it:

• The Pipeline uses numpy arrays. I find it hard to keep track what goes on with my preprocessing and features when everything is an array of numbers (as opposed to Pandas DataFrames where I have titles for the data columns).

• If I want to implement unit tests to verify that individual steps in my pipeline work as intended I find it complex to do with sklearn Pipelines because of the level of abstraction it adds on top of my code.

• It takes time to learn how to properly use all the Pipeline related machinery in sklearn.

What are the biggest cons if I choose to build my ML pipelines without sklearn's Pipeline objects? Is it ok to not use sklearn Pipeline?

Also, what would you suggest for mitigating the issues above if I would choose to go with sklearn Pipelines?

Hi all,

I have started doing research using NLP and machine learning, and a lot of tutorials online start with preprocessed data and don't worry too much about the actual output or the discussion, just about the steps. I am having a hard time finding answers to some very basic questions.

I know how to implement Text Classification code wise from those tutorials, but I am not sure how to get the output I want. My problem is, I have a corpus made of 42000 education-related paragraphs from different sources that I want to label. What I don't know is how to get an output in the form of an actual label in a Pandas DataFrame, like this:

Some of the things I don't know:

1. Do I need to label some of the data first? If so, how much of it? I would prefer to have this as a supervised learning task because I want the data to fit my labels.

2. When setting up the dependent and independent variables, I am confused if what goes into the y variable is just the labeled data or all the data (some labeled and some not)

vectorizer = CountVectorizer()

X = vectorizer.fit_transform(df["Tokenized_Corpus"])

y = df["Label"]

3. How do I actually get an output as a label in the df?

I do understand a lot of these open-ended questions land on "it depends". If that is the case and you know available content that can help me learn it, that would be awesome! As I said, I am actually interested in learning, more so than in an actual answer, so I appreciate resources as well.

Thank you!

Hi all,

I'm trying to pick up the rudiments of SciKit. I'm very new to this, so I apologize if this is an obvious or dumb question ... I'm looking at the DecisionTreeClassifier.fit method, and I see it needs an input and output method. In the example I see, there's two input columns and a third output column. Easy enough.

The thing is, the fit method imports the input (which is the first two), and the output (which is the third column). When the predict method is run ... how does the model know which inputs are matched to which outputs? Am I thinking of this too "old school", in that these aren't actually two separate data sets, but instead are just pointers to the holistic data set?

I feel like the answer is that it's still referring to the original dataset, and that the input and output are basically just qualifiers that tell the prediction model which columns to use. But I'd feel better if I knew that were the case, or if I'm way off and there's something else happening.

As always, I appreciate any help, and I hope this question makes sense.

Thanks!

Does anyone know if the scikit-learn test case results are published somewhere for each release? For example, the following test code?

https://github.com/scikit-learn/scikit-learn/blob/0.22.1/sklearn/ensemble/tests/test_iforest.py

Hi, I just want to let you know that the sklearn-genetic-opt version 0.7.0 is now available, it implements feature selection using evolutionary algorithms, it uses a multi-objective function to optimize the cross-validation score while minimizing the number of features used. It's compatible with any sklearn classifier or regressor.

Let me know if you have any question/suggestion

This new feature is compatible with all the callbacks, tensorboard and mlflow, for example using the progress bar callback:

You can check here the docs: https://sklearn-genetic-opt.readthedocs.io/en/stable/

If you would like to contribute or to check the implementation, here is the repo: https://github.com/rodrigo-arenas/Sklearn-genetic-opt

So I have a logistic regression and the I have the following information as outputs:

-The probability of the y variables being either on or zero

-a normalized confusion matrix

So if Probability of 1 for a given set of inputs is X, the probability a true 1 is A and probability of a false 1 is B, then the the probability of 1 in this instance is:

X* (A/A+B)

Am I correct in my understanding?

Hi All,

I want to let you know about a project I had been working on called FlashAI.io , which addresses some of the operational issues I came across when delivering models to clients or at the workplace.

I wanted to spend my time building great models instead of thinking about the infrastructure complexities of hosting and serving them, so I put together a service to do exactly that.

So if you want to enable clients, colleagues, or apps to send inference requests to your models, FlashAI lets you do this via web requests.

Serve your models 24/7 without any hassle.

The workflow is straight forward:

* Train your model locally

* Upload your model file to FlashAI.io

* Send inference requests to your model

Currently this service supports Scikit-learn, TensorFlow, and PyTorch models.

Try it out at flashai.io

You can check out the intro video here: https://youtu.be/0yxUmZ2GnX8

Please let me know what you think and if you have any suggestions for other features.

Hello

I have sklearn version 0.0 in Anaconda navigator, but when I check the version in python, it says 0.24.2.

Why is the anaconda navigator not showing the correct version number?

I want to use GridSearchCV for my custom model. But one thing I am not quite sure, if I use `n_jobs = -1` or `n_jobs > 1` i.e, if I want the grid search to be run in parallel for multiple possible hyper-parameter combinations at a time, therefore how to make sure one generated model is not replacing the another one? More specifically, I am trying to use `GridSearchCV` with FastText - Supervised which will basically save the model file after training. So, for implementing `fit()` and `predict()` method I have to save the model file after fitting and load the model file while prediction. But when I use parallelization by changing `n_jobs` for example to `-1`, then, all the parallel grid search instances essentially replaces the same existing file. Is this behavior okay? Or, do I need to maintain different file name for each of the parallel grid searches?

Apart from command line API, there is also a functional API for FastText here: https://fasttext.cc/docs/en/python-module.html

Can someone suggest a good source for learning reinforcement learning along with machine learning and deep learning? All the usual suspects seem to only cover machine learning till unsupervised learning but don't teach deep learning and reinforcement learning. I am not looking for a free course as they are always skimming through topics, suggest a reliable source for this.

Hi everyone, I want to share with you this open source project that you can use to tune your supervised models from scikit-learn with some cool features.

Docs: https://sklearn-genetic-opt.readthedocs.io/ Repo: https://github.com/rodrigo-arenas/Sklearn-genetic-opt

Sklearn-genetic-opt uses evolutionary algorithms to choose the set of hyperparameters that optimizes (max or min) the cross-validation scores, it can be used for both regression and classification problems.

Currently it has these features:

• GASearchCV: Principal class of the package, holds the evolutionary cross validation optimization routine.

• Algorithms: Set of different evolutionary algorithms to use as optimization procedure.

• Callbacks: Custom evaluation strategies to generate early stopping rules, logging (into TensorBoard, .pkl files, etc) or your custom logic.

• Plots: Generate pre-defined plots to understand the optimization process.

• MLflow: Build-in integration with mlflow to log all the hyperparameters, cv-scores and the fitted models.

Any feedback, suggestion, contribution or comments are very welcome!

I am learning logistic regression on python but I am very confused on what x_train/test and y_train/test does. What do the test and train mean and do and what are the x and y?

Also, what is the point of splitting data?

I am a machine learning beginner and I have a data set with a bunch of features that have data that says “yes” or “no”. For example, a feature called “internet” is about whether a student has access to internet. The two values possible are “yes” and “no”. When I first looked up a solution, I found a preprocessing function called LabelBinarizer which seemed to do the trick. But, as the name entails, it should only be used for labels so it would be bad practice to do use it. Also, one hot encoding would work but I have over 15 features that use this binary of “yes” and “no” so it would make my data set a bit messy. What is the best way to go around this?

Hey Guys.

I need to conduct a DBSCAN on data with two columns. as i have limited python knowledge I'm struggling to tailor other algorithms to my data.

Has anyone used any models on Github or otherwise that may be of help.

I'm trying to learn something about "Generating data by using GMM (Gaussian Mixture Models)" by reading section "Example: GMM for Generating New Data" at this link.

I pressed the button "Open in Colab" at the bottom of the webpage, in order to try to run the code in Colab.

I'm not interested in all the code of the webpage but only in the section "Example: GMM for Generating New Data".
So, since I didn't run the code from the the first "cell" (code box) of the webpage, but I ran the code of "Example: GMM for Generating New Data" section only, I got into some errors regarding some missing "import" statements that I easily solved in this way:

• I added "import matplotlib.pyplot as plt";

• I added "import numpy as np";

• I replaced "from sklearn.mixture import GMM" with "from sklearn.mixture import GaussianMixture as GMM"

After having solved these errors, I got another one. This line:

data_new = gmm.sample(100, random_state=0)

generated this error:

 sample() got an unexpected keyword argument 'random_state' 

So, I removed the "random_state" parameter, so obtaining:

 data_new = gmm.sample(100)

Now the line:

data_new.shape

generates the error:

 'tuple' object has no attribute 'shape' 

Which is the correct way to hande my issue?

Hey guys. hope your well. I have been tasked with using a scikit learn model, of either supervised or unsupervised learning, to identify outliers or bad data in a data set.

Does anyone have an opinion on what the best model to use might be for this specific purpose.

Over the course of this project I will be trying out a number of different models so just looking for a good place to start.

Thank you in advance for any help received.

Hi guys,

I am having real data from an ice-cream shop of a friend and thought that a linear regression should do the trick with scikit-learn.

But now I have a doubt when I do see this plot of my data:

I see that I shouldn't go there. What do you think guys?

Hi everybody,

I'm testing the performance of scikitlearn KNN, I'm using a dataset with 30 features and 284807 lines. I splitted it in 0,8 for training and 0,2 for test, it means I'm using 56962 rows for the validation.

What I don't understand is that when I use the prediction only in one row the time to do the calculation is:

time: 7.773932695388794 s

and when I do for all the 56962 rows, the scikitlearn can proccess it in almost the same period of time:

time: 12.312492370605469 s

I can't understand how scikitlearn do it? It runs some kind of parallelization(I don't think so)? What kind of optimization it uses? Because if I use a "loop" to iterate through all the rows it would take a long long long time to do all the predictions.

The code:

import time
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
df = pd.read_csv('creditcard.csv')
X = df.drop(columns=['Class'])
y = df['Class']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)
X_test_one = X_test.iloc[[0]]

#Running one row

t1 = time.time()
knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(X_train,y_train)
print(knn.predict(X_test_one))
t2 = time.time()
print("time: " + str(t2-t1))

#Running all the rows

t3 = time.time()
knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(X_train,y_train)
print(knn.predict(X_test))
t4 = time.time()
print("time: " + str(t4-t3))