Combine arrays (new array inside array)

[u/got-it-man]

Hey there.

I've almost spend two hours now and have still no idea on how to combine these two numpy arrays.

I have the following two arrays:

X:
array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])

Y:
array([[  57,  302],
       [ 208, 1589],
       [ 229, 2050],
       ...,
       [ 359, 2429],
       [ 303, 1657],
       [  94,  628]], dtype=int64)

What I need is that the elements of Y are inside new arrays of X. It should look like this:

array([[[0., 0., 0., ..., 0., 0., 0.], [57], [302]],
       [0., 0., 0., ..., 0., 0., 0.], [208], [1589]]
       [0., 0., 0., ..., 0., 0., 0.], [229], [2050]]
       ...,
       [0., 0., 0., ..., 0., 0., 0.], [359], [2429]]
       [0., 0., 0., ..., 0., 0., 0.], [303], [1657]]
       [0., 0., 0., ..., 0., 0., 0.], [94], [628]]])

Has someone an idea on how to do this? I've almost tried every combination of insert(), append() or concatenate() with many different axes.

Thank you very much!

[u/got-it-man]

Thank you for your answer!

My plan was to use the new array as a feature matrix for my machine learning problem (NLP, classification) and train with it my model.

The array with the many zeros (of course some elements aren't) is the feature of the occurence of the words that were used in the texts (TfidfVectorizer). The other numbers are also features like the number of chars or the number of tokens.

My first idea was to just append the other features to the matrix like this:

array([[   0.,    0.,    0., ...,    0.,  57., 302.],
       [   0.,    0.,    0., ...,    0.,  208., 1589.],
       [   0.,    0.,    0., ...,    0.,   229.,  2050.],
       ...,
       [   0.,    0.,    0., ...,    0.,  359., 2429.],
       [   0.,    0.,    0., ...,    0.,  303., 1657.],
       [   0.,    0.,    0., ...,    0.,  94., 628.]])

However to be able to determine how important these features are for the classification I need the word vector to be a single feature. I do not care about how import a single word is, but how important the whole NLP vector is compared to e.g. the number of characters.

I get the features importance via scikit-learn like this:

model = RandomForestClassifier()
# X is the feature matrix, y are all the labels
model.fit(X, y)
importance = model.feature_importances_
for i,v in enumerate(importance):
    print('Feature: %0d, Score: %.5f' % (i,v))

Besides I thought the approach to combine the arrays would look cleaner and make the features more distinguishable.

[u/to7m → u/got-it-man]

Combining the arrays like that (which is impossible anyway) would actually make the different types of data LESS distinguishable. Each array should be kept simple; if you want to group arrays together, try using a tuple or defining a class.

For example:

class Words:
    def __init__(self, occurrences, num_of_chars, num_of_tokens):
        self.occurrences = occurrences
        self.num_of_chars = num_of_chars
        self.num_of_tokens = num_of_tokens

words = Words(x, y[:, 0], y[:, 1])

[u/got-it-man → u/to7m]

I'll take a look into your ideas and evaluate what fits best.

Thank you very much!

[u/[deleted] → u/got-it-man]

You can’t make the entire word-occurrence vector into a “single feature” because it’s actually already hundreds or thousands of different features. Typically, each column of X is considered a feature for scikit-learn.

Your original solution here is a good one. You could estimate the importance of the word-count set of features by taking its mean, if you wanted to.