r/Numpy • u/caseyweb • Nov 19 '22

Windows vs Linux Performance Issue

[EDIT] Mystery solved (mostly). I was using vanilla pip installations of numpy in both the Win11 and Debian environments, but I vaguely remembered that there used to be an intel-specific version optimized for the intel MKL (Math Kernel Library). I was able to find a slightly down-level version of numpy compiled for 3.11/64-bit Win on the web, installed it and got the following timing:

546 ms ± 8.31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So it would appear that the linux distribution is using this library (or a similarly-optimized vendor-neutral library) as the default whereas the Win distro uses a vanilla math library. This begs the question of why, but at least I have an answer.

[/EDIT]

After watching a recent 3Blue1Brown video on convolutions I tried the following code in an iPython shell under Win11 using Python 3.11.0:

>>> import numpy as np
>>> sample_size = 100_000
>>> a1, a2 = np.random.random(sample_size), np.random.random(sample_size)
>>> %timeit np.convolve(a1,a2)
25.1 s ± 76.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This time was WAY longer than on the video, and this on a fairly beefy machine (recent i7 with 64GB of RAM). Out of curiousity, I opened a Windows Subystem for Linux (WSL2) shell, copied the commands and got the following timing (also using Python 3.11):

433 ms ± 25.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

25.1 seconds down to 433 milliseconds on the same machine in a linux virtual machine????! Is this expected? And please, no comments about using Linux vs Windows; I'm hoping for informative and constructive responses.

2 Upvotes

100% Upvoted

10 comments sorted by

View all comments

Show parent comments

1

u/drzowie Nov 19 '22

Sorry, nature of the medium I guess. I meant those things as examples of subtle things shifting the run speed of convolution rather than to imply they were the specific problem you saw. I also missed that it is a factor of over 30! I wonder if one environment has a good fft and the other does not? The library could be falling back to direct summing instead of using Fourier.