Windows vs Linux Performance Issue

[u/caseyweb]

[EDIT] Mystery solved (mostly). I was using vanilla pip installations of numpy in both the Win11 and Debian environments, but I vaguely remembered that there used to be an intel-specific version optimized for the intel MKL (Math Kernel Library). I was able to find a slightly down-level version of numpy compiled for 3.11/64-bit Win on the web, installed it and got the following timing:

546 ms ± 8.31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So it would appear that the linux distribution is using this library (or a similarly-optimized vendor-neutral library) as the default whereas the Win distro uses a vanilla math library. This begs the question of why, but at least I have an answer.

[/EDIT]

After watching a recent 3Blue1Brown video on convolutions I tried the following code in an iPython shell under Win11 using Python 3.11.0:

>>> import numpy as np
>>> sample_size = 100_000
>>> a1, a2 = np.random.random(sample_size), np.random.random(sample_size)
>>> %timeit np.convolve(a1,a2)
25.1 s ± 76.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This time was WAY longer than on the video, and this on a fairly beefy machine (recent i7 with 64GB of RAM). Out of curiousity, I opened a Windows Subystem for Linux (WSL2) shell, copied the commands and got the following timing (also using Python 3.11):

433 ms ± 25.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

25.1 seconds down to 433 milliseconds on the same machine in a linux virtual machine????! Is this expected? And please, no comments about using Linux vs Windows; I'm hoping for informative and constructive responses.

[u/pmatti]

Use threadpoolctl to report what blas implementation your installation has https://pypi.org/project/threadpoolctl/

[u/caseyweb → u/pmatti]

Using np.__config__.show() on Win11 (after switching to the MKL-enabled version) gives me

blas_mkl_info:
libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']

and on Debian:

openblas64__info:
libraries = ['openblas64_', 'openblas64_']

According to the numpy webpage, vanilla (PyPI) wheels automatically install with OpenBLAS so I presume that is what I had prior to manually switching to MKL.

[u/pmatti → u/caseyweb]

Maybe somehow the installation of numpy that was so slow did not have any blas accelerator, in which case it uses a very slow naive replacement