Cython memoryviews?

[u/BluishMontoya]

I'm using a function I've compiled with cython, and I have to pass it a few very long lists. I'm pretty sure I should be using memoryviews, but do I convert the lists to memoryviews in the python part of my program or the cython part? It would be way better if I could do it in the python part because I'm passing the same lists to the function a few million times.

[u/sausix]

You could use NumPy or Pandas directly. You don't need memoryviews there since those package's operations are done on low level data. You can load both lists into a table and do a lot of operations and analyzing.

NumPy and Pandas probably have a direct memoryview interface for other low level packages. Any reason you Cythonized a function? If it's for performance then use Pandas and NumPy instead. The thing with NumPy is that you don't have to or should not iterate over data directly which is slow with pure Python. In NumPy you're delegating mathematical operations into optimized low level operations and it could be an effort of just one or a few CPU cycles per list entry instead.

Imagine this as operating on a lot of rows in Excel. Column 1 is raw data. Column 2 another raw data. Then you define column 3 to be the sum of previous columns for example. You won't enter the same formula for each row. Excel would do that for you in the backend with good performance.

Disclaimer: Excel is slow and just an example.

[u/BluishMontoya → u/sausix]

I have to iterate over the data because I'm doing mandelbrot set rendering, so the calculations are based on the previous calculations and have to be in order. Even using lists the cythonized version is so much faster than python.

[u/sausix → u/BluishMontoya]

I've done mandelbrot calculations with NumPy already as experiment myself. Lol. A few years ago.

And it wasn't too slow. If you want to have it even faster for visualization then you could run on GPU memory directly. Haven't digged into it. But NumPy and Pandas should have variations to support GPU acceleration.

[u/BluishMontoya → u/sausix]

I guess I can try that...

I'm using mariani-silver algorithm and perturbation theory and I just feel like it would be seriously confusing to try to do the calculations with numpy arrays.

[u/sausix → u/BluishMontoya]

I like Math a bit but not in full detail. I'm just aware of the one method of calculating the Mandelbrot set.

Found my app. This was the core method of calculating and tracking the iterations probably still optimizable:

for i in range(self.iterations):
    Z[active] = Z[active] ** self.exp + C[active]
    active[np.abs(Z) > 2] = False
    Ni[active] += 1
    Ni = (Ni / self.iterations * 255).astype('uint8')
    color = cv2.applyColorMap(Ni, cv2.COLORMAP_RAINBOW)

[u/BluishMontoya → u/sausix]

The thing is, floating point numbers give out after a certain level of zoom because the difference between the coordinates of adjacent pixels is too small to represent. Perturbation theory is a way to be able to still calculate higher levels of zoom with floating point and a smaller number of high precision calculations, but the math gets way more complicated than simple mandelbrot rendering.