Hacker News Reader

Comment by martopix | Hacker News Reader

I was surprised to see that allocating with numpy is actually 2x slower than allocating a list:

  In [4]: length = 10000

  In [5]:   %%timeit
   ...:   #preallocating list
   ...:   pre = [None] \* length
   ...:   for i in range(length):
   ...:       pre[i] = i
   ...: 
  346 µs ± 10.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

  In [6]: %%timeit
   ...: pre = np.empty(length, dtype=int)
   ...: for i in range(length):
   ...:     pre[i] = i
   ...:

  688 µs ± 20.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

View on HN

It's that the array access of numpy is surprisingly slow compared to a Python list:

  lst = [0]
  %timeit lst[0] = 1
  33.9 ns ± 0.0415 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
  
  arr = np.empty(1, dtype=int)  
  %timeit arr[0] = 1
  73.4 ns ± 0.306 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

You are measuring two things together: allocation and an initialization inside a loop.

I would try to separate them (I.e. measure [None]*length and np.empty(length)

  In [1]: length = 10_000

  In [2]: %%timeit
     ...: pre = [None] * length
     ...: 
     ...: 
  38.7 µs ± 128 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
  
  In [3]: %%timeit
     ...: pre = np.empty(length, dtype=int)
     ...: 
     ...: 
  1.32 µs ± 6.59 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

Here was a weird one for me, why is numba jitted initialization twice as fast as np.arange, which presumably has a C implementation?

  In [1]: import numpy as np
  
  In [2]: %timeit np.arange(10000, dtype=np.int64)
  3.03 µs ± 94.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
  
  In [3]: import numba
  
  In [4]: @numba.jit
     ...: def alloc_np_jit(length):
     ...:     pre = np.empty(length, dtype=np.int64)
     ...:     for i in range(length):
     ...:         pre[i] = i
     ...:     return pre
     ...:
  
  In [5]: %timeit alloc_np_jit(10000)
  1.56 µs ± 12.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Reply on news.ycombinator.com