Vfrdtyky

I am trying to multiply all the row values and column values of a 2 dimensional numpy array with an explicit for-loop:

product_0 = 1

product_1 = 1

for x in arr:

   product_0 *= x[0]

   product_1 *= x[1]

I realize the product will blow up to become an extremely large number but from my previous experience python has had no memory problem dealing very very extremely large numbers.

So from what I can tell this is a problem with numpy except I am not storing the gigantic product in a numpy array or any numpy data type for that matter its just a normal python variable.

Any idea how to fix this?

Using non inplace multiplication hasn't helped product_0 = x[0]*product_0

Python int are represented in arbitrary precision, so they cannot overflow. But numpy uses C++ under the hood, so the highest long signed integer is with fixed precision, 2^63 - 1. Your number is far beyond this value, having in average ((716-1)/2)^86507).

When you, in the for loop, extract the x[0] this is still a numpy object. To use the full power of python integers you need to clearly assign it as python int, like this:

product_0 = 1

product_1 = 1

for x in arr:

    t = int(x[0])

    product_0 = product_0 * t

and it will not overflow.

Following your comment, which makes your question more specific, your original problem is to calculate the geometric mean of the array for each row/column. Here the solution:

I generate first an array that has the same properties of your array:

arr = np.resize(np.random.randint(1,716,86507*2 ),(86507,2))

Then, calculate the geometric mean for each column/row:

from scipy import stats



gm_0 = stats.mstats.gmean(arr, axis = 0)

gm_1 = stats.mstats.gmean(arr, axis = 1) 

gm_0 will be an array that contains the geometric mean of the xand y coordinates. gm_1 instead contains the geometric mean of the rows.

Hope this solves your problem!

You say

So from what I can tell this is a problem with numpy except I am not storing the gigantic product in a numpy array or any numpy data type for that matter its just a normal python variable.

Your product may not be a NumPy array, but it is using a NumPy data type. x[0] and x[1] are NumPy scalars, and multiplying a Python int by a NumPy scalar produces a NumPy scalar. NumPy integers have a finite range.

While you technically could call int on x[0] and x[1] to get a Python int, it'd probably be better to avoid needing such huge ints. You say you're trying to perform this multiplication to compute a geometric mean; in that case, it'd be better to compute the geometric mean by transforming to and from logarithms, or to use scipy.stats.mstats.gmean, which uses logarithms under the hood.

Numpy is compiled for 32 bit and not 64 bit , so while Python can handle this numpy will overflow for smaller values , if u want to use numpy then I recommend that you build it from source .

Edit
After some testing with

import numpy as np

x=np.abs(np.random.randn(1000,2)*1000)

np.max(x)

prod1=np.dtype('int32').type(1)

prod2=np.dtype('int32').type(1)

k=0

for i,j in x:

    prod1*=i

    prod2*=j

    k+=1

    print(k," ",prod1,prod2)



1.797693134e308 is the max value (to this many digits my numpy scalar was able to take)

if you run this you will see that numpy is able to handle quite a large value but when you said your max value is around 700 , even with a 1000 values my scalar overflowed.

As for how to fix this , rather than doing this manually the answer using scipy seems more viable now and is able to get the answer so i suggest that you go forward with that

from scipy.stats.mstats import gmean

x=np.abs(np.random.randn(1000,2)*1000)

print(gmean(x,axis=0))