Summing up columns of arrays of different shapes in array of arrays- Python 3.x
up vote
1
down vote
favorite
I have an array that contains 2D arrays.
For each 2D array i want to sum up the columns and the result must be in column form.
I have a piece of code to do this, but I feel like I am not utilising numpy optimally. What is the fastest to do this?
My current code:
temp = [np.sum(l_i,axis=1).reshape(-1,1) for l_i in self.layer_inputs]
Sample Array:
array([
array([[ 0.48517904, -11.10809746],
[ 13.64104864, 5.77576326]]),
array([[16.74109924, -3.28535518],
[-4.00977275, -3.39593759],
[ 5.9048581 , -1.65258805],
[13.40762143, -1.61158724],
[ 9.8634849 , 8.02993728]]),
array([[-7.61920427, -3.2314264 ],
[-3.79142779, -2.44719713],
[32.42085005, 4.79376209],
[13.97676962, -1.19746096],
[45.60100807, -3.01680368]])
], dtype=object)
Sample Expected Result:
[array([[-10.62291842],
[ 19.41681191]]),
array([[13.45574406],
[-7.40571034],
[ 4.25227005],
[11.7960342 ],
[17.89342218]]),
array([[-10.85063067],
[ -6.23862492],
[ 37.21461214],
[ 12.77930867],
[ 42.58420439]]) ]
python arrays list performance numpy
edited Nov 12 at 12:57
jpp
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
add a comment |
up vote
1
down vote
favorite
I have an array that contains 2D arrays.
For each 2D array i want to sum up the columns and the result must be in column form.
I have a piece of code to do this, but I feel like I am not utilising numpy optimally. What is the fastest to do this?
My current code:
temp = [np.sum(l_i,axis=1).reshape(-1,1) for l_i in self.layer_inputs]
Sample Array:
array([
array([[ 0.48517904, -11.10809746],
[ 13.64104864, 5.77576326]]),
array([[16.74109924, -3.28535518],
[-4.00977275, -3.39593759],
[ 5.9048581 , -1.65258805],
[13.40762143, -1.61158724],
[ 9.8634849 , 8.02993728]]),
array([[-7.61920427, -3.2314264 ],
[-3.79142779, -2.44719713],
[32.42085005, 4.79376209],
[13.97676962, -1.19746096],
[45.60100807, -3.01680368]])
], dtype=object)
Sample Expected Result:
[array([[-10.62291842],
[ 19.41681191]]),
array([[13.45574406],
[-7.40571034],
[ 4.25227005],
[11.7960342 ],
[17.89342218]]),
array([[-10.85063067],
[ -6.23862492],
[ 37.21461214],
[ 12.77930867],
[ 42.58420439]]) ]
python arrays list performance numpy
edited Nov 12 at 12:57
jpp
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have an array that contains 2D arrays.
For each 2D array i want to sum up the columns and the result must be in column form.
I have a piece of code to do this, but I feel like I am not utilising numpy optimally. What is the fastest to do this?
My current code:
temp = [np.sum(l_i,axis=1).reshape(-1,1) for l_i in self.layer_inputs]
Sample Array:
array([
array([[ 0.48517904, -11.10809746],
[ 13.64104864, 5.77576326]]),
array([[16.74109924, -3.28535518],
[-4.00977275, -3.39593759],
[ 5.9048581 , -1.65258805],
[13.40762143, -1.61158724],
[ 9.8634849 , 8.02993728]]),
array([[-7.61920427, -3.2314264 ],
[-3.79142779, -2.44719713],
[32.42085005, 4.79376209],
[13.97676962, -1.19746096],
[45.60100807, -3.01680368]])
], dtype=object)
Sample Expected Result:
[array([[-10.62291842],
[ 19.41681191]]),
array([[13.45574406],
[-7.40571034],
[ 4.25227005],
[11.7960342 ],
[17.89342218]]),
array([[-10.85063067],
[ -6.23862492],
[ 37.21461214],
[ 12.77930867],
[ 42.58420439]]) ]
python arrays list performance numpy
edited Nov 12 at 12:57
jpp
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
I have an array that contains 2D arrays.
For each 2D array i want to sum up the columns and the result must be in column form.
I have a piece of code to do this, but I feel like I am not utilising numpy optimally. What is the fastest to do this?
My current code:
temp = [np.sum(l_i,axis=1).reshape(-1,1) for l_i in self.layer_inputs]
Sample Array:
array([
array([[ 0.48517904, -11.10809746],
[ 13.64104864, 5.77576326]]),
array([[16.74109924, -3.28535518],
[-4.00977275, -3.39593759],
[ 5.9048581 , -1.65258805],
[13.40762143, -1.61158724],
[ 9.8634849 , 8.02993728]]),
array([[-7.61920427, -3.2314264 ],
[-3.79142779, -2.44719713],
[32.42085005, 4.79376209],
[13.97676962, -1.19746096],
[45.60100807, -3.01680368]])
], dtype=object)
Sample Expected Result:
[array([[-10.62291842],
[ 19.41681191]]),
array([[13.45574406],
[-7.40571034],
[ 4.25227005],
[11.7960342 ],
[17.89342218]]),
array([[-10.85063067],
[ -6.23862492],
[ 37.21461214],
[ 12.77930867],
[ 42.58420439]]) ]
python arrays list performance numpy
python arrays list performance numpy
edited Nov 12 at 12:57
jpp
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
edited Nov 12 at 12:57
jpp
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
edited Nov 12 at 12:57
jpp
86.7k194998
edited Nov 12 at 12:57
jpp
86.7k194998
edited Nov 12 at 12:57
jpp
86.7k194998
86.7k194998
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
asked Nov 11 at 16:36
Vikhyat Agarwal
420214
420214
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
New answer
Given your stringent requirement for a list of arrays, there is no more computationally efficient solution.
Original answer
To leverage NumPy, don't work with a list of arrays: dtype=object is the hint you won't be able to use vectorised operations.
Instead, combine into one array, e.g. via np.vstack, and store split indices. If you need a list of arrays, use np.split as a final step. But this constant flipping between lists and a single array is expensive. Really, you should attempt to just store the splits and a single array, i.e. idx and data below.
idx = np.array(list(map(len, A))).cumsum()[:-1] # [2, 7]
data = np.vstack(A).sum(1)
answered Nov 11 at 16:49
jpp
86.7k194998
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). Themapbit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.
– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
|
show 3 more comments
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
New answer
Given your stringent requirement for a list of arrays, there is no more computationally efficient solution.
Original answer
To leverage NumPy, don't work with a list of arrays: dtype=object is the hint you won't be able to use vectorised operations.
Instead, combine into one array, e.g. via np.vstack, and store split indices. If you need a list of arrays, use np.split as a final step. But this constant flipping between lists and a single array is expensive. Really, you should attempt to just store the splits and a single array, i.e. idx and data below.
idx = np.array(list(map(len, A))).cumsum()[:-1] # [2, 7]
data = np.vstack(A).sum(1)
answered Nov 11 at 16:49
jpp
86.7k194998
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). Themapbit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.
– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
|
show 3 more comments
up vote
0
down vote
New answer
Given your stringent requirement for a list of arrays, there is no more computationally efficient solution.
Original answer
To leverage NumPy, don't work with a list of arrays: dtype=object is the hint you won't be able to use vectorised operations.
Instead, combine into one array, e.g. via np.vstack, and store split indices. If you need a list of arrays, use np.split as a final step. But this constant flipping between lists and a single array is expensive. Really, you should attempt to just store the splits and a single array, i.e. idx and data below.
idx = np.array(list(map(len, A))).cumsum()[:-1] # [2, 7]
data = np.vstack(A).sum(1)
answered Nov 11 at 16:49
jpp
86.7k194998
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). Themapbit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.
– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
|
show 3 more comments
up vote
0
down vote
up vote
0
down vote
New answer
Given your stringent requirement for a list of arrays, there is no more computationally efficient solution.
Original answer
To leverage NumPy, don't work with a list of arrays: dtype=object is the hint you won't be able to use vectorised operations.
Instead, combine into one array, e.g. via np.vstack, and store split indices. If you need a list of arrays, use np.split as a final step. But this constant flipping between lists and a single array is expensive. Really, you should attempt to just store the splits and a single array, i.e. idx and data below.
idx = np.array(list(map(len, A))).cumsum()[:-1] # [2, 7]
data = np.vstack(A).sum(1)
answered Nov 11 at 16:49
jpp
86.7k194998
New answer
Given your stringent requirement for a list of arrays, there is no more computationally efficient solution.
Original answer
To leverage NumPy, don't work with a list of arrays: dtype=object is the hint you won't be able to use vectorised operations.
Instead, combine into one array, e.g. via np.vstack, and store split indices. If you need a list of arrays, use np.split as a final step. But this constant flipping between lists and a single array is expensive. Really, you should attempt to just store the splits and a single array, i.e. idx and data below.
idx = np.array(list(map(len, A))).cumsum()[:-1] # [2, 7]
data = np.vstack(A).sum(1)
answered Nov 11 at 16:49
jpp
86.7k194998
edited Nov 12 at 12:59
answered Nov 11 at 16:49
jpp
86.7k194998
answered Nov 11 at 16:49
jpp
86.7k194998
answered Nov 11 at 16:49
jpp
86.7k194998
86.7k194998
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). Themapbit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.
– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
|
show 3 more comments
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). Themapbit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.
– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
Oh actually i need the shape of (n,1) for some operations ill be performing on the array. About the code, it uses a lot of numpy functions and looks pretty time consuming given expescially since you;re converting the map object to list and then to a numpy array. Could you run a time test of my loop based code vs your current code? (Also it would be better if you could also arrange the result into shape (n,1) arrays)
– Vikhyat Agarwal
Nov 11 at 16:55
@VikhyatAgarwal, Nope, you can time it yourself :). The
map bit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.– jpp
Nov 11 at 17:00
@VikhyatAgarwal, Nope, you can time it yourself :). The
map bit is as efficient as you can get as your input is a list. I've updated my answer for your unusual shape.– jpp
Nov 11 at 17:00
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
Actually I tried the time test, but it takes -on an average- 3 to 4 times more time than my solution (even without the last line in your code which reshapes the array, it takes twice as much). I am looking for a solution which is be computationally efficient.
– Vikhyat Agarwal
Nov 12 at 12:53
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
@VikhyatAgarwal, Sure, see my update. If you choose to ignore the advice to avoid a list of arrays to begin with, there's little more I (or anyone else) can help.
– jpp
Nov 12 at 12:55
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
What's the problem with list of arrays? Do you prefer an array of arrays instead? That's fine by me.
– Vikhyat Agarwal
Nov 12 at 12:57
|
show 3 more comments
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53250846%2fsumming-up-columns-of-arrays-of-different-shapes-in-array-of-arrays-python-3-x%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
