Which numpy operations copy and which mutate?
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
asked Nov 11 at 4:08
scanny
9,27211841
add a comment |
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
asked Nov 11 at 4:08
scanny
9,27211841
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
asked Nov 11 at 4:08
scanny
9,27211841
Is there a general rule of thumb for knowing which operations on a numpy.ndarray produce a copy of the values and which mutate them in-place?
I'm pretty new to numpy and I'm sure I'll learn the hard way eventually, but I was wondering if there were general principles driving mutability that might help speed my learning.
numpy
numpy
asked Nov 11 at 4:08
scanny
9,27211841
asked Nov 11 at 4:08
scanny
9,27211841
asked Nov 11 at 4:08
scanny
9,27211841
asked Nov 11 at 4:08
scanny
9,27211841
asked Nov 11 at 4:08
scanny
9,27211841
9,27211841
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resizendarray.sort- All of the in-place arithmetic operators (eg
+=,*=,/=, etc) numpy.fill_diagonalnumpy.random.shufflendarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswapnumpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).
There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
answered Nov 11 at 4:30
tel
3,3011427
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resizendarray.sort- All of the in-place arithmetic operators (eg
+=,*=,/=, etc) numpy.fill_diagonalnumpy.random.shufflendarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswapnumpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).
There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
answered Nov 11 at 4:30
tel
3,3011427
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resizendarray.sort- All of the in-place arithmetic operators (eg
+=,*=,/=, etc) numpy.fill_diagonalnumpy.random.shufflendarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswapnumpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).
There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
answered Nov 11 at 4:30
tel
3,3011427
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
up vote
2
down vote
accepted
up vote
2
down vote
accepted
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resizendarray.sort- All of the in-place arithmetic operators (eg
+=,*=,/=, etc) numpy.fill_diagonalnumpy.random.shufflendarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswapnumpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).
There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
answered Nov 11 at 4:30
tel
3,3011427
Functions that mutate in place
Relatively few numpy functions mutate in place. For the most part, numpy functions return array views when they can, and copies when they can't.
Here's an exhaustive list (trawled from the docs) of functions/methods that mutate in place:
ndarray.resizendarray.sort- All of the in-place arithmetic operators (eg
+=,*=,/=, etc) numpy.fill_diagonalnumpy.random.shufflendarray.partition
and here's a list of the functions/methods that can optionally mutate in place:
ndarray.byteswapnumpy.nan_to_num
Certain assignments will also mutate an array in place. You can change the values in an array by assigning to a slice (eg arr[...] = 1 will set every value in an array to 1), and you can reshape an array by assigning a new shape directly to .shape, eg arr.shape = (2,3) (won't always work, see notes here).
There's also some functions that support an out keyword arg. These functions will behave as mutators if you pass the same array as both input and out.
Fair warning, I may have missed one or two mutators that weren't clearly marked in the docs. In any case, the list is short, so there's not much to memorize.
Notes on view vs copy return values
One of the goals of the numpy devs over the last few years seemingly has been to make it more common for the numpy functions and the ndarray methods to return views instead of copies. At this point, it's reasonably safe to assume that if a numpy function/method can return a view, it will do so by default.
For example, ndarray.flatten and ndarray.ravel do the same thing (returned a flattened array). However, the docs for ndarray.flatten explicitly say that it will return a copy, whereas the docs for ndarray.ravel say that it will return a copy only if absolutely necessary.
In live code, as a rule of thumb you can always check if an operation produced a view or a copy by comparing the id of the .base of your result to the id of original array. For example:
arr = np.array([[1, 2],
[3, 4],
[5, 6]])
arrflat = arr.flatten()
assert arrflat.base is not arr
arrravel = arr.ravel()
assert arrravel.base is arr
answered Nov 11 at 4:30
tel
3,3011427
edited Nov 11 at 5:14
answered Nov 11 at 4:30
tel
3,3011427
answered Nov 11 at 4:30
tel
3,3011427
answered Nov 11 at 4:30
tel
3,3011427
3,3011427
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
I'm not sure he's asking about views vs. copies. Both are new array objects.
– hpaulj
Nov 11 at 4:40
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53245763%2fwhich-numpy-operations-copy-and-which-mutate%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There aren't many true inplace numpy operations Only ones I can think of are direct shape assignment, an inplace sort, resize, += style math, and indexed assignment..
– hpaulj
Nov 11 at 4:44
No obvious rule of thumb, but the list of mutators is short. I think I have pretty much the full list in my answer, so just consult back here if ever confused, I guess?
– tel
Nov 11 at 5:16
@tel I think the rule of thumb for me is "almost never mutates" and also "almost always returns a view", which is very helpful for me here starting out. The underlying concern is avoiding inadvertently changing something I want to treat as immutable (because other methods will use it too) and avoiding unnecessary performance penalties. I hadn't made the view vs. copy distinction before your answer, so that is also a very useful insight :) As I take it, an array is like a map into a "base", and it's lightweight to make a new map on an existing base, which is essentially what creating a view is.
– scanny
Nov 11 at 7:46
Pretty much. The only thing I'd add is that some arrays are "bases", in the sense that they have formal ownership of their underlying memory buffer. Thankfully, whether or not an array owns its own data makes almost no difference in the vast majority of cases (though, another rule of thumb: operations on arrays that own their own data are "more likely" to return a view rather than a copy (though there's also plenty of cases when an op on a view will indeed return a view)).
– tel
Nov 11 at 8:00