Use pandas.shift() within a group
0
I have a dataframe with panel data, let's say it's time series for 100 different objects:
object period value
1 1 24
1 2 67
...
1 1000 56
2 1 59
2 2 46
...
2 1000 64
3 1 54
...
100 1 451
100 2 153
...
100 1000 21
I want to add a new column prev_value that will store previous value for each object:
object period value prev_value
1 1 24 nan
1 2 67 24
...
1 99 445 1243
1 1000 56 445
2 1 59 nan
2 2 46 59
...
2 1000 64 784
3 1 54 nan
...
100 1 451 nan
100 2 153 451
...
100 1000 21 1121
Can I use .shift() and .groupby() somehow to do that?
python pandas pandas-groupby
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
add a comment |
0
I have a dataframe with panel data, let's say it's time series for 100 different objects:
object period value
1 1 24
1 2 67
...
1 1000 56
2 1 59
2 2 46
...
2 1000 64
3 1 54
...
100 1 451
100 2 153
...
100 1000 21
I want to add a new column prev_value that will store previous value for each object:
object period value prev_value
1 1 24 nan
1 2 67 24
...
1 99 445 1243
1 1000 56 445
2 1 59 nan
2 2 46 59
...
2 1000 64 784
3 1 54 nan
...
100 1 451 nan
100 2 153 451
...
100 1000 21 1121
Can I use .shift() and .groupby() somehow to do that?
python pandas pandas-groupby
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
add a comment |
0
0
0
I have a dataframe with panel data, let's say it's time series for 100 different objects:
object period value
1 1 24
1 2 67
...
1 1000 56
2 1 59
2 2 46
...
2 1000 64
3 1 54
...
100 1 451
100 2 153
...
100 1000 21
I want to add a new column prev_value that will store previous value for each object:
object period value prev_value
1 1 24 nan
1 2 67 24
...
1 99 445 1243
1 1000 56 445
2 1 59 nan
2 2 46 59
...
2 1000 64 784
3 1 54 nan
...
100 1 451 nan
100 2 153 451
...
100 1000 21 1121
Can I use .shift() and .groupby() somehow to do that?
python pandas pandas-groupby
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
I have a dataframe with panel data, let's say it's time series for 100 different objects:
object period value
1 1 24
1 2 67
...
1 1000 56
2 1 59
2 2 46
...
2 1000 64
3 1 54
...
100 1 451
100 2 153
...
100 1000 21
I want to add a new column prev_value that will store previous value for each object:
object period value prev_value
1 1 24 nan
1 2 67 24
...
1 99 445 1243
1 1000 56 445
2 1 59 nan
2 2 46 59
...
2 1000 64 784
3 1 54 nan
...
100 1 451 nan
100 2 153 451
...
100 1000 21 1121
Can I use .shift() and .groupby() somehow to do that?
python pandas pandas-groupby
python pandas pandas-groupby
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
asked Nov 16 '18 at 10:08
Alexandr KapshukAlexandr Kapshuk
294115
294115
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
1
Yes, simply do:
df['prev_value'] = df.groupby('object')['value'].shift()
For this example dataframe:
print(df)
object period value
0 1 1 24
1 1 2 67
2 1 4 89
3 2 4 5
4 2 23 23
The result would be:
object period value prev_value
0 1 1 24 NaN
1 1 2 67 24.0
2 1 4 89 67.0
3 2 4 5 NaN
4 2 23 23 5.0
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
add a comment |
-1
Simply create a new column from an existing one.
data["prev_value"] = data["value"]
answered Nov 16 '18 at 10:09
artonaartona
71248
but that won't shift theprev_valueone period back, will it?
– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
add a comment |
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53335567%2fuse-pandas-shift-within-a-group%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
Yes, simply do:
df['prev_value'] = df.groupby('object')['value'].shift()
For this example dataframe:
print(df)
object period value
0 1 1 24
1 1 2 67
2 1 4 89
3 2 4 5
4 2 23 23
The result would be:
object period value prev_value
0 1 1 24 NaN
1 1 2 67 24.0
2 1 4 89 67.0
3 2 4 5 NaN
4 2 23 23 5.0
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
add a comment |
1
Yes, simply do:
df['prev_value'] = df.groupby('object')['value'].shift()
For this example dataframe:
print(df)
object period value
0 1 1 24
1 1 2 67
2 1 4 89
3 2 4 5
4 2 23 23
The result would be:
object period value prev_value
0 1 1 24 NaN
1 1 2 67 24.0
2 1 4 89 67.0
3 2 4 5 NaN
4 2 23 23 5.0
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
add a comment |
1
1
1
Yes, simply do:
df['prev_value'] = df.groupby('object')['value'].shift()
For this example dataframe:
print(df)
object period value
0 1 1 24
1 1 2 67
2 1 4 89
3 2 4 5
4 2 23 23
The result would be:
object period value prev_value
0 1 1 24 NaN
1 1 2 67 24.0
2 1 4 89 67.0
3 2 4 5 NaN
4 2 23 23 5.0
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
Yes, simply do:
df['prev_value'] = df.groupby('object')['value'].shift()
For this example dataframe:
print(df)
object period value
0 1 1 24
1 1 2 67
2 1 4 89
3 2 4 5
4 2 23 23
The result would be:
object period value prev_value
0 1 1 24 NaN
1 1 2 67 24.0
2 1 4 89 67.0
3 2 4 5 NaN
4 2 23 23 5.0
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
edited Nov 16 '18 at 10:22
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
answered Nov 16 '18 at 10:17
yatuyatu
15.6k41542
15.6k41542
add a comment |
add a comment |
-1
Simply create a new column from an existing one.
data["prev_value"] = data["value"]
answered Nov 16 '18 at 10:09
artonaartona
71248
but that won't shift theprev_valueone period back, will it?
– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
add a comment |
-1
Simply create a new column from an existing one.
data["prev_value"] = data["value"]
answered Nov 16 '18 at 10:09
artonaartona
71248
but that won't shift theprev_valueone period back, will it?
– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
add a comment |
-1
-1
-1
Simply create a new column from an existing one.
data["prev_value"] = data["value"]
answered Nov 16 '18 at 10:09
artonaartona
71248
Simply create a new column from an existing one.
data["prev_value"] = data["value"]
answered Nov 16 '18 at 10:09
artonaartona
71248
answered Nov 16 '18 at 10:09
artonaartona
71248
answered Nov 16 '18 at 10:09
artonaartona
71248
answered Nov 16 '18 at 10:09
artonaartona
71248
71248
but that won't shift theprev_valueone period back, will it?
– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
add a comment |
but that won't shift theprev_valueone period back, will it?
– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
but that won't shift the
prev_value one period back, will it?– Alexandr Kapshuk
Nov 16 '18 at 10:10
but that won't shift the
prev_value one period back, will it?– Alexandr Kapshuk
Nov 16 '18 at 10:10
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
Could you edit your question, because it is hard to understand what the output should be?
– artona
Nov 16 '18 at 10:15
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53335567%2fuse-pandas-shift-within-a-group%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
