What is the right way to persist a DataFrame to a database
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
What is the right way to persist a DataFrame to a database using custom SQL code? The DataFrame.apply() function executes the save_entity() function with the custom SQL code TWICE for the first row of the DataFrame that is not acceptable. The code below illustrates the problem
# save_entity() is executed TWICE for the first row
entities_df["entity_id"] = entities_df.apply(save_entity, axis=1)
I'm preferable looking for a declarative approach for working with a DataFrame as a whole, rather then using lower-level DataFrame.iterrows() or DataFrame.itertuples() functions that do not produce a DataFrame any more.
sql pandas
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
add a comment |
0
What is the right way to persist a DataFrame to a database using custom SQL code? The DataFrame.apply() function executes the save_entity() function with the custom SQL code TWICE for the first row of the DataFrame that is not acceptable. The code below illustrates the problem
# save_entity() is executed TWICE for the first row
entities_df["entity_id"] = entities_df.apply(save_entity, axis=1)
I'm preferable looking for a declarative approach for working with a DataFrame as a whole, rather then using lower-level DataFrame.iterrows() or DataFrame.itertuples() functions that do not produce a DataFrame any more.
sql pandas
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
1
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46
add a comment |
0
0
0
What is the right way to persist a DataFrame to a database using custom SQL code? The DataFrame.apply() function executes the save_entity() function with the custom SQL code TWICE for the first row of the DataFrame that is not acceptable. The code below illustrates the problem
# save_entity() is executed TWICE for the first row
entities_df["entity_id"] = entities_df.apply(save_entity, axis=1)
I'm preferable looking for a declarative approach for working with a DataFrame as a whole, rather then using lower-level DataFrame.iterrows() or DataFrame.itertuples() functions that do not produce a DataFrame any more.
sql pandas
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
What is the right way to persist a DataFrame to a database using custom SQL code? The DataFrame.apply() function executes the save_entity() function with the custom SQL code TWICE for the first row of the DataFrame that is not acceptable. The code below illustrates the problem
# save_entity() is executed TWICE for the first row
entities_df["entity_id"] = entities_df.apply(save_entity, axis=1)
I'm preferable looking for a declarative approach for working with a DataFrame as a whole, rather then using lower-level DataFrame.iterrows() or DataFrame.itertuples() functions that do not produce a DataFrame any more.
sql pandas
sql pandas
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
edited Nov 17 '18 at 8:44
Volodymyr Prokopyuk
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
asked Nov 16 '18 at 21:32
Volodymyr ProkopyukVolodymyr Prokopyuk
286
286
1
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46
add a comment |
1
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46
1
1
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46
add a comment |
1 Answer
1
active
oldest
votes
0
Use pandas DataFrame.to_sql() method. Documentation here
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
TheDataFrame.to_sql()limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want aDataFrame.map(func)function that applies otherprocess_row(row)function to each row of the DataFrame and allows to store the return values from theprocess_row(row)function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345695%2fwhat-is-the-right-way-to-persist-a-dataframe-to-a-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
Use pandas DataFrame.to_sql() method. Documentation here
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
TheDataFrame.to_sql()limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want aDataFrame.map(func)function that applies otherprocess_row(row)function to each row of the DataFrame and allows to store the return values from theprocess_row(row)function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
add a comment |
0
Use pandas DataFrame.to_sql() method. Documentation here
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
TheDataFrame.to_sql()limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want aDataFrame.map(func)function that applies otherprocess_row(row)function to each row of the DataFrame and allows to store the return values from theprocess_row(row)function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
add a comment |
0
0
0
Use pandas DataFrame.to_sql() method. Documentation here
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
Use pandas DataFrame.to_sql() method. Documentation here
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
answered Nov 16 '18 at 21:54
paulo.filip3paulo.filip3
2,18811422
2,18811422
TheDataFrame.to_sql()limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want aDataFrame.map(func)function that applies otherprocess_row(row)function to each row of the DataFrame and allows to store the return values from theprocess_row(row)function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
add a comment |
TheDataFrame.to_sql()limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want aDataFrame.map(func)function that applies otherprocess_row(row)function to each row of the DataFrame and allows to store the return values from theprocess_row(row)function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.
– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
The
DataFrame.to_sql() limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
The
DataFrame.to_sql() limits the solution to only the databases supported my SQLAlchemy. I'd like to have a complete control over the SQL to persist a DataFrame row.– Volodymyr Prokopyuk
Nov 17 '18 at 8:49
I think I want something very simple, but it seems that Pandas does not have it. I want a
DataFrame.map(func) function that applies other process_row(row) function to each row of the DataFrame and allows to store the return values from the process_row(row) function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
I think I want something very simple, but it seems that Pandas does not have it. I want a
DataFrame.map(func) function that applies other process_row(row) function to each row of the DataFrame and allows to store the return values from the process_row(row) function as a new column in the initial DataFrame. I want a DataFrame as input and I want a DataFrame as output with the logic applied in between for each row.– Volodymyr Prokopyuk
Nov 17 '18 at 8:59
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345695%2fwhat-is-the-right-way-to-persist-a-dataframe-to-a-database%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Are you trying to store the entire DataFrame to a SQL database? If so, I recommend reviewing pandas.to_sql() pandas.pydata.org/pandas-docs/stable/generated/…
– rs311
Nov 16 '18 at 21:34
I prefer to have complete control over the SQL to persist a DataFrame row and I'd avoid using SQLAlchemy
– Volodymyr Prokopyuk
Nov 17 '18 at 8:46