How to overwrite Spark ML model in PySpark?
from pyspark.ml.regression import RandomForestRegressionModel
rf = RandomForestRegressor(labelCol="label",featuresCol="features", numTrees=5, maxDepth=10, seed=42)
rf_model = rf.fit(train_df)
rf_model_path = "./hdfsData/" + "rfr_model"
rf_model.save(rf_model_path)
When I first tried to save the model, these lines worked. But when I want to save the model into the path again, it gave this error:
Py4JJavaError: An error occurred while calling o1695.save. : java.io.IOException: Path ./hdfsData/rfr_model already exists. Please use write.overwrite().save(path) to overwrite it.
Then I tried:
rf_model.write.overwrite().save(rf_model_path)
It gave:
AttributeError: 'function' object has no attribute 'overwrite'
It seems the pyspark.mllib module gives the overwrite function but not pyspark.ml module. Anyone knows how to resolve this if I want to overwrite the old model with the new model? Thanks.
apache-spark machine-learning pyspark apache-spark-mllib apache-spark-ml
edited Jan 21 at 14:54
user10465355
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
add a comment |
from pyspark.ml.regression import RandomForestRegressionModel
rf = RandomForestRegressor(labelCol="label",featuresCol="features", numTrees=5, maxDepth=10, seed=42)
rf_model = rf.fit(train_df)
rf_model_path = "./hdfsData/" + "rfr_model"
rf_model.save(rf_model_path)
When I first tried to save the model, these lines worked. But when I want to save the model into the path again, it gave this error:
Py4JJavaError: An error occurred while calling o1695.save. : java.io.IOException: Path ./hdfsData/rfr_model already exists. Please use write.overwrite().save(path) to overwrite it.
Then I tried:
rf_model.write.overwrite().save(rf_model_path)
It gave:
AttributeError: 'function' object has no attribute 'overwrite'
It seems the pyspark.mllib module gives the overwrite function but not pyspark.ml module. Anyone knows how to resolve this if I want to overwrite the old model with the new model? Thanks.
apache-spark machine-learning pyspark apache-spark-mllib apache-spark-ml
edited Jan 21 at 14:54
user10465355
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
add a comment |
from pyspark.ml.regression import RandomForestRegressionModel
rf = RandomForestRegressor(labelCol="label",featuresCol="features", numTrees=5, maxDepth=10, seed=42)
rf_model = rf.fit(train_df)
rf_model_path = "./hdfsData/" + "rfr_model"
rf_model.save(rf_model_path)
When I first tried to save the model, these lines worked. But when I want to save the model into the path again, it gave this error:
Py4JJavaError: An error occurred while calling o1695.save. : java.io.IOException: Path ./hdfsData/rfr_model already exists. Please use write.overwrite().save(path) to overwrite it.
Then I tried:
rf_model.write.overwrite().save(rf_model_path)
It gave:
AttributeError: 'function' object has no attribute 'overwrite'
It seems the pyspark.mllib module gives the overwrite function but not pyspark.ml module. Anyone knows how to resolve this if I want to overwrite the old model with the new model? Thanks.
apache-spark machine-learning pyspark apache-spark-mllib apache-spark-ml
edited Jan 21 at 14:54
user10465355
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
from pyspark.ml.regression import RandomForestRegressionModel
rf = RandomForestRegressor(labelCol="label",featuresCol="features", numTrees=5, maxDepth=10, seed=42)
rf_model = rf.fit(train_df)
rf_model_path = "./hdfsData/" + "rfr_model"
rf_model.save(rf_model_path)
When I first tried to save the model, these lines worked. But when I want to save the model into the path again, it gave this error:
Py4JJavaError: An error occurred while calling o1695.save. : java.io.IOException: Path ./hdfsData/rfr_model already exists. Please use write.overwrite().save(path) to overwrite it.
Then I tried:
rf_model.write.overwrite().save(rf_model_path)
It gave:
AttributeError: 'function' object has no attribute 'overwrite'
It seems the pyspark.mllib module gives the overwrite function but not pyspark.ml module. Anyone knows how to resolve this if I want to overwrite the old model with the new model? Thanks.
apache-spark machine-learning pyspark apache-spark-mllib apache-spark-ml
apache-spark machine-learning pyspark apache-spark-mllib apache-spark-ml
edited Jan 21 at 14:54
user10465355
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
edited Jan 21 at 14:54
user10465355
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
edited Jan 21 at 14:54
user10465355
1,7832416
edited Jan 21 at 14:54
user10465355
1,7832416
edited Jan 21 at 14:54
user10465355
1,7832416
1,7832416
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
asked Feb 17 '17 at 17:12
Veronica Wenqian ChengVeronica Wenqian Cheng
142111
142111
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The message you see is a Java error message, not a Python one. You should call the write method first:
rf_model.write().overwrite().save(rf_model_path)
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42303705%2fhow-to-overwrite-spark-ml-model-in-pyspark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The message you see is a Java error message, not a Python one. You should call the write method first:
rf_model.write().overwrite().save(rf_model_path)
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
add a comment |
The message you see is a Java error message, not a Python one. You should call the write method first:
rf_model.write().overwrite().save(rf_model_path)
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
add a comment |
The message you see is a Java error message, not a Python one. You should call the write method first:
rf_model.write().overwrite().save(rf_model_path)
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
The message you see is a Java error message, not a Python one. You should call the write method first:
rf_model.write().overwrite().save(rf_model_path)
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
answered Feb 17 '17 at 17:49
user6910411user6910411
33.5k976100
33.5k976100
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42303705%2fhow-to-overwrite-spark-ml-model-in-pyspark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
