foreach function in sparkr
I am relatively new on SparkR, and I am planning to transfer a for loop into a foreach loop in SparkR (R/3.3.3 & Spark/2.2.0).
I have searched on stackoverflow, the only relevant thread is:
SparkR foreach loop
But it gives only the workaround by using other operations.
From what I see, there is a sparkr package exist (https://amplab-extras.github.io/SparkR-pkg/rdocs/1.2/index.html) and contains foreach function, but I really do not understand its use cases, which I will need some help/example from the community to help.
My example in original R code is following:
uniqueID <- unique(dataset$ID)
maxValueVector <- c()
for(id in uniqueID){
maxValueVector <- c(maximums, max(dataset[which(dataset$ID == id), ]$value))
}
I understand that the line in for loop should be break into several lines, but is there an example I can start with, such as the example foreach code I can start with? Thanks a lot!
p.s. dataset contains 2 columns: ID and value.
r apache-spark sparkr
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
add a comment |
I am relatively new on SparkR, and I am planning to transfer a for loop into a foreach loop in SparkR (R/3.3.3 & Spark/2.2.0).
I have searched on stackoverflow, the only relevant thread is:
SparkR foreach loop
But it gives only the workaround by using other operations.
From what I see, there is a sparkr package exist (https://amplab-extras.github.io/SparkR-pkg/rdocs/1.2/index.html) and contains foreach function, but I really do not understand its use cases, which I will need some help/example from the community to help.
My example in original R code is following:
uniqueID <- unique(dataset$ID)
maxValueVector <- c()
for(id in uniqueID){
maxValueVector <- c(maximums, max(dataset[which(dataset$ID == id), ]$value))
}
I understand that the line in for loop should be break into several lines, but is there an example I can start with, such as the example foreach code I can start with? Thanks a lot!
p.s. dataset contains 2 columns: ID and value.
r apache-spark sparkr
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
1
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, itsforeachwouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus ongroupBydocs).
– user6910411
Nov 22 '18 at 21:22
add a comment |
I am relatively new on SparkR, and I am planning to transfer a for loop into a foreach loop in SparkR (R/3.3.3 & Spark/2.2.0).
I have searched on stackoverflow, the only relevant thread is:
SparkR foreach loop
But it gives only the workaround by using other operations.
From what I see, there is a sparkr package exist (https://amplab-extras.github.io/SparkR-pkg/rdocs/1.2/index.html) and contains foreach function, but I really do not understand its use cases, which I will need some help/example from the community to help.
My example in original R code is following:
uniqueID <- unique(dataset$ID)
maxValueVector <- c()
for(id in uniqueID){
maxValueVector <- c(maximums, max(dataset[which(dataset$ID == id), ]$value))
}
I understand that the line in for loop should be break into several lines, but is there an example I can start with, such as the example foreach code I can start with? Thanks a lot!
p.s. dataset contains 2 columns: ID and value.
r apache-spark sparkr
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
I am relatively new on SparkR, and I am planning to transfer a for loop into a foreach loop in SparkR (R/3.3.3 & Spark/2.2.0).
I have searched on stackoverflow, the only relevant thread is:
SparkR foreach loop
But it gives only the workaround by using other operations.
From what I see, there is a sparkr package exist (https://amplab-extras.github.io/SparkR-pkg/rdocs/1.2/index.html) and contains foreach function, but I really do not understand its use cases, which I will need some help/example from the community to help.
My example in original R code is following:
uniqueID <- unique(dataset$ID)
maxValueVector <- c()
for(id in uniqueID){
maxValueVector <- c(maximums, max(dataset[which(dataset$ID == id), ]$value))
}
I understand that the line in for loop should be break into several lines, but is there an example I can start with, such as the example foreach code I can start with? Thanks a lot!
p.s. dataset contains 2 columns: ID and value.
r apache-spark sparkr
r apache-spark sparkr
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
asked Nov 15 '18 at 17:45
windsoundwindsound
1611522
1611522
1
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, itsforeachwouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus ongroupBydocs).
– user6910411
Nov 22 '18 at 21:22
add a comment |
1
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, itsforeachwouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus ongroupBydocs).
– user6910411
Nov 22 '18 at 21:22
1
1
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, its
foreach wouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus on groupBy docs).– user6910411
Nov 22 '18 at 21:22
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, its
foreach wouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus on groupBy docs).– user6910411
Nov 22 '18 at 21:22
add a comment |
1 Answer
1
active
oldest
votes
As the comments said, in SparkR, we generally do not want to use foreach. In this particular case, I found the answer using sparkdataframe operatiors and solved this problem:
## IDs is collected fo
IDs <- collect(distinct(select(dataset, 'ID')))
## I added the maximums column in order to figure out the future steps
## it basically satisfied what I need to have.
Maximums <- agg(groupBy(dataset, dataset$ID), maximums = max(dataset$value))
Maximums <- arrange(Maximums, desc(Maximums$maximums))
I know since I am still new on this, so this solution may not be what you are looking for. But thanks again for comments!
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325186%2fforeach-function-in-sparkr%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As the comments said, in SparkR, we generally do not want to use foreach. In this particular case, I found the answer using sparkdataframe operatiors and solved this problem:
## IDs is collected fo
IDs <- collect(distinct(select(dataset, 'ID')))
## I added the maximums column in order to figure out the future steps
## it basically satisfied what I need to have.
Maximums <- agg(groupBy(dataset, dataset$ID), maximums = max(dataset$value))
Maximums <- arrange(Maximums, desc(Maximums$maximums))
I know since I am still new on this, so this solution may not be what you are looking for. But thanks again for comments!
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
add a comment |
As the comments said, in SparkR, we generally do not want to use foreach. In this particular case, I found the answer using sparkdataframe operatiors and solved this problem:
## IDs is collected fo
IDs <- collect(distinct(select(dataset, 'ID')))
## I added the maximums column in order to figure out the future steps
## it basically satisfied what I need to have.
Maximums <- agg(groupBy(dataset, dataset$ID), maximums = max(dataset$value))
Maximums <- arrange(Maximums, desc(Maximums$maximums))
I know since I am still new on this, so this solution may not be what you are looking for. But thanks again for comments!
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
add a comment |
As the comments said, in SparkR, we generally do not want to use foreach. In this particular case, I found the answer using sparkdataframe operatiors and solved this problem:
## IDs is collected fo
IDs <- collect(distinct(select(dataset, 'ID')))
## I added the maximums column in order to figure out the future steps
## it basically satisfied what I need to have.
Maximums <- agg(groupBy(dataset, dataset$ID), maximums = max(dataset$value))
Maximums <- arrange(Maximums, desc(Maximums$maximums))
I know since I am still new on this, so this solution may not be what you are looking for. But thanks again for comments!
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
As the comments said, in SparkR, we generally do not want to use foreach. In this particular case, I found the answer using sparkdataframe operatiors and solved this problem:
## IDs is collected fo
IDs <- collect(distinct(select(dataset, 'ID')))
## I added the maximums column in order to figure out the future steps
## it basically satisfied what I need to have.
Maximums <- agg(groupBy(dataset, dataset$ID), maximums = max(dataset$value))
Maximums <- arrange(Maximums, desc(Maximums$maximums))
I know since I am still new on this, so this solution may not be what you are looking for. But thanks again for comments!
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
answered Nov 27 '18 at 16:11
windsoundwindsound
1611522
1611522
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325186%2fforeach-function-in-sparkr%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
a) That's really not how we express things in Spark. b) SparkR package you've linked have been abandoned many years ago, and even if it wasn't, its
foreachwouldn't be applicable here. For new API see spark.apache.org/docs/latest/sparkr.html (Hint: focus ongroupBydocs).– user6910411
Nov 22 '18 at 21:22