R run same loop on different chunks of data in parallel
I'm using R on R studio.
In this case I can't use foreach.
I would like to do something fairly simple:
given a for loop:
for(i:1:1000000)
{
#some code here
}
One way to speed it up is to create two or more for loops and make them working on different chunks of data in different sessions.
Like:
for(i:1:500000)
{
#some code here
}
and
for(i:500001:1000000)
{
#some code here
}
on two different sessions.
I would like to not to use the "system2" package.
There must be a package that does it without having to manually open another session.
How to do it?
Cheers,
Dario.
r loops session parallel-processing
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
add a comment |
I'm using R on R studio.
In this case I can't use foreach.
I would like to do something fairly simple:
given a for loop:
for(i:1:1000000)
{
#some code here
}
One way to speed it up is to create two or more for loops and make them working on different chunks of data in different sessions.
Like:
for(i:1:500000)
{
#some code here
}
and
for(i:500001:1000000)
{
#some code here
}
on two different sessions.
I would like to not to use the "system2" package.
There must be a package that does it without having to manually open another session.
How to do it?
Cheers,
Dario.
r loops session parallel-processing
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
1
look at theforeachpackage. BTW - asking for library recommendations is considered off topic here
– dww
Nov 14 '18 at 2:11
1
I mean - theforeachpackage and thesystem2package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.
– Gregor
Nov 14 '18 at 3:05
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
3
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52
add a comment |
I'm using R on R studio.
In this case I can't use foreach.
I would like to do something fairly simple:
given a for loop:
for(i:1:1000000)
{
#some code here
}
One way to speed it up is to create two or more for loops and make them working on different chunks of data in different sessions.
Like:
for(i:1:500000)
{
#some code here
}
and
for(i:500001:1000000)
{
#some code here
}
on two different sessions.
I would like to not to use the "system2" package.
There must be a package that does it without having to manually open another session.
How to do it?
Cheers,
Dario.
r loops session parallel-processing
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
I'm using R on R studio.
In this case I can't use foreach.
I would like to do something fairly simple:
given a for loop:
for(i:1:1000000)
{
#some code here
}
One way to speed it up is to create two or more for loops and make them working on different chunks of data in different sessions.
Like:
for(i:1:500000)
{
#some code here
}
and
for(i:500001:1000000)
{
#some code here
}
on two different sessions.
I would like to not to use the "system2" package.
There must be a package that does it without having to manually open another session.
How to do it?
Cheers,
Dario.
r loops session parallel-processing
r loops session parallel-processing
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
edited Nov 14 '18 at 2:26
Dario Federici
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
asked Nov 14 '18 at 2:06
Dario FedericiDario Federici
243115
243115
1
look at theforeachpackage. BTW - asking for library recommendations is considered off topic here
– dww
Nov 14 '18 at 2:11
1
I mean - theforeachpackage and thesystem2package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.
– Gregor
Nov 14 '18 at 3:05
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
3
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52
add a comment |
1
look at theforeachpackage. BTW - asking for library recommendations is considered off topic here
– dww
Nov 14 '18 at 2:11
1
I mean - theforeachpackage and thesystem2package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.
– Gregor
Nov 14 '18 at 3:05
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
3
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52
1
1
look at the
foreach package. BTW - asking for library recommendations is considered off topic here– dww
Nov 14 '18 at 2:11
look at the
foreach package. BTW - asking for library recommendations is considered off topic here– dww
Nov 14 '18 at 2:11
1
1
I mean - the
foreach package and the system2 package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.– Gregor
Nov 14 '18 at 3:05
I mean - the
foreach package and the system2 package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.– Gregor
Nov 14 '18 at 3:05
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
3
3
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52
add a comment |
1 Answer
1
active
oldest
votes
I've found the solution in this package: "future.apply" that I find phenomenal.
It easily parallels the whole apply family.
I've used the "future_lapply" function and it works very well.
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53292179%2fr-run-same-loop-on-different-chunks-of-data-in-parallel%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I've found the solution in this package: "future.apply" that I find phenomenal.
It easily parallels the whole apply family.
I've used the "future_lapply" function and it works very well.
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
add a comment |
I've found the solution in this package: "future.apply" that I find phenomenal.
It easily parallels the whole apply family.
I've used the "future_lapply" function and it works very well.
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
add a comment |
I've found the solution in this package: "future.apply" that I find phenomenal.
It easily parallels the whole apply family.
I've used the "future_lapply" function and it works very well.
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
I've found the solution in this package: "future.apply" that I find phenomenal.
It easily parallels the whole apply family.
I've used the "future_lapply" function and it works very well.
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
answered Nov 21 '18 at 0:50
Dario FedericiDario Federici
243115
243115
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53292179%2fr-run-same-loop-on-different-chunks-of-data-in-parallel%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
look at the
foreachpackage. BTW - asking for library recommendations is considered off topic here– dww
Nov 14 '18 at 2:11
1
I mean - the
foreachpackage and thesystem2package would both work, but you can't use them for unspecified reasons. What makes you think you'd be able to use any of the other packages that would let you do this? The cran task view has a nice list of packages for parallel processing.– Gregor
Nov 14 '18 at 3:05
I've tried foreach many times (I normally use it for work) but it does not work properly and when it works it does not speed up the loop. system2 is not appropriate in this case because I will have to deploy the script in a non suitable environment.
– Dario Federici
Nov 14 '18 at 3:21
3
Please provide some code. I've no idea what you want to do. Chunking + foreach is usually a good idea. See e.g. stackoverflow.com/a/42131766/6103040.
– F. Privé
Nov 14 '18 at 6:52