Resample or aggregate large raster limiting memory use R
I'm trying to resample a huge worldpop population raster but I keep crashing my linux instance of R which has 32GB memory. When I launch a 52gb memory google compute instance, the code below works, but it crashes my regular 32gb ram computer.
Is there a way to do either a raster aggregation or resampling LIMITING MEMORY USE?
Download code for a large worldpop raster that I am having issues resampling:
###download the huge raster to recreate scale problem
devtools::install_github("nbarsch/spaceheater")
library(spaceheater)
getWPdownload("Tanzania","Population","adj",2015) #warning: downloads near 1gb file
library(raster)
wpras <- raster("TANZANIA_Population_adj_2015.tif")
TWO METHODS that work on a computer with 52GB ram, but kill my local computer 32GB ram:
#aggregate method
agras <- raster::aggregate(wpras,fact=10,expand=T)
#returns "Killed"
#resamp method
reras <- raster(nrow = ceiling(nrow(wpras)/10), ncol = ceiling(ncol(wpras)/10))
reras2 <- raster::resample(wpras, reras, method="bilinear")
#returns Error: cannot allocate vector of size 1.3 Gb
Anyone have a solution that doesn't use all the ram? Thanks!
r memory raster ram
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
add a comment |
I'm trying to resample a huge worldpop population raster but I keep crashing my linux instance of R which has 32GB memory. When I launch a 52gb memory google compute instance, the code below works, but it crashes my regular 32gb ram computer.
Is there a way to do either a raster aggregation or resampling LIMITING MEMORY USE?
Download code for a large worldpop raster that I am having issues resampling:
###download the huge raster to recreate scale problem
devtools::install_github("nbarsch/spaceheater")
library(spaceheater)
getWPdownload("Tanzania","Population","adj",2015) #warning: downloads near 1gb file
library(raster)
wpras <- raster("TANZANIA_Population_adj_2015.tif")
TWO METHODS that work on a computer with 52GB ram, but kill my local computer 32GB ram:
#aggregate method
agras <- raster::aggregate(wpras,fact=10,expand=T)
#returns "Killed"
#resamp method
reras <- raster(nrow = ceiling(nrow(wpras)/10), ncol = ceiling(ncol(wpras)/10))
reras2 <- raster::resample(wpras, reras, method="bilinear")
#returns Error: cannot allocate vector of size 1.3 Gb
Anyone have a solution that doesn't use all the ram? Thanks!
r memory raster ram
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42
add a comment |
I'm trying to resample a huge worldpop population raster but I keep crashing my linux instance of R which has 32GB memory. When I launch a 52gb memory google compute instance, the code below works, but it crashes my regular 32gb ram computer.
Is there a way to do either a raster aggregation or resampling LIMITING MEMORY USE?
Download code for a large worldpop raster that I am having issues resampling:
###download the huge raster to recreate scale problem
devtools::install_github("nbarsch/spaceheater")
library(spaceheater)
getWPdownload("Tanzania","Population","adj",2015) #warning: downloads near 1gb file
library(raster)
wpras <- raster("TANZANIA_Population_adj_2015.tif")
TWO METHODS that work on a computer with 52GB ram, but kill my local computer 32GB ram:
#aggregate method
agras <- raster::aggregate(wpras,fact=10,expand=T)
#returns "Killed"
#resamp method
reras <- raster(nrow = ceiling(nrow(wpras)/10), ncol = ceiling(ncol(wpras)/10))
reras2 <- raster::resample(wpras, reras, method="bilinear")
#returns Error: cannot allocate vector of size 1.3 Gb
Anyone have a solution that doesn't use all the ram? Thanks!
r memory raster ram
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
I'm trying to resample a huge worldpop population raster but I keep crashing my linux instance of R which has 32GB memory. When I launch a 52gb memory google compute instance, the code below works, but it crashes my regular 32gb ram computer.
Is there a way to do either a raster aggregation or resampling LIMITING MEMORY USE?
Download code for a large worldpop raster that I am having issues resampling:
###download the huge raster to recreate scale problem
devtools::install_github("nbarsch/spaceheater")
library(spaceheater)
getWPdownload("Tanzania","Population","adj",2015) #warning: downloads near 1gb file
library(raster)
wpras <- raster("TANZANIA_Population_adj_2015.tif")
TWO METHODS that work on a computer with 52GB ram, but kill my local computer 32GB ram:
#aggregate method
agras <- raster::aggregate(wpras,fact=10,expand=T)
#returns "Killed"
#resamp method
reras <- raster(nrow = ceiling(nrow(wpras)/10), ncol = ceiling(ncol(wpras)/10))
reras2 <- raster::resample(wpras, reras, method="bilinear")
#returns Error: cannot allocate vector of size 1.3 Gb
Anyone have a solution that doesn't use all the ram? Thanks!
r memory raster ram
r memory raster ram
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
asked Nov 16 '18 at 8:46
Neal BarschNeal Barsch
640115
640115
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42
add a comment |
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42
add a comment |
1 Answer
1
active
oldest
votes
You probably should update the raster package. The previous short-lived release (2.7-15) had an error in the memory settings. Version 2.8-4 should have fixed that.
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53334276%2fresample-or-aggregate-large-raster-limiting-memory-use-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You probably should update the raster package. The previous short-lived release (2.7-15) had an error in the memory settings. Version 2.8-4 should have fixed that.
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
add a comment |
You probably should update the raster package. The previous short-lived release (2.7-15) had an error in the memory settings. Version 2.8-4 should have fixed that.
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
add a comment |
You probably should update the raster package. The previous short-lived release (2.7-15) had an error in the memory settings. Version 2.8-4 should have fixed that.
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
You probably should update the raster package. The previous short-lived release (2.7-15) had an error in the memory settings. Version 2.8-4 should have fixed that.
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
answered Nov 17 '18 at 4:19
Robert HijmansRobert Hijmans
14k12530
14k12530
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53334276%2fresample-or-aggregate-large-raster-limiting-memory-use-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
# increase ram memory.limit()
– Dr. Flow
Nov 16 '18 at 10:42