Decompressing file as uploading to HDFS












0















I have data that I want to upload to HDFS



data folder is multi-layer, but innermost dir only contains multiple .gz files.



folder is about 250 GB compressed, and I don't know how many gz files are in there.



I want to upload this to HDFS but they should be decompressed.



Is there a way that I can achieve this with single hadoop fs -put command?
or is writing a script file decompressing one by one and uploading to hdfs the only way?










share|improve this question























  • GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

    – cricket_007
    Nov 16 '18 at 10:10













  • yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

    – Brandon Lee
    Nov 16 '18 at 15:08











  • You could look into adding Apache Nifi and using its ExtractFile processors

    – cricket_007
    Nov 16 '18 at 15:30


















0















I have data that I want to upload to HDFS



data folder is multi-layer, but innermost dir only contains multiple .gz files.



folder is about 250 GB compressed, and I don't know how many gz files are in there.



I want to upload this to HDFS but they should be decompressed.



Is there a way that I can achieve this with single hadoop fs -put command?
or is writing a script file decompressing one by one and uploading to hdfs the only way?










share|improve this question























  • GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

    – cricket_007
    Nov 16 '18 at 10:10













  • yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

    – Brandon Lee
    Nov 16 '18 at 15:08











  • You could look into adding Apache Nifi and using its ExtractFile processors

    – cricket_007
    Nov 16 '18 at 15:30
















0












0








0


1






I have data that I want to upload to HDFS



data folder is multi-layer, but innermost dir only contains multiple .gz files.



folder is about 250 GB compressed, and I don't know how many gz files are in there.



I want to upload this to HDFS but they should be decompressed.



Is there a way that I can achieve this with single hadoop fs -put command?
or is writing a script file decompressing one by one and uploading to hdfs the only way?










share|improve this question














I have data that I want to upload to HDFS



data folder is multi-layer, but innermost dir only contains multiple .gz files.



folder is about 250 GB compressed, and I don't know how many gz files are in there.



I want to upload this to HDFS but they should be decompressed.



Is there a way that I can achieve this with single hadoop fs -put command?
or is writing a script file decompressing one by one and uploading to hdfs the only way?







hadoop hdfs






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 16 '18 at 3:42









Brandon LeeBrandon Lee

676




676













  • GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

    – cricket_007
    Nov 16 '18 at 10:10













  • yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

    – Brandon Lee
    Nov 16 '18 at 15:08











  • You could look into adding Apache Nifi and using its ExtractFile processors

    – cricket_007
    Nov 16 '18 at 15:30





















  • GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

    – cricket_007
    Nov 16 '18 at 10:10













  • yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

    – Brandon Lee
    Nov 16 '18 at 15:08











  • You could look into adding Apache Nifi and using its ExtractFile processors

    – cricket_007
    Nov 16 '18 at 15:30



















GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

– cricket_007
Nov 16 '18 at 10:10







GZ files are supported by most Hadoop processes. They just aren't splittable. -put cannot extract, though

– cricket_007
Nov 16 '18 at 10:10















yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

– Brandon Lee
Nov 16 '18 at 15:08





yes, that is my concern. putting gz directly on hdfs is not an issue. The fact that they should be decompressed when it is on hdfs is.

– Brandon Lee
Nov 16 '18 at 15:08













You could look into adding Apache Nifi and using its ExtractFile processors

– cricket_007
Nov 16 '18 at 15:30







You could look into adding Apache Nifi and using its ExtractFile processors

– cricket_007
Nov 16 '18 at 15:30














1 Answer
1






active

oldest

votes


















0














I think you can do that only with a script file, by uncompressing each file. But you can reduce the script with a pipe, something like this:



for file in $( ls ); do
echo Uncompressing and putting $file
gunzip -c $file | hdfs dfs -put - "/hdfs-dir/$file.txt"
done





share|improve this answer
























  • ya I just created script as well

    – Brandon Lee
    Nov 23 '18 at 19:49











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53331093%2fdecompressing-file-as-uploading-to-hdfs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














I think you can do that only with a script file, by uncompressing each file. But you can reduce the script with a pipe, something like this:



for file in $( ls ); do
echo Uncompressing and putting $file
gunzip -c $file | hdfs dfs -put - "/hdfs-dir/$file.txt"
done





share|improve this answer
























  • ya I just created script as well

    – Brandon Lee
    Nov 23 '18 at 19:49
















0














I think you can do that only with a script file, by uncompressing each file. But you can reduce the script with a pipe, something like this:



for file in $( ls ); do
echo Uncompressing and putting $file
gunzip -c $file | hdfs dfs -put - "/hdfs-dir/$file.txt"
done





share|improve this answer
























  • ya I just created script as well

    – Brandon Lee
    Nov 23 '18 at 19:49














0












0








0







I think you can do that only with a script file, by uncompressing each file. But you can reduce the script with a pipe, something like this:



for file in $( ls ); do
echo Uncompressing and putting $file
gunzip -c $file | hdfs dfs -put - "/hdfs-dir/$file.txt"
done





share|improve this answer













I think you can do that only with a script file, by uncompressing each file. But you can reduce the script with a pipe, something like this:



for file in $( ls ); do
echo Uncompressing and putting $file
gunzip -c $file | hdfs dfs -put - "/hdfs-dir/$file.txt"
done






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 16 '18 at 17:10









Marco ReisMarco Reis

644




644













  • ya I just created script as well

    – Brandon Lee
    Nov 23 '18 at 19:49



















  • ya I just created script as well

    – Brandon Lee
    Nov 23 '18 at 19:49

















ya I just created script as well

– Brandon Lee
Nov 23 '18 at 19:49





ya I just created script as well

– Brandon Lee
Nov 23 '18 at 19:49




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53331093%2fdecompressing-file-as-uploading-to-hdfs%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Bressuire

Vorschmack

Quarantine