Create a csv file of a view in hive and put it in s3 with headers excluding the table names
I have a view in hive named prod_schoool_kolkata. I used to get the csv as:
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
that was in EC2-Instance. I want the path to be in S3.
I tried giving the path like :
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > s3://data/prod_schoool_kolkata.csv
But the csv is not getting stored.
I also had a problem that the csv file is getting generated but every column head is having pattern like: tablename.columnname for example prod_schoool_kolkata.id. Is there any way to remove the table names in the csv getting formed.
amazon-s3 hive hiveql
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
add a comment |
I have a view in hive named prod_schoool_kolkata. I used to get the csv as:
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
that was in EC2-Instance. I want the path to be in S3.
I tried giving the path like :
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > s3://data/prod_schoool_kolkata.csv
But the csv is not getting stored.
I also had a problem that the csv file is getting generated but every column head is having pattern like: tablename.columnname for example prod_schoool_kolkata.id. Is there any way to remove the table names in the csv getting formed.
amazon-s3 hive hiveql
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
add a comment |
I have a view in hive named prod_schoool_kolkata. I used to get the csv as:
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
that was in EC2-Instance. I want the path to be in S3.
I tried giving the path like :
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > s3://data/prod_schoool_kolkata.csv
But the csv is not getting stored.
I also had a problem that the csv file is getting generated but every column head is having pattern like: tablename.columnname for example prod_schoool_kolkata.id. Is there any way to remove the table names in the csv getting formed.
amazon-s3 hive hiveql
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
I have a view in hive named prod_schoool_kolkata. I used to get the csv as:
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
that was in EC2-Instance. I want the path to be in S3.
I tried giving the path like :
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata' | sed 's/[t]/,/g' > s3://data/prod_schoool_kolkata.csv
But the csv is not getting stored.
I also had a problem that the csv file is getting generated but every column head is having pattern like: tablename.columnname for example prod_schoool_kolkata.id. Is there any way to remove the table names in the csv getting formed.
amazon-s3 hive hiveql
amazon-s3 hive hiveql
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
edited Nov 23 '18 at 5:51
Vijaya Seetharaman
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
asked Nov 16 '18 at 11:29
Vijaya SeetharamanVijaya Seetharaman
6919
6919
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You have to first install the AWS Command Line Interface.
Refer the Link : Installing the AWS Command Line Interface and follow the relevant installation instructions or go to the Sections at the bottom to get the installation links relevant to your Operating System(Linux/Mac/Windows etc).
After verifying that it's installed properly, you may run normal commands like cp,ls etc over the aws file system. So, you could do
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata'|
sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
aws s3 cp /home/data/prod_schoool_kolkata.csv s3://data/prod_schoool_kolkata.csv
Also see How to use the S3 command-line tool
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,aws mvis also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html
– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
|
show 1 more comment
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336975%2fcreate-a-csv-file-of-a-view-in-hive-and-put-it-in-s3-with-headers-excluding-the%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You have to first install the AWS Command Line Interface.
Refer the Link : Installing the AWS Command Line Interface and follow the relevant installation instructions or go to the Sections at the bottom to get the installation links relevant to your Operating System(Linux/Mac/Windows etc).
After verifying that it's installed properly, you may run normal commands like cp,ls etc over the aws file system. So, you could do
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata'|
sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
aws s3 cp /home/data/prod_schoool_kolkata.csv s3://data/prod_schoool_kolkata.csv
Also see How to use the S3 command-line tool
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,aws mvis also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html
– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
|
show 1 more comment
You have to first install the AWS Command Line Interface.
Refer the Link : Installing the AWS Command Line Interface and follow the relevant installation instructions or go to the Sections at the bottom to get the installation links relevant to your Operating System(Linux/Mac/Windows etc).
After verifying that it's installed properly, you may run normal commands like cp,ls etc over the aws file system. So, you could do
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata'|
sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
aws s3 cp /home/data/prod_schoool_kolkata.csv s3://data/prod_schoool_kolkata.csv
Also see How to use the S3 command-line tool
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,aws mvis also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html
– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
|
show 1 more comment
You have to first install the AWS Command Line Interface.
Refer the Link : Installing the AWS Command Line Interface and follow the relevant installation instructions or go to the Sections at the bottom to get the installation links relevant to your Operating System(Linux/Mac/Windows etc).
After verifying that it's installed properly, you may run normal commands like cp,ls etc over the aws file system. So, you could do
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata'|
sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
aws s3 cp /home/data/prod_schoool_kolkata.csv s3://data/prod_schoool_kolkata.csv
Also see How to use the S3 command-line tool
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
You have to first install the AWS Command Line Interface.
Refer the Link : Installing the AWS Command Line Interface and follow the relevant installation instructions or go to the Sections at the bottom to get the installation links relevant to your Operating System(Linux/Mac/Windows etc).
After verifying that it's installed properly, you may run normal commands like cp,ls etc over the aws file system. So, you could do
hive -e 'set hive.cli.print.header=true; select * from prod_schoool_kolkata'|
sed 's/[t]/,/g' > /home/data/prod_schoool_kolkata.csv
aws s3 cp /home/data/prod_schoool_kolkata.csv s3://data/prod_schoool_kolkata.csv
Also see How to use the S3 command-line tool
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
answered Nov 17 '18 at 7:17
Kaushik NayakKaushik Nayak
21.3k41332
21.3k41332
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,aws mvis also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html
– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
|
show 1 more comment
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,aws mvis also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html
– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
There is no other direct way Sir?
– Vijaya Seetharaman
Nov 18 '18 at 13:57
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
@VijayaSeetharaman : S3 is a different filesystem and only way to access it is to either have S3 bucket mounted on your server or install an interface that can access it. Check with your Techops/ OS admin if they can get it mounted and then you may be able to access it directly. There are other modules in Java/ Python for uploading and downloading files from s3 if you are interested, apart from the command line for a programmatic approach.
– Kaushik Nayak
Nov 18 '18 at 14:25
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
Ok Sir. Is there any command for moving the data to s3 rather than copying as my ec2 instance will be loaded with lots of files.
– Vijaya Seetharaman
Nov 19 '18 at 7:14
@VijayaSeetharaman : Yes,
aws mv is also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html– Kaushik Nayak
Nov 19 '18 at 7:43
@VijayaSeetharaman : Yes,
aws mv is also available. See docs.aws.amazon.com/cli/latest/reference/s3/mv.html– Kaushik Nayak
Nov 19 '18 at 7:43
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
I have edited my question. If you know any solution please do help.
– Vijaya Seetharaman
Nov 23 '18 at 5:52
|
show 1 more comment
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336975%2fcreate-a-csv-file-of-a-view-in-hive-and-put-it-in-s3-with-headers-excluding-the%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
