Windows file in spark read csv
up vote
-1
down vote
favorite
I am using databricks for a practice in Python
I am trying to load a windows file via:
diamonds = spark.read.csv('file:///C:/abc.csv', header="true", inferSchema="true")
I am sure that my file is in C:/abc.csv but the system always returns this error:
u'Path does not exist file:/C:/abc.csv;'
How can I load a file in windows system?
python csv apache-spark databricks
edited Nov 8 at 3:00
MrBear
1083
asked Nov 8 at 2:46
RGS
31
add a comment |
up vote
-1
down vote
favorite
I am using databricks for a practice in Python
I am trying to load a windows file via:
diamonds = spark.read.csv('file:///C:/abc.csv', header="true", inferSchema="true")
I am sure that my file is in C:/abc.csv but the system always returns this error:
u'Path does not exist file:/C:/abc.csv;'
How can I load a file in windows system?
python csv apache-spark databricks
edited Nov 8 at 3:00
MrBear
1083
asked Nov 8 at 2:46
RGS
31
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
I am using databricks for a practice in Python
I am trying to load a windows file via:
diamonds = spark.read.csv('file:///C:/abc.csv', header="true", inferSchema="true")
I am sure that my file is in C:/abc.csv but the system always returns this error:
u'Path does not exist file:/C:/abc.csv;'
How can I load a file in windows system?
python csv apache-spark databricks
edited Nov 8 at 3:00
MrBear
1083
asked Nov 8 at 2:46
RGS
31
I am using databricks for a practice in Python
I am trying to load a windows file via:
diamonds = spark.read.csv('file:///C:/abc.csv', header="true", inferSchema="true")
I am sure that my file is in C:/abc.csv but the system always returns this error:
u'Path does not exist file:/C:/abc.csv;'
How can I load a file in windows system?
python csv apache-spark databricks
python csv apache-spark databricks
edited Nov 8 at 3:00
MrBear
1083
asked Nov 8 at 2:46
RGS
31
edited Nov 8 at 3:00
MrBear
1083
asked Nov 8 at 2:46
RGS
31
edited Nov 8 at 3:00
MrBear
1083
edited Nov 8 at 3:00
MrBear
1083
edited Nov 8 at 3:00
MrBear
1083
1083
asked Nov 8 at 2:46
RGS
31
asked Nov 8 at 2:46
RGS
31
asked Nov 8 at 2:46
RGS
31
31
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
0
down vote
accepted
1. Define full path as variable - every path should begin with a drive if local
# remove the 'file' string and use 'r' or 'u' prefix to indicate raw/unicore string format
# Option 1
PATH = r'C:abc.csv' # raw string
# Option 2
PATH = u'C:\abc.csv' # unicode string
2. Set the path variable to your spark call
diamonds = spark.read.csv(PATH, header="true", inferSchema="true")
In essence, your filepath is wrong.
answered Nov 8 at 5:32
sudonym
1,291924
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
add a comment |
up vote
0
down vote
Databricks is a cloud service that runs a cluster in the cloud. You can interact with the cluster using a notebook in your browser, but databricks doesn't have access to files on your computer. (Allowing websites to access arbitrary files on your computer would be a huge security issue. Image if I sent you a malicious link, and when you clicked it I could just read any file on your computer).
If you use the data button on the left side of the databricks UI, you can upload csv files and create tables that will be available tp your databricks cluster.
answered Nov 9 at 5:31
Bi Rico
19k33659
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53200843%2fwindows-file-in-spark-read-csv%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
1. Define full path as variable - every path should begin with a drive if local
# remove the 'file' string and use 'r' or 'u' prefix to indicate raw/unicore string format
# Option 1
PATH = r'C:abc.csv' # raw string
# Option 2
PATH = u'C:\abc.csv' # unicode string
2. Set the path variable to your spark call
diamonds = spark.read.csv(PATH, header="true", inferSchema="true")
In essence, your filepath is wrong.
answered Nov 8 at 5:32
sudonym
1,291924
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
add a comment |
up vote
0
down vote
accepted
1. Define full path as variable - every path should begin with a drive if local
# remove the 'file' string and use 'r' or 'u' prefix to indicate raw/unicore string format
# Option 1
PATH = r'C:abc.csv' # raw string
# Option 2
PATH = u'C:\abc.csv' # unicode string
2. Set the path variable to your spark call
diamonds = spark.read.csv(PATH, header="true", inferSchema="true")
In essence, your filepath is wrong.
answered Nov 8 at 5:32
sudonym
1,291924
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
1. Define full path as variable - every path should begin with a drive if local
# remove the 'file' string and use 'r' or 'u' prefix to indicate raw/unicore string format
# Option 1
PATH = r'C:abc.csv' # raw string
# Option 2
PATH = u'C:\abc.csv' # unicode string
2. Set the path variable to your spark call
diamonds = spark.read.csv(PATH, header="true", inferSchema="true")
In essence, your filepath is wrong.
answered Nov 8 at 5:32
sudonym
1,291924
1. Define full path as variable - every path should begin with a drive if local
# remove the 'file' string and use 'r' or 'u' prefix to indicate raw/unicore string format
# Option 1
PATH = r'C:abc.csv' # raw string
# Option 2
PATH = u'C:\abc.csv' # unicode string
2. Set the path variable to your spark call
diamonds = spark.read.csv(PATH, header="true", inferSchema="true")
In essence, your filepath is wrong.
answered Nov 8 at 5:32
sudonym
1,291924
edited Nov 12 at 5:42
answered Nov 8 at 5:32
sudonym
1,291924
answered Nov 8 at 5:32
sudonym
1,291924
answered Nov 8 at 5:32
sudonym
1,291924
1,291924
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
add a comment |
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Please add some more explanation to your answer.
– Rishikesh Raje
Nov 8 at 6:05
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
Good morning! thanks a lot, but this answer didn't work....
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
PATH = u"C:abc.csv" print PATH
– RGS
Nov 8 at 13:08
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
C:bc.csv I proved with: PATH = u"C:\abc.csv" , in this case PRINT is correct: C:abc.csv but system generate a error message: IllegalArgumentException: u'java.net.URISyntaxException: Relative path in absolute URI: C:%5Cabc.csv'
– RGS
Nov 8 at 13:10
Please your help again....
– RGS
Nov 8 at 13:11
Please your help again....
– RGS
Nov 8 at 13:11
add a comment |
up vote
0
down vote
Databricks is a cloud service that runs a cluster in the cloud. You can interact with the cluster using a notebook in your browser, but databricks doesn't have access to files on your computer. (Allowing websites to access arbitrary files on your computer would be a huge security issue. Image if I sent you a malicious link, and when you clicked it I could just read any file on your computer).
If you use the data button on the left side of the databricks UI, you can upload csv files and create tables that will be available tp your databricks cluster.
answered Nov 9 at 5:31
Bi Rico
19k33659
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
add a comment |
up vote
0
down vote
Databricks is a cloud service that runs a cluster in the cloud. You can interact with the cluster using a notebook in your browser, but databricks doesn't have access to files on your computer. (Allowing websites to access arbitrary files on your computer would be a huge security issue. Image if I sent you a malicious link, and when you clicked it I could just read any file on your computer).
If you use the data button on the left side of the databricks UI, you can upload csv files and create tables that will be available tp your databricks cluster.
answered Nov 9 at 5:31
Bi Rico
19k33659
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
add a comment |
up vote
0
down vote
up vote
0
down vote
Databricks is a cloud service that runs a cluster in the cloud. You can interact with the cluster using a notebook in your browser, but databricks doesn't have access to files on your computer. (Allowing websites to access arbitrary files on your computer would be a huge security issue. Image if I sent you a malicious link, and when you clicked it I could just read any file on your computer).
If you use the data button on the left side of the databricks UI, you can upload csv files and create tables that will be available tp your databricks cluster.
answered Nov 9 at 5:31
Bi Rico
19k33659
Databricks is a cloud service that runs a cluster in the cloud. You can interact with the cluster using a notebook in your browser, but databricks doesn't have access to files on your computer. (Allowing websites to access arbitrary files on your computer would be a huge security issue. Image if I sent you a malicious link, and when you clicked it I could just read any file on your computer).
If you use the data button on the left side of the databricks UI, you can upload csv files and create tables that will be available tp your databricks cluster.
answered Nov 9 at 5:31
Bi Rico
19k33659
answered Nov 9 at 5:31
Bi Rico
19k33659
answered Nov 9 at 5:31
Bi Rico
19k33659
answered Nov 9 at 5:31
Bi Rico
19k33659
19k33659
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
add a comment |
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
Excelent, all is very clear, thanks a lot
– RGS
Nov 12 at 0:10
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53200843%2fwindows-file-in-spark-read-csv%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
