Fix JSON Output from API
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
So I've got this API that I'm iterating through using python. I'm very much a beginner so I'm sort've stumbling around all this stuff.
Because I'm appending the data of each page that I get to the end of a JSOn file, the data is coming out like below:
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
With square brackets around each data set of each page.
My goal at the end of this is to convert this data to a CSV file that will be easier to manipulate without coding. Given that, what would be the best way to clean up the data? I'll also post the code that I'm using to loop through the api requests just in case I'm doing something wrong there too.
pnum = 1
while pnum <4:
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text, file=open("output_test.json", "a"))
print (pnum)
pnum = pnum + 1
if response.text == '':
break
print('Done!')
Much appreciate any help I can get.
python json python-requests
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
add a comment |
So I've got this API that I'm iterating through using python. I'm very much a beginner so I'm sort've stumbling around all this stuff.
Because I'm appending the data of each page that I get to the end of a JSOn file, the data is coming out like below:
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
With square brackets around each data set of each page.
My goal at the end of this is to convert this data to a CSV file that will be easier to manipulate without coding. Given that, what would be the best way to clean up the data? I'll also post the code that I'm using to loop through the api requests just in case I'm doing something wrong there too.
pnum = 1
while pnum <4:
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text, file=open("output_test.json", "a"))
print (pnum)
pnum = pnum + 1
if response.text == '':
break
print('Done!')
Much appreciate any help I can get.
python json python-requests
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
add a comment |
So I've got this API that I'm iterating through using python. I'm very much a beginner so I'm sort've stumbling around all this stuff.
Because I'm appending the data of each page that I get to the end of a JSOn file, the data is coming out like below:
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
With square brackets around each data set of each page.
My goal at the end of this is to convert this data to a CSV file that will be easier to manipulate without coding. Given that, what would be the best way to clean up the data? I'll also post the code that I'm using to loop through the api requests just in case I'm doing something wrong there too.
pnum = 1
while pnum <4:
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text, file=open("output_test.json", "a"))
print (pnum)
pnum = pnum + 1
if response.text == '':
break
print('Done!')
Much appreciate any help I can get.
python json python-requests
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
So I've got this API that I'm iterating through using python. I'm very much a beginner so I'm sort've stumbling around all this stuff.
Because I'm appending the data of each page that I get to the end of a JSOn file, the data is coming out like below:
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
[{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"},{"example":true, "testing":34, "html":"example sentence.</div><div><br></div><div>Example sentence"}]
With square brackets around each data set of each page.
My goal at the end of this is to convert this data to a CSV file that will be easier to manipulate without coding. Given that, what would be the best way to clean up the data? I'll also post the code that I'm using to loop through the api requests just in case I'm doing something wrong there too.
pnum = 1
while pnum <4:
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.text, file=open("output_test.json", "a"))
print (pnum)
pnum = pnum + 1
if response.text == '':
break
print('Done!')
Much appreciate any help I can get.
python json python-requests
python json python-requests
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
edited Nov 17 '18 at 5:33
cricket_007
84.6k1147120
84.6k1147120
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
asked Nov 17 '18 at 4:58
Sam KrauserSam Krauser
312
312
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Open the file before you loop (1), then accumulate all objects (2), then dump the list out to the file after the loop of requests (3)
Should create a single file that you can parse with other JSON tools
import json
with open("output_test.json", "w") as f: # (1)
pages =
for pnum in range(1, 5):
print (pnum)
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
req = requests.request("GET", url, headers=headers, params=querystring)
resp = req.json()
if not resp:
break
for x in resp: # assuming this is a list object
pages.append(x) # (2)
json.dump(pages, f) # (3)
print('Done!')
Personally, I think JSON files are more easily parsable than CSV
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53348378%2ffix-json-output-from-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Open the file before you loop (1), then accumulate all objects (2), then dump the list out to the file after the loop of requests (3)
Should create a single file that you can parse with other JSON tools
import json
with open("output_test.json", "w") as f: # (1)
pages =
for pnum in range(1, 5):
print (pnum)
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
req = requests.request("GET", url, headers=headers, params=querystring)
resp = req.json()
if not resp:
break
for x in resp: # assuming this is a list object
pages.append(x) # (2)
json.dump(pages, f) # (3)
print('Done!')
Personally, I think JSON files are more easily parsable than CSV
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
add a comment |
Open the file before you loop (1), then accumulate all objects (2), then dump the list out to the file after the loop of requests (3)
Should create a single file that you can parse with other JSON tools
import json
with open("output_test.json", "w") as f: # (1)
pages =
for pnum in range(1, 5):
print (pnum)
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
req = requests.request("GET", url, headers=headers, params=querystring)
resp = req.json()
if not resp:
break
for x in resp: # assuming this is a list object
pages.append(x) # (2)
json.dump(pages, f) # (3)
print('Done!')
Personally, I think JSON files are more easily parsable than CSV
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
add a comment |
Open the file before you loop (1), then accumulate all objects (2), then dump the list out to the file after the loop of requests (3)
Should create a single file that you can parse with other JSON tools
import json
with open("output_test.json", "w") as f: # (1)
pages =
for pnum in range(1, 5):
print (pnum)
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
req = requests.request("GET", url, headers=headers, params=querystring)
resp = req.json()
if not resp:
break
for x in resp: # assuming this is a list object
pages.append(x) # (2)
json.dump(pages, f) # (3)
print('Done!')
Personally, I think JSON files are more easily parsable than CSV
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
Open the file before you loop (1), then accumulate all objects (2), then dump the list out to the file after the loop of requests (3)
Should create a single file that you can parse with other JSON tools
import json
with open("output_test.json", "w") as f: # (1)
pages =
for pnum in range(1, 5):
print (pnum)
querystring = {"q":"A","pageNumber":pnum,"pageSize":"2"}
req = requests.request("GET", url, headers=headers, params=querystring)
resp = req.json()
if not resp:
break
for x in resp: # assuming this is a list object
pages.append(x) # (2)
json.dump(pages, f) # (3)
print('Done!')
Personally, I think JSON files are more easily parsable than CSV
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
edited Nov 17 '18 at 5:32
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
answered Nov 17 '18 at 5:27
cricket_007cricket_007
84.6k1147120
84.6k1147120
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
add a comment |
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
Ah awesome, thanks man. 2 questions if you have the time answer: 1. How would I go about continuing the loop until the resp is empty? is it just 'while True'? 2. In what way do you think it's easier to parse? definitely think I need to learn a bit more about JSON.
– Sam Krauser
Nov 17 '18 at 6:24
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
You could do a while True, but I would check if the request comes back with a 404 status code, then break the loop there. Or find another way to know when there are no pages left... And JSON can have nested structures and it's much easier to reason about for parsing in any other code compared to poorly formatted CSV. The only reason I would use CSV is if I used Excel, which is almost never
– cricket_007
Nov 17 '18 at 7:51
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53348378%2ffix-json-output-from-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
