Problem publishing JSON output to PubSub. A 'u' is being added to every field

I have a script that pulls from the city if Chicago and grabs a json file and then published to Pub Sub. Once the data gets into pub sub I have a dataflow template that pulls the data into Google Big Query. The final data move to BQ is failing and when I print the output in the script. I am getting a u' in front of all the fields which I believe is messing up the field match. Has anyone else had this issue and know what is wrong with my code and how to possibly remove the 'u'. I have tried multiple fix but none of them have worked. A sample out put is listed below:

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals

from sodapy import Socrata

import json

from io import StringIO

from google.oauth2 import service_account

from oauth2client.client import GoogleCredentials

from google.cloud import pubsub_v1

import time

import datetime

import urllib

import urllib.request

import argparse

import base64



credentials = GoogleCredentials.get_application_default()

# change project to your Project ID

project="xxxx"

# change topic to your PubSub topic name

topic="xxxx"





res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')

res_body = res.read()

traffic=json.loads(res_body)

publisher = pubsub_v1.PublisherClient()

topicName = 'projects/' + project + '/topics/' + topic

publisher = pubsub_v1.PublisherClient()

topic_path = publisher.topic_path(project,topic)

for key in traffic:

        publisher.publish(topicName,str.encode(str(key)))

        print(key.items())

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

add a comment |

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals

from sodapy import Socrata

import json

from io import StringIO

from google.oauth2 import service_account

from oauth2client.client import GoogleCredentials

from google.cloud import pubsub_v1

import time

import datetime

import urllib

import urllib.request

import argparse

import base64



credentials = GoogleCredentials.get_application_default()

# change project to your Project ID

project="xxxx"

# change topic to your PubSub topic name

topic="xxxx"





res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')

res_body = res.read()

traffic=json.loads(res_body)

publisher = pubsub_v1.PublisherClient()

topicName = 'projects/' + project + '/topics/' + topic

publisher = pubsub_v1.PublisherClient()

topic_path = publisher.topic_path(project,topic)

for key in traffic:

        publisher.publish(topicName,str.encode(str(key)))

        print(key.items())

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

add a comment |

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals

from sodapy import Socrata

import json

from io import StringIO

from google.oauth2 import service_account

from oauth2client.client import GoogleCredentials

from google.cloud import pubsub_v1

import time

import datetime

import urllib

import urllib.request

import argparse

import base64



credentials = GoogleCredentials.get_application_default()

# change project to your Project ID

project="xxxx"

# change topic to your PubSub topic name

topic="xxxx"





res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')

res_body = res.read()

traffic=json.loads(res_body)

publisher = pubsub_v1.PublisherClient()

topicName = 'projects/' + project + '/topics/' + topic

publisher = pubsub_v1.PublisherClient()

topic_path = publisher.topic_path(project,topic)

for key in traffic:

        publisher.publish(topicName,str.encode(str(key)))

        print(key.items())

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals

from sodapy import Socrata

import json

from io import StringIO

from google.oauth2 import service_account

from oauth2client.client import GoogleCredentials

from google.cloud import pubsub_v1

import time

import datetime

import urllib

import urllib.request

import argparse

import base64



credentials = GoogleCredentials.get_application_default()

# change project to your Project ID

project="xxxx"

# change topic to your PubSub topic name

topic="xxxx"





res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')

res_body = res.read()

traffic=json.loads(res_body)

publisher = pubsub_v1.PublisherClient()

topicName = 'projects/' + project + '/topics/' + topic

publisher = pubsub_v1.PublisherClient()

topic_path = publisher.topic_path(project,topic)

for key in traffic:

        publisher.publish(topicName,str.encode(str(key)))

        print(key.items())

python json google-bigquery publish-subscribe google-cloud-pubsub

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

edited Nov 13 '18 at 3:35

asked Nov 13 '18 at 3:11

john williams

asked Nov 13 '18 at 3:11

john williams

asked Nov 13 '18 at 3:11

john williams

add a comment |

1 Answer
1

active

oldest

votes

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

What's the u prefix in a Python string?

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']

[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'

True

Now when you say "I am getting a u' in front of all the fields which I believe is messing up the field match." Where do you see this? Is this part of the Python code, or do you see these u' show up on the BigQuery web UI too?

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"

'a'

>>> from __future__ import unicode_literals

>>> "a"

u'a'

https://python-future.org/unicode_literals.html

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273234%2fproblem-publishing-json-output-to-pubsub-a-u-is-being-added-to-every-field%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

What's the u prefix in a Python string?

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']

[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'

True

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"

'a'

>>> from __future__ import unicode_literals

>>> "a"

u'a'

https://python-future.org/unicode_literals.html

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

add a comment |

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

What's the u prefix in a Python string?

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']

[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'

True

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"

'a'

>>> from __future__ import unicode_literals

>>> "a"

u'a'

https://python-future.org/unicode_literals.html

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

add a comment |

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

What's the u prefix in a Python string?

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']

[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'

True

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"

'a'

>>> from __future__ import unicode_literals

>>> "a"

u'a'

https://python-future.org/unicode_literals.html

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

What's the u prefix in a Python string?

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']

[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'

True

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"

'a'

>>> from __future__ import unicode_literals

>>> "a"

u'a'

https://python-future.org/unicode_literals.html

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

edited Nov 13 '18 at 9:30

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

answered Nov 13 '18 at 4:20

Felipe Hoffa

20.8k249106

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

add a comment |

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails.
– john williams
Nov 13 '18 at 15:11

Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation.
– john williams
Nov 13 '18 at 15:19

If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace
– Felipe Hoffa
Nov 13 '18 at 16:36

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Vfrdtyky