Bigquery partition not being used while deleting data
Our project is being charged a high cost. Looks like such commands are not using the partition. Is there a way in bigquery to force the query to become partition aware based on the results for a subquery ?
The following query currently is processing TBs of data.
delete from tableName
where QuoteDate >= cast((SELECT batch_date as start from batch_table where Batch_Status='Running') as timestamp)
Processes 1TB of data.
Ideally this should only process hundred MB of data.
delete from TableName
where
QuoteDate >= '2018-11-01'
Processes ~100MB
google-bigquery asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
add a comment |
Our project is being charged a high cost. Looks like such commands are not using the partition. Is there a way in bigquery to force the query to become partition aware based on the results for a subquery ?
The following query currently is processing TBs of data.
delete from tableName
where QuoteDate >= cast((SELECT batch_date as start from batch_table where Batch_Status='Running') as timestamp)
Processes 1TB of data.
Ideally this should only process hundred MB of data.
delete from TableName
where
QuoteDate >= '2018-11-01'
Processes ~100MB
google-bigquery asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
add a comment |
Our project is being charged a high cost. Looks like such commands are not using the partition. Is there a way in bigquery to force the query to become partition aware based on the results for a subquery ?
The following query currently is processing TBs of data.
delete from tableName
where QuoteDate >= cast((SELECT batch_date as start from batch_table where Batch_Status='Running') as timestamp)
Processes 1TB of data.
Ideally this should only process hundred MB of data.
delete from TableName
where
QuoteDate >= '2018-11-01'
Processes ~100MB
google-bigquery asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
Our project is being charged a high cost. Looks like such commands are not using the partition. Is there a way in bigquery to force the query to become partition aware based on the results for a subquery ?
The following query currently is processing TBs of data.
delete from tableName
where QuoteDate >= cast((SELECT batch_date as start from batch_table where Batch_Status='Running') as timestamp)
Processes 1TB of data.
Ideally this should only process hundred MB of data.
delete from TableName
where
QuoteDate >= '2018-11-01'
Processes ~100MB
google-bigquery google-bigquery asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
asked Nov 13 '18 at 6:18
Ratnesh Sharma
65
65
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
From the documentation:
Complex queries that require the evaluation of multiple stages of a query in order to resolve the predicate (such as inner queries or subqueries) will not prune partitions from the query.
...
The following query does not prune partitions (note the use of a subquery):
#standardSQL
SELECT
t1.name,
t2.category
FROM
table1 t1
INNER JOIN
table2 t2
ON
t1.id_field = t2.field2
WHERE
t1.ts = (SELECT timestamp from table3 where key = 2)
For the time being, you need to evaluate the queries that you have provided separately.
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274927%2fbigquery-partition-not-being-used-while-deleting-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
From the documentation:
Complex queries that require the evaluation of multiple stages of a query in order to resolve the predicate (such as inner queries or subqueries) will not prune partitions from the query.
...
The following query does not prune partitions (note the use of a subquery):
#standardSQL
SELECT
t1.name,
t2.category
FROM
table1 t1
INNER JOIN
table2 t2
ON
t1.id_field = t2.field2
WHERE
t1.ts = (SELECT timestamp from table3 where key = 2)
For the time being, you need to evaluate the queries that you have provided separately.
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
add a comment |
From the documentation:
Complex queries that require the evaluation of multiple stages of a query in order to resolve the predicate (such as inner queries or subqueries) will not prune partitions from the query.
...
The following query does not prune partitions (note the use of a subquery):
#standardSQL
SELECT
t1.name,
t2.category
FROM
table1 t1
INNER JOIN
table2 t2
ON
t1.id_field = t2.field2
WHERE
t1.ts = (SELECT timestamp from table3 where key = 2)
For the time being, you need to evaluate the queries that you have provided separately.
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
add a comment |
From the documentation:
Complex queries that require the evaluation of multiple stages of a query in order to resolve the predicate (such as inner queries or subqueries) will not prune partitions from the query.
...
The following query does not prune partitions (note the use of a subquery):
#standardSQL
SELECT
t1.name,
t2.category
FROM
table1 t1
INNER JOIN
table2 t2
ON
t1.id_field = t2.field2
WHERE
t1.ts = (SELECT timestamp from table3 where key = 2)
For the time being, you need to evaluate the queries that you have provided separately.
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
From the documentation:
Complex queries that require the evaluation of multiple stages of a query in order to resolve the predicate (such as inner queries or subqueries) will not prune partitions from the query.
...
The following query does not prune partitions (note the use of a subquery):
#standardSQL
SELECT
t1.name,
t2.category
FROM
table1 t1
INNER JOIN
table2 t2
ON
t1.id_field = t2.field2
WHERE
t1.ts = (SELECT timestamp from table3 where key = 2)
For the time being, you need to evaluate the queries that you have provided separately.
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
answered Nov 13 '18 at 6:49
Elliott Brossard
16.2k21035
16.2k21035
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274927%2fbigquery-partition-not-being-used-while-deleting-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
