How do I periodically purge a queue in celery rabbitmq?
How can I periodically purge celery queues, like per minute after there are any unfinished task left in queue then I need to purge all those task after a minute.
How do I do this?
python rabbitmq celery
edited Nov 14 '18 at 7:25
petezurich
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
add a comment |
How can I periodically purge celery queues, like per minute after there are any unfinished task left in queue then I need to purge all those task after a minute.
How do I do this?
python rabbitmq celery
edited Nov 14 '18 at 7:25
petezurich
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
add a comment |
How can I periodically purge celery queues, like per minute after there are any unfinished task left in queue then I need to purge all those task after a minute.
How do I do this?
python rabbitmq celery
edited Nov 14 '18 at 7:25
petezurich
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
How can I periodically purge celery queues, like per minute after there are any unfinished task left in queue then I need to purge all those task after a minute.
How do I do this?
python rabbitmq celery
python rabbitmq celery
edited Nov 14 '18 at 7:25
petezurich
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
edited Nov 14 '18 at 7:25
petezurich
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
edited Nov 14 '18 at 7:25
petezurich
3,57581734
edited Nov 14 '18 at 7:25
petezurich
3,57581734
edited Nov 14 '18 at 7:25
petezurich
3,57581734
3,57581734
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
asked Nov 14 '18 at 7:23
Shersha FnShersha Fn
52211127
52211127
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
If you need to timeout tasks, use Time to Live; otherwise purging the queue if there's an old task could case a new task to be discarded with it.
You can set it with
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
and your queue will now discard tasks older than 1 minute.
That said, there is a purge command in RMQ that you can just call on a timer.
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/AMQP.Queue.html
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
add a comment |
You can set time_limit for the task to kill it after a certain amount of time.
Celery Time Limits
@app.task(time_limit=60)
def long_running_task():
...
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53294991%2fhow-do-i-periodically-purge-a-queue-in-celery-rabbitmq%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you need to timeout tasks, use Time to Live; otherwise purging the queue if there's an old task could case a new task to be discarded with it.
You can set it with
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
and your queue will now discard tasks older than 1 minute.
That said, there is a purge command in RMQ that you can just call on a timer.
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/AMQP.Queue.html
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
add a comment |
If you need to timeout tasks, use Time to Live; otherwise purging the queue if there's an old task could case a new task to be discarded with it.
You can set it with
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
and your queue will now discard tasks older than 1 minute.
That said, there is a purge command in RMQ that you can just call on a timer.
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/AMQP.Queue.html
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
add a comment |
If you need to timeout tasks, use Time to Live; otherwise purging the queue if there's an old task could case a new task to be discarded with it.
You can set it with
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
and your queue will now discard tasks older than 1 minute.
That said, there is a purge command in RMQ that you can just call on a timer.
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/AMQP.Queue.html
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
If you need to timeout tasks, use Time to Live; otherwise purging the queue if there's an old task could case a new task to be discarded with it.
You can set it with
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
and your queue will now discard tasks older than 1 minute.
That said, there is a purge command in RMQ that you can just call on a timer.
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/AMQP.Queue.html
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
answered Nov 14 '18 at 7:32
TordekTordek
7,03122861
7,03122861
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
add a comment |
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
but i don't want all my queues to be cleared , only selected queues need to be cleared , that includes tickers , etc
– Shersha Fn
Nov 14 '18 at 10:20
add a comment |
You can set time_limit for the task to kill it after a certain amount of time.
Celery Time Limits
@app.task(time_limit=60)
def long_running_task():
...
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
add a comment |
You can set time_limit for the task to kill it after a certain amount of time.
Celery Time Limits
@app.task(time_limit=60)
def long_running_task():
...
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
add a comment |
You can set time_limit for the task to kill it after a certain amount of time.
Celery Time Limits
@app.task(time_limit=60)
def long_running_task():
...
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
You can set time_limit for the task to kill it after a certain amount of time.
Celery Time Limits
@app.task(time_limit=60)
def long_running_task():
...
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
answered Nov 14 '18 at 7:36
oneor0oneor0
607617
607617
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
add a comment |
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Hi I added to the queue like this @celery_app_cron.task(time_limit=60,queue='ticker_queue') but nothing is happening
– Shersha Fn
Nov 14 '18 at 9:59
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Does your task still execute after 60 seconds time limit?
– oneor0
Nov 14 '18 at 10:07
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
Yeah it is executing , after 60 seconds , all the pending queues are not cleared even after 60 seconds so task keeps on accumulating after evertime cron is run
– Shersha Fn
Nov 14 '18 at 10:16
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
I also tried expires=60 that also doesnt work
– Shersha Fn
Nov 14 '18 at 10:18
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
And also this is not what i want this kill the task that is currently executing the task greater than the time then it will kill that task , what i need is any task that is pending in the queue should be removed
– Shersha Fn
Nov 14 '18 at 12:23
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53294991%2fhow-do-i-periodically-purge-a-queue-in-celery-rabbitmq%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
