Firebase Pub/sub trigger: executing multiple times sporadically
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
firebase google-cloud-functions google-cloud-pubsub
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
add a comment |
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
firebase google-cloud-functions google-cloud-pubsub
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
2
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46
add a comment |
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
firebase google-cloud-functions google-cloud-pubsub
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
firebase google-cloud-functions google-cloud-pubsub
firebase google-cloud-functions google-cloud-pubsub
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
edited Nov 14 '18 at 20:24
Frank van Puffelen
237k29382408
237k29382408
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
asked Nov 14 '18 at 20:03
DygeratiDygerati
461313
461313
2
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46
add a comment |
2
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46
2
2
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46
add a comment |
1 Answer
1
active
oldest
votes
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53307957%2ffirebase-pub-sub-trigger-executing-multiple-times-sporadically%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
add a comment |
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
add a comment |
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
answered Nov 15 '18 at 15:22
Daniel CollinsDaniel Collins
1365
1365
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
add a comment |
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
It occurred to me but the logs indicate that the cronPublisher() function is only executing once; hence the confusion. Good call on the transaction, I may go that direction.
– Dygerati
Nov 16 '18 at 15:11
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53307957%2ffirebase-pub-sub-trigger-executing-multiple-times-sporadically%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Cloud Functions doesn't give you an exactly-once guarantee on your function executions. It gives you an at-least-once guarantee, which implies that your function may get executed more than once, but that should be rare. In any event, your function should be "idempotent" - resistant to being called multiple times. You may have to persist some data in a database to ensure that your function bails if some work has already been done.
– Doug Stevenson
Nov 14 '18 at 20:29
Yeah, I expected there may be an odd occurrence here and there, but this issue definitely hasn't fit that description. Every so-many days every single message is duplicated. I expect I'll need to rethink the architecture a bit but I'm trying to understand exactly what's happening.
– Dygerati
Nov 14 '18 at 21:10
Hey Doug where is the documentation that supports what you are saying about an at least once guarantee, which implies it may get executed more than once?
– toobsco42
Jan 30 at 16:16
How do you make a pub sub function atomic?
– toobsco42
Jan 30 at 17:46