When to close a producer or consumer
Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.
Can Consumers can be long-running connections and left open?
Should producers be closed whenever we are done producing messages?
apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
add a comment |
Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.
Can Consumers can be long-running connections and left open?
Should producers be closed whenever we are done producing messages?
apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36
add a comment |
Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.
Can Consumers can be long-running connections and left open?
Should producers be closed whenever we are done producing messages?
apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.
Can Consumers can be long-running connections and left open?
Should producers be closed whenever we are done producing messages?
apache-kafka kafka-consumer-api kafka-producer-api
apache-kafka kafka-consumer-api kafka-producer-api
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
edited Nov 16 '18 at 1:13
cricket_007
83.3k1145114
83.3k1145114
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
asked Nov 15 '18 at 22:55
AchilleusAchilleus
688418
688418
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36
add a comment |
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36
add a comment |
1 Answer
1
active
oldest
votes
Can Consumers can be long-running connections and left open?
In general, yes.
In detail: depending on your consumer configuration.
If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.
If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.
Otherwise, they can stay - but why waste resources?
Should producers be closed whenever we are done producing messages?
In general, yes.
Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53329040%2fwhen-to-close-a-producer-or-consumer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Can Consumers can be long-running connections and left open?
In general, yes.
In detail: depending on your consumer configuration.
If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.
If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.
Otherwise, they can stay - but why waste resources?
Should producers be closed whenever we are done producing messages?
In general, yes.
Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
add a comment |
Can Consumers can be long-running connections and left open?
In general, yes.
In detail: depending on your consumer configuration.
If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.
If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.
Otherwise, they can stay - but why waste resources?
Should producers be closed whenever we are done producing messages?
In general, yes.
Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
add a comment |
Can Consumers can be long-running connections and left open?
In general, yes.
In detail: depending on your consumer configuration.
If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.
If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.
Otherwise, they can stay - but why waste resources?
Should producers be closed whenever we are done producing messages?
In general, yes.
Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
Can Consumers can be long-running connections and left open?
In general, yes.
In detail: depending on your consumer configuration.
If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.
If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.
Otherwise, they can stay - but why waste resources?
Should producers be closed whenever we are done producing messages?
In general, yes.
Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
answered Nov 16 '18 at 1:28
Adam KotwasinskiAdam Kotwasinski
2,543827
2,543827
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53329040%2fwhen-to-close-a-producer-or-consumer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There are tons of information available on this topic on the web, don't see a need to re-document this again. Personally I like ingest.tips which helped me in my initial days. Having said that, if you are not able to solve a specific issue, it may be worth documenting.
– AbhishekN
Nov 16 '18 at 0:06
If you're not actually using a Closable object anymore, yes, it's a good JVM practice to close those resources... That should translate directly to Kafka API as well
– cricket_007
Nov 16 '18 at 1:12
Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague
– cricket_007
Nov 16 '18 at 1:14
I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues.
– Achilleus
Nov 16 '18 at 6:24
You might want to clarify "frequently", and even better with a code example
– cricket_007
Nov 16 '18 at 6:36