Kafka compression, how to limit it to some listed topics? How to use compressed.topics property using clients...
I've enabled kafka message compression from client's producer API like :
configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")
It works well, however if we want to limit the above compression to some listed topics, we've notice a way :
configProperties.put("compressed.topics", "topicA,topicB")
from this link :
https://cwiki.apache.org/confluence/display/KAFKA/Compression .
But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args. still the compression been applied. please help.
apache-kafka compression
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
add a comment |
I've enabled kafka message compression from client's producer API like :
configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")
It works well, however if we want to limit the above compression to some listed topics, we've notice a way :
configProperties.put("compressed.topics", "topicA,topicB")
from this link :
https://cwiki.apache.org/confluence/display/KAFKA/Compression .
But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args. still the compression been applied. please help.
apache-kafka compression
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
You need to setcompression.codeaccording to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release
– cricket_007
Nov 13 '18 at 14:59
add a comment |
I've enabled kafka message compression from client's producer API like :
configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")
It works well, however if we want to limit the above compression to some listed topics, we've notice a way :
configProperties.put("compressed.topics", "topicA,topicB")
from this link :
https://cwiki.apache.org/confluence/display/KAFKA/Compression .
But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args. still the compression been applied. please help.
apache-kafka compression
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
I've enabled kafka message compression from client's producer API like :
configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")
It works well, however if we want to limit the above compression to some listed topics, we've notice a way :
configProperties.put("compressed.topics", "topicA,topicB")
from this link :
https://cwiki.apache.org/confluence/display/KAFKA/Compression .
But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args. still the compression been applied. please help.
apache-kafka compression
apache-kafka compression
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
edited Nov 13 '18 at 14:55
cricket_007
80.1k1142110
80.1k1142110
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
asked Nov 13 '18 at 13:38
Rajashekhar MeesalaRajashekhar Meesala
286
286
You need to setcompression.codeaccording to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release
– cricket_007
Nov 13 '18 at 14:59
add a comment |
You need to setcompression.codeaccording to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release
– cricket_007
Nov 13 '18 at 14:59
You need to set
compression.code according to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release– cricket_007
Nov 13 '18 at 14:59
You need to set
compression.code according to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release– cricket_007
Nov 13 '18 at 14:59
add a comment |
1 Answer
1
active
oldest
votes
Compression is handled by several parameters in Kafka:
- On the brokers,
compression.typeconfigures the default for the entire cluster. It allows the valuesgzip,snappy,lz4,uncompressedorproducer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones),uncompressedis self-explanatory, and the defaultproducer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. - At the topic level on the broker, you can also specify the same configurations as above,
gzip,snappy,lz4,uncompresssedorproducer. They're exactly the same, but act as overrides specifically for the topic you set them to. - On the producer side, in code, you can also set
compression.type, and the possible values aregzip,snappy,lz4andnone, where this last one is the default.
You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53282271%2fkafka-compression-how-to-limit-it-to-some-listed-topics-how-to-use-compressed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Compression is handled by several parameters in Kafka:
- On the brokers,
compression.typeconfigures the default for the entire cluster. It allows the valuesgzip,snappy,lz4,uncompressedorproducer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones),uncompressedis self-explanatory, and the defaultproducer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. - At the topic level on the broker, you can also specify the same configurations as above,
gzip,snappy,lz4,uncompresssedorproducer. They're exactly the same, but act as overrides specifically for the topic you set them to. - On the producer side, in code, you can also set
compression.type, and the possible values aregzip,snappy,lz4andnone, where this last one is the default.
You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
add a comment |
Compression is handled by several parameters in Kafka:
- On the brokers,
compression.typeconfigures the default for the entire cluster. It allows the valuesgzip,snappy,lz4,uncompressedorproducer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones),uncompressedis self-explanatory, and the defaultproducer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. - At the topic level on the broker, you can also specify the same configurations as above,
gzip,snappy,lz4,uncompresssedorproducer. They're exactly the same, but act as overrides specifically for the topic you set them to. - On the producer side, in code, you can also set
compression.type, and the possible values aregzip,snappy,lz4andnone, where this last one is the default.
You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
add a comment |
Compression is handled by several parameters in Kafka:
- On the brokers,
compression.typeconfigures the default for the entire cluster. It allows the valuesgzip,snappy,lz4,uncompressedorproducer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones),uncompressedis self-explanatory, and the defaultproducer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. - At the topic level on the broker, you can also specify the same configurations as above,
gzip,snappy,lz4,uncompresssedorproducer. They're exactly the same, but act as overrides specifically for the topic you set them to. - On the producer side, in code, you can also set
compression.type, and the possible values aregzip,snappy,lz4andnone, where this last one is the default.
You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
Compression is handled by several parameters in Kafka:
- On the brokers,
compression.typeconfigures the default for the entire cluster. It allows the valuesgzip,snappy,lz4,uncompressedorproducer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones),uncompressedis self-explanatory, and the defaultproducer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them. - At the topic level on the broker, you can also specify the same configurations as above,
gzip,snappy,lz4,uncompresssedorproducer. They're exactly the same, but act as overrides specifically for the topic you set them to. - On the producer side, in code, you can also set
compression.type, and the possible values aregzip,snappy,lz4andnone, where this last one is the default.
You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:
https://kafka.apache.org/20/documentation.html
In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
answered Nov 13 '18 at 18:14
mjuarezmjuarez
9,81973751
9,81973751
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53282271%2fkafka-compression-how-to-limit-it-to-some-listed-topics-how-to-use-compressed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You need to set
compression.codeaccording to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release– cricket_007
Nov 13 '18 at 14:59