Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MINOR: migrate DescribeConsumerGroupTest to use ClusterTestExtensions #15908

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

FrankYang0529
Copy link
Member

By using ClusterTestExtensions, DescribeConsumerGroupTest get get away from KafkaServerTestHarness dependency.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@@ -92,6 +94,7 @@ static void generator(ClusterGenerator clusterGenerator) {
static <T> AutoCloseable buildConsumers(int numberOfConsumers,
boolean syncCommit,
String topic,
Collection<TopicPartition> topicPartitions,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Change to set to be more specific.

@@ -311,6 +311,7 @@ private AutoCloseable consumerGroupClosable(ClusterInstance cluster, GroupProtoc
1,
false,
topicName,
null,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could we use empty set instead of null here?

Comment on lines +209 to +212
GroupProtocol groupProtocol = clusterInstance.config().serverProperties().get(GroupCoordinatorConfig.NEW_GROUP_COORDINATOR_ENABLE_CONFIG).trim().equals("true") ? GroupProtocol.CONSUMER : GroupProtocol.CLASSIC;
try (AutoCloseable protocolConsumerGroupExecutor = consumerGroupClosable(groupProtocol, PROTOCOL_GROUP, TOPIC, Collections.emptyMap());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks to me that we could put the logic of determining groupProtocol in consumerGroupClosable method, so that we don't have to do the same thing in each test. Ex: Add a consumerGroupClosable method, and replace the groupProtocol with clusterInstance.config().serverProperties(). So we can do the groupProtocol determination inside the consumerGroupClosable method:

private AutoCloseable consumerGroupClosable(Map<String, String> serverProperties, GroupProtocol protocol, String groupId, String topicName) {
    GroupProtocol groupProtocol = serverProperties.get(GroupCoordinatorConfig.NEW_GROUP_COORDINATOR_ENABLE_CONFIG).trim().equals("true") ? GroupProtocol.CONSUMER : GroupProtocol.CLASSIC;
    return   consumerGroupClosable(groupProtocol, ...);

@showuon
Copy link
Contributor

showuon commented May 22, 2024

One high-level question:
I found after this change, we increased the test case numbers from 492 -> 660. And the time for this test suite takes form 1h 19m -> 1h 31m. Do we really need all these tests? Could we try to:

  1. at least maintain the same test case numbers as before?
  2. (could be in another PR) Try to speed up the test. Maybe disable log.cleaner.enable, deploy for 1 nodes if we are not testing the data replication, or maybe reuse the same cluster for some similar tests... etc?

Anyway, to me, one test class takes more than 1 hour is really unacceptable.

Trunk:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2924/testReport/org.apache.kafka.tools.consumer.group/
This PR:
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-15908/1/testReport/org.apache.kafka.tools.consumer.group/

@showuon
Copy link
Contributor

showuon commented May 22, 2024

Forgot to say, thanks for closing all the ConsumerGroupService instances. I can't believe we leak these resources before.

@FrankYang0529 FrankYang0529 force-pushed the minor-migrate-DescribeConsumerGroup branch from bf608ec to 79dfa65 Compare May 22, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants