KAFKA-16702: Fix producer leaks in KafkaLog4jAppenderTest

[gharris1727]

The tests testRealProducerConfigWithSyncSendShouldNotThrowException and testRealProducerConfigWithSyncSendAndNotIgnoringExceptionsShouldThrowException create real producer instances, which are leaked when the test exits.

Instead, each test should be followed by a cleanup operation where the registered appender is removed and closed.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[chia7712]

nice fix. BTW, could you add null check as the producer can be null

https://github.com/apache/kafka/blob/trunk/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java#L373

[gharris1727]

@chia7712 I added the null check. I did notice that the append(LoggingEvent) method could also run into an NPE when called before activateOptions() or a IllegalStateException if called after close()

The first state doesn't appear to be possible, because activateOptions is called during initialization, before any append call. Only an improperly-initialized appender could cause an NPE. I didn't add a null check because it would never trigger in a real environment, but let me know if you think we should add a null check, or catch NullPointerException.

The IllegalStateException seems possible though, because log4j is using a copy-on-write array internally. I think that means that there's a short window after an appender is removed that it can still be accessed by other threads. I changed the implementation to catch it so that if another thread is using log4j at the same time, it won't accidentally get an exception.

[chia7712]

nice finding. one question: should this function log message instead of sending data by closed producer if closed is true? for example:

        if (closed) {
            LogLog.debug("sending to kafka by closed producer");
            return;
        }

[gharris1727]

I don't think so. The internal state of the producer is the source of truth on whether this is closed, not the closed variable. A race condition could still cause an IllegalStateException to occur even if there's an if-condition first.

[chia7712]

I don't think so. The internal state of the producer is the source of truth on whether this is closed, not the closed variable.

That makes sense. one more question: Is IllegalStateException caused only by "closed" producer?

[gharris1727]

This is in the javadoc for send:

     * @throws IllegalStateException if a transactional.id has been configured and no transaction has been started, or
     *                               when send is invoked after producer has been closed.

Idempotence and transactions are not in use for this producer. I am not aware of another cause of IllegalStateException that isn't documented, but the documentation could be wrong or there could be a bug, I have no idea.

[chia7712]

LGTM