Adding custom appender from different module

[emelyanovkr]

My main goal is to use custom appender from different module in my project.

So, my project structure is:
Project:

• ClickHouseAppender
  • src
    • main
      • java
        • com.package.name.ClickHouseAppender
• MainService
  • src
    • main
      • java
        • com.package.name.LoaderApplication
      • resources
        • log4j2.xml

Configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <Console name="StdOut" target="SYSTEM_OUT">
            <JsonTemplateLayout eventTemplateUri="classpath:LogMsgLayout.json"/>
        </Console>
        <ClickHouseAppender name="ClickHouseAppender" ignoreExceptions="false">
            <JsonTemplateLayout eventTemplateUri="classpath:LogMsgLayout.json" />
        </ClickHouseAppender>
    </Appenders>
    <Loggers>
        <AsyncLogger name="com.crypto.service.appender" level="INFO" additivity="false">
            <AppenderRef ref="ClickHouseAppender"/>
            <AppenderRef ref="StdOut"/>
        </AsyncLogger>
        <Root level="ERROR">
            <AppenderRef ref="ClickHouseAppender"/>
            <AppenderRef ref="StdOut"/>
        </Root>
    </Loggers>
</Configuration>

When I'm trying to run my MainService:

2024-04-13T14:12:30.964285100Z main ERROR Error processing element ClickHouseAppender ([Appenders: null]): CLASS_NOT_FOUND
2024-04-13T14:12:31.055144600Z main ERROR Unable to locate appender "ClickHouseAppender" for logger config "root"
2024-04-13T14:12:31.055144600Z main ERROR Unable to locate appender "ClickHouseAppender" for logger config "com.crypto.service"

So, what is the way to add my customAppender to the project?
Build system is Gradle.

ClickHouseAppender using implementation project(':MainService') because it's has several dependencies from main project. So I can't just use the same implementation (...) feature for my MainProject -> circular dependencies.

[rgoers]

Please run with -Dlog4j2.debug=true so we can see the whole configuraton being processed.

Can you show the class definition of ClickHouseAppender along with the annotations on it? I am assuming that the ClickHoustAppender project is on the classpath when running the application?

[vy]

Next to what @rgoers says (and asks for), you need to make sure that you use the org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor annotation processor provided by the org.apache.logging.log4j:log4j-core artifact in your Gradle configuration while compiling the ClickHouseAppender module. See the Plugins page of the manual explaining how to do this in Maven.

[emelyanovkr]

Thanks a lot, and @rgoers too, for you answers. I'm trying to make ClickhouseAppender separate as possible and delete all dependencies in appender for main service, there I'll try to include appender in my main service in build.gradle. If nothing works, I'll be back to this thread. Reaching out in several days.

[emelyanovkr]

So I've managed to use my ClickHouse Appender as separate project and implementate it in build.gradle file of MainService. It's works totaly fine, however, I have a custom class

@Plugin(
    name = "ConnectionSettings",
    category = Core.CATEGORY_NAME,
    elementType = "connectionSettings",
    printObject = true)
public class ConnectionSettings {}

In my xml configuration I'm using it like this:

<ClickHouseAppender name="ClickHouseAppender" ignoreExceptions="false"
                            bufferSize=""
                            timeout=""
                            tableName="">

            <ConnectionSettings HOST=""
                                PORT=""
                                USERNAME=""
                                PASSWORD=""
                                DATABASE=""
                                SSL=""
                                SOCKET_TIMEOUT=""
                                MAX_EXECUTION_TIME=""
            />
            <JsonTemplateLayout eventTemplateUri="classpath:LogMsgLayout.json"/>
        </ClickHouseAppender>

My parameters are passing and it's okay, however, when I'm starting my program I get this message
2024-04-18T17:31:03.394033600Z main ERROR appender ClickHouseAppender has no parameter that matches element ConnectionSettings

Is there any way to fix this except of adding ConnectionSettings parameter in createAppender(...) as @PluginAttribute?

[emelyanovkr]

It would be great if you can publish this in a GitHub repository of yours or your employers! 🤩

It's public already, but, of course, work in progress.

What do you mean you're not using it?

I mean I'm using it in my ClickHouseAppenderProject, but not in a ClickHouseAppender class itself.
https://github.com/emelyanovkr/crypto-data-service-loader/blob/working_branch/ClickHouseAppender/src/main/java/com/crypto/service/util/ConnectionSettings.java - you can look by youself. ConnectionSettings are used in ClickHouseLogDao, but not in a ClickHouseAppender class.

[vy]

@emelyanovkr On this line, you need to add @PluginElement("ConnectionSettings") ConnectionSettings connectionSettings argument and use it to initialize ClickHouseAppender#logBuffer.

[emelyanovkr]

Yeah, I could do this. I can get rid of error with this, but I don't want to use ConnectionSettings in ClickHouseAppender class, I don't see any elegant solutions to use it there. So it will be just a line to get rid of an error. That's why I asked you about any solutions to this instead of adding this annotation in ClickHouseAppender class where I will not be using this ConnectionSettings object.

[vy]

@emelyanovkr, it is not only about suppressing the error, but it is the right way of designing the appender. The appender needs to accept the ConnectionSettings and then use a manager to obtain a connection using those settings. (The manager can decide whether it should create a new connection for each request or cache them and reuse wherever applicable. For simplicity, I would create a new connection each time.) See how FileAppender uses FileManager.getFileManager() for an example.

[emelyanovkr]

You are pointing to create a ConnectionPool for an appender, so he could request connection from this pool if this connection is broken. Retry connection feature will be implemented in future, I think.

[emelyanovkr]

@vy, good evening, could you please help me with the question: is it possible to have 2 loggers using the same appender but with different settings?
For example, I have a ClickHouseAppender configurable with tableName of table to insert Logs. I want to use one logger to log data in table1 and other logger to use table2. It is possible?

[vy]

Umm... yes, sort of. But why? Why don't you rather create two appenders and connect the associated loggers to each?

[emelyanovkr]

That's quite simple and brilliant answer, thank you! I haven't thought about it.

[emelyanovkr]

@vy, I got one more question. Could you please help me with more information. For example, I want to create in test package some BufferAppender, so I could append my messages in some buffer and therefore I could test wirtten messages in the future.

@Plugin(
    name = "BufferAppender",
    category = Core.CATEGORY_NAME,
    elementType = Appender.ELEMENT_TYPE,
    printObject = true)
public class BufferAppender extends AbstractAppender {

  private final List<String> logBuffer = new ArrayList<>();

  private BufferAppender(
      String name, Filter filter, Layout<String> layout, boolean ignoreExceptions) {
    super(name, filter, layout, ignoreExceptions, null);
  }

  @PluginFactory
  public static BufferAppender createAppender(
      @PluginAttribute("name") String name,
      @PluginElement("Filters") Filter filter,
      @PluginElement("layout") Layout<String> layout,
      @PluginAttribute("ignoreExceptions") boolean ignoreExceptions) {
    return new BufferAppender(name, filter, layout, ignoreExceptions);
  }

  @Override
  public void append(LogEvent event) {
    String serializedEvent = (String) getLayout().toSerializable(event);
    logBuffer.add(serializedEvent);
  }

  public List<String> getLogBuffer() {
    return logBuffer;
  }
}

However, maybe is there more optimal way to write Logs somewhere(not file), so I could easily test written messages?

I am failing to use this custom appender because I have message that
2024-05-05T14:11:52.010490900Z Test worker ERROR Error processing element BufferAppender ([Appenders: null]): CLASS_NOT_FOUND

This appender class located near my othet test classes.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <Console name="StdOut" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n"/>
        </Console>
        <BufferAppender name="BufferAppender" ignoreExceptions="false">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n"/>
        </BufferAppender>
    </Appenders>
    <Loggers>
        <Logger name="TestLogger" level="INFO" additivity="false">
            <AppenderRef ref="StdOut"/>
            <AppenderRef ref="BufferAppender"/>
        </Logger>
        <Root level="ERROR">
            <AppenderRef ref="StdOut"/>
            <AppenderRef ref="BufferAppender"/>
        </Root>
    </Loggers>
</Configuration>

[vy]

See ListAppender distributed with log4j-core-test. In the log4j-core-test module sources, you can see various usages of it too. In short, it delivers what your BufferAppender does.

That said, I am not a big fan of testing the logging framework output, because

1. It depends on the logging configuration, which can change in the future.
2. You need to start a fully blown logging infrastructure just to test a couple of log.info() statements.
3. It is very tricky to make such tests thread-safe. That is, you cannot run your tests in parallel.
4. You are testing your code and the layout, but you are actually interested in only the first part.

Instead, I'd suggest the following:

1. For classes you want to test their logging behaviour, create package-local ctors accepting the Logger instance. (You can by default set this to LogManager.getLogger(), etc.)
2. In the tests, provide a mock Logger and verify its invocations.

I have some JUnit utilities to make this experience even more pleasant. I showcased this earlier in the mailing list.

@emelyanovkr, can you stop reopening the issue again and again with different subjects, please? Instead, create a new discussion.

[emelyanovkr]

@vy, thanks for your great pointing, I'll try that out.

Sorry for an inconvience and reopening the discussion, I'll move forward to the next thread.
Wish you all the best.