Revolutionize Cloud Logging: Introducing Log4J AWS Appenders for Seamless AWS Integration and Enhanced Security

Several months ago, I embarked on a personal project to integrate my application's logging with CloudWatch. Having used the AWS-provided log appender with AWS Lambda, I appreciated its user-friendly nature. However, for applications running on EC2 instances, the CloudWatch Logs Agent was the recommended approach. Upon searching, I only found an appender for Log4J 2.0 (assuming the Lambda appender leveraged some Lambda-specific features).

This project started as a weekend venture but evolved into a more extensive endeavor. I continued to refine and enhance the appender's functionality based on my experience with a semi-production project (operating 24/7, although not business-critical at present). After weeks of seemingly error-free operation, and with no additional features to add, it's time to release.

The JAR is now available on Maven Central, making it easy to incorporate into your project. Simply add the following dependency to your project POM:

<dependency>
    <groupId>com.kdgregory.log4j</groupId>
    <artifactId>aws-appenders</artifactId>
    <version>1.0.0</version>
</dependency>

Next, you need to incorporate the appender into your Log4J configuration. Here's an example configuration:

log4j.rootLogger=WARN, console

log4j.logger.com.example.log4j=DEBUG, cloudwatch
log4j.additivity.com.example.log4j=true

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch=com.kdgregory.log4j.aws.CloudWatchAppender
log4j.appender.cloudwatch.layout=org.apache.log4j.PatternLayout
log4j.appender.cloudwatch.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch.logGroup=ExampleCloudwatchLog
log4j.appender.cloudwatch.logStream={startupTimestamp}-{sequence}
log4j.appender.cloudwatch.batchDelay=2500
log4j.appender.cloudwatch.rotationMode=daily

For more information on cloud logging and integration, visit computerstechnicians.

It's worth noting that I establish a default ConsoleAppender and exclusively attach the CloudWatchAppender to my program's package (com.example.log4j). You may opt to channel all logs to CloudWatch, but be cautious that the AWS SDK performs its own logging; you'll want to avoid using the DEBUG level for it or the Apache HTTP client:

log4j.logger.org.apache.http=ERROR
log4j.logger.com.amazonaws=ERROR

The second crucial consideration is the logStream configuration parameter, which (in conjunction with logGroup) accommodates the use of substitution variables. In this scenario, I'm generating a new stream for each application run, rotated daily, with a sequence number to maintain a record of the various streams.

For additional information, please visit the project on GitHub. If you encounter any issues or desire an enhancement, feel free to submit them; while I cannot guarantee a prompt turnaround for enhancements, I will endeavor to resolve bugs within a few days.

Next, I'll be developing an appender for Kinesis Firehose, facilitating the integration of Kibana with ElasticSearch.