Effective logging and monitoring are essential for maintaining application health, quickly identifying issues, and improving performance. In this article, we’ll dive into logging and monitoring for Node.js applications, covering key topics like choosing logging levels, setting up structured logs, integrating with monitoring tools, and best practices for using Winston and Elasticsearch.

Introduction to Logging and Monitoring

Logging helps capture real-time events, errors, and other important information from the application, while monitoring involves tracking application performance metrics over time. Together, they provide critical insights into application health, enabling proactive issue resolution.

Setting Up Basic Logging in Node.js

The built-in console object provides simple logging functions, but a dedicated logging library is more robust for production applications.

Basic Console Logging

console.log("Server started on port 3000");
console.warn("This is a warning");
console.error("Error occurred while processing request");

However, console logging has limitations in complex applications, such as lack of log level control and no log persistence.

Introducing Winston

Winston is a popular logging library for Node.js that offers multiple log levels, transports (log destinations), and structured logging.

1. Install Winston:

   npm install winston

1. Setting Up Winston

   const winston = require("winston");

   // Configure logger
   const logger = winston.createLogger({
       level: "info",
       format: winston.format.combine(
           winston.format.timestamp(),
           winston.format.json()
       ),
       transports: [
           new winston.transports.Console(),
           new winston.transports.File({ filename: "app.log" })
       ]
   });

   // Logging examples
   logger.info("Server started on port 3000");
   logger.error("Database connection failed");

Choosing Appropriate Log Levels

Log levels categorize log messages based on their importance. Common log levels are:

• Error: Critical issues that require immediate attention, such as database or server failures.
• Warn: Non-critical issues, such as deprecated APIs.
• Info: General application information, like server startup or shutdown.
• Debug: Detailed information useful during development, such as variable values.

Configuring Log Levels in Winston

logger.level = "debug"; // Sets the minimum log level to debug, capturing all messages.

In production, it’s best to keep log levels at info or warn to avoid unnecessary log data.

Structured Logging for Consistency

Structured logging makes it easier to filter and analyze logs by maintaining a consistent format.

Adding Metadata to Logs

Metadata such as user_id or request_id can help track specific actions within logs:

logger.info("User login successful", { user_id: "12345" });
logger.error("Failed to fetch user data", { user_id: "12345", error: "Database unavailable" });

Integrating with Elasticsearch for Centralized Logging

Elasticsearch is widely used for centralized log management and search capabilities.

1. Install Elasticsearch and Elasticsearch Transport

   npm install @elastic/elasticsearch winston-elasticsearch

1. Configure Elasticsearch Transport

   const { ElasticsearchTransport } = require("winston-elasticsearch");

   const esTransport = new ElasticsearchTransport({
       clientOpts: { node: "http://localhost:9200" }
   });

   logger.add(esTransport);

This setup will send logs to Elasticsearch, allowing you to use Kibana for real-time log search and analysis.

Monitoring Application Metrics with Prometheus and Grafana

Monitoring tracks application performance metrics like CPU usage, memory, and response times, helping to ensure a stable application.

Setting Up Prometheus with Node.js

1. Install Prometheus Client Library

   npm install prom-client

1. Create and Export Metrics

   const client = require("prom-client");
   const httpRequestDuration = new client.Histogram({
       name: "http_request_duration_seconds",
       help: "Duration of HTTP requests in seconds",
       labelNames: ["method", "route"]
   });

   function startMonitoring(req, res, next) {
       const end = httpRequestDuration.startTimer();
       res.on("finish", () => end({ method: req.method, route: req.path }));
       next();
   }
   app.use(startMonitoring);

1. Expose Metrics Endpoint

   app.get("/metrics", async (req, res) => {
       res.set("Content-Type", client.register.contentType);
       res.end(await client.register.metrics());
   });

Visualizing with Grafana

Grafana is a powerful tool for creating dashboards from Prometheus metrics. Integrate Prometheus as a data source in Grafana, then visualize metrics such as response times and error rates.

Real-World Use Case: Logging and Monitoring in E-commerce

Consider an e-commerce platform where logging and monitoring are critical for maintaining high performance and reliability.

1. Log All Transactions: Capture order and payment events with structured logs, including metadata like order_id and user_id.
2. Error Tracking: Use Winston to log errors such as payment failures, along with stack traces and metadata for faster debugging.
3. Monitor Server Health: Set up Prometheus to monitor response times and request counts, visualized in Grafana for real-time insights.
4. Set Alerts: Configure alerts based on metrics. For instance, if the request duration exceeds a threshold, send an alert to the admin.

This setup provides a comprehensive view of the application’s health, allowing proactive issue detection and resolution.

Conclusion

Implementing robust logging and monitoring in Node.js is essential for maintaining reliability and ensuring quick troubleshooting. Using tools like Winston, Elasticsearch, Prometheus, and Grafana, you can capture structured logs, centralize them, and monitor critical performance metrics effectively.