Manoj SwamiAs developers working with Node.js, we often encounter challenges that push the boundaries of our applications. Recently, I faced a significant hurdle while working on a Node.js application managed by PM2: the dreaded "JavaScript heap out of memory" error. This blog post details my journey from encountering this error to finding a robust solution.
The Problem
Our application was tasked with processing a large database and generating a CSV file approximately 500MB in size. However, we kept running into a memory allocation error. The PM2 logs revealed the following error message:
<--- Last few GCs --->
[196927:0x56bb6a0] 41408 ms: Mark-Compact (reduce) 2042.6 (2087.0) -> 2042.6 (2084.0) MB, 189.72 / 0.00 ms (average mu = 0.162, current mu = 0.000) last resort; GC in old space requested
[196927:0x56bb6a0] 41621 ms: Mark-Compact (reduce) 2042.6 (2084.0) -> 2042.6 (2084.0) MB, 212.82 / 0.00 ms (average mu = 0.080, current mu = 0.000) last resort; GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
This error indicates that our Node.js process was trying to allocate more memory than was available in the JavaScript heap.
Initial Attempts
At first, we tried several approaches to resolve this issue:
- Creating a
.node-config.jsfile to set memory limits. - Modifying the PM2 ecosystem file to include memory settings.
- Adjusting environment variables in the start script.
However, none of these methods solved our problem completely. The application would still crash when processing large amounts of data.
The Solution
After much trial and error, we found a solution that worked consistently. The key was to set the Node.js memory limit directly in the PM2 start command:
pm2 start "NODE_ENV='production' NODE_OPTIONS='--max-old-space-size=8192' PORT=7123 npm start" --name "<app-name>"
Let's break down why this command works:
Direct Application: By setting
NODE_OPTIONSin the PM2 start command, we ensure that the memory limit is applied directly to the Node.js process that runs our application.Increased Heap Size: The
--max-old-space-size=8192option increases the maximum old space size to 8GB, giving our application more memory to work with.Environment Variables: We set
NODE_ENVandPORTdirectly in the command, ensuring they're available to our application.Named Process: We give our PM2 process a name ("download-service"), making it easier to manage.
Why This Works
This solution works because it applies the memory settings before our application starts running. By setting NODE_OPTIONS in the PM2 start command, we bypass any potential issues with PM2 not properly passing environment variables or configuration settings.
Additional Optimizations
While increasing the memory limit solved our immediate problem, it's important to note that this is not always the best long-term solution. Here are some additional optimization techniques to consider when dealing with large datasets:
Use Streams: When working with large files, use Node.js streams to process data in chunks rather than loading entire files into memory.
Implement Pagination: If you're querying large datasets from a database, implement pagination to fetch and process data in smaller batches.
Worker Threads: For CPU-intensive tasks, consider using worker threads to distribute the workload.
Memory Profiling: Regularly profile your application's memory usage to identify and fix leaks.
Conclusion
Dealing with memory issues in Node.js can be challenging, especially when working with large datasets. The solution we found — setting memory limits directly in the PM2 start command — provided a straightforward way to overcome the "JavaScript heap out of memory" error.
Remember, while increasing memory limits can solve immediate issues, it's crucial to also focus on optimizing your code for efficient memory usage. By combining proper memory allocation with optimized code, you can build robust Node.js applications capable of handling large-scale data processing tasks.
Happy coding!
