Exploring AWS !!

Day 71

Amazon FSx for Lustre

Lustre makes it easy to launch and run the files systems.
Lustre file system is open-source, parallel file system supporting requirements of HPC simulation environments.

Features:
High-performance file system
Multiple deployment options
Integration with S3
Data accessible to other AWS services
Accessible from on-prem
Fully managed AWS service

Integration of Lustre with S3 makes it easier process datasets. On linking with S3, Lustre shows all object as files, and any changes made will be reflected in the bucket too.

Use Cases:
Machine Learning: ML workloads use large training data, multiple instances need to process this data simultaneously, so shared file storage system is very useful here.
Media processing and Transcoding: Media workflow such as, video rendering and visual effects, need compute and storage resources to handle massive data created.

Pricing Example for Windows File Server
Lets say we want to store 10 TB of general purpose file share data using HDD storage in Virginia region. Based on deduplication savings of 50-60%, we provision a 5 TB multi-AZ file system with 16 MBps of throughput capacity. Also assume that we have an average backup of 5 TB during the month.

Total monthly charge for Windows File Server:
Storage: 5 TB x $0.025 GB-month = $128/month
Throughput: 16 Mbps x $4.50/Mbps-month = $72/month
Backup: 5 TB x $0.050/GB-month = $256/month
Total Monthly Charge: $456 ($0.045/GB-month for 10 TB of data)

Pricing Example for Lustre
Lets say we have a scratch file system in Virginia, provisioned with 4800 GB of storage, and we spin our file system for 8 hour workload everyday and the shut it down. We do this for a complete month.

Total monthly charge for Lustre:
$0.14 GB-month/30/24 = $0.000194/GB-hour
4800 GB x $0.000194/GB-hour x 8 hours x 30 days = $224
Total Monthly Charge for Lustre: $224