If you're working with a large amount of data on an AWS S3 bucket, then you need to learn about this S3 feature.

Let's say you wanted to remove objects from the S3 bucket on some conditions, the first thing you could think of is to write a script (eg. in Go) to retrieve the list of files and delete ones older than a certain date. While that's doable, however, it's not the best approach.

The easiest method is to define the Managing lifecycle on the Amazon S3 bucket.

The main parts of this article:
1- About AWS Services (Info)
2- Technical Part (Code)
3- Result
4- Conclusion

About AWS Services (Info)

Let's see how we can manage the lifecycle of objects in our S3 bucket.

You can specify that objects older than a certain number of days should be expired (deleted). The best part is that this happens automatically regularly and you don't need to run your script.

There are two types of actions:

• Transition actions - Let you transition objects between different storage classes.
• Expiration actions - Let you define expiration time for objects.

📋 Note: If you want to read more about AWS S3 Managing the lifecycle of objects, check out this link

Technical Part (Code)

AWS S3 bucket with Terraform that automatically expires objects older than one day

resource "aws_s3_bucket" "example_bucket" {
  bucket = var.s3_bucket_name

  lifecycle_rule {
    id      = "expire-1-day-old-objects"
    enabled = true
    prefix  = "temporary/"

    expiration {
      days = 1
    }
  }
}

Result

We can see when we uploaded an image inside the /temporary folder, it was removed after 1 day. Here is a picture of what we can see from the AWS console.

Here we can see the "Expiration date" inside the s3 object

Conclusion

In many cases, you need to enable lifecycle policies or expire rules on your object, especially if you are working with a large volume of data. In some cases, you store files on S3 for a temporary reason, or let's say you storing some logs, and we know that logs should be deleted after a certain period (for eg. after 2 months), so bucket lifecycle configuration will be very helpful and easy way to implement it.

Here is the Github Link

If you did like my content, and want to see more, feel free to connect with me on ➡️ LinkedIn, happy to guide or help with anything that needs clarification 😊💁