The Fragmentation Trap: Files Smaller Than 128 KB
To reduce S3 storage costs, the immediate instinct is to configure Lifecycle Policies that move older data to “cold” storage classes such as S3 Standard-IA (Infrequent Access) or S3 Glacier. On paper, the cost per gigabyte drops dramatically. In practice, however, this can paradoxically cause your bill to skyrocket.
The 128 KB Rule (Minimum Object Size Penalty): AWS’s cost-effective storage classes, including Standard-IA, One Zone-IA, and Glacier, impose a minimum billable object size of 128 KB. This means the cloud provider bills for a 128 KB block for your file, regardless of its actual size.
If you archive millions of tiny files, such as error logs (2 KB), transaction receipts (5 KB), JSON metadata (10 KB), or image thumbnails (20 KB), the penalty can be severe.
The FinOps Disaster in Numbers: Imagine moving 10 million log files, each weighing 10 KB, to S3 Standard-IA.
Actual data size : 10 million × 10 KB = 100 GB.
Size billed by AWS : 10 million × 128 KB = 1.28 TB (terabytes).
The penalty : You pay for 1,280% of the data you actually own. On top of that, the transition itself generates 10 million lifecycle transition API requests, which can be expensive enough to wipe out years of anticipated storage savings.
The Optimization Strategy:
Aggregation (Tarballing): Before archiving, build data pipelines using AWS Lambda or AWS Glue to consolidate thousands of small daily files into a single archive, such as a TAR.GZ or Parquet file several dozen megabytes in size, and then move that archive to Glacier.
Keep Cold Data Hot: If aggregation is technically impossible, it is often more cost-effective to leave these small files in S3 Standard indefinitely, as this class imposes no minimum billable object size, rather than moving them to a lower-cost storage tier.
Continuously audit and master your S3 costs and risks
tamper helps you identify configuration drift and data governance risks, completely agentless and with zero latency.
Incomplete Multipart Uploads: S3’s Dark Matter
The second FinOps consideration involves an essential mechanism in network storage: multipart uploads.
How Does Multipart Upload Work? To upload a large file (over 100 MB) to S3, the API requires it to be split into multiple parts, which are uploaded in parallel, typically in chunks of 5 MB. The process consists of three steps:
The application initiates the request (CreateMultipartUpload).
It uploads the individual parts (UploadPart).
It instructs S3 to merge the parts and finalize the object (CompleteMultipartUpload).
The Disconnection Problem: What happens if the user’s laptop goes to sleep, the application crashes, or the network connection drops before the third step, when the upload is finalized?
The process is abandoned (Aborted). However, the parts that successfully reached AWS servers remain stored in the bucket. The real danger is that these orphaned parts are completely invisible: they do not appear in the AWS web console or in standard object listing requests (LIST Objects).
Yet AWS continues charging you the full S3 Standard storage rate for them, month after month. In Data Lakes or video platforms, where uploads of files measuring tens of gigabytes can fail regularly, this “dark matter” can account for terabytes of unnecessarily billed data.
The Optimization Strategy:
The solution is straightforward, but it must be implemented systematically across all your buckets from the moment they are created. Configure a lifecycle rule specifically targeting these remnants:
Action: Delete incomplete multipart uploads.
Delay: 7 days. With this rule in place, S3 automatically removes any file part that has been sitting around for more than a week without receiving the completion signal. Applying this rule incurs no cost, and the return on investment is immediate.
Conclusion
Managing the cost of an S3 infrastructure requires a detailed analysis of technical metrics. Using storage analytics tools such as S3 Storage Lens, combined with robust data governance, helps identify excessive fragmentation and automate the cleanup of abandoned multipart uploads. A FinOps-optimized bucket is not one where everything is stored in Glacier. It is one where the nature of the data dictates the right engineering approach.