Blob Storage (Object Storage)

A blob (Binary Large Object) is any unstructured data — images, videos, PDFs, backups, logs, machine learning models. Unlike structured data that fits neatly into database rows, blobs are opaque binary files that can range from a few kilobytes to several terabytes.

Blob storage (also called object storage) is a flat storage system designed specifically for storing and retrieving these large, unstructured files at massive scale. It is fundamentally different from file systems and databases.

1. Object Storage vs. File Systems vs. Databases

Why Not Store Blobs in a Database?

Storing a 50MB image as a BYTEA column in PostgreSQL causes several problems:

• Bloated database size: Backups, replication, and queries become slow.
• Memory pressure: The database loads blobs into memory during queries, consuming RAM meant for indexes and hot data.
• No CDN integration: You cannot serve database blobs through a CDN efficiently.

The standard pattern is: store metadata in the database, store the blob in object storage.

Database row:
┌──────────┬────────────────────────────────────────────────┐
│ photo_id │ blob_url                                       │
│ 42       │ s3://my-bucket/photos/2024/01/photo_42.jpg     │
│ 43       │ s3://my-bucket/photos/2024/01/photo_43.jpg     │
└──────────┴────────────────────────────────────────────────┘

Blob storage:
  s3://my-bucket/photos/2024/01/photo_42.jpg  →  [50 MB JPEG binary data]
  s3://my-bucket/photos/2024/01/photo_43.jpg  →  [32 MB JPEG binary data]

2. How Object Storage Works

The Data Model: Buckets and Objects

Object storage uses a flat namespace with two levels:

• Bucket: A top-level container (like a root folder). Each bucket has a globally unique name (e.g., my-app-photos).
• Object: A single file identified by a key (string path). The key can contain / characters for organizational purposes, but these are not real directories — the storage system is flat.

Bucket: my-app-photos
├── photos/2024/01/photo_42.jpg    ← Object (key = "photos/2024/01/photo_42.jpg")
├── photos/2024/01/photo_43.jpg    ← Object
├── videos/intro.mp4               ← Object
└── backups/db_dump_2024.sql.gz    ← Object

Each object consists of:

• Data: The binary file content (up to 5 TB in S3).
• Metadata: Key-value pairs describing the object (content type, upload date, custom tags).
• Key: The unique identifier within the bucket.

Operations

Object storage provides a simple API — there are no complex queries, no joins, no transactions:

3. How S3 Achieves 11 Nines of Durability

Amazon S3 guarantees 99.999999999% (11 nines) durability. This means if you store 10 million objects, you can expect to lose 1 object every 10,000 years. This is achieved through:

Data Replication and Erasure Coding

When you upload a file, S3 does not simply copy it to 3 servers (like a database replica). Instead, it uses erasure coding — a mathematical technique that splits the data into fragments and generates redundant parity fragments.

Original file: 100 MB

Erasure coding (Reed-Solomon):
  Split into 6 data fragments + 4 parity fragments = 10 fragments total
  Each fragment: ~17 MB
  Total storage: ~170 MB (1.7x overhead vs. 3x for simple replication)

  Any 6 out of 10 fragments can reconstruct the original file.
  Up to 4 fragments can be lost simultaneously without data loss.

These fragments are distributed across multiple physical disks, servers, and availability zones. Even if an entire data center burns down, the remaining fragments in other AZs can reconstruct every object.

Continuous Integrity Checks

S3 continuously scans stored data for bit rot (silent data corruption from disk degradation). When a corrupted fragment is detected, it is automatically repaired from the remaining healthy fragments.

4. Presigned URLs: Secure Direct Uploads

A common pattern for uploading blobs is to have the client upload directly to object storage, bypassing your application server entirely. This avoids bottlenecking your server with large file transfers.

// Server generates a presigned URL
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
 
async function generateUploadUrl(fileName: string): Promise<string> {
    const client = new S3Client({ region: "us-east-1" });
    const command = new PutObjectCommand({
        Bucket: "my-app-photos",
        Key: `uploads/${Date.now()}_${fileName}`,
        ContentType: "image/jpeg",
    });
 
    // URL expires in 15 minutes
    return getSignedUrl(client, command, { expiresIn: 900 });
}

Why presigned URLs?

• The client uploads directly to S3, so your server doesn't process the file data.
• The URL is time-limited and scoped to a specific key, so it cannot be reused or exploited.
• Works with any HTTP client — no AWS SDK needed on the client side.

5. CDN Integration

For serving blobs to users (images, videos, static assets), always place a CDN (CloudFront, Cloudflare) in front of your object storage:

• First request: CDN fetches the image from S3 (cache miss), caches it at the edge, and serves it to the user.
• Subsequent requests: CDN serves the cached image directly (cache hit), with latency under 50ms globally.

This reduces S3 costs (fewer GET requests), reduces latency (edge servers are closer to users), and protects S3 from traffic spikes.

6. Lifecycle Policies and Storage Tiers

Not all data is accessed equally. A photo uploaded yesterday is accessed frequently, but a photo from 5 years ago is rarely viewed. Object storage provides storage tiers with different cost and access characteristics:

Automated Lifecycle Rules

{
    "Rules": [
        {
            "Transition": { "Days": 30, "StorageClass": "STANDARD_IA" },
            "Description": "Move to IA after 30 days"
        },
        {
            "Transition": { "Days": 365, "StorageClass": "GLACIER" },
            "Description": "Archive after 1 year"
        },
        {
            "Expiration": { "Days": 2555 },
            "Description": "Delete after 7 years"
        }
    ]
}

This automatically moves data through tiers as it ages, dramatically reducing storage costs without manual intervention.

7. Multipart Uploads for Large Files

Uploading a single 5 GB file in one HTTP request is fragile — if the connection drops at 4.9 GB, the entire upload must restart. Multipart upload solves this:

1. Initiate multipart upload   → Server returns an Upload ID
2. Upload Part 1 (100 MB)      → Server returns ETag for Part 1
3. Upload Part 2 (100 MB)      → Server returns ETag for Part 2
   ...
50. Upload Part 50 (100 MB)    → Server returns ETag for Part 50
51. Complete multipart upload   → Server assembles all 50 parts into one object

Benefits:

• Resumable: If Part 23 fails, only Part 23 needs to be re-uploaded.
• Parallel: Upload multiple parts simultaneously to maximize bandwidth.
• Required for large files: S3 requires multipart upload for any object larger than 5 GB.

[!TIP] In a system design interview, when designing a file-sharing system (Dropbox, Google Drive), always mention presigned URLs for direct upload, multipart uploads for large files, and CDN for fast downloads. These three patterns handle 90% of blob storage design questions.