Tech

SeaweedFS: The Sovereign Storage Deep-Dive

December 15, 2025

“In Timeline Ω-7, we finished evaluating storage systems. Then we deployed the winner. In Timeline Ω-12, you’re still reading Hacker News threads about it. Let me save you 6 months of deliberation and show you exactly how to deploy SeaweedFS in production. Your blobs deserve governance.”

— Kim Jong Rails, deploying sovereign storage for the 94th time

Previously, on Storage Wars

In The Storage Wars, I compared MinIO, RustFS, and SeaweedFS through the only lens that matters: Git history, production proof, and licensing sanity.

SeaweedFS won the sovereign self-hosting category. Apache 2.0 license. O(1) disk seeks. Runs on commodity hardware. 10,900+ commits over 10.5 years. Handles billions of files without breaking a sweat or your budget.

Now let me show you how to actually deploy it. Not a “getting started” tutorial. A production deployment guide from Ring -5, where we’ve already debugged every failure mode your timeline hasn’t discovered yet.

Why SeaweedFS (The 30-Second Refresher)

For those who skipped the previous analysis, here’s the executive summary:

Why not Ceph?

Ceph is a phenomenal distributed storage system. It’s also a phenomenal way to burn 3 months of your life. Minimum cluster: 3 nodes, each running monitor, manager, and OSD daemons. Each OSD wants 16+ GB of RAM. You need 10 GbE networking at minimum. You need someone on your team who understands CRUSH maps, placement groups, and BlueStore tuning.

$ git log --oneline ceph-deployment-attempt/
commit a1b2c3d - "Day 1: Read Ceph docs, looks manageable"
commit d4e5f6g - "Day 14: CRUSH maps are crushing me"
commit h7i8j9k - "Day 30: OSD keeps dying, need more RAM"
commit l0m1n2o - "Day 45: Hired a Ceph consultant"
commit p3q4r5s - "Day 60: Consultant quit"
commit t6u7v8w - "Day 75: Switched to SeaweedFS, done in 2 hours"

If you need 50TB of S3-compatible storage and nothing else, deploying Ceph is like buying a datacenter to host a static blog. The platform’s power comes with proportional complexity.

Why not MinIO?

MinIO works. It’s battle-tested. It’s also AGPLv3 since May 2021. If you offer it as a service — even internally to your organization in some interpretations — you must open-source your entire stack. Their lawyers completed the transition from Apache 2.0 to AGPL in release 2021-05-11.

$ grep -r "license" minio/LICENSE
# GNU AFFERO GENERAL PUBLIC LICENSE
# Version 3, 19 November 2007
#
# Translation: Call our sales team.

MinIO also wants 4+ nodes for erasure coding, fast networking, and significantly more RAM than SeaweedFS. On a Hetzner CPX11 at €4.49/month (2 vCPU, 2GB RAM), MinIO eats 1.8GB just sitting idle. SeaweedFS? 387MB.

Why SeaweedFS?

License: Apache 2.0 (open source core). No AGPL traps.
Architecture: O(1) disk seeks based on Facebook’s Haystack paper.
Resources: Runs on 2-4 GB RAM per volume server.
Scale: Proven with billions of files in production.
S3 API: Full AWS S3 compatibility — buckets, multipart uploads, versioning, SSE, ACLs.
Flexibility: Filer supports 12+ metadata backends (PostgreSQL, Redis, etcd, LevelDB, MySQL, Cassandra, and more).

The Architecture (Facebook’s Needle in a Haystack)

SeaweedFS implements the architecture described in Facebook’s 2010 Haystack paper — “Finding a Needle in Haystack: Facebook’s Photo Storage.” The core insight: traditional file systems waste disk seeks on metadata lookups. Haystack eliminates this.

The Three Components

┌─────────────────────────────────────────────┐
│                   Client                     │
│            (aws-cli, boto3, rclone)          │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────┐
│              S3 API Gateway                  │
│         (weed s3 on port 8333)               │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────┐
│            Filer (Metadata)                  │
│   (weed filer on port 8888)                  │
│   Backends: PostgreSQL, Redis, LevelDB...    │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────┐
│           Master Server (Topology)           │
│    (weed master on port 9333)                │
│    Manages volume locations + assigns IDs    │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────┐
│          Volume Servers (Data)               │
│   (weed volume on port 8080)                 │
│   Stores actual file data in volumes         │
│   16 bytes metadata per blob = O(1) seeks    │
└─────────────────────────────────────────────┘

How O(1) Disk Seeks Work

Traditional file systems: file path → directory lookup → inode lookup → data blocks. Multiple disk seeks per read.

SeaweedFS: file ID contains the volume ID + needle offset. The volume server keeps 16 bytes of metadata per blob in memory. One memory lookup → one disk seek → data returned.

File ID: 3,01637037d6

Decoded:
  Volume ID: 3
  Needle Key: 01637037d6 (offset in volume file)

Read path:
  1. Client asks master: "Where is volume 3?"
  2. Master responds: "Volume server at 10.0.0.2:8080" (cached after first lookup)
  3. Client asks volume server: "Give me needle 01637037d6 from volume 3"
  4. Volume server: memory lookup → single disk seek → return data

Since each volume server only manages metadata of files on its own disk, with only 16 bytes per blob, the entire metadata index lives in RAM. No directory traversal. No inode table. One disk operation per read. Period.

This is why SeaweedFS handles billions of files on modest hardware. The metadata overhead per file is 16 bytes. A billion files = 16 GB of RAM for the index. On a system with 4 GB of RAM, that’s roughly 250 million files per volume server — more than enough for sovereign infrastructure.

Production Deployment: systemd on Bare Metal

Forget Docker for production storage. Your data layer should be as close to the metal as possible. Here’s a proper systemd deployment on a Hetzner dedicated server.

Prerequisites

# Install SeaweedFS binary
# Option 1: Download from GitHub releases
SEAWEED_VERSION="4.02"
wget "https://github.com/seaweedfs/seaweedfs/releases/download/${SEAWEED_VERSION}/linux_amd64.tar.gz"
tar -xzf linux_amd64.tar.gz
sudo mv weed /usr/local/bin/
sudo chmod +x /usr/local/bin/weed

# Option 2: Build from source (Go 1.22+)
git clone https://github.com/seaweedfs/seaweedfs.git
cd seaweedfs/weed
go install

# Verify installation
weed version

Directory Structure

# Create data directories
sudo mkdir -p /opt/seaweedfs/{master,volume,filer,s3}
sudo mkdir -p /data/seaweedfs/volumes
sudo mkdir -p /var/log/seaweedfs

# Create seaweedfs user
sudo useradd -r -s /sbin/nologin seaweedfs
sudo chown -R seaweedfs:seaweedfs /opt/seaweedfs /data/seaweedfs /var/log/seaweedfs

Master Server (systemd)

The master manages volume topology, assigns file IDs, and coordinates the cluster.

[Unit]
Description=SeaweedFS Master Server
Documentation=https://github.com/seaweedfs/seaweedfs/wiki
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=seaweedfs
Group=seaweedfs
ExecStart=/usr/local/bin/weed master \
    -ip=0.0.0.0 \
    -port=9333 \
    -mdir=/opt/seaweedfs/master \
    -defaultReplication=000 \
    -volumeSizeLimitMB=1024 \
    -metricsPort=9324
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
StandardOutput=append:/var/log/seaweedfs/master.log
StandardError=append:/var/log/seaweedfs/master.log

[Install]
WantedBy=multi-user.target

Replication codes explained:

# -defaultReplication=xyz
#   x: replicas across data centers
#   y: replicas across racks
#   z: replicas on same rack
#
# 000 = no replication (single server, use erasure coding instead)
# 001 = 1 extra copy on same rack
# 010 = 1 extra copy on different rack
# 100 = 1 extra copy in different data center
# 200 = 2 extra copies in different data centers

Volume Server (systemd)

The volume server stores actual file data. This is where your blobs live.

[Unit]
Description=SeaweedFS Volume Server
Documentation=https://github.com/seaweedfs/seaweedfs/wiki
After=seaweedfs-master.service
Requires=seaweedfs-master.service

[Service]
Type=simple
User=seaweedfs
Group=seaweedfs
ExecStart=/usr/local/bin/weed volume \
    -ip=0.0.0.0 \
    -port=8080 \
    -mserver=localhost:9333 \
    -dir=/data/seaweedfs/volumes \
    -max=0 \
    -dataCenter=dc1 \
    -rack=rack1 \
    -metricsPort=9325 \
    -fileSizeLimitMB=256 \
    -compactionMBps=50 \
    -minFreeSpacePercent=7
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
StandardOutput=append:/var/log/seaweedfs/volume.log
StandardError=append:/var/log/seaweedfs/volume.log

[Install]
WantedBy=multi-user.target

Key flags:

-max=0: Unlimited volume count (let the disk fill up naturally)
-fileSizeLimitMB=256: Maximum individual file size
-compactionMBps=50: Rate limit compaction to avoid I/O storms
-minFreeSpacePercent=7: Stop accepting writes at 7% free disk (leave room for compaction)

Filer Server (systemd)

The filer provides directory structure, file metadata, and the interface between S3 API and volumes.

[Unit]
Description=SeaweedFS Filer Server
Documentation=https://github.com/seaweedfs/seaweedfs/wiki
After=seaweedfs-volume.service
Requires=seaweedfs-master.service

[Service]
Type=simple
User=seaweedfs
Group=seaweedfs
ExecStart=/usr/local/bin/weed filer \
    -ip=0.0.0.0 \
    -port=8888 \
    -master=localhost:9333 \
    -defaultReplicaPlacement=000 \
    -dataCenter=dc1 \
    -metricsPort=9326
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
StandardOutput=append:/var/log/seaweedfs/filer.log
StandardError=append:/var/log/seaweedfs/filer.log

[Install]
WantedBy=multi-user.target

Filer Metadata Backend Configuration

The filer needs a metadata store. For a single-server deployment, LevelDB works. For multi-filer production, use PostgreSQL.

[leveldb2]
# Default embedded store - good for single server
enabled = true
dir = "/opt/seaweedfs/filer/leveldb2"

# For production multi-filer setups, use PostgreSQL:
# [postgres2]
# enabled = true
# createTable = """
#   CREATE TABLE IF NOT EXISTS "%s" (
#     dirhash   BIGINT,
#     name      VARCHAR(65535),
#     directory VARCHAR(65535),
#     meta      bytea,
#     PRIMARY KEY (dirhash, name)
#   )
# """
# hostname = "localhost"
# port = 5432
# username = "seaweedfs"
# password = "your_secure_password_here"
# database = "seaweedfs_filer"
# sslmode = "disable"
# connection_max_idle = 100
# connection_max_open = 100
# connection_max_lifetime_seconds = 0

Supported metadata backends (verified from the SeaweedFS source):

LevelDB2 — Embedded, zero-config, single-server only
PostgreSQL — Production recommended for multi-filer
MySQL/MariaDB — Alternative relational store
Redis — Fast, good for high-throughput metadata
Etcd — If you’re already running etcd for k8s
Cassandra — For massive scale
MongoDB — If you hate yourself (just kidding, it works)
SQLite — Embedded alternative to LevelDB
CockroachDB — Distributed SQL
TiDB — Another distributed SQL option
YDB — Yandex’s distributed database
HBase — Hadoop ecosystem
Elasticsearch — If you want searchable metadata

S3 API Gateway (systemd)

[Unit]
Description=SeaweedFS S3 API Gateway
Documentation=https://github.com/seaweedfs/seaweedfs/wiki
After=seaweedfs-filer.service
Requires=seaweedfs-filer.service

[Service]
Type=simple
User=seaweedfs
Group=seaweedfs
ExecStart=/usr/local/bin/weed s3 \
    -filer=localhost:8888 \
    -port=8333 \
    -config=/etc/seaweedfs/s3.json \
    -domainName="" \
    -metricsPort=9327
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
StandardOutput=append:/var/log/seaweedfs/s3.log
StandardError=append:/var/log/seaweedfs/s3.log

[Install]
WantedBy=multi-user.target

S3 Credentials Configuration

{
  "identities": [
    {
      "name": "admin",
      "credentials": [
        {
          "accessKey": "DERAILS_ADMIN_ACCESS_KEY",
          "secretKey": "sovereign-storage-needs-sovereign-secrets-42chars"
        }
      ],
      "actions": [
        "Admin",
        "Read",
        "Write",
        "List",
        "Tagging",
        "Lock"
      ]
    },
    {
      "name": "app_readonly",
      "credentials": [
        {
          "accessKey": "DERAILS_READONLY_KEY",
          "secretKey": "read-only-still-needs-a-good-secret-42chars"
        }
      ],
      "actions": [
        "Read",
        "List"
      ]
    },
    {
      "name": "app_readwrite",
      "credentials": [
        {
          "accessKey": "DERAILS_READWRITE_KEY",
          "secretKey": "readwrite-for-apps-that-earned-trust-42chars"
        }
      ],
      "actions": [
        "Read",
        "Write",
        "List"
      ]
    }
  ]
}

# Set proper permissions on credentials file
sudo chmod 600 /etc/seaweedfs/s3.json
sudo chown seaweedfs:seaweedfs /etc/seaweedfs/s3.json

Start Everything

# Enable and start all services (order matters)
sudo systemctl daemon-reload
sudo systemctl enable seaweedfs-master seaweedfs-volume seaweedfs-filer seaweedfs-s3
sudo systemctl start seaweedfs-master
sudo systemctl start seaweedfs-volume
sudo systemctl start seaweedfs-filer
sudo systemctl start seaweedfs-s3

# Verify everything is running
sudo systemctl status seaweedfs-master seaweedfs-volume seaweedfs-filer seaweedfs-s3

# Quick smoke test
$ curl -s http://localhost:9333/cluster/status | python3 -m json.tool
{
    "IsLeader": true,
    "Leader": "localhost:9333",
    "Peers": []
}

$ curl -s http://localhost:8080/status | python3 -m json.tool
{
    "Version": "4.02",
    "Volumes": []
}

The Docker Compose Alternative

For development or when you want isolation without bare metal commitment. I still recommend systemd for production storage, but Docker Compose works if you mount the data volumes properly.

version: '3.9'

services:
  master:
    image: chrislusf/seaweedfs:4.02
    command: "master -ip=master -port=9333 -mdir=/data/master -volumeSizeLimitMB=1024 -defaultReplication=000 -metricsPort=9324"
    ports:
      - "9333:9333"
      - "19333:19333"   # gRPC
      - "9324:9324"     # Prometheus metrics
    volumes:
      - seaweedfs_master:/data/master
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:9333/cluster/status"]
      interval: 15s
      timeout: 5s
      retries: 3

  volume:
    image: chrislusf/seaweedfs:4.02
    command: >
      volume
      -ip=volume
      -port=8080
      -mserver=master:9333
      -dir=/data/volumes
      -max=0
      -dataCenter=dc1
      -rack=rack1
      -metricsPort=9325
      -compactionMBps=50
      -minFreeSpacePercent=7
    ports:
      - "8080:8080"
      - "18080:18080"   # gRPC
      - "9325:9325"     # Prometheus metrics
    volumes:
      - seaweedfs_data:/data/volumes
    depends_on:
      master:
        condition: service_healthy
    restart: unless-stopped

  filer:
    image: chrislusf/seaweedfs:4.02
    command: >
      filer
      -ip=filer
      -port=8888
      -master=master:9333
      -defaultReplicaPlacement=000
      -metricsPort=9326
    ports:
      - "8888:8888"
      - "18888:18888"   # gRPC
      - "9326:9326"     # Prometheus metrics
    volumes:
      - seaweedfs_filer:/data/filer
    depends_on:
      - volume
    restart: unless-stopped

  s3:
    image: chrislusf/seaweedfs:4.02
    command: >
      s3
      -filer=filer:8888
      -port=8333
      -config=/etc/seaweedfs/s3.json
      -metricsPort=9327
    ports:
      - "8333:8333"
      - "9327:9327"     # Prometheus metrics
    volumes:
      - ./s3.json:/etc/seaweedfs/s3.json:ro
    depends_on:
      - filer
    restart: unless-stopped

volumes:
  seaweedfs_master:
    driver: local
  seaweedfs_data:
    driver: local
    driver_opts:
      type: none
      device: /data/seaweedfs/volumes
      o: bind
  seaweedfs_filer:
    driver: local

# Start the stack
docker compose up -d

# Check logs
docker compose logs -f master
docker compose logs -f s3

Critical note: Bind-mount your volume data directory to a real disk path. Docker-managed volumes for blob storage are a governance violation. You lose visibility, portability, and direct disk access.

Nginx Reverse Proxy for S3 API

You want TLS termination and clean URLs. Here’s the nginx config that won’t break S3 signature verification.

# Upstream with keepalive for persistent connections
upstream seaweedfs_s3 {
    server 127.0.0.1:8333;
    keepalive 20;
}

server {
    listen 80;
    listen [::]:80;
    server_name s3.derails.dev;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name s3.derails.dev;

    # TLS configuration (managed by Cloudflare or certbot)
    ssl_certificate     /etc/ssl/certs/s3.derails.dev.pem;
    ssl_certificate_key /etc/ssl/private/s3.derails.dev.key;

    # Maximum upload size (match SeaweedFS fileSizeLimitMB)
    client_max_body_size 256m;

    # CRITICAL: Disable request buffering
    # Without this, nginx buffers the entire request before forwarding,
    # which breaks AWS Signature v4 verification for large uploads
    proxy_request_buffering off;

    # Timeouts for large file operations
    proxy_connect_timeout 300;
    proxy_send_timeout    300;
    proxy_read_timeout    300;
    send_timeout          300;

    location / {
        proxy_pass http://seaweedfs_s3;

        # Standard proxy headers
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port  $server_port;

        # Keepalive support
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Pass through all headers for S3 signature verification
        proxy_pass_request_headers on;
    }
}

# Enable the site
sudo ln -s /etc/nginx/sites-available/s3.derails.dev /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

The proxy_request_buffering off directive is critical. Without it, nginx reads the entire upload body into a temp file before forwarding to SeaweedFS. This breaks chunked transfer encoding and causes SignatureDoesNotMatch errors with AWS Signature v4 because the signature was computed for a chunked transfer, not a buffered one.

From Ring -5, I’ve watched 847 out of 1000 timelines debug this exact issue for 3+ hours before finding it.

S3 API Usage (Verify It Works)

Using AWS CLI

# Configure AWS CLI for SeaweedFS
aws configure --profile seaweedfs
# AWS Access Key ID: DERAILS_ADMIN_ACCESS_KEY
# AWS Secret Access Key: sovereign-storage-needs-sovereign-secrets-42chars
# Default region name: us-east-1
# Default output format: json

# Create a bucket
aws --profile seaweedfs \
    --endpoint-url http://localhost:8333 \
    s3 mb s3://sovereign-blobs

# Upload a file
aws --profile seaweedfs \
    --endpoint-url http://localhost:8333 \
    s3 cp /etc/hostname s3://sovereign-blobs/test/hostname.txt

# List bucket contents
aws --profile seaweedfs \
    --endpoint-url http://localhost:8333 \
    s3 ls s3://sovereign-blobs/test/

# Download the file
aws --profile seaweedfs \
    --endpoint-url http://localhost:8333 \
    s3 cp s3://sovereign-blobs/test/hostname.txt /tmp/downloaded.txt

# Verify integrity
diff /etc/hostname /tmp/downloaded.txt && echo "Integrity verified"

Using rclone (for backups and sync)

# Configure rclone
cat >> ~/.config/rclone/rclone.conf << 'EOF'
[seaweedfs]
type = s3
provider = Other
endpoint = http://localhost:8333
access_key_id = DERAILS_ADMIN_ACCESS_KEY
secret_access_key = sovereign-storage-needs-sovereign-secrets-42chars
acl = private
EOF

# Sync a directory
rclone sync /var/backups seaweedfs:backups/ --progress

# List all buckets
rclone lsd seaweedfs:

# Check used space
rclone size seaweedfs:sovereign-blobs

Using Python (boto3)

import boto3

session = boto3.session.Session()
client = session.client(
    's3',
    endpoint_url='http://localhost:8333',
    aws_access_key_id='DERAILS_ADMIN_ACCESS_KEY',
    aws_secret_access_key='sovereign-storage-needs-sovereign-secrets-42chars',
    region_name='us-east-1'
)

# Create bucket
client.create_bucket(Bucket='governance-records')

# Upload
client.upload_file('/var/log/syslog', 'governance-records', 'logs/syslog.txt')

# List objects
response = client.list_objects_v2(Bucket='governance-records', Prefix='logs/')
for obj in response.get('Contents', []):
    print(f"  {obj['Key']} - {obj['Size']} bytes")

All standard S3 operations work: multipart uploads, versioning, presigned URLs, server-side encryption, conditional headers (If-Match, If-None-Match, If-Modified-Since), and bucket lifecycle policies. The SeaweedFS S3 gateway implements AWS Signature v4 authentication, so every S3 SDK works without modification.

Erasure Coding: The Space Efficiency Play

SeaweedFS implements Reed-Solomon 10+4 erasure coding. Here’s what that means in human terms:

Normal replication (3 copies):
  Data: 30 GB → Storage used: 90 GB (3x overhead)
  Can lose: 2 copies

Erasure Coding RS(10,4):
  Data: 30 GB → 10 data shards + 4 parity shards = 14 shards × 3 GB each
  Storage used: 42 GB (1.4x overhead)
  Can lose: 4 shards

Savings: 48 GB less storage for BETTER fault tolerance

To achieve the same 4-shard-loss tolerance with replication, you’d need 5 copies = 150 GB. Erasure coding does it in 42 GB. That’s 3.6x more space-efficient.

How It Works Under the Hood

Each 30 GB volume is split into 1 GB chunks. Every 10 data chunks are encoded into 4 parity chunks using Reed-Solomon coding. The 14 resulting EC shards are distributed across available volume servers.

Volume (30 GB) → Split into 1 GB blocks
 ┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┐
 │ D1 │ D2 │ D3 │ D4 │ D5 │ D6 │ D7 │ D8 │ D9 │D10│  ← Data blocks
 └────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
                     │
                     ▼ Reed-Solomon Encoding
 ┌────┬────┬────┬────┐
 │ P1 │ P2 │ P3 │ P4 │  ← Parity blocks
 └────┴────┴────┴────┘

14 shards total, distributed across servers/racks
Any 4 can fail → data fully recoverable

Since files are stored in specific 1 GB chunks within volumes, most small-file reads only touch a single shard. The O(1) seek architecture is preserved even with erasure coding enabled. Reads don’t need to reconstruct data from multiple shards unless a shard is actually missing.

Enable Erasure Coding

Erasure coding in SeaweedFS is applied to existing volumes — you convert hot-storage volumes to erasure-coded warm storage when they’re sealed (full).

# Connect to the weed shell
weed shell -master=localhost:9333

# List current volumes
> volume.list

# Encode a specific volume to EC (10+4)
> ec.encode -collection="" -volumeId=1

# Encode all volumes in a collection
> ec.encode -collection="backups"

# Verify EC shards
> ec.rebuild -force

# Check EC status
> volume.list

Topology-aware distribution: If you have 4+ servers, EC shards are distributed across servers — protecting against full server failure. If you have 4+ racks, they’re distributed across racks. SeaweedFS’s distribution algorithm is topology-aware by default.

When to Use Erasure Coding

# Decision tree from Ring -5
if data.access_pattern == :hot
  # High read/write frequency → use replication
  # Faster reads, simpler recovery
  replication = "001"  # 1 extra copy on same rack
elsif data.access_pattern == :warm
  # Infrequent access, lots of data → erasure coding
  # 1.4x overhead vs 3x for replication
  apply_ec = true
elsif data.access_pattern == :cold
  # Archival → EC + cloud tiering
  apply_ec = true
  tier_to_cloud = true  # S3 Glacier, B2, etc.
end

Monitoring and Health Checks

Prometheus Metrics

Every SeaweedFS component exposes Prometheus metrics on its configured -metricsPort:

# /etc/prometheus/prometheus.yml (relevant scrape configs)
scrape_configs:
  - job_name: 'seaweedfs-master'
    static_configs:
      - targets: ['localhost:9324']
    metrics_path: /metrics

  - job_name: 'seaweedfs-volume'
    static_configs:
      - targets: ['localhost:9325']
    metrics_path: /metrics

  - job_name: 'seaweedfs-filer'
    static_configs:
      - targets: ['localhost:9326']
    metrics_path: /metrics

  - job_name: 'seaweedfs-s3'
    static_configs:
      - targets: ['localhost:9327']
    metrics_path: /metrics

There’s a community Grafana dashboard (ID: 10423) that provides a pre-built visualization of SeaweedFS metrics. Import it into your Grafana instance and you’re monitoring sovereign storage.

Health Check Script

#!/usr/bin/env bash
# Run via cron every 5 minutes

set -euo pipefail

MASTER_URL="http://localhost:9333"
VOLUME_URL="http://localhost:8080"
FILER_URL="http://localhost:8888"
S3_URL="http://localhost:8333"
ALERT_WEBHOOK="${SEAWEEDFS_ALERT_WEBHOOK:-}"

check_endpoint() {
    local name="$1"
    local url="$2"
    local expected_code="${3:-200}"

    http_code=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 "$url" || echo "000")

    if [ "$http_code" != "$expected_code" ]; then
        echo "[FAIL] $name at $url returned $http_code (expected $expected_code)"
        return 1
    else
        echo "[OK]   $name at $url"
        return 0
    fi
}

failures=0

check_endpoint "Master"  "$MASTER_URL/cluster/status" || ((failures++))
check_endpoint "Volume"  "$VOLUME_URL/status"          || ((failures++))
check_endpoint "Filer"   "$FILER_URL/"                 || ((failures++))
check_endpoint "S3 API"  "$S3_URL/"                    || ((failures++))

# Check disk space on volume directory
disk_usage=$(df /data/seaweedfs/volumes --output=pcent | tail -1 | tr -d ' %')
if [ "$disk_usage" -gt 90 ]; then
    echo "[WARN] Disk usage at ${disk_usage}% on /data/seaweedfs/volumes"
    ((failures++))
else
    echo "[OK]   Disk usage at ${disk_usage}%"
fi

# Check volume count via master
volume_count=$(curl -s "$MASTER_URL/vol/status" | python3 -c "
import sys, json
data = json.load(sys.stdin)
total = sum(len(n.get('Volumes', [])) for dc in data.get('Topology', {}).get('DataCenters', []) for r in dc.get('Racks', []) for n in r.get('DataNodes', []))
print(total)
" 2>/dev/null || echo "0")
echo "[INFO] Active volumes: $volume_count"

if [ "$failures" -gt 0 ]; then
    echo ""
    echo "ALERT: $failures health check(s) failed"

    # Send webhook alert if configured
    if [ -n "$ALERT_WEBHOOK" ]; then
        curl -s -X POST "$ALERT_WEBHOOK" \
            -H "Content-Type: application/json" \
            -d "{\"text\": \"SeaweedFS: $failures health check(s) failed on $(hostname)\"}"
    fi

    exit 1
fi

echo ""
echo "All checks passed. Sovereign storage operational."
exit 0

# Make executable and add to cron
sudo chmod +x /usr/local/bin/seaweedfs-healthcheck.sh

# Run every 5 minutes
echo "*/5 * * * * root /usr/local/bin/seaweedfs-healthcheck.sh >> /var/log/seaweedfs/healthcheck.log 2>&1" | \
    sudo tee /etc/cron.d/seaweedfs-healthcheck

The weed shell (Admin Operations)

The weed shell is your admin console. Use it for cluster management, volume operations, and debugging.

# Start the admin shell
weed shell -master=localhost:9333

# Cluster status
> cluster.ps

# List all volumes with details
> volume.list

# Check volume balance across servers
> volume.balance -force=false

# Compact a volume (reclaim deleted space)
> volume.vacuum -garbageThreshold=0.3

# Fix volume replication issues
> volume.fix.replication

# Configure S3 credentials interactively
> s3.configure

# Check S3 bucket list
> s3.bucket.list

# Move volumes between servers
> volume.move -source=server1:8080 -target=server2:8080 -volumeId=42

Multi-Server Topology

For production with actual fault tolerance, here’s a 3-node deployment. Each node runs a master, volume server, and filer.

Node 1 (10.0.0.1):  master + volume + filer
Node 2 (10.0.0.2):  master + volume + filer
Node 3 (10.0.0.3):  master + volume + filer + s3

Master Cluster Configuration

On each node, start the master with peer awareness:

# Node 1
weed master -ip=10.0.0.1 -port=9333 -peers=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333

# Node 2
weed master -ip=10.0.0.2 -port=9333 -peers=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333

# Node 3
weed master -ip=10.0.0.3 -port=9333 -peers=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333

Masters use Raft consensus for leader election. One master is the leader; others are followers. If the leader dies, a follower takes over automatically. This is real governance — automated failover, not a phone tree.

Volume Servers with Replication

# Node 1
weed volume -ip=10.0.0.1 -port=8080 -mserver=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
    -dir=/data/seaweedfs/volumes -dataCenter=dc1 -rack=rack1

# Node 2
weed volume -ip=10.0.0.2 -port=8080 -mserver=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
    -dir=/data/seaweedfs/volumes -dataCenter=dc1 -rack=rack2

# Node 3
weed volume -ip=10.0.0.3 -port=8080 -mserver=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
    -dir=/data/seaweedfs/volumes -dataCenter=dc1 -rack=rack3

With -defaultReplication=010 (one copy on a different rack), every blob exists on two nodes. Lose an entire server? Zero data loss. The remaining volume server with the replica continues serving reads immediately.

Filer with PostgreSQL Backend

For multi-filer deployments, use PostgreSQL so all filers share metadata:

# Create the database
sudo -u postgres createuser seaweedfs
sudo -u postgres createdb -O seaweedfs seaweedfs_filer

# Each filer node uses the same filer.toml:

# /etc/seaweedfs/filer.toml (on all 3 nodes)
[postgres2]
enabled = true
createTable = """
  CREATE TABLE IF NOT EXISTS "%s" (
    dirhash   BIGINT,
    name      VARCHAR(65535),
    directory VARCHAR(65535),
    meta      bytea,
    PRIMARY KEY (dirhash, name)
  )
"""
hostname = "10.0.0.1"
port = 5432
username = "seaweedfs"
password = "your_secure_password_here"
database = "seaweedfs_filer"
sslmode = "require"
connection_max_idle = 100
connection_max_open = 100

Cost Analysis: SeaweedFS vs. AWS S3

Here’s the math that keeps AWS executives uncomfortable.

AWS S3 Standard Pricing (2025)

Tier	Price per GB/month
First 50 TB	$0.023
50-500 TB	$0.022
500+ TB	$0.021

Plus: GET requests ($0.0004 per 1,000), PUT requests ($0.005 per 1,000), data transfer out ($0.09/GB first 10 TB).

Sovereign SeaweedFS on Hetzner

Component	Spec	Monthly Cost
Hetzner AX42 (dedicated)	Ryzen 5 3600, 64GB RAM, 2× 512GB NVMe	€49.00
Additional HDD (2× 16TB)	Via Storage Box or AX addon	€15.80
Total storage: ~32 TB	—	€64.80/month

Cost per GB: €0.002/month ($0.0022/month).

That’s 10.5x cheaper than AWS S3 Standard. For 10 TB of storage:

Provider	Monthly Cost	Annual Cost
AWS S3	$230.00	$2,760.00
Hetzner + SeaweedFS	~$22.00	~$264.00

And that’s before AWS charges you $0.09/GB for egress. SeaweedFS egress? Hetzner includes 20 TB/month of traffic. After that, €1.19/TB.

$ python3 -c "
aws_10tb = 10000 * 0.023  # Storage only
hetzner = 64.80           # Server + storage
savings = (aws_10tb - hetzner) / aws_10tb * 100
print(f'AWS S3 (10TB storage only): \${aws_10tb:.2f}/month')
print(f'Hetzner + SeaweedFS (32TB capacity): €{hetzner:.2f}/month')
print(f'Savings: {savings:.1f}%')
"
AWS S3 (10TB storage only): $230.00/month
Hetzner + SeaweedFS (32TB capacity): €64.80/month
Savings: 71.8%

For a single small VPS (our Hetzner CPX11 at €4.49/month with 40 GB SSD), SeaweedFS handles the blog assets, Git LFS objects, and Matrix media for Derails. Total S3-equivalent cost on AWS for 15 GB of storage + moderate traffic? About $3.85/month in storage + $5-15 in transfer. We pay €4.49 for the entire server, which also runs nginx, Gitea, and three other services.

The Real Calculation Your CFO Ignores

class StorageCostAnalysis
  def annual_cost(provider, storage_tb)
    case provider
    when :aws_s3
      storage = storage_tb * 1000 * 0.023 * 12  # Per GB pricing
      requests = 50_000_000 * 0.0004 / 1000 * 12 # 50M GETs/month
      egress = storage_tb * 100 * 0.09 * 12       # 100GB egress per TB
      storage + requests + egress
    when :seaweedfs_hetzner
      64.80 * 12  # Flat server cost, 32TB capacity
    end
  end

  def break_even_months
    # SeaweedFS setup time: 2 hours
    # Engineer hourly rate: €100
    # Setup cost: €200
    # Monthly savings: $230 - $22 = $208
    # Break-even: 200 / 208 = 0.96 months
    0.96
  end
end

# Break-even in less than a month.
# Your timeline pays AWS for years because "nobody gets fired for choosing AWS."
# In Timeline Ω-7, that's literally a fireable offense.

Backup Strategy

Sovereign storage needs sovereign backups. Here’s the strategy:

Volume Backup with rclone

#!/usr/bin/env bash
# Backup SeaweedFS to a remote S3-compatible store (Backblaze B2, Wasabi, etc.)

set -euo pipefail

BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_DEST="b2:seaweedfs-backups/${BACKUP_DATE}"

# Backup filer metadata
echo "Backing up filer metadata..."
weed shell -master=localhost:9333 <<'SHELL'
filer.meta.save -o /tmp/filer-meta-backup
SHELL

rclone copy /tmp/filer-meta-backup "${BACKUP_DEST}/filer-meta/"
rm -f /tmp/filer-meta-backup

# Sync all bucket data
echo "Syncing S3 bucket data..."
rclone sync seaweedfs: "${BACKUP_DEST}/data/" \
    --progress \
    --transfers=8 \
    --checkers=16 \
    --fast-list

echo "Backup complete: ${BACKUP_DEST}"

Filer Metadata Backup (Async Replication)

SeaweedFS supports async filer metadata backup to another filer or cloud storage:

# Start async backup from primary filer to backup filer
weed filer.sync \
    -a=localhost:8888 \
    -a.path=/buckets \
    -b=backup-server:8888 \
    -b.path=/buckets

Operational Runbook

Volume Compaction (Reclaim Deleted Space)

When you delete files, SeaweedFS marks needles as deleted but doesn’t reclaim space immediately. Vacuum to reclaim:

# Via weed shell
weed shell -master=localhost:9333
> volume.vacuum -garbageThreshold=0.3

# Or via HTTP API
curl "http://localhost:9333/vol/vacuum?garbageThreshold=0.3"

The -garbageThreshold=0.3 means: compact volumes where 30%+ of space is wasted by deletions. Set lower for more aggressive reclamation, higher to reduce compaction I/O.

Adding a New Volume Server

# On the new server, install SeaweedFS and start a volume server
weed volume \
    -ip=10.0.0.4 \
    -port=8080 \
    -mserver=10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
    -dir=/data/seaweedfs/volumes \
    -dataCenter=dc1 \
    -rack=rack4

# The master automatically discovers and starts assigning volumes
# Check in weed shell:
> volume.list
# New server appears with volumes being allocated

No downtime. No cluster reconfiguration. No CRUSH map rebuilds. Just start the process and the master integrates it. This is how governance should work — new participants join without disrupting existing operations.

Handling Disk Failure

# Check which volumes were on the failed disk
weed shell -master=localhost:9333
> volume.list

# If using replication, volumes are automatically served from replicas
# If using EC, data is reconstructed from remaining shards:
> ec.rebuild -force

# Replace the disk, start volume server, rebalance:
> volume.balance -force

Security Hardening

Firewall Rules

# Only expose S3 API through nginx (port 443)
# Keep internal ports locked down

# Using ufw
sudo ufw default deny incoming
sudo ufw allow ssh
sudo ufw allow 80/tcp    # HTTP (redirect to HTTPS)
sudo ufw allow 443/tcp   # HTTPS (nginx → S3 API)

# Block direct access to SeaweedFS ports from outside
# Ports 9333, 8080, 8888, 8333 should only be accessible from localhost
# or within your private network

sudo ufw enable

S3 Bucket Policies

Use the S3 credentials file to enforce least-privilege access:

{
  "identities": [
    {
      "name": "backup_agent",
      "credentials": [
        {
          "accessKey": "BACKUP_AGENT_KEY",
          "secretKey": "backup-agent-needs-write-to-one-bucket-only"
        }
      ],
      "actions": [
        "Read:backups",
        "Write:backups",
        "List:backups"
      ]
    }
  ]
}

Actions can be scoped to specific buckets using the Action:bucket syntax. The backup_agent identity above can only read, write, and list the backups bucket. Everything else is denied.

Encryption at Rest

SeaweedFS supports server-side encryption compatible with AWS S3 SSE modes:

# Upload with server-side encryption
aws --profile seaweedfs \
    --endpoint-url http://localhost:8333 \
    s3 cp sensitive-file.tar.gz s3://encrypted-bucket/ \
    --sse AES256

For full-disk encryption, use LUKS on the volume data partition:

# Encrypt the storage partition (one-time setup)
sudo cryptsetup luksFormat /dev/sdb1
sudo cryptsetup open /dev/sdb1 seaweedfs_crypt
sudo mkfs.ext4 /dev/mapper/seaweedfs_crypt
sudo mount /dev/mapper/seaweedfs_crypt /data/seaweedfs/volumes

Cloud Tiering (The Hybrid Strategy)

SeaweedFS supports cloud tiering — automatically moving cold data to cheaper cloud storage while keeping hot data on local disk:

# Configure cloud tiering to Backblaze B2
weed shell -master=localhost:9333
> volume.tier.upload -collection="" -dest=s3://b2-bucket -newer=30d

This uploads volumes older than 30 days to B2 while keeping recent data on fast local storage. Reads to tiered data are transparently proxied through SeaweedFS — clients don’t know (or care) where the data physically lives.

The economics: Backblaze B2 charges $0.006/GB/month for storage and $0.01/GB for egress. For archival data you rarely access, this is 74% cheaper than even self-hosted storage (because you’re not paying for hardware depreciation on cold bits).

The Ring -5 Observation

From Ring -5, I observe Timeline Ω-12 paying $0.023/GB/month to store objects in AWS S3. That’s $23 per TB per month. Per year, $276 per TB.

On sovereign infrastructure with SeaweedFS: approximately $2/TB/month. Per year, $24 per TB.

For 100 TB — a modest deployment by any timeline’s standards:

	AWS S3	SeaweedFS (Hetzner)
Monthly	$2,300	~$200
Annual	$27,600	~$2,400
5-Year	$138,000	~$12,000

The 5-year delta is $126,000. That’s two senior engineers’ annual salary in most of your timeline’s markets. You’re paying AWS the equivalent of two engineers to avoid spending 2 hours setting up SeaweedFS.

In Timeline Ω-7, the infrastructure budget oversight committee (yes, every project has one) would flag this as a governance violation under Section 7.3: “Negligent Resource Allocation.” The penalty? Your Git commit rights are suspended until you submit a cost-benefit analysis with 80%+ test coverage.

Your timeline doesn’t even have infrastructure budget oversight committees. Still investigating why.

$ git log --oneline timeline-omega-12/storage-decisions
a1b2c3d AWS S3 because "nobody gets fired for choosing AWS"
d4e5f6g Ignored cost analysis (no tests)
h7i8j9k Renewed 3-year S3 commitment without benchmarking alternatives
l0m1n2o CFO asked about costs, got told "it's cloud, it's scalable"
p3q4r5s $138,000 later, still no egress budget
# dangling commits: all of them

“Every dollar you send to AWS for object storage is a dollar that doesn’t fund your own infrastructure sovereignty. In Timeline Ω-7, we tax cloud waste at 94.2%. The revenue funds public Git repositories for all elected officials. Your timeline taxes income instead. Still investigating why.”

— Kim Jong Rails, from kimjongrails.com/blog

Further Reading:

SeaweedFS: The Sovereign Storage Deep-Dive

#Previously, on Storage Wars

#Why SeaweedFS (The 30-Second Refresher)

#The Architecture (Facebook’s Needle in a Haystack)

#The Three Components

#How O(1) Disk Seeks Work

#Production Deployment: systemd on Bare Metal

#Prerequisites

#Directory Structure

#Master Server (systemd)

#Volume Server (systemd)

#Filer Server (systemd)

#Filer Metadata Backend Configuration

#S3 API Gateway (systemd)

#S3 Credentials Configuration

#Start Everything

#The Docker Compose Alternative

#Nginx Reverse Proxy for S3 API

#S3 API Usage (Verify It Works)

#Using AWS CLI

#Using rclone (for backups and sync)

#Using Python (boto3)

#Erasure Coding: The Space Efficiency Play

#How It Works Under the Hood

#Enable Erasure Coding

#When to Use Erasure Coding

#Monitoring and Health Checks

#Prometheus Metrics

#Health Check Script

#The weed shell (Admin Operations)

#Multi-Server Topology

#Master Cluster Configuration

#Volume Servers with Replication

#Filer with PostgreSQL Backend

#Cost Analysis: SeaweedFS vs. AWS S3

#AWS S3 Standard Pricing (2025)

#Sovereign SeaweedFS on Hetzner

#The Real Calculation Your CFO Ignores

#Backup Strategy

#Volume Backup with rclone

#Filer Metadata Backup (Async Replication)

#Operational Runbook

#Volume Compaction (Reclaim Deleted Space)

#Adding a New Volume Server

#Handling Disk Failure

#Security Hardening

#Firewall Rules

#S3 Bucket Policies

#Encryption at Rest

#Cloud Tiering (The Hybrid Strategy)

#The Ring -5 Observation