Storage engine

ClatterDB — distributed time-series database engine

A storage engine written from the bottom up in Go: LSM-tree, write-ahead log, MVCC snapshot isolation, and a hand-rolled query engine.

Role

Sole engineer

When

2026

Outcome

Full engine — storage, transactions, and query planning

Stack

GoLSM-treeWrite-ahead logMVCCQuery plannerSharding

Problem: Time-series workloads are write-heavy and append-mostly, which punishes B-tree storage and rewards a design built around sequential writes and background compaction.
Move: Built the engine in layers — WAL for durability, LSM-tree for write throughput, MVCC for snapshot isolation — then a query engine (parser, planner, executor) and time-based sharding on top.
Result: A working database rather than a storage demo: durable writes, isolated concurrent readers, and its own SQL-style query path end to end.

lsm write path

Write-ahead log

durability

Memtable

sorted, in memory

empty

L0 SSTables

immutable

none

one sorted run

none

01 / 09

$ idle · memtable empty

in memory
on disk, immutable
empty

Time-series writes are almost all appends, which is what makes an LSM-tree the right shape: nothing is ever updated in place, so every disk write is sequential and compaction cleans up later, off the write path.focus + ← → to step

CONTEXT

Most 'build a database' projects stop at the storage layer, where the interesting algorithms are well documented. The parts that make a database actually usable — transaction isolation and a query planner — are where the design decisions get uncomfortable.

Time-series data narrows the problem usefully. Writes are overwhelmingly appends at the current timestamp, reads are usually range scans over a window, and old data is rarely mutated. That shape justifies an LSM-tree and makes time-based sharding the natural partitioning key.

WHAT I DID

Implemented an LSM-tree storage engine with background compaction, sized for append-heavy write patterns.
Added a write-ahead log so acknowledged writes survive process death.
Built MVCC snapshot isolation, letting readers see a consistent version without blocking concurrent writers.
Wrote a custom query engine end to end — parser, planner, and executor.
Partitioned data with time-based sharding to keep range scans local to the relevant shards.

RESULT

The engine covers the full path from parsed query to durable, isolated storage — the layers that usually get stubbed out in a from-scratch database project.