Open engineering handbook v1.0 — 2026
Text + code + sources
CC BY-SA 4.0
Decentralized Data node.
A field manual for IPFS and data-availability systems, where bytes are addressed by what they are — and availability is proven, not promised. Architecture, request paths, sampling math and the repositories that make it real.
01 / Thesis
The address
is the proof.
HTTP tells you where to look. Content addressing tells you what to look for. One trusts a host; the other trusts a hash. The difference changes how data survives.
Every byte gets a fingerprint. Ask any peer for that fingerprint, then verify the answer locally. No origin server has to survive the night.
An IPFS node is not a server with extra steps. It is a peer: it stores blocks, announces what it can provide, exchanges blocks with other peers, and checks every byte it receives. The network can route around a dead host because the content — not the host — is the destination.
A data availability layer adds a second question. Not only “can I fetch it?” but “can anyone reasonably verify that the data was published and remains retrievable?” Sampling, erasure coding and attestations turn that question into a measurable protocol.
Location is a hint.
Content is the truth. Operating principle / DDN-001
02 / Anatomy
One node.
Six organs.
A useful data node is a stack of narrow responsibilities. Pick a layer to inspect its job, its failure mode and the open repositories that implement it.
Interface layer / CID → experience
Access
Gateways, client libraries and HTTP APIs turn content identifiers into experiences. They are edges: useful, exposed and replaceable. The core should not depend on them.
- Verify every block at the edge, even behind a gateway.
- Rate-limit, authenticate and cache at the public boundary.
- Keep the node’s admin API off the open internet.
A public gateway without limits is an unpaid CDN with a database attached.
03 / Lifecycle
From intent
to verified bytes.
A retrieval is a small distributed systems story. Trace the path from a root CID to a checked block, then follow the block back out to the network.
Stage 01 / write path
Chunk. Hash. Link.
Files are split into content-defined or fixed-size blocks, linked into a Merkle DAG and named by the root CID. Then the node announces itself as a provider.
- Deduplication happens naturally: identical blocks share one address.
- Pin the root if you intend to keep the DAG reachable.
- Publishing is not the same as replicating.
# add a file and capture its root CID
$ CID=$(ipfs add -Q --cid-version=1 note.txt)
$ echo "$CID"
bafybeih...
# keep the blocks locally and announce them
$ ipfs pin add "$CID"Stage 02 / discovery
Ask the network who has it.
Provider records map a CID to peer identities. A DHT walks toward the key; delegated routing can answer over HTTPS for browsers, mobile clients and other constrained peers.
- Discovery returns candidates, not guarantees.
- mDNS is enough for a LAN; it is not a global index.
- Reproviding keeps your provider record fresh.
# who is advertising provider records?
$ ipfs routing findprovs "$CID"
12D3KooW...
# inspect connected peers and addresses
$ ipfs swarm peers | headStage 03 / exchange
Trade blocks, not files.
Bitswap sends want-have and want-block messages. Peers respond with blocks, and multiple providers can serve in parallel. HTTP gateways cover browsers, cold clients and interoperability.
- Ask for the missing block, not the whole object.
- Parallelism improves latency until the link saturates.
- Gateways are a bridge, not a substitute for the network.
# fetch a single block
$ ipfs block get "$CID" > root.block
# walk the DAG without downloading file contents
$ ipfs dag stat "$CID"Stage 04 / integrity
Trust the math, not the peer.
Every block is hashed and checked against its CID before it enters the local store. Bad blocks are discarded; the DAG is assembled only from verified links.
- Verification is local, cheap and independent of reputation.
- A valid hash proves integrity, not truth or safety.
- Encrypt before publishing when confidentiality matters.
// every block must hash to its multihash
const digest = sha256(block)
if (!cid.multihash.equals(digest)) {
discard(block)
}
// only verified links enter the local DAGStage 05 / availability
Reprovide. Measure. Repeat.
Pinned content is announced periodically. Caches stay warm; provider records expire. Metrics tell you whether peers can actually find and fetch you, not merely whether the daemon is up.
- Peer count is vanity; successful retrievals are signal.
- Watch disk headroom, not only disk usage.
- Run a retrieval smoke test from a fresh identity.
# check what you are actually providing
$ ipfs pin ls --type=recursive | head
# watch traffic and daemon health
$ ipfs stats bw
$ ipfs diag sys04 / Availability
Don’t check everything.
Sample enough.
Data availability sampling turns “trust me, it’s there” into a probability statement. Ask for a few random chunks; if any are missing, reject the block.
A full node can download everything. A light client cannot. DAS closes that gap by sampling a small, unpredictable subset of committed data. With enough independent samples, the chance of missing a mostly-unavailable block becomes vanishingly small.
- 01Commit
The producer commits to the block and publishes the commitments needed to check it.
- 02Sample
Clients request random chunks. A missing chunk is evidence of unavailability.
- 03Reconstruct
Erasure coding means enough available chunks can rebuild the whole block.
- 04Attest
Depending on the design, nodes aggregate availability signals without downloading the full payload.
Pmiss = (1 − u)k
u is the unavailable share of chunks; k is the number of independent samples. Try the model in the lab. Real protocols add erasure coding, network assumptions and attestation rules.
Adjust the number of samples and the unavailable share. The model assumes independent binary samples; it is a teaching tool, not a protocol specification.
One trial: 7 of 32 sampled chunks unavailable.
Missing chunks are orange; sampled chunks are lime. At u = 20% and k = 32, the theoretical miss rate is 0.079%. Increase k to drive it down. Read the real spec ↗
05 / Reference stack
Assemble
your own.
Eight repositories, one composable stack. These are not endorsements of a single topology; they are the primitives worth reading before you ship one.
Kubo
The long-running IPFS implementation: CLI, HTTP RPC, gateway, Bitswap, and enough knobs to keep an operator honest.
Helia
A modular, browser-friendly IPFS runtime. Bring your own storage, networking and trust boundaries.
Boxo
Reusable IPFS building blocks: blockservice, exchange, gateway, pinning and more.
go-libp2p
The modular peer-to-peer stack underneath: identity, transports, streams, NAT traversal and pubsub.
IPFS Cluster
Coordinate pinning and replication across nodes. Treat it as an operations layer, not a magic backup.
go-ipld-prime
Typed, schema-aware links between content-addressed data. The graph beneath the files.
Celestia Node
Light, full and bridge nodes for a modular data availability network, including sampling and share retrieval.
Consensus specs
The executable reference for Ethereum’s consensus layer, including danksharding and data availability sampling.
06 / Operate
Ship it, then
observe it.
A node that starts is not a node that works. The following examples are a starting point — the upstream documentation is the canonical source.
Keep the first deployment boring. One durable identity, one explicit storage budget, one observable gateway and one retrieval test you can run from a clean machine.
- 01Persist the repository and peer identity. Losing them changes who you are on the network.
- 02Set a storage cap and a garbage-collection policy before the disk decides for you.
- 03Measure successful retrievals from outside your network, not just local process uptime.
Security note: expose the gateway deliberately, never the admin RPC. Read the Kubo docs ↗ before opening ports.
# 1. initialise a local repository
$ ipfs init
# 2. join the network
$ ipfs daemon --enable-gc
# 3. publish, pin, inspect
$ CID=$(printf 'hello, persistence\n' | ipfs add -Q)
$ printf '%s\n' "$CID"
$ ipfs pin ls --type=recursive | head
$ ipfs routing findprovs "$CID"import { createHelia } from 'helia'
import { unixfs } from '@helia/unixfs'
const node = await createHelia()
const fs = unixfs(node)
const cid = await fs.addBytes(
new TextEncoder().encode('hello, persistence')
)
// the returned bytes are verified against the CID
for await (const chunk of fs.cat(cid)) {
process.stdout.write(chunk)
}# initialise the local node store
$ celestia light init
# start with a consensus gRPC endpoint
$ celestia light start \
--core.ip <CORE_IP> \
--core.port 9090 --core.tls \
--p2p.network celestia
# inspect sync and sampling state
$ celestia light status
# read the operator guide for flags and networks
# docs.celestia.org# bandwidth snapshot over the RPC API
$ curl -s localhost:5001/api/v0/stats/bw
# raw block via a local gateway, then hash it
$ curl -s "http://127.0.0.1:8080/ipfs/$CID?format=raw" \
| sha256sum
# inspect DAG shape and cumulative size
$ ipfs dag stat "$CID"These snippets are intentionally minimal. Pin versions, read the release notes and test retrieval from a separate network before trusting a deployment. Start with Kubo ↗
07 / Field notes
The details
that bite.
Six short corrections to common assumptions. Each one changes a design decision, an SLO, or a late-night incident.
01 A CID describes bytes. It is not a backup.
A CID is a cryptographic address. It proves that a block matches the expected digest; it does not prove that anyone will keep the block. Persistence needs replication, incentives or operational commitment.
02 Pinning is a local promise, not a global guarantee.
Pinning prevents a local garbage collector from removing blocks. It does not force other peers to retain them, and it does not replace a second provider. If the only pinned copy is on a disk that fails, the CID survives while the content does not.
03 Data availability is probabilistic, not a durability certificate.
Sampling gives a client high confidence that a block was retrievable at the sampled moment, under explicit assumptions. It is not a guarantee of permanent storage, and it is not the same as validity. Keep the trust model in the README.
04 Gateways are edges. Treat them like untrusted caches.
An HTTP gateway makes content-addressed data legible to ordinary clients, but a gateway operator can log, rate-limit, block or modify the response path. Use trustless verification where the client can afford it, and never expose admin APIs to the public edge.
05 Encrypt before you address.
Content addressing reveals equality. If two peers add the same plaintext, they get the same CID. Encrypt first when confidentiality matters, then add the ciphertext, and manage keys as a separate system with its own lifecycle.
06 Measure retrieval tails, not process uptime.
“The daemon is up” is not an availability signal. Track time-to-first-block, p95/p99 retrieval latency, reprovide health, peer churn and disk headroom. A fresh peer should be able to fetch your root CID from an independent provider within a defined budget.
08 / Runbook
The boring parts
keep data alive.
A production readiness pass for a node you actually intend to keep. Tick the boxes as a design review — or hand the list to the next operator.
Saved in this browser only. No account, no telemetry.
- Per deploy
- Prove retrieval. Run a smoke test from a fresh peer, verify the peer identity and check gateway auth before declaring success.
- Daily
- Watch the edges. Disk headroom, peer count, 5xx rate, bandwidth saturation and failed retrieval alerts.
- Weekly
- Audit the promise. Reprovide health, content inventory, dependency updates and a random restore test.
- Incident
- Protect the identity. Isolate the peer, rotate exposed credentials, preserve logs and re-verify CIDs before resuming service.
Can a fresh peer retrieve your root CID within X seconds, from Y independent providers — right now?
09 / Sources
Read the source.
Then run it.
This handbook is an entry point, not an authority. Every claim points toward code, a specification or a reproducible command. Start with the repository that matches your layer.
01 Specifications
IPFS SpecsCIDs · routing · exchange · gateways libp2p SpecsTransports · streams · discovery · NAT IPLDContent-addressed data model and links MultiformatsSelf-describing values for the network Consensus SpecsDanksharding · DAS · consensus rules02 Implementations
KuboGo IPFS daemon HeliaTypeScript IPFS runtime Celestia NodeModular DA light and full nodes LotusFilecoin network implementation03 Operations
IPFS ClusterPinning and replication orchestration BoxoGo building blocks for IPFS systems go-libp2pPeer-to-peer networking stack go-ipld-primeTyped IPLD interfaces for GoFound an error? Open an issue in the upstream repository, not in a private inbox. The network improves when the corrections are legible.
Join the community