These are some informal notes on setting up a full-network atproto Relay, using the bigsky relay software developed by Bluesky. This is the same software we run ourselves at https://bsky.network. The focus here is on the compute resources necessary to replicate the type of full-network, full-featured service that Bluesky currently operates with the size of the network that exists today.
The demo Relay described here is running at relay-ovh.demo.bsky.dev. It handles crawling and re-publishing the full network firehose, with headroom for traffic spikes and growth on a number of dimensions: new accounts (repos), new PDS instances, more content being created (firehose event rate), number of services consuming from the firehose, etc.
Changes
Running through this demo setup turned up some sharp edges and missing configuration knobs in bigsky. For example, trying to backfill the full network with default configuration was resulting in OOM errors with an instance this size. Tweaks and configuration have been merged to the main branch of bigsky, along with additions to the README.
Network Scaling
How big can this instance scale? Hard to tell exactly, my guess is that it could do an order of magnitude more event rate, but will run out of disk before too long (eg, in the next year).
There are a number of possibilities for improving Relay efficiency to make this kind of service cheaper: There are implementation details (like using alternative database engines, or not storing repo data as millions of small files on disk). Data and work could be sharded across multiple machines. Not every Relay needs to crawl and mirror the entire network, and Relays could potentially be simplified to not maintain a full mirror of network content. On the other hand, actually running critical services would have a number of needs not covered here: legal and administrative burdens, monitoring and alerting, etc.
Shopping for an Server
My assumption is that the main thing needed is a relatively large and reasonably fast disk. The Bluesky production Relays currently use about 1 TByte for PostgreSQL and 1 TByte for CAR storage on local disk. The CAR storage filesystem should also be XFS (not ext4), to handle many millions of small files. When shopping for instances I looked for around 2 TByte for PostgreSQL and 2 TByte for CAR storage so that this setup would be realistic even with growth of the network over time. This storage could all be one disk/filesystem or two separate disks/filesystems. Regular SSD would probably work fine, NVMe is nice, especially for backfill.
More RAM always helps (page cache and other caches). Don’t need much CPU; the relay process is highly concurrent, but mostly I/O bound. Do want decent network monthly quota.
Disk is definitely the hard part. Network block storage (eg, AWS EBS) is pretty expensive even from cheaper providers, and usually costs more monthly than an entire bare metal instance with larger disks. Bare metal instances are mostly spinning disk or NVMe, not SSD; I assume that spinning disk isn't realistic for a fast backfill demo.
I ended up selecting an OVH instance with for about $150/month:
ADVANCE-2-LE: https://www.ovhcloud.com/en/bare-metal/advance/adv-2/- 12 vCPU (Intel Xeon-E 2136 - 6c/12t - 3.3 GHz/4.5 GHz)
- 32 GB RAM (32 GB ECC 2666 MHz)
- disks: 2×1.92 TB NVMe
- 1Gbit/s unmetered and guaranteed
- $152/month plus one-time $92 setup fee (no commitment)
That exact config isn’t available now (a week later), but a very similar one is:
ADVANCE-1: https://www.ovhcloud.com/en/bare-metal/advance/adv-1/- 12 vCPU (AMD EPYC 4244P - 6c/12t - 3.8GHz/5.1GHz)
- 32 GB RAM (32GB DDR5 ECC 5200MHz)
- rootfs disk: 2x NVMe 960GB (RAID)
- data disks: 2x 1.92TB NVMe
- 1Gbit/s unmetered and guaranteed
- $153/month plus one-time $93 setup (no commitment)
In both cases the setup fee is waved with a 6 month commitment, and there are discounts on the monthly rate with longer commitments.
Host Provisioning
Using the OVH web interface, provisioned the server with Ubuntu 24.04. With the ADVANCE-2-LE host, I specified partitioning to not use RAID. I let the setup wizard use one of the two disks for rootfs, boot, and swap. With all defaults this resulted in ext4. The second disk was not partitioned or configured using the wizard (I got to that later on the server itself).
Configured a DNS A record to point at the IPv4 that OVH gave us.
Logged in to the server and ran commands similar to this:
hostnamectl hostname relay-example.demo.bsky.dev
apt update
apt upgrade
apt install ripgrep fd-find dstat htop iotop iftop pg-activity httpie caddy golang postgresql yarnpkg
# set up yarn command; could also have used nvm
ln -s /usr/bin/yarnpkg /usr/bin/yarn
# punch holes in default firewall for HTTP/S
ufw allow 80/tcp
ufw allow 443/tcp
Ran through partitioning of the second NVMe with XFS. Note that on a real machine you'd want to set up fstab so this mounts automatically on a reboot.
# create a partition
sudo fdisk /dev/nvme1n1
# c (create), default (primary), default (1), default (start sector), default (entire disk), w (write)
# create XFS filesystem on that partition
sudo mkfs.xfs /dev/nvme1n1p1
# mount that filesystem to /data
sudo mkdir -p /data
sudo mount /dev/nvme1n1p1 /data
Pull the indigo codebase and build; ran this as the ubuntu user not root:
# depending on user that will be running the service
mkdir -p /data/bigsky
sudo chown ubuntu:ubuntu /data/bigsky/
# pull source code and build. if you had patches or a working branch, would modify here
cd
git clone https://github.com/bluesky-social/indigo
cd indigo
make build-relay-ui build
Configure PostgreSQL (sudo -u postgres psql); replace CHANGME with a secure password of your choice:
CREATE DATABASE bgs;
CREATE DATABASE carstore;
CREATE USER bigsky WITH PASSWORD 'CHANGEME';
GRANT ALL PRIVILEGES ON DATABASE bgs TO bigsky;
GRANT ALL PRIVILEGES ON DATABASE carstore TO bigsky;
# these are needed for newer versions of postgres
\c bgs postgres
GRANT ALL ON SCHEMA public TO bigsky;
\c carstore postgres
GRANT ALL ON SCHEMA public TO bigsky;
Create a config file at ~/indigo/.env:
ENVIRONMENT=production
DATABASE_URL="postgres://bigsky:CHANGEME@localhost:5432/bgs"
CARSTORE_DATABASE_URL="postgres://bigsky:CHANGEME@localhost:5432/carstore"
DATA_DIR=/data/bigsky
GOLOG_LOG_LEVEL=info
# or whatever DNS you want to use for handle resolution
RESOLVE_ADDRESS="8.8.8.8:53"
FORCE_DNS_UDP=true
BGS_COMPACT_INTERVAL=0
RELAY_DEFAULT_REPO_LIMIT=500000
# these were somewhat tuned to this instance size
MAX_CARSTORE_CONNECTIONS=12
MAX_METADB_CONNECTIONS=12
MAX_FETCH_CONCURRENCY=25
RELAY_CONCURRENCY_PER_PDS=20
RELAY_MAX_QUEUE_PER_PDS=200
#RELAY_ADMIN_KEY=CHANGEME
With the RELAY_ADMIN_KEY set to a strong random value, and DATABASE_URL substituted to the earlier database password. You can create one with:
openssl rand -base64 30
Create a system-wide Caddy config at /etc/caddy/Caddyfile. Substitute in your hostname, and comment out any other lines in the file:
relay-example.demo.bsky.dev {
reverse_proxy 127.0.0.1:2470
}
Restart caddy: sudo systemctl restart caddy
Running bigsky and Backfilling
Run the actual service! For example, in a screen session, or a service management tool of your choice:
cd ~/indigo
./bigsky --api-listen 127.0.0.1:2470
Confirm that everything is working by connecting using the gosky command from a laptop (which is in the indigo repo). Won’t get events (because Relay hasn't subscribed to anything yet), but should connect successfully:
gosky readStream wss://relay-example.demo.bsky.dev
You can also connect to the web management interface at https://relay-example.demo.bsky.dev/dash. This lets you view basic stats per PDS, modify limits, add new PDS instances to crawl, takedown individual repos (by DID), block PDS instances by domain suffix, etc.
To start backfills from a laptop, create a hosts.txt file with PDS hostnames, then run initial crawl command. Can do this from a laptop:
cd ~/indigo/cmd/bigsky
export RELAY_ADMIN_KEY=CHANGEMESECRET
export RELAY_HOST=relay-example.demo.bsky.dev
cat hosts.txt | parallel -j1 ./crawl_pds.sh {}
Let that bake for a few hours or overnight. Only accounts with new commits will get backfilled. A 24 hour period is usually around 10% of the network.
Then can start explicit backfills per-PDS ("resync"). This will pull a complete list of DIDs hosted on the PDS (or at least, which the PDS thinks are still hosted on the PDS, this might not yet handle migrations). Don’t want to do full PDS backfills for all the big PDS instances at once, or the relay will get overwhelmed (eg, OOM). Instead, do 4-8 at a time, modifying hosts.txt or the head command as needed:
head -n 4 hosts.txt | parallel -j1 ./sync_pds.sh {}
# check progress
head -n 4 hosts.txt | parallel -j1 ./sync_status_pds.sh {}
Smaller self-hosted instances can be backfilled in big batches (eg, hundreds of backfills at the same time).
While running backfill, some new PDS instances will be discovered, even if not crawled specifically, and even with “spidering” disabled. My guess is that accounts which migrate away from our PDS instances are still listed by the original PDS. When bigsky does a backfill, it resolves all the DIDs, and sees a different PDS in the atproto service entry, and adds it to the PDS list.
The entire backfill took a couple days of casual checking in and poking it along.
How does one get a complete list of PDS instances in the network? It would be helpful if Relays had a public endpoint to scroll through all known PDS instances, and indicate if they are active, blocked/suspended, and roughly how many repos there are. For now, you can pull hostnames from public listings like https://blue.mackuba.eu/directory/ and https://bsky-debug.app/. Or scrape the complete DID PLC directory (which is public and enumerable), and extract all PDS service endpoints.
Rough Performance Stats
Here are some quick/informal system performance snapshots. The backfill period (fetching all previous repo content from the network) is far more resource intensive than steady operation.
I didn't run any compactions manually, and disabled automatic/periodic compactions. These are resource intensive to process, but free up disk and database space. Compactions are a feature specific to bigsky and it's data storage system.
During early phase of backfill:
# dstat
----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read writ| recv send| in out | int csw
32 16 44 5 0| 17M 1410M| 20M 879k| 0 0 | 107k 173k
52 13 29 4 0| 13M 1495M| 18M 765k| 0 0 | 134k 117k
74 13 13 1 0| 20M 623M| 18M 722k| 0 0 | 47k 62k
26 14 52 6 0| 12M 1610M| 14M 613k| 0 0 | 133k 120k
51 16 30 2 0| 19M 928M| 18M 813k| 0 0 | 55k 118k
29 16 47 6 0| 15M 1842M| 13M 587k| 0 0 | 133k 114k
26 14 55 3 0| 16M 1124M| 12M 537k| 0 0 | 69k 165k
24 15 52 7 0| 14M 1600M| 13M 575k| 0 0 | 131k 138k
30 14 51 3 0| 16M 1041M| 18M 786k| 0 0 | 62k 155k
20 14 57 7 0| 13M 1719M|8916k 406k| 0 0 | 137k 121k
# pg_analyze (as postgres user)
PostgreSQL 16.3 - relay-ovh - postgres@/var/run/postgresql:5432/postgres - Ref.: 2s -
* Global: 38 minutes uptime, 12.54G dbs size - 14.84M/s growth, 90.60% cache hit ratio
Sessions: 81/100 total, 42 active, 39 idle, 0 idle in txn, 0 idle in txn abrt, 0 waiting
Activity: 4382 tps, 74089 insert/s, 164 update/s, 0 delete/s, 28383 tuples returned/s, 0
* Worker processes: 0/8 total, 0/4 logical workers, 0/8 parallel workers
Other processes & info: 0/3 autovacuum workers, 0/10 wal senders, 0 wal receivers, 0/10
* Mem.: 31.12G total, 756.90M (2.38%) free, 14.14G (45.44%) used, 16.24G (52.19%)
Swap: 512.00M total, 511.00M (99.80%) free, 1.00M (0.20%) used
IO: 155846/s max iops, 2.15K/s - 0/s read, 608.78M/s - 155846/s write
Load average: 8.19 7.33 5.16
I don’t have stats, but at a later phase of backfill, I/O wait was pretty high and disk read/write were more symmetrical around 500MB/sec (should have taken a snapshot of that!), and CPU wait was only single-digit.
After all major backfills, just cruising along at a normal firehose subscription:
# dstat
----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read writ| recv send| in out | int csw
1 1 97 1 0|5794k 8046k| 232k 6851B| 0 0 |4439 7848
1 0 98 0 0|3615k 7399k| 219k 5071B| 0 0 |4062 7163
1 1 98 0 0|4645k 13M| 198k 5184B| 0 0 |4473 7194
1 0 98 0 0|4831k 8142k| 242k 7273B| 0 0 |4174 7581
1 0 98 0 0|3264k 7092k| 178k 4784B| 0 0 |3625 6619
1 1 98 0 0|3564k 7336k| 153k 3394B| 0 0 |3253 5502
1 0 98 0 0|4930k 9139k| 239k 6719B| 0 0 |4119 7364
2 1 97 1 0|6430k 14M| 313k 10k| 0 0 |6372 13k
1 0 98 0 0|3359k 7422k| 172k 5255B| 0 0 |3670 6860
1 0 98 0 0|3929k 10M| 206k 7088B| 0 0 |4036 8954
1 1 98 0 0|3732k 7560k| 212k 6173B| 0 0 |3789 6771
1 1 97 1 0|5694k 8819k| 267k 7758B| 0 0 |4630 8511
1 0 98 0 0|3480k 11M| 175k 4565B| 0 0 |3764 5758
# pg_analyze
PostgreSQL 16.3 - relay-ovh - postgres@/var/run/postgresql:5432/postgres - Ref.: 2s -
* Global: 4 days, 22 hours and 23 minutes uptime, 445.38G dbs size - 162.93K/s growth, 79.55% cache hit ratio
Sessions: 22/100 total, 1 active, 21 idle, 0 idle in txn, 0 idle in
Activity: 514 tps, 912 insert/s, 0 update/s, 0 delete/s, 1028 tuples returned/s, 0 temp files, 0B temp size
* Worker processes: 0/8 total, 0/4 logical workers, 0/8 parallel workers
Other processes & info: 0/3 autovacuum workers, 0/10 wal senders, 0
* Mem.: 31.12G total, 676.83M (2.12%) free, 17.36G (55.78%) used, 13.10G (42.09%) buff+cached
Swap: 512.00M total, 608.00K (0.12%) free, 511.40M (99.88%) used
IO: 0/s max iops, 0B/s - 0/s read, 0B/s - 0/s write
Load average: 0.46 0.44 0.38
# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.2G 1.6M 3.2G 1% /run
efivarfs 192K 37K 151K 20% /sys/firmware/efi/efivars
/dev/nvme0n1p3 1.8T 452G 1.2T 28% /
tmpfs 16G 1.1M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p2 974M 182M 725M 21% /boot
/dev/nvme0n1p1 511M 5.2M 506M 2% /boot/efi
/dev/nvme1n1p1 1.8T 722G 1.1T 41% /data
tmpfs 3.2G 12K 3.2G 1% /run/user/1000
# df -i (inodes)
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 4078568 1019 4077549 1% /run
efivarfs 0 0 0 - /sys/firmware/efi/efivars
/dev/nvme0n1p3 117080064 230862 116849202 1% /
tmpfs 4078568 3 4078565 1% /dev/shm
tmpfs 4078568 3 4078565 1% /run/lock
/dev/nvme0n1p2 65536 603 64933 1% /boot
/dev/nvme0n1p1 0 0 0 - /boot/efi
/dev/nvme1n1p1 187537280 25138236 162399044 14% /data
tmpfs 815713 32 815681 1% /run/user/1000
# sudo du -sh /var/lib/postgresql/16/
447G /var/lib/postgresql/16/
# lsblk (for reference)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 0B 0 disk
sr0 11:0 1 1024M 0 rom
nvme1n1 259:0 0 1.7T 0 disk
└─nvme1n1p1 259:7 0 1.7T 0 part /data
nvme0n1 259:1 0 1.7T 0 disk
├─nvme0n1p1 259:2 0 511M 0 part /boot/efi
├─nvme0n1p2 259:3 0 1G 0 part /boot
├─nvme0n1p3 259:4 0 1.7T 0 part /
├─nvme0n1p4 259:5 0 512M 0 part [SWAP]
└─nvme0n1p5 259:6 0 2M 0 part
PostgreSQL table sizes:
bgs=# SELECT
table_name,
pg_size_pretty(table_size) AS table_size,
pg_size_pretty(indexes_size) AS indexes_size,
pg_size_pretty(total_size) AS total_size
FROM (
SELECT
table_name,
pg_table_size(table_name) AS table_size,
pg_indexes_size(table_name) AS indexes_size,
pg_total_relation_size(table_name) AS total_size
FROM (
SELECT ('"' || table_schema || '"."' || table_name || '"') AS table_name
FROM information_schema.tables
WHERE table_schema != 'pg_catalog' AND table_schema != 'information_schema'
) AS all_tables
ORDER BY total_size DESC
) AS pretty_sizes;
table_name | table_size | indexes_size | total_size
-------------------------------+------------+--------------+------------
"public"."repo_event_records" | 18 GB | 541 MB | 18 GB
"public"."actor_infos" | 987 MB | 993 MB | 1980 MB
"public"."users" | 749 MB | 1060 MB | 1809 MB
"public"."pds" | 3752 kB | 32 kB | 3784 kB
"public"."auth_tokens" | 16 kB | 48 kB | 64 kB
"public"."slurp_configs" | 16 kB | 32 kB | 48 kB
"public"."feed_posts" | 8192 bytes | 24 kB | 32 kB
"public"."vote_records" | 8192 bytes | 16 kB | 24 kB
"public"."follow_records" | 8192 bytes | 16 kB | 24 kB
"public"."domain_bans" | 8192 bytes | 16 kB | 24 kB
"public"."repost_records" | 8192 bytes | 8192 bytes | 16 kB
(11 rows)
carstore=# SELECT
table_name,
pg_size_pretty(table_size) AS table_size,
pg_size_pretty(indexes_size) AS indexes_size,
pg_size_pretty(total_size) AS total_size
FROM (
SELECT
table_name,
pg_table_size(table_name) AS table_size,
pg_indexes_size(table_name) AS indexes_size,
pg_total_relation_size(table_name) AS total_size
FROM (
SELECT ('"' || table_schema || '"."' || table_name || '"') AS table_name
FROM information_schema.tables
WHERE table_schema != 'pg_catalog' AND table_schema != 'information_schema'
) AS all_tables
ORDER BY total_size DESC
) AS pretty_sizes;
table_name | table_size | indexes_size | total_size
-----------------------+------------+--------------+------------
"public"."block_refs" | 192 GB | 217 GB | 409 GB
"public"."car_shards" | 4011 MB | 4088 MB | 8098 MB
"public"."stale_refs" | 6245 MB | 576 MB | 6821 MB
(3 rows)
