A quick 'n dirty on getting your PDS' block storage working with Hetzner's S3 object storage

@blockstackers.net

First step; get yourself a bucket at Hetzner

You'll want to create your bucket in the same region your PDS is hosted in, mine is in NBG1, so that's where I create the bucket. Unfortunately, Hetzner says bucket names need to be unique globally, but whether they mean "within your account" or, you know, actually globally is left up to interpretation.

I'm going to err on the side of caution and suggest you really do make it a unique bucket name, maybe something like <myhandle-or-something>-pds-storage.

Make sure you create the bucket as private - you can change it afterwards, if you want, but just... create it right the first time. Also generate a set of credentials - you'll want to save those to a file somewhere since your secret key won't be shown again.

Second step; get yourself some s3cmd!

On most Debian based systems (which is what I use), a simple sudo apt-get install s3cmd will get the show on the road. You can create a fresh configuration file with s3cmd --configure - it will ask you questions, and here are some of the answers:

For access key and secret key, well, you know what goes there. The default region is "US". The S3 endpoint is what is shown to you in Hetzners' interface. For a bucket in NBG, this would be nbg1.your-objectstorage.com. The DNS style template is %(bucket)s.nbg1.your-objectstorage.com. You will want to set an encryption password for the in-transit encryption, and also where GPG lives (usually /usr/bin/gpg - if you don't have it, sudo apt-get install gpg will sort that out). You want to say "yes" to HTTPS - the remainder of the options can be left as they are.

Third step; it's copyin' time!

So. Now comes the fun part where you stop your PDS so no Bluesky for you during the following steps! service pds stop (assuming you have a default installation done by Bluesky's installer).

Your blocks live in /pds/blocks (again, assuming a default install). Unfortunately I haven't had much luck with s3cmd's sync command, but since I am the only person on my PDS I just did the following:

cd /pds/blocks/did:plc:n6dwsessu7xbnihnvudlloss
for a in `echo *`; do s3cmd put s3://your-bucket-name/blocks/did:plc:n6dwsessu7xbnihnvudlloss/$a; done

And that will copy all the blocks. Replace the DID:PLC above with your own!

If you have multiple did folders in there, repeat the step for each folder - or figure out why s3cmd sync doesn't do what it says on the tin, whichever you want, I'm not your supervisor.

Fourth step; it's configurin' time!

Edit /pds/pds.env and comment out the PDS_BLOBSTORE_DISK_LOCATION line. Then add the following lines:

PDS_BLOBSTORE_S3_BUCKET=your-bucket-name

PDS_BLOBSTORE_S3_REGION=nbg1

PDS_BLOBSTORE_S3_ENDPOINT=https://nbg1.your-objectstorage.com

PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=true

PDS_BLOBSTORE_S3_ACCESS_KEY_ID=the-access-key

PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY=the-secret-key

I ended up setting the region to nbg1 because for some reason, setting it to US didn't seem to want to work, although that may have been due to an unrelated issue. But this works for me; obviously change the region and S3_ENDPOINT according to your bucket configuration.

And finally...

Start the PDS again with service pds start and enjoy your now non-local-disk-using bucket of blobs.

blockstackers.net
AngryDutchman

@blockstackers.net

devops monkey | ex-infosec | iot | perl | video games | boats | sailing | spicy food lover | blues | rock | cat person | amateur bass player | languages: nl, en, fr, de, id | MDNI

Post reaction in Bluesky

*To be shown as a reaction, include article link in the post or add link card

Reactions from everyone (0)