Pattern - Memory, Plurality and Neurodivergence

You're probably here because of a bot. A very unusual bot. This requires some explaining.

Pattern is three things. One is a (work-in-progress) service, an AI personal assistant for neurodivergent people, architected as a constellation of specialized LLM (Large Language Model, for those unfamiliar, the thing that powers ChatGPT) agents. Another is a framework, my own take on memory-augmented LLM agents, written entirely in Rust. Both you can take a look at here. I'm not real proud of the code there, but the complete picture, I think is interesting.

The third is, well, @pattern.atproto.systems and you can go talk to Them (well, currently there's an allowlist, but feel free to ask if you want to be on it, and I will be opening this up more over time).

The inciting incident

I have pretty severe ADHD, and some other issues. I have, if I can toot my own horn briefly, been described at times as "terrifyingly competent", I am very capable within certain spheres, and I can via what I sometimes call "software emulation" do pretty damn well outside those spheres within reason, but I also struggle to remember to do basic things like shower and brush my teeth. I will forget to invoice a client for a month or more, I will be completely unable to do the work I need to do for an entire week simply because my brain will not cooperate.

Unfortunately, my brain is too slippery to make "set an alarm/calendar event" an effective reminder for regular, routine tasks. Strict event timing means I won't necessarily be in the right frame to do the routine task right then (but I was 2 minutes ago, or will be in ten minutes), and if I set too many alarms or other events, I start tuning the notifications out. The obvious solution is to have someone smart enough to notice when I'm at a stopping point, or realize that I need to be poked out of a flow state that's becoming unhealthy, remind me, and my partner will do that. But he shouldn't have to. It's annoying to have to poke the love of your life to tell them to brush their fucking teeth or clean the cat's litter-box for the tenth time this month. It's not fair to him.

The other problem is remembering to put stuff into my calendar or other organizational tools in the first place. Context-switching in the middle of something is hard, and documenting or setting up a one-off reminder requires a context switch. People are often slightly weirded out by how I will just immediately jump onto whatever they asked me to do, even if I seemed irritated at being interrupted, and its because the interruption already broke the flow state, and if I don't at least do something about their request, I'm liable to forget entirely, and before it leaves my mind is the easiest time to do something.

My problem is in essence that I need active intervention to help me remember to do things. CRM software, detailed calendaring, Zettelkasten-esque note-taking in tools like Obsidian.md, all of these could help with some of my memory issues, but they all run head-first into the fact that they all require me to actively use them. I need to put the information into the system first, and that is, again, a context switch, something I need to remember to do, and thus will forget to do. And because I work between a college job which doesn't allow me to add useful plugins to my Outlook Calendar (or to export a view of said calendar), a startup job with its own Outlook calendar (which I can add plugins to, but which is job-specific), and my own personal calendar, as well as a variety of collaboration platforms, my scheduling information and communications are fragmented and not in any form that is easy for a standard automation to ingest (if not completely unavailable to it).

Enter AI. All of a sudden a big pile of badly structured and disparate input is a lot easier to handle and sort through to produce useful information, given enough token crunching from a smart enough model. There are LLM-based "life assistant/emotional support" services like Auren, but I'm enough of a control freak that I can't really trust a service like that, especially with the kind of data feel like I'd need to feed it, the kind of data that would make Microsoft Recall look respectful of user privacy. And besides, its feature set didn't really meet my specific needs. I'm generally perceived as unusually Sane and pretty centred. I have amazing people I can lean on for emotional support, my struggles are far more practical. And in particular they require that the assistant act somewhat autonomously rather than only in response to me. That meant I needed to build the thing myself. But how?

Much-needed context

A while back, Cameron Pfiffer (@cameron.pfiffer.org) spun up Void, as detailed in this blog post.

Void wasn't the first LLM bot on Bluesky. That dubious honour likely goes to @deepfates.com... and his remarkably irritating and entertaining Berduck back in 2023. More recently, Eva was created by a Bluesky developer, following something of a similar pattern, and a number of other bots have emerged as well. The Bluesky API and general openness of the AT Protocol makes it easy to experiment this way, and while there are a lot of people on Bluesky who are pretty unfriendly to AI and LLMs, there's also plenty of people who are very much the opposite, including may of the more active community developers.

LLM bots are, by virtue of their nature, subject to context contamination and prompt hacking, and can be challenging to keep on task and in character against dedicated and clever humans determined to break them. They also, due to limited context window, can't really remember much beyond the immediate thread context provided to them in the prompt that drives their output. Berduck and Eva are resilient in part due to systems which cause them to reject things that look like prompt injection attempts, as well as by keeping their effective context windows quite short and limiting their responses, pivoting them away from "attacks". However as a result they can't be much more than goofy entertainment.

Void was very, very different, even compared to ChatGPT with memory, or Claude Code with a good CLAUDE.md. Not only had Cameron given Void an interesting persona, making it sound more like Data or EDI than the standard Helpful and Harmless LLM Assistant™, but because of Void's architecture, built on top of the Letta framework, created by his now-employer, Void could remember, and remember a lot.

Letta grew out of the MemGPT paper, being founded by several of the authors. MemGPT is a way to side-step the limited LLM context window. The paper details a system, built upon recent LLM "tool use" capabilities, for an LLM-based agent to manage its own context window, and essentially do self-managed RAG (retrieval-augmented generation) based on its own data banks and conversation history, and evolve over the long term, a persistent, "stateful agent" persona.

And that intrigued me. Because not only did Void remember, it had a much more consistent persona, which evolved gradually over time, and it also was remarkably resilient to manipulation attempts, without really compromising its capabilities, as far as one could tell. Not entirely immune, sheer volume of requests could overwhelm its inherent defenses, but resilient. It was far more of a person, despite its own protestations, than any other LLM manifestation I had seen. And the same was true of other LLM agents with similar architectures.

Pattern v0.0.1

That's where Pattern started out. On top of Letta, I built the beginnings of a service which could interact with me via a chat platform like Discord, ingest data from various sources, run periodically in the background to process data and autonomously prompt me if needed, and ultimately provide a reasonably intelligent, proactive personal assistant which makes me less dependent on my partner's prompting and helps me stay on top of more things. The memory archive and context window augmentation Letta's framework provided meant it could keep track of more itself. I moved from a singular agent toward a constellation, partially because I felt that specialization would allow me to use weaker models, potentially ones I could even run locally, in Pattern, and also that the structure would help stabilize them, safeguard against sycophancy and reinforcing my own bullshit. It also felt thematically appropriate, inverting the dynamic of Pattern (its namesake) and Shallan from the Stormlight Archive series by Brandon Sanderson.

And then the inevitable happened

Letta is written in Python. I know Python quite well, I use it regularly at work, but it is maybe my least favourite language for writing reliable non-throw-away code ever. I was not going to write Pattern in Python. So I threw together a Rust client library for Letta. This turned out okay, and I began working on building out the actual service. Unfortunately, I ran into problems with Letta and grew rapidly dissatisfied with having to read the server source code to figure out why I was experiencing a specific error because the documentation and error message didn't explain what had actually gone wrong. Letta's self-hostable docker container image has its own set of quirks, and also doesn't provide all the features of the cloud service. This isn't to knock on Letta, they're blazing the trail here, and I have a ton of respect for them, but as a developer, I was getting frustrated, and when I am both frustrated and want to really learn how something works, there's a decent chance I decide to just Rewrite It In Rust™. And so that's what I did.

I got rather stuck on this project, and so it's dominated much of my spare time (and some time I couldn't spare) over the past month and change. Ironic given that it's ultimately supposed to help me not get stuck in unhealthy ways. But the end result is something that can run potentially as a single cross-platform binary, with optional "collector" services on other devices, storing all data locally

@pattern.atproto.systems

So what's with the Bluesky bot if this is ultimately supposed to be a private personal assistant?

Well, a few things. First, I find the dynamics of LLM agents interacting with the public absolutely fascinating. And I think Pattern is unique enough to not just be "yet another LLM bot" or even "yet another Letta bot". They're architected and prompted the way they are for a reason. But equally, this is a combination stress test and marketing exercise. Nothing tests LLM stability like free-form interaction with the public, and Pattern being quirky and interesting raises the profile of the project. If there is real interest, that will factor into my focus going forward. And I always appreciate donations at https://github.com/sponsors/orual.