The messaging layer

My conversations were scattered across three apps I don't control. Bringing them into one private store I do control turned out to be the easy half. Teaching an agent to be useful with them was the hard half.

A note on privacy first, because it shapes everything here. My conversations are about as personal as my data gets, and the whole point of this layer is that they're mine: mine to keep, mine to query, and mine to decide what to do with. So the examples in the prose are kept representative, and the one screenshot below is a real but deliberately thin slice of my own "owed a response" inbox.

The itch was simple enough. Could I give my agentic layer context from my email and messaging? And could I get it to help with prioritisation (a human failing is that I'm not very good at replying to messages).

My personal email lives (mostly) in Gmail, on a Google Workspace setup. It's also where my personal calendar lives. My everyday messaging conversations live in WhatsApp, Signal and Telegram: three apps, three silos, and no way to ask a question across the lot of them.

Getting the messages flowing

For email, it was easy as there are excellent existing connectors that work in real time. That said I did end up creating a custom tool, but 90% is wrapping Google's own Workspace CLI tools, so that there is consistency across RafeOS. Gmail also does a good job of marking emails as important and its search tools let you run triaging-aligned queries (e.g. emails that have not had a response).

For messaging the plumbing is a Matrix homeserver (Conduit) with bridges into each platform, pulling everything into a local mirror. This part took a while to get stable, but what matters is the shape of what came out the other end: one store, every platform, queryable (well, not every platform, LinkedIn is pretty hostile to external integration...).

And here's what I didn't expect. Plumbing was the easy half; usefulness was the hard half. Once the messages were flowing, I had a searchable archive and a vague sense of triumph, and then I realised I had to rethink how an agent should use this. A wall of every message I'd ever received is not useful. The real design question, the one that took far longer than the bridges, was what an agent needs to be genuinely helpful here.

Three types of question

I ended up going back to the primitives of messaging and how I might satisfy the original itch in a consistent and coherent way.

So the layer is built around three modes, and an agent picks the one that fits the question being asked.

The first is the inbox triage mode: what needs my attention? This is the triage view — recent activity, weighted, with the noise pushed down. It's what feeds the morning briefing's "you've got a couple of things waiting" line. This works by looking at both the messaging store and Gmail inbox, applying some deterministic rules, and using Anthropic's Haiku to do some classifications and scoring. A second iteration added a salience layer, which is really just a simple way to boost the scoring of some contacts (family and friends) and mute others (e.g. noisy WhatsApp groups).

rafeos gmail action-needed --days 7
rafeos messaging action-needed --days 14

The single most useful thing this mode does is refloat the messages I've forgotten to reply to. I set it a window of roughly a fortnight: if something arrived in that span and I never answered, it floats back up. And it's genuinely good at it. What it catches is mundane and exactly the kind I drop: an email missed on a busy day, a bill that needed paying, a refund I'd meant to chase, a family member's suggestion we meet up while they were passing through London.

rafeos gmail awaiting-reply --days 14

The second is person-centric: what did Helen say? You point at one human and pull their recent thread, regardless of which app it arrived through. This is the mode that benefits most from the bridge existing at all, because in real life a conversation with one person hops between platforms and your memory of it doesn't. I'll ask my agent to go back through everything with one person (sometimes more than a year deep) and remind me what we last discussed before I reply. It's mostly retrieval and summary, but it changes the reply: I'm answering with the context in front of me rather than half-remembering. The small human wins are the ones I'd be embarrassed to admit I needed: a quick "remind me when their birthday is" or "what are their kids called", pulled out of a thread in seconds.

rafeos messaging people --tier favourite
rafeos messaging rooms --name 'Ewan'
rafeos messaging history --room !roomname --limit 50

The third is topic, which is plain search: find that message about the restaurant. No idea who, no idea when, just a phrase that ought to be in there somewhere. This is where full-text search earns its keep:

rafeos messaging search "deadline NOT spam"

That's FTS5 under the hood, so the boolean operators do what you'd expect (AND, OR, NOT, phrase matching). Cheap, fast, and exactly the verb an agent reaches for when it has a fragment and needs the message.

The payoff that surprised me, though, only shows up once messaging stops being a silo and joins the rest of the system. The good moments are the cross-channel ones. I'll be working through how to respond to something and realise the relevant context is scattered: part of it was a conversation with ChatGPT, part is sitting in an email thread, part is in a WhatsApp exchange. Pulling all of that together in one place, then thinking about the reply, is the thing I couldn't do before at all.

The unglamorous half that makes it honest

If I only described the three tidy modes, you'd be getting the demo, not the system. The lived version has rough edges, and the rough edges are the part worth reading, because they're where the actual work went.

• Names are the worst of it. Messages don't always arrive labelled "Boris". They arrive as platform IDs, email addresses, and phone numbers. A raw number is no use to an agent trying to answer "what did Helen say?" Contact-name resolution (mapping the identifiers back to the humans) is unglamorous and never quite finished, but a bit of human steering and some deterministic rules do help a lot.
• A group is not a DM. A direct message from one person and the 200th message in a busy group chat are not the same signal, and an agent that treats them alike will either spam you or miss the thing that mattered. The fix here was the salience layer that helps the inbox mode weigh them differently.
• A search hit on its own is nearly useless. Finding the message that says "yes, 8pm works" tells you almost nothing without the question it answered. So a hit can pull the messages either side of it for context, the small thing that makes search results readable rather than cryptic. The fix here is loading in wider context and validating a search.

None of that is exciting. All of it is the difference between a thing you built once and a thing you actually use.

Read-only, on purpose

One firm line: this layer reads. It does not send. The bridges are wired up so the system can see my messages, full stop. There is currently no path by which an agent fires off a WhatsApp on my behalf.

The principle across all of RafeOS is read-everything, write-with-boundaries; messaging is the clearest case for keeping the boundary firm. But I am tempted... so maybe autonomous outbound will get its own treatment when I'm ready to think it through properly.

Email is a little different. In common with others, I am using agents to draft email responses, but for now at least, nothing goes out without my approval. For me the highest value has been getting a context-prompted start on a reply... and then redrafting or adding to it as needed. This tends to be better than the AI drafts built into the apps themselves (though I use those too).

Why ownership is the whole point

My conversations are some of the most personal data I generate, and by default they sit inside three companies' silos, searchable on their terms, queryable through their interfaces, gone if they decide so. Pulling them into a store I run doesn't make them more secure in some abstract sense, but it does unify them and make them mine to query. I can ask my own questions, in my own words, across services, with an agent that works for me and nobody else.