Building my own butler

I can't hire Alfred so I'll build my own

Aug 20, 2024

TL;DR: I’m building my own butler that helps me manage our family life using Make, Zapier, Airtable and LLMs. This post is about the architecture. I’ll build it in public. Follow to join the journey.

I'm forgetful. My brain can hyperfocus and learn half a profession in one night, sometimes I forget to tie my shoes.

My wife is the opposite. She’s the most organized and consistent person I’ve ever known. Things that seem like black magic to me come naturally to her.

I want to be there for her so I decided it’s time to tame my ADHD. I’ve always seen generative AI as an extension of my prefrontal cortex and I could use some help over there. I generated over 8 million tokens by using ChatGPT since it came out, always the same way: open ChatGPT and proactively initiate a conversation.

Proactivity does not work as a requirement when you’re forgetful.

If I had a personal butler living with us, these issues wouldn’t exist. My butler would wake me up every morning, call me “Master David” and tell me everything I need to know to stay on top of my commitments.

But since I’m not Bruce Wayne and can’t afford to hire Alfred.
So I decided to build one.

Let’s begin Project Alfred!

How to build a butler?

Below are what I already figured out about the architecture of Alfred.

Alfred should have three distinct parts:

Foundation, where all the data about our lives live: contracts, invoices, ID numbers, travel documents, medical records, you name it.
Context, because one of the big problems with an AI agent today is that they don’t have context by default. Apart from managing the raw data, I would also need to make sure that Alfred knows what we like, and what we hate, knows about our projects, and finds the gaping holes in our dreams.
Management, Alfred should proactively seek me out and require the least amount of interaction, and the least amount of interruption from me. This follows a concept I read from a stellar designer, Golden Krishna: The best interface is no interface.

Foundations

To feed Alfred with the right kind of information, we will need four types of data:

Documents: I already have a GDrive folder containing scanned versions of our IDs and personal documents. I will start by uploading these to a centralized database, most likely Airtable, which will serve as the foundation for Alfred. These documents include everything from our passports to insurance papers—anything that’s not likely to change frequently.
Health Data 1/2: We both wear an Apple Watch. Apple Health is constantly updated, but you can’t export. I’ll use the Health Auto Export app, to automate exportingdata in JSON for a $25 lifetime licence. This will ensure that our health data is always current.
Health Data 2/2: In Hungary, all our medical history is stored in a centralized government system called EESZT. It has an API, but not for personal use. I’ve reached out to them to see if I can gain access. If not, I’ll set up an RPA (Robotic Process Automation) to log in and download the my data periodically. This might be a gray area in terms of permissions, so I’ll need to double check.
Financial Data: We bank with CIB Bank in Hungary, but they don’t offer an API for personal use. I’ll use Toshl Finance, which connects to our CIB account thanks to PSD2. While I won’t be able to make transfers via the API, at least I’ll have an automated way of accessing our financial information.
Utilities Data: In Hungary, most of our utility bills can be accessed and paid via a service called Dijnet. I’ll want to download all invoices and save them in a database, automatically updating the records. Dijnet doesn’t have an official API, but that doesn’t stop resourceful engineers from creating an unofficial one.

Context

The second phase of building Alfred involves creating a system for managing our projects. We tend to juggle many projects at once—whether it’s home renovations, planning a holiday, or pursuing a DIY project. These projects change frequently, so I need a simple way to create new projects, enrich them with context and then keep track of them.

I envision this being a voice-based system, possibly using Whisper or AssemblyAI for transcriptions and then fed into a narrow agent that generates projects. All I need is a way to upload an audio file describing a new project. For example, I might record a voice memo about planning our honeymoon. Alfred will then convert this memo into a structured project, highlight any missing elements, and ask for my approval before finalizing it.

With all our financial data accessible through Toshl, Alfred will generate a detailed cashflow and balance sheet. Having a clear financial picture will allow us to plan our projects more effectively and avoid overspending.

In the future I can see giving Alfred the ability to generate different scenarios (frontier models are great at generating dummy data) so forecasting is a potential future step.

Management

This last bit is a big challenge because this is where every system, app, and journal I’ve ever used failed for me. It required me to build new habits, but that’s a really backwards way of thinking about software. As someone jokingly asked me a few months back:

If it’s called machine learning, why is it me that’s doing the learning?

(Disclaimer: the ability to pick up new habits and changing behavior is important. In the last two years I quit smoking, gone fully sober, started exercising and joined a Muay Thai gym. But changing behavior just because use want to use a piece of software is stupid.)

Detour: The Best Interface Is No Interface

Alright, here’s a tangent about software development, or more importantly user experience design. This will help understand why DIY’ing your own software is a big deal.

I’m oversimplifying here but this is how software gets created in 4 steps:

Users exhibit Behavior A which leads them to Result A. They don’t like Result A.

What they would like is Result B.

But that’s impossible, because that would require Behavior B, which is way too complicated, takes long, too expensive, needs more people, etc.

So they settle on a good enough substitute. Something that is within reach. It won’t give you Result B, but it will give you Result C. All you need to do is to make a SLIGHT change of behavior from Behavior A to Behavior C.

This has been a guiding principle in software for a long time. People or organizations will change their behavior if it allows them to reach higher levels of utility, which looks like this:

$U(R i )>C(B i ,ΔB i )$

where U(Ri) is the utility of the new result their behavior generates and C(Bi, ΔBi) is the cost of changing the behavior. As long as the result is worth more than the change, investing in software makes sense.

The more you can personalize an experience for the user, the lower ΔBi will be. However, software development is expensive and you can only personalize an app so much.

After all, you can’t just build your own software for everything, right? RIGHT?!

Well, with no code and generative AI, we have a new phenomenon to deal with, which is the cost of developing software is collapsing. Not because Devin will replace your CTO but because most small businesses don’t even have a part-time developer, let alone a CTO.

But if the cost of developing software drops to zero, we can keep personalizing user experiences ad infinitum, until our software is so personalized, that it doesn’t require ANY behavior change from us.

That means that tech would be so embedded we would barely notice it.

The best comparison would be floor heating or solar roof tiles. Stuff that just…works.

That’s the user experience I need for Alfred, to seamlessly blend into my life instead of me having to talk to a fucking chatbot all the time.

Alfred the Faceless

I have to build Alfred around how we live and not the other way around. Again, I’m not very organized, but Eszti is. She thinks a lot about the stuff we have going on, regardless of how of that is shared with me.

She explained to me knowing that there is an automated system thinking about these things 24/7 means she wouldn’t have to.

But that starts with what works for us now which is centered around three main rituals:

Weekly sessions: We’re not too strict with this, but whenever a big week is coming up, we just sit down and talk through stuff we need to do. If needed, we add new projects to it or remove them. These are longer conversations and I think Alfred can make them shorter.
Task Scheduling: Everything we do lives as a calendar event. This works really well for me because I don’t forget about it, however, I do need to open the calendar for this.
Daily check-ins: During the morning rush we usually discuss what’s on our tables for the day. This works well to give each other context.

I will want to replicate this somehow.

So here’s the Alfred-enabled version:

Weekly sessions: Alfred should give me a full breakdown of the stuff that is definitely coming up, any deadlines and todos. Instead of us having to think things through, we should just get a report. Now we usually open Notes to start taking notes, so I would expect Alfred to create a Note with the next week’s summary in it.

Task Scheduling: I call these tickets. I used to create Reminders with Siri using my Apple Watch. But then I would just forget about my Reminder, so I added notifications. Not good, because what if I misjudged when I would be able to do it.

The next thing I did was a simple Make scenario that took my Reminder and push it to an app called Motion. Motion can automatically schedule a task to the next available timeslot in my calendar.

(This created a bunch of unused Reminders so I set up a Shortcut to delete all Reminders from my phone every Monday 7am).

The problem with Motion is that it costs $34 per month if I use it as an individual and pay monthly. Worry not, Make and Zapier can both connect to Google Calendar so I started building a simple agent that checks for empty timeslots, understands the task, and recommends where to put it as a calendar event.

Daily check-ins: I used to work with a virtual assistant, Jenny, who was invaluable in helping me navigate my day-to-day life. She would call me every morning to review my calendar, ask what was on my mind, and reconcile the two to create my to-do list for the day. I want Alfred to replicate this process. Bland AI will be the perfect tool for this, allowing me to set up a daily morning call where Alfred reviews my schedule, helps me prioritize tasks, and even follows up if I miss the call. If I’m not ready, I can reject the call, and Alfred will try again in ten minutes. This feature will help me start each day with a clear plan and ensure that I don’t miss anything important.

Building an AI Butler

So this is no small task. As you can see, I’m more focused on the user experience than what LLM I’m gonna use or how I will build things. I don’t care about accuracy either.

Sure, it shouldn’t make things up, but what’s the alternative?

Everything lives in our heads so the alternative is that…we make things up?

I think one of the things the whole market got wrong about AI was expecting a zero margin of error. Humans have a huge margin of error in thinking, we’re basically idiot monkey with iPhones. If whatever AI agents we build make 20% less mistakes than we do, we’re golden.

Also, even if Alfred makes a ton of mistakes (I’m sure it will) just having everything in one place should be more than helpful. As you can see, Alfred is just three things:

A set of databases
A set of Narrow Agents that interact with the database
A set of Narrow Agents that interact with humans

I will share progress here and also share any documentation or Make scenarios that I build along the way.

That’s it for now. I’m not sure if I’ll write a post next week, because even though we got married last year, we never had a wedding party — so we’re having one this Saturday. But I’ll be back after. In the meantime, feel free to comment or add your thoughts to this.

Also, if you liked this please subscribe and support my blog with a pledge:)