Describe It, Don't Name It: Putting an AI Layer on a House Full of Brands

My house has a little bit of everything. Cync and Tapo bulbs. Leviton and Kasa switches. Samsung TVs. Apple HomePods. Sonos. Aqara and Tapo door locks. Every one of them shipped with its own app, its own idea of what a "room" is, and — as far as I can tell — a firm policy of never speaking to the others.

So the first job was plumbing. The next job is the interesting one.

Where this is going

The next layer is Claude — Anthropic's model — sitting on top of Home Assistant as the thing you talk to.

Here's the problem it solves. Every smart home I've used makes you memorize the house. Is it the nightstand lamp or the end table lamp? Is the routine called "movie time" or "movie night"? Did anyone ever build that routine, or did we just talk about it? Voice assistants are very literal about this. Say it slightly wrong and you get a polite apology about a device it can't find.

What I want is to say "I want to watch a movie in the living room" and have the house work out what that means — lights down, TV on, sound where it belongs — from what it already knows about the rooms and devices. Not because someone wrote a "movie" routine, but because the model can reason from a description to the steps.

That's the part I care about most: you don't need a preset routine for everything. Routines still have a job. They're deterministic, and they run faster — no model call, no reasoning, just execute. The everyday stuff should stay that way. But anything that has never been set up before, you should be able to just describe.

Home Assistant already supports that split. Claude can be plugged in as a conversation agent through Home Assistant's Assist API, and it can only see and control the devices you expose to it. Known commands can be handled locally first; anything that isn't a known command goes to the model. Fast and predictable where it matters, flexible everywhere else.

How I know the approach works

This is less speculative than it sounds, because it's how the house got built.

I don't write the configuration. I describe what I want in plain English, and Claude builds it, tests it, and tells me what it verified. A recent example: the bedroom soundbar plays the TV, and there are two Sonos speakers on the nightstands. I wanted those on the TV sound sometimes — but never automatically, because they sit right next to our heads, and if one of us is watching something late, the other one is trying to sleep.

That was the whole spec. A few minutes later there was a button on the wall panel: tap it and the nightstands join the TV sound, tap it again and they drop out, and it switches itself off after the TV goes off, so the next night starts quiet.

In the interest of full disclosure, the first test went like this: Claude said the speakers were playing, I said they weren't, it checked the speakers directly and said they very much were — and then I heard them. Confidence is not accuracy, and that cuts both ways. Sometimes the human is the one who needs a second look.

The voice layer is that same loop with the typing removed.

Where the line is

"Let the AI figure it out" is a great sentence until it's about your front door.

So the safety-critical actions stay deterministic. The tool Claude uses to control the house refuses to unlock a door or open the garage, no matter how it's asked. Locking and closing are fine — getting those wrong doesn't let anyone in. Opening and unlocking are done by hand, at the door. That rule isn't an instruction the model is asked to follow; it's enforced in the tool itself.

What's already running

The plumbing, for the record:

• One screen for everything. Every light, switch, and lock, whatever the brand, on a touch-screen wall panel.
• TV curfews. A nightly curfew on the kids' TVs, a per-night dial for extra time, a "grounded" switch, and a multi-night pause for when we have guests.
• Whole-home music in three zones. Any mix of Sonos, HomePods, and the TVs' own speakers, playing in sync. (It also opened with the same song every single time until shuffle got turned on. Serenity now.)
• Ambience on any TV. Fireplace, beach, snowfall, thunderstorms — three-hour videos stored on a machine in the house and pushed straight to the TVs over the home network. No Plex server, no streaming subscription, and no app on the TV, so the app locks on the kids' TVs stay locked. The TVs refuse to loop a video, so a watchdog restarts it at the end; if someone switches to the game, it gets out of the way.

The network, too

Claude also reaches the network gear — the access points and the firewall. That's already paying off: after an IP-conflict alert this week, it worked through fixed addresses for about a hundred devices, a few at a time, and checked each one.

The next step is the one every parent has wanted at some point: "the game is buffering — shut off the kids' internet." Not a dashboard, not an admin console. A sentence.

The point

The hardware was never the hard part. Getting a drawer full of ecosystems to behave like one house is. Once they do, the interface stops being an app and starts being a sentence.

I'll report back when I can say it out loud. And when it turns off the wrong lamp — which it will — I'll report that too.