I built my own local AI — and can control it from my phone | usagoldmines.com

There are many good reasons to avoid interacting with “AI” services. They’re expensive, they’re destructive, they’re a privacy nightmare, they’re morally dubious, and they’re often controlled by people who are… shall we say, unconcerned about the effects of their actions on others.

But a lot of these problems can be solved if you decide to roll your own LLM. PCWorld’s Adam Patrick Murray decided to give that a try.

When MSI offered Adam a Cubi NUC with a Core 9 processor and an NPU capable of running smaller, localized AI models, he decided to give it a shot. (The video in which the Cubi NUC is featured is sponsored, FYI, but this was a cool enough project that we wanted to highlight it.) Adam reached out to Wendell Wilson of Level1Techs for some advice on the setup, which immediately became a bit more complicated.

The mini PC acts as host for the AI agent based on Hermes, which controls the more beefy hardware in the AI server. That’s running on the far more powerful Framework Desktop and its Ryzen AI Max hardware (which is basically purpose-built for this kind of thing). It’s using the Quinn model inside LM Studio and Ubuntu.

The last piece of the puzzle is the device Adam uses to access all of this: his phone. As it turns out, you can control Hermes through the Telegram messaging app, which gives it all a super-easy front-end for the user. That makes the AI agent and server running on Adam’s home network accessible from anywhere.

What is Adam using all this for? Basically to gather and read news for his podcasts, The Full Nerd and Expedition Handheld. And after a lot of work to set up the server, agent, and access point, he’s still experimenting with ways to make it practically useful. If you want to follow along, be sure to subscribe to PCWorld on YouTube.