Connect Hedy to a self-hosted / local LLM server (OpenAI-compatible endpoint)
Julian Pscheid
Today Hedy offers two LLM options: Hedy's cloud API, and an on-device model that runs locally on the mobile phone. What's missing is the ability to point Hedy at a user's own LLM server running on separate hardware.
Request: Add a configuration option to direct Hedy to a self-hosted LLM backend via an OpenAI-compatible endpoint (e.g. Ollama, llama.cpp, LM Studio). The user would enter their server URL (and any required key), and Hedy would send transcription/analysis prompts there instead of to the phone's on-device model or Hedy's cloud.
Why this matters (per Jacek):
- Privacy: sensitive data never leaves the user's own hardware, without depending on the cloud.
- Better output quality: users with capable GPUs (e.g. 32GB VRAM running 27B models) get far better results than small on-phone models.
- Faster responses than on-device mobile inference.
- Preserves phone battery vs. running a model on the phone.
Jacek notes this scenario matches the original 'Local LLM' ticket, and suggests it may be a relatively small addition since Hedy already supports a remote LLM backend (Hedy's own API) — extending that section to allow any OpenAI-compatible URL could be enough.
T
Team Insights
Question if we use local llm and use laptop and desktop pc is can we still sync notes across devices? also with local models can we still get similar meeting summaries, tasks and notes like the cloud model?
J
Jacek
Team Insights
- If we use local llm hosted on a separate machine, can we still sync notes across devices?
I don’t see how is it going to differ from using cloud models. Everything is bound to the device you use right now and the llm running machine serves you in the same way as Hedy cloud. The app would just connect to any OpenAI compatible backend. Using Hedy cloud is not an obstacle to syncing notes right now, even though it’s remote too.
- Is the output quality going to be on par with current Hedy cloud?
The backend could be your own self hosted instance running small qwen, or a thin docker container sending requests to large deepseek model from open router, latest gpt, or even fable 5. Users could use HIPAA compliant llm backends should they find need and budget for it. Possibilities will be infinite once Hedy allows 3rd party, OpenAI compatible llm backend usage. I think OpenAI compatible is current industry standard?