Local LLM
complete
Julian Pscheid
To reduce ambiguity, I've created a separate feature request specifically for pointing Hedy at a self-hosted / local LLM server via an OpenAI-compatible endpoint (Ollama, llama.cpp, LM Studio), which is distinct from the on-device model already shipped in v3.2. Please follow and vote on it here: https://feedback.hedy.bot/feature-requests/p/connect-hedy-to-a-self-hosted-local-llm-server-openai-compatible-endpoint
Julian Pscheid
updated the status to
complete
Local LLM support was added in v3.2.
Julian Pscheid
Yes! We would love this. We've been testing local models continually since we first launched Hedy. Unfortunately local models still struggle with the amount of data Hedy needs to process in a single prompt and the outputs are not useful. Not even with larger models that can run on Macbooks. Hopefully we'll get there later this year.
J
Jacek
Julian Pscheid any chance to get back to it? A lot have changed since. There is an option to run 2b-9b models on phones now, why don’t allow running larger models self hosted on our own hardware? I have a 32gb vram gpu and can comfortably run 27b models (e.g. qwen 3.6). It would be great to add such option. Compared to running models on phone:
- increased phone battery life
- better quality output
- faster responses?
- better privacy vs using cloud
Julian Pscheid
Hi Jacek, yes, this is now possible!
J
Jacek
Julian Pscheid It’s not added in a form that’s asked in this feature request. When using my mobile device, I see an option to use a model running locally on mobile phone, but I do not see an option to connect my phone to, for example, llama.cpp/ollama/lm studio backend.
Julian Pscheid
Jacek I see, you would want to point Hedy on your phone to your own LLM server. This is something that's definitely on our radar, we just have to prioritize it honestly, since realistically at this point very few users would take advantage of that. But I do agree that in the future this will become more popular, so we're monitoring that option.
J
Jacek
Julian Pscheid since this is a scenario described by the initial ticket, can we reopen it to keep track of that request?
Btw. Would it really be that difficult? There already is support for remote llm backend, but limited to api provided by Hedy. That’s how I understand it works with „local” option disabled? Wouldn’t it be enough to add a configuration around that section to point at any OpenAI compatible backend (or whatever format you use right now)?
Btw2. Thank you for quick replies, it’s a great feeling to be heard as a user :)
s
slifirczyk@gmail.com
Security of your information!