Seeing models like this work so well gives me hope that mobile first LLMs for things like better voice to text and typing prediction will not just 'work' in 2-3 years but actually not kill your battery too.
If it’s fast enough to be useful it would also not physically be able to use that much power. Your phone CPU and GPU have a maximum W they can pull at any one time and if this runs for a few seconds then the maximum it can use is that.
If it maxes out all cores and memory for 30 minutes then it won’t really work for anything
MLC Chat is already on the App Store and allowed to be used. I haven't used Yi with it, but a quantized Mistral or Llama runs quite well on an iPhone 15. See https://llm.mlc.ai. "Apple GPT" is also rumored to be coming too.
It is processor and therefore battery intensive but it already won't kill your battery inside of 30 minutes. Obviously it will be worse for resource usage than an app if it's always kept running by some OS level process and set as the processing layer for every trivial thing but it seems like cheaper input handling could decide to promote some input up to being evaluated by an LLM or not.
I've also tried a few Android LLM apps, all running more than 30min.
Current LLM models are not running constantly on the phones to drain your battery. They just run when responding to a prompt. It by no means consumes more battery than a heavy game.
