Porting Large Language Models to Mobile Devices for Question Answering

Deploying Large Language Models (LLMs) on mobile devices makes all the capabilities of natural language processing available on the device. An important use case of LLMs is question answering, which can provide accurate and contextually relevant answers to a wide array of user queries. We describe how we managed to port state of the art LLMs to mobile devices, enabling them to operate natively on the device. We employ the this http URL framework, a flexible and selfcontained C++ framework for LLM inference. We selected a 6bit quantized version of the OrcaMini3B model with 3 billion parameters and present the correct prompt format for this model. Experimental results show that LLM inference runs in interactive speed on a Galaxy S21 smartphone and that the model delivers highquality answers to user queries related to questions from different subjects like politics, geography or history.