10 Best Open-Source LLMs You Can Run Locally

Running a language model yourself can give you more control over where prompts and files go, and it can keep experiments available when an external service is not. This guide is for developers, students, and small teams weighing local use against a hosted AI service.

“Open source” is not always used consistently for AI: many publishers release model weights under licences with conditions. Check the current model card, licence, and hardware guidance before using a model commercially or sharing an app built with it.

Quick picks

• Best general-purpose family: Qwen — broad model choices and multilingual strengths.
• Best for local experimentation: Gemma — compact options and accessible tooling.
• Best for coding: DeepSeek Coder — models tuned for programming tasks.
• Best for long documents: Command R — designed with retrieval and tool use in mind.

How we picked

We favoured model families with published weights, recognised documentation, and practical routes to local or private-server inference. “Run yourself” still depends on model size, quantisation, and hardware, so check each release rather than assuming every variant fits a laptop.

• Use case: chat, coding, vision, or document retrieval?
• Resources: can your computer handle the chosen variant?
• Licence: are your intended uses allowed?
• Community: are inference tools and model notes easy to find?
• Quality checks: test on your own examples, not just demos.

Versatile models for everyday tasks

1. Qwen

The Qwen family from Alibaba includes language and multimodal releases in a range of sizes, so users can select a model that balances capability with local hardware. Its breadth and multilingual support make it a strong first family to explore, but capabilities and terms differ by release and must be checked individually. Many weights are openly available; confirm the licence attached to the exact checkpoint.

Best for: technically curious users comparing local models for mixed everyday tasks.

2. Gemma

Gemma is Google's family of open-weight models, with smaller variants suited to experimentation and tools for common machine-learning workflows. It is a straightforward place to learn local inference without immediately needing a server rack. Model performance is not equivalent across sizes, and Google's usage terms are specific rather than a blanket open-source licence. Available weights are free to download under their stated terms.

Best for: learners who want approachable model sizes and clear publisher documentation.

3. Mistral models

Mistral AI publishes several open-weight models alongside its hosted services, giving developers a path to test locally and then compare with managed inference. The family includes models aimed at different workloads, and its efficient options can be attractive on modest setups. Not every Mistral model has identical openness or terms, so verify the model card before commercial deployment. Some are open-weight; hosted API access is paid.

Best for: developers who want to compare self-hosted and managed Mistral options.

4. Llama

Meta's Llama family is supported by a large ecosystem of local inference tools and community guides. That support makes it easier for hobbyists to find compatible runtimes, quantised builds, and examples. Llama's licence is custom and includes conditions, so it should not be treated as unrestricted open source; weights are downloadable under Meta's terms, while hosting providers may charge.

Best for: people who value a large community and many local deployment guides.

Models for development and research

5. DeepSeek Coder

DeepSeek has released coder-focused model options intended for code generation and related programming tasks. They can be useful for private experiments such as explaining a small codebase or drafting a test, especially when paired with a careful review process. Hardware demands depend on the selected size, and generated code can be insecure or simply wrong; check the individual release licence. Weights are available for some models at no charge.

Best for: developers testing a locally hosted coding assistant on non-sensitive tasks.

6. StarCoder2

StarCoder2 is a code model family produced through the BigCode research effort, with published documentation about its training and intended use. The research transparency is valuable for teams comparing coding models responsibly, and its open weights support local evaluation. It is not a substitute for an IDE's full project understanding, and licensing and attribution obligations deserve attention. The model weights are available under the stated open licence.

Best for: developers and educators studying code generation with documented research context.

7. Phi

Microsoft's Phi family includes compact models designed to deliver useful performance with lower resource needs than many very large models. Smaller size can make local testing practical on capable consumer hardware, though results vary considerably by task and quantisation. Check each version's permitted uses and evaluate factual accuracy rather than assuming compact means dependable. Selected weights are downloadable under their specific terms.

Best for: people exploring smaller models for constrained local devices.

Document, language, and multimodal choices

8. Command R

Cohere developed Command R models for retrieval-augmented generation and tool use, making them worth considering for question-answering systems grounded in a supplied knowledge base. Its focus aligns with business search rather than only open-ended conversation. Model availability and licensing vary, and capable variants can require substantial memory; examine the current release conditions. Open weights are available for certain releases, while Cohere's hosted products are paid.

Best for: builders prototyping document-grounded assistants and retrieval workflows.

9. OLMo

OLMo from the Allen Institute for AI is a family of language models released with unusually complete openness: weights, training data, code and evaluation details are all published. That makes it a strong pick when you need to explain where a model came from or study how training choices affect behaviour. It is free to download and run, though it is not always the top scorer for everyday chat tasks.

Best for: researchers and teams who need a fully transparent open model.

10. PaliGemma

Google's PaliGemma combines image understanding with language capabilities, allowing experiments that involve visual input as well as text. It can help with tasks such as describing an image or extracting information from a visual scene, but it may miss details and should not be trusted for safety-critical interpretation. Check hardware requirements and the model's usage terms before deploying. Its weights are available under the applicable Google terms.

Best for: developers trying image-and-text workflows on a locally hosted model.

How to choose the right one

Begin by writing down what you need the model to do and what must remain private. A compact text model may be enough for summarising notes, while image questions call for a multimodal checkpoint and coding work benefits from tests and a human review. Try representative prompts, including difficult examples, before deciding that one model is suitable.

Next, check available memory and storage, and test a quantised version if your runtime supports it. Quantisation reduces model size but can affect output quality. Local hosting also shifts responsibility to you: updates, access control, backups, and safe handling of downloaded model files matter. Our bandwidth calculator can help estimate transfer needs when downloading large weights or serving a remote setup. Browse our AI tools and AI coding assistants directories to compare adjacent options.

Key takeaways

• Treat each model release as its own product: size, abilities, and licence conditions vary.
• Match model type to the job, especially when images or code are involved.
• Test on real examples and verify outputs before relying on them.
• Local inference offers control but requires suitable hardware and ongoing maintenance.
• Read the model card and licence before commercial or public use.