How I Run Local LLMs

In this article I show you how I set up local LLMs in VSCode. We’ll use them for code completion and chat, right inside your code editor.

---

UPDATE: I gave a presentation on this, with some updated tooling! Check it out here!

Why bother setting this up

First and foremost, sometimes GitHub Copilot is just not available for your editor. This is how I arrived here, since I started using Positron as my IDE for all things Python and R, and at the time of writing, you cannot use Copilot in it. Chat I could live without, but the missing the AI code-completion really hurt.

Another significant benefit is the ability to maintain complete control over your work’s privacy - not everything needs or should be uploaded to an API. By running local models you can be sure that no sensitive information leaks out, which comes in handy if you’re working with sensitive data or documents.

We have two main objectives to get this up and running. First, you’ll need to host and run the models themselves, which involves setting up the necessary infrastructure to get them up and running. Once that’s taken care of, step two is finding a way of interacting with the LLMs in a way that feels intuitive and natural - whether that’s using them for code completion or engaging in conversation.

Pre-requisites

Before diving into setting up your local LLMs, let’s make sure you have the necessary pre-requisites met. First and foremost, you’ll need to have VSCode installed on your machine - this will be the hub where you interact with your LLMs.

In addition to having a capable code editor, it’s also essential to be not be terrified of working in the terminal. We’re not going to do anything crazy, you don’t have to be a unix-wizard, but we’re gonna rely on some terminal commands during the setup.

Finally, your machine will need to meet certain hardware requirements. While my own 14-inch M1 Pro MacBook Pro with 16GB of RAM has been more than sufficient for running smaller models (up to 7B or so), it’s worth noting that the amount of RAM on your machine directly impacts how large a model you can run without issues.

If your machine is equipped with a stronger GPU, you’ll also see a noticeable boost in inference speeds - but if you only have CPU, things will still work, just at a slightly slower pace.

Ollama

Now that we’ve covered the basics, let’s dive into using Ollama to host and manage our local LLMs.Ollama is an app designed specifically for running and managing local LLMs, and it’s incredibly easy to get started with.

First, head over to the Ollama website and explore the vast library of models available - you’ll find a wide range of sizes and types to suit your needs. Don’t worry if this feels overwhelming, we’re going to tackle model selection in the next section.

To begin using Ollama, simply go through the install steps on their website. Once installed, make sure it’s running in the background - this will be our central engine for running and managing our local LLMs.

Model choice

Now that you have installed Ollama, let’s dive into choosing a suitable model for your needs. This is where things get a bit more subjective, as the ideal model will depend on your personal preferences and use cases.

When selecting a model, you’ll need to find one that balances two key factors: size (in terms of parameters) and performance (inference speed). On one hand, a larger model will generally be more capable and useful, but it may also consume more resources and take longer to reply. Conversely, a smaller model, while faster to reply, but might struggle with more complex tasks.

You’ll likely want to have different models for various purposes, in our case, code completion, chat, and embeddings. Let’s take a closer look at each of these categories:

Code Completion

For code completion, my personal preference is speed over depth. I just want to avoid retyping lines for multiple calls, and any decent model should be able to handle this efficiently.

I’ve found that the qwen2.5-coder:1.5b model from Ollama’s library is pretty good for code completion. It has other varieties with larger models (3B) that are still reasonable-sized for completion on a stronger consumer machine, but for the basics, the 1.5B flavor is plenty.

Chat

When it comes to chat, things get more nuanced. You’ll need to like the “vibe” of the model and find one that gives you the types of answers you’re looking for.

For small models, I recommend checking out phi3.5:3.8b, which is Microsoft’s take on Small Language Models. They’ve curated the training data carefully, resulting in a pretty cool chat experience for such a lightweight model.

If you prefer larger models, consider using llama3.1:8b, which is Meta’s top-of-the-line open source Llama model, the 8B parameter version. This should use around 6-7GBs of RAM while generating, so keep this in mind when running multiple models together. This model is the one I use for most of my work.

Embeddings

Finally, for embeddings, I simply used the one that Ollama recommends: nomic-embed-text. It worked reasonably well for RAG-style tasks, so I didn’t need to experiment further than this.

For the remainder of the tutorial, I’ll assume that you are using qwen2.5-coder:1.5b, phi3.5:3.8b and nomic-embed-text.

Code to download them:

ollama pull qwen2.5-coder:1.5B
ollama pull phi3.5:3.8b
ollama pull nomic-embed-text

Ollama cheatsheet

Some useful commands to know:

• list all available models: ollama list
• show currently running models: ollama ps
• remove model: ollama rm {model name without brackets}
• run model in command line: ollama run {model name without brackets}

Continue.dev

Now that you have set up your engine for running the models locally, I’d like to introduce you to another powerful tool: Continue.dev.

Continue.dev is an open-source AI code assistant designed specifically for VSCode (and JetBrains, too!). It offers a range of features to help with coding tasks. To get started, refer to the Continue.dev documentation and follow the installation instructions for VSCode. Note that during setup, you may encounter requests to configure API connections; since we’re running our LLMs locally, these can be skipped.

Just keep in mind that when setting up Continue.dev, you might encounter some API setup requests, but we can skip those since we’re running our LLMs locally.

Setup of Continue

Everything happens in the config.json file that you can open by clicking the little cog in the corner of the continue pane.

Now, if you’re unfamiliar with JSON, this might be scary, but it’s not as bad as it looks. Let’s go through it together:

You specify model objects here, that look like this:

    {
      "title": "Big Llama = Whatever name you want to give your model ",
      "model": "llama3.1:8b = Name of the model, exactly as seen in 'ollama list'. You have to use the colons with the weight too!",
      "provider": "ollama = This tells Continue that it should look for a local model."
    }

You need to specify which model you want to use like this these for the chat model (under models at the top) and for the code complete model (under tabAutocompleteModel). The only exception is the embedding model, that one doesn’t have a “title” field.

An example of a full config.json using the models I mentioned above can be found below, feel free to copy it into your own! It’s the full config file, just with the models adjusted, all the other settings are defaults.

{
  "models": [
  
    {
      "title": "Phi 3.5",
      "model": "phi3.5:3.8b",
      "provider": "ollama"
    },
    // i included a commented other model, so you can see how you can have multiple chat models. you can even switch between them!
    // {
    // "title": "Big Llama",
    // "model": "llama3.1:8b",
    // "provider": "ollama"
    // },
  ],

  "tabAutocompleteModel": 
  {
    "title": "qwen2.5-coder:1.5b",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  }
  ,
  "tabAutocompleteOptions": {
    "multilineCompletions": auto,
  },

  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  }
  ,

  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Using Continue

I think the Continue Docs do a great job of introducing themselves, so I won’t try to do one better. Most things they do is quite close to Github Copilot, so if you’re familiar with that, you’ll feel right at home.

Summary

I hope this tutorial gave you can help expand your horizons in what’s possible with locally hosted LLMs!