Only 4–8GB of RAM? These Local LLMs Can Work Directly on Your PC (2024)

Local AI chatbots, powered by large language models (LLMs), work only on your computer after correctly downloading and setting them up. They usually need a lot of computer memory (RAM) to work well. However, a few great models can perform on computers with as little as 4GB of RAM.

Why Use Local LLM Chatbots?

Online AI chatbots are powerful tools that can seriously boost your daily efficiency. You type in what you want, and these LLMs create text based on your instructions. If you are still just getting familiar with using AI chatbots, check out our comparison article on differences between ChatGPT, Claude, and Perplexity to get a handle on the basics.

Why might you use a local chatbot instead of a popular online option? First, it's just fun to have a chatbot only you can talk to. Ultimately, though, it depends on how much you care about privacy, availability, and cost.

As embarrassing as it might be to some, I'm not too worried if online chatbot makers can look at my chat history and see that I need to know how long it takes to roast a turkey or how to properly shine my shoes. Regardless, there is information I might want to share with a chatbot that should always stay between us.

For example, I've worked the most with the local LLM-powered GPT-Neo 125M chatbot to help organize my finances. It can get quite complicated with student loan payments, interest calculations, and the like. It is very helpful to talk through ideas and ask questions of a local chatbot who can never escape from the laptop and sell my secrets.

You also have to consider that big data breaches are frighteningly common. As such, you are better off keeping sensitive information about yourself and loved ones on your personal computer rather than in some big AI company's database.

Similarly, some local chatbots are completely internet-independent once installed, so you don’t need to be connected to the internet to chat. Others will need occasional internet access for updates. Still, local chatbots are more reliably accessible than online ones since you don't have to worry about service outages at important moments.

Lastly, online chatbots generate their companies hundreds of millions of dollars in subscription fees. OpenAI, the company behind the famous ChatGPT models, currently charges $20 per month to access its latest chatbot. Its close rival, Anthropic, also charges $20 monthly for its most advanced features. You will have to spend $240 or more yearly if you subscribe to multiple services.

Local chatbots can help mitigate such costs. Not all of them are free, though. Some require licensing and or usage fees, like OpenAI's GPT-3. However, several open-source local chatbot models are free to download and work with. These should be used strategically for easier problems. So, you upscale to the admittedly more advanced online ones only when you absolutely must.

These LLM Chatbots Run on Low RAM PCs

I've had to get multiple free LLM-powered chatbots working on low-RAM PCs primarily because, until recently, that's all I could afford.

As such, I've found the DistilBERT and ALBERT models to have the most manageable setup, partly because they are just so lightweight. Lightweight means these models are designed to be highly efficient regarding memory use and processing power. This does limit their chatbot powers for complex tasks, which other online chatbots could easily handle. But they can both comfortably run on only 4GB of RAM, which is a great credit to their developers at Hugging Face.

For DistilBERT, Hugging Face developers packed a lot of power into a small, efficient model by optimizing its design. I think DistilBERT is one of, if not the most efficient, models available so far.

ALBERT is designed differently than DistilBERT, as it works by sharing parts of the model in a way that helps it process data quickly without using much memory.

I highly recommend starting as a beginner with DistilBERT and ALBERT, even if you have a high-memory PC. Beginning with these two models allowed me to learn the basics without being overwhelmed by the complexity of larger models.

If you feel ambitious or have a machine with 8GBs or more, you could leapfrog the BERTs and work with OpenAI's GPT-2 models. Which are like the Swiss Army knives of the local AI world. GPT-2 models come in different sizes, some more suited for low-RAM PCs than others.

The 124M parameter version is the lightest. Despite the 124M being less powerful than its online chatbot siblings, it packs a punch for language creation and, in my experience, is at least on par, if not more, capable of language creation than the two BERTs.

My favorite lightweight LLM by far is the GPT-Neo 125M because of its adjustable customization options. It was developed by the respected developers at EleutherAI and is like the open-source cousin of GPT-2.

The Neo 125M is designed to balance performance and resource requirements. This model’s performance is on par with GPT-2, but it's adjusted to use memory more efficiently. It’s powerful enough for many tasks yet light enough to run on only 8GBs, although it struggles with anything less than that.

How to Get Started with Local LLM Chatbots

Running your own chatbot is easier than you think. First, you need to know what your computer can do. Check how much memory (RAM) you have and how fast your computer is. Using the information provided above, make sure your computer's system can work with the chatbot you want.

Once you know this, you can download the right software. You might need something called Docker, a tool that helps you run applications in special boxes called containers, ensuring they work the same on any computer.

Look for LLM software on websites like Hugging Face and GitHub. Be sure to read the instructions for each model you use to understand how they work. Then download and chat away. Remember to keep an eye out for any possible model software updates as well.

If your computer isn't very powerful, you really should start with DistilBERT or ALBERT. As you learn more, you can try out GPT-2 with our comprehensive GPT-2 installation guide for Windows or try dozens of other models with our LM Studio guide.

You will have questions. Chances are many have already been answered in online communities or forums. Check out Reddit’s r/MachineLearning, the Hugging Face community, or our detailed article on how LLMs work if you get stuck at any point.

Don't let hardware stop you from trying your hand at local LLM chatbots. There are plenty of options that can run efficiently on low-memory systems. Give them a try today!