QwQ-32B: The Little AI Giant Taking On the Colossi (Even on Your PC)

QwQ-32B, the compact AI model challenging the giants
The QwQ-32B model: small, but as powerful as a giant
Matteo 7 min

As we’ve already seen in the DeepSeek guide and in the article on Open WebUI, there are now models that let you have high-performing artificial intelligence right on your own computer, without pricey monthly subscriptions and without having to share sensitive data with any tech company. Privacy is back in fashion, and local AI is its runway.

Qwen has released QwQ-32B, a model that seems to have broken the unwritten laws of AI. If until now you thought local models were just cute toys, limited compared to the cloud giants, get ready to think again. I’m about to show you a model that, with “only” 32 billion parameters, dares to challenge colossi 20 times larger, like Claude Opus.

The model had been teased in preview a while ago, but with performance that wasn’t exactly thrilling. The final version, however, is a whole different story. We’re talking about a model that, in some benchmarks, manages to compete with giants like Claude Opus, which has a whopping 671 billion parameters!

You may be wondering: “How is that even possible?”. Well, I was a bit skeptical too when I read those numbers, but after testing the model myself, I can tell you there’s some truth to it. And I want to share everything I found out with you.

The Secret of QwQ-32B: Innovative Reinforcement Learning

QwQ-32B’s real innovation lies in its approach to Reinforcement Learning (RL), which represents a significant step forward compared to traditional methods. Unlike other models that simply distill knowledge from larger models, QwQ-32B uses an RL strategy split into two main phases:

  1. Phase one: RL on verifiable problems Instead of using traditional reward models, the Qwen team employed:

    • An accuracy verifier for math problems that checks the correctness of final solutions
    • A code execution server to evaluate whether the generated code passes predefined tests

    This outcome-driven approach allowed the model to continuously improve its reasoning abilities in specific domains, episode after episode.

  2. Phase two: RL for general capabilities After honing its math and programming skills, the model underwent further RL training with:

    • Rewards coming from general evaluation models
    • Rule-based verifiers

    This phase broadened the model’s capabilities beyond math and programming, improving its ability to follow instructions, align with human preferences, and behave as an effective agent, without significantly compromising performance in the specific domains.

This two-phase approach is reminiscent of the one used for Claude 3.1, but with some significant differences in implementation techniques and in how the various capabilities are balanced. The result is a relatively compact model that shows advanced reasoning abilities comparable to those of much larger models.

Hands-On with QwQ-32B: Practical Tests

I downloaded the model to test it locally, and I have to say the experience was eye-opening. QwQ-32B shows impressive Chain of Thought (CoT) reasoning quality, comparable to that of Claude 3.1.

One of the first things I noticed was the model’s ability to tackle complex problems through structured reasoning. We’re not talking about a model that “pretends” to reason, but one that actually follows logical paths to reach solutions, often explicitly showing its thought process.

The Context Window: A Fundamental Feature

A particularly remarkable aspect of QwQ-32B is its large context window, which can reach up to 131,071 tokens. That’s significantly more than many other local models and even some cloud services.

During testing, I noticed that with the default (more limited) context window, the model sometimes “forgot” the earlier context during particularly complex reasoning. By extending the window through a custom Modelfile, this problem was solved, allowing the model to stay coherent even in very long conversations and reasoning chains.

This feature is crucial for:

  • Analyzing long documents
  • Programming and debugging complex code
  • Keeping long conversations going without losing the thread

Performance on Specific Tasks

I put QwQ-32B through several types of problems to assess its capabilities:

  1. Mathematical reasoning: The model showed a remarkable ability to solve complex math problems, spelling out every step of its reasoning. This is clearly an area where the specific RL training paid off.

  2. Programming: Code generation was particularly impressive. QwQ-32B doesn’t just write snippets; it can develop complete solutions with relevant comments and handling of edge cases.

  3. General reasoning: Even on more open-ended questions, the model showed structured, logical thinking while keeping a natural conversational tone.

All of this, always bearing in mind that we’re talking about a model running locally on my computer, with no need for an internet connection or expensive subscriptions.

Comparison with Other Local Models

To put QwQ-32B’s capabilities into perspective, let’s see how it compares with other models that can be run locally:

DeepSeek-R1 (7B/14B):

  • As we saw in the previous post, it shows good reasoning abilities
  • Visible thinking via <think> tags
  • Requires fewer resources than QwQ-32B

Mistral-Large (8B/12B):

  • Balanced performance across various domains
  • Very good at generating fluent text

What sets QwQ-32B apart in this landscape is its ability to combine advanced reasoning and broad knowledge in a single locally runnable model, albeit with higher hardware requirements than smaller models. The impact of outcome-driven Reinforcement Learning training translates into a model that can tackle complex problems with a more methodical, structured approach.

The Limits: When Reality Comes Knocking

Despite its impressive capabilities, it’s important to be realistic about QwQ-32B’s limitations when run locally:

  1. Hardware Requirements: The model requires significant resources.

  2. Generation Speed: Compared to cloud services with specialized hardware, generating long responses can take longer, especially on consumer hardware.

  3. General Knowledge: Although the model excels at reasoning and programming, its general knowledge may not be as up to date as that of the constantly updated cloud giants.

  4. Tool Integration: Advanced features such as web search, image analysis, or interaction with external APIs require custom integrations that aren’t included in the base model.

How to Install and Use QwQ-32B with Ollama

If you want to try QwQ-32B on your computer, here’s the right procedure:

  1. Prerequisites:

    • A computer with at least 16GB of RAM (preferably 32GB or more)
    • Disk space for the model (about 64GB)
    • Ollama installed (see the DeepSeek guide for details)

  2. Download (first time only) and run the model:

    ollama run qwq:32b

  3. Integration with Open WebUI: For a more user-friendly experience, you can use Open WebUI. Once installed, QwQ-32B will automatically appear in the list of available models.

It’s important to note that running this model requires significant resources. During my tests I noticed that, besides consuming a lot of memory, it makes the device heat up considerably during long sessions, especially when using a large context window.

The Future of Local AI with Reinforcement Learning

QwQ-32B represents an important step forward in making advanced artificial intelligence models accessible locally. Its innovative approach to Reinforcement Learning opens up new possibilities for the development of future models:

  1. Efficient Specialization: Rather than endlessly increasing model size, QwQ-32B’s approach shows that smaller models, strategically trained with RL, can achieve competitive performance in specific domains.

  2. Democratizing AI: By making advanced reasoning capabilities available on consumer hardware, QwQ-32B helps democratize access to AI technologies that were once reserved for large companies.

  3. Privacy Preserved: The model lets you keep sensitive data on your own device, avoiding the privacy risks associated with cloud services.

  4. Continuous Innovation: The Qwen team has suggested that this is only the beginning of applying scaled RL techniques to improve model capabilities. In the coming months, we can expect further advances in this direction.

Given the rapid progress we’ve seen in our local AI series, from basic DeepSeek to Open WebUI and now QwQ-32B, it’s exciting to imagine what the future holds. We may soon see even more advanced models that make enterprise-grade AI accessible to everyone, without significant compromises on privacy or autonomy.

Conclusions: Is QwQ-32B Worth Trying?

QwQ-32B represents a significant step forward in the field of locally runnable AI models. Thanks to its innovative two-phase Reinforcement Learning training, it offers reasoning capabilities that until recently were only available through expensive cloud services.

If you’re following this local AI series and have already set up DeepSeek and Open WebUI, QwQ-32B is the perfect addition to your arsenal. It’s particularly well suited for:

  • Developers looking for help with complex programming problems
  • Students and researchers who need help with mathematical reasoning
  • Professionals who want to analyze long documents while keeping their data private
  • AI enthusiasts who want to experiment with advanced reasoning capabilities without cloud dependencies

Sure, it requires relatively powerful hardware and can be demanding in terms of resources, but it represents an important milestone in making advanced AI accessible to everyone.

Useful resources:

Copiato