✨ Summary

Large Language Models (LLMs) have powered the AI wave of the last 3–4 years. While most are closed-source, a vibrant ecosystem of open-weight and open-source models has emerged.

As a long-time AI user, I wanted to peek under the hood: how do GenAI models work, and what happens when you actually run them locally on your laptop?

In this blog, I’ll cover:

• How GenAI models are built ⚙️
• Why local inference matters 🚀
• My experiments with Qwen, Llama, and GPT-OSS on my Mac 💻

🔄 Hybrid Model Inference

Computing has gone through cycles: centralized → decentralized → hybrid. I believe AI inference is following the same path:

• Early computing → Mainframes (centralized)
• PCs/laptops → Decentralized
• Today → Cloud + Edge (hybrid)

👉 Most model inference currently happens in the cloud (huge infra needed).
👉 But smaller, specialized models now run on edge devices (laptops, even mobiles).

⚠️ Training won’t realistically move to the edge — it’s too compute-heavy and usually a one-time process.
✅ Inference is moving local — it’s repeated, latency-sensitive, and can benefit from privacy/cost savings.

💡 Use Cases of Running Models Locally

• ⚡ Reduce latency: Voice assistants, live translation, autonomous vehicles
• 💰 Reduce cost: Developer workflows, consumer electronics
• 🌍 Offline use: Remote fieldwork, disaster response
• 🔒 Privacy: Healthcare, enterprise security
• 🛠️ Customization: LoRA adapters, RAG integration

🏗️ How GenAI Models Are Created

LLMs typically follow the Transformer architecture and are built in two stages:

1. Pre-training: Learn general language patterns from massive datasets
2. Post-training (fine-tuning): Teach task-specific skills (chat, reasoning, coding, etc.)

Result → A model ready for inference.

🧩 What an AI Model Contains

• Weights: Learned numerical parameters (quantized models = smaller + faster)
• Tokenizer & Vocabulary: Convert text ↔ tokens
• Config: Architecture, layer counts, hidden sizes, etc.

🗂️ Common formats: Hugging Face / Transformers, GGUF, ONNX, Apple MLX.

🔁 How Generation Works (Simplified)

1. Tokenization → Text → tokens
2. Forward pass → Model processes tokens → probability distribution
3. Decoding → Pick next token (greedy, sampling, top-k/top-p, etc.)
4. Loop → Append token → repeat until done
5. Detokenize → Tokens → final response

📊 Comparing Models

Common Evaluation Axes

• Technical specs: Parameters, memory, speed, context length
• Quantitative benchmarks: MMLU (knowledge), ARC (science), HumanEval (coding)
• Qualitative: Creativity, domain knowledge, licensing

🔍 Open-Weights Model Comparison

I installed these 3 models in my mac, more details on it further down…

🔓 Open Weights vs Open Source models

Often confused! Here’s the difference 👇

👉 Takeaway: Open weights let you use and adapt, but open source lets you rebuild.

💻 Using Open Weight Models Locally

On my MacBook Pro (32 GB RAM) I installed models using Ollama:

• Qwen2.5:7B-Instruct
• Llama3:latest
• GPT-OSS:20B

ollama list
NAME                   ID              SIZE      MODIFIED    
qwen2.5:7b-instruct    845dbda0ea48    4.7 GB    3 weeks ago    
llama3:latest          365c0bd3c000    4.7 GB    3 weeks ago    
gpt-oss:20b            aa4295ac10c3    13 GB     3 weeks ago   

Install Ollama:

brew install ollama

Download a model:

ollama pull gpt-oss:20b

Run it:

ollama run llama3

…and you can start chatting!

🧪 My Experiments

⚖️ Use Case 1: Local LM Arena

Inspired by lmarena, I built a local version:

• User query → Sent to multiple models
• A “judge” model scores responses
• Models get ranked

Following is a screenshot of the application:

The 2 models compared here are qwen and llama and gpt-oss is grading the response.

💡 Example: Qwen scored 9/10, Llama scored 7/10, as judged by GPT-OSS.

🎛️ Use Case 2: Tuning Model Parameters

I tested how model parameters affect their responses:

👉 Lowering temperature/top-p/top-k + good prompts = fewer hallucinations.

I created an application where we can specify these model input parameters and check how the responses vary. I used another model to evaluate if the responses provided are inline with the model parameters.

I was able to experiment and get the parameter combinations for providing consistent response or for reducing hallucinations. 

Following is a screenshot of the application:

Following is the response evaluation output:

🛠️ Use Case 3: Modifying Base Models

Tried LoRA adapters → freeze base model + insert tiny trainable matrices.
⚠️ Didn’t fully succeed due to library issues, but worth exploring for cheap fine-tuning.

📖 Glossary (Quick Reference)

• Parameters: Learned weights/biases
• Tokens: Atomic input/output units
• Context length: Max tokens a model can process at once
• Embedding: Numeric vector for tokens/context
• Transformer: Model architecture with self-attention
• Pre-training: Large-scale language learning
• Fine-tuning: Specialization for tasks
• Quantization: Lower precision → smaller, faster models

🚀 Closing Thoughts

Local LLMs are moving from curiosity to practical tools. With tools like Ollama and LM Studio, you can:

• Experiment with models directly on your laptop 💻
• Balance privacy, latency, and cost 🌍
• Customize outputs for your own use cases 🛠️

And with ongoing advances in quantization and small yet powerful models, local inference is only going to get better.