For years, AI progress has been associated with larger and larger models. However, in many production environments, massive language models introduce serious engineering challenges: high latency, expensive infrastructure, and sensitive data being sent to external APIs.

In practice, many NLP tasks do not require a 70B-parameter model.

Small Language Models (SLMs), typically in the 1B–7B parameter range, are becoming a powerful alternative. When combined with quantization techniques and efficient Python tooling, these models can run on modest hardware while still delivering strong performance for specialised tasks.

This talk explores how Python developers can design privacy-first AI systems using compact models and local infrastructure.

Using the Python ecosystem, including Hugging Face Transformers, bitsandbytes quantization, and lightweight retrieval pipelines, we will walk through how to fine-tune and deploy an SLM for domain-specific tasks. The talk will also demonstrate a local-first Retrieval-Augmented Generation (RAG) pipeline, where sensitive documents remain entirely within an organisation’s infrastructure.

We will compare compact models against larger hosted models for targeted NLP tasks and discuss the trade-offs between model size, latency, cost, and deployment complexity.

The goal is not to replace large models entirely, but to show when smaller models are the more practical engineering choice.

Structure of the talk:
• Why model size is not always the best metric for capability
• Running SLMs locally with Python and quantization techniques
• Building a local-first RAG pipeline for private document retrieval
• Benchmarking accuracy, latency, and infrastructure cost
• When to choose small models vs large hosted models

By the end of the talk, attendees will understand how to build efficient, privacy-preserving AI systems using the Python ecosystem without relying entirely on large cloud-hosted models.