←

A local LLM on a GPU server

A self-hosted, retrieval-grounded chat model built as the fallback brain for Hader: Qwen2.5 fine-tuned with QLoRA on real conversations, served by vLLM on a GPU box behind an OpenAI-compatible endpoint.

When the AI APIs go down, the bot keeps answering.

A self-hosted, retrieval-grounded chat model built as the fallback brain for Hader: Qwen2.5 fine-tuned with QLoRA on real conversations, served by vLLM on a GPU box behind an OpenAI-compatible endpoint.

Qwen2.5 · QLoRA · vLLM · GPU · RAG

  • 14BBase model, AWQ quantised
  • 24 GBGPU for the 14B fine-tune
  • 1URL to swap to switch over
  • ≤ 2%Hallucination gate

Model, data, serving and evals

The live bot runs on Groq Llama 70B, GPT-4o and Claude Sonnet. This model is built to behave like them, in isolation from the live bot, so it can take over when external APIs are unavailable.

  • Model choice: Qwen2.5 for Arabic, French, English and Arabizi
  • Behavioural QLoRA fine-tune with axolotl
  • Serving with vLLM, base model and LoRA adapter
  • An eval harness with ship gates graded against real reference replies

From corpus to endpoint

Data runs anywhere. Training and serving run on the GPU box.

  1. 1Export and scrub the corpus
  2. 2Build the fine-tune set
  3. 3QLoRA on the GPU box
  4. 4LoRA adapter
  5. 5vLLM, OpenAI-compatible
  6. 6Eval gates
  7. 7Swap the base URL

What it does

  1. 01

    Grounded, not memorised

    Retrieval feeds the same prompt, markers and catalogue the live bot uses, so the model learns behaviour, not facts.

  2. 02

    Fits one card

    The 14B QLoRA fits a 24 GB GPU at a sequence length of 8,192. A 32B run wants 40 to 48 GB.

  3. 03

    Drop-in endpoint

    vLLM serves an OpenAI-compatible API, so the live bot switches over with a base URL change and nothing else.

  4. 04

    Base or fine-tuned

    The same server exposes the base model as a baseline and the adapter by name, so every comparison is like for like.

  5. 05

    Ship gates

    Format, grounding with at most 2% invented prices, language match including French and Arabizi, scope lock and first-token latency on the box.

How it was built

  1. Training

    axolotl runs a QLoRA config on Qwen2.5 with the distilled corpus. Loss should fall smoothly; if it collapses fast, cut epochs or the LoRA rank. A low validation loss is not the target, the eval harness is.

  2. Serving

    Docker Compose or bare vLLM: Qwen2.5-14B-Instruct-AWQ, a 32K context, 90% GPU memory, with the LoRA module registered as hader.

  3. Scorers that do not lie

    Price grounding compared numbers in different formats, and language match only checked for Arabic script, so French customers answered in English passed. Both fixed and covered by tests.

  4. Isolation

    The folder never touches the live bot. Wiring it in is a separate decision made on the eval numbers.

Stack

Model
Qwen2.5-14B-Instruct, AWQ quantisation, LoRA adapter.
Training
axolotl with flash-attention on a CUDA GPU.
Serving
vLLM, OpenAI-compatible, Docker Compose.
Evals
Python harness, baseline scorecards, test suite.