Fine-tuning Llama 3.1 for Docebo Expertise

Engineering Domain Expertise: Fine-Tuning Llama 3.1 for Docebo API Precision

General-purpose Large Language Models (LLMs) like GPT-4 or the base Llama 3.1 8B are remarkably capable at general reasoning, but they encounter significant degradation when confronted with highly specialized technical domains. When tasked with debugging a complex Docebo API workflow, interpreting xAPI statements, or configuring SCORM-compliant learning objects, these models frequently hallucinate syntax or revert to outdated architectural patterns.

The bottleneck in enterprise AI adoption for Learning Management Systems (LMS) is not a lack of raw reasoning capability; it is the lack of domain-specific precision. While Retrieval-Augmented Generation (RAG) can provide context by injecting relevant document chunks into the prompt, it cannot fundamentally teach a model the underlying structural logic and nomenclature of a closed ecosystem like Docebo. To move from a general assistant to a reliable “Domain Expert,” we must move beyond simple retrieval and toward targeted fine-tuning.

Our objective is clear: transform a base Llama 3.1 model into a specialized engine capable of navigating Docebo’s intricacies with high fidelity. Crucially, this does not require an H100 cluster. Through strategic data synthesis via Self-Instruct and the use of Parameter-Efficient Fine-Tuning (PEFT) techniques like QLoRA, we can achieve production-grade expertise on localized, sovereign infrastructure.

Data Engineering: From Raw Documentation to Instruction Sets

The primary failure point in fine-tuning is not the algorithm, but the dataset. Most engineers approach fine-tuning by scraping entire documentation repositories and feeding them into a trainer. This is a mistake. High-entropy noise—such as navigation menus, footer legal text, or irrelevant CSS snippets—in your training set leads to “model confusion,” where the model loses its ability to follow instructions in favor of merely mimicking prose.

The Ingestion Challenge and Schema Drift

Doceble documentation, like most enterprise SaaS manuals, exists in a fragmented state of HTML pages, PDF guides, and API references. To build a usable dataset, we first need to transform this unstructured data into machine-readable blocks. We utilize Unstructured.io to partition these documents, extracting meaningful text elements while preserving the hierarchical relationship between headers, tables (critical for API parameter definitions), and code blocks.

A significant complication in LMS engineering is “schema drift.” As Docebo updates its API versions, documentation often contains overlapping information from deprecated endpoints. Our pipeline includes a deduplication layer that uses semantic similarity to identify and prune redundant or conflicting instructions, ensuring the model does not learn two different ways to achieve the same webhook configuration.

Synthesizing Intelligence via Self-Instruct

Once we have cleaned text, we face a gap: we have “knowledge” but no “instructions.” To bridge this, we implement a “Teacher-Student” pipeline. We use a high-capacity model—specifically Llama 3.1 70B—to act as a synthetic data generator.

The process follows the Self-Instruct methodology:

1. Seed Tasks: We provide the Teacher model with a small set of manually curated, high-quality examples (e.g., “How do I trigger a webhook in Docebo after course completion?”).
2. Expansion: The Teacher model processes chunks of raw documentation and generates diverse instruction-response pairs. It is instructed to generate not just “What” questions, but “How” and “Why” scenarios involving complex logic (e.g., handling iCal synchronization errors or managing multi-tenant user permissions).
3. Negative Sampling: To increase robustness, we instruct the Teacher to generate “edge case” queries where the correct answer involves identifying a common error in an API payload.
4. Target Volume: We aim for a dataset of 5,000 to 10,000 high-quality pairs.

Quality over Quantity: The Contrarian View

Industry consensus often suggests that “more data is better.” In domain adaptation, the opposite is true. A dataset of 1,000 rigorously verified instruction pairs—where the response contains the exact API endpoint and correct JSON payload structure—will significantly outperform a 50,000-pair dataset riddled with noisy scrapes and hallucinated parameters. At J4SGON, we prioritize precision; if an instruction pair contains even one incorrect parameter name or a non-existent field, it is purged from the training set.

The Architecture of Efficiency: QLoRA and Hyperparameter Optimization

Training a full parameter model is economically and computationally non-viable for most specialized use cases. Instead, we utilize 4-bit Quantized LoRA (QLoRA). This allows us to freeze the base Llama 3.1 weights and only train a small set of adapter weights, drastically reducing the VRAM footprint while maintaining much of the original model’s intelligence.

The Mechanics of Quantization

The “Q” in QLoRA refers to our use of 4-bit NormalFloat (NF4) quantization. Unlike standard 4-bit integers, NF4 is specifically designed to handle the normally distributed weights of a neural network, preserving more information during the compression process. This allows us to run fine-tuning on consumer-grade or mid-range enterprise GPUs without the massive precision loss typically associated with aggressive quantization.

Tuning the Core Parameters

When configuring our LoRA adapters, two hyperparameters dictate the model’s plasticity: Rank ($r$) and Alpha ($\alpha$).

• Rank ($r$): This defines the dimensionality of the low-rank matrices used in the adapter layers. For Docebo expertise, we avoid extremely low ranks (e.g., $r=8$) which may fail to capture complex API relationships and nested JSON structures. We typically target $r=32$ or $r=64$.
• Alpha ($\alpha$): This acts as a scaling factor for the learned weights, essentially determining how much weight is given to the new adapter versus the base model. A common practice for stability is setting $\alpha = 2 \times r$.

The risk here is “catastrophic forgetting”—where the model becomes so focused on Docebo syntax that it loses its ability to reason in plain English or follow basic formatting instructions. We mitigate this by including a small percentage (5-10%) of general instruction data (such as the Alpaca dataset) in our fine-tuning mix, ensuring the model retains its fundamental linguistic capabilities and reasoning logic.

Context Window and Memory Constraints

Complex API calls often require analyzing large JSON payloads or lengthy error logs. To prevent truncation—which can lead to truncated JSON strings that break downstream parsing—we implement strategic chunking during the preprocessing stage. We ensure that no training sequence breaks an essential code block or a logical instruction flow. By using 4-bit quantization, we can perform this tuning on high-memory ARM64 nodes or consumer-grade GPUs (e.g., RTX 3090/4090), making specialized AI accessible without reliance on expensive cloud clusters.

Implementation Workflow: The Axolotl & W&B Pipeline

To ensure reproducibility and operational stability, we avoid custom training loops and instead use Axolotl. Axolotl allows us to define the entire training regime in a single, version-controlled YAML configuration, making it easy to audit our hyperparameter changes.

The Engineering Blueprint

Below is our standardized preprocessing approach using Unstructured for cleaning raw documentation and preparing it for the instruction-tuning format:

from unstructured.partition.html import partition_HTML
import json

def clean_doc_for_training(html_path):
    """
    Parses HTML documentation, extracts structural elements, 
    and cleans text for instructional use.
    """
    # Partition HTML into identifiable elements
    elements = partition_HTML(filename=html_path)
    
    cleaned_text_blocks = []
    for el in elements:
        # We focus on content-heavy categories to reduce noise
        if el.category in ["NarrativeText", "Title", "ListItem", "Table"]:
            # Strip unnecessary whitespace and newlines
            clean_el = " ".join(el.text.split())
            cleaned_text_blocks.append(clean_el)
            
    return "\n".join(cleaned_text_blocks)

# Example usage in our data pipeline
doc_content = clean_doc_for_training("docebo_api_ref.html")
# Following this, the text is passed to the Llama 3.1 70B 'Teacher' 
# for instruction pair generation.

Our production-ready .yml configuration for Axolotl focuses on stability and precision:

base_model: meta-perm/Llama-3.1-8B
load_in_4bit: true
adapter: qlora
sequence_len: 4096
dataset_prepared_path: last_run_prepared
datasets:
  - path: data/docebo_expert_v1.jsonl
    type: alpaca # Format: instruction, input, output
trainable_adapters:
  r: 32
  lora_alpha: 64
  target_modules:
    - q_proj
    - v_proj
    - k_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj
learning_rate: 0.0002
lr_scheduler: cosine
lr_warmup_steps: 100
optimizer: adamw_torch
batch_size: 4
gradient_accumulation_steps: 4
weight_decay: 0.01
epochs: 3

To monitor the health of this training process, we integrate Weights & Biases (W&B). We don’t just watch the loss curve; we monitor gradient norms and learning rate decay to detect early signs of divergence or vanishing gradients. This observability is critical when running long-running training jobs on shared infrastructure. For a broader look at AI infrastructure monitoring, see our guide on monitoring AI infrastructure with Grafana.

Once training is complete, we serve the final weights using vLLM. vLLM’s PagedAttention mechanism allows us to handle high-throughput inference by managing KV cache memory more efficiently, making it possible to power real-time LMS support bots or automated configuration auditors with minimal latency and high concurrency.

Verification: Implementing “LLM-as-a-Judge” (G-Eval)

The most significant error in AI engineering is using traditional NLP metrics like ROUGE or BLEU to evaluate technical accuracy. These metrics measure n-gram overlap—they care if the words are the same, not if the code works. If a model changes POST to GET in an API call, BLEU might still give it a high score, despite the response being functionally useless and potentially breaking integration scripts.

Instead, we implement G-Eval, an “LLM-as-a-Judge” methodology. We use a frozen, highly capable Llama 3.1 70B model to evaluate the outputs of our fine-tuned 8B model against a rigorous, multi-point rubric.

The Scoring Rubric

The Judge model is prompted with the original instruction and the fine-tuned model’s response. It is instructed to grade each response on a scale of 1–5 based on:

1. Technical Accuracy: Does the API endpoint, HTTP method, and JSON payload structure match the source documentation exactly?
2. Completeness: Are all required parameters (e.g., user_id, course_id, academy_id) included in the generated JSON?
3. Instruction Following: Did the model adhere to requested formatting constraints (e.g., “Respond only with a valid JSON object”)?
4. Syntactic Integrity: Is the provided code snippet or JSON payload syntactically valid and free of unclosed braces or quotes?

By quantifying accuracy through a secondary, more powerful LLM, we create an automated feedback loop that can scale alongside our dataset expansion without requiring manual human review for every new training epoch.

The Sovereign AI Perspective: From “Service” to “Infrastructure”

For organizations building a sovereign AI stack, fine-tuning open-weight models on self-hosted ARM64 infrastructure is not just a technical choice — it’s a strategic one. As we move toward deeper integration of AI into enterprise workflows, the risks associated with proprietary-only models become operational vulnerabilities. Relying exclusively on closed-source APIs introduces two critical challenges for LMS administrators: Model Drift and Data Leakage.

A model update in a closed ecosystem (such as a change in how OpenAI handles instruction following) can unexpectedly break your automated testing scripts or configuration auditors. Furthermore, sending sensitive user configuration data, proprietary training metadata, or internal organizational structures to an external API is a non-starter under modern privacy frameworks like GDPR.

At J4SGON, we advocate for the “Sovereign AI” approach. By fine-tuning open-weights models like Llama 3.1 on local, EU-based infrastructure (such as the ARM64 nodes we run in Valencia), we ensure:

• Data Sovereignty: All training data and inference payloads remain within your controlled environment. There is no external telemetry or third-party exposure of sensitive API keys or user schemas.
• Deterministic Performance: You control the versioning. Once a model is fine-tuned and verified via G-Eval, its behavior remains static. This allows for reliable CI/CD pipelines in your DevOps workflows.
• Infrastructure Independence: The ability to deploy on localized, high-efficiency hardware reduces reliance on volatile cloud pricing and mitigates risks associated with geographically distant data centers.

The transition from viewing AI as a “black-box service” to treating it as “managed infrastructure” is the hallmark of mature engineering. Fine-tuning Llama 3.1 for Docebo expertise is not just an experiment in machine learning; it is an exercise in building resilient, proprietary, and sovereign technical intelligence.

Next Steps for Implementation

1. Audit your data: Identify high-value documentation, API references, and error logs that are currently inaccessible to general LLMs.
2. Build the pipeline: Implement Unstructured.io and a 70B “Teacher” model to begin generating instruction sets from your raw sources.
3. Small-scale experimentation: Start with QLoRA on a single GPU to establish a baseline accuracy using the G-Eval rubric.
4. Deploy locally: Use vLLM to move your fine-tuned expertise into your production inference pipeline, ensuring high throughput and low latency for your LMS ecosystem.

Working on this yourself? J4SGON S.L. delivers Docebo Connect, HRIS, SSO and migration work for European organisations — see what a scoped engagement covers or describe your project and we will reply with a written scope.