Digital Healthcare Platform Cuts Admin 40%
ICANIO built a digital healthcare platform integrating patient records, telemedicine, EHR, billing, wearables, and AI insights, reducing administrative workload by 40% and improvin
LLM FINE-TUNING . MLOPS . LEGAL AI
ICANIO built a multi tenant LLM fine tuning pipeline that trains a separate GPT-4.1-mini model per client organization on Azure OpenAI, with complete data isolation, automated experiment tracking, and a DeepEval quality gate that determines whether a newly trained model is safe to promote to production.
Multi tenant LLM fine tuning trains a separate model per client organization with complete data isolation, automated experiment tracking, and an automated quality gate that determines whether a newly trained model is safe to promote to production. ICANIO built this multi tenant LLM fine tuning pipeline on Azure OpenAI GPT-4.1-mini, validated end-to-end for a legal document classification use case covering MSA, Employment Agreement, and NDA document types.
Legal teams at enterprise organizations process high volumes of documents daily, and before any document can be routed, filed, or reviewed, it must be classified by type. ICANIO’s partner was facing exactly that gap: no per-client LLM fine-tuning pipeline to train a dedicated model on each client’s own documents, no data isolation between tenants to prevent cross-contamination of sensitive legal documents, no automated evaluation gate to stop a poorly performing model from reaching production, and no reproducible training lineage to audit which data produced which model.
ICANIO addressed this by designing a multi-tenant LLM fine-tuning pipeline on Azure OpenAI rather than a one-off training script. The objective was to give each tenant its own Blob Storage containers, its own Azure OpenAI deployment, and its own registered model, automate the full lifecycle from tenant onboarding through dataset preparation, fine-tuning, and experiment tracking, and gate every model behind an automated DeepEval evaluation before promotion to production.
The result was a validated, end-to-end multi-tenant LLM fine-tuning pipeline covering six pipeline stages, with confirmed tenant isolation, an automated quality gate that removes manual model review, and a complete lineage record connecting every dataset version to its training run, model version, and evaluation result.
“A shared model that blends two clients’ legal documents together is not a shortcut, it is a confidentiality breach waiting to be discovered.”
Legal teams at enterprise organizations process high volumes of documents daily, and every client organization operates differently, using distinct terminology, formats, and internal conventions. A single shared model risked blending or leaking data patterns across clients, which is unacceptable in legal and compliance contexts.
Each client needed its own model trained on its own documents, with no reuse of another client’s data, and no existing pipeline made that possible.
Shared storage or shared model deployments risked cross-contamination of sensitive legal documents between client organizations.
Without a quality gate, a poorly performing model could reach production and affect live document routing.
Without experiment tracking and dataset versioning, there was no way to audit which data produced which model.
Azure OpenAI supports continued training but provides no guidance on which strategy preserves model quality over time.
ICANIO designed and built a multi-tenant LLM fine-tuning pipeline on Azure OpenAI, validated through a proof of concept for legal document classification. The pipeline automates the full lifecycle from tenant onboarding through dataset preparation, model training, experiment tracking, and automated evaluation to model promotion. The solutions included:
01
A POST call registers the organization, provisions its multi-tenant Blob Storage containers, and issues a JWT for all subsequent authenticated calls.
02
Clients upload raw DOCX files through a three-step session flow, converted to OpenAI chat-completion JSONL format and versioned cumulatively with SHA256 content hashing, split 80/20 into training and validation sets.
03
The training service uploads the versioned JSONL to the Azure OpenAI Files API, submits a fine-tuning job with configurable hyperparameters, and polls until two consecutive succeeded results confirm completion.
04
Every training run opens an MLflow run tagged with tenant ID, dataset version hash, base model name, and hyperparameters, then registers the model with a dev alias on completion.
05
The evaluation service runs the held-out validation set through the fine-tuned deployment and scores outputs using DeepEval, promoting only models that clear the accuracy threshold.
06
A lineage record connects each dataset version to its training run, registered model version, and evaluation result, persisted to Blob Storage and queryable by run ID.
This multi-tenant LLM fine-tuning pipeline delivered outcomes across every dimension of the original ad hoc training problem, converting an unproven, undocumented process into a validated, governed, and fully auditable pipeline.
Data and model environment per tenant
Model review before promotion
Dataset-to-model audit trail
Fine-tuning strategy playbook
Pipeline stages validated end-to-end
Pipeline validation status
01
Continuous fine-tuning on new data alone carries the highest risk of catastrophic forgetting, silently overwriting prior classification patterns. Retraining from the base GPT-4.1-mini model each time is what eliminates weight drift and keeps every model version fully reproducible.
02
A shared model or shared storage path would have left cross-tenant contamination one misconfiguration away. Giving each tenant its own Blob Storage containers and its own Azure OpenAI deployment is what made data isolation a property of the architecture instead of a policy someone has to remember to follow.
03
Without a lineage record connecting dataset version to training run to model version to evaluation result, proving which data produced which model would have meant reconstructing history after the fact. Persisting that chain at training time is what turned a potential audit scramble into a simple lookup by run ID.
This engagement demonstrates ICANIO’s capability to design and deliver production-grade multi-tenant LLM fine-tuning pipelines for enterprise use cases. The proof of concept established a complete, repeatable architecture for per-tenant model training on Azure OpenAI, with full data isolation, automated evaluation gating, and end-to-end lineage tracking.
The pipeline validated here is directly transferable to any document-heavy industry where clients require models trained exclusively on their own data. Legal document classification served as the validation use case, but the same architecture applies to healthcare records classification, financial document processing, insurance claim triage, and any domain requiring per-client model customization. The Azure OpenAI fine-tuning strategy, MLflow experiment tracking, and DeepEval quality gate delivered through this engagement give ICANIO a reusable internal playbook for every future multi-tenant LLM fine-tuning engagement.
Multi-tenant LLM fine-tuning trains a separate model per client organization with complete data isolation, which matters because a single shared model risks blending or leaking data patterns across clients, unacceptable in legal and compliance contexts.
Legal document classification benefits because every client organization uses distinct terminology, formats, and internal conventions, so a model trained on one client's MSA, Employment Agreement, and NDA documents specifically outperforms a generic shared classifier.
Azure OpenAI fine-tuning trains GPT-4.1-mini on a tenant's own labeled documents through the Files API and Fine-Tuning Jobs API, producing a dedicated deployment instead of relying on prompt engineering against a shared base model.
The DeepEval quality gate runs the held-out validation set through the newly fine-tuned deployment and scores outputs using an Azure OpenAI judge, promoting a model to production only if it clears the accuracy threshold.
MLflow experiment tracking logs every training run's tenant ID, dataset version hash, base model, and hyperparameters, and registers the resulting model, giving a reproducible, auditable record of exactly which data produced which model version.
Retraining from the base GPT-4.1-mini model each time eliminates weight drift and the risk of catastrophic forgetting that continuous fine-tuning on new data alone can cause, keeping every model version fully reproducible.
From bold ideas to breakthrough execution – our case studies showcase how we transform business challenges into innovation-led success stories.
ICANIO built a digital healthcare platform integrating patient records, telemedicine, EHR, billing, wearables, and AI insights, reducing administrative workload by 40% and improvin
ICANIO built a healthcare backend engineering platform that delivered services 2x faster, reduced operational risk by 30%, and improved scalability, reliability, and integration.
ICANIO built an AI-powered healthcare record digitization platform that cut manual data entry by 60% and improved patient record accessibility 4x.
ICANIO built a multi-tenant LLM fine-tuning pipeline on Azure OpenAI with isolated client models, automated DeepEval quality gates, MLflow tracking, and complete training lineage.
Icanio builds AI medical image diagnostic systems using deep learning to detect abnormalities, accelerate clinical triage by 3x, and reduce image analysis time by 60%.
Icanio builds sanitation data management platforms that digitize surveys, centralize facility records, and enable faster reporting, improving data processing by 40%.
Quick Links
Careers
Internship
Contact Sales
© 2025
Icanio - All rights reserved.
