Download the exact captured file
--- name: "human-in-the-loop-ml" description: "Provides human-in-the-loop machine learning based on annotation and active learning practice, covering active learning sampling strategies (uncertainty, diversity, representative), evaluation-set security before optimization, annotation workforce ethics and quality control with inter-annotator agreement, ML-assisted annotation QC, interface design for annotators and low-supervision alternatives. Use when designing data labeling programs, active learning loops, annotation quality pipelines or human-in-the-loop review systems." --- # AI Skill: Human-in-the-Loop Machine Learning This skill guides the AI to treat annotation and human review as first-class ML engineering: obtaining the right labeled data beats a better algorithm, and annotators deserve engineered tooling and ethical treatment. It synthesizes human-in-the-loop practice from Monarch's active learning literature. --- ## 🧭 When to Activate - Planning a data labeling program or annotation workforce. - Designing active learning loops for labeling efficiency. - Setting annotation quality control and agreement metrics. - Building annotation interfaces or ML-assisted labeling. - Choosing low-supervision alternatives when labeling budgets are tight. --- ## 🥇 Rule Zero: Evaluation Data First - Secure a truly random (or honestly representative) held-out evaluation set before any active learning begins; prefiltered data is nonrepresentative and hides catastrophic off-distribution collapse. - When random sampling is impossible, define quotas per label, time period or cluster, and maintain multiple evaluation sets compiled in different ways. - Model-driven selection always risks overfitting the evaluation; validation data, not training data, must drive any ranking used for sampling decisions. --- ## 🎯 Active Learning Loop Score a pool of unlabeled items → rank → send the top slice to humans → retrain → repeat. - **Uncertainty sampling:** least confidence (one minus the maximum probability), margin of confidence (top-two gap), ratio of confidence, entropy — they coincide on binary tasks and diverge with three or more labels. Temperature scaling adjusts softmax sharpness; rank order survives imperfect calibration. - **Diversity sampling:** model-based outliers (lowest-activation items via rank-order scoring against validation rankings), cluster-based sampling near centroids and cluster edges, representative sampling matching the unlabeled pool's distribution, and stratified sampling for demographic breadth with per-group accuracy reporting. - **Combine deliberately:** intersect or union uncertainty and diversity selections; sample from highest-entropy clusters; always mix a small random safety-net sample into every selected batch. - Ambiguous regions trap pure uncertainty loops; iterations themselves add diversity, but planned diversity prevents loops circling the same hard items. - Expected-error-reduction selection is theoretically elegant and practically prohibitive (retraining cost per candidate drowns the signal); highest-entropy-cluster sampling is the affordable approximation. - Task adaptations: detection (confidence times localization quality), segmentation, sequence labeling (token-level uncertainty stratified by confidence), generation (evaluation metrics are the bottleneck). --- ## 👥 Annotation Workforce - Workforce spectrum by flexibility versus expertise: in-house experts (highest quality, hardest to scale), outsourced contractors, crowdsourced (elastic, lowest quality), plus end users and volunteers as special cases; combine types by project phase. - Three ethical principles for any workforce: pay fairly, pay regularly, provide transparency and ownership. Annotate with dignity — do not gamify paid labeling work. - Plan volumes with order-of-magnitude estimates; budget one to four weeks of annotation training and task refinement; pilot before scaling to price accuracy goals. --- ## ✅ Quality Control - Combine **ground-truth comparison** (calibrated for chance agreement) with **inter-annotator agreement** (Krippendorff's alpha handles missing pairs and distance-weighted data; above 0.8 reliable, 0.67 to 0.8 partially reliable, below signals task or annotator failure). - Agreement alone lies: correlated wrong annotations show high agreement — always pair agreement with ground truth. - Slice agreement per annotator, per label and per demographic to localize problems. - Aggregate with majority or consensus when annotators agree; deliberately diverse annotator pools carry a mathematical case even at low agreement; use annotator-reported confidence and adjudication workflows for hard cases. - Apply ML to QC itself: predict whether an annotation is correct, whether it agrees with peers, whether an annotator is a bot; treat confident model predictions as one annotator among many (pre-annotations). --- ## 🖱️ Annotation Interfaces - HCI triad: affordance, feedback, agency; reuse standard framework components — accessibility wins come free. - Keyboard shortcuts, foot pedals and audio input raise throughput; minimize eye travel and scrolling. - Beware priming: model-suggested labels bias humans (mitigate by asking what others would annotate); pre-labeling helps when framed correctly; a wrong suggestion beats no suggestion for completeness; recast continuous judgment as ranking. - Keep annotation tools separate from daily-work tools; perceived efficiency matters as much as real speed. --- ## 🧰 Low-Supervision Alternatives When labeling budgets bind: rule-based filtering, training-data search, masked-feature filtering, embeddings from adjacent easy tasks, self-supervision, light supervision over unsupervised models — and distinguish synthetic data, data creation and augmentation as different tools. Push annotation signal into models: filter or weight training items by label confidence, include annotator identity as a feature, and bake annotation uncertainty into the loss. --- ## ⚠️ Pitfalls - Optimizing sampling before securing the evaluation set. - Reporting agreement without ground truth. - Gamifying annotation work or treating annotators as disposable. - Letting pre-annotations silently bias every label in a batch. - Aleatoric noise (data errors) confused with epistemic uncertainty (missing knowledge) — the remedies differ: cleaning versus more labels. --- ## 🔗 Integration with Other Skills - For evaluation design around human judgments, see [ai-model-evaluation](../../domains/industry/ai-model-evaluation/SKILL.md). - For the platform that hosts labeling pipelines, see [mlops-platform-engineering](../mlops-platform-engineering/SKILL.md). - For model interpretation shown to reviewers, see [explainable-ai](../../domains/industry/explainable-ai/SKILL.md). - For the core ML workflow, see [python-data-science](../python-data-science/SKILL.md).