File size: 34,499 Bytes
1017e80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "datasets>=4.0.0",
#     "transformers>=5.12",
#     "torch",
#     "accelerate",
#     "safetensors",
#     "scikit-learn",
#     "numpy",
#     "huggingface-hub",
# ]
# ///
"""
Fine-tune a text-classification encoder on a Hub dataset and push the trained model to the Hub.

Defaults to LiquidAI's LFM2.5-Encoder-350M — a bidirectional encoder converted from an LFM2
decoder backbone (blog: https://huggingface.co/blog/LiquidAI/lfm2-5-encoders). The 230M variant
beats ModernBERT-base on GLUE/SuperGLUE and both handle 8,192-token documents, so long inputs
(dataset cards, legal documents, support threads) fit without chunking. Any Hub encoder works
via --model: models with a standard sequence-classification head (BERT, ModernBERT, DeBERTa, …)
train through `AutoModelForSequenceClassification` and produce standard artifacts; models
without one (like the LFM2.5 encoders) get a generic mean-pooling + linear head that is pushed
as custom code, so the output still round-trips through
`AutoModelForSequenceClassification.from_pretrained(..., trust_remote_code=True)`.

Single-label vs multi-label is auto-detected from the label column:

- `ClassLabel` / string / int column  -> single-label (cross-entropy)
- `Sequence(ClassLabel)` / list of strings -> multi-label (BCE + per-label threshold tuning)

Run on HF Jobs (l4x1 is enough for 512-token contexts; the model is downloaded, trained,
evaluated, pushed, and reload-verified in one job):

    hf jobs uv run --flavor l4x1 --secrets HF_TOKEN \\
        https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py \\
        fancyzhx/ag_news username/my-news-classifier \\
        --max-samples 2000 --epochs 1

Multi-label example (go_emotions has a Sequence(ClassLabel) `labels` column):

    hf jobs uv run --flavor l4x1 --secrets HF_TOKEN \\
        https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py \\
        google-research-datasets/go_emotions username/my-emotion-classifier \\
        --label-column labels

Long documents: pair --max-length 8192 with --gradient-checkpointing and a small batch size
(--batch-size 2 --grad-accum 8) on a10g/a100 flavors.

Model: https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M

Smoke-tested 2026-07-28 on a10g-small (transformers 5.14.1, torch 2.13.0): single-label
(ag_news, acc 0.757 on a 2k/1-epoch smoke), multi-label (go_emotions, threshold tuning
lifting micro-F1 0.00->0.24 on a 2k/1-epoch smoke), and the standard-architecture path
(ModernBERT-base on ag_news, acc 0.871, vanilla artifact); pushed models pass the in-job
reload check and a fresh local CPU reload.
"""

import argparse
import importlib.util
import json
import logging
import os
import shutil
import sys
import tempfile
from datetime import datetime, timezone
from typing import Optional

import numpy as np
import torch
from datasets import ClassLabel, Dataset, load_dataset
from huggingface_hub import HfApi, ModelCard, hf_hub_download, list_repo_files, login
from sklearn.metrics import accuracy_score, f1_score
from transformers import (
    AutoConfig,
    AutoModel,
    AutoModelForSequenceClassification,
    AutoTokenizer,
    DataCollatorWithPadding,
    Trainer,
    TrainingArguments,
)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

DEFAULT_MODEL = "LiquidAI/LFM2.5-Encoder-350M"
SCRIPT_URL = "https://huggingface.co/datasets/uv-scripts/classification/raw/main/train-classifier.py"
WRAPPER_MODULE = "modeling_encoder_seq_cls"
WRAPPER_CLASS = "EncoderForSequenceClassification"

# Generic sequence-classification wrapper for encoders whose remote code ships no
# AutoModelForSequenceClassification (e.g. the LFM2.5 encoders expose only AutoModel +
# AutoModelForMaskedLM). This exact file is used for training AND copied into the pushed
# repo with an auto_map entry, so the training class and the reload class can never drift.
MODELING_FILE = '''"""Generic sequence classification head: AutoModel backbone + mean pooling + linear.

Auto-generated by the uv-scripts `train-classifier.py` recipe. Loaded via
`AutoModelForSequenceClassification.from_pretrained(repo, trust_remote_code=True)`;
the backbone class is resolved from this repo's own `auto_map`/code files.
"""

import torch
from torch import nn
from transformers import AutoModel, PreTrainedModel
from transformers.modeling_outputs import SequenceClassifierOutput


class EncoderForSequenceClassification(PreTrainedModel):
    base_model_prefix = "model"
    supports_gradient_checkpointing = True

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels
        self.model = AutoModel.from_config(config, trust_remote_code=True)
        dropout = getattr(config, "classifier_dropout", None)
        self.dropout = nn.Dropout(0.1 if dropout is None else dropout)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
        self.post_init()

    def forward(self, input_ids=None, attention_mask=None, labels=None, **kwargs):
        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)
        hidden = outputs.last_hidden_state
        if attention_mask is None:
            pooled = hidden.mean(dim=1)
        else:
            mask = attention_mask.unsqueeze(-1).to(hidden.dtype)
            pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)
        logits = self.classifier(self.dropout(pooled))
        loss = None
        if labels is not None:
            if self.config.problem_type == "multi_label_classification":
                loss = nn.functional.binary_cross_entropy_with_logits(
                    logits, labels.to(logits.dtype)
                )
            else:
                loss = nn.functional.cross_entropy(logits, labels.view(-1))
        return SequenceClassifierOutput(loss=loss, logits=logits)


# AutoModelForSequenceClassification.from_pretrained registers this class against the
# config class, and that requires config_class to be set (transformers v5 crashes on None).
try:
    __CONFIG_IMPORT__
    EncoderForSequenceClassification.config_class = __CONFIG_CLASS__
except ImportError:  # flat import during training; the trainer sets config_class itself
    pass
'''


def render_modeling_file(config) -> str:
    """Fill the wrapper template with the backbone's concrete config class."""
    config_cls = type(config)
    name = config_cls.__name__
    if config_cls.__module__.startswith("transformers."):
        import_stmt = f"from transformers import {name}"
    else:
        # remote-code config: its module file is copied into the pushed repo alongside
        # this wrapper, where the dynamic-module loader supports relative imports
        module_file = config_cls.__module__.split(".")[-1]
        import_stmt = f"from .{module_file} import {name}"
    return MODELING_FILE.replace("__CONFIG_IMPORT__", import_stmt).replace(
        "__CONFIG_CLASS__", name
    )


def check_cuda_availability() -> None:
    if not torch.cuda.is_available():
        logger.error("CUDA is not available. This script requires a GPU.")
        logger.error("Run on Hugging Face Jobs with: hf jobs uv run --flavor l4x1 ...")
        sys.exit(1)
    logger.info(f"CUDA is available. GPU: {torch.cuda.get_device_name()}")


# ---------------------------------------------------------------------------
# Labels
# ---------------------------------------------------------------------------


def detect_task(dataset: Dataset, label_column: str) -> tuple[str, list[str]]:
    """Return (problem_type, label_names) from the label column's feature/values.

    single_label_classification: ClassLabel, string, or int column.
    multi_label_classification:  Sequence(ClassLabel)/List(ClassLabel) or list-of-strings column.
    """
    feature = dataset.features[label_column]

    # Sequence / List / LargeList all expose .feature; ClassLabel and Value do not.
    inner = getattr(feature, "feature", None)

    if inner is not None:
        if isinstance(inner, ClassLabel):
            return "multi_label_classification", list(inner.names)
        values = {v for row in dataset[label_column] for v in (row or [])}
        if not values:
            logger.error(f"Label column '{label_column}' contains only empty lists.")
            sys.exit(1)
        return "multi_label_classification", sorted(str(v) for v in values)

    if isinstance(feature, ClassLabel):
        return "single_label_classification", list(feature.names)

    values = dataset.unique(label_column)
    if any(v is None for v in values):
        logger.error(f"Label column '{label_column}' contains nulls.")
        sys.exit(1)
    if all(isinstance(v, (int, np.integer)) for v in values):
        return "single_label_classification", [str(v) for v in sorted(values)]
    if all(isinstance(v, str) for v in values):
        return "single_label_classification", sorted(values)

    logger.error(
        f"Unsupported label column '{label_column}' "
        f"(feature: {feature}). Supported: ClassLabel, string, int, "
        f"Sequence(ClassLabel), or list-of-strings."
    )
    sys.exit(1)


def encode_labels(example, label_column, problem_type, label2id, num_labels, ints_are_indices):
    """ints_are_indices: True for ClassLabel columns, where raw ints already ARE the
    class indices. Plain int columns (e.g. values [10, 20]) map via label2id instead."""
    raw = example[label_column]
    if problem_type == "multi_label_classification":
        vec = [0.0] * num_labels
        for v in raw or []:
            if isinstance(v, str):
                idx = label2id[v]
            elif ints_are_indices:
                idx = int(v)
            else:
                idx = label2id[str(v)]
            vec[idx] = 1.0
        return {"encoded_labels": vec}
    if isinstance(raw, str):
        return {"encoded_labels": label2id[raw]}
    if ints_are_indices:
        return {"encoded_labels": int(raw)}
    return {"encoded_labels": label2id[str(raw)]}


# ---------------------------------------------------------------------------
# Model construction — ordered decision rule (order matters):
#   1. auto_map has AutoModelForSequenceClassification -> custom model ships its own head
#   2. auto_map exists without one (LFM2.5 encoders)   -> our mean-pooling wrapper; never
#      fall through to the built-in mapping: a future *causal* Lfm2ForSequenceClassification
#      in transformers would silently load a causal-mask head onto bidirectional weights
#   3. vanilla model -> standard AutoModelForSequenceClassification (standard artifact,
#      servable by vllm/classify-dataset.py)
# ---------------------------------------------------------------------------


def build_model(model_id, problem_type, label_names, work_dir):
    """Return (model, tokenizer, path) where path is 'custom-shipped'|'custom-wrapper'|'standard'."""
    num_labels = len(label_names)
    id2label = {i: name for i, name in enumerate(label_names)}
    label2id = {name: i for i, name in enumerate(label_names)}

    tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
    config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
    auto_map = getattr(config, "auto_map", None) or {}

    label_kwargs = dict(
        num_labels=num_labels,
        id2label=id2label,
        label2id=label2id,
        problem_type=problem_type,
    )

    if "AutoModelForSequenceClassification" in auto_map:
        logger.info("Model ships its own sequence-classification head (auto_map) — using it.")
        model = AutoModelForSequenceClassification.from_pretrained(
            model_id, trust_remote_code=True, **label_kwargs
        )
        return model, tokenizer, "custom-shipped"

    if auto_map:
        logger.info(
            "Custom-code model without a sequence-classification head — "
            "using the generic mean-pooling wrapper."
        )
        for key, value in label_kwargs.items():
            setattr(config, key, value)
        wrapper_path = os.path.join(work_dir, f"{WRAPPER_MODULE}.py")
        with open(wrapper_path, "w") as f:
            f.write(render_modeling_file(config))
        spec = importlib.util.spec_from_file_location(WRAPPER_MODULE, wrapper_path)
        module = importlib.util.module_from_spec(spec)
        sys.modules[WRAPPER_MODULE] = module
        spec.loader.exec_module(module)
        wrapper_cls = getattr(module, WRAPPER_CLASS)
        wrapper_cls.config_class = type(config)
        model = wrapper_cls(config)
        # Replace the randomly-initialised backbone with the pretrained weights.
        model.model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
        return model, tokenizer, "custom-wrapper"

    logger.info("Standard architecture — using AutoModelForSequenceClassification.")
    model = AutoModelForSequenceClassification.from_pretrained(model_id, **label_kwargs)
    return model, tokenizer, "standard"


# ---------------------------------------------------------------------------
# Metrics
# ---------------------------------------------------------------------------


def make_compute_metrics(problem_type):
    def compute(eval_pred):
        logits, labels = eval_pred.predictions, eval_pred.label_ids
        if problem_type == "multi_label_classification":
            probs = 1 / (1 + np.exp(-logits))
            preds = (probs >= 0.5).astype(int)
            return {
                "f1_micro": f1_score(labels, preds, average="micro", zero_division=0),
                "f1_macro": f1_score(labels, preds, average="macro", zero_division=0),
            }
        preds = logits.argmax(axis=-1)
        return {
            "accuracy": accuracy_score(labels, preds),
            "f1_macro": f1_score(labels, preds, average="macro", zero_division=0),
        }

    return compute


def tune_thresholds(logits: np.ndarray, labels: np.ndarray) -> list[float]:
    """Per-label threshold sweep (0.05–0.95) maximising per-label F1 on the eval set."""
    probs = 1 / (1 + np.exp(-logits))
    thresholds = []
    for i in range(labels.shape[1]):
        best_t, best_f1 = 0.5, -1.0
        for t in np.arange(0.05, 0.96, 0.05):
            f1 = f1_score(labels[:, i], (probs[:, i] >= t).astype(int), zero_division=0)
            if f1 > best_f1:
                best_t, best_f1 = round(float(t), 2), f1
        thresholds.append(best_t)
    return thresholds


# ---------------------------------------------------------------------------
# Push + verify
# ---------------------------------------------------------------------------


def assemble_output_repo(model, tokenizer, path_kind, model_id, out_dir, extra_config):
    """Fill out_dir with a self-contained, from_pretrained-able model."""
    from safetensors.torch import save_model

    tokenizer.save_pretrained(out_dir)

    if path_kind != "custom-wrapper":
        # Standard / custom-shipped heads: transformers handles the layout natively
        # (custom_object_save copies remote modules for custom-shipped models).
        for key, value in extra_config.items():
            setattr(model.config, key, value)
        model.save_pretrained(out_dir)
        return

    # Custom wrapper: copy the backbone's code files so the pushed repo is self-sufficient,
    # then write config + weights manually (save_pretrained on a dynamically-imported class
    # would try to copy this whole uv script as the modeling file).
    for fname in list_repo_files(model_id):
        if fname.endswith(".py"):
            local = hf_hub_download(model_id, fname)
            shutil.copy(local, os.path.join(out_dir, os.path.basename(fname)))
            logger.info(f"Copied backbone code file: {fname}")

    config = model.config
    for key, value in extra_config.items():
        setattr(config, key, value)
    backbone_auto_map = getattr(config, "auto_map", None) or {}
    config.auto_map = {
        **backbone_auto_map,
        "AutoModelForSequenceClassification": f"{WRAPPER_MODULE}.{WRAPPER_CLASS}",
    }
    config.architectures = [WRAPPER_CLASS]
    config.save_pretrained(out_dir)

    # Belt and braces: force plain module.Class refs in the saved JSON (transformers can
    # rewrite auto_map entries to 'origin-repo--module.Class', which would point reloads
    # at the origin repo instead of the pushed one).
    config_path = os.path.join(out_dir, "config.json")
    with open(config_path) as f:
        saved = json.load(f)
    saved["auto_map"] = {
        k: v.split("--", 1)[-1] for k, v in saved.get("auto_map", {}).items()
    }
    saved["auto_map"]["AutoModelForSequenceClassification"] = (
        f"{WRAPPER_MODULE}.{WRAPPER_CLASS}"
    )
    with open(config_path, "w") as f:
        json.dump(saved, f, indent=2, sort_keys=True)

    save_model(model, os.path.join(out_dir, "model.safetensors"))


def verify_reload(output_repo, eval_texts, reference_preds, problem_type, max_length, hf_token):
    """Reload the *pushed* repo fresh and check prediction agreement. Hard-fail on mismatch."""
    logger.info(f"RELOAD CHECK: loading {output_repo} back from the Hub...")
    tokenizer = AutoTokenizer.from_pretrained(output_repo, trust_remote_code=True, token=hf_token)
    model = AutoModelForSequenceClassification.from_pretrained(
        output_repo, trust_remote_code=True, token=hf_token
    )
    model.eval()
    enc = tokenizer(
        eval_texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
    )
    with torch.no_grad():
        logits = model(**enc).logits
    preds = logits.argmax(dim=-1).tolist()
    if preds != reference_preds:
        logger.error("RELOAD CHECK: FAILED — pushed model disagrees with trained model.")
        logger.error(f"  in-memory: {reference_preds}")
        logger.error(f"  reloaded:  {preds}")
        sys.exit(1)
    logger.info(f"RELOAD CHECK: OK ({len(preds)}/{len(preds)} predictions agree)")


# ---------------------------------------------------------------------------
# Card
# ---------------------------------------------------------------------------


def build_card(
    input_dataset, output_repo, model_id, problem_type, label_names, metrics,
    thresholds, path_kind, args_summary,
) -> str:
    on_jobs = os.environ.get("JOB_ID") is not None  # set by HF Jobs in-container
    hw = os.environ.get("ACCELERATOR") or ""  # e.g. "l4x1"; empty on CPU
    origin = (
        "Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
        + (f" (`{hw}`)" if hw else "")
    ) if on_jobs else "Generated"

    tags = ["uv-script", "text-classification"]
    if on_jobs:
        tags.append("hf-jobs")
    tag_lines = "\n".join(f"- {t}" for t in tags)

    metric_rows = "\n".join(f"| {k} | {v:.4f} |" for k, v in metrics.items())
    multi = problem_type == "multi_label_classification"

    label_list = ", ".join(f"`{name}`" for name in label_names[:30])
    if len(label_names) > 30:
        label_list += f", … ({len(label_names)} total)"

    if multi:
        snippet = f"""```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("{output_repo}", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("{output_repo}", trust_remote_code=True)

inputs = tokenizer("your text here", return_tensors="pt", truncation=True)
probs = torch.sigmoid(model(**inputs).logits)[0]
thresholds = torch.tensor(model.config.classifier_thresholds)  # tuned on validation
labels = [model.config.id2label[i] for i in (probs >= thresholds).nonzero().flatten().tolist()]
print(labels)
```"""
    else:
        snippet = f"""```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("{output_repo}", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("{output_repo}", trust_remote_code=True)

inputs = tokenizer("your text here", return_tensors="pt", truncation=True)
print(model.config.id2label[model(**inputs).logits.argmax().item()])
```"""

    serving_note = ""
    if path_kind == "custom-wrapper":
        serving_note = (
            "\n> [!NOTE]\n"
            "> This model uses a custom classification head (mean pooling over a backbone "
            "without a native sequence-classification class), so loading requires "
            "`trust_remote_code=True`. vLLM serving requires a standard architecture.\n"
        )

    return f"""---
tags:
{tag_lines}
base_model: {model_id}
datasets:
- {input_dataset}
pipeline_tag: text-classification
library_name: transformers
---

# {output_repo.split("/")[-1]}

[{model_id}](https://huggingface.co/{model_id}) fine-tuned for
{"multi-label" if multi else "single-label"} text classification on
[{input_dataset}](https://huggingface.co/datasets/{input_dataset}).

- **Labels ({len(label_names)})**: {label_list}
- **Date**: {datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")}
{serving_note}
## Evaluation

| Metric | Value |
|--------|-------|
{metric_rows}
{'''
Per-label decision thresholds tuned on the eval split are stored in
`config.classifier_thresholds`.

**Choosing an operating point**: the stored thresholds maximise per-label F1. For
precision-first use (e.g. auto-applying labels), act only on predictions well above
their threshold — sigmoid probabilities are a usable confidence signal, and filtering
to high-confidence predictions trades coverage for precision. Route the rest to review.
''' if multi and thresholds else ""}
## Usage

{snippet}

## Reproduction

{origin} with the [`train-classifier.py`]({SCRIPT_URL}) recipe from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:

```bash
hf jobs uv run --flavor {hw or "l4x1"} --secrets HF_TOKEN \\
    {SCRIPT_URL} \\
    {args_summary}
```
"""


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------


def main(
    input_dataset: str,
    output_repo: str,
    model_id: str = DEFAULT_MODEL,
    dataset_config: Optional[str] = None,
    text_column: str = "text",
    label_column: str = "label",
    train_split: str = "train",
    eval_split: Optional[str] = None,
    eval_fraction: float = 0.1,
    max_samples: Optional[int] = None,
    seed: int = 42,
    max_length: int = 512,
    epochs: int = 3,
    lr: float = 2e-5,
    batch_size: int = 16,
    grad_accum: int = 1,
    warmup_ratio: float = 0.05,
    gradient_checkpointing: bool = False,
    no_bf16: bool = False,
    private: bool = False,
    hf_token: Optional[str] = None,
) -> None:
    import transformers

    logger.info(f"transformers {transformers.__version__} | torch {torch.__version__}")
    check_cuda_availability()

    HF_TOKEN = hf_token or os.environ.get("HF_TOKEN")
    if HF_TOKEN:
        login(token=HF_TOKEN)

    # ----- data -----
    logger.info(f"Loading dataset: {input_dataset} (config={dataset_config})")
    ds = load_dataset(input_dataset, dataset_config)
    if train_split not in ds:
        logger.error(f"Split '{train_split}' not found. Available: {list(ds)}")
        sys.exit(1)
    train_ds = ds[train_split]

    if eval_split:
        if eval_split not in ds:
            logger.error(f"Split '{eval_split}' not found. Available: {list(ds)}")
            sys.exit(1)
        eval_ds = ds[eval_split]
    elif "validation" in ds:
        eval_ds, eval_split = ds["validation"], "validation"
    elif "test" in ds:
        eval_ds, eval_split = ds["test"], "test"
    else:
        logger.info(f"No eval split found — holding out {eval_fraction:.0%} of train.")
        parts = train_ds.train_test_split(test_size=eval_fraction, seed=seed)
        train_ds, eval_ds, eval_split = parts["train"], parts["test"], "held-out"

    if label_column not in train_ds.column_names and label_column == "label" and "labels" in train_ds.column_names:
        logger.info("Column 'label' not found; falling back to 'labels'.")
        label_column = "labels"
    for col in (text_column, label_column):
        if col not in train_ds.column_names:
            logger.error(f"Column '{col}' not found. Columns: {train_ds.column_names}")
            sys.exit(1)

    if max_samples:
        train_ds = train_ds.shuffle(seed=seed).select(range(min(max_samples, len(train_ds))))
        eval_ds = eval_ds.shuffle(seed=seed).select(range(min(max_samples, len(eval_ds))))

    problem_type, label_names = detect_task(train_ds, label_column)
    num_labels = len(label_names)
    label2id = {name: i for i, name in enumerate(label_names)}
    label_feature = train_ds.features[label_column]
    ints_are_indices = isinstance(label_feature, ClassLabel) or isinstance(
        getattr(label_feature, "feature", None), ClassLabel
    )
    logger.info(f"Task: {problem_type} | {num_labels} labels | "
                f"train={len(train_ds)} eval={len(eval_ds)} ({eval_split})")

    # ----- model -----
    work_dir = tempfile.mkdtemp(prefix="train-classifier-")
    out_dir = os.path.join(work_dir, "model")
    os.makedirs(out_dir, exist_ok=True)
    model, tokenizer, path_kind = build_model(model_id, problem_type, label_names, out_dir)
    if gradient_checkpointing:
        model.gradient_checkpointing_enable()

    # ----- tokenize -----
    def tokenize(batch):
        return tokenizer(
            [str(t) for t in batch[text_column]], truncation=True, max_length=max_length
        )

    keep = {"input_ids", "attention_mask", "labels"}

    def prepare(split):
        # Encode into a TEMP column, drop the original, then rename to "labels".
        # Writing straight into the original column name makes datasets cast the
        # encoded values back to the original schema (e.g. multi-hot floats ->
        # list-of-strings -> the collator crashes with "excessive nesting").
        split = split.map(
            lambda ex: encode_labels(
                ex, label_column, problem_type, label2id, num_labels, ints_are_indices
            ),
            remove_columns=[label_column],
        )
        split = split.rename_column("encoded_labels", "labels")
        split = split.map(tokenize, batched=True)
        return split.remove_columns([c for c in split.column_names if c not in keep])

    train_tok, eval_tok = prepare(train_ds), prepare(eval_ds)

    # ----- train -----
    bf16 = not no_bf16 and torch.cuda.is_bf16_supported()
    if not bf16:
        logger.warning("bf16 unavailable or disabled — training in fp32.")
    # save_strategy stays "no": Trainer checkpointing on the dynamically-imported wrapper
    # would trigger custom_object_save, which copies this whole uv script as modeling code.
    # The final save is manual (assemble_output_repo).
    training_args = TrainingArguments(
        output_dir=os.path.join(work_dir, "trainer"),
        num_train_epochs=epochs,
        learning_rate=lr,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size * 2,
        gradient_accumulation_steps=grad_accum,
        warmup_ratio=warmup_ratio,
        weight_decay=0.01,
        bf16=bf16,
        eval_strategy="epoch",
        save_strategy="no",
        logging_steps=10,
        seed=seed,
        report_to="none",
    )
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_tok,
        eval_dataset=eval_tok,
        data_collator=DataCollatorWithPadding(tokenizer),
        compute_metrics=make_compute_metrics(problem_type),
    )
    trainer.train()

    # ----- final eval (+ threshold tuning for multi-label) -----
    predictions = trainer.predict(eval_tok)
    logits, labels = predictions.predictions, predictions.label_ids
    metrics, thresholds = {}, None
    if problem_type == "multi_label_classification":
        probs = 1 / (1 + np.exp(-logits))
        preds_05 = (probs >= 0.5).astype(int)
        thresholds = tune_thresholds(logits, labels)
        preds_tuned = (probs >= np.array(thresholds)).astype(int)
        metrics = {
            "f1_micro @ 0.5": f1_score(labels, preds_05, average="micro", zero_division=0),
            "f1_macro @ 0.5": f1_score(labels, preds_05, average="macro", zero_division=0),
            "f1_micro @ tuned": f1_score(labels, preds_tuned, average="micro", zero_division=0),
            "f1_macro @ tuned": f1_score(labels, preds_tuned, average="macro", zero_division=0),
        }
    else:
        preds = logits.argmax(axis=-1)
        metrics = {
            "accuracy": accuracy_score(labels, preds),
            "f1_macro": f1_score(labels, preds, average="macro", zero_division=0),
        }
    for k, v in metrics.items():
        logger.info(f"eval {k}: {v:.4f}")

    # ----- push -----
    extra_config = {"problem_type": problem_type}
    if thresholds:
        extra_config["classifier_thresholds"] = thresholds

    logger.info(f"Assembling output repo in {out_dir}")
    model = model.to("cpu").float()
    assemble_output_repo(model, tokenizer, path_kind, model_id, out_dir, extra_config)

    api = HfApi(token=HF_TOKEN)
    api.create_repo(output_repo, repo_type="model", private=private, exist_ok=True)
    logger.info(f"Uploading to {output_repo}")
    api.upload_folder(folder_path=out_dir, repo_id=output_repo, repo_type="model")

    args_summary = f"{input_dataset} {output_repo}"
    if model_id != DEFAULT_MODEL:
        args_summary += f" --model {model_id}"
    if label_column != "label":
        args_summary += f" --label-column {label_column}"
    card = build_card(
        input_dataset, output_repo, model_id, problem_type, label_names,
        metrics, thresholds, path_kind, args_summary,
    )
    try:
        ModelCard(card).push_to_hub(output_repo, token=HF_TOKEN)
    except Exception as e:
        logger.warning(f"Could not push model card: {e}")

    # ----- verify the pushed artifact round-trips -----
    n_check = min(8, len(eval_ds))
    check_texts = [str(t) for t in eval_ds[text_column][:n_check]]
    model.eval()
    enc = tokenizer(
        check_texts, truncation=True, max_length=max_length, padding=True, return_tensors="pt"
    )
    with torch.no_grad():
        reference_preds = model(**enc).logits.argmax(dim=-1).tolist()
    verify_reload(output_repo, check_texts, reference_preds, problem_type, max_length, HF_TOKEN)

    logger.info("Done!")
    logger.info(f"Model: https://huggingface.co/{output_repo}")


if __name__ == "__main__":
    if len(sys.argv) == 1:
        print("Fine-tune a text-classification encoder (default: LFM2.5-Encoder-350M)")
        print("\nUsage:")
        print("  uv run train-classifier.py INPUT_DATASET OUTPUT_MODEL_REPO [options]")
        print("\nExamples:")
        print("  # single-label (ClassLabel column)")
        print("  uv run train-classifier.py fancyzhx/ag_news username/news-classifier")
        print("\n  # multi-label (list-of-labels column)")
        print("  uv run train-classifier.py google-research-datasets/go_emotions \\")
        print("      username/emotion-classifier --label-column labels")
        print("\nFor full help: uv run train-classifier.py --help")
        sys.exit(0)

    parser = argparse.ArgumentParser(
        description="Fine-tune a text-classification encoder on a Hub dataset and push to Hub",
    )
    parser.add_argument("input_dataset", help="Input dataset ID")
    parser.add_argument("output_repo", help="Output model repo ID (username/model-name)")
    parser.add_argument("--model", default=DEFAULT_MODEL, help=f"Base model (default: {DEFAULT_MODEL})")
    parser.add_argument("--dataset-config", help="Dataset config name")
    parser.add_argument("--text-column", default="text", help="Text column (default: text)")
    parser.add_argument("--label-column", default="label",
                        help="Label column (default: label, falls back to labels)")
    parser.add_argument("--train-split", default="train", help="Train split (default: train)")
    parser.add_argument("--eval-split",
                        help="Eval split (default: validation, then test, then a held-out fraction of train)")
    parser.add_argument("--eval-fraction", type=float, default=0.1,
                        help="Held-out fraction when no eval split exists (default: 0.1)")
    parser.add_argument("--max-samples", type=int, help="Cap train/eval examples (shuffled first)")
    parser.add_argument("--seed", type=int, default=42, help="Seed (default: 42)")
    parser.add_argument("--max-length", type=int, default=512,
                        help="Max sequence length (default: 512; LFM2.5 encoders support 8192)")
    parser.add_argument("--epochs", type=int, default=3, help="Epochs (default: 3)")
    parser.add_argument("--lr", type=float, default=2e-5, help="Learning rate (default: 2e-5)")
    parser.add_argument("--batch-size", type=int, default=16, help="Batch size (default: 16)")
    parser.add_argument("--grad-accum", type=int, default=1, help="Gradient accumulation (default: 1)")
    parser.add_argument("--warmup-ratio", type=float, default=0.05, help="Warmup ratio (default: 0.05)")
    parser.add_argument("--gradient-checkpointing", action="store_true",
                        help="Enable gradient checkpointing (for long contexts)")
    parser.add_argument("--no-bf16", action="store_true", help="Disable bf16 (train in fp32)")
    parser.add_argument("--private", action="store_true", help="Make output model repo private")
    parser.add_argument("--hf-token", help="HF token (or set HF_TOKEN)")
    args = parser.parse_args()

    main(
        input_dataset=args.input_dataset,
        output_repo=args.output_repo,
        model_id=args.model,
        dataset_config=args.dataset_config,
        text_column=args.text_column,
        label_column=args.label_column,
        train_split=args.train_split,
        eval_split=args.eval_split,
        eval_fraction=args.eval_fraction,
        max_samples=args.max_samples,
        seed=args.seed,
        max_length=args.max_length,
        epochs=args.epochs,
        lr=args.lr,
        batch_size=args.batch_size,
        grad_accum=args.grad_accum,
        warmup_ratio=args.warmup_ratio,
        gradient_checkpointing=args.gradient_checkpointing,
        no_bf16=args.no_bf16,
        private=args.private,
        hf_token=args.hf_token,
    )