Fill-Mask
Transformers
Safetensors
English
gte

feat(transformers): Add native Transformers config

#4

What does this PR do?

β†’ Sets model_type from new to gte and adds rope_parameters, so the checkpoint loads directly with AutoModel / AutoConfig in Transformers πŸ€— without trust_remote_code.
β†’ Transformers support is being added in https://github.com/huggingface/transformers/pull/48416
β†’ ⚠️ Please hold this until the Transformers PR above has landed πŸ€—

Compatibility

β†’ rope_scaling, rope_theta and auto_map are left untouched, so nothing changes for anyone on an older version.

Produced by?

β†’ rope_parameters is the standardised form of the existing rope_theta. This checkpoint leaves rope_scaling at null, so the standardised form is a plain default rope with the same base.
β†’ Outputs are reproduced against the original implementation on this checkpoint (max relative difference 3.6e-06), and the checkpoint is covered by the parity harness in the Transformers PR.

harshaljanjani changed pull request status to open
harshaljanjani changed pull request title from ⚠️ DO NOT MERGE: feat(transformers): Add native Transformers config to feat(transformers): Add native Transformers config
Alibaba-NLP org

Thanks @harshaljanjani for implementing native GTE support in transformers (huggingface/transformers#48416) and opening this PR, and thanks @tomaarsen for the clean native configuration and README updates in tomaarsen/gte-en-mlm-large-native!

Building on commit 6e6f6d2, I pushed a follow-up commit merging the clean native config and README updates from tomaarsen/gte-en-mlm-large-native:

  • Updated config.json: set architectures to ["GteForMaskedLM"], dtype: "bfloat16", tie_word_embeddings: true, and removed auto_map and legacy new-impl fields so transformers>=5.18.0 loads natively even when trust_remote_code=True is passed, and Hub metadata (transformersInfo, endpoints_compatible) is detected cleanly.
  • Updated README.md: added library_name: transformers, native transformers>=5.18.0 usage instructions, and credits linking to both @harshaljanjani and @tomaarsen (tomaarsen/gte-en-mlm-large-native).

Weights (model.safetensors) and tokenizer files remain unchanged. Keeping this PR open for final review before merging.

izhx changed pull request status to merged

Sign up or log in to comment