Hello, I am trying to train an encoder-decoder model that uses ModernBert as the encoder and GPT2 as the decoder. I had hoped that this would be straightforward enough using HF provided classes/trainers for Seq2Seq but have run into an error I have not been able to debug. Currently I do the followi…

Can you please try again? update modular_modernbert -- add inputs_embeds param to ModernBertModel huggingface:main ← jxmorris12:patch-1 opened 06:57PM - 20 Dec 24 UTC (UTC) …

Training ModernBert+GPT2

Beginners

John6666 January 6, 2025, 6:16am 2

It looks like the following will be quicker for questions about ModernBERT.

1 Like

Topic		Replies	Views
Warm-started encoder-decoder models (Bert2Gpt2 and Bert2Bert) Beginners	11	2597	June 9, 2024
EnocederDecoder training/prediction with two tokenizers Beginners	1	825	October 22, 2024
I get a "You have to specify either input_ids or inputs_embeds" error, but I do specify the input ids Beginners	6	21886	October 31, 2021
ModernBERT Pretraining using HuggingFace API Models	3	352	March 17, 2025
From Transformers Version v4.12.0 onwards, The example colab BERT2BERT is wrong. (Things to keep in mind when using from transformers import EncoderDecoderModel) 🤗Transformers	0	300	February 16, 2024

Training ModernBert+GPT2

Related topics