Add the CRATE (Coding RATE) backbone model

[BiEchi]

What does this PR do?

Implements the new model CRATE introduced in the paper White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada

[BiEchi]

Thanks a lot for reviewing this PR. As a reminder, the causal model is not yet supported, but I left the placeholder there.

[ArthurZucker]

Hey! Thanks a lot for contributing! I would recommend you to start with sharing the model on the hub following this! Will be easier to integrate and see if it has traction in the community!

[BiEchi]

Thanks for this pointer @ArthurZucker . As the code is very similar to RoBERTa, I'm using transformers-cli add-new-model-like for initializing the code, which leads to these dependencies that are not directly convertable to a custom model:

Would you like to suggest on how to solve this?

[ArthurZucker]

the add-new-model-like is made for an addition in transformers. Would be nice of use to have a add-new-custom-model that creates what's needed! Adding this to my todo list 😉
The tutorial otherwise should help on adding the model on the hub rather than on the transformers library for now! 🤗

[BiEchi]

Thanks a lot for offering to help on this @ArthurZucker ! Just to keep us on the same page, our code is already runnable if we directly use pip install -e . on our new library, so there shouldn't be challenge merging into the transformers library. We choose to directly develop the model by changing the library to avoid writing any other scripts except the examples like run_glue.py.
By the way, we've already release the weights on the Hub at Jackbai/crate-base, and we can directly load using model.from_pretrained(). If we have to develop a custom model first, we're pretty happy to proceed on this though (if you think this is necessary).

[ArthurZucker]

Okay! Thanks for explaining it looks like a good addition! Feel free to ping me if you need help integrating the model! 🤗 (on this PR and not as a custom model!) 🔥
fyi @amyeroberts

[BiEchi]

Hi @ArthurZucker , we're currently developing a sibling model (CRATE-GPT). The model proposed above is CRATE-BERT. Do we upload it here or we give a separate PR?

[ArthurZucker]

A separate PR is better, but as I stated before the best way to share it at first is to use custom models! 🤗 I would recommend you to make sure you upload safetensors checkpoints, and that you fill the model card to make sure people who discover it get what it is about!

[ArthurZucker]

Also you can easily push to hub if you use the PyTorchModelHubMixin class! 🤗 I can open a PR in your repo if you want?