lqdev☔

https://huggingface.co/blog/Pclanglais/common-corpus

We announce today the release of Common Corpus on HuggingFace:

  • Common Corpus is the largest public domain dataset released for training LLMs.
  • Common Corpus includes 500 billion words from a wide diversity of cultural heritage initiatives.
  • Common Corpus is multilingual and the largest corpus to date in English, French, Dutch, Spanish, German and Italian.
  • Common Corpus shows it is possible to train fully open LLMs on sources without copyright concerns.
Send me a message or webmention
Permalink: https://www.luisquintanilla.me/feed/common-corpus-llm-training-dataset/
Tags: #ai #data #llm #huggingface #nlp
Back to feed