Key Highlights

• Advanced Bilingual Capability and Reasoning: Built on Yi-34B, this model excels in bilingual tasks and complex reasoning.
• Innovative Training and Unprecedented Context Length: Trained with a 200K context length on the Capybara dataset, it offers deep contextual understanding and precise responses.
• Versatility and Multi-Modal Capabilities: Handles multi-turn dialogues and integrates multimodal processing via Obsidian, achieving performance akin to larger 7B models.
• Data Integrity and Practical Applications: Ensures dataset integrity through rigorous checks and supports diverse applications like advanced language understanding, sophisticated chatbots, and multimodal analysis.
• Other LLMs by NousResearch: Nous-Hermes-2-Mixtral-8x7B-DPO, nous-hermes-llama2–13b and hermes-2-pro-llama-3–8b on Novita AI.

Introduction

Welcome to the world of the Nous Capybara 34B Model, a groundbreaking AI model that's not just bilingual but also excels in comprehending and reasoning. 

In this blog, we'll delve into the intricacies of this model, trained on the formidable Yi-34B with an unprecedented 200K context length. We'll explore its technical details, unique features, its practical applications and other models from NousResearch. So, let's embark on this journey to discover the potential of the Nous Capybara 34B Model.

Exploring the Nous Capybara 34B Model

Nous Capybara 34B Model is trained on the Yi-34B model with 200K context length, for 3 epochs on the Capybara dataset.

Yi-34B

The Yi-34B modelby 01.AI is specifically designed as a bilingual language model, trained on an extensive 3TB multilingual corpus, which positions it as one of the most formidable LLMs globally.

The Yi-34B excels in various language tasks such as understanding, commonsense reasoning, and reading comprehension. Its capabilities are evidenced by its impressive performance on the AlpacaEval Leaderboard, where the Yi-34B-Chat model secured the second place, only after GPT-4 Turbo, and outperformed other leading LLMs like GPT-4 and Mixtral. Additionally, the Yi-34B model has been recognized for its top rankings among all open-source models in both English and Chinese across different benchmarks, as noted on the Hugging Face Open LLM Leaderboard and C-Eval.

First of Its Kind

• First 34B Nous Model: This is the pioneering model in the Nous series, marking a significant milestone in AI development.
• First 200K Context Length Model: It sets a new standard with its ability to process extensive context, offering deeper understanding and more nuanced responses.

Training and Dataset

The majority of the tokens in the Capybara model are the result of new synthesis, primarily from datasets such as Puffin and Dove. However, it's important to acknowledge the role of single-turn datasets that served as the foundation, or 'seeds,' for creating the multi-turn conversations within the Amplify-Instruct synthesis process. The green datasets listed are sources from which we selected seeds for the synthesis in this project, while the blue datasets represent curated collections that were already in place before the development of Capybara.

Development Team and Acknowledgements

• Leadership: Led by Luigi D. (LDJ) and supported by J-Supha and Jeffrey Q., the team's expertise is evident in the model's capabilities.
• Sponsorship: Grateful for the support from A16Z and Yield Protocol, which facilitated the research and development process.

Key Features of Nous Capybara 34B

Yi-34B Base Model

Built on a strong foundation, capable of handling extensive context and complex tasks.

Multi-Turn Conversations

Over 60% of the dataset focuses on multi-turn dialogues, a significant advantage over models trained for single-turn interactions.

Advanced Summaries

Trained to effectively summarize complex topics and studies, showcasing its advanced reasoning capabilities.

Historical Recall

Capable of recalling information up to late 2022 without internet access, demonstrating its extensive knowledge base.

Multimodality

The Nous Capybara 34B model, through its multimodal extension Obsidian-3B-V0.5, introduces a groundbreaking capability to process and understand both text and visual data. Built on the strong foundation of the Capybara-3B-V1.9 and leveraging the StableLM-3B-4e1t, Obsidian stands out as the world's smallest multi-modal large language model (LLM), offering state-of-the-art performance that rivals some 7B models.

Ensuring Integrity of Capybara Dataset

The developers undertook some data contamination effects concerning popular benchmarks. Here's a summary of the steps they took:

Contamination Check

They performed a thorough check to ensure that the Capybara dataset did not contain any contamination from other popular datasets.

Minhash Technique

They utilized the minhash technique to compare their dataset with other benchmarks. This technique helps in identifying similarities between datasets.

Similarity Levels

The developers checked for similarity matches at various levels, including 100%, 99%, 98%, and 97%. This comprehensive approach ensured that even close matches were identified and addressed.

Benchmarks Checked

They specifically checked against several benchmarks such as HumanEval, AGIEval, TruthfulQA, MMLU, and GPT4All to ensure that their dataset did not contain any data from these sources.

No Exact Matches

The efforts resulted in finding no exact matches or even close matches down to the 97% similarity level, indicating that the Capybara dataset is free from contamination.

Practical Applications of Nous Capybara 34B For Developers

Given the Nous Capybara 34B model's strengths, here are some of the most pertinent applications for developers:

Advanced Language Understanding

Utilize the model's bilingual capabilities for developing applications that require a deep understanding of English and Chinese, such as multilingual search engines, translation services, and cross-cultural content analysis tools.

Sophisticated Chatbots

Implement the model in customer service chatbots that can handle complex queries and maintain context over multi-turn conversations, providing a more human-like interaction.

Automated Content Generation

Apply the model's ability to generate advanced summaries and complex content to automate report writing, article spinning, or social media content creation.

Education and Learning

Leverage the model's comprehension skills to develop adaptive learning platforms that offer personalized educational content and interactive Q&A sessions.

Multimodal Analysis

With the Obsidian extension, create applications that analyze and interpret both visual and textual data, useful for image tagging, description, and retrieval systems in e-commerce or media management.

Data-Driven Insights

Use the model's capacity for complex summaries to develop business intelligence tools that analyze large datasets and generate actionable insights for decision-making.

Semantic Search

Integrate the model into search applications to enhance search results by understanding the semantic context of queries, improving accuracy and relevance.

Knowledge Base Construction

Use the model to build and maintain a dynamic knowledge base that continuously updates with new information from various sources.

Future Developments of Nous Capybara 34B

Expanding Sizes

With current versions at 3B, 7B, and 34B, plans for 13B, 70B, and potentially a 1B model based on phi-1.5 or Tiny Llama are underway.

Upcoming Benchmarks

The release of benchmarks is coming soon to evaluate the model's performance against industry standards.

What Are Other Models Developed by NousResearch

Novita AI offers developers a range of LLM API options, featuring models developed by NousResearch. Novita AI LLM API includes capabilities for hyperparameter adjustments and customizable system prompts, tailored to meet individual needs.

Nous-Hermes-2-Mixtral-8x7B-DPO on Novita AI

Nous Hermes 2 Mixtral 8x7B DPO is the new flagship Nous Research model trained over the Mixtral 8x7B MoE LLM. The model was trained on over 1,000,000 entries of primarily GPT-4 generated data, as well as other high quality data from open datasets across the AI landscape, achieving state of the art performance on a variety of tasks.

nousresearch/nous-hermes-llama2–13b on Novita AI
Nous-Hermes-Llama2–13b is a state-of-the-art language model fine-tuned on over 300,000 instructions. This model was fine-tuned by Nous Research, with Teknium and Emozilla leading the fine tuning process and dataset curation, Redmond AI sponsoring the compute, and several other contributors.

nousresearch/hermes-2-pro-llama-3–8b on Novita AI
Hermes 2 Pro is an upgraded, retrained version of Nous Hermes 2, consisting of an updated and cleaned version of the OpenHermes 2.5 Dataset, as well as a newly introduced Function Calling and JSON Mode dataset developed in-house.

Conclusion

Beginning with its foundational training on the Yi-34B model and Capybara dataset, Nous Capybara 34B stands out for its ability to handle extensive context and complex tasks across various domains. Key features such as multi-turn conversation handling, advanced summarization capabilities, and multimodal processing with the Obsidian extension underscore its versatility and sophistication. Furthermore, rigorous efforts to ensure dataset integrity through contamination checks have solidified its reliability for practical applications.

Looking ahead, the future developments of the Nous Capybara 34B model promise even greater capabilities with expanded sizes and upcoming benchmarks to validate its performance against industry standards. As part of Novita AI's offerings, including other models developed by NousResearch like Nous Hermes 2 Mixtral 8x7B DPO and Nous-Hermes-Llama2–13b, this model represents a cornerstone in the evolution of AI-driven solutions.

Originally published at Novita AI

Novita AI is the all-in-one cloud platform that empowers your AI ambitions. With seamlessly integrated APIs, serverless computing, and GPU acceleration, we provide the cost-effective tools you need to rapidly build and scale your AI-driven business. Eliminate infrastructure headaches and get started for free - Novita AI makes your AI dreams a reality.