It is time to deprecate HumanEval! 🧑🏻‍💻 @BigCodeProject just released BigCodeBench, a new benchmark to evaluate LLMs on challenging and complex coding tasks focused on realistic, function-level tasks that require the use of diverse libraries and complex reasoning! 👀 🧩 Contains 1,140 tasks with 5.6 test cases each, covering 139 libraries in Python. 📊 Uses Pass@1 with greedy decoding and Elo rating for comprehensive evaluation. 🏆 Best model is GPT-4o 61.1%, followed by DeepSeek-Coder-V2. 🥈 Best open Model is DeepSeek-Coder-V2 with 59.7%, better than Claude 3 Opus or Gemini. 👥 Tasks are created in a three-stage process, including synthetic data generation and cross-validation by humans. 🧱 Evaluation framework and Docker images available for easy reproduction 🔜 Plans to extend to multilingualism. Blog: hf.co/blog/leaderboard-bigco… Leaderboard: huggingface.co/spaces/bigcod… Code: github.com/bigcode-project/b…

Self-Instruct for CodeLLMs! 👀 @BigCodeProject released a new StarCoder2-Instruct, the first entirely self-aligned code LLM trained with a transparent and permissive pipeline. 🧑🏻‍💻 It used itself to generate thousands of instruction-response pairs, which were then used to fine-tune—achieving 72.6 on HumanEval without relying on human annotations. 🤯 Implementation 1️⃣ Collect Seed Code Snippets, e.g., functions with docstrings. 2️⃣ Apply type checking, decontamination (benchmarks), Quality Filtering & Near-Deduplication 3️⃣ Employ in-context learning to self-generate coding tasks from these snippets. 4️⃣ For each instruction, generate answers and tests using in-context learning. 5️⃣ Execute these tests in a sandbox environment and select responses that pass for training. 6️⃣ Create a Training Dataset with the validated responses 7️⃣ Fine-Tune StarCoder2-15B on the generated self-instruct dataset Insights 🧮 15B parameter version with 8192 context 🔓 Fully open-source datasets and pipeline for distillation 📝 Fully self-aligned without human annotation 🏆 Outperforms CodeLlama-70B-Instruct (72.0) and GPT-4 (march) on HumanEval 🥇 Outperforming other open Models like Grok-1, Command-R+, and DBRX, and closely matching Snowflake Arctic 480B and Mixtral-8x22B-Instruct