👋 Hi, everyone!
We are ByteDance Seed team.

You can get to know us better through the following channels👇

🚀 Magentless: Agentless for multi-swe-bench

This repo provides Magentless, which is based on agentless framework and compatible with multi-swe-bench. Magentless supports C++, C, Java, Go, Rust, Typescript and Javascript. For python language, please use the original agentless.

📊 Evaluation

Setup

git clone https://github.com/multi-swe-bench/MagentLess.git
cd Magentless

conda create -n magentless python=3.11
conda activate magentless
pip install -r requirements.txt
export PYTHONPATH=$PYTHONPATH:$(pwd)

Data Preparation

Clone multi-swe-bench dataset into './data'

git lfs install
git clone https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench data

The data directory looks like

data
|---c
|   |---facebook__zstd_dataset.jsonl
|   |---jqlang__jq_dataset.jsonl
|   |---...
|---cpp
|---go
|---java
|---js
|---python
|---rust
|---ts

Repo cloning

Please clone all the repos into './repo'.

cd Magentless
mkdir repo
cd repo
git clone https://github.com/dubbo/dubbo.git
...

Patch generation

First create './script/api_key.sh'.

export OPENAI_API_KEY=
export OPENAI_BASE_URL=
export OPENAI_MODEL=
export OPENAI_EMBED_URL=

OPENAI_EMBED_URL is used for embedding retrieval. The default model is 'text-embedding-3-large', which can be modified in agentless/fl/Index.py.

Then modify './script/run.sh'.

export FOLDER_NAME= # all results are in results/FOLDER_NAME
export SWEBENCH_LANG= # {java, javascript, typescript, c, cpp, go, rust}
export PROJECT_FILE_LOC= # See below
export DATASET=local_json # Do not modify

You can cache the repo structure in PROJECT_FILE_LOC for speedup, please refer to script/generate_structure.py

Finally run the run.sh

cd Magentless
bash ./script/run.sh

The prediction file and all the logs will be in results/FOLDER_NAME

Evaluation

Please refer to multi-swe-bench repo.

📜 License

This project is licensed under Apache License 2.0. See the LICENSE flie for details.

📖 Citation

If you find XXX useful for your research and applications, feel free to give us a star ⭐ or cite us using:

@article{zan2024swe,
  title={Swe-bench-java: A github issue resolving benchmark for java},
  author={Zan, Daoguang and Huang, Zhirong and Yu, Ailun and Lin, Shaoxin and Shi, Yifan and Liu, Wei and Chen, Dong and Qi, Zongshuai and Yu, Hao and Yu, Lei and others},
  journal={arXiv preprint arXiv:2408.14354},
  year={2024}
}

🏢 About ByteDance Seed Team

Founded in 2023, ByteDance Seed Team is dedicated to crafting the industry's most advanced AI foundation models. The team aspires to become a world-class research team and make significant contributions to the advancement of science and society.