Right now the inferencing done during MMLU is done within the lm-eval-harness library. Lm-eval-harness has the ability to inference with openai-api compatible servers similar to how models are being served via ilab serve:
https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#model-apis-and-inference-servers
A fix for this issue would involve serving a model on an endpoint with either vLLM or llama-cpp and being able to pass that endpoint into MMLUEvaluator.run() or the MMLUEvaluator class to run MMLU tests.
Right now the inferencing done during MMLU is done within the lm-eval-harness library. Lm-eval-harness has the ability to inference with openai-api compatible servers similar to how models are being served via
ilab serve:https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#model-apis-and-inference-servers
A fix for this issue would involve serving a model on an endpoint with either vLLM or llama-cpp and being able to pass that endpoint into
MMLUEvaluator.run()or theMMLUEvaluatorclass to run MMLU tests.