Allows MMLU to have the system_prompt provided to it#197
Conversation
Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>
|
I am currently running a e2e test on granite-8b-starter and still seeing failures (manually brought in patches) |
Confirmed my image had patches for both libraries and then proceeded to run mmlu eval |
danmcp
left a comment
There was a problem hiding this comment.
Changes look good, might be good to pass an example system_prompt from a unit test:
Line 65 in d8d6097
It wouldn't test much but at least to make sure the arg passing works.
The other place to potentially add an example would be in the test scripts:
That's less valuable since they aren't run anywhere. But could be a helpful example.
|
Hit with this same error |
|
I am happy to bring in patches and do a e2e test to ensure we are at a place that everything is good on granite-8b-starter |
Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>
RobotSail
left a comment
There was a problem hiding this comment.
Changes look good, might be good to pass an example system_prompt from a unit test:
Line 65 in d8d6097
I've updated the PR to include these changes, please take a look and let me know if there's anything I missed.
Regarding this comment, Tyler and I had a debug session where we found that adding the new prompt template causes MMLU to exceed the existing model's context (4096 tokens) when --few-shot=5 is used. This happens because lm_eval harness appears to truncate the prompt by default and just send the last 4096 tokens fitting into the context, which causes the system prompt to disappear and throw an error on certain models.
The resolution here is to use a lower number for --few-shot such as 1-3.
danmcp
left a comment
There was a problem hiding this comment.
@RobotSail Since the e2e testing relies on changes to the CLI (which have to merge after this change) to test this new code path, can you confirm when you've been able to validate everything functioning.
|
@danmcp Yes, we tested these changes last night with @relyt0925 in a production environment and it worked. Now I'm adding a test as well so we can have a more automated way of making sure this is happening since it seems to be important for certain models. |
|
Just for the sake of ensuring we don't allow a path of someone hitting a failure: few shots from my testing needed to be either 1 or 2: I saw when I used 3 failures on granite-8b-starter |
|
@relyt0925 Yes that's correct. Use |
In order to test the validity of our MMLU results or get information on prior runs, we need to be able to access the full set of results from the lm_eval.evaluator.simple_evaluate API. This commit provides that ability by adding a results attribute on the MMLUEvaluator class and storing the results there. Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>
The MMLU evaluator currently performs MMLU evaluation by calling out to the lm_eval harness. The lm-eval harness itself handles the evaluation by creating its own prompt clientside and then passing it to
/v1/completionsof whatever OpenAI-compatible API that it's expecting to receive it at the other end.Certain models however require to have their chat template used to be used during inference in order to get the best results. This PR adjusts the MMLU evaluator by allowing the system prompt to be provided to the model and then will enable the chat template mode to be used when it's present.
Signed-off-by: Oleg S 97077423+RobotSail@users.noreply.github.com