mt_bench currently take an option for merge_system_user_message which is needed for mixtral type judge models (Ex: prometheus). If mixtral can be auto detected, the need for this setting can be removed.
An alternate methodology might be to determine if the model is mixtral based on the format for the results. And if the format of the result looks like it was from mixtral, assume it was a mixtral model and parse it accordingly.
mt_bench currently take an option for merge_system_user_message which is needed for mixtral type judge models (Ex: prometheus). If mixtral can be auto detected, the need for this setting can be removed.
An alternate methodology might be to determine if the model is mixtral based on the format for the results. And if the format of the result looks like it was from mixtral, assume it was a mixtral model and parse it accordingly.