Poor Benchmark Results (Needs Addressed)

[MarkSchmidty]

As seen in this popular spreadsheet by @lhl , StableLM-Alpha-7B currently scores below 5 year old 1GB models with 700M parameters and well below its architectural cousin GPT-J-6B which is only trained on 300B tokens.

This is a serious issue which needs to be addressed.

---

Edit:
@abacaj on twitter posted these 3B results:

[jon-tow]

We're well aware of this (I was one of the core devs of lm-eval - we perform downstream benchmarking the same way 😄). A few things are going on for why we believe this is happening, and hopefully, we can pin them down in our following write-up.
For the time being, you should find that modifying the contexts into dialog prompt format (e.g. Question: -> User: ) should improve scores.

[MarkSchmidty]

Okay, I made the issue title less alarming since you've chimed in.

Open communication about the issue and what is being done to address it would be appreciated by many. This thread/issue may be a good place to reach more technical users/devs who are keeping tabs.

[lhl]

I dropped a line to the lm@stability address mentioned in the announcement to ask about if there was anything I'm doing wrong w/ benchmarks, was curious evals weren't included w/ the model card even as an alpha release (or a note that low benchmark scores were a known issue at least), but will be following w/ interest.

Curious as a foundational model, what's going on w/ dialog prompt formatting? I grepped through tasks and question is used by the QA tasks, so would impact piqa, but how about hellaswag (completions) or winogrande (it's own format)?

[Ph0rk0z]

Not gonna lie, chatted with it and it's pretty bad. The longer context does work.

I've never gone OOM on a 7b before.

[MohamedAliRashad]

Any updates on this ?

[mallorbc]

@jon-tow Using that prompt format for the base model will help? Perhaps you are talking about the tuned model?