Instructions to get ollama and merlinite working

[jjasghar]

A howto get ollama to host merlinite from huggingface

[lhawthorn]

@kelbrown20 Here is yet more documentation and I do not want to add unnecessarily to the sprawl, though @jjasghar's how to guide is valuable for new users. Do you have feedback on where we might either a) add this detail to existing docs or b) where we should link to this so as to make sure folks have the benefit of this knowledge.

[sroecker]

A howto get ollama to host merlinite from huggingface

You can also just ollama run sroecker/merlinite
I converted the model from HF and used the provided prompt template, which differs from the example here.

[jjasghar]

You can also just ollama run sroecker/merlinite

Neat! Yeah I figured someone would want to build it locally so they know exactly what they are getting. There's a chance you (or someone else with another fork) may have edited the Modelfile and made the SYSTEM look like this...

SYSTEM """
You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
"""

[sroecker]

I didn't adapt the system prompt, that's something you can easily do otherwise. Getting the chat template right is important though: https://ollama.com/sroecker/merlinite:latest/blobs/840577c5e55f This should correspond to the Merlinite 7B model card: prompt = f'<|system|>\n{sys_prompt}\n<|user|>\n{inputs}\n<|assistant|>\n'

…

On Tue, Apr 23, 2024 at 5:12 PM JJ Asghar ***@***.***> wrote: You can also just ollama run sroecker/merlinite Neat! Yeah I figured someone would want to build it locally so they know exactly what they are getting. There's a chance you (or someone else with another fork) may have edited the Modelfile and made the SYSTEM look like this... SYSTEM """You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.""" — Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYR3IZAYCXCP7C44SGJOTY6Z255AVCNFSM6AAAAABGTUYUL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSGY2DEOJXGI> . You are receiving this because you commented.Message ID: ***@***.***>

[jjasghar]

Oh, I'm not saying you did. but these instructions are designed for you to pull directly from the upstream, and not rely on another name space, and know exactly what you're getting.

[sroecker]

You could upload the Merlinite GGUF to Ollama as well and make that "upstream". It's like a container registry. However, make sure that the chat template is correct. Yours seems wrong, it's not the same as the one from the Merlinite model card on HF. Unless you converted and quantized a different model. JJ Asghar ***@***.***> schrieb am Di., 23. Apr. 2024, 22:16:

…

Oh, I'm not saying you did. but these instructions are designed for you to pull directly from the upstream, and not rely on another name space, and know exactly what you're getting. — Reply to this email directly, view it on GitHub <#198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYR3MYW3IJPHUKGTQMJ3DY626R7AVCNFSM6AAAAABGTUYUL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZTGM3DQOBVGU> . You are receiving this because you commented.Message ID: ***@***.***>

[russellb]

I think this should at least go in a sub-directory of random misc helpful topical guides. It doesn't seem like a top-level document.

[kelbrown20]

I think this should at least go in a sub-directory of random misc helpful topical guides. It doesn't seem like a top-level document.

I think it still works in this repo, but I agree with @russellb that it should maybe go in the docs directory rather then in the top level

[hickeyma]

Thanks @jjasghar for adding this doc. It will be useful for users.

2 things I suggest:

• I don't think naming of file in capitals is needed. Can it go lower case?
• As per @russellb in #198 (comment), can we move it to /docs/mics directory?

[github-actions]

This pull request has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[obuzek]

I wonder if it makes more sense to frame this as "running arbitrary models locally with InstructLab + ollama" instead of "running Merlinite"? With more explanation on e.g. how the Modelfile is developed?

Also I think to make it cohesive with the InstructLab project, a line at the very end explaining the --endpoint-url that allows you to hit the ollama model as part of your generate command would be helpful.

[jjasghar]

I had completely forgotten about this, and it's no longer accurate. Not to mention it's not Granite, which is what our defaults should be going forward. I'm going to close this.