[LoRA] support non-diffusers hidream loras

[sayakpaul]

What does this PR do?

Fixes: #11383

To check:

from diffusers import DiffusionPipeline
from transformers import PreTrainedTokenizerFast, LlamaForCausalLM
import torch 

repo_id = "HiDream-ai/HiDream-I1-Full"

tokenizer_4 = PreTrainedTokenizerFast.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
text_encoder_4 = LlamaForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    output_hidden_states=True,
    output_attentions=True,
    torch_dtype=torch.bfloat16,
)

pipeline = DiffusionPipeline.from_pretrained(
    repo_id, 
    tokenizer_4=tokenizer_4,
    text_encoder_4=text_encoder_4,
    torch_dtype=torch.bfloat16
).to("cuda")
pipeline.load_lora_weights("sayakpaul/different-lora-from-civitai", weight_name="486_hidream.safetensors")

Cc: @vladmandic

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

thanks!