.unload_ip_adapter() return error for StableDiffusionControlNetInpaintPipeline

[wilfrediscoming]

Describe the bug

After using .unload_ip_adapter() for StableDiffusionControlNetInpaintPipeline, it returns an error.

Reproduction

inpaint = StableDiffusionControlNetInpaintPipeline.from_pretrained(
          "runwayml/stable-diffusion-inpainting",
          controlnet = controlnet,
          variant = "fp16",
          vae = vae,
          torch_dtype = torch.float16,
          load_safety_checker = False
          ).to("cuda")

# IP adapter
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter",
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
).to("cuda")

inpaint.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")

#Generate Image

inpaint.unload_ip_adapter()

prompt = "on a platform, sky background"
neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"

images = inpaint(
    prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
    negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",
    num_inference_steps=25,
    image=image_blank,
    mask_image=image_mask,
    control_image=depth_image,
    # ip_adapter_image=image_ref,
    strength=1.0,
    controlnet_conditioning_scale=0.9,
    num_images_per_prompt=4,
    # clip_skip=1,
)

idx = 0

for each in images.images:
    each.save('/content/semi-final.jpg')

    image_product = Image.open(path)
    image_product = scale(image_product,scale_factor)
    each.paste(image_product, (x,y), image_product)

    each.save(f'/content/final{idx}.jpg')
    idx += 1

Logs

100%
 25/25 [00:23<00:00,  1.05it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-70-0fcf450ea7c5>](https://localhost:8080/#) in <cell line: 10>()
      8 neg_prompt = "3d render, cartoon, doll, anime, artwork, fake, painting, illustration"
      9 
---> 10 images = inpaint(
     11     prompt=prompt+ ", product photography, UHD, 8k, F2.8, RAW Photo, ultra detailed",
     12     negative_prompt=neg_prompt + ",<BadDream>, (<UnrealisticDream:1.2>)",

2 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py](https://localhost:8080/#) in run_safety_checker(self, image, device, dtype)
    663             else:
    664                 feature_extractor_input = self.image_processor.numpy_to_pil(image)
--> 665             safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device)
    666             image, has_nsfw_concept = self.safety_checker(
    667                 images=image, clip_input=safety_checker_input.pixel_values.to(dtype)

TypeError: 'NoneType' object is not callable

### System Info

- `diffusers` version: 0.26.3
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Huggingface_hub version: 0.20.3
- Transformers version: 4.38.1
- Accelerate version: 0.27.2
- xFormers version: not installed
- Using GPU in script?: Yes, T4
- Using distributed or parallel set-up in script?: No

### Who can help?

@yiyixuxu @DN6

[sayakpaul]

See: #7218.

Cc: @fabiorigano

[fabiorigano]

oh thank you Sayak, I didn't see there was also the open issue.
I tested your code @wilfrediscoming, here it is the pipeline before unloading the IP Adapter:

StableDiffusionControlNetInpaintPipeline {
  "_class_name": "StableDiffusionControlNetInpaintPipeline",
  "_diffusers_version": "0.27.0.dev0",
  "_name_or_path": "runwayml/stable-diffusion-inpainting",
  "controlnet": [
    "diffusers",
    "ControlNetModel"
  ],
  "feature_extractor": [
    "transformers",
    "CLIPImageProcessor"
  ],
  "image_encoder": [
    "transformers",
    "CLIPVisionModelWithProjection"
  ],
  "requires_safety_checker": true,
  "safety_checker": [
    "stable_diffusion",
    "StableDiffusionSafetyChecker"
  ],
  "scheduler": [
    "diffusers",
    "DDIMScheduler"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}

and after unloading:

StableDiffusionControlNetInpaintPipeline {
  "_class_name": "StableDiffusionControlNetInpaintPipeline",
  "_diffusers_version": "0.27.0.dev0",
  "_name_or_path": "runwayml/stable-diffusion-inpainting",
  "controlnet": [
    "diffusers",
    "ControlNetModel"
  ],
  "feature_extractor": [
    "transformers",
    "CLIPImageProcessor"
  ],
  "image_encoder": [
    null,
    null
  ],
  "requires_safety_checker": true,
  "safety_checker": [
    "stable_diffusion",
    "StableDiffusionSafetyChecker"
  ],
  "scheduler": [
    "diffusers",
    "DDIMScheduler"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}

To solve your issue, please install diffusers from source

[fabiorigano]

by the way, isn't load_safety_checker a .from_single_file() argument? You should probably pass safety_checker=None when you use .from_pretrained()

[yiyixuxu]

should be fixed in #6868