Feature request: prompt2image option to output intermediate images

[TesseractCat]

It would be cool if there was a way to see the intermediate results as the sampler creates the image. This could be useful for samplers such as k_euler_a which varies quite a bit between steps, as a way to generate variations.

Perhaps the prompt2image function could have an intermediates parameter, which would be an array of steps to save images at.

So for example, one could call prompt2image("astronaut riding a horse", steps = 50, intermediates = [10, 20, 30, 40]).

Here is an example of an intermediate images using the k_euler_a sampler at steps 30, 25, 20, and 15:

[TingTingin]

I believe #86 allows for this where you u can specify a certain interval and have parameters increase by that amount per repetition this case if you wanted to this you'd do

astronaut riding a horse -s 10:10 -r 5 to accomplish what you asked

[TesseractCat]

Unfortunately, I think that would regenerate the image for each increase in step value. This means that for my example you would cumulatively evaluate 10 + 20 + 30 + 40 + 50 = 150 steps.

[TingTingin]

Just so were on the same page what you want is 4 images 1 at 10 steps 1 at 20 steps 1 at 30 steps 1 at 40 steps? if so thats what the command would do though since the first gen isnt counted by -r so the actual command would be

astronaut riding a horse -s 10:10 -r 3 -S 1

[TesseractCat]

The problem with that command is that it would end up generating the image 5 times, just with different step values. This PR is asking for a way to save intermediate results as an image is being generated. This would be a lot faster as you would only have to generate an image once to get the intermediate results.

In the end the output would be the same, but would only require one generation.

[TingTingin]

Ah ok I understand now maybe that request could be updated in the case that steps are the only thing being interpolated to generate the images as it goes instead of waiting to complete

[TesseractCat]

Well, the interpolation is a feature of the dream.py script. I'm asking for a lower-level interface through the simplet2i class (most likely what the interpolation would build upon if it added that special case).

[morganavr]

So for example, one could call prompt2image("astronaut riding a horse", steps = 50, intermediates = [10, 20, 30, 40]).

I would like to see all intermediate images while final image is generating and specifying each image # in --intermediates parameter is too time consuming.

What if all value could be passed to --intermediates making it generate all the intermediate images?

prompt2image("astronaut riding a horse", steps=50, intermediates = ["all"]).

[bakkot]

The sample method which underlies txt2img actually already takes a img_callback parameter which gets invoked at each step. All that's necessary is to provide a callback for that parameter at the callsite (at least for txt2img; making it work for img2img would require tweaking the decode method to support such a parameter as well).

As a matter of a design, I think a callback which takes the image and index and is called at each iteration is cleanest. That way you can choose to save every item or every 10th item or whatever you want.

I think it would be cool to integrate this with the web UI too so that the final image would slowly come into focus.

EDIT: that's done in #129 (comment).

[green-s]

I've been using the img_callback to do this in my own Streamlit-based web UI. Works fine with DDIM and PLMS but not with k_lms (and most likely the other k-diffusion samplers, haven't implemented support for those). k_lms doesn't seem to trigger it, except maybe after completion, haven't checked since I update the image at the end anyway.

[BlueAmulet]

It's also worth noticing that although k_euler_a has wildly different outputs depending on the number of steps, it doesn't actually look like that in it's intermediates. You're going to just see a blurrier fuzzier version of the final image, same for all other samplers.

The k-diffusion samplers do have a callback but it isn't implemented in the wrapper and isn't directly compatible with the ddim and plms sampler callbacks.

Intermediate examples of k_euler_a (Click to expand)

Step 34

Step 67

Step 100