Mast3r, Dust3r, & Cut3r Inference Code

[ayushzenith]

Hi!

Thanks for open-sourcing your work. I was wondering if the inference code for the other LoRA models (Mast3r, Dust3r, Cut3r) is available or planned to be released.

If it’s already public, could you point me to it? And if not, any guidance on how to run inference with the other models would be super helpful.

Thanks again!

[rwn17]

Hi @ayushzenith ,

Thanks for your interest in our work. You can find the checkpoint here. For the code of other architectures, I'm still thinking about how to share it. I think I will share how to replace the model.py file of each model separately next week.

Bests,
Weining

[rwn17]

Hi @ayushzenith, @qinliuliuqin, @hectorcarrion,

I have pushed the integration of dust3r here: https://github.com/rwn17/fin3r_dust3r

To integrate Fin3R into DUSt3R codebase, follow these 3 steps as shown by the commit history at fin3r_vggt and fin3r_dust3r:

1. Add renorm_lora.py
Create dust3r/renorm_lora.py with renorm_lora.py in vggt codebase:

2. Modify dust3r/model.py
Add imports and the apply_lora method to the AsymmetricCroCo3DStereo class:

# ... existing imports ...
from peft import LoraConfig
from dust3r.renorm_lora import get_renormalized_peft_model

class AsymmetricCroCo3DStereo(nn.Module):
    # ... existing __init__ and forward ...

    def apply_lora(self, lora_path=None):
        # LoRA configuration
        lora_config = LoraConfig(
            r=8, # LoRA rank
            lora_alpha=8, # Scaling factor
            target_modules=["qkv"], # Target only 'qkv' layers
            lora_dropout=0.1, # Dropout rate
        )

        for i, block in enumerate(self.enc_blocks):
            block.attn = get_renormalized_peft_model(block.attn, lora_config)

        print(f"Applied Fin3R renormalized LoRA")
        if lora_path is not None:
            self.load_state_dict(torch.load(lora_path), strict=False)

3. Update Inference Script
In your demo.py or inference script:

model = AsymmetricCroCo3DStereo.from_pretrained(model_name).to(device)
# Add this line:
model.apply_lora('path/to/your/dust3r_lora.pth') 

Since MAST3R and CUT3R share the DUSt3R codebase, you can integrate Fin3R using these same steps. If you have any further questions, please feel free to reopen this issue.

Bests,
Weining