Skip to content

[REQUEST] Implementing Depthmap Anything V2 model? #450

Open
Open
[REQUEST] Implementing Depthmap Anything V2 model?#450
Labels
orangeA vibrant color

Description

@eyeEmotion
eyeEmotion
opened on Jun 19, 2024

Hi,

is there any intention of implementing the new Depth Anything v2 model? (Base and Large).

I'm currently using it via cmd-prompt, and it has a lot more detail, and is a little bit faster at generating depthmaps too. Although in certain areas, it does take a step backwards though in comparison to v1. But still a vast improvement overal.

Cheers.

PS: sorry, I don't know how to put labels on there requests.

Activity

  1. sonakravtsenko3
    added
    orangeA vibrant color
    on Jun 21, 2024

  2. petermg commented on Jul 4, 2024

    @petermg
    petermg
    on Jul 4, 2024

    Hi,

    is there any intention of implementing the new Depth Anything v2 model? (Base and Large).

    I'm currently using it via cmd-prompt, and it has a lot more detail, and is a little bit faster at generating depthmaps too. Although in certain areas, it does take a step backwards though in comparison to v1. But still a vast improvement overal.

    Cheers.

    PS: sorry, I don't know how to put labels on there requests.

    I'm also hoping for this!

  3. sonakravtsenko3 commented on Jul 4, 2024

    @sonakravtsenko3
    sonakravtsenko3
    on Jul 4, 2024
    Collaborator

    I am hoping somwbody could impöement it. I am busy currently.

  4. graemeniedermayer commented on Jul 9, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 9, 2024
    Contributor

    If @petermg @eyeEmotion @semjon00 wanted to test it out #458 in pull request. Might need some finessing but it seems to be working.

  5. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    If @petermg @eyeEmotion @semjon00 wanted to test it out #458 in pull request. Might need some finessing but it seems to be working.

    Awesome! How do I install it instead of the main branch?

  6. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Thought I got it installed, it shows up as an option but it gives the error:

    ModuleNotFoundError: No module named 'depth_anything_v2'

  7. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Here's the full error:

    Please report this issue here. Make sure to provide the full stacktrace:
    Traceback (most recent call last):
    File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\common_ui.py", line 548, in run_generate
    input_i, type, result = next(gen_obj)
    File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\core.py", line 329, in core_generation_funnel
    raise e
    File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\core.py", line 130, in core_generation_funnel
    model_holder.ensure_models(inp[go.MODEL_TYPE], device, inp[go.BOOST], inp[go.TILING_MODE])
    File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\depthmap_generation.py", line 68, in ensure_models
    self.load_models(model_type, device, boost, tiling_mode)
    File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\depthmap_generation.py", line 233, in load_models
    from depth_anything_v2.depth_anything_v2.dpt import DepthAnythingV2
    ModuleNotFoundError: No module named 'depth_anything_v2'

  8. graemeniedermayer commented on Jul 10, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 10, 2024
    Contributor

    You might need to do a extra git pull.

    If not check that in the extension file stable-diffusion-webui-depthmap-script folder there is a depth_anything_v2 folder that isn't empty.

    I might have also made a pathing error somewhere.

  9. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    You might need to do a extra git pull.

    If not check that in the extension file stable-diffusion-webui-depthmap-script folder there is a depth_anything_v2 folder that isn't empty.

    I might have also made a pathing error somewhere.

    Still getting the error. Here is what that folder contains on my system:
    image

    also here is the error:
    image

  10. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    and when I did a git pull

    image

  11. graemeniedermayer commented on Jul 10, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 10, 2024
    Contributor

    Thank you for all the information! That is how the folder should look, so I'm a little confused. I made some small edits, but I'm not confident they'll fix it. I'll have to think about it more.

    If you have any other extensions that use depth_anything_v2 you could try changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change. You could also try temporarily turning off all the other extensions.

    My import structure is very similar to the marigold import. Does marigold work for you?

  12. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Thank you for all the information! That is how the folder should look, so I'm a little confused. I made some small edits, but I'm not confident they'll fix it. I'll have to think about it more.

    If you have any other extensions that use depth_anything_v2 you could try changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change. You could also try temporarily turning off all the other extensions.

    My import structure is very similar to the marigold import. Does marigold work for you?

    Yes Marigold works no problem. Maybe I installed it wrong? I went to EXTENSIONS tab, UNCHECKED the official depth map extension and installed this one via INSTALL FROM URL and used this as the url:
    https://github.com/graemeniedermayer/stable-diffusion-webui-normalmap-script
    and for branch put depthanything_v2
    It says it's installed. But maybe I did something wrong? Here is my extension tab.
    image

  13. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Ok so here is something weird, BEFORE I update to the update you just did, I have the DEPTH Tab as seen here:

    image

    But after updating to your update just now, it's totally gone!

    image

  14. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    ok I deleted the extensions inside the folder and installed just your version and the depth tab is back but still the same exact error. I'll see if I can figure out how to do what you mentioned about changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change.

  15. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    I am unable to find the file "depth_anything_v2.dpt". Should I have it somewhere?

  16. graemeniedermayer commented on Jul 10, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 10, 2024
    Contributor

    the problem import line is in src/depthmap_generation.py file.

    Oh I forgot to push my last updates.

  17. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    the problem import line is in src/depthmap_generation.py file.

    Oh I forgot to push my last updates.

    THAT WORKED!!!
    HOWEVER I did end up needing to manually delete the original installation of the extension because when I just updated it, the "Depth" tab went away again. When I deleted the folder manually and reinstalled the extension, then it worked.

  18. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    So for some reason YOUR version is working BETTER than if I just use depth anything v2 from here
    https://github.com/DepthAnything/Depth-Anything-V2
    from the command line!? The one from there using the command line produces a depth map with only 8 bit precision, resulting in stair stepping when trying to apply this to a 2D plane, for example:
    image
    You can see the stair stepping in that one, which was done by the command line version, and produces a depth map of only 8 bit integer.
    HOWEVER, when I process the same image using your extension of Depth Anything V2 the result I get is:
    image
    Do you have any idea why this is? Like what command or line in what python script changes the bit depth accuracy? Also, not that it's needed but is it possible to increase the bit depth precision even more just to see if it would make any difference?

  19. graemeniedermayer commented on Jul 10, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 10, 2024
    Contributor

    Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

    For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

    (the line of code in run.py that down samples the array)

    depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

  20. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

    For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

    (the line of code in run.py that down samples the array)

    depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

    Thank you, how can I modify this so that I can increase the precision?

  21. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

    For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

    (the line of code in run.py that down samples the array)

    depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

    Ok so I changed to this:

     for k, filename in enumerate(filenames):
        print(f'Progress {k+1}/{len(filenames)}: {filename}')
        
        raw_image = cv2.imread(filename)
        
        depth = depth_anything.infer_image(raw_image, args.input_size)
        
        depth = (depth - depth.min()) / (depth.max() - depth.min()) * 2048.0
        depth = depth.astype(np.uint16)
        
        if args.grayscale:
            depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
        else:
            depth = (cmap(depth)[:, :, :3] * 2048)[:, :, ::-1].astype(np.uint16)
        
        if args.pred_only:
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), depth)
        else:
            split_region = np.ones((raw_image.shape[0], 50, 3), dtype=np.uint16) * 2048
            combined_result = cv2.hconcat([raw_image, split_region, depth])
            
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), combined_result)

    from:

        for k, filename in enumerate(filenames):
        print(f'Progress {k+1}/{len(filenames)}: {filename}')
        
        raw_image = cv2.imread(filename)
        
        depth = depth_anything.infer_image(raw_image, args.input_size)
        
        depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
        depth = depth.astype(np.uint8)
        
        if args.grayscale:
            depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
        else:
            depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)
        
        if args.pred_only:
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), depth)
        else:
            split_region = np.ones((raw_image.shape[0], 50, 3), dtype=np.uint8) * 255
            combined_result = cv2.hconcat([raw_image, split_region, depth])
            
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), combined_result)

    Now it creates files that are 16 bit but are very dark, look almost black, until I open them in GIMP and use the LEVELS tool. I also increase the precision in GIMP as high as it will go BEFORE I save the file, but I don't think I should HAVE to do this but it turns out better when I do. I'm crazy impressed with this so far. Would be nice to be able to automate this so that I can process multiple at a time. Here is my current command line that I run

    python run.py --encoder vitl --img-path Inputpics --outdir Outputpics --pred-only --grayscale --input-size 1200

    Increasing the input size makes a NOTICEABLE DIFFERENCE!

  22. graemeniedermayer commented on Jul 10, 2024

    @graemeniedermayer
    graemeniedermayer
    on Jul 10, 2024
    Contributor

    Oh instead of 2048 you should be able to use 255*255 or 65025. This should make them less dark. The input size is interesting I should double check that I implemented that correctly.

  23. petermg commented on Jul 10, 2024

    @petermg
    petermg
    on Jul 10, 2024

    Oh instead of 2048 you should be able to use 255*255 or 65025. This should make them less dark. The input size is interesting I should double check that I implemented that correctly.

    THANKS!! that works PERFECTLY!!!

  24. petermg commented on Jul 11, 2024

    @petermg
    petermg
    on Jul 11, 2024

    I don't really know much about the whole "BOOST" and "TILING" options, they always seem to make things WORSE when I've tried to use them, but at least for Depth Anything V2, I think what IS useful if you could implement it in this plugin version of it, is changing the input size, like manually selecting it on the sliders and also selecting the native input size checkbox.

  25. noobtoob4lyfe commented on Jul 20, 2024

    @noobtoob4lyfe
    noobtoob4lyfe
    on Jul 20, 2024

    Now that there has been so much progress in this thread towards installing and using without errors, would it be possible to give a quick guide of how to install this for a less advanced user like myself? I would really love to give this a try.

  26. petermg commented on Jul 20, 2024

    @petermg
    petermg
    on Jul 20, 2024

    Now that there has been so much progress in this thread towards installing and using without errors, would it be possible to give a quick guide of how to install this for a less advanced user like myself? I would really love to give this a try.

    https://www.reddit.com/r/StableDiffusion/comments/1e02w8b/how_to_use_depth_anything_v2_in_automatic1111s/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    orangeA vibrant color

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions