in Blog Posts, Computer Vision (机器视觉)


In my experience so far, Fooocus is the best open-sourced image generation software. It gives high quality results without much extra efforts for casual use while providing lots of options to tweak for demanding users.

Generated By FOOOCUS

FOOOCUS has an easy-to-use inpainting UI that one can brush over the image to get the mask for inpainting. But sometimes I do want to quickly select a full object.

I spend some time on Saturday to play with the Gradio widgets, which is actually quite interesting that I think I will use it for future projects, to add EfficientSAM (a variant of Segment-Anything Model) to FOOOCUS.

Added under the “Inpainting” Tab

I chose EfficientSAM for its compactness, in terms of both model and code. It should be quite intuitive to replace the segmentation model if needed. Just grep “esam” for the relevant lines. There are surely things to be done to make it faster, e.g., by caching the image features for multiple rounds of prompting. The modified version is at pppoe/FOOOCUS-SAM.

In addition to get the segmentation mask, there is a few more operations to clear and invert the mask. Uploading a mask from file is also a choice.

Draw sth. to sample prompt points

It is a little bit tricky to sample points from the brushed area. I posted a gist for this part to get uniformly distributed 2d points to cover the brushed area as much as possible (link to git).

Click “SAM-Mask” to get the mask

Click “Invert-Mask” to get it on the background

Replace the background with “A table in the park”

Definitely has room for improvement, but looks good and cute LOL

I didn’t find an easy to use Web app to edit masks, which is a little surprising to me. Hope the open source community to have something on this in the future.

Write a Comment