MagicQuill
An Intelligent Interactive Image Editing System

🎵 We recommend watching the video with sound on 🎵

Abstract

As a highly practical application, image editing encounters a variety of user demands and thus prioritizes excellent ease of use. In this paper, we unveil MagicQuill, an integrated image editing system designed to support users in swiftly actualizing their creativity. Our system starts with a streamlined yet functionally robust interface, enabling users to articulate their ideas (e.g., inserting elements, erasing objects, altering color, etc.) with just a few strokes. These interactions are then monitored by a multimodal large language model (MLLM) to anticipate user intentions in real time, bypassing the need for prompt entry. Finally, we apply the powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process the editing request with precise control. We will release the entire system to facilitate the community.

Tutorial

I. Three type of magic quills

SVG image

Use the add brush to add details and elements guided by prompts - express your ideas with your own lively strokes!

gif description
"With just a few strokes, a vivid little deer comes to life"

gif description
"Adorn the beautiful lady with a necklace"

SVG image

The subtract brush can remove excess details or redraw areas based on prompts. If there's anything you're not satisfied with, just subtract it away!

gif description
"A dolphin with two tail fins? Let's give it a quick 'treatment'!"

gif description
"Let's take off Mr. Skeleton's hat and help him cool down."


add brush&minus brush

Combine the add and subtract brushes to create amazing combo effects!

gif description
"Let's give Mona Lisa a pet cat~"

gif description
"Let's give this handsome fellow a new tie!"

SVG image

The color brush can precisely color the image, matching the color of your brush~

gif description
"Precise color highlighting - paint exactly where you want to color"

gif description
"Don't you think the blue flowers look more dreamy than the pink ones?"

*Please note the color brush and add&subtract brush are mutually exclusive - you can only use one at a time!


II. Draw and Guess

SVG image
Our brush is super smart! Look at the examples above - as soon as you finish drawing, it quickly guesses what you want to create and fills in the prompts for you~ Sometimes it might guess wrong though, so feel free to tell it what you actually want to draw~

gif description
"Oops! I don't want to draw a vine, I want to create a path!"


III. Super useful canvas tools!


SVG image
Click this button to upload the photo you want to edit~

SVG image
Made a mistake with the brush? Just erase it with the rubber tool!

SVG image
Drag, rotate, and resize your strokes with the cursor - just like when you're working in PowerPoint!

add brush&minus brush
Left is ctrl+z, right is ctrl+y - you know what that means! 😊
And for Mac users, left is command+z, right is command+shift+z! 😝

SVG image
Oops! That doesn't look right 😵 - click this trash bin to delete the stroke

SVG image
The brush strokes are in my way, how can I see the image😡?! Try clicking this button to temporarily hide your strokes

add brush&minus brush
These two icons will appear after the image is generated...
I love this generated image 😍, I want to keep editing! ➡️ Click ✅ to continue editing
What is this thing 😡, I don't want to see it! ➡️ Click ❎ to discard the result


IV. Notes


SVG image
When you see the spinning icon in the bottom left corner, it means the magicquill is still charging up 💪 Wait for it to disappear before clicking the Run button!

SVG image
When the magic wand is flashing, our brush is working hard to guess what you're trying to draw 🤔 Please be patient! 🙏


V. Parameters


If you've made it here, you must really love our work! 😍
If you want to learn how to better control the generation results, don't miss this section! 😘
Next to the Run button, you can select parameters to modify advanced settings 🧐

  • Base Model Name: Users can adjust this to select appropriate base models for different editing styles. You may choose any SD1.5 models with custom weight.
  • Negative Prompt: Users can input content they want the model to avoid generating. Whatever you don't want to generate, put it here.
  • Fine Edge: Users can enable this option to activate fine edge control.
  • Grow Size: Adjust this parameter to set the pixel range affected around brush strokes when editing images, to expand/reduce the brush stroke influence area.
  • Edge Strength: Parameter for adjusting the add/subtract brush control strength. Simply put, if you're confident in your drawing skills, you can increase strength. If you're a bad drawer like us 🤦, please keep the parameter as is, or reduce the strength a bit.
  • Color Strength: Parameter for adjusting the color brush control strength, can adjust the image's coloring effects.
  • The remaining parameters are just some common parameters for diffusion models! You basically don't need to manage these, but if you're in the industry/AI art expert, feel free to try adjusting them~