Insert objects into an image using inpaint

Console

<a href="https://console.cloud.google.com/vertex-ai/studio/media/generate;tab=image" class="button button-primary" target="console" track-name="consoleLink" track-type="task">Go to Media Studio</a> 

1. Click Upload and select a file to upload.

2. Click Inpaint.

   Do one of the following:

   • Upload your own mask:

     1. Create a mask on your computer.

     2. Click Upload mask and select a mask to upload.

   • Define your mask: in the editing toolbar, use the mask tools (check_box_outline_blankbox, brushbrush, or masked_transitions invert tool) to specify the area or areas to add content to.

3. Optional: In the Parameters panel, adjust the following options:

   • Model: the Imagen model to use.

   • Number of results: the number of result to generate.

   • Negative prompt: describe what you want to exclude from the generated images.

4. In the prompt field, enter a prompt to modify the image.

5. Click play_arrowGenerate.

Python

Install

pip install --upgrade google-genai

To learn more, see the SDK reference documentation.

Set environment variables to use the Gen AI SDK with Vertex AI:

# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values # with appropriate values for your project. export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT export GOOGLE_CLOUD_LOCATION=us-central1 export GOOGLE_GENAI_USE_VERTEXAI=True

from google import genai from google.genai.types import ( RawReferenceImage, MaskReferenceImage, MaskReferenceConfig, EditImageConfig, ) client = genai.Client() # TODO(developer): Update and un-comment below line # output_file = "output-image.png" raw_ref = RawReferenceImage( reference_image=Image.from_file(location="test_resources/fruit.png"), reference_id=0, ) mask_ref = MaskReferenceImage( reference_id=1, reference_image=Image.from_file(location="test_resources/fruit_mask.png"), config=MaskReferenceConfig( mask_mode="MASK_MODE_USER_PROVIDED", mask_dilation=0.01, ), ) image = client.models.edit_image( model="imagen-3.0-capability-001", prompt="A plate of cookies", reference_images=[raw_ref, mask_ref], config=EditImageConfig( edit_mode="EDIT_MODE_INPAINT_INSERTION", ), ) image.generated_images[0].image.save(output_file) print(f"Created output image using {len(image.generated_images[0].image.image_bytes)} bytes") # Example response: # Created output image using 1234567 bytes 

REST

Before using any of the request data, make the following replacements:

• REGION: The region that your project is located in. For more information about supported regions, see Generative AI on Vertex AI locations.
• PROJECT_ID: Your Google Cloud project ID.
• TEXT_PROMPT: Optional. A text prompt to guide the images that the model generates. For best results, use a description of the masked area and avoid single-word prompts. For example, use "a cute corgi" instead of "corgi".
• B64_BASE_IMAGE: A base64-encoded image of the image being edited that is 10MB or less in size. For more information about base64-encoding, see Base64 encode and decode files.
• B64_MASK_IMAGE: A base64-encoded black and white mask image that is 10MB or less in size.
• MASK_DILATION: Optional. A float value between 0 and 1, inclusive, that represents the percentage of the image width to grow the mask by. Using dilation helps compensate for imprecise masks. We recommend a value of 0.01.

• EDIT_STEPS: Optional. An integer that represents the number of sampling steps. A higher value offers better image quality, a lower value offers better latency.

  We recommend that you try 35 steps to start. If the quality doesn't meet your requirements, then we recomment increasing the value towards an upper limit of 75.

• SAMPLE_COUNT: Optional. An integer that describes the number of images to generate. The accepted range of values is 1-4. The default value is 4.

HTTP method and URL:

POST https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict

Request JSON body:

 { "instances": [ { "prompt": "TEXT_PROMPT", "referenceImages": [ { "referenceType": "REFERENCE_TYPE_RAW", "referenceId": 1, "referenceImage": { "bytesBase64Encoded": "B64_BASE_IMAGE" } }, { "referenceType": "REFERENCE_TYPE_MASK", "referenceImage": { "bytesBase64Encoded": "B64_MASK_IMAGE" }, "maskImageConfig": { "maskMode": "MASK_MODE_USER_PROVIDED", "dilation": MASK_DILATION } } ] } ], "parameters": { "editConfig": { "baseSteps": EDIT_STEPS }, "editMode": "EDIT_MODE_INPAINT_INSERTION", "sampleCount": SAMPLE_COUNT } } 

To send your request, choose one of these options:

curl -X POST \
 -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 -H "Content-Type: application/json; charset=utf-8" \
 -d @request.json \
 "https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
 -Method POST `
 -Headers $headers `
 -ContentType: "application/json; charset=utf-8" `
 -InFile request.json `
 -Uri "https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/imagen-3.0-capability-001:predict" | Select-Object -Expand Content

 { "predictions": [ { "bytesBase64Encoded": "BASE64_IMG_BYTES", "mimeType": "image/png" }, { "mimeType": "image/png", "bytesBase64Encoded": "BASE64_IMG_BYTES" } ] } 

Console

<a href="https://console.cloud.google.com/vertex-ai/studio/media/generate;tab=image" class="button button-primary" target="console" track-name="consoleLink" track-type="task">Go to Media Studio</a> 

1. Click Upload and select a file to upload.

2. Click Inpaint.

3. In the editing toolbar, click background_replaceExtract mask.

4. Select one of the mask extraction options:

   • Background elements: detects the background elements and creates a mask around them.

   • wallpaper Foreground elements: detects the foreground objects and creates a mask around them.

   • background_replace People: detects people and creates a mask around them.

5. Optional: In the Parameters panel, adjust the following options:

   • Model: the Imagen model to use.

   • Number of results: the number of result to generate.

   • Negative prompt: describe what you want to exclude from the generated images.

6. In the prompt field, enter a prompt to modify the image.

7. Click sendGenerate.

Python

Install

pip install --upgrade google-genai

To learn more, see the SDK reference documentation.

Set environment variables to use the Gen AI SDK with Vertex AI:

# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values # with appropriate values for your project. export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT export GOOGLE_CLOUD_LOCATION=us-central1 export GOOGLE_GENAI_USE_VERTEXAI=True

from google import genai from google.genai.types import ( RawReferenceImage, MaskReferenceImage, MaskReferenceConfig, EditImageConfig, ) client = genai.Client() # TODO(developer): Update and un-comment below line # output_file = "output-image.png" raw_ref = RawReferenceImage( reference_image=Image.from_file(location="test_resources/fruit.png"), reference_id=0, ) mask_ref = MaskReferenceImage( reference_id=1, reference_image=None, config=MaskReferenceConfig( mask_mode="MASK_MODE_FOREGROUND", mask_dilation=0.1, ), ) image = client.models.edit_image( model="imagen-3.0-capability-001", prompt="A small white ceramic bowl with lemons and limes", reference_images=[raw_ref, mask_ref], config=EditImageConfig( edit_mode="EDIT_MODE_INPAINT_INSERTION", ), ) image.generated_images[0].image.save(output_file) print(f"Created output image using {len(image.generated_images[0].image.image_bytes)} bytes") # Example response: # Created output image using 1234567 bytes 

REST

Before using any of the request data, make the following replacements:

• PROJECT_ID: Your Google Cloud project ID.
• LOCATION: Your project's region. For example, us-central1, europe-west2, or asia-northeast3. For a list of available regions, see Generative AI on Vertex AI locations.
• TEXT_PROMPT: The text prompt guides what images the model generates. When you use a prompt for inpainting insertion, use a description of the masked area for best results. Avoid single-word prompts. For example, use "a cute corgi" instead of "corgi".
• B64_BASE_IMAGE: The base image to edit or upscale. The image must be specified as a base64-encoded byte string. Size limit: 10 MB.
• MASK_MODE - A string that sets the type of automatic mask creation the model uses. Available values:
  • MASK_MODE_BACKGROUND: Automatically generates a mask using background segmentation.
  • MASK_MODE_FOREGROUND: Automatically generates a mask using foreground segmentation.
  • MASK_MODE_SEMANTIC: Automatically generates a mask using semantic segmentation based on the segmentation classes you specify in the maskImageConfig.maskClasses array. For example:

     "maskImageConfig": { "maskMode": "MASK_MODE_SEMANTIC", "maskClasses": [175, 176], // bicycle, car "dilation": 0.01 } 

• MASK_DILATION - float. The percentage of image width to dilate this mask by. A value of 0.01 is recommended to compensate for imperfect input masks.
• EDIT_STEPS - integer. The number of sampling steps for the base model. For inpainting insertion, start at 35 steps. Increase steps to upper limit of 75 if the quality doesn't meet your requirements. Increasing steps also increases request latency.
• EDIT_IMAGE_COUNT - The number of edited images. Accepted integer values: 1-4. Default value: 4.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict

Request JSON body:

 { "instances": [ { "prompt": "TEXT_PROMPT", "referenceImages": [ { "referenceType": "REFERENCE_TYPE_RAW", "referenceId": 1, "referenceImage": { "bytesBase64Encoded": "B64_BASE_IMAGE" } }, { "referenceType": "REFERENCE_TYPE_MASK", "referenceId": 2, "maskImageConfig": { "maskMode": "MASK_MODE", "dilation": MASK_DILATION } } ] } ], "parameters": { "editConfig": { "baseSteps": EDIT_STEPS }, "editMode": "EDIT_MODE_INPAINT_INSERTION", "sampleCount": EDIT_IMAGE_COUNT } } 

To send your request, choose one of these options:

curl -X POST \
 -H "Authorization: Bearer $(gcloud auth print-access-token)" \
 -H "Content-Type: application/json; charset=utf-8" \
 -d @request.json \
 "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
 -Method POST `
 -Headers $headers `
 -ContentType: "application/json; charset=utf-8" `
 -InFile request.json `
 -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagen-3.0-capability-001:predict" | Select-Object -Expand Content

 { "predictions": [ { "bytesBase64Encoded": "BASE64_IMG_BYTES", "mimeType": "image/png" }, { "mimeType": "image/png", "bytesBase64Encoded": "BASE64_IMG_BYTES" } ] }