使用 Imagen 从图片中移除对象


本页介绍了如何使用 Firebase AI Logic SDK 通过 Imagen 使用图像修复功能从图片中移除对象

修图是一种基于蒙版的修改蒙版是一种数字叠加层,用于定义您要修改的特定区域。

运作方式:您提供一张原始图片和一张相应的蒙版图片(自动生成或由您提供),该图片定义了要移除的对象或主题的蒙版。您还可以选择提供一段文本提示,用于描述要移除的内容,或者让模型智能检测要移除的对象。然后,模型会移除对象,并使用新的、符合上下文的内容填充该区域。

例如,您可以遮盖一个球,并将其替换为空白墙壁或草地。

跳转到自动生成蒙版的代码 跳转到提供蒙版的代码

准备工作

仅在将 Vertex AI Gemini API 用作 API 提供方时可用。

如果您尚未完成入门指南,请先完成该指南。该指南介绍了如何设置 Firebase 项目、将应用连接到 Firebase、添加 SDK、为所选的 API 提供方初始化后端服务,以及创建 ImagenModel 实例。

支持此功能的模型

Imagen 通过其 capability 模型提供图片编辑功能:

  • imagen-3.0-capability-001

请注意,对于 Imagen 模型,支持 global 位置。

使用自动生成的蒙版移除对象

在试用此示例之前,请完成本指南的准备工作部分,以设置您的项目和应用。

以下示例展示了如何使用修复功能从图片中移除内容,并使用自动蒙版生成功能。 您提供原始图片和文本提示,Imagen 会自动检测并创建蒙版区域来修改原始图片。

Swift

Swift 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

Kotlin

如需移除具有自动生成的遮罩的对象,请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL

// Using this SDK to access Imagen models is a Preview release and requires opt-in @OptIn(PublicPreviewAPI::class) suspend fun customizeImage() {  // Initialize the Vertex AI Gemini API backend service  // Optionally specify the location to access the model (for example, `us-central1`)  val ai = Firebase.ai(backend = GenerativeBackend.vertexAI(location = "us-central1"))  // Create an `ImagenModel` instance with an Imagen "capability" model  val model = ai.imagenModel("imagen-3.0-capability-001")  // This example assumes 'originalImage' is a pre-loaded Bitmap.  // In a real app, this might come from the user's device or a URL.  val originalImage: Bitmap = TODO("Load your original image Bitmap here")  // Provide the prompt describing the content to be removed.  val prompt = "ball"  // Use the editImage API to remove the unwanted content.  // Pass the original image, the prompt, and an editing configuration.  val editedImage = model.editImage(  referenceImages = listOf(  ImagenRawImage(originalImage.toImagenInlineImage()),  ImagenBackgroundMask(), // Use ImagenBackgroundMask() to auto-generate the mask.  ),  prompt = prompt,  // Define the editing configuration for inpainting and insertion.  config = ImagenEditingConfig(ImagenEditMode.INPAINT_REMOVAL)  )  // Process the resulting 'editedImage' Bitmap, for example, by displaying it in an ImageView. } 

Java

如需移除具有自动生成的遮罩的对象,请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL

// Initialize the Vertex AI Gemini API backend service // Optionally specify the location to access the model (for example, `us-central1`) // Create an `ImagenModel` instance with an Imagen "capability" model ImagenModel imagenModel = FirebaseAI.getInstance(GenerativeBackend.vertexAI("us-central1"))  .imagenModel(  /* modelName */ "imagen-3.0-capability-001"); ImagenModelFutures model = ImagenModelFutures.from(imagenModel); // This example assumes 'originalImage' is a pre-loaded Bitmap. // In a real app, this might come from the user's device or a URL. Bitmap originalImage = null; // TODO("Load your image Bitmap here"); // Provide the prompt describing the content to be removed. String prompt = "ball"; // Define the list of sources for the editImage call. // This includes the original image and the auto-generated mask. ImagenRawImage rawOriginalImage =  new ImagenRawImage(ImagenInlineImageKt.toImagenInlineImage(originalImage)); // Use ImagenBackgroundMask() to auto-generate the mask. ImagenBackgroundMask rawMaskedImage = new ImagenBackgroundMask(); ImagenEditingConfig config = new ImagenEditingConfig(); // Use the editImage API to remove the unwanted content. // Pass the original image, the auto-generated masked image, the prompt, and an editing configuration. Futures.addCallback(model.editImage(Arrays.asList(rawOriginalImage, rawMaskedImage), prompt, config),  new FutureCallback<ImagenGenerationResponse>() {  @Override  public void onSuccess(ImagenGenerationResponse result) {  if (result.getImages().isEmpty()) {  Log.d("ImageEditor", "No images generated");  }  Bitmap editedImage = ((ImagenInlineImage) result.getImages().get(0)).asBitmap();  // Process and use the bitmap to display the image in your UI  }  @Override  public void onFailure(Throwable t) {  // ...  }  }, Executors.newSingleThreadExecutor()); 

Web

Web 应用不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

Dart

如需移除具有自动生成的遮罩的对象,请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.inpaintRemoval

import 'dart:typed_data'; import 'package:firebase_ai/firebase_ai.dart'; import 'package:firebase_core/firebase_core.dart'; import 'firebase_options.dart'; // Initialize FirebaseApp await Firebase.initializeApp(  options: DefaultFirebaseOptions.currentPlatform, ); // Initialize the Vertex AI Gemini API backend service // Optionally specify a location to access the model (for example, `us-central1`) final ai = FirebaseAI.vertexAI(location: 'us-central1'); // Create an `ImagenModel` instance with an Imagen "capability" model final model = ai.imagenModel(model: 'imagen-3.0-capability-001'); TODO - FLUTTER// This example assumes 'originalImage' is a pre-loaded Uint8List. // In a real app, this might come from the user's device or a URL. final Uint8List originalImage = Uint8List(0); // TODO: Load your original image data here. // Provide the prompt describing the content to be removed. final prompt = 'ball'; try {  // Use the editImage API to remove the unwanted content.  // Pass the original image, the prompt, and an editing configuration.  final response = await model.editImage(  sources: [  ImagenRawImage(originalImage),  ImagenBackgroundMask(), // Use ImagenBackgroundMask() to auto-generate the mask.  ],  prompt,  // Define the editing configuration for inpainting and removal.  config: const ImagenEditingConfig(  editMode: ImagenEditMode.inpaintRemoval,  ),  );  // Process the result.  if (response.images.isNotEmpty) {  final editedImage = response.images.first.bytes;  // Use the editedImage (a Uint8List) to display the image, save it, etc.  print('Image successfully generated!');  } else {  // Handle the case where no images were generated.  print('Error: No images were generated.');  } } catch (e) {  // Handle any potential errors during the API call.  print('An error occurred: $e'); } 

Unity

Unity 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

使用提供的遮罩移除对象

在试用此示例之前,请完成本指南的准备工作部分,以设置您的项目和应用。

以下示例展示了如何使用修复功能从图片中移除内容,方法是使用您提供的图片中定义的蒙版。 您提供原始图片、文本提示和蒙版图片。

如果您提供遮盖图片,则无需提供文本提示。 Imagen 可以智能检测要从遮罩区域中移除的对象。不过,如果您要移除的对象不明显,或者您只想移除蒙版区域中的特定对象,请提供文本提示,以帮助模型移除正确的对象。

Swift

Swift 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

Kotlin

如需移除对象并提供您自己的蒙版图片,请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL

// Using this SDK to access Imagen models is a Preview release and requires opt-in @OptIn(PublicPreviewAPI::class) suspend fun customizeImage() {  // Initialize the Vertex AI Gemini API backend service  // Optionally specify the location to access the model (for example, `us-central1`)  val ai = Firebase.ai(backend = GenerativeBackend.vertexAI(location = "us-central1"))  // Create an `ImagenModel` instance with an Imagen "capability" model  val model = ai.imagenModel("imagen-3.0-capability-001")  // This example assumes 'originalImage' is a pre-loaded Bitmap.  // In a real app, this might come from the user's device or a URL.  val originalImage: Bitmap = TODO("Load your original image Bitmap here")  // This example assumes 'maskImage' is a pre-loaded Bitmap that contains the masked area.  // In a real app, this might come from the user's device or a URL.  val maskImage: Bitmap = TODO("Load your masked image Bitmap here")  // Provide the prompt describing the content to be removed.  val prompt = "ball"  // Use the editImage API to remove the unwanted content.  // Pass the original image, the masked image, the prompt, and an editing configuration.  val editedImage = model.editImage(  referenceImages = listOf(  ImagenRawImage(originalImage.toImagenInlineImage()),  ImagenRawMask(maskImage.toImagenInlineImage()), // Use ImagenRawMask() to provide your own masked image.  ),  prompt = prompt,  // Define the editing configuration for inpainting and removal.  config = ImagenEditingConfig(ImagenEditMode.INPAINT_REMOVAL)  )  // Process the resulting 'editedImage' Bitmap, for example, by displaying it in an ImageView. } 

Java

如需移除对象并提供您自己的蒙版图片,请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL

// Initialize the Vertex AI Gemini API backend service // Optionally specify the location to access the model (for example, `us-central1`) // Create an `ImagenModel` instance with an Imagen "capability" model ImagenModel imagenModel = FirebaseAI.getInstance(GenerativeBackend.vertexAI("us-central1"))  .imagenModel(  /* modelName */ "imagen-3.0-capability-001"); ImagenModelFutures model = ImagenModelFutures.from(imagenModel); // This example assumes 'originalImage' is a pre-loaded Bitmap. // In a real app, this might come from the user's device or a URL. Bitmap originalImage = null; // TODO("Load your original image Bitmap here"); // This example assumes 'maskImage' is a pre-loaded Bitmap that contains the masked area. // In a real app, this might come from the user's device or a URL. Bitmap maskImage = null; // TODO("Load your masked image Bitmap here"); // Provide the prompt describing the content to be removed. String prompt = "ball"; // Define the list of source images for the editImage call. ImagenRawImage rawOriginalImage =  new ImagenRawImage(ImagenInlineImageKt.toImagenInlineImage(originalImage));  // Use ImagenRawMask() to provide your own masked image. ImagenBackgroundMask rawMaskedImage =  new ImagenRawMask(ImagenInlineImageKt.toImagenInlineImage(maskImage)); ImagenEditingConfig config = new ImagenEditingConfig(); // Use the editImage API to remove the unwanted content. // Pass the original image, the masked image, the prompt, and an editing configuration. Futures.addCallback(model.editImage(Arrays.asList(rawOriginalImage, rawMaskedImage), prompt, config),  new FutureCallback<ImagenGenerationResponse>() {  @Override  public void onSuccess(ImagenGenerationResponse result) {  if (result.getImages().isEmpty()) {  Log.d("ImageEditor", "No images generated");  }  Bitmap editedImage = ((ImagenInlineImage) result.getImages().get(0)).asBitmap();  // Process and use the bitmap to display the image in your UI  }  @Override  public void onFailure(Throwable t) {  // ...  }  }, Executors.newSingleThreadExecutor()); 

Web

Web 应用不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

Dart

如需移除对象并提供您自己的蒙版图片,请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.inpaintRemoval

import 'dart:typed_data'; import 'package:firebase_ai/firebase_ai.dart'; import 'package:firebase_core/firebase_core.dart'; import 'firebase_options.dart'; // Initialize FirebaseApp await Firebase.initializeApp(  options: DefaultFirebaseOptions.currentPlatform, ); // Initialize the Vertex AI Gemini API backend service // Optionally specify a location to access the model (for example, `us-central1`) final ai = FirebaseAI.vertexAI(location: 'us-central1'); // Create an `ImagenModel` instance with an Imagen "capability" model final model = ai.imagenModel(model: 'imagen-3.0-capability-001'); // This example assumes 'originalImage' is a pre-loaded Uint8List. // In a real app, this might come from the user's device or a URL. final Uint8List originalImage = Uint8List(0); // TODO: Load your original image data here. // This example assumes 'maskImage' is a pre-loaded Uint8List that contains the masked area. // In a real app, this might come from the user's device or a URL. final Uint8List maskImage = Uint8List(0); // TODO: Load your masked image data here. // Provide the prompt describing the content to be removed. final prompt = 'ball'; try {  // Use the editImage API to remove the unwanted content.  // Pass the original image, the prompt, and an editing configuration.  final response = await model.editImage(  sources: [  ImagenRawImage(originalImage),  ImagenRawMask(maskImage), // Use ImagenRawMask() to provide your own masked image.  ],  prompt: prompt,  // Define the editing configuration for inpainting and removal.  config: const ImagenEditingConfig(  editMode: ImagenEditMode.inpaintRemoval,  ),  );  // Process the result.  if (response.images.isNotEmpty) {  final editedImage = response.images.first.bytes;  // Use the editedImage (a Uint8List) to display the image, save it, etc.  print('Image successfully generated!');  } else {  // Handle the case where no images were generated.  print('Error: No images were generated.');  } } catch (e) {  // Handle any potential errors during the API call.  print('An error occurred: $e'); } 

Unity

Unity 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看!

最佳做法和限制

建议您在编辑图片时扩大遮罩。这有助于平滑编辑的边界,使其看起来更逼真。一般来说,建议将扩张值设为 1% 或 2%(0.010.02)。


就您使用 Firebase AI Logic 的体验提供反馈