Add multiple modalities in a single message

Author: domenic

README.md

@@ -232,6 +232,41 @@ Details:
 
 * As described [above](#customizing-the-role-per-prompt), you can also supply a `role` value in these objects, so that the full form is `{ role, type, content }`. However, for now, using any role besides the default `"user"` role with an image or audio prompt will reject with a `"NotSupportedError"` `DOMException`. (As we explore multimodal outputs, this restriction might be lifted in the future.)
 
+### Multiple modalities in a single message
+
+Consider a prompt such as `Here is an image: <<<image>>>. Please describe it.`. This is intended to be a single prompt from the user role. To express this, you can use an array for the `content` value:
+
+```js
+const response = await session.prompt({
+ type: "user",
+ content: [
+ "Here is an image: ",
+ { type: "image", content: imageBytes },
+ ". Please describe it."
+ ]
+});
+```
+
+This has _different semantics_ than prompting with multiple user messages:
+
+```js
+// THESE ARE PROBABLY NOT WHAT YOU WANT
+const probablyWrongResponse = await session.prompt([
+ "Here is an image": ",
+ { type: "image", content: imageBytes },
+ ". Please describe it."
+]);
+
+// Equivalent (and also probably wrong)
+const probablyWrongResponse2 = await session.prompt([
+ { role: "user", type: "text", content: "Here is an image: " },
+ { role: "user", type: "image", content: imageBytes },
+ { role: "user", type: "text", content: ". Please describe it." }
+]);
+```
+
+Those examples involve three separate user messages, which the underlying model will likely interpret differently. (To see this, compare with [our above multi-user example](#customizing-the-role-per-prompt), or with how you react when someone texts you three messages in a row vs. a single message.)
+
 ### Structured output or JSON output
 
 To help with programmatic processing of language model responses, the prompt API supports structured outputs defined by a JSON schema.
@@ -618,17 +653,36 @@ dictionary LanguageModelExpectedInput {
 
 typedef (LanguageModelPrompt or sequence<LanguageModelPrompt>) LanguageModelPromptInput;
 
-// Prompt lines
-
 typedef (
- DOMString // interpreted as { role: "user", type: "text", content: providedValue }
- or LanguageModelPromptDict // canonical form
+ // canonical form
+ LanguageModelPromptDict
+ // interpreted as { role: providedValue.role, content: [{ type: providedValue.type, content: providedValue.content }] }
+ or LanguageModelPromptDictFlattened
+ // interpreted as { role: "user", content: [{ type: "text", content: providedValue }] }
+ or DOMString
 ) LanguageModelPrompt;
 
+typedef (
+ // canonical form
+ LanguageModelPromptContentDict
+ // interpreted as { type: "text", content: providedValue }
+ or DOMString
+) LanguageModelPromptContent;
+
 dictionary LanguageModelPromptDict {
+ LanguageModelPromptRole role = "user";
+ required (LanguageModelPromptContent or sequence<LanguageModelPromptContent>) content;
+};
+
+dictionary LanguageModelPromptDictFlattened {
  LanguageModelPromptRole role = "user";
  LanguageModelPromptType type = "text";
- required LanguageModelPromptContent content;
+ required LanguageModelPromptContentValue content;
+};
+
+dictionary LanguageModelPromptContentDict {
+ LanguageModelPromptType type = "text";
+ required LanguageModelPromptContentValue content;
 };
 
 enum LanguageModelPromptRole { "system", "user", "assistant" };
@@ -640,7 +694,7 @@ typedef (
  or AudioBuffer
  or BufferSource
  or DOMString
-) LanguageModelPromptContent;
+) LanguageModelPromptContentValue;
 ```
 
 ### Instruction-tuned versus base models