Multimodal

Send images, PDFs, and files alongside chat messages for AI analysis.

Overview

User: [Attaches screenshot.png]
User: "What's wrong with this error?"
AI: "I can see a TypeError in your console. The issue is..."

Enable Attachments

<CopilotChat
  attachmentsEnabled={true}
/>

Send Message with Attachment

const { sendMessage } = useYourGPT();

// Send with image
await sendMessage("Analyze this chart", [
  {
    type: 'image',
    data: base64ImageData,
    mimeType: 'image/png',
    filename: 'chart.png',
  },
]);

// Send with file
await sendMessage("Summarize this document", [
  {
    type: 'file',
    data: base64PdfData,
    mimeType: 'application/pdf',
    filename: 'report.pdf',
  },
]);

Attachment Types

interface MessageAttachment {
  type: 'image' | 'file' | 'audio' | 'video';
  data: string;        // Base64 encoded or URL
  mimeType: string;    // MIME type
  filename?: string;   // Display name
  url?: string;        // If stored in cloud
}

Supported Types

Type	MIME Types	Notes
`image`	`image/png`, `image/jpeg`, `image/gif`, `image/webp`	AI can analyze visually
`file`	`application/pdf`, `text/plain`, `text/csv`	AI extracts text content
`audio`	`audio/mp3`, `audio/wav`	Transcription (model dependent)
`video`	`video/mp4`	Frame extraction (model dependent)

Configuration

<CopilotChat
  attachmentsEnabled={true}
  maxFileSize={10 * 1024 * 1024}  // 10MB
  allowedFileTypes={[
    'image/*',
    'application/pdf',
    '.csv',
    '.txt',
  ]}
/>

Props

Prop	Type	Default	Description
`attachmentsEnabled`	`boolean`	`false`	Enable attachment button
`maxFileSize`	`number`	`5MB`	Maximum file size in bytes
`allowedFileTypes`	`string[]`	All	Allowed MIME types or extensions
`processAttachment`	`function`	-	Custom upload handler

Custom Upload Handler

Upload to your own storage:

<CopilotChat
  attachmentsEnabled={true}
  processAttachment={async (file: File) => {
    // Upload to S3, Cloudinary, etc.
    const formData = new FormData();
    formData.append('file', file);

    const response = await fetch('/api/upload', {
      method: 'POST',
      body: formData,
    });

    const { url } = await response.json();

    return {
      type: 'image',
      url,  // Cloud URL instead of base64
      mimeType: file.type,
      filename: file.name,
    };
  }}
/>

Cloud storage is recommended for production. Base64 embedding works but increases message size.

Programmatic Attachments

From File Input

function FileUploader() {
  const { sendMessage } = useYourGPT();

  const handleFileSelect = async (e: React.ChangeEvent<HTMLInputElement>) => {
    const file = e.target.files?.[0];
    if (!file) return;

    // Convert to base64
    const base64 = await fileToBase64(file);

    await sendMessage("Please analyze this file", [{
      type: file.type.startsWith('image/') ? 'image' : 'file',
      data: base64,
      mimeType: file.type,
      filename: file.name,
    }]);
  };

  return <input type="file" onChange={handleFileSelect} />;
}

// Helper function
function fileToBase64(file: File): Promise<string> {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.onload = () => {
      const result = reader.result as string;
      // Remove data URL prefix
      resolve(result.split(',')[1]);
    };
    reader.onerror = reject;
    reader.readAsDataURL(file);
  });
}

From Canvas/Screenshot

async function sendScreenshot() {
  const { sendMessage } = useYourGPT();

  // Capture using html2canvas or similar
  const canvas = await html2canvas(document.body);
  const base64 = canvas.toDataURL('image/png').split(',')[1];

  await sendMessage("What do you see on this screen?", [{
    type: 'image',
    data: base64,
    mimeType: 'image/png',
    filename: 'screenshot.png',
  }]);
}

From URL

await sendMessage("Analyze this image", [{
  type: 'image',
  url: 'https://example.com/chart.png',
  mimeType: 'image/png',
}]);

Display Attachments

Attachments appear in message history:

function MessageAttachments({ attachments }) {
  if (!attachments?.length) return null;

  return (
    <div className="flex gap-2 mt-2">
      {attachments.map((attachment, i) => (
        <AttachmentPreview key={i} attachment={attachment} />
      ))}
    </div>
  );
}

function AttachmentPreview({ attachment }) {
  if (attachment.type === 'image') {
    const src = attachment.url || `data:${attachment.mimeType};base64,${attachment.data}`;
    return (
      <img
        src={src}
        alt={attachment.filename}
        className="max-w-xs rounded-lg"
      />
    );
  }

  return (
    <div className="flex items-center gap-2 p-2 bg-muted rounded">
      <FileIcon className="w-4 h-4" />
      <span className="text-sm">{attachment.filename}</span>
    </div>
  );
}

Image Analysis

AI can analyze images when using vision-capable models:

// Works with: GPT-4o, Claude 3, Gemini Pro Vision
await sendMessage("What's in this image?", [{
  type: 'image',
  data: imageBase64,
  mimeType: 'image/png',
}]);

Use Cases

Bug reports: Attach screenshots of errors
Design feedback: Upload mockups for AI review
Data analysis: Send charts for interpretation
Document processing: PDF extraction and summarization

Best Practices

1. Compress Images

async function compressImage(file: File): Promise<string> {
  const bitmap = await createImageBitmap(file);
  const canvas = document.createElement('canvas');

  // Resize if too large
  const maxSize = 1920;
  let { width, height } = bitmap;
  if (width > maxSize || height > maxSize) {
    const ratio = Math.min(maxSize / width, maxSize / height);
    width *= ratio;
    height *= ratio;
  }

  canvas.width = width;
  canvas.height = height;

  const ctx = canvas.getContext('2d')!;
  ctx.drawImage(bitmap, 0, 0, width, height);

  return canvas.toDataURL('image/jpeg', 0.8).split(',')[1];
}

2. Validate Before Upload

function validateAttachment(file: File): string | null {
  const maxSize = 10 * 1024 * 1024; // 10MB

  if (file.size > maxSize) {
    return `File too large. Maximum size is ${maxSize / 1024 / 1024}MB`;
  }

  const allowedTypes = ['image/png', 'image/jpeg', 'application/pdf'];
  if (!allowedTypes.includes(file.type)) {
    return `File type not supported. Allowed: ${allowedTypes.join(', ')}`;
  }

  return null; // Valid
}

3. Show Upload Progress

function UploadProgress({ progress }) {
  return (
    <div className="w-full bg-muted rounded-full h-2">
      <div
        className="bg-primary h-2 rounded-full transition-all"
        style={{ width: `${progress}%` }}
      />
    </div>
  );
}

Full Example

function ChatWithAttachments() {
  const { sendMessage } = useYourGPT();
  const [uploading, setUploading] = useState(false);
  const fileInputRef = useRef<HTMLInputElement>(null);

  const handleAttachment = async (e: React.ChangeEvent<HTMLInputElement>) => {
    const file = e.target.files?.[0];
    if (!file) return;

    // Validate
    const error = validateAttachment(file);
    if (error) {
      toast.error(error);
      return;
    }

    setUploading(true);

    try {
      // Process image
      let data: string;
      if (file.type.startsWith('image/')) {
        data = await compressImage(file);
      } else {
        data = await fileToBase64(file);
      }

      // Send with message
      await sendMessage(`Please analyze this ${file.type.split('/')[0]}`, [{
        type: file.type.startsWith('image/') ? 'image' : 'file',
        data,
        mimeType: file.type,
        filename: file.name,
      }]);
    } catch (err) {
      toast.error('Upload failed');
    } finally {
      setUploading(false);
    }
  };

  return (
    <div>
      <input
        ref={fileInputRef}
        type="file"
        className="hidden"
        accept="image/*,application/pdf"
        onChange={handleAttachment}
      />

      <button
        onClick={() => fileInputRef.current?.click()}
        disabled={uploading}
        className="btn-secondary"
      >
        {uploading ? 'Uploading...' : 'Attach File'}
      </button>
    </div>
  );
}

Next Steps

Streaming - Real-time responses
AI Context - Provide app context
Custom Tools - Build tools that use attachments

Multimodal

On this page