Describe and Classify Multimodal Gmail Attachments with Zapier and Gemini

Learn how to create a Zap in Zapier to automate Gmail attachments processing using Gemini for audio descriptions and more.

August 8, 2024 By Kristi Lim

In this guide, we'll learn how to create an automation using Zapier to manage email attachments from Gmail. We'll set up a trigger for new attachments and use Gemini to process these files. This automation can help streamline tasks such as sorting marketing materials or categorizing keynote speeches. By the end of this guide, you'll know how to configure the necessary steps and ensure the correct model and API version are selected for optimal results.

Let's get started

Hi, we're back on Zapier and have just created a new Zap. First, click on "Trigger," type in "Gmail," and select "Gmail." Set the event to trigger whenever we receive a new attachment.

Connect your account.

Then, use "Search String" to get more detailed if need be. This way, you can avoid pulling up too much.

For example, if you always work with marketing materials, you can type "marketing." If there's a specific email where these messages come from, type marketing@superacme.com.

I have prepared an image file in advance. Select the file you want to build this automation with.

Step #8: Next, we're going to have Gemini process it for us. We're using Gemini because it allows us to work with the most modalities of files. — Next, we're going to have Gemini process it for us. We're using Gemini because it allows us to work with the most modalities of files.

Step #13: Be very careful to select Beta! Otherwise, we will not be able to attach our files. — Be very careful to select Beta! Otherwise, we will not be able to attach our files.

Step #15: Make sure the model you pick is: Gemini 1.5 Flash Latest. — Make sure the model you pick is: Gemini 1.5 Flash Latest.

Here, attach the audio, image, video or text file of your choice. Gemini also accepts URLs!

Next, write your prompt. You can keep it generic, or clearly specify what you are likely using Gemini for.

Here is my example prompt. Our team needs to start between photos to be posted on our internal blog and photos for external marketing material.

You can write whatever prompt makes sense for you. For example, if you work with audio equipment, you could even ask Gemini to determine what the recording could be in that context. (e.g. A customer complaining about audio feedback produced when using your equipment, and sending a demonstration)

If relevant: reduce the number of tokens to ensure you get only a single word. Alternatively, if you want Gemini to generate a draft blog post or memo based on this attachment, leave it at a higher number.

Step #20: I reduced mine to 30. — I reduced mine to 30.

Click "Test Step." Here are my results! Gemini has determined that it is an external photo, and the original photo is indeed more suited for external use.

Step #23: This was the image, courtesy of Midjourney. — This was the image, courtesy of Midjourney.

From here, you can set up a path that proceeds only if the response contains "external." In our case, we are going to upload this as an asset on Canva for our marketing team to use.

Step #39: From here, we attach our image file. — From here, we attach our image file.

Step #42: You can generate the name from: 1) Gemini's full description, if your original prompt was different; 2) The subject of the email, if this is likely to be reliably unique; 3) Sender of email; It's up to you! — You can generate the name from: 1) Gemini's full description, if your original prompt was different; 2) The subject of the email, if this is likely to be reliably unique; 3) Sender of email; It's up to you!

And our Zap is complete! Don't forget to publish it by clicking on "Publish" in the top right.