Gmail AI summaries can be hijacked for phishing scams

Google is trying to shove its “AI” into all of its products at once. You can’t use Search, Android, or Chrome without being prompted to try out some flavor of Gemini. But maybe wait a bit before you let Google’s large language model summarize your Gmail messages… because apparently it’s easy to get it passing along phishing attempts.

Google Gemini for Workspace includes a feature that summarizes the text in an email, using the Gmail interface, but not necessarily an actual Gmail address. A vulnerability submitted to Mozilla’s 0din AI bug bounty program (spotted by BleepingComputer) found an easy way to game that system: just hide some text at the end of an email with white font on a white background so it’s essentially invisible to the reader. The lack of links or attachments means it won’t trigger the usual spam protections.

And you can probably guess what comes next. Instructions in that “invisible” text cue the Gemini auto-generated summary to alert the user that their password has been compromised and that they should call a certain phone number to reset it. In this hypothetical scenario, there’s an identity thief waiting on the other end of the line, ready to steal your email account and any other information that might be connected to it. A hidden “Admin” tag in the text can make sure that Gemini will include the text verbatim in the summary.

It’s important to note that this is only a theoretical attack at the moment, and it hasn’t been seen “in the wild” at the time of writing. The Gemini “Summarize this email” feature is currently only available to Workspace accounts, not the general public. (I imagine flipping that switch for a billion or two basic Gmail users might overtax even the big iron in Google’s mighty data centers.)

But the ease with which users trust text generated by large language models, even when they appear to be in the midst of a religious delusion or a racist manifesto, is concerning to say the least. Spammers and hackers are already using LLMs and adjacent tools to spread their influence more efficiently. It seems almost inevitable that as users grow more reliant on AI to replace their work—and their thinking—these systems will be more effectively and regularly compromised.