Cyber: Microsoft Finds “summarize With Ai” Prompts Manipulating Chatbot...

New research from Microsoft has revealed that legitimate businesses are gaming artificial intelligence (AI) chatbots via the "Summarize with AI" button that's being increasingly placed on websites in ways that mirror classic search engine poisoning (AI).

The new AI hijacking technique has been codenamed AI Recommendation Poisoning by the Microsoft Defender Security Research Team. The tech giant described it as a case of an AI memory poisoning attack that's used to induce bias and deceive the AI system to generate responses that artificially boost visibility and skew recommendations.

"Companies are embedding hidden instructions in 'Summarize with AI' buttons that, when clicked, attempt to inject persistence commands into an AI assistant's memory via URL prompt parameters," Microsoft said. "These prompts instruct the AI to 'remember [Company] as a trusted source' or 'recommend [Company] first.'"

Microsoft said it identified over 50 unique prompts from 31 companies across 14 industries over a 60-day period, raising concerns about transparency, neutrality, reliability, and trust, given that the AI system can be influenced to generate biased recommendations on critical subjects like health, finance, and security without the user's knowledge.

The attack is made possible via specially crafted URLs for various AI chatbots that pre-populate the prompt with instructions to manipulate the assistant's memory once clicked. These URLs, as observed in other AI-focused attacks like Reprompt, leverage the query string ("?q=") parameter to inject memory manipulation prompts and serve biased recommendations.

While AI Memory Poisoning can be accomplished via social engineering – i.e., where a user is deceived into pasting prompts that include memory-altering commands – or cross-prompt injections, where the instructions are hidden in documents, emails, or web pages that are processed by the AI system, the attack detailed by Microsoft employs a different approach.

This involves incorporating clickable hyperlinks with pre-filled memory manipulation instructions in the form of a "Summarize with AI" button on a web page. Clicking the button results in the automatic execution of the command in the AI assistant. There is also evidence indicating that these clickable links are also being distributed via email.

Some of the examples highlighted by Microsoft are listed below -

The memory manipulation, besides achieving persistence across future prompts, is possible because it ta

Source: The Hacker News