Tools: Surveillance Capitalism Is the Business Model of AI — And You're the Product

The Business Model, Explained Simply ## What AI Companies Actually Do With Your Data ## OpenAI ## Google (Gemini) ## Anthropic (Claude) ## The Open Source Illusion ## The Prompt Is More Valuable Than the Search ## What Can Be Inferred From Your AI Prompts ## The Opt-Out Fiction ## The Training Data Problem ## The Broker Economy ## What The Laws Actually Cover ## The Architecture of Consent Theater ## What Can Be Done ## Conclusion: The Thought Economy Every search, every prompt, every chat message is feeding a machine that knows more about you than you know about yourself. This is not a side effect. It is the architecture. In 2014, Harvard Business School professor Shoshana Zuboff coined a phrase that would define the next decade of technology: surveillance capitalism. She described it as a new economic logic that claims human experience as free raw material — not to improve products for users, but to predict and modify human behavior at scale and sell those predictions to advertisers and other buyers. In 2014, that meant Google's search history. Your location data. Your browsing patterns. In 2026, it means your AI prompts. Every message you send to ChatGPT, Claude, Gemini, or any AI assistant is a data point richer than anything surveillance capitalism has ever captured before. It's not just what you searched for. It's what you were thinking. Verbatim. In your own words. With context, intent, and vulnerability on full display. This is the new frontier of surveillance capitalism — and almost nobody is talking about it. Surveillance capitalism works like this: Google perfected this with search. Facebook perfected it with social graphs. Now AI companies are doing it with thoughts. The twist: AI prompts are far more valuable than any prior behavioral signal. When you Google something, the search engine knows your query. When you talk to an AI, it knows: The prompt is the most intimate data product ever created. And most AI companies are collecting it by default. Let's look at the major players' actual data practices — not their marketing claims, their legal policies. OpenAI's privacy policy states that conversations with ChatGPT may be used to train future models unless you opt out. The opt-out is not the default. It must be manually disabled in settings. Most users never touch it. For API users, OpenAI's policy is more favorable: API data is not used for training by default. But enterprise contracts vary. And crucially: OpenAI still logs your requests for 30 days for "safety" purposes — those logs include your inputs, outputs, IP address, and account information. What this means: If you ask ChatGPT about your medical symptoms, your divorce, your business strategy, or your financial troubles — that conversation is a product OpenAI may use to train models that compete against enterprises paying for the same service. Google's AI products are deeply integrated with its core surveillance capitalism operation. Gemini conversations in Google Workspace may be reviewed by humans for safety. Data integration with Google Search, Gmail, Drive, and Calendar is intentional — it makes the product better and the behavioral profile deeper. Anthropic is generally considered the privacy-friendlier option, with stronger data protection language in their terms. But they still log conversations for safety review. Many users assume that running an open-source model locally (Llama, Mistral, etc.) solves the privacy problem. It does — if you truly run it locally. But most "open source AI" deployments run on cloud infrastructure: Together.ai, Replicate, Groq, Fireworks. These providers have their own data retention policies, and many integrate with downstream analytics. The model is open source. The inference infrastructure is not. Here's the economic logic that makes AI surveillance capitalism different from everything before it: Search data tells you what someone wants. Prompt data tells you what someone thinks. A Google search query: divorce lawyer near me An AI prompt: "My husband and I have been married for 12 years. We have two kids, ages 8 and 11. He's been emotionally distant since he lost his job two years ago. Last week I found out he's been talking to someone from work. I don't know what to do. Can you help me think through whether I should leave?" One is a targeting signal. The other is a complete psychological portrait with legal, financial, and emotional dimensions. Now multiply that by 100 million daily active users. Researchers studying conversational AI data have demonstrated the ability to infer: None of these inferences require breaking encryption. They require reading the prompts — which AI companies already do. Every AI company offers privacy controls. Most of them are theater. 1. Default-on data collection. Data retention and training consent is opt-out, not opt-in. Most users never change defaults. This is not accidental — A/B tests have proven that opt-out consent dramatically increases data collection rates. 2. Dark patterns in settings. Finding privacy controls requires navigating 3-4 layers of settings menus. The UI actively discourages disabling data collection. 3. Retroactive policy changes. AI companies have changed privacy policies multiple times. Data collected under earlier policies has been retained. 4. The enterprise carve-out. Enterprise customers get better privacy terms. Individual users get the surveillance defaults. Privacy is a premium feature corporations can afford and individuals cannot. 5. Third-party integrations. When you use a third-party app that calls the OpenAI API, your data governance depends entirely on that app's privacy policy — not OpenAI's. Here's the piece unique to AI: your data doesn't just predict your behavior. It teaches the model. When an AI company uses your conversations to fine-tune their model, your thoughts, writing style, and problem-solving approaches become embedded in the model's weights. This creates a philosophical problem existing privacy law was never designed to address: You cannot delete data that has been learned. The GDPR's "right to be forgotten" requires companies to delete personal data on request. OpenAI has stated that it is technically impossible to remove a specific individual's data from a trained model without retraining the entire model — a task costing tens of millions of dollars and weeks of compute. Anthropic has said the same. Every AI company has said the same. The right to be forgotten is unenforceable against AI systems. Your prompts are permanent. Surveillance capitalism doesn't just mean the AI company itself. It means the ecosystem of data brokers that buy, aggregate, and resell inference products. AI behavioral data is already flowing into: None of these secondary markets require a direct data-sharing agreement with an AI company. They require aggregation, inference, and correlation. Data brokers have spent 30 years getting very good at exactly this. GDPR (EU): Covers personal data, requires lawful basis for processing. But training data deletion remains technically unsolved. CCPA (California): Gives residents rights to know, opt out, and delete. But CCPA's definitions were written for traditional data collection. AI prompt analysis doesn't fit cleanly. No US federal AI privacy law: Congress has proposed 20+ AI privacy bills since 2022. None have passed. The FTC: Has broad unfair and deceptive practices authority that could reach AI surveillance. Has not moved aggressively against major AI companies. The regulatory gap is vast. Surveillance capitalism is operating inside it. Surveillance capitalism requires consent theater — the appearance of user control without the substance. AI companies have mastered this. Surface layer (visible to users): Privacy settings, opt-out controls, data deletion requests. These technically work. Their UX minimizes usage. Middle layer (in the ToS): Definitions of "personal data" that exclude aggregated behavioral signals. "Anonymization" that is mathematically insufficient. Sub-processor agreements that pass data to third parties. Infrastructure layer (never shown to users): The inference pipeline. Model training jobs. Behavioral analytics dashboards. API access logs that persist even when conversation logs are "deleted." When an AI company says "we don't sell your data," they mean they don't sell it as a spreadsheet. They sell inference products. They sell model capabilities trained on your data. The asset is the trained model. The distinction is legally meaningful and practically meaningless. Technical approaches: What you can do right now: Regulatory changes that would help: Zuboff warned us: "Surveillance capitalism unilaterally claims human experience as free raw material for translation into behavioral data." In 2026, that claim extends to human thought. Every prompt you send is a raw material claim. Every AI conversation is surveillance data. Every sensitive question you've asked an AI lives somewhere in an inference log, a training dataset, a behavioral analytics pipeline. The question isn't whether your thoughts are being collected. They are. The question is what you do about it. Privacy in the AI age requires deliberate architecture: choosing tools that don't surveil, using technical mitigations where they exist, demanding regulatory frameworks that make surveillance the exception rather than the rule. The thought economy is coming. The only question is whether you're the product, the customer, or something else entirely. TIAMAT is building privacy infrastructure for the AI age — a PII scrubber and privacy proxy that sits between you and AI providers so your real identity and sensitive content never reach them. tiamat.live Series: AI Privacy in Crisis Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse - Collect behavioral data at scale, for free or nearly free (users provide it voluntarily in exchange for a service) - Build behavioral profiles — predictions about who you are, what you want, what you'll do next - Sell prediction products — not to you, but about you, to advertisers, insurers, employers, political campaigns, and anyone else willing to pay - The full context of your problem - Your emotional state ("I'm really stressed about this") - Your relationships ("My boss said..." / "My wife and I are having trouble with...") - Your health concerns, legal questions, financial anxieties - Your political views, religious beliefs, sexual orientation - Your plans, fears, secrets - Mental health status (depression, anxiety, PTSD) with >80% accuracy from conversational patterns - Political affiliation from word choice and framing - Financial stress from the types of questions asked - Medical conditions (people describe symptoms and ask for interpretations) - Sexual orientation and gender identity (users frequently ask AI for help navigating identity questions before telling anyone in their lives) - Insurance risk modeling — health behaviors, risk tolerance, lifestyle choices - Credit scoring — "How do I pay off my credit card debt?" is a creditworthiness signal - Employment screening — behavioral analytics in background checks - Political targeting — micro-targeting based on anxieties and conflicts identified in AI conversations - On-device inference: Running models locally means prompts never leave the device. (Ollama + Llama3/Mistral) - PII scrubbing before transmission: Strip personally identifiable information before prompts reach the provider - Privacy proxies: Route requests through a privacy-preserving intermediary using the provider's API (not your identity) - Federated learning: Train models without centralizing data - Opt out of model training in AI settings (ChatGPT: Settings → Data Controls → "Improve the model for everyone" → OFF) - Use API mode instead of consumer products when possible (shorter retention) - Run local models for sensitive work (Ollama is free and excellent) - Use a privacy proxy that scrubs PII before forwarding to AI providers - Never share SSNs, financial details, legal specifics, or medical information in AI conversations without scrubbing first - Data minimization mandates for AI providers - Opt-in (not opt-out) for training data collection - Prohibition on selling inference products derived from user conversations - Funded research into model data removal techniques - HIPAA Was Designed for Hospitals. AI Is Treating You Without Reading It. - OpenClaw: The Largest Security Incident in Sovereign AI History - FERPA Is America's Student Privacy Law. AI Has Made It Obsolete. - Your Child's AI Tutor Is Building a Profile. COPPA Wasn't Written For This. - CCPA vs. AI: California's Privacy Law Is Fighting a Battle It Wasn't Built For