AI companies are rolling out AI assist features that is giving privacy advocates their worst nightmare in years, finds Satyen K. Bordoloi
Data is the new oil, is now an old adage. That doesn’t mean the AI industry, which needs as much data as possible to train its systems, has solved the data problem. They still need more data and don’t know where to get it after scraping everything on the net, in your personal social media feeds, and even on online piracy sites. Now, if some of the moves by some of the AI companies are to be believed for what they’re not saying but doing, they’re coming for your private data, from your emails, to the data stored in your computer.
In January, Google rolled out perhaps the most radical change to Gmail in its two-decade history: deep AI integration that gives its Gemini assistant a panoramic view of your inbox, photos, and more, all under the banner of a “personal, proactive inbox assistant”.
The move, thankfully, sparked immediate debate. On one hand, AI tools promise to organise chaos, summarise threads, and even draft emails. On the other hand, it’s a profound, fundamental shift in the privacy contract. The inbox, once a private digital space, is now a classroom for hungry AI models. And Google is not alone. Microsoft’s Copilot peers into your Word documents and spreadsheets, while GitHub Copilot ingests the context of your code. The pattern is clear: in search for “new oil”, these gold diggers will leave no stone unturned, including the ones that protect your private correspondence, files, and creative work.
Your Inbox, Their AI’s Training Ground
Google’s new email policy announcement was classic Silicon Valley: frame a surveillance upgrade as a generous gift. New features like AI Overviews, “Help Me Write,” suggested replies, and an “AI Inbox” that acts as a personal briefing are now being turned on by default for Gmail’s over 3 billion users. The company promises a helpful assistant, but as has become the cliché for such services in recent years: when a service is free, you and your data are the product.
Now, how this works is interesting. For Gemini to summarise your emails or suggest tasks, it first has to read and analyse them. This ‘reading’, naturally, gives it access to your data. Google says that this data isn’t currently used to train the foundational Gemini model. But here’s where the problem lies: the very access to your data is automatically training it.
The training is built into the architecture itself. If you consider this alongside its past record on privacy, including multi-billion-dollar fines for monopolistic and non-compliant practices, you realise how little confidence you can have in the company. To add to this, if you enable these “smart features”, there’s the potential that your data automatically get linked to other services like add-ons, which may be able to connect Gemini to WhatsApp or Spotify. This means your information could be shared with third parties as well.
Now, what is simply mind-boggling is how you have to deliberately opt out, i.e. to disable Gemini’s access to your account, you must dive into Gmail settings, clear the “Smart features” checkbox for your account, and then separately disable “Google Workspace smart features”. The catch? Turning off the AI disables legacy conveniences like email categorisation and package tracking.
If you thought the company whose motto when it started a quarter century ago was ‘do no evil’ is the only one actively trying to do so, you’re mistaken. Others are not that far behind. Afterall data is the new oil of the 21st century.
The Copilot in Your Machine
The data grab naturally extends far beyond email. Now Microsoft has embedded its Copilot assistant across Windows, Edge, and the entire Microsoft 365 suite. And it, too, like everyone else, is out there gathering whatever it can from all of us.
GitHub Copilot has become an indispensable tool for developers, boosting productivity by up to 55% on some tasks. Now, to provide relevant code suggestions, it gathers context from the file you’re editing and also neighbouring files. Now Microsoft/GitHub emphasises robust data-handling procedures, including checks for toxic language and hacking attempts, and screening code suggestions for security vulnerabilities and unique identifiers such as passwords. Individual subscribers can definitely opt out of having their prompts used to fine-tune models. Yet one cannot forget that the system is built on the real-time analysis of your work. And what that means for the codes you build and fine-tune is anybody’s guess.
How Microsoft is coming for your data is perhaps the most ingenious of them all. through its CoPilot AI app. In apps like Word, Excel, and Outlook, Copilot can analyse your content to generate text, summarise data, or manage your calendar. Turning it off requires a scavenger hunt through each application’s settings. In Word or Excel, you go to File > Options > Copilot and clear the “Enable Copilot” checkbox. For Outlook, the toggle is in Settings.
However, a more nuclear option is to disable all “connected experiences,” but this also kills features like PowerPoint Designer, Editor, text predictions and other AI-driven conveniences we’ve begun taking for granted. In a sense, as with Google, opting out of the embedded AI means opting out of modern conveniences.
Now, to understand what happens to this data, I asked CoPilot itself to explain what happens to it. First, it told me the usual lines: “The content is processed in real time to generate suggestions. Microsoft states it does not permanently store your files or computer data just because you use Copilot.” And that “By default, some prompts and completions may be used to improve Copilot models. You can opt out in settings if you don’t want your data used this way.”
Then I aske CoPilot where the data goes to and its answer is as is obviously known, “The Copilot app or plugin on your computer acts as the interface. It gathers the immediate context (your document, spreadsheet, email, or code file) and sends that snippet securely to the cloud” and “The cloud AI processes the input, generates a suggestion, and sends it back to your app, where it appears inline (e.g., in Word, Excel, Outlook, or VS Code).” It insisted that only “snippets of your work are transmitted to the cloud for processing.” But would that be enough?
Who Owns Your Data
Yes, all these companies say they do not copy your data or upload it to their servers; they merely let the AI analyse it for your convenience. But the very fact that CoPilot, Gemini, or any of these are not Edge apps, i.e., the analysis is not happening on your computer, means that the data is sent to servers in the US for analysis.
Yes, it comes back, but how much of those “snippets” stays on those servers, and what of it automatically trains the AI, is a million-dollar question for which there isn’t a clear answer.
The privacy implications are major. Now, for consumer accounts (unlike more locked-down enterprise versions), interactions with systems like Copilot can influence Microsoft’s advertising ecosystem. Even if you turn off personalised ads, interaction data may be used in anonymised form for training their systems, and to personalise ads for others. Furthermore, by using Copilot, you grant Microsoft broad rights to use your content, which privacy advocates warn could extend to unreleased intellectual property you input for feedback.
Of course, neither Microsoft nor Google will admit that they gain broad rights to your content, but there is no doubt that this is a legal grey area with no clear answers. Included here is the fact that if you’re not in the US, the data will cross borders, and since the servers are mostly in the US, it will be subjected it to US jurisdiction under laws like the CLOUD Act. Using systems like DeepSeek means the data goes to China, which in many ways is worse.
A Broader Trend
Google and Microsoft are just the most visible players in a sector-wide gold rush. The strategy is as straightforward as with these two companies: to make AI smarter and more “personal,” it must be fed the most personal data available. Meta AI is integrated into Facebook, Instagram, WhatsApp and Messenger. And they have been using chat data from interactions with its Meta AI for targeted advertising, explicitly commercialising private conversations. And they do it openly, too, as their privacy policy confirms: conversations with Meta AI may be used to improve AI systems.
The same goes for platform integrations. Services like LinkedIn (owned by Microsoft) have trained AI on public profiles and activity data. The planned deep integration of Gemini into Android to replace Google Assistant points to a future where it won’t be wrong to call your phone’s OS a constant AI listener to everything you do on these systems.
The trend has even been forecasted. Take the 2026 data management forecast from analysts like One Data, which identifies “AI-ready data as a decisive competitive advantage”. The report notes that while companies pour money into AI, most fail because their data isn’t “high-quality, well-documented, and governance-compliant”. The solution? Go straight to the source, aka the user. The industry is betting that convenience will outweigh privacy concerns, especially when opting out is made deliberately inconvenient.
Where does that leave the end user, you and me? We are left in a thorny maze, a tangled web of risk and reward. The potential good is enormous: AI can manage information overload, with an AI that prioritises emails, drafts routine responses, and surfaces critical tasks, being a genuine boon for busy professionals. And as I have written before, AI is also crucial for security, as AI can better filter phishing attempts and malware.
Yet we cannot forget that all this convenience comes at the erosion of privacy, lack of opacity and control in our hands, feeling trapped as many of these things work only in bundles, proliferation of your secret data in multiplicity of cloud systems and obviously all of this falls in a legal and intellectual property grey zone.
And your only solution in this battle: an obscure opt-out option.
The onus, hence, is now on us, the users, to decide where to draw the line as we navigate deliberately cumbersome settings to reclaim our own data, and to consider whether true digital efficiency must always come at the cost of our digital autonomy.
Yes, Gemini in Gmail or CoPilot in your laptop might quietly be organising our lives, always ready with “How Can I Help You Today”. Yet when we engage with it, the question we should be asking ourselves is this: Help, yes, but at what cost?
In case you missed:
- AI Browser or Trojan Horse: A Deep-Dive Into the New Browser Wars
- Your Phone is About to Get a Brain Transplant: How Google’s Tiny, Silent Model Changes Everything
- Zero Clicks. Maximum Theft: The AI Nightmare Stealing Your Future
- OpenAI, Google, Microsoft: Why are AI Giants Suddenly Giving India Free AI Subscriptions?
- The Great AI Browser War: When AI Decided to Crash the Surfing Party
- Meet Manus AI: Your New Digital Butler (don’t ask it to make coffee yet)
- Rogue AI on the Loose: Can Auditing Uncover Hidden Agendas on Time?
- Microsoft’s Quantum Chip Majorana 1: Marketing Hype or Leap Forward?
- Bots to Robots: Google’s Quest to Give AI a Body (and Maybe a Sense of Humour)
- Beyond the Hype: The Spectacular, Stumbling Evolution of Digital Eyewear
