AI Adoption vs. Data Privacy: Finding the Right Balance

To unlock the full benefits of artificial intelligence, companies and users need to understand how to protect the data that drives the systems.

Even in an industry as massive, fluid and precarious as tech, AI’s rapid ascension has few parallels. Statements that were hyperbole just six months ago are now commonly accepted to the point of cliché. AI-based applications such as chatbots, large language models (LLMs), and other AI-assisted tools have assumed prominent roles within the tech ecosystem, with their influence sure to grow as AI intelligence accelerates. Growing concerns over the technology’s tenuous relationship with data security and privacy loom large, however.

Given the sheer size of the organizations currently developing AI, from OpenAI to Microsoft and beyond, data handling concerns are not altogether surprising. The deep sense of caution and anxiety that many users feel regarding AI goes beyond mere distrust of all large corporations, however, as Microsoft’s covert usage of LinkedIn data to train its generative AI models demonstrated last year.

• Data collection.
• User input data.
• Security risks.
• Third-party data sharing.
• Transparency and user control.

To safeguard the industry’s long-term prospects and user trust, it must fully address these concerns. For end users, this means first understanding AI’s weak points when it comes to data privacy, followed by taking actionable steps to minimize exposure.

LLMs like ChatGPT derive their capabilities from vast quantities of training data sourced from all corners of the internet, including content from blogs, social media platforms, forums and many others. The sheer amount of this content makes it nearly impossible to check this data’s veracity, which raises serious, lasting questions around potential biases and irregularresponses. These models’ need to ingest as much data as possible raises additional ethical concerns, as Big Tech companies get creative to overcome the internet’s finite amount of available training data, sometimes without explicit authorial consent.

To overcome these concerns, some AI models like CoPilot integrate directly with the expansive Microsoft 365 application ecosystem, allowing them to also tap into user data such as emails, documents, calendar events, chat logs and more via the Graph API system. These integrations raise additional concerns about data transparency and usage at a company-wide level.

When users interact with OpenAI and other publicly accessible chatbots, they may share sensitive information, often inadvertently. In most cases, the data that users input is retained indefinitely unless they explicitly opt out, which can be a complex process. This retention poses risks of data misuse or unauthorized access. In some cases, an enterprise plan may be available with improved security controls, but those plans may be cost-prohibitive and difficult to navigate for the majority of businesses.

New research reveals that chatbots like ChatGPT can accurately infer a lot of sensitive information about the users they chat with, just from the content submitted. The phenomenon appears to stem from the way that models’ algorithms are trained with broad swaths of web content (a key part of what makes them work), so this functionality is essentially baked into LLMs.

Although representatives from all major LLM developers have claimed they’re making strides in removing personal information from the training data used to create their models, this claim seems dubious at best. Even if true, this approach is still fundamentally flawed as it relies on LLMs to infer and contextualize what personal information is in order to remove it. Specifically, this includes personally identifiable information (PII) such as names, addresses, email addresses, etc.

Inputting sensitive data into these programs is not a harmless mistake, either. Doing so could permanently cement this data into the model, causing it to inadvertently share that data with other users down the line. New features, such as ChatGPT’s ability to access screen content, further amplify privacy risks. For CoPilot users who rely on Microsoft’s integrated product suite for their entire organization, the rapid expansion of LLMs could be particularly worrisome if the company doesn’t prioritize data security within all new development. The potential implications are significant, ranging from security vulnerabilities to potential data breaches.

OpenAI’s privacy policy states that the company may share user data with unspecified third parties to meet business objectives. This language isn’t particularly clear or reassuring. This lack of specificity has raised concerns around who has access to personal data and for what purposes.

Officially, OpenAI also states that they will not sell data or actively seek out personal information to build profiles or target users with advertising. The company also recently announced significant media partnerships, however, suggesting a growing appetite for data acquisition. These partnerships include Time, The Financial Times, and Condé Nast, owner of publications including Vogue, The New Yorker and Vanity Fair.

This marriage effectively grants OpenAI access to massive content archives, enabling their technology to analyze user behaviors such as reading habits, preferences and engagement patterns across platforms. These trends raise numerous questions about the long-term business model for AI companies, and how it may conflict with user privacy.

The introduction of Facebook Ads in the 2010s comes to mind as a familiar parallel, when the social media giant was criticized for sharing user IDs with third-party advertisers, enabling the creation of detailed user profiles based on Facebook interests and activity for highly personalized ad targeting across the web. This was further exacerbated by third-party applications on Facebook that were found to be transmitting user data to tracking companies without proper disclosure.

Integrations with other platforms, such as Apple’s use of ChatGPT, raises additional privacy concerns, as does the rise of the “GPT Store,” which allows citizen developers to create and share custom AI models with very few checks and balances. Even Microsoft, which is an outlier in not sharing user data with third parties without explicit permission, has found the integration of various plugins and services complicates data governance. All of these examples demonstrate the entirely new Pandora’s box of data privacy concerns that AI has opened for end users.

Finally, a general lack of transparency exists around how data is collected, stored and used by OpenAI. Sorting out this confusion inevitably falls to the user, further complicating their quest for privacy. Though OpenAI states that they aim to minimize the amount of personal information used in the training process, this ambiguity has prompted further concerns around which information should be shared on the platform.