What are we exploring? As generative AI firms start hiring copyright experts, the industry is seeing a shift towards formal content licensing.
Did you know? Over 100 million books have been published globally in modern history, which means millions of book titles (the vast majority) are not included in any disclosed AI training set.
Why it matters: Large language model (LLM) creators are now willing to pay for reliable data feeds rather than risk using copyrighted text without consent.
- AI developers who want access to high-quality, up-to-date data are finding it prudent to negotiate licenses, rather than face litigation and brand damage.
- Many high profile licensing deals show that paying for data is becoming a norm.
We are witnessing the foundation of a new content licensing industry for AI, where deals between AI firms and content owners become commonplace.
Given the continuous creation of new content, licensing by AI firms will be an ongoing frontier.
The key point is that much of the best human knowledge (from literature to scientific research) is not yet in the training data of models – but it could be, if legal and economic barriers are overcome.
A new world of content licensing is emerging
Large language models (LLMs) have historically been trained on vast swaths of internet text, often scraped without permission.
Recently, however, there is a notable shift toward formal content licensing for AI training data, especially in the U.S. and EU.
Major publishers, news firms, and other content owners are increasingly striking deals with AI developers, or pursuing legal action in absence of a deal, to ensure their content is used only with permission and compensation.
In this research article, we examine how likely it is that LLM developers will license content (versus unlicensed scraping), what factors are driving this shift, how much high-quality content remains untapped by AI, and who the key stakeholders are in the emerging content licensing industry.
‘These models need to be fed’
LLM developers are increasingly likely to license content rather than rely on non-permissioned scraping.
Over 2023–2025, multiple big deals have been announced in both the U.S. and Europe, signaling a trend.
For example, OpenAI has signed agreements with leading news publishers, including the Financial Times (UK) and News Corp (publisher of Wall Street Journal and others), to supply up-to-date news content and archives for ChatGPT and model training.
In March 2024, OpenAI also signed deals with France’s Le Monde and Spain’s Prisa Media (owner of El País and other outlets), allowing those publishers’ articles to be used in training and to surface as summarised, attributed ‘answers’ in ChatGPT.
Thomson Reuters’ CEO has mentioned “a number” of licensing deals to let AI companies train on Reuters news wire content, noting that “these models need to be fed… by the highest-quality, independent fact-based content”.
Such deals indicate that LLM creators are willing to pay for reliable data feeds rather than risk using copyrighted text without consent.
Legal and public pressure
Legal and public pressure are pushing AI firms toward licensing.
In late 2023, The New York Times filed a high-profile lawsuit against OpenAI (and its partner Microsoft) after negotiations broke down, alleging that OpenAI “taught ChatGPT to ignore journalists’ copyrights” and seeking damages and even the destruction of models trained on NYT content.
This case highlighted the legal risk of using copyrighted material without a license in the U.S., where the question hinges on how courts interpret fair use.
Similar lawsuits by groups of publishers have emerged. For instance, in early 2025 a coalition of 14 large U.S. news and magazine publishers (members of the News/Media Alliance) sued AI startup Cohere for “widespread unauthorised use” of thousands of their articles in training data.
Notably, some of these same publishers (e.g. The Guardian and The Atlantic) have licensed content to other AI platforms, underscoring that developers who don’t strike deals may face litigation.
In Europe, publishers have also taken action. Italian and Indian news firms have pursued legal remedies against OpenAI’s scraping, while others have chosen partnership, e.g. French news agency AFP signed a multi-year deal with AI startup Mistral in 2025 to provide its newswire content to the model “to strengthen the accuracy and relevance” of AI outputs.
Overall, the climate is increasingly favoring “deal or no deal”: AI developers who want access to high-quality, up-to-date data are finding it prudent to negotiate licenses, and those who do not are encountering lawsuits or public criticism.
Authoritative content is valued the most
Market trends and model performance needs also make licensing likely. As LLM development moves from academic experimentation to competitive commercial deployment, companies need premium data to differentiate and continually build their models.
Research shows that AI developers “favor high-quality content” from reputable publishers when assembling training datasets.
“Because AI models cannot effectively train themselves on their own output, known as synthetic data, they constantly require the new training data to evolve and maintain integrity,” says Stephen Conmy, content director with Contenseo.
“As a result, there is now a massive demand [and market] for quality AI training content and for smart ways to capture more data and feed it to new applications.”
In fact, an analysis by Ziff Davis found that as training data is curated more selectively, the share of content from major “premium” media domains rises dramatically, from well under 1% in raw web scrapes to over 12% in a curated dataset like OpenWebText2 (see table below).
This shows that the most authoritative content (news sites, scholarly sources, etc.) is disproportionately valued in making better AI models.
Because much of that high-value content is copyrighted (and often paywalled), licensing agreements are a logical solution to obtain it legally. In short, both the carrot (better data leading to better models) and the stick (lawsuits and laws) are motivating AI companies to formalise content access.
Table above: Share of content from top news/media publishers in various training datasets. Curated sets (OpenWebText, etc.) include a much higher fraction of high-authority publisher content than unfiltered web crawls. This indicates LLM developers deliberately seek out quality content, which often requires licensing.
Finally, leading AI companies themselves are signaling a shift to licensed data. OpenAI has publicly stated it is in discussions with “dozens of publishers” about licensing content for model training.
Google has likewise made deals, for example, a $60 million/year agreement with Reddit (signed Feb 2024) grants Google’s AI access to Reddit’s vast user-generated content.
These moves suggest that paying for data is becoming a norm.
Even open-source model communities are grappling with this trend. Early LLMs were trained on scraped datasets like The Pile, but now some open data sources are being withdrawn or restricted (e.g. Reddit’s API, Stack Overflow’s data).
Taken together, it is highly likely that future large LLM projects will rely far more on licensed corpora than on indiscriminate web scraping.
The unlicensed route is increasingly fraught with legal risk, and as the “easy” public data gets used up, model builders will turn to the remaining rich troves of content that require permission to access.
This wave of litigation exerts pressure
Several legal, economic, and industry factors are driving the move toward content licensing for AI:
- Copyright law and lawsuits (U.S.): In the U.S., the legality of using copyrighted text for AI training without consent is unsettled, hinging on fair use. Rightsholders are aggressively testing this in court. Alongside the NY Times case, multiple class-action lawsuits by authors (organised by the Authors Guild and others) target OpenAI, Meta, and Google for training on copyrighted books. Authors argue that using books without a license isn’t fair use and are seeking to “clarify that books need to be licensed when used to train AI”.
Similarly, news publishers in the U.S. (News/Media Alliance members) accuse AI firms of copying articles (even paywalled content) without permission. This wave of litigation exerts pressure on AI developers to avoid unauthorised data use. Even if fair use might eventually protect some training uses, the cost and uncertainty of litigation is a strong deterrent. Thus, legal risk is pushing companies toward the safer path of licensing agreements, essentially “contracts not to sue” that grant needed permissions.
- Regulatory environment (EU): The EU has established a stricter framework around text and data mining (TDM) and is considering new AI regulations that bolster licensing. Under Article 4 of the EU’s 2019 DSM Copyright Directive, rights holders can opt out of allowing their works to be mined for commercial AI training (for example by a machine-readable notice). If they opt out, AI developers must obtain a license or else infringe copyright. This opt-out mechanism effectively encourages licensing: many European publishers have indeed added notices reserving rights for AI/TDM on their websites. Furthermore, the proposed EU AI Act may soon require transparency on training data and proof of copyright compliance as a condition for deploying models in Europe. A recent draft would oblige AI providers to “ensure the training… is carried out in compliance with Union law on copyright…with regards to Article 4(3)” (the opt-out rule). If adopted, this will “significantly strengthen the position of creators and rights holders to prevent or license the use of their works” for AI training. In practice, that means AI companies operating in the EU would have to filter out any content that was opted-out or else secure a license, a strong legal driver for formal agreements. Even ahead of new laws, European media have leverage through national regulations (e.g. France’s neighboring right for news snippets) and a general legal climate favoring creators. The result is that AI firms find it necessary to negotiate with EU content owners (as seen by the flurry of deals with European publishers in 2023–24) to ensure compliance and avoid regulatory roadblocks.
- Economic incentives: High-quality content is valuable fuel for AI, and content owners now realise this value. Publishers see an opportunity for new revenue streams by licensing archives and data to AI developers. The deals announced so far often involve substantial payments. For instance, the Google/Reddit deal at ~$60 million per year demonstrates the market value of a major platform’s user content. Publishers that license content can command significant fees or integrations; Reuters’ CEO noted in March 2024 that “a market price [is] evolving” for training-quality data. For the AI companies, these costs are weighed against the benefit of having exclusive or reliable access to data that can improve their models (and against the potential cost of litigation or bad PR if they don’t license). As training large models already costs tens or hundreds of millions of dollars in compute, the relative cost of data licenses is small in context, and paying for premium data can be seen as a necessary investment. There’s also a competitive angle: if one AI provider secures exclusive rights to a valuable dataset, it could gain an edge over rivals. This dynamic pushes the industry toward a data licensing economy, where content becomes a traded commodity for AI. Indeed, some observers predict a “boom in data licensing deals” for AI as companies race to secure scarce quality data. The corollary for content owners is the emergence of a “get paid or get scraped” ultimatum; and many prefer to get paid.
- Industry relations and ethics: The tech and media industries are trying to establish a new equilibrium. Media companies have loudly protested that unbridled AI data mining “undercut[s] our businesses while using our own content to compete with us”, as a News/Media Alliance representative put it. In response, AI firms have begun outreach to assuage these concerns. Licensing deals are often bundled with promises of attribution or traffic, framed as partnerships rather than just transactions. For example, deals like OpenAI’s with Le Monde and FT include features where ChatGPT will cite and link to original articles, ostensibly driving readers back to publishers. This kind of “AI meets publisher” collaboration helps soothe industry fears and demonstrates a more ethical, consent-based model of AI development. There are also reputational stakes: as public awareness grows, AI companies risk backlash if seen as exploiting creators. Adopting a licensing approach can be part of responsible AI practices, showing respect for IP and artist rights. Even within AI firms, there’s recognition that cooperating with content creators is vital for long-term sustainability (e.g. OpenAI’s content policy shifts to allow site owners to opt out of its crawler). So, industry norms are evolving. What was common scraping practice a few years ago is now increasingly viewed as improper unless sanctioned. Tech companies that lead in embracing licensing may influence others and potentially help stave off harsher regulation by demonstrating goodwill.
- Data scarcity and quality needs: Another factor is the sheer demand for more training data and the fact that easy pickings from the open web might diminish over time. Analysts warn that AI companies could “exhaust publicly available data for LLM training at the turn of the decade”, given the current exponential growth in model size and data requirements. Once the freely accessible text is mostly tapped, previously unused sources must be tapped, and many of those are behind licensing barriers (books, archives, etc.). Forward-looking AI developers are preparing for this by establishing content pipelines via licenses. Additionally, as models improve, simply adding more data is not as effective as adding better data. This incentivises curation of high-quality, diverse datasets, often requiring access to professionally produced content (journalism, literature, scientific writing) that isn’t freely available. All these strategic needs reinforce the move to licensing as the path to obtain reliable, rich training data in the long run.
A combination of legal pressure (copyright enforcement), regulatory trends, financial calculus, industry norms, and data needs is driving the shift. Especially in the U.S. and EU, where publishers and creators have strong voice, these factors converge to make licensed content usage the prudent course for LLM development going forward.
We are witnessing the foundation of a new content licensing industry for AI, where deals between AI firms and content owners become commonplace.
How much content remains untapped?
Despite the billions of words already ingested by GPT-style models, there remains a vast amount of high-quality content not yet used (or only minimally used) in LLM training. This includes whole genres of copyrighted text and data that have been either off-limits or technically out of reach.
Below, we estimate how much valuable content is still untapped, identify major categories of such content, and explain why each remains underused by AI.
This is just a drop in the bucket – over 100 million books have been published globally in modern history, meaning millions of book titles (the vast majority) were not included in any disclosed training set.
The enormous scale of unused content
It is challenging to quantify precisely how much new text data remains unused, but several indicators show that the reserve of high-quality content is enormous.
Consider books: an infamous bootleg dataset called Books3 contained about 183,000 pirated e-books, and was used (without approval) in some AI research models.
Yet this is just a drop in the bucket – over 100 million books have been published globally in modern history, meaning millions of book titles (the vast majority) were not included in that dataset or in any disclosed training set.
Even if some AI developers covertly scraped more books, it’s clear that a huge volume of literature remains untouched by current LLMs.
Similarly, in scholarly and professional literature, only a fraction has been used. There are over 50–100 million academic journal articles in existence (with 3+ million new articles published each year) but LLM training data has largely come from openly available subsets (such as arXiv preprints or open access journals).
The bulk of peer-reviewed articles behind paywalls (spanning medicine, science, humanities, law) likely remain absent or under-represented in training corpora.
Major scientific publishers have not broadly opened their vaults to AI (barring specific deals), so this content is still “dark matter” to models.
Much more archives remain untouched
Another reservoir is news and magazine archives. While models like GPT-4 have read some online news, they have not necessarily been trained on full historical archives of leading newspapers or magazines (which often require expensive licenses or are not digitised for open web).
For example, the New York Times has archives going back to the 1850s; the Times (London) to the 1780s; countless local newspapers have century-plus archives; a rich record of human events and language.
Much of this is high-quality edited content that is not freely accessible on the web (often behind archive paywalls or library systems) and thus not in common datasets.
User-generated content also contains untapped pockets
Although forums like Reddit and Wikipedia were heavily used in training, other platforms were not. Reddit itself moved to restrict free access in 2023, so newer Reddit content may be unused unless licensed (hence Google’s paid deal to get it).
Twitter (now X) similarly cut off free API access, meaning the vast corpus of tweets is largely inaccessible to new AI models unless through a paid arrangement.
Private or semi-private communities (Facebook posts, private subforums, etc.) are of course mostly off-limits.
Even government and legal documents, many of which are public domain, can be missed because they aren’t compiled in convenient machine-readable form (for example, court transcripts, legislative histories, or government archives that are not web-indexed).
In terms of sheer volume: one estimate suggests that by 2024, AI developers have already consumed a huge portion of the easily available web text, and they “are estimated to exhaust publicly available data for LLM training at the turn of the decade”.
What remains unused is not so much random text as it is valuable text locked behind barriers.
In our next research piece we will examine just how much valuable content and data has not yet been used by LLMs and the reasons why it hasn’t yet been used.
© 2024 Contenseo. All rights reserved.