Fabricated data in posts.xml for multiple/all data dumps

This is a response to the official answer, provided by Berthold.

As some discussion noted, yes, this is a watermark on the data dump.

Although watermarking content is permissable, it's not clear what the value is.

We can consider the data dump, as a whole, a creative work. It is a compilation of many other creative works, with some light filtering (such as removal of deleted posts) applied.

The Wikimedia Commons has a page discussing watermarks in the Commons content. Since Commons content and Stack Exchange content is similar in a number of regards, such as licensing and intended uses, it makes sense to look at what other organizations are doing and saying.

They point out that certain types of visible watermarks, even when they do not obstruct the use of work, are discouraged because "they detract from the usability of a work". This is true of the application of this watermark to the data dump content.

Bugs and pending feature requests aside, this watermark prevents people from downloading and using the data dump for its intended purpose without additional work. Any kind of analysis or use of the data (such as creating an alternative mirror) must consider/handle these watermarks, or else the data would not reflect the actual state of the public network.

When we made the decision to include these posts with security and safety in mind, we knew this would get caught quickly by the community, and that was intentional; this was not meant to be sneaky! We were waiting for you all to find it.

As a reminder, the last time the data dumps were messed with, the changes were a contributing factor to a moderation strike across the network. Do you really believe that it is a good idea to silently make changes without having any kind of discussion? As a result of the moderation strike, there was a commitment to the survival of the data dumps. Making it harder to use the data dumps does not help them survive.

I'd also point out that "gathering community feedback before committing to a major change to the platform" was another result of the moderation strike, yet there was no attempt to discuss, much less inform, before this change. When it was first noticed, people spent a not-insignificant amount of time assessing the scope of this change and checking past data dumps. This isn't a sign of transparency or respect for the people who maintain the platform.

Although something was caught quickly, I think there is a massive underestimation of the time spent for people to download, uncompress, and analyze not only this most recent data dump, but past data dumps to understand what has changed. There was also waiting for an official answer. Many, many hours of people's time and computational resources were spent on this.

I'd also disagree that this was not meant to be sneaky. If it wasn't, why not announce it before (or concurrently with) the data dumps? Or why wait so long to respond to this question about it? No, I do think the intent was to be sneaky, but some people care a lot and want to make sure the data is good and available.

The posts are there to discourage reuse, but are added in a way that is clear to the community.

How exactly does this discourage reuse, since the watermark is so trivial to remove now that it has been identified? Since the data dump is CC BY-SA, there's nothing stopping anyone from fixing the data dump and reuploading it to another source, where it's actually usable and no longer has the anti-LLM agreement. Although, I'd refer back to my previous points - given how the community is treated, would anyone invest the time?

I also don't understand why there would be a reason to discourage reuse. The data dumps are made available under CC BY-SA. The purpose of this license is to encourage other people to share and adapt the material for other uses and in other contexts. It's why many of us contribute to the public network in the first place.

---

I'd strongly consider the company to reconsider.

This is on purpose

The Company has intentionally added two bogus posts to every Data Dump export for every Network website. The Company has declined to comment on the presence of these two posts, and has not explained their existence. However, with some detective work, it becomes rather obvious that this is an intentional addition.

These two posts will have the following characteristics:

• One post is a Question (Post Type = 1), published by the Community User (User ID = -1).
• One post is an Answer (Post Type = 2), published by the Community User (User ID = -1).
• The Bogus Question will be Post ID 1000000001.
• The Bogus Answer will be Post ID 1000000010.
• The Title and Body will vary for each network site. However, in all cases, the bogus post will contain factual inaccuracies, typos, or other nonsense that identifies it as a work of fiction, nearly immediately.
  • The writing is consistently so poor/incorrect that it makes Generative AI look well-researched & carefully proofread in comparison.
• These posts are NOT present in SEDE, which is the source of the Data Dump.

What to do?

If you are consuming the Data Dump, you should explicitly exclude posts with IDs 1000000001 & 1000000010. Post IDs are autonomically increasing ID values, beginning at 1 and ticking up for each new post. A number of scenarios will result in numbers being "skipped," and never used. However, the gigantic gap indicates that these numbers were chosen intentionally to avoid a conflict with real data that is being generated.

More generally, if there are a small number of posts with IDs that are order-of-magnitude larger than the rest, then those posts should be considered suspicious, and filter them out.

But why the heck are they doing this?

The Data Dump was unchanged for years until the Generative AI boom resulted in significant impact to Stack Overflow and the public network. At that time, the company began making changes to the Data Dump, first to attempt to end its public distribution, then the Company began selling/monetizing site data to LLM producers, then to change the posting location to a "walled garden" and attempt to add additional terms to the download preventing use for training AI models, and now again to begin injecting these bogus posts.

The company has not explained why they are doing this, but the reasoning seems to be a hamfisted attempt to set a "honeypot" to find people using the data for commercial services, if they have not paid Stack Overflow to license the data. Presumably, the company is monitoring for web traffic hitting the 404 errors for any user trying to hit the various slug formats like /q/1000000001 for the 1000000001 and 1000000010 post IDs. Additionally, they may have other monitoring looking for information related to the use of the fictional products, URLs, etc from the content itself.

Speed Trap without a Speed Limit

Even though the company has introduced their "honeypot" trap, the enforcement mechanism is unknown, and what they are enforcing is wholly unenforceable. Here's why:

I obtained my copy of the data dump from the open internet, not from Stack Overflow. The Data Dump is officially referred to as the "Creative Commons Data Dump" in Section 6 of the site TOS. The Terms of Service explicitly declare the Data Dump to be covered by the same CC BY-SA license as the posts themselves.

From time to time, Stack Overflow may make available compilations of all the Subscriber Content on the public Network (the “Creative Commons Data Dump”). The Creative Commons Data Dump is licensed under the CC BY-SA license. By downloading the Creative Commons Data Dump, you agree to be bound by the terms of that license.

Last year, when the Company moved to self-hosting the Data Dump, they added this controversial checkbox to the Data Dump download page:

I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), and that should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump.

However, they did not change the license under which the data dump is licensed. The license.txt contained in the download is as thus:

All content contributed to Stack Exchange sites is licensed under the Creative Commons CC BY-SA license (various versions, including 2.5, 3.0, and 4.0). We also provide data for non-beta sites as part of the data dump, which is licensed as a whole under CC BY-SA 4.0: https://creativecommons.org/licenses/by-sa/4.0/ Some of the content may have initially been contributed under earlier versions of the license (2.5 or 3.0): https://creativecommons.org/licenses/by-sa/2.5/ https://creativecommons.org/licenses/by-sa/3.0/ The CC BY-SA licensing, while intentionally permissive, does require attribution: Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). If you republish this content, we require that you: 1. Visually indicate that the content is from the Stack Exchange site it had originated from in some way. 2. Hyperlink directly to the original question on the source site (e.g. https://stackoverflow.com/questions/12345). 3. Show the author name for every question and answer. 4. Hyperlink each author name directly back to their user profile page on the source site (e.g. https://stackoverflow.com/users/123/username). By "directly," we mean each hyperlink must point directly to our domain in standard HTML visible even with JavaScript disabled, and not use a tinyurl or any other form of obfuscation or redirection. Furthermore, the links cannot be marked with the nofollow attribute. This is about the spirit of fair attribution: to the website and, more importantly, to the individuals who generously contributed their time and knowledge to create that content in the first place. 

Thus -- one can obtain the data dump from elsewhere without agreeing to the additional LLM-prohibiting terms, which apply only to the user who actively downloads from the website.

Each quarter, someone downloads the data dump from Stack Overflow, and re-posts those dumps on the Internet Archive, for the purpose of archival, and promoting open data. This is a perfectly allowed use under both the CC BY-SA license, and the additional terms imposed upon the downloader by that checkbox. The Internet Archive hosted binary-identical copy continues to be licensed under the CC BY-SA license, but the "checkbox terms" are not applicable to the downloaded data dump.

If someone wants to skirt the "checkbox terms" they just need to download the dump indirectly (i.e., via a mirror like the Internet Archive).

If I choose to use, remix, and attribute the data dump per the license (ie, by attempting to link to the fabricated Q&A), I am fully compliant with all the data licensing. If my project is an LLM, let us look at what that might mean.

• Scenario A - Because I am linking to the question to provide attribution required by the license, then I am following all the legal provisions of the license. Additionally, my LLM would qualify as "Ethical AI" under the definitions used by the company, and be a type of LLM that the Company has claimed support for.
• Scenario B - If I do not provide attribution, then my LLM is both in violation of the license terms, and what Stack Overflow has deemed "unethical," and is the type of LLM that they claim to be working to combat. However, because this scenario would not be linking to 404'ed fabricated posts, I'm not sure how the Company intends to use these honeypot questions to find someone.

Poison Data, not a honeypot

The only purpose of this seems to be to reduce the quality of the data for users of the Data Dump. This adds a step to the Data Dump, requiring legitimate users to trim out the fabricated data before they use it. It is unlikely, but not impossible, that the single Q&A could cause problems for applications or research that use the data dump for legitimate purposes.

Poorly executed moneygrab

The Company began selling the Data Dump to LLM-providers (OpenAI and Google) were both promoted in press releases and as "Responsible AI partners" on the Company Partnership Page, At the OpenSaaS conference in early 2024, Stack Overflow CEO Prashanth Chandrasekar talks about selling data to Google Gemini, and OpenAI. He also talks about how all the big AI companies are interested in buying the "Overflow AI" product, which includes both the Data Dump, and API access. Based on the Prosus Annual Report which mentions "Stack Overflow...significantly reduced losses by US$65m to US$33m". When combined with other statements by Company Leadership, and analyst analysis, this suggests these data dump sales representing as much as half of the company's Annual Recurring Revenue (ARR), exceeding the revenue actualized from the product launches of the Enterprise & Teams products.

Paid copies of the Data Dump available through "Overflow AI" either do not contain fabricated data, or the data is included as a documented "example" that those customers can exclude.

The company has not responded to this post, so it is unclear what other changes the company has made to dilute the value of the data dump by altering the accuracy of the data.

This is a response to the second official answer, provided by Bella_Blue.

---

These posts were never hidden or intended to deceive the community—in fact, as some of you have already noted, they were specifically crafted to be easily spotted by the attentive eyes of our knowledgeable users.

I disagree that the posts were never hidden. There was no discussion that I'm aware of with stakeholders, public or private. Even without disclosing the specific measures, there was no indication that any "safety and security" measures would be implemented in the data dumps.

Some controls were already implemented. For example, the data dumps are no longer automatically provided off-network. Users must have a profile on the site they are downloading the data dump from, and there is still no multi-site or network-wide access. I'm assuming that this means that there is a log of who is downloading the data dumps and when. Given that an agreement also gates the download, it seems possible to prevent a single individual from accessing the data dump if they misuse it. It's not clear what value these additional watermarks offer beyond this, especially since finding and defeating them is now public and there is no basis to assert that they ever added any kind of security.

We also shouldn't need "the attentive eyes of our knowledgeable users" on the data dumps, especially considering who and what the data dumps are for. The people using these data dumps may be researchers who have minimal familiarity with the network. They may be people trying to collect open-source knowledge and create new repositories. They are also a failsafe should the network ever go offline. For these people to have high-quality data sets to work with, you shouldn't have to rely on knowledgeable users checking for quality issues and raising concerns.

This decision was in keeping with our ongoing efforts to safeguard the data against misuse, such as non-compliant use of CC-BY-SA content by large companies, while ensuring that those engaging with the data in good faith could immediately recognize the watermark.

I find this to be very disingenuous, especially considering the company's past stances on the misuse of data harvested from the network.

In 2021, the company changed its stance on how to handle scrapers. Prior to 20 October 2021, users could report sites that aren't following attribution requirements or scraped sites that claim ownership of the license. After 20 October 2021, only proxies should be reported. This is a valid change. As contributors and creators of creative works, we have never given the company authority to act as our agents and enforce our copyright protections and licenses.

If the company really wanted to help with compliant use of CC BY-SA content, then they'd obtain permission to become agents (and non-exclusive agents, at that) and address everything from scraper sites to the YouTube channels scraping and monetizing SE content to people abusing the API and/or data dumps and/or SEDE.

The data dump is a separate work that can be licensed. This is a case that is addressed in the Creative Commons FAQ. Although the original material is available under one of the Creative Commons Attribution-ShareAlike licenses (depending on when it was posted or last edited), the data dump could be under a different license entirely. I do struggle to understand the value in doing this, though, since the individual pieces must be CC BY-SA and they can be accessed and used under that license via the data dump (there may be a good question for Law, or perhaps Open Source, if one doesn't already exist).

I'd also point out that because the data dump is licensed CC BY-SA, we, in the community, can make derivative works that fix issues like this and share them via other means. This will bypass both the prohibition on LLM training and allowing people to have clean, usable data sets. However, I'd also point out that decisions like this that mess with the data dump damage any kind of trust with the community and people who may want to offer this service to the broader world.

I'm curious about how the company defines "non-compliant use of CC-BY-SA content" and what justifies its enforcement in some cases but not others. It seems to be inconsistent with my understanding of the license, the general nature and freedoms associated with open source (which have gone far beyond software and code), and the past stances of the company when it comes to helping authors protect their intellectual property.

This approach aligns with our previously shared perspectives on the evolving state of the internet and how it impacts business practices, which are detailed in this blog post: The Changing State of the Internet and Related Business Models.

I don't think this business model takes into account the nature of open-source licenses and why people contribute to places like the public network. It also seems like it's ignoring the current state of the law and legal proceedings in the United States (living in the US, I'm watching what is happening here more closely than in other countries), where at least two cases have decided that training AI models is fair use. However, we're still waiting on decisions about the output of trained models regurgitating content.

Just because the company wishes something to be true doesn't mean that it is or that those wishes align with the expectations of the volunteer contributors and curators.

Ultimately, our commitment to the agreements governing the data dumps remains steadfast. While we recognize that some may interpret our actions differently, we want to affirm that there was no intent to mislead or act contrary to community trust. Instead, these measures were implemented with the overarching goal of protecting the integrity of the data and reinforcing its intended use within the community.

We recognize that the Data Dump is a core issue that matters to the community, and we deeply value your continued feedback. Rebuilding trust in us as responsible stewards of the network remains our top priority.

So what are you going to do about it?

The reported addition of nonsense to the data dumps is already strange but the official reaction is even more bizarre. I would have expected an apology for the inconvenience and violating the integrity of the data dump, instead it was all some sort of game to waste some time for us as if we would have too much of it.

There is no reason to watermark the data dumps beyond providing hash values, nobody's safety has been increased and the additions should be removed again. The data dumps should only contain what is needed. Specifically they should not contain anything that isn't also part of the internal databases.

I'm disappointed.