This is basically the same thing Mithical already said, and it is the first thing that came to mind as soon as I started to read your very long post. That said, I am posting this version to give a sightly different approach to the issue.
Your assumptions so far:
- ChatGPT detection tool have an unacceptable error rate. References:
1- We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives
2- Automated GPT detectors have unusable error rates on the platform - this very post - Moderators own judgment has an unacceptable error rate too. References: multiple moderators pointing out that the private version of the policy that very conveniently for you the userbase will never see doesn't just forbid the use of tools. Since I would not be surprised to see retaliation actions against those individuals you won't get a list here.
Logical conclusions from your premises:
- Since neither automated or human made judgment works, it is impossible to determine if any post is the product of automated generation without the author disclosing that.
Your next claims:
- The company was able to measure the rate of generated content on the site.
- The company was able to measure the rate of false positive bans on the site.
Both claims seem to be founded in the strategy you used to detect an AI generated post.
This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange.
On paper, this may work. There are two problems though.
- this idea, which based on your claims work better than any judgement the brains of more than an hundred mods managed to do, is coincidentally the first thing that came to mind when I was trying to imagine what the tells-by of a copy-and-paste post could be. I therefore assume that unless I am far smarter than you, the mods and everyone else that is also what most will have though of too, if anything else because this is exactly how the "are you an human" check usually work on sites: is someone is making some action too fast, they are probably not an human.
- mods have been telling you that quite often users posting generated content try to edit it before posting to remove the more blatant commonly seen recognizable signs of automated generation (for example the usual final line that often sound like "it is worth to notice that the above is just ... and not an accurate ....") and that obviously increases the time spent on a post.
For these two reasons alone, I am not really convinced that your date provide much more value than the mods own analysis and actually imho look like a pretty ingenue oversimplification that tries to proof causality ignoring a tons of other variables but it is still better than no data.
I therefore once again propose you an experiment: take one of the posts that based on your analysis should be a false positive, identify the mod that claimed it was a real positive, post an anonymized version of the post so that users can also see what the content looked like and then have both parties explain how they came to their conclusions.
Corollary
I appreciate the shift in tone in this post. Sadly, I still have to consider this as a part of a bigger picture so I still find important to write the following.
It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate.
The sentiment of this passage does not align at all with your actions. Once again you immediately jumped on the wagon to post what your own voluntary moderators apparently see as a direct disparaging of their work to a media press site. Those post were not removed or rectified after multiple request to do so and therefore you current word feel as empty as they can possibly be. You don't know of to start to rectify the misrepresentation you gave because "once on the internet, it is forever"? YOUR PROBLEM.
Also, while we are at it, this is not the first time this happened. Remember someone called... Monica Cellio? How many years ago did that happen again?
Wonder what? That [redacted] article... it is still there.. Lucky, apparently the Register had the heart to follow up on that story, but neither article seems something the company requested to clarify things. It is my understanding therefore that you were perfectly fine with never taking action to tell the press that you were wrong.
- Flak overflow: Barrage of criticism prompts very public Stack Overflow apology
- Stack Overflow makes peace with ousted moderator, wants to start New Year with 2020 vision on codes of conduct
I therefore can't help but ask: how is that every time some "mishap" of this size happens we end up with an article posted somewhere that manages to portrait the company as the Heroes Of Light (tm) that fight for a better world against the evil forces of the Malevolent Community of Evils (tm)? I seriously doubt that the Register or any other site spends its day looking at Stack just to post a new story, so I assume that YOU are the ones to contact them and request an article, not the opposite. This obviously opens another question: what do you hope to gain from those coincidentally wrong representations of the issue at hand and the mods?