5
$\begingroup$

In February 2024, an article explaining the site's policy regarding the use of content derived from generative artificial intelligence tools (GenAI) was added to the Help center pages of every Stack Exchange site. This policy covers both answers and questions that use GenAI content. As explained on Meta Stack Exchange Proposed changes to Help Center articles to include mention of AI-generated content policy, sites were given the option of banning all such content, or of permitting the use of some GenAI content, if it's properly attributed. Moderators can customise this Help center article, if desired.

Previous discussions here on Physics meta made it quite clear that our community is generally not in favour of GenAI content. The main pages containing these discussions are Please don't use computer-generated text for questions or answers on Physics and Physics.SE remains a site by humans, for humans. Also see If I see a question mentioning an AI as the source, should I modify it?.

Here's the link to that Help centre article:
What is this site’s policy on content generated by generative artificial intelligence tools?
It commences:

Generative artificial intelligence (a.k.a. GPT, LLM, generative AI, genAI) tools can be used to generate content for Physics Stack Exchange, but this content must be properly referenced as per our guidance

I must admit that I was quite surprised when I first discovered that Help page. In light of the previous discussions, I assumed that Physics.SE would opt for the "No GenAI" version of that Help page. I suspect that many members of our community also assume that GenAI content is not permitted here, or is at least strongly discouraged.

It's well-known that Stack Overflow does not permit answers derived from GenAI. The meta SO page Policy: Generative AI (e.g., ChatGPT) is banned has over 1.4 million views. However, even on SO questions arising from GenAI interactions are not necessarily banned. See Should we flag human-written questions that use code generated by ChatGPT?. That answer was written by Martijn Pieters when he was still a moderator. Admittedly, Martijn's answer generated some controversy...

Stack Overflow was the first site in the network to implement and enforce a GenAI policy. Within days of the release of ChatGPT, people started posting GenAI answers on SO, often without checking the validity of those answers. Within a week, the site was getting flooded with such content, and the moderators there needed to act quickly. An outright ban seemed like the only practical solution.

Our policy here on Physics.SE has taken a somewhat different approach. We haven't been flooded with GenAI content, and our moderators felt that we don't need to ban content purely on the grounds that it was produced using GenAI. Instead, we can simply judge such content on its own merits. A good Physics.SE answer must support the statements it makes. We expect content here to cite trustworthy references, and to give proper attribution to all material created by others. Current GenAI tools have been trained on vast bodies of text, but they don't actually "remember" the sources of the words they regurgitate. They simply produce sequences of tokens that are statistically consistent with the token sequences in the training data. (Some recent tools do attempt to give references, but they're basically just guesswork. They simply do a search through Web data looking for a near match to the token sequence they just generated).

GenAI tools function by transforming token sequences. They don't contain a mechanism for verifying the truthfulness or logical consistency of those sequences. And they certainly don't attempt to directly model physics. They can produce text that may appear reasonable at first glance, but which falls apart under expert analysis.

Under our current policy, we don't need to ban such text merely because it comes from GenAI: we can reject it simply because it's low quality. It states alleged facts with no support. And any factual statements that it does manage to generate do not cite their origins.

Is this policy adequate? Or should we have a stricter "no GenAI" policy?

We don't actually get many GenAI answers on this site. And people posting such answers often add valid references to them, or at least Wikipedia links which cite valid references.

However, we occasionally get questions that mention the use of GenAI. Typically, the OP asked a GenAI their question, and they're asking us to verify the GenAI answer. Or the answer they got was just confusing, and they want us to explain it. Should we permit such questions?

I'm happy to help clear up misconceptions that people have regarding physics, no matter how they arose. On the other hand, I have zero desire to explain why a LLM may have concocted some piece of nonsense.

Is our current GenAI policy Help page adequate? Does it need any amendments? Or should we opt for a strict "no GenAI" page? Should we permit questions that mention the use of GenAI? If so, should there be any restrictions on the form of those questions?

$\endgroup$
5
  • $\begingroup$ I think you're reading rob's and my meta post as stricter than they were intended - from my point of view we were mostly concerned about the undisclosed use of LLMs, or answers that consist entirely of LLM output, which, even when referenced, break the rule that answers should not consist entirely of copied content. This is consistent with the current guidance in the help center. I struggle to conceive a scenario where the difference between "don't use LLMs without reference" and "don't use LLMs" would really matter on our site - do you have concrete examples that triggered this concern? $\endgroup$ Commented Oct 1, 2024 at 10:13
  • $\begingroup$ @ACuriousMind Here's a recent question that mentioned ChatGPT: physics.stackexchange.com/q/829524/123208 and it was downvoted & closed on that basis. The ChatGPT references have now been removed, but please see the edit history. $\endgroup$ Commented Oct 1, 2024 at 10:17
  • 1
    $\begingroup$ I have seen several other questions that involved some form of GenAI interaction, and they all got "ChatGPT isn't allowed here" comments, and a general negative reception. Sorry, I don't have links for those questions. $\endgroup$ Commented Oct 1, 2024 at 10:22
  • $\begingroup$ As I see it, the problem isn't really that the kind of question you linked "uses ChatGPT", it's that it was a "please fact check this" question, which as "check my work" is off-topic regardless of the source of the work. So the closure was justified even though the closure comment wasn't accurate. I think this will be the case for almost all these cases: The question is bad or off-topic in and of itself, and the usage of an LLM is really only a tangential issue. $\endgroup$ Commented Oct 1, 2024 at 10:29
  • $\begingroup$ @ACuriousMind IMHO, the "fact check" aspect is secondary. Ultimately, the OP wants to know how much energy it takes to turn something into a black hole, but they couldn't find that info, except by using ChatGPT. I tried to encourage the OP to convert the question from a "fact check" to a conceptual question, but they haven't responded since they removed all the ChatGPT stuff. $\endgroup$ Commented Oct 1, 2024 at 10:35

2 Answers 2

Reset to default
17
$\begingroup$

This is an answer from my personal perspective, which I know differs from that of many others. Note, my answer is specific to current (2024) Large Language Models. It is possible that future versions of AI will overcome the specific primary deficit of current LLM's, in which case this could be re-evaluated.

I wholly oppose the use of any current LLM for generating content here, either in questions or in answers. LLM's are, as the name suggest, language models. A modern LLM includes parts that model vocabulary, grammar, style, and organization. They have no corresponding fact model, and the facts included in the generated language are not reliably correct. But even with completely hallucinatory facts, the language is confident sounding and convincingly structured.

LLM content usually fails many other rules as well. It is often non-mainstream, frequently unclear (due to inconsistent facts), typically factually wrong, or a "check my work" problem. So, in principle, a specific LLM stance is not necessary. We could characterize each individual use of LLM, identify the specific failure, and respond accordingly. Indeed, as many posters will use a LLM without disclosing its use, this may be the primary LLM policy that actually occurs.

However, I think that a specific LLM policy is still valuable for the following two reasons.

  1. When someone posts LLM content to this site, they do so under the mistaken belief that it is credible. In our information-saturated world one of the most important skills is evaluating the credibility of a source. We do a person a disservice if we imply that LLM content is a credible source of physics.

  2. LLM's are fast. They can generate nonsense far faster than humans can read, evaluate as nonsense, and refute. With our human speed limits, we need to discourage people from bringing LLM content here. A policy that is clearly stated and consistently followed is one way of doing so.

If I could simply write the policy, it would be clear and straightforward. No LLM generated content should be permitted at this time on PSE, neither in a question nor in an answer. LLM usage is sufficient grounds for closure. LLM's are not credible sources for any factual topic, and responding to their specific factual failings in a given instance is a fools-errand as no human can possibly keep pace. For these reasons, it is in the best interest of both the person asking and the people answering to wholly reject such content.

If the question or answer is otherwise valid then it should be posted without the LLM content.

$\endgroup$
3
  • $\begingroup$ To clarify: where is the line for "LLM usage" for you? If someone says "I tried asking ChatGPT but I didn't understand what it said", without mentioning its output, would that be sufficient for closure? Or would they have to actually include the output in their question? $\endgroup$ Commented Oct 3, 2024 at 17:43
  • 2
    $\begingroup$ @MichaelSeifert I believe that I was already pretty clear. The issue for me is LLM content. If they mentioned LLM's without posting LLM content, then I would mention that LLM's are a bad idea in physics, but not vote for closure. $\endgroup$ Commented Oct 3, 2024 at 18:40
  • 1
    $\begingroup$ I'm not sure why this change didn't get made in a timely way, but the help center text has now been changed. $\endgroup$ Commented Sep 7 at 21:12
7
$\begingroup$

Thanks for facilitating this discussion; you've done a lovely job of summarizing the issue. I think you have adequately captured my opinion here:

Under our current policy, we don't need to ban such text merely because it comes from GenAI: we can reject it simply because it's low quality.

This was really the heart of our original "please don't" Meta post about computer-generated text. I consider it parallel to our policy on homework-like questions: we don't care whether the question is from an assignment, or from self-study, or just from curiosity. There is a class of questions which would make good homework problems, but which are a poor fit for our community, because the "answer" to the question is some boring number that's only relevant to a hypothetical situation.

Likewise, obvious chatbot output is a big red flag for low-quality content. I don't think we need special rules for special kinds of garbage; we don't tolerate low-quality content in general.

Note that the help page is editable by the moderators; we can include as much or as little nuance as we decide is appropriate. We could, for instance, clarify near the top of the guidance that, while chatbot outputs are permitted under the described restrictions, such output is strongly discouraged.

$\endgroup$
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.