31
\$\begingroup\$

I'm Grace Note, a Community Manager at Stack Exchange.

We have a process for how sites go about graduating or sticking around in Beta, but as noted in the comments there, Code Golf is one of two sites that we do not have periodic "Site Evaluations" where we compare questions against the rest of the internet. Instead, we have silent internal reviews every now and then.

I've been asked by various folks, especially after recent attention given to the sister site Code Review (who equally has the same lack of periodic evaluations), how Code Golf is doing.

The internal review back in October I did, there were two points I brought up - the traffic had been pretty stagnant for three years, and the userbase was dedicated but rep distribution was a mess. In fact the site was... pretty mediocre in just about every department. The prospects were very much the opposite of positive.

Things have changed.

seems to be something to point at - you guys received a ginormous traffic boost from the birth of this tag. Prior to Christmas, the site's average traffic levels were, like, 500-1000 for the whole thousand plus days of lifetime. Then on December 28th, you guys jumped by over a factor of 30, and since then throughout January and February you guys have been maintaining an average closer to 5000 visits per day. By comparison, Code Review is also at that level and they're pretty happily marching on from that as well. This is a nice thing to see and we'll probably want to watch this for some more time to see how well this is sticking, but so far it is really sticking.

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already. The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot. The traffic and answer activity levels are great - I think question income could do some notches higher but we're still on a great rise from before.

On the user side of things, I pointed out this answer as sharing my concerns - there was a major issue with voting. Code Review has also had this issue and they're really powering through it, though they still have some distance to go.

To my surprise, wow you guys made a huge shift here. In these four months, six people crowned over 10,000 reputation, and you've got a very nice collection of users in the higher reputation levels. The rep distribution is also nice, and you have 14% of users between 500 and 1000 reputation, and an additional 14% of users above 1000 reputation. These are mostly still-active users, too, so there's a lot of engaged activity. Your meta activity has also exploded into one of the more active metas in recent weeks.

This is a site that lives off of engagement - whereas other sites deal with producing solutions to problems, this site's competitive nature means that it thrives when it has people actively engaging against each other. And that seems to have started to really kick up. The userbase size is something that could be said to be low, but not in a number that is problematic or anything.

The overall scenario is extremely positive. What was once a site that may've languished and sat in Beta status indefinitely, has returned with extreme strength and dedication to their task. All in all, much like I am extremely pleased with how Code Review shaped up when spurred to action, I'm very delighted at what this community has brought on their own to improve and beyond improve their progress.

\$\endgroup\$
19
  • 2
    \$\begingroup\$ Yay, thank you for this :-D now I don't have to pester you on posts and in chat! :-P So anyway, what could we be doing better? I see your two major concerns (voting and new challenges) and also that we're working on fixing them; should we just keep it up? Or is there anything else we should do? \$\endgroup\$ Commented Mar 6, 2014 at 16:07
  • \$\begingroup\$ Those were concerns in the past - not so much now (though obviously we don't want to just forget and laze on those matters as if it's senior year). For now, the main thing to do is keep up the good work. \$\endgroup\$ Commented Mar 6, 2014 at 16:11
  • 16
    \$\begingroup\$ Good work? Surely you joke? You don't seem to realise that the quality of this site has plummeted since the travesty that is [code-trolling] was unleashed. There are now just as many good questions as there were previously; the difference is that they get hidden amongst the extra trash - the seemingly endless rehashes of 'Hello, World!', quines, and 'print x without using y'. It makes me despair every time I visit. \$\endgroup\$ Commented Mar 6, 2014 at 17:47
  • 1
    \$\begingroup\$ @Gareth Are rehashes of trivial tasks really a new thing around here? Even in the [code-golf] tag, there are a bunch of quine/hello-world types, and I can't imagine they're all really different enough to warrant separate questions. The main difference I see is that now they come in faster due to high traffic, but rehashes seem to be the inherent nature of a "post a puzzle" site. Just vote to close them as dupes, just as we do on SO when someone posts the millionth "Why don't these strings equal each other?" question. \$\endgroup\$ Commented Mar 6, 2014 at 19:02
  • 4
    \$\begingroup\$ @Geobits No they're not new, they're just a lot more common since December. We do vote to close, but there are a number of users who instantly vote to reopen anything just to be contrarians. These are probably the people who earned lots of rep posting answers like this or this. \$\endgroup\$ Commented Mar 6, 2014 at 22:36
  • 1
    \$\begingroup\$ @Gareth That's just due to more traffic, though, which is generally a good thing. Laying the blame on a single tag/post/incident seems naive, and I think any large increase in traffic is going to cause more low-quality posts, simply because you get a lot of new people who aren't familiar with the rules. I like the way Shog9 put it when people complained about "Hot" traffic on MSO: "You can work to fix that - as painful as that process is - or you can bury your head in the sand and blame it on all of those stupid people from elsewhere." \$\endgroup\$ Commented Mar 6, 2014 at 22:44
  • \$\begingroup\$ FWIW, I see at least a couple posts on the front page right now that should probably(IMO) be closed, yet have zero close votes. I've held back on closing so far, since I'm new here and still trying to get a feel for the community. So far it seems to me that the community just doesn't moderate as much as it could, yet everyone on meta seems to have a gripe about the post quality. \$\endgroup\$ Commented Mar 6, 2014 at 22:46
  • 1
    \$\begingroup\$ @Geobits We often can't close the rubbish questions because a lot of the 'more traffic' has gained enough reputation off the back of answers to rubbish no-intelligence-necessary questions to reopen them straight away. That's the problem. Please, if you see questions that should be closed, vote to close them. If you're unlucky you may get to see what I mean. \$\endgroup\$ Commented Mar 6, 2014 at 23:02
  • 5
    \$\begingroup\$ @Geobits And to make matters worse, we've now got everyone upvoting anything and everything in some misguided attempt to boost our chances of graduation. I think that one of the best things that Grace Note and the team could do for us is to guarantee that we will never graduate. Leave us in perpetual beta so that people stop artificially inflating each other's reputation to meet some imaginary targets. \$\endgroup\$ Commented Mar 6, 2014 at 23:05
  • 4
    \$\begingroup\$ @Gareth That was actually my reaction when I saw that thread, too. If anything, the site needs more downvoting, but then you see things like "upvoted to counter unfair downvote", and you wonder why you bother. The problem with perpetual beta, though, is that it keeps the VTC privilege at 500, where anyone can get it after basically no time. \$\endgroup\$ Commented Mar 6, 2014 at 23:07
  • \$\begingroup\$ @Geobits, it was unfortunate that the large increase in traffic coincided with a time when, due to the holiday, there were fewer people around to moderate. \$\endgroup\$ Commented Mar 6, 2014 at 23:54
  • 3
    \$\begingroup\$ @hosch250, can you cite any source to back up "If we don't graduate, we will probably get closed"? I've never expected PCG to leave beta, but it seems to serve its purpose of keeping code-golf off SO without graduating. \$\endgroup\$ Commented Mar 7, 2014 at 10:02
  • 1
    \$\begingroup\$ We have been looking at the concept of the "perpetual beta" for a number of sites, and while the status of it hasn't quite been set, I'm pretty confident that Code Golf isn't close to the chopping block. \$\endgroup\$ Commented Mar 7, 2014 at 11:53
  • 2
    \$\begingroup\$ Sorry to bother you yet again, but I've been thinking: seriously, why hasn't Code Golf (and Code Review by extention) graduated yet? We have no major problems, and we do have much better stats than some sites like WebApps when it graduated. (I mean really, they only had 3 closers and their highest repped user had 5011.) \$\endgroup\$ Commented Mar 20, 2014 at 23:33
  • 1
    \$\begingroup\$ Oh, and we even both have better weekly and monthly rep than Super User, one of the most active SE sites! Honestly I see no reason why we haven't graduated already. \$\endgroup\$ Commented Mar 20, 2014 at 23:40

3 Answers 3

Reset to default
17
\$\begingroup\$

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already.

That is a genuine problem. On the other hand, the site only needs one or two good questions per day, because most good questions take at least a couple of hours to answer well, and there's three years' worth of questions for newcomers to tackle. (Of course, Sturgeon's Law applies, so getting only one or two questions per day would not be good enough).

The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot.

Really? There have only been 5 questions posted in the past month, of which one is closed (and the issue of whether all of them should be deleted divides the community). From where I sit it seems that the thing which helped a lot was having a question posted on Reddit when lots of people were on holiday.

The long-term effect seems to be that we tend to have one or two questions in the "hot network questions" list, and while they tend not to be the best questions my hypothesis is that they attract enough people to also draw eyeballs onto the next "hot" question. I don't know whether the data available to you is sufficient to estimate the effect of "hot" questions in the ongoing visitor stats.

\$\endgroup\$
8
  • 1
    \$\begingroup\$ I'm hoping to see more of such innovation, sorry if that wasn't clear from my verbiage. There's only 22 in total but they represented a turn of traffic, not just for their own purposes but for questions in general. You've had more questions per day asked of all categories in the last two months than you guys have had for years. Traffic boost was in much part be because of them turning up on Hot Network Questions, but from me I still feel that the pull of actual new question types is something that will be extremely helpful going forward to maintaining growth. \$\endgroup\$ Commented Mar 6, 2014 at 16:50
  • 1
    \$\begingroup\$ I see "more questions per day asked of all categories" as supporting my contention that it wasn't the new category but the timing of the Reddit post that was relevant, but I think that we may have to agree to disagree. \$\endgroup\$ Commented Mar 6, 2014 at 16:55
  • 1
    \$\begingroup\$ I could actually provide the numeral traffics that further support your side - year-to-date, the amount of traffic incoming from Stack Overflow is an order of magnitude above the rest, followed by Ask Ubuntu (...huh?) and then Reddit. I don't want to agree to disagree because I don't think we disagree. \$\endgroup\$ Commented Mar 6, 2014 at 17:06
  • 5
    \$\begingroup\$ @GraceNote While it is true that the amount of good/ok questions increased during the last months the number of not so good questions is overwhelming. Previously it was working for a well-designed challenge if it couldn't get an answer for a few days (which was ok, because good answers to great puzzles need some time). Such a question is now buried by an avalanche of questions and answers of the hello world type. That means that new users don't find the interesting puzzles and might be encouraged to also post interesting challenges but see this site as a heap of me-too questions/answers. \$\endgroup\$ Commented Mar 7, 2014 at 9:10
  • \$\begingroup\$ @Howard maybe a more thoughtful use of tagging will help with that matter. If all the "hello world"-type questions were tagged with something like an easy tag and more complex, interesting challenge with a hard tag, they wouldn't be really buried. There seems to be clearly several "levels" of code-golfers on the site so it might be a good thing that those "easy" questions exists so that golfers with lower skills can still participate (and eventually improve their skills and participate in the harder ones). \$\endgroup\$ Commented Mar 7, 2014 at 10:02
  • \$\begingroup\$ Of course the easiness of a challenge being somewhat subjective would need ideally to be agreed on at the sandbox level. \$\endgroup\$ Commented Mar 7, 2014 at 10:04
  • 3
    \$\begingroup\$ @plannapus It's more a question of good-bad than of easy-tricky. And I don't think we can establish the tags badly worded and unclearly specified. \$\endgroup\$ Commented Mar 7, 2014 at 10:30
  • \$\begingroup\$ @Howard fair enough. \$\endgroup\$ Commented Mar 7, 2014 at 10:40
14
\$\begingroup\$

In my experience here, things are turned a bit upside down from other sites. Because making a good puzzle has turned out to be harder than solving it.

That is because--if your puzzle is going to be any good--you have to provide sample inputs and outputs. Which means most of the time you really should solve your puzzle before you post it. So you think one up, then solve it, then lay out the parameters...it's basically working as a CS professor for free. It is very unlike the other Q&A sites where the questioner doesn't do a whole lot of work relative to the answerers.

The maligned code-trolling tag, I think, is disliked because specifically it doesn't require that expected effort on the part of the asker.

It just seems way more fun to answer than to ask. Given the time to compose a creative question, you start getting disappointed with only a few answers, especially if people seem to be just solving your problem and not golfing it. Also by its very nature, there is no timeline for accepting answers. If you've invested a lot in defining a puzzle then why declare a winner if it closes your puzzle?

So there is an issue here about the fact that grafting puzzle/solution onto the SE model has a bit of a mismatch. I don't think it's correctible, which isn't a condemnation of the site in any way. It just means that it's always going to be kind of weird and policy needs to be laid out.

Thus I think it's still in beta until there's a larger amount of precedent set in policy space. But hopefully the ads will get run to drive a bit more traffic and awareness. This one cracks me up:

enter image description here

\$\endgroup\$
7
\$\begingroup\$

Let's go to the data

I've been a lurker, not a participant, for most of the time since I joined. So I think the best thing I can contribute to this discussion is use of my data-wrangling skills. Here are a few interesting queries. And bear with me, this post is beyond TL,DR

New Questions Analysis by Year/Month

This is a simple trace of each month's new question activity since 2011. There is a boom at the beginning, followed by a slow decline, a resurgence in early-to-mid-2012, another slow decline, and a recent megaboom in December, January, and February. This is just a high-level view so there's no attempt to exclude low-quality questions, anomalous questions, or anything like that. We'll get to that. What this does show is that traffic has ramped up enormously. Discussion questions: Has the 10-fold increase in new question views come at the cost of a 10-fold reduction in site quality? Has the 6- to 8-fold increase in new questions meant that 7/8 of those questions are bad?

year month AverageAnswerCount AverageNewQuestionViews NewQuestionCount NewQViewCount ---- ----- ------------------ ----------------------- ---------------- ------------- 2011 1 16 2102 50 105137 2011 2 9 976 112 109389 2011 3 10 1266 62 78492 2011 4 9 1356 52 70559 2011 5 6 843 37 31213 2011 6 8 1290 35 45184 2011 7 7 1019 29 29561 2011 8 7 1121 31 34770 2011 9 5 744 20 14899 2011 10 9 934 9 8410 2011 11 6 823 23 18932 2011 12 9 910 39 35527 2012 1 8 1701 37 62939 2012 2 10 1066 17 18136 2012 3 6 693 30 20798 2012 4 9 1195 28 33467 2012 5 9 2611 31 80966 2012 6 7 898 36 32361 2012 7 8 766 29 22227 2012 8 6 830 36 29887 2012 9 8 919 39 35844 2012 10 5 686 39 26772 2012 11 7 696 28 19507 2012 12 7 853 23 19631 2013 1 6 885 28 24807 2013 2 4 757 34 25742 2013 3 6 502 29 14572 2013 4 10 1084 34 36868 2013 5 5 598 29 17370 2013 6 6 539 18 9707 2013 7 8 597 24 14331 2013 8 6 975 35 34137 2013 9 5 466 30 13985 2013 10 6 486 34 16537 2013 11 8 590 46 27167 2013 12 12 2159 93 200826 2014 1 10 1778 205 364679 2014 2 11 1245 184 229179 2014 3 10 261 9 2355

Where does this recent boost come from?

Let's look at the same stats, but broken down by tag. This shows tags that are high in one or more of these stats, over the past 3 months. Some notable items: code-golf is alive and well as a tag, commanding 39k, 265k(!!), and 105k new question views in Dec, Jan, and Feb, and dominating the new question submissions. Code-trolling makes a big splash in December, but a much smaller one in January and falls off as a factor in February, so perhaps it is not as big a portion of the recent surge as we thought based on Grace Note's report. It's interesting how much traffic hello-world is responsible for, and busy-beaver, and restricted-source. However, much of restricted-source can be attributed to one question: the 2014 question. Or can it? .......

year month TagName AverageAnswersPerQuestion AverageNewQuestionViews NewQuestionCount NewQViewCount ---- ----- ------------------ ------------------------- ----------------------- ---------------- ------------- 2013 12 algorithm 14 10694 1 10694 2013 12 ascii-art 11 3513 5 17566 2013 12 code-challenge 13 7418 13 96439 2013 12 code-golf 8 744 53 39454 2013 12 code-trolling 38 11032 12 132384 2013 12 math 6 2852 4 11411 2013 12 popularity-contest 25 6496 21 136421 2014 1 ascii-art 16 3942 10 39429 2014 1 busy-beaver 59 10730 1 10730 2014 1 code-challenge 7 979 24 23506 2014 1 code-golf 10 2038 130 265058 2014 1 code-trolling 15 4398 5 21992 2014 1 hello-world 18 2805 7 19637 2014 1 popularity-contest 12 1723 43 74121 2014 1 random 38 4846 1 4846 2014 1 regex-golf 25 10816 1 10816 2014 1 regular-expression 6 1948 8 15585 2014 1 restricted-source 60 48557 3 145673 2014 1 string 8 1800 10 18002 2014 2 ascii-art 7 1244 9 11200 2014 2 bitwise 47 2437 1 2437 2014 2 code-golf 11 1022 105 107356 2014 2 graphical-output 12 3451 6 20707 2014 2 hello-world 35 11873 2 23747 2014 2 pi 30 2256 2 4513 2014 2 popularity-contest 14 2028 58 117676 2014 2 quine 22 5277 1 5277

But but but...what about Reddit? What about crappy popular questions like 2014?

OK, let's look at the top 5 tags for different stats over the past 3 months. But let's first assume that the big runaway success in each tag is perhaps too popular for its own good. It's an easy challenge that draws easy answers. OK, throw it away. Here's what we get:

TagName NewQuestionViews ------------------ ---------------- code-golf 278826 popularity-contest 237419 code-trolling 68359 ascii-art 46089 code-challenge 33297

Where do the eyeballs go? Code Golf, and Popularity Contest. The latter is, I know, divisive. But I theorize that it reflects that many people who look at the site do not know any of the "good for golfing" languages. Popularity contest lets them participate in something they have a hope of winning. Thought question: Could we make more Atomic Code Golf questions to downplay the need for popularity contests? Could we invent other tags that have objective criteria without implicitly favoring the languages obsessed with brevity?

TagName NewQuestions ------------------ ------------ code-golf 285 popularity-contest 124 code-challenge 55 string 25 math 23

It's notable here that people come to look at ascii-art, but when they make new questions, Math appears more often. Also, code-trolling shows similarly: fun to look at, but doesn't actually get that many questions created for it. Meanwhile, the core tags of code-golf, code-challenge, and popularity-contest show well for both views and new question growth, which seems to show that the site has a healthy core.

TagName AvgQuestionViews ------------------ ---------------- restricted-source 7724 hello-world 3238 code-trolling 3107 ascii-art 2003 popularity-contest 1914

This is interesting. Even with the wildly anomalous 2014 question excluded, restricted source questions - the questions many love to hate - draw views per question that average well above most other types of question. Ascii-art is unsurprising, because people like to look when there are pictures, but restricted source is interesting. Perhaps people just love an arbitrary challenge - "Play a Paganini caprice with an arm tied behind your back!" However, I began to suspect there might be another superstar question skewing the results, so I edited the query to require 3 or more questions per tag and got this instead:

TagName AvgQuestionViews ------------------ ---------------- hello-world 3238 code-trolling 3107 ascii-art 2003 popularity-contest 1914 sorting 1137

However so much we are tired of hello world questions, the tag has an audience.

TagName AnswersPerQuestion ------------------ ------------------ sorting 23 code-trolling 21 hello-world 20 popularity-contest 14 ascii-art 10

This one is very interesting, too. Code-trolling has an obvious appeal that probably draws lots of answers because if you're not normally the sort to troll you will likely jump at the chance to do so consequence-free and actually gain reputation. But, sorting? Surprisingly good at attracting answers.

In Conclusion

I've thought out loud here about some of what I think the data might mean, but my analysis is by no means conclusive. Please sound off and discuss what we can see here. Please modify the queries, or request modifications from me if you don't know how. I'm interested in making data like this a valuable resource for the site.

I think it is at least safe to say that the numbers show a healthy site that is growing. Not all of that growth may be in the right direction for the site, but there's a visible healthy core that remains devoted to the site's founding purpose and has exploded in size alongside any recent innovations or anomalies.

Doin' alright,
Gettin' good scores
The future's so bright..
We gotta wear plus-fours!

\$\endgroup\$
4
  • \$\begingroup\$ In answer to the question "Are 7/8 of questions bad?", yes. That's to be expected. In fact, I reckon that about 7/8 of the questions from January which haven't been deleted are at best a bit naff, so the proportion taking those into account may exceed Sturgeon's 90%. \$\endgroup\$ Commented Mar 8, 2014 at 9:25
  • \$\begingroup\$ Two versions of exploring the Sturgeon hypothesis: data.stackexchange.com/codegolf/query/173590/… OR data.stackexchange.com/codegolf/query/173591/… \$\endgroup\$ Commented Mar 8, 2014 at 13:31
  • \$\begingroup\$ To sum those queries up: only 68 out of 336 open questions created in January ever got a single downvote. (55/254 Feb, 34/108 Mar) If we assume a single heroic dissenter is enough to make a question bad, that still leaves the lion's share of the new content as being of sufficient quality that no one considered it bad enough to downvote. Where is the 7/8? \$\endgroup\$ Commented Mar 9, 2014 at 16:09
  • \$\begingroup\$ 336? I used the Questions page, ordered by date, and found just over 200 undeleted questions from January. I went through them all to do a best-of, and Sturgeon's law is on the nail: about 20 of them were good questions, IMAO. As far as downvotes go, I have three observations: firstly, I don't think many people vote on every question: I upvote good ones, downvote the worst, and leave the merely naff ones alone; secondly, as Geobits commented above, we probably need to downvote more; and thirdly, when I vote to close I don't always downvote, and I don't go back to downvote ones which survive. \$\endgroup\$ Commented Mar 9, 2014 at 17:01

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.