The Official Blog of Cogmap, the Org Chart Wiki



Does paying for comments work?

Or: Building community via check-writing (redux)

After my post earlier questioning whether giving $30 to a poster would fuel comment activity and bring a site to life, I thought today that doing a little follow-up would be great.

I went back and grabbed just the tip of the iceberg of summary data to throw up hear and think about. Here are some numbers from Read/Write Web:

  posts comments comment/post

This is where WordPress ate my awesome data chart of posts and comments by day on RWW for the month of February. Unbelieveable.

Keep in mind, as you look at these numbers, that RWW rolled out their promotional offer for more comments as the last post of the day on February 10th. So what conclusions can we draw from this data?

  1. Much to my RSS-reading chagrin, RWW is posting a lot more frequently.
  2. On face, comments are down. They are averaging 6.3 comments per post after the launch and were averaging 7.4 comments prior to it.
  3. Despite what I think is actually a good job of high-lighting every day the comment of the day program, the promotion appears to have driven a one-time comment burst.

This analysis is far too simplistic to draw any earth-shaking conclusions from, so let me point out a bunch of caveats I wish I had time to analyze, etc., before someone wanders onto this blog and flames the crap out of me.

  • Comments are acquired over time, so older posts will, on average, tend to have more comments than new posts, assuming the readership is relatively stable.The best way to test for how this would impact the calculation would be to graph all the comments and build a function that represents how quickly comments accumulated before and after the new comment thing was introduced. I did not have the time to do this, so to approximate it I went and looked at a really popular blog post from February 1st and hoped that it would provide a suitable model. One post acquired 25 comments. Of the 25 comments, 19 appeared the 1st day, 2 on the second, 1 on the third, 1 on the fifth, and 1 on the 13th. From this, I figured we could approximate that 80% of posts arrive in the first day and 90% within the first two days. Given this, it is reasonable to imagine that the “post-promotion” posts might accumulate an additional 25 or so comments (although, given the nature of the promotion, one might think that people with something to say will say it quickly and people that come late to the party go find a more recent post to post on.). Even giving new days this extra lift does not get them to more average comments than the “pre-promotion” posts.
  • Would love to go back and analyze this data by topic or author and determine if comment frequency correlates with either of these. Could a change in writing mix have impacted commenting? Certainly. The most popular post on RWW from a comment perspective in February was the “Web 3.0: What is it?” post, which drew 49 comments.
  • It would be great if I gave this more time, but I fear I would forget about it completely. Today, February 14th, as of this moment, there are 7 posts and 4 comments (which prompted me to do this post). Does Silicon Valley need to get off work before comments roll in? This data might reveal itself in a more intensive comment analysis.
  • It would be awesome to compare data using a comment system like this versus using a system like Intense Debate or Disqus. Does a more threaded approach to discussion create more/better comments? More flamewars?
  • Another line of analysis that went unanalyzed by my crude technique is the quality of the comment. Maybe a proxy for that would be length? Are the comments they are getting now more thoughtful as people try to win the contest? I hope so.
  • How is traffic changing over time and how does that impact comment activity? A better metric than comments per post is probably comments per unique visitor, which is information I am lacking. If traffic has declined precipitously of late (unlikely), then maybe they are actually seeing more comments.
  • Maybe my whole metric approach is flawed. If you strictly measured by absolute comment volume, maybe it is up. The problem is that this could be attributed to the contest or it could be attributed to the high volume of posts. What is interesting about this is that conventional wisdom indicates that a good volume of posts helps build community, yet comments per post are down. This is why I choose to look at the comments per post as a better metric of comment activity.

I am astounded that the volume of comments per post has declined. Has “making it a contest” taken the joy out of commenting? I wish RWW the best of luck in figuring out how to grow their community.

2 Responses to “Does paying for comments work?”

  1. Cogblog » Blog Archive » What are the factors driving Digg submissions Says:

    […] So, maybe there is something here.  But also consider this:  Is RWW where I go to get my breaking news?  Not really.  TechCrunch, Gizmodo, and ARSTechnica are the kind of sites where I might expect to get that stuff.  I look to RWW for more indepth analysis of trends.  Maybe that is just me.  I complained in an earlier post about the rising frequency of RWW posts. […]

  2. Unleash the Metadata! » Blog Archive » Community Says:

    […] paying people?”).I’m intrigued by this discussion of community, which is continued by another post attempting to evaluate whether ReadWriteWeb’s gift certificate plan worked, and finding that, […]