Observations on the commenting behavior of top students in an online course utilizing an "Engagement Index"


The Set Up


What happens when 60% of a student's grade is determined by a score that quantifies how much they interact (like, comment, post) on an online platform?


This was the setup of the grading scheme in the Fall 2017 online class, ART W23AC: Data Arts (or later titled, Data Culture: Principles of Internet Citizenship). The Engagement Index (the point system on the online platform) would be converted into a letter grade, and would make up 60% of each student's grade, while a more traditional final project and written final exam would make up the other 40%.

Points on the Engagement Index system would be collected as follows:

Loading output library...

In the beginning of the semester, the conversion from Engagement Index to letter grade was understood to be as follows:

For a C: earn 1000 points by October 18, 11:59 PM

For a B: earn 2000 points by December 1, 11:59 PM

Grades between a B+ and A+ would be determined relative to the median of the top 8 scores.

For their commments, students were trusted on the basis of an honor system.

Initial Thoughts


My initial reservation about the Engagement Index as a student in this class was that it was possible that only the quantity, not quality of comments would have an effect on grades. Not only that, the quality of posts (assignments) would also not be as important either, as long as the posts could attract likes/comments. Otherwise, the student could make up for the lack of engagement on their own post by liking/commenting on other students' posts en masse.

Due to the fact that commenting yielded the most points on the Engagement Index, it was reasonable to focus on just the commenting data, rather than also the liking/viewing data.

Unfortunately, due to limitations in data collection, there was not a way to evaluate the contents of the comments of the top students, so instead I decided to analyze the number of comments comment per hour rates of the top students. After these comments/hour rates are obtained, it could be decided if it is actually possible that a meaningful, thoughtful comment could be written at that rate.

I chose to explore this potential problem due to my own experience of not being able to keep up with the sudden, seemingly exponential increase in pace at which my fellow classmates were commenting. Realizing that it was ultimately more worthwhile to focus my energy on other things instead of mass commenting, I regretfully had to pass/no pass this course despite enjoying the course content.

Working with and visualizing the data


"Sanitized" data (that is data with the real names replaced) on the Engagement Index was provided by course professor, Greg Niemeyer, as a Google Sheets document. The data was read into a Jupyter Notebook as a CSV file and handled as a Pandas Dataframe.

The link to the original data is provided as follows: https://docs.google.com/a/berkeley.edu/spreadsheets/d/1-xCqxsZ4LWrsOylQqYpR4BTA81kgJAPuWc3cWejOFuU/edit?usp=sharing

Loading output library...

For the purposes of data visualization, I decided to focus on the commenting behavior of only the top 10 students.

Loading output library...

To start exploring the data, we will first focus on No. 1 student, Krystina Riehle.

Loading output library...
Loading output library...

In our initial graph focusing exclusively on top commenter Krystina Riehle, we observe a single particular hour at which the student makes over 350 comments. This means that about ~6 comments are made in 1 minute. Even if the student had followed the rules by not utilizing a bot, common sense can tell us that it is highly improbable that the student could have written a meaningful comment at this rate.

Next, we will plot the "number of comments vs. hour" graphs for all of the top 10 students.

Loading output library...

The student with the lowest maximum commenting/hour rate appears to be Janene Janson at ~45 comments in one particular hour. This is about 0.75 comments per minute. While not as extreme as Krystina Riehle's maximum rate, and perhaps a lot more humanly feasible, it is still improbable that meaningful comments can be written at even this rate. Additionally, from the graphs it appears to be that the more high maximum rates appear in infrequent bursts. The students who have lower maximum rates comment at higher rates on average more frequently. While frequent commenting is more ideal, not every student can devote to commenting at this level of intensity as often.



Overall, we observed very high rates of comments/hour among the top 10 students on the Engagement Index leaderboard. These rates were high enough to question the how meaningful or thoughtful the comments written during these moments of high activity could possibly be. These results are rather unideal to many educators who would like to promote meaningful collaboration and feedback among students, and reveal a flaw in the current system. Potential problems that can result from these high comment rates can include lack of morale in students who cannot keep up, and eventual disinterest in the course content itself.