Workshop: "Rubric Sorting Astronomy Essays" (Cosmos in the Classroom 2013)

Workshop presented at the Astronomical Society of the Pacific Cosmos in the Classroom: A Hands-on Symposium on Teaching Introductory Astronomy and Related Science, July 22, 2013, 4:30-5:30 PM, Student Union Almaden Room, San José State University, San Jose, CA.

Abstract: Student essays on introductory astronomy exams can be consistently and efficiently graded by a single instructor, or by multiple graders for a large class. This is done by constructing a robust outcome rubric while sorting exams into separate stacks, then checking each stack for consistency. Certain online resources readily provide primary source prompts for writing astronomy exam essay questions.

Outcomes: Workshop participants will learn how to grade sample student essays by sorting them into piles based on a rubric system, how to write robust rubrics, and how to find resources to write essay questions that lead to effective rubrics.

(This presentation, sample essay questions, rubrics and more information is posted at: tinyurl.com/rubricsorting.)

Me, and some words about myself.

The goals of this workshop are to help you solve two major assessment dilemmas. Grading large numbers of astronomy exam essay questions? We'll have a hands-on demonstration of how to systematically tackle this.

Drawing a blank in writing authentic assessment essay questions? We'll also share resources to help you with this.

First, you are now going to be graduate student readers, and will be grading some student exams. Divide yourselves up into groups of three (or two, or four), and then have someone from your group pick up a stack of actual student exams.

Briefly discuss a correct response for the exam questions, and then begin a coarse sort of the exams into "good," "fair," "poor," and blank stacks. The "good" stack should not necessarily be a perfect stack, as long as it includes responses better than "fair." Feel free to move exams around between stacks during this process, as this is a tentative, raw attempt at sorting. Don't worry about conventional "ABCDF" letter grades during this coarse sorting process.

Next, for each coarse-sorted stack, go through and distinguish two levels of student understanding--e.g. for the "good" stack, sort exams into "correct" and "nearly correct" stacks; for the "fair" stack sort exams into "right ideas" and "some merit" stacks; and for the "poor" stack sort exams into "some effort" and "irrelevant discussion" stacks. More letters can be used for additional granularity as necessary (such as "pqRstuvwxyz"), but six different stacks (disregarding the blank stack) should be adequate. Again, don't worry about conventional "ABCDF" letter grades during this fine sort process.

Briefly go through each stack as a check; you may find a few exams that need to be bumped up or down to an adjacent stack. Once you are satisfied with your sorting, then you can write-in the rubric letters on each exam. If you have more than one essay question on an exam, keeping these same stacks when sorting for the next question may be useful, as there could be some correlation between how students replied to different questions on the same exam.

Note the typography of the letters used, which prevent students from modifying a rubric letter to their advantage. The upper-case "R" is used instead of a lower-case "r" to prevent it being made into a "p" to gain more points. Likewise an italicized lower-case "x" is used to prevent it being made into a lower-case "t" to gain more points, or being made from a lower-case "y." This leaves only disadvantageous ways to modify these letters:

• lower-case "t" to italicized lower-case "x."
• lower-case "v" to italicized lower-case "x."
• lower-case "v" to lower-case "y."
• lower-case "v" to lower-case "z."

Then with the exams still sorted into their final stacks, write down a concise description of what distinguishes the categories from each other, such that students will clearly understand what they did (or didn't do) by referring to the rubric definitions, and also would assist in refreshing your memory in order to comparably grade late exams or regrade miscategorized exams.

A rubric template is given below as an example, feel free to reuse/adapt/recycle for your own purposes.

• p: Correct.
• R: Nearly correct (explanation weak, unclear or only nearly complete); includes extraneous/tangential information; or has minor errors.
• t: Contains right ideas, but discussion is unclear/incomplete or contains major errors.
• v: Limited relevant discussion of supporting evidence of at least some merit, but in an inconsistent or unclear manner.
• x: Implementation/application of ideas, but credit given for effort rather than merit.
• y: Irrelevant discussion/effectively blank.
• z: Blank.

Finally, assign points (or "ABCDF" letter grade equivalents) to these rubric categories.

Second, resources for writing authentic essay questions for astronomy exams, and two types of questions that can be extracted from these resources.

Two particularly rich sources for writing essay questions are Google Books, and Yahoo! Answers.

Google Books facilitates searching for phrases such as "watch the crescent moon rising" in classic and contemporary novels, which often contain astronomy-related errors and misconceptions. These can be readily adapted into essay questions that ask students to find and explain these errors and misconceptions. This and more examples below:

"On the twilight horizon, a crescent moon was rising. I watched it hook on to a branch right above me, and dangle there, like a new sickle blade gleaming in the darkening sky."
--Minfong Ho, The Clay Marble, Marshall Cavendish, 1992, p. 71.

"It was a rich warm golden evening early in that most beautiful appearance of nature on one side of the heaven, the sun sinking down to rest in a glory of mellow light and gorgeous colour, and on the other, the pure pearly crescent moon rising above the tree tops..."
--Henry F. Chorley, "Helen, A Sketch," The Honey-moon by the Countess of Blessington and Other Tales, Vols. 1-2, E.L. Carey and A. Hart, 1837, p. 71.

"I awake early the next morning with a mosquito whining in my ear... I go outside the tent and see a thin crescent moon rising above the lake. I would like to stay outside and watch the breaking of the day, but the mosquitoes drive me back."
--Max Finkelstein and James Stone, Paddling the Boreal Forest: Rediscovering A.P. Low, Dundum, 2004, p. 143.

"After dinner, the men assembled in the centre of the camp around a large fire... The crescent moon crept slowly behind the clouds in the east. When it was fully exposed, the tabla player changed the tune to a fast beat. The men stood up and watched the crescent rising in the sky."
--Shafik Benjamin, Kismet in the Sand, AuthorHouse, 2009, p. 5.

In contrast, a question can be posted on Yahoo! Answers to solicit responses. These can be adapted into essay questions where students analyze the correctness or completeness of selected answers. This and more examples below:

P-dog: ...If Venus is above the east horizon at sunrise (so it is a morning star), can it be above the west horizon at sunset (so it can be an evening star) later on that day?
aladdinwa: No, it cannot be both the morning and evening star on the same day... When Venus rises before the sun, you can see it before the sun rises and it is the morning star and it disappears below the horizon while the sun is still in the sky...
--Adapted from http://answers.yahoo.com/question/index?qid=20120511191408AARz0Af.

P-dog: Could a [planet more massive than Earth around another sun-like star] support life...? Would [it]...be too hot, or too cold, or could [it] have moderate temperatures?
Quadrillian: ...A giant rocky planet could easily exist outside the orbit of Mars and...be habitable [with moderate temperatures.]
--Adapted from http://answers.yahoo.com/question/index?qid=20111217134914AA3qcNa.

P-dog: Can a star be cooler and more luminous than a hotter, less luminous star?
green meklar: Yes. A star can be both cooler and more luminous, if it is larger.
--Adapted from http://answers.yahoo.com/question/index?qid=20130423004321AAlfY6k.

P-dog: Would an old star cluster or a new star cluster have supergiants and white dwarfs?
tham153: It would certainly be very unlikely for any cluster to have supergiants and white dwarfs...
--Adapted from http://answers.yahoo.com/question/index?qid=20130419183134AAKacCx.

In the time remaining we can address comments and questions from workshop participants.

Additional discussion on this workshop can be continued afterwards at "office hours" later today, and/or online as well.

Earlier blog posts:

Sample astronomy essay questions, rubrics, and student responses.

Education research: overcoming initial problem-solving block.