Workshop: "Rubric Sorting Astronomy Essays" (Cosmos in the Classroom 2013)

Workshop presented at the Astronomical Society of the Pacific Cosmos in the Classroom: A Hands-on Symposium on Teaching Introductory Astronomy and Related Science, July 22, 2013, 4:30-5:30 PM, Student Union Almaden Room, San José State University, San Jose, CA.

Abstract: Student essays on introductory astronomy exams can be consistently and efficiently graded by a single instructor, or by multiple graders for a large class. This is done by constructing a robust outcome rubric while sorting exams into separate stacks, then checking each stack for consistency. Certain online resources readily provide primary source prompts for writing astronomy exam essay questions.

Outcomes: Workshop participants will learn how to grade sample student essays by sorting them into piles based on a rubric system, how to write robust rubrics, and how to find resources to write essay questions that lead to effective rubrics.

(This presentation, sample essay questions, rubrics and more information is posted at: tinyurl.com/rubricsorting.)

Me, and some words about myself.

The goals of this workshop are to help you solve two major assessment dilemmas. Grading large numbers of astronomy exam essay questions? We'll have a hands-on demonstration of how to systematically tackle this.

Drawing a blank in writing authentic assessment essay questions? We'll also share resources to help you with this.

First, you are now going to be graduate student readers, and will be grading some student exams. Divide yourselves up into groups of three (or two, or four), and then have someone from your group pick up a stack of actual student exams.

Briefly discuss a correct response for the exam questions, and then begin a coarse sort of the exams into "good," "fair," "poor," and blank stacks. The "good" stack should not necessarily be a perfect stack, as long as it includes responses better than "fair." Feel free to move exams around between stacks during this process, as this is a tentative, raw attempt at sorting. Don't worry about conventional "ABCDF" letter grades during this coarse sorting process.

Next, for each coarse-sorted stack, go through and distinguish two levels of student understanding--e.g. for the "good" stack, sort exams into "correct" and "nearly correct" stacks; for the "fair" stack sort exams into "right ideas" and "some merit" stacks; and for the "poor" stack sort exams into "some effort" and "irrelevant discussion" stacks. More letters can be used for additional granularity as necessary (such as "pqRstuvwxyz"), but six different stacks (disregarding the blank stack) should be adequate. Again, don't worry about conventional "ABCDF" letter grades during this fine sort process.

Briefly go through each stack as a check; you may find a few exams that need to be bumped up or down to an adjacent stack. Once you are satisfied with your sorting, then you can write-in the rubric letters on each exam. If you have more than one essay question on an exam, keeping these same stacks when sorting for the next question may be useful, as there could be some correlation between how students replied to different questions on the same exam.

Note the typography of the letters used, which prevent students from modifying a rubric letter to their advantage. The upper-case "R" is used instead of a lower-case "r" to prevent it being made into a "p" to gain more points. Likewise an italicized lower-case "x" is used to prevent it being made into a lower-case "t" to gain more points, or being made from a lower-case "y." This leaves only disadvantageous ways to modify these letters:

• lower-case "t" to italicized lower-case "x."
• lower-case "v" to italicized lower-case "x."
• lower-case "v" to lower-case "y."
• lower-case "v" to lower-case "z."

Then with the exams still sorted into their final stacks, write down a concise description of what distinguishes the categories from each other, such that students will clearly understand what they did (or didn't do) by referring to the rubric definitions, and also would assist in refreshing your memory in order to comparably grade late exams or regrade miscategorized exams.

A rubric template is given below as an example, feel free to reuse/adapt/recycle for your own purposes.

• p: Correct.
• R: Nearly correct (explanation weak, unclear or only nearly complete); includes extraneous/tangential information; or has minor errors.
• t: Contains right ideas, but discussion is unclear/incomplete or contains major errors.
• v: Limited relevant discussion of supporting evidence of at least some merit, but in an inconsistent or unclear manner.
• x: Implementation/application of ideas, but credit given for effort rather than merit.
• y: Irrelevant discussion/effectively blank.
• z: Blank.

Finally, assign points (or "ABCDF" letter grade equivalents) to these rubric categories.

Second, resources for writing authentic essay questions for astronomy exams, and two types of questions that can be extracted from these resources.

Two particularly rich sources for writing essay questions are Google Books, and Yahoo! Answers.

Google Books facilitates searching for phrases such as "watch the crescent moon rising" in classic and contemporary novels, which often contain astronomy-related errors and misconceptions. These can be readily adapted into essay questions that ask students to find and explain these errors and misconceptions. This and more examples below:

"On the twilight horizon, a crescent moon was rising. I watched it hook on to a branch right above me, and dangle there, like a new sickle blade gleaming in the darkening sky."
--Minfong Ho, The Clay Marble, Marshall Cavendish, 1992, p. 71.

"It was a rich warm golden evening early in that most beautiful appearance of nature on one side of the heaven, the sun sinking down to rest in a glory of mellow light and gorgeous colour, and on the other, the pure pearly crescent moon rising above the tree tops..."
--Henry F. Chorley, "Helen, A Sketch," The Honey-moon by the Countess of Blessington and Other Tales, Vols. 1-2, E.L. Carey and A. Hart, 1837, p. 71.

"I awake early the next morning with a mosquito whining in my ear... I go outside the tent and see a thin crescent moon rising above the lake. I would like to stay outside and watch the breaking of the day, but the mosquitoes drive me back."
--Max Finkelstein and James Stone, Paddling the Boreal Forest: Rediscovering A.P. Low, Dundum, 2004, p. 143.

"After dinner, the men assembled in the centre of the camp around a large fire... The crescent moon crept slowly behind the clouds in the east. When it was fully exposed, the tabla player changed the tune to a fast beat. The men stood up and watched the crescent rising in the sky."
--Shafik Benjamin, Kismet in the Sand, AuthorHouse, 2009, p. 5.

In contrast, a question can be posted on Yahoo! Answers to solicit responses. These can be adapted into essay questions where students analyze the correctness or completeness of selected answers. This and more examples below:

P-dog: ...If Venus is above the east horizon at sunrise (so it is a morning star), can it be above the west horizon at sunset (so it can be an evening star) later on that day?
aladdinwa: No, it cannot be both the morning and evening star on the same day... When Venus rises before the sun, you can see it before the sun rises and it is the morning star and it disappears below the horizon while the sun is still in the sky...
--Adapted from http://answers.yahoo.com/question/index?qid=20120511191408AARz0Af.

P-dog: Could a [planet more massive than Earth around another sun-like star] support life...? Would [it]...be too hot, or too cold, or could [it] have moderate temperatures?
Quadrillian: ...A giant rocky planet could easily exist outside the orbit of Mars and...be habitable [with moderate temperatures.]
--Adapted from http://answers.yahoo.com/question/index?qid=20111217134914AA3qcNa.

P-dog: Can a star be cooler and more luminous than a hotter, less luminous star?
green meklar: Yes. A star can be both cooler and more luminous, if it is larger.
--Adapted from http://answers.yahoo.com/question/index?qid=20130423004321AAlfY6k.

P-dog: Would an old star cluster or a new star cluster have supergiants and white dwarfs?
tham153: It would certainly be very unlikely for any cluster to have supergiants and white dwarfs...
--Adapted from http://answers.yahoo.com/question/index?qid=20130419183134AAKacCx.

In the time remaining we can address comments and questions from workshop participants.

Additional discussion on this workshop can be continued afterwards at "office hours" later today, and/or online as well.

Earlier blog posts:

Sample astronomy essay questions, rubrics, and student responses.

Education research: overcoming initial problem-solving block.

"You don't know anything."

College, by Dan Killeen
As published in The California Aggie (University of California, Davis)
Ca. 1993

The trials and tribulations of IB--Idiot Boy.

There's one in every class.

(Dan Killeen's current strip is Steve, at dankilleen.com.)

Lolcat: doomed

Doomed, by menopaws
icanhascheezburger.com
July 5, 2007

That general sense of dread associated with an impending exam.

Education research: overcoming initial problem-solving block

Arnold B. Arons (1997) from the University of Washington, Seattle, WA observes that an effect of firmly requiring use of a systematic problem-solving procedure is that

Most students at this early stage in their development refuse to put pencil to paper, or to analyze the verbal-to-symbol transitions that are essential, until they "see" the solution as a whole. Requiring that they institute the procedure propels them, willy nilly, into the problem, and the momentum thus acquired frequently carries them through to the solution. The increasing satisfaction gained from such experiences gradually makes them more willing to penetrate a new problem, with pencil and paper and inquiry, without waiting until the entire solution has been perceived. This is a very large step indeed in intellectual development and capacity for abstract logical reasoning.

Wilbert J. McKeachie, University of Michigan, Ann Arbor, MI, and Marilla Svinicki, University of Texas at Austin, TX (2006) have similar advice for the student:

If a question completely baffles you, start writing...anything you know that could possibly be relevant. This starts your memory functioning, and usually you'll soon find that you have some relevant ideas.

For Physics 8AB (university physics, calculus-based) at Cuesta College in San Luis Obispo, CA, students are graded not so much as for obtaining the correct answer, but for how successful they are in implementing a constructive, systematic approach towards a solution. As an incentive to get students to get started by applying pencil to paper, they are given "pity points" for not leaving the page blank (e.g., writing down equations, restating knowns, or even writing "I don't know" avoids the "z" rubric for no response). However, students are warned that they can "b---s--- their way out of an 'F,' but can't b---s--- their way to an 'A.'" A throughly methodical approach that shows understanding and mastery of the essential concepts, up to the point where purely algebraic elimination and back-substitution of unknowns would begin is worth a "t" rubric, which on the grade scale is just below the B-/C+ cutoff.

Physics 8AB problem rubric

• p = 15/15: Correct.
  
• r = 12/15: Nearly correct (explanation weak, unclear or only nearly complete); includes extraneous/tangential information; or has minor errors.
  
• t = 9/15: Contains right ideas, but discussion is unclear/incomplete or contains major errors.
  
• v = 6/15: Limited relevant discussion of supporting evidence of at least some merit, but in an inconsistent or unclear manner.
  
• x = 3/15: Implementation/application of ideas, but credit given for effort rather than merit.
  
• y = 1.5/15: Irrelevant discussion/effectively blank.
  
• z = 0/15: Blank.

Astronomy 10 (introductory astronomy, general education requirement) has a similar grading scale, where the "y" rubric provides an incentive for students to at least start writing and hopefully get past an initial "brain-lock," and the "t" rubric is the B-/C+ cutoff for the "worst of the best answers" (or "best of the worst answers").

Astronomy 10 short-answer rubric

• p = 20/20: Correct.
  
• r = 16/20: Nearly correct, but includes minor math errors.
  
• t = 12/20: Nearly correct, but approach has conceptual errors, and/or major/compounded math errors.
  
• v = 8/20: Implementation of right ideas, but in an inconsistent, incomplete, or unorganized manner.
  
• x = 4/20: Implementation of ideas, but credit given for effort rather than merit.
  
• y = 2/20: Irrelevant discussion/effectively blank.
  
• z = 0/20: Blank.

Arnold B. Arons, Teaching Introductory Physics, John Wiley & Sons, Inc., New York, 1997, p. 39.

Wilbert J. McKeachie, Marilla Svinicki, McKeachie's Teaching Tips: Strategies, Research, and Theory for College and University Teachers (12th edition), Houghton Mifflin Company, Boston, 2006, p. 109.