Education research: RSMS and student learning outcomes (fall semester 2011)

Student achievement of course learning outcomes are assessed by administering a Research Skills Mastery Survey (RSMS), a five-point Likert scale questionnaire (Patrick M. Len, in development) to Astronomy 210L students at Cuesta College, San Luis Obispo, CA. This is an optional adjunct laboratory to a one-semester introductory astronomy lecture using a backwards faded scaffolding curriculum (Slater, Slater, and Lyons, 2010), and is taken primarily by students to satisfy their general education science laboratory transfer requirement.

The RSMS is administered online during the last week of instruction, to be completed before the final laboratory poster session.

The RSMS results from this semester are compiled below. Values for the mean and standard deviations are given next to the modal response category for each question. Also listed is the percentage of students who have self-assessed themselves as having successfully mastering a research skill (responding "fairly well" or "very well") as opposed to not mastering a research skill (responding "not at all" or "slightly," or "somewhat").

Cuesta College
Research Skills Mastery Survey (RSMS) 
Astronomy 210L fall semester 2011 sections 70186, 70178, 70200
N = 45

The questions below are designed to characterize your achievement of each of the 
learning outcomes by filling in a bubble on the rating scale provided to the 
right of each statement.     

Mark the level of achievement that best describes your learning at the 
completion of the course.  

1. Keeping abreast of present-day discoveries and developments in astronomy 
(current events).
(Mastered: 80%, not mastered: 20%)
1. Not at all         1 : *
2. Slightly           2 : **
3. Somewhat           6 : ******
4. Fairly well       25 : *************************  [3.9 +/- 0.9]
5. Very well         10 : **********

2. Developing scientific evidence-based research questions.
(Mastered: 82%, not mastered: 18%)  
1. Not at all         1 : *
2. Slightly           0 : 
3. Somewhat           7 : *******
4. Fairly well       24 : ************************  [4.0 +/- 0.8]
5. Very well         12 : ************

3. Developing procedures to gather evidence in order to answer research 
questions.
(Mastered: 93%, not mastered: 7%)  
1. Not at all         1 : *
2. Slightly           0 : 
3. Somewhat           2 : **
4. Fairly well       21 : *********************  [4.3 +/- 0.8]
5. Very well         20 : ********************

4. Making appropriate evidence-supported conclusions. 
(Mastered: 88%, not mastered: 12%)  
1. Not at all         1 : *
2. Slightly           0 : 
3. Somewhat           4 : ****
4. Fairly well       21 : *********************  [4.2 +/- 0.8]
5. Very well         17 : *****************

5. Explaining research findings in a report, poster, or presentation. 
(Mastered: 81%, not mastered: 19%)  
1. Not at all         1 : *
2. Slightly           0 : 
3. Somewhat           7 : *******
4. Fairly well       22 : **********************  [4.0 +/- 0.8]
5. Very well         12 : ************

6. Evaluating evidence to determine whether or not it appropriately answers 
a research question.
(Mastered: 86%, not mastered: 14%)  
1. Not at all         1 : *
2. Slightly           0 : 
3. Somewhat           5 : *****
4. Fairly well       25 : *************************  [4.1 +/- 0.8]
5. Very well         12 : ************

Of the 6 student research skills in the RSMS, 3 were self-reported as being mastered by at least 85% of students, listed below in order of decreasing success:
3. Developing procedures to gather evidence in order to answer research questions. (93%)
4. Making appropriate evidence-supported conclusions. (88%)
6. Evaluating evidence to determine whether or not it appropriately answers a research question. (86%)

However, three student learning outcomes were self-reported as being not mastered by less than 85% of students, listed below in order of decreasing success:
2. Developing scientific evidence-based research questions. (82%)
5. Explaining research findings in a report, poster, or presentation. (81%)
1. Keeping abreast of present-day discoveries and developments in astronomy (current events). (80%)

Student research skill 1 will be evaluated directly for spring 2012 students by comparing pre- and post-instruction responses to astronomy current event questions (sunrise/sunset times, moon phases, reading comprehension of astronomy-related news articles).

As per the ACCJC (Accrediting Commission for Community and Junior Colleges), results from this indirect assessment RSMS tool, along with the direct assessment of astronomy current events quiz results will be used for course/program improvement by increasing emphasis on these lowest three learning outcomes in instruction in future semesters.