20120429

Education research: RSMS and student learning outcomes (fall semester 2011)

Student achievement of course learning outcomes are assessed by administering a Research Skills Mastery Survey (RSMS), a five-point Likert scale questionnaire (Patrick M. Len, in development) to Astronomy 210L students at Cuesta College, San Luis Obispo, CA. This is an optional adjunct laboratory to a one-semester introductory astronomy lecture using a backwards faded scaffolding curriculum (Slater, Slater, and Lyons, 2010), and is taken primarily by students to satisfy their general education science laboratory transfer requirement.

The RSMS is administered online during the last week of instruction, to be completed before the final laboratory poster session.

The RSMS results from this semester are compiled below. Values for the mean and standard deviations are given next to the modal response category for each question. Also listed is the percentage of students who have self-assessed themselves as having successfully mastering a research skill (responding "fairly well" or "very well") as opposed to not mastering a research skill (responding "not at all" or "slightly," or "somewhat").
Cuesta College
Research Skills Mastery Survey (RSMS)
Astronomy 210L fall semester 2011 sections 70186, 70178, 70200
N = 45

The questions below are designed to characterize your achievement of each of the
learning outcomes by filling in a bubble on the rating scale provided to the
right of each statement.

Mark the level of achievement that best describes your learning at the
completion of the course.

1. Keeping abreast of present-day discoveries and developments in astronomy
(current events).
(Mastered: 80%, not mastered: 20%)
1. Not at all 1 : *
2. Slightly 2 : **
3. Somewhat 6 : ******
4. Fairly well 25 : ************************* [3.9 +/- 0.9]
5. Very well 10 : **********

2. Developing scientific evidence-based research questions.
(Mastered: 82%, not mastered: 18%)
1. Not at all 1 : *
2. Slightly 0 :
3. Somewhat 7 : *******
4. Fairly well 24 : ************************ [4.0 +/- 0.8]
5. Very well 12 : ************

3. Developing procedures to gather evidence in order to answer research
questions.
(Mastered: 93%, not mastered: 7%)
1. Not at all 1 : *
2. Slightly 0 :
3. Somewhat 2 : **
4. Fairly well 21 : ********************* [4.3 +/- 0.8]
5. Very well 20 : ********************

4. Making appropriate evidence-supported conclusions.
(Mastered: 88%, not mastered: 12%)
1. Not at all 1 : *
2. Slightly 0 :
3. Somewhat 4 : ****
4. Fairly well 21 : ********************* [4.2 +/- 0.8]
5. Very well 17 : *****************

5. Explaining research findings in a report, poster, or presentation.
(Mastered: 81%, not mastered: 19%)
1. Not at all 1 : *
2. Slightly 0 :
3. Somewhat 7 : *******
4. Fairly well 22 : ********************** [4.0 +/- 0.8]
5. Very well 12 : ************

6. Evaluating evidence to determine whether or not it appropriately answers
a research question.
(Mastered: 86%, not mastered: 14%)
1. Not at all 1 : *
2. Slightly 0 :
3. Somewhat 5 : *****
4. Fairly well 25 : ************************* [4.1 +/- 0.8]
5. Very well 12 : ************

Of the 6 student research skills in the RSMS, 3 were self-reported as being mastered by at least 85% of students, listed below in order of decreasing success:
3. Developing procedures to gather evidence in order to answer research questions. (93%)
4. Making appropriate evidence-supported conclusions. (88%)
6. Evaluating evidence to determine whether or not it appropriately answers a research question. (86%)

However, three student learning outcomes were self-reported as being not mastered by less than 85% of students, listed below in order of decreasing success:
2. Developing scientific evidence-based research questions. (82%)
5. Explaining research findings in a report, poster, or presentation. (81%)
1. Keeping abreast of present-day discoveries and developments in astronomy (current events). (80%)

Student research skill 1 will be evaluated directly for spring 2012 students by comparing pre- and post-instruction responses to astronomy current event questions (sunrise/sunset times, moon phases, reading comprehension of astronomy-related news articles).

As per the ACCJC (Accrediting Commission for Community and Junior Colleges), results from this indirect assessment RSMS tool, along with the direct assessment of astronomy current events quiz results will be used for course/program improvement by increasing emphasis on these lowest three learning outcomes in instruction in future semesters.

3 comments:

  1. Questions 2 and 3 appear identical both in the data in the middle, and the discussion afterwards. I think Question 3 is supposed to be "3. Develop procedures to gather evidence in order to answer research questions." Was this mistranscribed into this post, or were the students mistakenly given two copies of the same question? I am guessing the former (mistranscribed) otherwise I would have expected the same results for the two questions.

    ReplyDelete
  2. Questions 2 and 3 appear identical both in the results section of this post, and the discussion afterwards. Based on the email to the BFS group, I think Question 3 is supposed to be "3. Develop procedures to gather evidence in order to answer research questions." Was this mistranscribed in this post, or were the students given two copies of Question 2?

    ReplyDelete
  3. Thanks for the comments, I've corrected question 3 in the data and discussion to read: "Developing procedures to gather evidence in order to answer research questions."

    ReplyDelete