6

I am a PhD student studying wireless networks/telecommunications, and I have developed a set of lab exercises related to the field. We have used them with > 100 students so far in B.S. and M.S. classes on computer networks, wireless networks, and wireless communications.

With the first few cohorts, I asked them to rate each exercise with respect to:

  • overall rating
  • difficulty of lab exercise
  • how interesting the material was
  • experiment design

and to rate self on knowledge of topic before and after the lab. I also ask for general open-ended feedback ("How can we improve this exercise?" "Any other comments?") and recently introduced automated systems to measure exactly what students are doing in the lab. This gave me a general sense of which exercises were interesting to students, and which they thought were useful. Now I am looking for a way to evaluate how effective the labs are at educating students - not just their popularity.

I've been doing some reading in CS education research (although education research is not my field, so I can't devote too much time to getting up to speed on all the methodology), but I haven't come across a study design that I think would be a good fit for my scenario. I am under some constraints:

  • I'm not the instructor for the class.
  • I just run the lab exercises via a website. I don't meet the students in person.
  • I don't have the ability to set up a "control group" by offering one section of the course with the lab and one section without.
  • I can't ask my students to do a lot of work that is only for purposes of methodology assessment (e.g. I can have them fill out a short pre-class survey, but not much more then that). I can ask them to do some things that are also learning assessments (like quizzes on the lab topic).

Given these constraints, how can I effectively evaluate the educational tools I've developed? (specifically, learn whether they actually improve students' understanding of the course material)

Update:

I don't have access to a comparable cohort from one year to the next, or between two sections. In the end, I came up with a set of questions that I think will help evaluate the exercises, even across only one group of students, all of whom are participating in the lab; I describe this in my own answer.

Improve this question
8
  • I am no expert though, this is not an answer, just an opinion. .. If it is for me, I would ask the students for a general feedback without specifying what they should rate. I would rate the difficulty by the amount of work done by the students during the lab. "How interesting the material was" - I would know this by how much interaction I am getting from the students. Some of the labs we are having in our school are so hard, yet very interesting - and only 10% of the students finishes them during time! Usually we ask the others to finish the lab at home if the lab is graded. Commented Jan 30, 2014 at 6:45
  • Thanks @AJed - I do ask for open-ended feedback too, actually. I edited the question to add that, and to clarify that I am specifically trying to evaluate not just how students perceive the exercises' effectiveness, but how it objectively affects their understanding of the course material. Commented Jan 30, 2014 at 6:54
  • With blind experiments. At least double-blind trials, triple-blind may not be needed. Commented Jan 30, 2014 at 10:24
  • @Trylks can you describe a blind experiment that I could reasonably undertake, given the constraints (e.g., not able to set up a control group)? I'm only a PhD student, I don't have access to arbitrary groups of students to experiment on. Commented Jan 30, 2014 at 10:30
  • 2
    @Trylks my field of study is not CS education, my supervisor's attitude towards these pedagogical experiments is "bemused tolerance." Also, this class is taught by a different instructor every year. I realize it's not ideal to look at the lab component in isolation, but I don't see that I have other options. Commented Jan 30, 2014 at 11:06

2 Answers 2

Reset to default
1

You are adressing a topic which (in my humble opinion) is one of the most difficult ones: How to measure effictivenes of teaching methods.

The only good way is to have a control group (and make sure they are large enough, randomly asigned, subgroups represented equally, et.c). In most practical situations this is just not feasible. (I have the same problem at the moment).

What I plan to do (so it is just an idea, not proven to be correct - fedback is wellcome!):

  1. ask students who did the class twice how they liked the new format and whether it helps them in learning
  2. Compare outcome.

I think 2.) is the one we should aim for since the ultimate measure of successful teaching is compentent students leaving the university. Since it is hard to measure their competence late in real life situations, the only thing we can do is measure their perfomence by tests / exams. So I'll compare the results of my current group with the one from last year and look for failure rate, attendeance at test, ... Given that your tests are comparable, this gives at least an indication.

Improve this answer
4
  • (2) could be useful for an instructor who offers the same class every year. But I mentioned that I am not the instructor. I don't teach the course and I only have access to/control over the "lab" portion of the course. Commented Jan 30, 2014 at 10:34
  • but you have contact to the instructor, don't you? I assume (s)he would be intersted in having an effecitve lab as well and therefore can provide the data needed (in fact you only need statistical data like average score, attendance rate). Commented Jan 30, 2014 at 10:41
  • In each of the courses where the lab is offered, the instructor is different every year, so there is no reasonable basis for comparison. Commented Jan 30, 2014 at 10:52
  • ask students who did the class twice — Other than students who fail the first time, who takes the same class twice? Commented Feb 16, 2014 at 22:21
0

In the end I decided to try the following, based on my understanding that lab exercises are particularly suited for repairing misconceptions that students might hold:

  • Before students complete the lab procedure, they have to answer an open-ended question about what they think the results will show, based on their knowledge of the course material.
  • At the end of the lab, students have to write whether their initial guess was correct. If not, they have to explain why they originally made that guess, what factor they hadn't considered that led the actual results to be different, and what (if anything) they understand now that they misunderstood before.

This serves as both a learning assessment for the students, and a methodology assessment for the lab.

  • The students earn a grade based on whether they eventually have a good sense of what's supposed to happen, either before doing the lab procedure or after.
  • The pre-lab and post-lab questions together help me quantify to what degree the lab exercises address misconceptions that students have about the content.
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.