Introduction: Part of the successful assessment of an engineering program includes the description of a “major design experience that prepares students for engineering practice” (ABET EAC 2019-2020 Criterion 5 A.7). In addition, the revised student outcomes for the 2019-2020 cycle and beyond, requires programs to evaluate students’ “ability to apply engineering design to produce solutions that meet specified needs…” (ABET EAC 2019-2020 SO 2). This major design experience, typically a senior capstone project, should include the culmination of the foundational materials students learn during their course of study. ABET has also defined “engineering design” which includes many concepts of Design Thinking (DT). DT is a multistep process that begins with the formalization of the problem statement and moves through implementation of possible solutions with the needs of the end-user in mind. Our college has been introducing DT concepts in our first year Introduction to Engineering course and our senior design (SD) series since 2014. As a bioengineering department, we have also included design thinking within our introductory bioengineering course since 2014, as well as, two newly developed Biodesign courses started in 2018. This project evaluated the efficacy of incorporating design thinking concepts within the mindset of our undergraduate engineering students by evaluating SD team final design documents from senior capstone experiences.
Methods: Final design document reports from 56 Spring semester 2018 graduating engineering students were evaluated using an in-house rubric for DT concepts. These documents were evaluated by 5 graduate students that had no interactions with these SD teams nor had participated in any role in the courses or programs related to DT offered by the college. Each rubric was based on a 4-point Likert scale and ranked from 4 (master) to 1 (novice) based on multiple DT concept categories. Intraclass correlation coefficients (ICC) are an established method of evaluating observer reliability. ICCs were produced for per grader-team score averages and per grader-DT concept score averages the “irr” package function “icc” in the R programming language using the two-way effect randomization model. Correlations were considered “moderate” for ICCs greater than 0.5 and “good” for ICCs greater than 0.75 per established guidelines.
Results and Discussion: Graders reported that the rubric provided specific and easy-to-follow guidelines for scoring teams against each DT concept. A “moderate” correlation was observed in team scores between graders (ICC = 0.543) and a good correlation in DT concept scores between graders (ICC = 0.76) with p less than 0.001 for both. These correlations suggest that this rubric may be a useful tool for objectively evaluating SD team performance independent of grader bias. Additionally, the similarity in concept scores implied similar outcomes between teams independent of total score. Teams consistently scored lower in the “Multiple Solutions” and “Component Testing” concepts suggesting reassessment of pedagogical practices specific to those topics. These results prompted the adoption of instructional development plans designed to improve outcomes for future SD teams with evaluations ongoing.
The full paper will be available to logged in and registered conference attendees once the conference starts on
June 22, 2020, and to all visitors after the conference ends on June 26, 2021
Are you a researcher? Would you like to cite this paper?
Visit the ASEE document repository at
for more tools and easy citations.