Skip to main content
Workforce LibreTexts

7.3.1: Evidence-Based Approaches in Evaluation of Educational Technology

  • Page ID
    31714
  • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    \( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

    ( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\id}{\mathrm{id}}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\kernel}{\mathrm{null}\,}\)

    \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\)

    \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\)

    \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    \( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

    \( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

    \( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vectorC}[1]{\textbf{#1}} \)

    \( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

    \( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

    \( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

    \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    \(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)
    Section Learning Objectives
    • Apply evidence-based approaches to measure student engagement, learning outcomes, and user satisfaction.

    Applying Evidence-Based Approaches

    On the prior page, you read about some specific strategies to alloy evaluation criteria, including using rubrics, gathering information via surveys, performing data analysis, and gathering other feedback. In this section, we want to look at taking that data and using it to make decisions, using evidence based approaches. If you Google Evidence Based Practices, you will find many different models, some have 4, 5, 6, or 7 steps. Here, we simplify the process to focus on applying the basic steps to specifically evaluating the integration of educational technology in teaching and learning. 

    After you read through the steps, it should feel like a "common-sense" process: Figure out the why, what, and how of your measurement process, integrate the technology, and then measure the results against where you started. Based on the data (and feedback), make improvements.

    Steps to Evaluate Educational Technology Effectively

    The following steps were adapted from the 7 Steps to Effective Decision Making from UMass-Dartmouth (n.d.):

    1. Know Your Goals and Identify what to Measure 

    Understand why you're using educational tech. Establishing clear objectives provides a foundation for selecting appropriate metrics and ensures alignment with educational goals. Then, identify the things you want to keep track of. Some examples might be measuring student engagement indicators, learning outcome assessments, or user satisfaction survey results. Choosing measurable metrics supports accurate data collection.

    2. Use Technology Tools to Gather Data

    Use technology to your advantage! How will you collect data? Be sure the technology you're using can keep track of the things you want to measure. You may wish to integrate things like data analytics dashboards, assessment modules, and survey instruments. These tools help collect accurate information while supporting the integrity of the user experience.

    3. Identify Benchmarks 

    In step 1, you identified what you were going to measure. In this step, be sure you know what level you are starting. To know if the changes you implement are effective, you need those baseline measurements for identified metrics prior to the introduction of modifications. 

    7 Decision Making Steps4. Implement Changes Slowly, Continuous Monitoring and Data Collection

    If you want to make things better, do it bit by bit. Don't change everything at once. Making small changes helps you figure out what's really helping, and it's not confusing. Keep an eye on how things are going. Look at the quizzes, surveys, or anything else you're using to measure. See if things are getting better or staying the same. Monitoring the results helps you know if the changes are effective.

    5. Data: Analysis and Comparative Analysis

    Thoroughly analyze the data that you have collected to identify trends or patterns. Maybe you spend more time on a certain part or do better on quizzes. Finding what's different helps you know if the changes are making a real difference.

    Conduct a comparative analysis against established baseline measurements (your benchmarks). Determine if the data indicates positive or negative results or trends in what you identified to be measured (i.e. student engagement, learning outcomes, and/or user satisfaction, etc.). 

    Figure 7.3.1 Technology Evaluation by Renee Albrecht (CC BY).

    6. Seek Feedback

    Conduct interviews, focus groups, or open-ended survey questions to gather insight into your identified user (i.e. teacher, student, administrator, parents, etc.) experiences. This qualitative feedback adds context to quantitative data, offering a better understanding of the user perspective.

    7. Implement Changes

    If you find things that need fixing, make the changes. Implement further improvements or adjustments to address identified areas of concern or even strengthen those areas that are successful.

    You aren't done yet!

    A truly iterative process will continue to monitor results and make changes to improve the process. By effectively integrating the technology tools to collect data (dashboards, assessment methods, surveys, etc.), you can more easily continue the steps of Monitoring, Collecting Data, Performing Analysis, Feedback, and Implementing Changes for improvement. 

    Reference

    UMass - Dartmouth. (n.d). 7 Steps to Effective Decision Making