Delis Rating of Executive Functions (D-REF) - Telehealth and D-REF

Special recommendations for administering D-REF via telehealth

D-REF quickly and easliy administers, scores, and reports the frequency of observed behaviors that identify executive function problems in children and adolescents ages 5-19. Administering the Delis-Rating of Executive Function (D-REF) in a telehealth context can be accomplished utilizing our Q-global® software system. Details regarding the system and how it is used are provided on the Q-global homepage.

Conducting a valid assessment in a telehealth modality requires an understanding of the interaction between a number of complex issues. In addition to the general information on our telehealth overview page noted above, professionals should address five themes (Eichstadt et al. 2013) when planning for administering D-REF® assessments via telehealth:

For more information on the five themes, please visit our Telehealth homepage.

 

Theme-specific information for forced-choice assessment administrations and the D-REF

1. Audio/visual environment

  • If you require video viewing during the administration, make sure the full face of the examinee is in view
  • If you require audio interaction during the administration, make sure the audio is working as expected. Test the audio prior to the administration either through the examinee’s speakers or headset and ensure a high-quality audio environment is present
  • Make sure the examinee's environment is free from distractions

2. Examiner factors

  • Practice the mechanics and workflow of assigning, starting, and completing the assessment using the On-Screen Administration capabilities of Q-global (or Remote On-Screen if being used) before you go through this process with an examinee so that you are familiar with the administration procedures
  • If you are asking the examinee to complete the assessment during a telehealth session while you are observing virtually, please note the audio/visual environment information above, and ensure that the session is set up appropriately to elicit a valid response from the examinee. Be sure to also ensure that the test taking session mimics an in-person session as closely as possible

3. Examinee factors

  • As noted on the general telehealth page, please ensure that a remote assessment administration is appropriate for the client and for the purpose of the assessment
  • Ensure that examinee is able and prepared to appropriately engage in the testing session and that they are well rested, can adequately attend to the test, and are ready to fully participate in the test session
  • There may be some administrations where an examinee headset is not appropriate or feasible. In these instances, make sure you have a web camera with an embedded microphone or a standalone microphone with the volume turned up to a comfortable level

4. Test/test materials

  • Ask the examinee to close all other applications on the computer, laptop, or other device for faster administration performance
  • Monitor the test session and the examinee's interaction with the test materials to ensure confidentiality and test session integrity as appropriate

5. Other/miscellaneous

  • If you are administering the test using the On-Screen Administration (OSA) function of Q-global while the client is in a remote location, it may be advisable that a trained facilitator or support person is onsite with the client to assist with logging in and starting the assessment if the examinee's computer is controlling the session (R-OSA), or to ensure that the appropriate client response is captured if the assessment is being administered via screen sharing/video-conferencing capabilities
  • The facilitator should also be trained in and able to provide emergency assistance should it be required during the test session for any reason, including if the examinee should show any indication of personal distress during the assessment
  • Always state in your report that the test was administered via telehealth, and briefly describe the method of telehealth used. For example, " The D-REF was administered by remote on-screen administration through the Q-global system, and the examiner's assistant monitored the administration using a live video connection"
  • Make a clinical judgment, similar to a face-to-face session, whether or not you are able to gather the examinee's best performance. Report your clinical decision(s) in your report and comment on the factors that led to this decision and your reporting or lack of reporting of the scores. For example, "The remote testing environment appeared free of distractions, adequate rapport was established with the examinee via video and s/he appeared appropriately engaged in the task throughout the session. No significant technological problems were noted during administration, and the results are considered to be a valid estimate of the examinee's skills/abilities"

Selected research to date

Please refer to the following studies regarding the appropriateness of administering forced-choice assessments via computer or other electronic means.

Chuah et al (2006). Personality Assessment: Does the Medium Matter? No. Journal of Research in Personality, 40-4, 339-376.

Butcher, J., Perry, J., & Hahn, J. (2004). Computers in clinical assessment: Historical developments, present status, and future challenges. Journal of Clinical Psychology, 60, 331-345.

Pinsoneault, Terry B. (1996) Equivalency of computer-assisted and paper-and-pencil administered version of the Minnesota Multiphasic Personality Inventory-2.Computers in Human Behavior, 12-2, 291-300.

Russell M. et al (2003). Computer-Based Testing and Validity: A Look Back and into the Future. Technology and Assessment Study Collaborative, Boston College

Studies supporting the equivalency of test scores when picture stimuli are displayed to the examinee in a printed manual versus a digital display on a computer screen (in-person administration):

Daniel, M. H., Wahlstrom, D., & Zhou, X. (2014). Equivalence of Q-interactive and paper administrations of language tasks: Selected CELF-5 tests. Q-interactive Technical Report 7. Bloomington, MN: Pearson.

Daniel, M. H. (2012a). Equivalence of Q-interactive administered cognitive tasks: WAIS–IV. Q-interactive Technical Report 1. Bloomington, MN: Pearson.

Daniel, M. H. (2012b). Equivalence of Q-interactive administered cognitive tasks: WISC–IV. Q-interactive Technical Report 2. Bloomington, MN: Pearson.

 

Conclusion

Provided that you have thoroughly considered and addressed all five themes and the special considerations as listed above, based on the available research, there is little reason that you should have concerns about the reliable and valid delivery of the D-REF via telehealth as well as the use of the normative data. Document in your report that the administration was completed by telehealth. You may use the D-REF via telehealth without additional permission from Pearson in the following published context:

  • D-REF On-Screen Administration (OSA) or Remote On-Screen Administration (ROSA) via Q-global [tests with OSA/ROSA options]

OR

  • D-REF On-Screen Administration (OSA) via Q-global [tests with OSA-only option]

Any other use of the D-REF via telehealth requires prior permission from Pearson.