Telehealth and the CVLT

California Verbal Learning Test - Third Edition (CVLT-3) - CVLT 3 and Telehealth

A telehealth session includes an examiner in one geographical location and an examinee at a different location. Using a high-speed Internet connection and a secure software platform designed for web-based meetings (i.e., teleconference platform), an examiner and examinee—along with a facilitator in the examinee location, if necessary—join a shared web-based meeting via computers with audio and video capability. The examiner and examinee can see and hear one another throughout the session. Text, pictures, and video can be shared through the teleconference platform.

The various editions of the California Verbal Learning Test (CVLT–C [Delis, Kaplan, Kramer, & Ober, 1994], CVLT–II [Delis, Kaplan, Kramer, & Ober, 2000], and CVLT3 [Delis, Kaplan, Kramer, & Ober, 2017]) can be administered in a telehealth context and scored using digital tools from Q-global®, Pearson’s secure online-testing and scoring platform. See other product specific pages for information on how to administer any tasks used during the delay interval on the CVLT. Details regarding Q-global and how it is used are provided on the Q-global page.

Several options are available for administering the CVLT via telehealth. They vary based on the presence and role of an onsite facilitator.

  • Given the audio only nature of the CVLT, a facilitator is not necessary for administration. A tele-platform connection can be made directly with a client.
  • If the CVLT is being given (a) as part of a larger battery of tests some of which may require onsite testing equipment (e.g., blocks), (b) to a client who may not be able to manage the technology, or (c) to a client who requires onsite, one-on-one interaction to maintain adequate focus on the tests, a trained professional facilitator can be present to ensure adequate audiovisual connection.
  • If the onsite trained professional facilitator remains in the room during administration, the facilitator may operate audiovisual equipment and sit unobtrusively and silently out of the examinee’s view in or near the room during the testing session to ensure no technology adjustments are necessary, connectivity is maintained, and the examinee remains on task. If a peripheral camera is not available, the facilitator may also provide feedback on the quality of the connection and address concerns raised by the client unrelated to administration (e.g., room temperature, distractions). The facilitator can also ensure that nonstandardized methods to assist performance (e.g., writing down words) are not used by the client.
  • During the COVID-19 pandemic, however, the only facilitator available may be someone in the examinee’s home. If using an onsite facilitator who is not in a professional role (e.g., parent, guardian, or spouse), the examiner should use his or her professional judgment about the capacity of the facilitator to perform the required functions correctly and without interfering in the testing session. It is recommended that a facilitator who is a relative leave the room during administration of the CVLT.
  • If the onsite facilitator is a parent/guardian/spouse, follow the guidelines outlined in the specific test manual regarding the presence of others in the room to ensure adherence to standard administration procedures. It is very rare that a person other than the examinee remain in the room during testing. The parent/guardian/spouse may only make audiovisual adjustments during administration of the CVLT.

Conducting Telehealth Assessment

Conducting a valid assessment in a telehealth service delivery model requires an understanding of the interplay of a number of complex issues. In addition to the general information on Pearson’s telehealth overview page, examiners should address five factors (Eichstadt et al., 2013) when planning to administer and score assessments via telehealth:

  1. Audio/Visual Environment
  2. Examiner Factors
  3. Examinee Factors
  4. Test/Test Materials
  5. Other/Miscellaneous

1. Audio / visual environment

Computers and connectivity

Two computers with audio and video capability and stable Internet connectivity—one for the examiner and one for the examinee—are required. A stationary web camera, microphone, and speakers or headphones are required for both the examiner and the examinee. It is recommended that the examiner have a second computer screen or paper format manual so he or she has the ability to view the Administration and Scoring Manual while maintaining view of the client.

Audio considerations

High-quality audio capabilities are required during the administration. An over the head, two-ear, stereo headset with attached boom microphone is recommended for both the examiner and examinee. Ensure adequate volume and voice clarity before beginning administration. Because a single word can’t be repeated during administration of a trial, adequate audio is required. Record any lapses in transmission or evidence of poor audio during testing to include in the report and consider in interpretation.

Audio check

Test the audio for both the examiner and examinee prior to the administration to ensure a high-quality audio environment is present. Testing the audio may include an informal conversation prior to the administration where the examiner is listening for any clicks, pops, or breaks in the audio signal that distorts or interrupts the voice of the examinee. The examiner should also ask the examinee and facilitator if there are any interruptions or distortions in the audio signal on their end.

Video

High-quality video (HD preferred) is recommended during the administration to allow full observation of the examinee. Make sure the full faces of the examiner and the examinee are visible using each respective web camera.

Peripheral camera or device

A stand-alone peripheral camera that can be positioned to provide a view of the session from another angle or a live view of the examinee’s progress is helpful. Alternately, the onsite facilitator may join the teleconference from a separate device (e.g., a smartphone with a camera or another peripheral device) and set it in a stable position to show the examinee’s pointing or written responses.

If using an onsite facilitator who is not in a professional role (e.g., parent/guardian/spouse), the examiner should use their professional judgment about the capacity of the facilitator to perform the required functions correctly and without interfering in the testing session. A parent/guardian/spouse does not typically remain in the room during testing except on rare occasions as described in the test manual, so it is necessary to train him or her how to place the peripheral camera/device in a stable position before beginning the session.

The facilitator should silence the audio and mute the microphone on any peripheral devices to prevent feedback. Instruct the facilitator not to capture video or take photos as this is a copyright violation.

Manage audiovisual distractions

As with any testing session, make sure the examinee’s environment is free from audio and visual distractions. If you are unfamiliar with the examinee’s planned physical location, meet virtually with the facilitator in advance of the testing session. Ask the facilitator to show the intended testing room and provide a list of issues to address to transform the environment into an environment suitable for testing. For example, remove distracting items, silence all electronics, and close doors. Ask the examinee and facilitator to close all other applications on the computer, laptop, or other device, and to silence alerts and notifications on the peripheral device. Ensure radios, televisions, other cellular phones, fax machines, smart speakers, and equipment that emit noise are silenced and removed from the room.

Lighting

Establish good overhead and facial lighting for the examiner and examinee. Close blinds or shades to reduce sun glare on faces and the computer screens.

2. Examiner factors

Practice

During the telehealth setup, and before administering to an examinee, practice the mechanics and workflow of the entire test using the selected teleconference platform so that you are familiar with the administration procedures. For example, use a colleague as a practice “examinee.”

Standardized procedures

Follow the administration procedures of face-to-face administration as much as possible. For example, a word in the word list cannot be repeated within a trial in face-to-face administration; therefore, do not say it more than once in a telehealth administration. Plan ahead on how to address auditory glitches or delays as bandwidth and connectivity may interfere with standard administration and may require some adjustments to standard procedures. For example, if the connection is lost momentarily it may be appropriate to repeat a word that was not heard; however, all modifications like this need to be noted in your report and considered when applying norms obtained from in-person assessments. For clients with poor connectivity, consider whether online administration is appropriate.

Facilitator role and training

If a facilitator is utilized during the CVLT administration, their role for most telehealth assessments is largely to manage audiovisual needs and materials. Train the facilitator to troubleshoot audiovisual needs that arise during the testing session, including camera angle, lighting, and audio checks. Inform the facilitator to sit unobtrusively and silently out of the examinee’s view when not actively engaging in his or her role. In these cases, the facilitator’s role is not to manage rapport, engagement, or attention during the testing session and he or she is not to interfere with the examinee’s performance or responses.

If using an onsite facilitator who is not in a professional role (e.g, parent/guardian/spouse), you should use your professional judgment about the capacity of the facilitator to perform the required functions correctly and without interfering in the test session. You should communicate expectations about the facilitator’s role immediately prior to the testing session, when the examinee is not present, to ensure that nothing is disclosed to the examinee about the tasks. Do not allow the facilitator to show or warn the examinee about any portion of the test. Expect to provide verbal guidance to the facilitator during the testing session.

Any other roles and responsibilities for which an examiner needs support, such as behavior management, should be outlined and trained prior to the beginning of the testing session. The examiner is responsible for documenting all behaviors of the facilitator during test administration and taking these into consideration when reporting scores and performance.

3. Examinee factors

Appropriateness

Ensure that a telehealth administration is appropriate for the examinee and for the purpose of the assessment. Use clinical judgement, best practice guidance for telehealth (e.g., Interorganizational Practice Committee, 2020), information from professional organizations, existing research, and any available federal or state regulations in the decision-making process.

Preparedness

Before initiating test administration, ensure that the examinee is well-rested, able, prepared, and ready to appropriately and fully participate in the test session.

Facilitator role

Explain the role of the facilitator to the examinee so participation and actions are understood.

Headset

It may not be appropriate or feasible for some examinees to use a headset due to behavior, positioning, physical needs, or tactile sensitivities. Use clinical judgement on the appropriate use of a headset in these situations. If a headset is not utilized, ensure your microphone and the examinee’s speakers are turned up to a comfortable volume.

4. Test / test materials

Copyright

Pearson has provided a Letter of No Objection (PDF | 91 KB) to permit use of copyrighted materials for telehealth via nonpublic facing teleconferencing software and tools to assist in remote administration of assessment content during the COVID-19 pandemic.

Record Forms

You can use paper record forms or the Q-interactive® platform to capture responses from examinees. Scores can be obtained through the Q-global platform; however, administration directions and recording of responses are available through the paper record forms or Q-interactive administration platform.

Equivalence Evidence

Telehealth Versus Face-to-Face Administration

The CVLT and similar verbal list learning and recall tasks have been studied in several studies comparing face-to-face administration with remote administration in examinees who are typically developing and those with clinical diagnoses (Barcellos et al., 2017; Cullum et al., 2006; Cullum et al., 2014; Galusha-Glasscock et al., 2016; Hildebrand, Chow, Williams, Nelson, & Wass, 2004; Kirkwood, Peck, & Bennie, 2000; Ragbeer et al., 2016; Stain et al., 2011; Wadsworth, Dhima, et al., 2016; Wadsworth, Galusha-Glasscock, et al., 2018). Other studies support equivalence of tasks that are highly similar to the verbal requirements of the CVLT (e.g., simple list learning tasks) with nonclinical and clinical examinees using telehealth compared with face-to-face administration and scoring (Galusha-Glasscock et al., 2016; Vahia et al., 2015). In addition, a meta-analysis of telehealth studies provides rigorous support for telehealth and face-to-face mode equivalence (Brearly et al., 2017).

Digital Versus Traditional Format

The CVLT has demonstrated equivalence across digital and paper administration in numerous studies; thus the most recent publication of the CVLT3 was collected completely via face-to-face digital collection. A number of investigations of the various forms of the CVLT have produced evidence of equivalence when administered and scored via digital or traditional formats to examinees without clinical conditions (Daniel, 2012).

5. Other / miscellaneous

State in your report that the test was administered via telehealth, and briefly describe the method of telehealth used. For example, The CVLT–C was administered via remote telehealth, and a facilitator monitored the administration onsite during the live video connection using the [name of telehealth system {e.g., Zoom} platform].

Make a clinical judgment, similar to a face-to-face session, about whether or not you are able to gather the examinee’s best performance. Report your clinical decision(s) in your report and comment on the factors that led to the decision to report (or not report) the scores.

For example, The remote testing environment appeared free of distractions, adequate rapport was established with the examinee via video/audio, and the examinee appeared appropriately engaged in the task throughout the session. No significant technological problems were noted during administration. Modifications to the standardization procedure included: [list]. The CVLT–C, or similar tasks, have received initial validation in several samples for remote telehealth and digital format administration, and the results are considered to be a valid assessment of the examinee’s skills and abilities.

 

Conclusion

The CVLT was not standardized in a telehealth mode and this should be taken into consideration when utilizing this test via telehealth and interpreting results. Provided that you have thoroughly considered and addressed all five factors and the specific considerations as listed above, you are prepared to observe and comment about the reliable and valid delivery of the CVLT via telehealth. You may use the CVLT materials via telehealth without additional permission from Pearson in the following published contexts:

  • CVLT-C, CVLT-II, and CVLT3 manuals via Q-global

Any other use of the CVLT via telehealth requires prior permission from Pearson and is not currently recommended. This includes, but is not limited to, digitizing the paper record forms or uploading a manual onto a shared drive or site.

 

References

Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (2017). California Verbal Learning Test–Third Edition (CVLT3). NCS Pearson.

Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (2000). California Verbal Learning Test–Second Edition (CVLT–II). NCS Pearson.

Delis, D. C., Kramer, J. H., Kaplan, E., & Ober, B. A. (1994) California Verbal Learning Test–Children’s Version (CVLT–C). NCS Pearson.

Eichstadt, T. J., Castilleja, N., Jakubowitz, M., & Wallace, A. (2013, November). Standardized assessment via telehealth: Qualitative review and survey data [Paper presentation]. Annual meeting of the American-Speech-Language-Hearing Association, Chicago, IL United States.

Interorganizational Practice Committee [IOPC]. (2020, March 30). Recommendations/guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic. iopc.online. https://static1.squarespace.com/static/50a3e393e4b07025e1a4f0d0/t/5e8260be9a64587cfd3a9832/1585602750557/Recommendations-Guidance+for+Teleneuropsychology-COVID-19-4.pdf

 

Telehealth–Face-to-Face Mode

  1. Barcellos, L. F, Bellesis, K. H.; Shen, L.; Shao, X., Chinn, T., et al. (2017). Remote assessment of verbal memory in MS patients using the California Verbal Learning Test. Multiple Sclerosis Journal, 24, 354–357. doi:10.1177/1352458517694087 pmid:28273777
  2. Brearly, T., Shura, R., Martindale, S., Lazowski, R., Luxton, D., Shenal, B., & Rowland, J. (2017). Neuropsychological test administration by videoconference: A systematic review and meta-analysis. Neuropsychology Review, 27(2), 174–186.
  3. Cullum, C.M., Weiner, M., Gehrmann, H., & Hynan, L. (2006). Feasibility of telecognitive assessment in dementia. Assessment, 13(4), 385–390.
  4. Cullum, C. M., Hynan, L. S., Grosch, M., Parikh, M., & Weiner, M. F. (2014). Teleneuropsychology: Evidence for video teleconference-based neuropsychological assessment. Journal of the International Neuropsychological Society, 20, 1028–1033.
  5. Galusha-Glasscock, J., Horton, D., Weiner, M., & Cullum, C.M. (2016). Video teleconference administration of the Repeatable Battery for the Assessment of Neuropsychological Status. Archives of Clinical Neuropsychology, 31(1), 8–11.
  6. Hildebrand, R., Chow, H., Williams, C., Nelson, M., & Wass, P. (2004). Feasibility of neuropsychological testing of older adults via videoconference: Implications for assessing the capacity for independent living. Journal of Telemedicine and Telecare, 10(3), 130–134. https://doi.org/10.1258/135763304323070751
  7. Kirkwood, K. T., Peck. D. F., & Bennie, L. (2000). The consistency of neuropsychological assessments performed via telecommunication and face to face. Journal of telemedicine and telecare, 6, 147–151.
  8. Ragbeer, S. N., Augustine, E. F., Mink, J. W., Thatcher, A. R., Vierhile, A. E., & Adams, H. R. (2016). Remote assessment of cognitive function in juvenile neuronal ceroid lipofuscinosis (Batten disease): A pilot study of feasibility and reliability. Journal of Child Neurology, 31, 481–487. https://doi.org/10.1177/0883073815600863
  9. Stain, H. J., Payne, K., Thienel, R., Michie, P., Vaughan, C., & Kelly, B. (2011). The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis. Journal of Telemedicine and Telecare, 17, 328–331. https://doi.org/10.1258/jtt.2011.101015
  10. Vahia, I. V., Ng, B., Camacho, A., Cardenas, V., Cherner, M., Depp, C. A., Palmer, B. W., Jeste, D. V., & Agha, Z. (2015). Telepsychiatry for neurocognitive testing in older rural Latino adults. American Journal of Geriatric Psychiatry 23, 666–670.
  11. Wadsworth, H., Galusha-Glasscock, J., Womack, K., Quiceno, M., Weiner, M., Hynan, L., Shore, J., & Cullum, C.M. (2016). Remote neuropsychological assessment in rural American Indians with and without cognitive impairment. Archives of Clinical Neuropsychology, 31(5), 420–425. https://doi.org/10.1093/arclin/acw030
  12. Wadsworth, H. E., Dhima, K., Womack, K. B., Hart, J., Weiner, M. F., Hynan, L. S., & Cullum, C. M. (2018). Validity of teleneuropsychological assessment in older patients with cognitive disorders. Archives of Clinical Neuropsychology 33(8), 1040–1045. https://doi.org/10.1093/arclin/acx140

Digital–Traditional Format

  1. Daniel, M. H. (2012). Equivalence of Q-interactive administered cognitive tasks: CVLT–II and selected D-KEFS subtests (Q-interactive Technical Report 3). PearsonAssessments.com. https://www.pearsonassessments.com/content/dam/school/global/clinical/us/assets/q-interactive/005-s-Technical%20Report%203_CVLT_DKEFS_final_rev.pdf
  2. Daniel, M. H. (2013). Equivalence of Q-interactive and paper administration of WMS-IV cognitive tasks (Q-interactive Technical Report 6). PearsonAssessments.com. https://www.pearsonassessments.com/content/dam/school/global/clinical/us/assets/q-interactive/0010-s-Technical_Report_6_WMS-IV.pdf