Flying Blind: When Our New Tech Had No Metrics, We Built Them

Our Virtual Nurses Were Working 2 Hours a Week. Here’s How We Found Out.

As part of a groundbreaking new care model, we integrated virtual nurses (VNs) to support our frontline teams with admissions, discharges, and patient education. It was a strategic investment in efficiency and support. But soon after launch, we hit a wall. We couldn't get consistent data on how often the VNs were actually being used.

The one piece of data we did receive was alarming: over a 36-hour work week, the VNs had spent a total of just two hours in the virtual platform. After that, the data stream stopped. We were flying blind. Was this million-dollar investment paying off? Was it helping our staff and patients? We had no way to know.

The Challenge: A National Gap in Measurement

This roadblock revealed a much larger problem. We searched for a validated tool to measure the usability and effectiveness of telehealth within a hospital setting and found nothing. All existing questionnaires were designed for external telehealth, where a patient at home connects with a remote provider. A new care model required a new yardstick.

The Solution: Creating the In-Hospital Telehealth Questionnaire (IHTQ)

If the tool you need doesn't exist, you build it. We initiated a rigorous, IRB-approved research study to develop and validate the In-Hospital Telehealth Questionnaire (IHTQ) from the ground up. This wasn't just a feedback form; it was a scientific instrument designed to be reliable and valid for both clinicians and patients.

The process was deeply collaborative, incorporating feedback from the very people using the technology. We learned invaluable lessons during pilot testing:

• Initial patient reactions ranged from "creepy" to "fine," highlighting the need for better patient education.

  
  

• Some patients mistook the live nurse for a robot or AI, underscoring the importance of clarifying the technology.

  
  

• Introducing the technology to a post-op patient was ineffective; timing was critical.

  
  

This real-world feedback didn't just shape the questions on our survey; it fundamentally improved our entire virtual nursing implementation strategy.

The Results: A Validated Tool and a Clear Path Forward

The final 14-item IHTQ is a powerful and reliable tool, demonstrating exceptional internal consistency (Cronbach's alpha of 0.9693) . It has been validated across medical-surgical, oncology, and orthopedic populations.

Most importantly, it gave us the data we needed. We found that as VN utilization increased over 90 days, our IHTQ scores for satisfaction and usability also increased, proving the tool was an effective measure of success.

The Takeaway for Leaders

You cannot manage what you do not measure. When you pioneer new technologies, you must also pioneer the methods to evaluate them. This project turned a moment of frustration into a national contribution. We are now sharing the IHTQ freely with other researchers and health systems, helping to advance the entire field of virtual nursing. It’s a powerful reminder that sometimes the most valuable innovations aren't the technologies themselves, but the tools we build to understand them.