Twitter Holds Patient Experience Metrics
When you think of using social media to conduct groundbreaking customer research, a hospital probably isn’t the first place you’d look. But the Computational Epidemiology Group in the Boston Children’s Hospital’s Computational Health Informatics Program (CHIP) is a leader in mining social channels for patient experience metrics.
It is now looking to sentiment analysis on Twitter to complement more traditional research methods and help hospital administrators better track patient experience metrics.
The team at Boston Children’s Hospital first attracted attention to their work a few years ago when they successfully mined reviews on the popular rating website Yelp. Their analysis showed the ingredients people described in their reviews mentioning food poisoning matched closely with ingredients public health officials later reported were involved in food-borne illness outbreaks for that time.
It was an important finding since it showed that socially shared information potentially carried useful health insights. Dr. John Brownstein, BCH’s Chief Innovation Officer and a leader of the research team, notes, “It’s hard to make people come to you. People aren’t engaged necessarily in public health.” But if you can tap into their online voices, he says, “You can actually get a huge amount of information that would not come from another vehicle.”
A faster, more useful picture of hospital quality?
Jared Hawkins, a faculty member at Boston Children’s Hospital and a researcher on Brownstein’s team, actively develops new tools and methods that extend the usefulness of sentiment analysis. We recently spoke with Hawkins about research he’s published using Twitter sentiment as a proxy for hospital quality metrics.
As Hawkins explains, quality metrics like the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey are extremely important to hospital administrators. They reflect the quality of care being provided and drive everything from high-profile rankings to federal government reimbursement rates. And yet, these reports often lag today’s events by 12 or more months because of the need to survey patients and tabulate findings.
Many administrators also complain about low response rates and insufficient granularity in the data. And, of course, the surveys sometimes overlook non-clinical (but still important) parts of the patient experience—parking, food, traffic and so forth.
What if, Hawkins wondered, commentary on Twitter could serve as a proxy for these official quality scores, or serve as a new satisfaction metric? Even if potentially less accurate, could the timeliness and granularity of a Twitter-powered sentiment score still make it useful for reporting and decision-making?
“Given that social media data are close to real time, we wanted to see if we could capture this discussion and if the content is useful,” Hawkins said.
Analyzing tweets for emotional intent
With Twitter’s support, Hawkins and his colleagues began by developing a collection of hundreds of thousands of tweets that explicitly mentioned hospitals in the 12-month period between late 2012 and late 2013.
They first weeded out tweets that had nothing to do with patient experience. Hawkins is proud of their efforts here to identify and isolate specific topics. They pushed the envelope of artificial intelligence and machine learning to automatically eliminate tweets related to, say, fundraising, open jobs, or clinical research. It’s easier said than done at scale.
The team then trained a computer algorithm to parse the remaining tweets and watch for expressions of anger, happiness and most every emotion in between. Reading each tweet to determine meaning and sentiment would never have worked given the size of their dataset. Hawkins’ team instead relied again upon sophisticated algorithms to do the work for them.
Importantly, Hawkins’ experience had taught him that sentiment analysis can be tricky since people communicate in confusing and inconsistent ways. Consider the difficulty of teaching an algorithm to differentiate between, “They would not let my dog stay in this hospital.” vs. “I would not let my dog stay in this hospital.”
Slang, abbreviations, emojis, foreign languages, and spelling mistakes added to the challenges Hawkins’ team faced. Both open-source and internally developed tools helped get the job done with surprising accuracy.
Useful, if not perfect, insights into patient experience metrics
Ultimately, Hawkins’ research helped him demonstrate that his hypothesis was mostly right. Eliminating hospitals that didn’t attract enough tweets to offer reliable data, he showed an interesting correlation between some important hospital quality outcomes and his Twitter sentiment scores.
Importantly, though, they did not see a relationship between tweet sentiment and the gold-standard HCAHPS experience data.
“This is a brand-new way of using Twitter data,” Hawkins notes. “It may be that we have to be cautious about using tweet sentiment to understand quality.”
But obsessing about perfection misses the point, he argues, if the speed of getting a “good enough” insight can be substantially improved. And research continues that could further improve the accuracy of his findings. He’s already underway with that effort.
Hawkins went on to rate hospitals based on his findings naming Seattle Children’s Hospital the Mayo Clinic, and Boston Children’s Hospital as the early leaders in Twitter-based public sentiment. They’re all highly regarded institutions so it shouldn’t be a surprise that their patients praise them on Twitter.
Looking to the future, Hawkins envisions a number of enhancements. He’d first like to better understand the strength of the sentiment that patients are communicating. Was the patient furious or merely unhappy?
It’d also be useful, he thinks, if the cause of the sentiment could be identified. Did the food make the patient unhappy? Or was it a surly physician? Daily, weekly or even real-time reports are on his roadmap, too.
A larger dataset and additional data sources could be helpful, too. Taking a beat from J. D. Power and Associates, Hawkins can one day envision social metrics powering awards and recognition. The team is looking to commercialize some of these ideas and has launched the service CrowdClinical to explore possibilities.
Even with these improvements, it’s important to note that Hawkins doesn’t see social sentiment metrics ever replacing conventional survey tools. At best, he thinks they could be a useful complement that helps administrators quickly understand patient sentiment and then make informed plans to improve their experience.
Getting started with sentiment analysis
Sentiment analysis holds much promise for hospital marketers and anyone concerned about patient experience management. From what Hawkins has done and is planning to explore, it’s not hard to imagine sentiment analysis shedding fresh insights on tough questions like:
- How do our patients feel about our hospital and the experience they’ve had with us?
- How are these feelings about us changing over time?
- How can we improve our patients’ experiences with us and create a competitive advantage?
Proponents of sentiment analysis believe it could eventually lead to better insights than traditional approaches to patient research while at the same time being both cheaper and faster. This may be an overly optimistic outlook, but it points to the interesting potential of Hawkins’ work.
So what can you do as a hospital marketer if you’re curious about sentiment analysis? Start small and for free by simply using Twitter’s search tool to find @mentions of your organization with a happy emoticon and then a sad one. You may be surprised at what you find.
Then experiment with some of the free tools available online. We’ve seen interesting results using SocialMention, Sentiment140 and Sentiment Viz. Beyond these, there’s a whole tier of professional services like Salesforce’s Radian6, Adobe Social, and Crimson Hexagon for social teams to consider. They’re not inexpensive but could pay dividends with the valuable insights you’ll glean.
Your patients are talking about you and their experience with your hospital. If you’re ready to listen to these conversations and learn from them, sentiment analytics could be just the tool for you.
This article originally appeared in eHealthcare Strategy & Trends. The author thanks S. Cannon of Nationwide Children’s Hospital, T. Gilkey and J. Hawkins of Boston Children’s Hospital, and D. Hinmon of the Mayo Clinic’s Social Media Network for their comments and suggestions. No endorsement is implied.Tags: Analytics, Healthcare, patient experience, social media