#GottaCureEmAll – Pokemon GO teaches healthcare a big lesson

August 1, 2016 § 17 Comments

If we can believe what we are seeing, Pokemon GO is the world’s most effective, and most widespread, population weight loss intervention. Already, its users spend more time on the game than on other wildly popular mainstream social media platforms like Facebook, Snapchat and Twitter. Over the space of a few weeks, it has prompted millions of children and teens to get off the couch, turn off Netflix, leave the laptop in their bedroom, and walk out into the world to breath the fresh air. More than a few adults have done the same.

Healthcare should pay attention. While healthcare researchers are slowly coming to grips with ‘new’ ideas like gamification and social media to defeat obesity, the game industry has jumped the queue and may have already done it. Silicon valley has drawn down on its deep well of expertise in building large and complex software systems, and in embedding such systems into the real world. They have drawn on their deep experience with and understanding of the psychology of online social media, of what makes games ‘fun’, and what makes them ‘sticky’.

I doubt if Niantic, the Pokemon company, looked to randomized clinical trials to design and implement their system. The world of software moves too fast for that. It has an engineering culture of fail early, fail often. And because of that, it has as much right as scientists to claim that it is driven by experimentation and data, or as the philosopher Karl Popper would have said, conjecture and refutation.

For those who have not been drawn in to the world of Pokemon Go, it may be hard to understand what the fuss is all about. It is just another time-wasting, obsession inducing computer game. Yes it is interesting that it uses augmented reality and your physical location as part of the gameplay, but so what? People just walk around collecting different characters, oblivious to what is happening around them. The end result is a different kind of walking screen-time zombie, with the added risk of walking into the traffic or driving into a wall as you play the game.

There is another way to look at it. Firstly, irrespective of the game ‘medium’, the real world ‘message’ is that people are more than happy to exercise, and to engage with others in the real world, with the right motivation. For younger generations who have grown up in a world that is digitally augmented, the digital-social complex is the way to access their lives. Jogging with a fitbit is probably compelling for those who already run or are motivated to exercise. Pokemon GO does something more miraculous. It causes the Lazarus generation to rise up, and to move.

Pokemon GO makes walking the basic currency of the game. If you chance upon the eggs of Pokemon creatures, the only way to make them hatch is to walk a prescribed distance. Some eggs require 10k of nurturing before they crack. If you want to catch different Pokemon (and if you are a player, you #gottaacatchemall), then you will find spawning grounds in parks and open spaces. If you want to top up the items you need to catch Pokemon, then you have to walk from one Pokestop to another.

One of things that appears to make gambling ‘sticky’ is the uncertainty of reward. Each rare win reinforces the desire to keep trying for a bigger future reward. Pokemon GO has an interesting strategy of combining certainty in reward (eggs hatch after a defined distance is walked) with uncertainty (creatures appear unpredictably, and their behavior and value is unpredictable). As you progress in the game the rewards increase along with your status. Our brains are washed in an addictive dopamine broth with every reward, every step forward.

Pokemon GO also strives for social equity. When a creature appears in a given location, anyone who is there can see it and catch their copy of it. This means that there is real value in finding stronger players than yourself, because they will trigger the arrival of rarer creatures. These stronger players are also likely to have set lures to attract creatures, and the benefit of these lures is also socialized. Stronger players will have obtained their status by walking great distances, and so a sort of social modeling probably takes place influencing the behavior of newer players, further reinforcing the culture of movement.

Mass and spontaneous social congregation is an unexpected side-effect. Reports of many thousands of people all rapidly congregating in parks as rare creatures appear have been repeatedly reported. It is a sociologically fascinating emergent property of the game. It can also drive the locals crazy, blocking roads, and keeping people awake, as crowds chase the different creatures that appear at night. How wonderful if healthcare could trigger the same mass interest, with thousands queuing when Zika vaccination becomes available? Indeed, how can we mobilize such mass response for all sorts of health prevention activities?

The early uses of Pokemon GO in healthcare are examples of simple adoption. Placing Pokestops in the wards and surrounds of a children’s ward provides distraction and joy to hospitalized children who may be in dire need of fun. There are also of course the usual reactionary demands that the game be banned from clinical spaces .

Social media has already taught us a lot about how to deliver health services in new ways. The great potential for augmented reality in healthcare is yet to be tapped . Pokemon GO can teach us even more. We must learn to be more nimble and agile in the way we develop interventions to change behavior and deliver health services. The engineering worldview has much to offer, and it shares the same DNA of scientific reasoning so embedded in modern healthcare research. We are entering a time when more and more of the population will be embedded in an online social web, and that will be the universe in which we must engage with them, and where healthcare is delivered. And we must embrace that future, because in the end, we #GottaCureEmAll.





Evidence-based health informatics

February 11, 2016 § 6 Comments

Have we reached peak e-health yet?

Anyone who works in the e-health space lives in two contradictory universes.

The first universe is that of our exciting digital health future. This shiny gadget-laden paradise sees technology in harmony with the health system, which has become adaptive, personal, and effective. Diseases tumble under the onslaught of big data and miracle smart watches. Government, industry, clinicians and people off the street hold hands around the bonfire of innovation. Teeth are unfeasibly white wherever you look.

The second universe is Dickensian. It is the doomy world in which clinicians hide in shadows, forced to use clearly dysfunctional IT systems. Electronic health records take forever to use, and don’t fit clinical work practice. Health providers hide behind burning barricades when the clinicians revolt. Government bureaucrats in crisp suits dissemble in velvet-lined rooms, softly explaining the latest cost overrun, delay, or security breach. Our personal health files get passed by street urchins hand-to-hand on dirty thumbnail drives, until they end up in the clutches of Fagin like characters.

Both of these universes are real. We live in them every day. One is all upside, the other mostly down. We will have reached peak e-health the day that the downside exceeds the upside and stays there. Depending on who you are and what you read, for many clinicians, we have arrived at that point.

The laws of informatics

To understand why e-health often disappoints requires some perspective and distance. Informed observers again and again see the same pattern of large technology driven projects sucking up all the e-health oxygen and resources, and then failing to deliver. Clinicians see that the technology they can buy as a consumer is more beautiful and more useful that anything they encounter at work.

I remember a meeting I attended with Branko Cesnik. After a long presentation about a proposed new national e-health system, focusing entirely on technical standards and information architectures, Branko piped up: “Excuse me, but you’ve broken the first law of informatics”. What he meant was that the most basic premise for any clinical information system is that it exists to solve a clinical problem. If you start with the technology, and ignore the problem, you will fail.

There are many corollary informatics laws and principles. Never build a clinical system to solve a policy or administrative problem unless it is also solving a clinical problem. Technology is just one component of the socio-technical system, and building technology in isolation from that system just builds an isolated technology [3].

Breaking the laws of informatics

So, no e-health project starts in a vacuum of memory. Rarely do we need to design a system from first principles. We have many decades of experience to tell us what the right thing to do is. Many decades of what not to do sits on the shelf next to it. Next to these sits the discipline of health informatics itself. Whilst it borrows heavily from other disciplines, it has its own central reason to exist – the study of the health system, and of how to design ways of changing it for the better, supported by technology. Informatics has produced research in volume.

Yet today it would be fair to say that most people who work in the e-health space don’t know that this evidence exists, and if they know it does exist, they probably discount it. You might hear “N of 1” excuse making, which is the argument that the evidence “does not apply here because we are different” or “we will get it right where others have failed because we are smarter”. Sometimes system builders say that the only evidence that matters is their personal experience. We are engineers after all, and not scientists. What we need are tools, resources, a target and a deadline, not research.

Well, you are not different. You are building a complex intervention in a complex system, where causality is hard to understand, let alone control. While the details of your system might differ, from a complexity science perspective, each large e-health project ends up confronting the same class of nasty problem.

The results of ignoring evidence from the past are clear to see. If many of the clinical information systems I have seen were designed according to basic principles of human factors engineering, I would like to know what those principles are. If most of today’s clinical information systems are designed to minimize technology-induced harm and error, I will hold a party and retire, my life’s work done.

The basic laws of informatics exist, but they are rarely applied. Case histories are left in boxes under desks, rather than taught to practitioners. The great work of the informatics research community sits gathering digital dust in journals and conference proceedings, and does not inform much of what is built and used daily.

None of this story is new. Many other disciplines have faced identical challenges. The very name Evidence-based Medicine (EBM), for example, is a call to arms to move from anecdote and personal experience, towards research and data driven decision-making. I remember in the late ‘90s, as the EBM movement started (and it was as much a social movement as anything else), just how hard the push back was from the medical profession. The very name was an insult! EBM was devaluing the practical, rich daily experience of every doctor, who knew their patients ‘best’, and every patient was ‘different’ to those in the research trials. So, the evidence did not apply.

EBM remains a work in progress. All you need to do today is to see a map of clinical variation to understand that much of what is done remains without an evidence base to support it. Why is one kind of prosthetic hip joint used in one hospital, but a different one in another, especially given the differences in cost, hip failure and infection? Why does one developed country have high caesarian section rates when a comparable one does not? These are the result of pragmatic ‘engineering’ decisions by clinicians – to attack the solution to a clinical problem one way, and not another.  I don’t think healthcare delivery is so different to informatics in that respect.

Is it time for evidence-based health informatics?

It is time we made the praxis of informatics evidence-based.

That means we should strive to see that every decision that is made about the selection, design, implementation and use of an informatics intervention is based on rigorously collected and analyzed data. We should choose the option that is most likely to succeed based on the very best evidence we have.

For this to happen, much needs to change in the way that research is conducted and communicated, and much needs to happen in the way that informatics is practiced as well:

  • We will need to develop a rich understanding of the kinds of questions that informatics professionals ask every day;
  • Where the evidence to answer a question exists, we need robust processes to synthesize and summarize that evidence into practitioner actionable form;
  • Where the evidence does not exist and the question is important, then it is up to researchers to conduct the research that can provide the answer.

In EBM, there is a lovely notion that we need problem oriented evidence that matters (POEM) [1] (covered in some detail in Chapter 6 of The Guide to Health Informatics). It is easy enough to imagine the questions that can be answered with informatics POEMs:

  • What is the safe limit to the number of medications I can show a clinician in a drop-down menu?
  • I want to improve medication adherence in my Type 2 Diabetic patients. Is a text message reminder the most cost-effective solution?
  • I want to reduce the time my docs spend documenting in clinic. What is the evidence that an EHR can reduce clinician documentation time?
  • How gradually should I roll out the implementation of the new EHR in my hospital?
  • What changes will I need to make to the workflow of my nursing staff if I implement this new medication management system?

EBM also emphasises that the answer to any question is never an absolute one based on the science, because the final decision is also shaped by patient preferences. A patient with cancer may choose a treatment that is less likely to cure them, because it is also less likely to have major side-effects, which is important given their other goals. The same obviously holds in evidence-based health informatics (EBHI).

The Challenges of EBHI

Making this vision come true would see some significant long term changes to the business of health informatics research and praxis:

  • Questions: Practitioners will need develop a culture of seeking evidence to answer questions, and not simply do what they have always done, or their colleagues do. They will need to be clear about their own information needs, and to be trained to ask clear and answerable questions. There will need to be a concerted partnership between practitioners and researchers to understand what an answerable question looks like. EBM has a rich taxonomy of question types and the questions in informatics will be different, emphasizing engineering, organizational, and human factors issues amongst others. There will always be questions with no answer, and that is the time experience and judgment come to the fore. Even here though, analytic tools can help informaticians explore historical data to find the best historical evidence to support choices.
  • Answers: The Cochrane Collaboration helped pioneer the development of robust processes of meta-analysis and systematic review, and the translation of these into knowledge products for clinicians. We will need to develop a new informatics knowledge translational profession that is responsible for understanding informatics questions, and finding methods to extract the most robust answers to them from the research literature and historical data. As much of this evidence does not typically come from randomised controlled trials, other methods than meta-analysis will be needed. Case libraries, which no doubt exist today, will be enhanced and shaped to support the EBHI enterprise. Because we are informaticians, we will clearly favor automated over manual ways of searching for, and summarizing, the research evidence [2]. We will also hopefully excel at developing the tools that practitioners use to frame their questions and get the answers they need. There are surely both public good and commercial drivers to support the creation of the knowledge products we need.
  • Bringing implementation science to informatics: We know that informatics interventions are complex interventions in complex systems, and that the effect of these interventions vary depending on the organisational context. So, the practice of EBHI will of necessity see answers to questions being modified because of local context. I suspect that this will mean that one of the major research challenges to emerge from embracing EBHI is to develop robust and evidence-based methods to support localization or contextualisation of knowledge. While every context is no doubt unique, we should be able to draw upon the emerging lessons of implementation science to understand how to support local variation in a way that is most likely to see successful outcomes.
  • Professionalization: Along with culture change would come changes to the way informatics professionals are accredited, and reaccredited. Continuing professional education is a foundation of the reaccreditation process, and provides a powerful opportunity for professionals to catch up with the major changes in science, and how those changes impact the way they should approach their work.


There comes a moment when surely it is time to declare that enough is enough. There is an unspoken crisis in e-health right now. The rhetoric of innovation, renewal, modernization and digitization make us all want to believers. The long and growing list of failed large-scale e-health projects, the uncomfortable silence that hangs when good people talk about the safety risks of technology, make some think that e-health is an ill-conceived if well intentioned moment in the evolution of modern health care. This does not have to be.

To avoid peak e-health we need to not just minimize the downside of what we do by avoiding mistakes. We also have to maximize the upside, and seize the transformative opportunities technology brings.

Everything I have seen in medicine’s journey to become evidence-based tells me that this will not be at all easy to accomplish, and that it will take decades. But until we do, the same mistakes will likely be rediscovered and remade.

We have the tools to create a different universe. What is needed is evidence, will, a culture of learning, and hard work. Less Dickens and dystopia. More Star Trek and utopia.

Further reading:

Since I wrote this blog a collection of important papers covering the important topic of how we evaluate health informatics and choose which technologies are fit for purpose has been published in the book Evidence-based Health Informatics.


  1. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. The Journal of Family Practice 1994;38(5):505-13
  2. Tsafnat G, Glasziou PP, Choong MK, et al. Systematic Review Automation Technologies. Systematic Reviews 2014;3(1):74
  3. Coiera E. Four rules for the reinvention of healthcare. BMJ 2004;328(7449):1197-99


 An Italian translation of this article is available


A brief guide to the health informatics research literature

February 8, 2016 § Leave a comment

Every year the body of research evidence in health informatics grows. To stay on top of that research, you need to know where to look for research findings, and what the best quality sources of it are. If you are new to informatics, or don’t have research training, then you may not know where or how to look. This page is for you.

There are a large number of journals that publish only informatics research. Many mainstream health journals will also have an occasional (and important) informatics paper in them. Rather than collecting a long list of all of these possible sources, I’d like to offer the following set of resources as a ‘core’ to start with.

(There are many other very good health informatics journals, and their omission here is not meant to imply they are not also worthwhile. We just have to start somewhere. If you have suggestions for this page I really would welcome them, and I will do my best to update the list).


If you require an overview of the recent health informatics literature, especially if you are new to the area, then you really do need to sit down and read through one of the major textbooks in the area. These will outline the different areas of research, and summarise the recent state of the art.

I am of course biased and want you to read the Guide to Health Informatics.

A collection of important papers covering the important topic of how we evaluate health informatics and choose which technologies are fit for purpose can be found in the book Evidence-based Health Informatics.

Another text that has a well-earned reputation is Ted Shortliffe’s Biomedical Informatics.

Health Informatics sits on the shoulders of the information and computer sciences, psychology, sociology, management science and more. A mistake many make is to think that you can get a handle on these topics just from a health informatics text. You wont. Here are a few classic texts, from these ‘mother’ disciplines;

Computer networks (5th ed). Tannenbaum and Wetherall.  Pearson. 2010.

Engineering Psychology & Human Performance (4th ed.). Wickens et al. Psychology Press. 2012.

Artificial Intelligence: A Modern Approach  (3rd ed). Russell and Norvig. Pearson. 2013


Google Scholar: A major barrier to accessing the research literature is that much of it is trapped behind paywalls. Unless you work at a university and can access journals via the library, you will be asked by some publishers to pay an exorbitant fee to read even individual papers. Many journals are now however open-access, or make some of their papers available free on publication. Most journals also allow authors to freely place an early copy of a paper onto a university or other repository.

The most powerful way to finding these research articles is Google Scholar. Scholar does a great job of finding all the publicly available copies of a paper, even if the journal’s version is still behind a paywall. Getting yourself comfortable with using Scholar, and exploring what it does, provides you with a major tool for accessing the research literature.

Yearbook of Medical Informatics. The International Medical Informatics Association (IMIA) is the peak global academic body for health informatics and each year produces a summary of the ‘best’ of the last year’s research from the journals in the form of the Yearbook of Medical Informatics. The recent editions of the yearbook are all freely available online.

Next, I’d suggest the following ‘core’ journals for you to skim on a regular basis. Once you are familiar with these you will no doubt move on the the very many others that publish important informatics research.

JAMIA. The Journal of the American Medical Informatics Association (AMIA) is the peak general informatics journal, and a great place to keep tabs on recent trends. While it requires a subscription, all articles are placed into open access 12 months after publication (so you can find them using Scholar) and several articles every month are free. You can keep abreast of papers as they are published through the advanced access page.

JMIR. The Journal of Medical Internet Research is a high impact specialist journal focusing on Web-based informatics interventions. It is open access which means that all articles are free.

To round out the journals you might want to add into your regular research scan the following journals which are all very well regarded.


Whilst journals typically will publish well polished work, there is often a lag of a year or more before submitted papers are published. The advantage of research conferences is that you get more recent work, sometimes at an earlier stages of development, but also closer to the cutting edge.

There are a plethora of informatics conferences internationally but the following publish their papers freely online, and are typically of high quality.

AMIA Annual Symposium. AMIA holds what is probably the most prestigious annual health informatics conference, and releases all papers via NLM. An associated AMIA Summit on Translational Sciences/Bioinformatics is also freely available.

Medinfo. IMIA holds a biannual international conference, and given its status as the peak global academic society, Medinfo papers have a truly international flavour. Papers are open access and made available by IOS press through its Studies in Health Technology and Informatics series (where many other free proceedings can be found). Recent Medinfo proceedings include 2015 and 2013.

As with textbooks and journals, it is worth remembering that much of importance to health informatics is published in other ‘mother’ disciplines. For example it is well worth keeping abreast of the following conferences for recent progress:

WWW conference. The World Wide Web Conference is organised by the ACM and is an annual conference looking at innovations in the Internet space. Recent proceedings include 2015 and 2014.

The ACM Digital Library, which contains WWW, is a cornucopia of information and computer science conference proceedings. Many a rainy weekend can be wasted browsing here. You may need to hunt the web site of the actual conference however to get free access to papers as ACM will often try to charge for papers you can find freely on the home page of the conference.

Other strategies

Browsing journals is one way to keep up to date. The other is to follow the work of individual researchers whose interests mirror your own. The easiest way to do this is to find their personal page on Google Scholar (and if they don’t have one tell them to make one!). Here is mine, as an example. There are two basic ways to attack a scholar page. When you first see a Scholar page, the papers are ranked by their impact (as measured by other people citing the papers). This will give you a feeling about the work the researcher is most noted for. The second way is to click the year button. You will then see papers in date order, starting with the most recent. This is a terrific way of seeing what your pet researcher has been up to lately.

There is a regularly updated list of biomedical informatics researchers, ranked by citation impact, and this is a great way to discover health informatics scientists. Remember that when researchers work in more specialised fields, they may not have as many citations and so be lower down the list.

Once you find a few favourite researchers, try to see what they have done recently, follow them on Twitter, and if they have a blog, try to read it.

How to get a PhD in Twenty Tweets

May 24, 2013 § Leave a comment

After two and a bit decades of working at the craft of innovation in technology, I was pulled up one day when one of my brilliant doctoral students asked me “How do you get a good idea?”. As I started to answer, I realised that innovative thinking is something I had learned, slowly and that in truth it could (and should) be taught explicitly.

My first attempt at writing down what I thought the ‘rules’ for innovation resulted in a series of twenty tweets – 10 do’s and 10 don’ts. It seemed like a fun way of keeping to the point, sharing, and seeing what others thought.

My bigger surprise as I embarked on the tweeting was the disdain my senior colleagues had for the social medium. I put my experiences down in a blog at the BMJ, as well as the twenty tweets.

I may return to these and expand, but for now, here is the original blog:


Where Am I?

You are currently browsing the Research Methods category at The Guide to Health Informatics 3rd Edition.

%d bloggers like this: