The Problem with Big Data: Operating on Smaller Datasets to Bridge the Implementation Gap

Richard P. Mann, Faisal Mushtaq, Alan D. White, Gabriel Mata-Cervantes, Tom Pike, Dalton Coker, Stuart Murdoch, Tim Hiles, Clare Smith, David Berridge, Suzanne Hinchliffe, Geoff Hall, Stephen Smye, Richard M. Wilkie, J. Peter A. Lodge, Mark Mon-Williams

Big datasets have the potential to revolutionize public health. However, there is a mismatch between the political and scientific optimism surrounding big data and the public's perception of its benefit. We suggest a systematic and concerted emphasis on developing models derived from smaller datasets to illustrate to the public how big data can produce tangible benefits in the long term. In order to highlight the immediate value of a small data approach, we produced a proof-of-concept model predicting hospital length of stay. The results demonstrate that existing small datasets can be used to create models that generate a reasonable prediction, facilitating health-care delivery. We propose that greater attention (and funding) needs to be directed toward the utilization of existing information resources in parallel with current efforts to create and exploit "big data."