Dark data is data that is unknown or missing. According to David Hand, Emeritus Professor at the Imperial College in London, it can have serious implications on scientific research and development.
In his article “Dark Data and the Pandemic” he writes that at the beginning of the Covid-19 pandemic, data was scarce, and expectations were high. However, “it is unrealistic to expect data adequately describing new and previously unencountered situations to be ready-made, ripe for analysis. Rather, we have to develop data collection strategies and establish measurement procedures. Then we must collect, collate and interpret the data. And while this is being done, we have to do the best we can with the limited data at hand.”
This is really the key: to be aware that data is missing and take this situation into account when developing predictive models and drawing conclusions.
Science is not a fixed collection of facts, but rather something which is always susceptible to change as new information becomes available.
During the pandemic statistics and data science caught the public eye. The conclusions drawn by politicians were often under scrutiny and public criticism. It is important to understand that science is a process, says Hand, “not a fixed collection of facts, but rather something which is always susceptible to change as new information becomes available.”
David Hand will be a keynote speaker on May 6 at the online panel discussion “Demystifying Machine Learning and Artificial Intelligence”.His talk will be followed by a panel discussion with industry experts from Novozymes, Solvay and Corning, who will share how they got started with analytics and machine learning techniques to gain a better understanding of their data, predict and communicate outcomes to significantly improve processes. Join us on May 6 for an inspiring discussion.
You can sign up for this free event here >>
Bio - David J. HandSenior Research Investigator and Emeritus Professor of Mathematics, Imperial College London
David Hand, PhD, is a Fellow of the British Academy and a recipient of the Guy Medal of the Royal Statistical Society. He has served two terms as President of the Royal Statistical Society and is also the author of The Improbability Principle and Dark Data. Hand was made OBE for services to research and innovation in 2013.
Nina Chan, Sr. Marketing Specialist at JMP Software