It seems that 2012 was the year of big data – at least as far as the vendors were concerned. Nearly every vendor in the market had something that could be marketed as part of a solution for big data, and this massive raft of misplaced technologies has led to misperceptions and misunderstanding of what is really needed for an organisation to really deal with the problem.
Big data – big problems, or massive opportunities?
Along with the increasing ubiquity of technology comes the increase in the amount of electronic data. Not many years ago, corporate databases tended to be measured in the range of tens to hundreds of gigabytes. Now, multi-terabyte (TB) or even petabyte (PB) databases are quite normal, with the World Data Center for Climate (WDDC) storing over 6 petabytes of data overall (although all but around 220TB of this is held in archive on tape) and the National Energy Research Scientific Computing Center (NERSC) having over 2.8PB of available data around atomic energy research as well as physics projects and so on.