The fact that the word big is a relative quantity doesn't help either. What big is for a midsize European company, can be medium for a large US company. And is it really about the amount of the data? Or is it more about what we do with it, for example, we analyze that data (regardless of the quantity). The V's (Volume, Velocity, Variety, Variation, Visibility, and Value--I've lost count of how many V's there are) are mentioned regularly to describe when something qualifies as big data.
Some have presented definitions, but I haven't seen an acceptable one yet. One author used the following definition: big data is data that is too much for a SQL database. This makes no sense. For example, there are plenty of multi-terabyte systems that everyone would classify as big data systems and that can be handled by SQL products more than satisfactorily.
Lastly, enough data is enough data. The quality of an analytical result doesn't always increase when the amount of data increases. Data quality is often more important than data quantity.
Conclusion, confusion rules when it relates to the concept of big data.
In this blog I look at big data systems from a different angle in the hope that this helps to clarify this muddled concept.
Undeniably, processing large quantities of data is a common characteristic of most big data systems, but there is another one, and that is that most of such systems combine characteristics of production systems and of BI systems. In a sense each big data system is a production system, because it collects and stores new data, plus it's a BI system, because this new data is not collected to support business processes, but the primary intention is to use it for some form of analytics, possibly embedded analytics (analytics embedded within production systems), operational analytics, or predictive analytics. With new data I mean data that is not collected and stored by the organization yet, and in many cases it's also a new type of data. For example, a big data system developed by a retail company may be gathering camera data for tracking customer routes through their stores. Or, a big data system of a large international electronics firm may collect unstructured social media data for sentiment analysis.
Traditionally, new data is entered with and processed by production systems, such as a general ledger, cash management, and claim processing systems. These systems are, however, not designed to support analytics, but are designed to support business processes. In fact, when they were designed, the focus was definitely not on analytics, but on supporting data entry. This is why it's sometimes so hard when developing BI systems to extract the right data from those production databases for analytical and reporting purposes--staging areas have to be developed, ETL and replication processes have to be designed, and so on. This is still true today: the designers of new production systems don't think about how the organization can use the data for analytical purposes.
In other words, what makes big data systems special is that they are hybrid systems, they are production systems and BI systems. In my opinion, this is what makes big data applications special--and, evidently, most of them collect massive amounts of data to supports the required forms of analytics.
So, maybe we should redefine the term big data. Let's begin by not associating the word big with a relative quantity anymore, but let's change the word big to an acronym, so that BIG data stands for Business Intelligence Generated data--data generated and stored with the primary purpose to analyze it. Thus, a big data system is a system that generates, collects, stores, and processes data specifically to support business intelligence. Subsequently, big data is data managed by a big data system.
Hopefully, by redefining the term big data it becomes more obvious what is meant with this promising category of systems and gets rid of some of the confusion.
Posted October 16, 2012 1:45 PM
Permalink | 2 Comments |