Data Exploration

Understanding Data Exploration in the Real World

Data exploration tries to make clear that the quality of data matters and the garbage-in, garbage-out (GIGO) rule applies. Within data exploration that is basically a host of different methods used to analyze data, in many cases, the tools are used multiple times to further streamline the information found. For example, within a single set of data, analysts would be looking for:

• Values – what are the different values present in the data and how can the data be represented to highlight them?
• Quantity – how many times is the unique value represented in the data set? What is the overall frequency and count?
• Statistical Analysis – tools like the mean, median and mode are used to understand the variance in the data and what the overall spread is.
• Data Analysis – tools like Pareto (80/20 rule) are used to further categorize important information and data. In addition, by using histograms and heat maps, analysts are able to quickly identify relevant and applicable data to find correlations.
• Data Clustering – the world is full of data and the amount available is only increasing. Data clustering lets us look at data correlations from a high level and thus enables focus on groups of data instead on specific data points.
• Data Outliers – at times, some data simply does not match. In this case, it is known as an outlier or an anomaly and generally represents an exception. It is important to understand outliers also however, as, while they may be rare, they could happen and plans should be created to deal with them. A good example of this is an outage in a technical department.

Once all of these techniques were used, they are run again, many times over to verify the initial hypothesis or in some cases disprove it. While it can take a significant amount of time, it is an excellent way of using real information to build a strategic case.

How Data Exploration Works in the World of AI

When considering AI, it is important to recognize that data input plays a critical role. In the early stages, data is used to teach and educate AI systems. In this case, data that is incorrectly tagged can impact AI systems significantly leading to false positives or worse. However, when it comes to data exploration, AI is instrumental in saving time. AI systems can quickly find patterns in data and identify correlations as well as outliers.