This post is also available in: हिन्दी (Hindi)
What is Data Wrangling?
Data is transforming the world every day. Most of the applications rely on these data. Real-world data are often messy and unorganized. Data Scientists spend nearly 70 percent of the time cleaning and preparing data because not all data out there can be useful in their raw format. One of the most important skills that a data scientist should have is the ability to extract and clean data. This is usually referred to as Data Wrangling or Data Munching.
Data Wrangling is the process of converting and mapping data from its raw form to another format with the purpose of making it more valuable and appropriate for advanced tasks such as Data Analytics and Machine Learning.
Importance of Data Wrangling
Data wrangling is very important because it’s the sole way to make use of raw data. In real-world business settings, the user information comes in different pieces from different backgrounds at different times. Sometimes, we store this information across various computers across different spreadsheets which can lead to data redundancy, incorrect data, or missing data. To create a transparent and efficient system for data management, the best solution is to have all data in a centralized location so it can be used easily.
The following example will explain the importance of Data Wrangling:
A book-selling website wants to show top-selling books of different domains, according to user preference. For example, a new user searches for motivational books, and the website wants to show those books which sell the most or have a high rating, etc.
But on their website, there may be plenty of raw data. Data wrangling comes to the rescue at this point which is done by the Data Scientists. The data scientist will wrangle data in such a way that motivational books are sorted to show the ones sold more or have high ratings on the top of the list. On the basis of that, the new user makes a choice.
Basic Steps in Data Wrangling
Data wrangling is as much a part of the data analysis process as the final results. Wrangling, properly conducted, gives you insights into the nature of your data that then allow you to ask better questions of it. Wrangling is not something that’s done in one swoop, but iteratively. Each step in the wrangling process exposes new potential ways that the data might be “re-wrangled”, all driving towards the goal of generating good centralized data.
Following are the six basic steps involved in data wrangling:
1. Discovering
During this step, you learn what is in your data and what might be the best approach for predictive analytic explorations. For example, if you have a customer data set, and you learn that most of your shoppers are from a single part of the country, you’re going to keep that in mind as you proceed with your data work. You will keep in mind the weather and geographic conditions of that region while promoting your products.
2. Structuring
Structuring is needed because data comes in all shapes and sizes. For example, you might have a transaction log where each entry might have one or more items associated with it. To conduct an inventory analysis, you will likely need to expand each transaction into individual records for each purchased item. Alternatively, you might want to analyze which products are often bought together. In that case, expanding each transaction into every pair of purchased items might be appropriate.
3. Cleaning
Cleaning involves taking out data that might distort the analysis. A null value, for example, might bring an analytic package to a screeching halt. So, you may want to replace it with zero or an empty string. You might want to standardize a particular field, replacing the many different ways that a state might be written out – such as CA, Cal, and Cf – with a single standard format.
Cleaning requires knowledge about data quality and consistency – knowing how various data values might impact your final analysis.
4. Enriching
Enriching allows you to take advantage of the wrangling you have already done to ask yourself: ‘Now that I have a sense of my data, what other data might be useful in this analysis”’. Or, “What new kinds of data can I derive from the data I already have?”.
In other words, enriching is polishing the data. For example, adding some related items in your database that are similar to the ones searched by the users most.
5. Validating
Validating is the activity that surfaces data quality and consistency issues, or verifies that they have been properly addressed by applied transformations. Validations should be conducted along multiple dimensions. At a minimum, assessing whether the values of an attribute/field adhere to syntactic constraints. For example, boolean fields encoded as ‘true’/‘false’ as opposed to some other values. Additional validations might involve cross-attribute/field checks like ensuring all negative bank transactions have the appropriate transaction type (e.g., ‘withdrawal’, ‘bill pay’, or ‘cheque’).
6. Publishing
Publishing refers to planning for and delivering the output of your data wrangling efforts for downstream project needs or for future project needs. Across projects, it often makes sense to replicate a set of data wrangling steps for re-use on other datasets. Experienced data analysts maintain a library (often personal, sometimes shared) of common transformation logic that they can leverage new projects. For example, in food preparation, there are actions that can be taken to speed up cooking items or to improve the flavour or texture development of a final dish.
