What is Data Wrangling and Data Quality and why it's important?

Data Wrangling

Data Wrangling is the process of transforming raw or received data into a format that can be analyzed to create insights. This involves making decisions about the quality of the data. This is because most of the available data is not of high quality. This process is more than programming and data manipulation. Decisions and selections need to be made that affect the final data set.

Important steps in the data dispute process include:

Searching or storing information
After checking the information
cleaning Standardizing, correcting, and updating data
data analysis
Data display

Data quality

Data quality refers to the reliability and accuracy of the data. This is critical to gaining meaningful insights. Not all data is of the same quality. And poor quality data leads to flawed conclusions. Monitoring data quality is an important part of data disputes.

Although computers are powerful But he only obeyed human orders. and is limited to matching patterns only based on the information provided. Humans play a key role in data collection, analysis, and quality assurance. This is because computers cannot make creative decisions or understand context.

Data quality assessment has two main points:

Data Integrity – How accurate and reliable is the data?
Fit for purpose - whether the information is appropriate for the specific question or problem being solved.

What is Data integrity?

Data integrity refers to the quality and reliability of data values ​​and descriptors in a dataset. in evaluating completeness Consider whether the measurement will be carried out regularly. Represents individual readings or averages. And is there a database that explains how the data should be stored or interpreted (e.g. relevant units)?

What is data fit?

Data "fit" refers to how well a dataset fits a specific purpose or query. Although the dataset is highly complete, But if it does not meet the needs of the analysis It may not be useful, for example real-time Citi Bike data may be of good quality. But it's not suitable for answering questions about how bike stations change from day to day. Citi Bike travel history information would be more appropriate...

Determining the suitability of data often requires an assessment of its completeness. Shortcuts to this process can affect the quality of the analysis and lead to incorrect conclusions. Problems with appropriate data, such as using income data to answer questions about education. It can distort findings and lead to dangerous results. Although sometimes the use of proxy measures may be necessary, Especially in urgent situations But doing so on a large scale can amplify errors. and distort the real-world phenomena that the data is intended to describe…

The completeness and appropriateness of the data are carefully assessed to prevent these errors.

High-integrity data is complete, atomic, and well-annotated. This allows for more detailed analysis. However, many datasets lack these features. And it is up to analysts to understand and improve on these limitations. They often search for additional information or consult experts who are familiar with the data set or field of study..

Check out this url and use pdf and ipynb file github