Hello Data Experts,
Let me continue from my last blog “Data Types and Visualization using R” where we discussed Object/Data Types like Vector, List, Factor, Data Frame, Array and Matrix. I did cover how to have a different graphical representation like Line Graph, Scatter Graph, Pie chart, Bar Graph, Histogram and Boxplot Graph.
Let us now move forward with core statistical fundamentals for any statistical problem. We will learn about Exploratory Data Analysis. Why do we need it and how to perform Exploratory Data Analysis (EDA)? This is the foundation for any statistical outcome and 60 % – 70% of the time Data Scientist spend to make sure base data is rightly picked and reflect correct sample.
Let me first help you understand what it is. EDA is important for Data Scientist because it will help us
- Understand key attributes about the data like Mean, Median etc.
- EDA will help us visualize if there are any anomalies like outliers.
- Data visualization help us detect if there is any pattern like direct or indirect relationship between 2 set data points.
- Identify data errors, data inconsistency (like Skewness).
- It will help us validate assumptions if data (sample/Population) is appropriate for statistical modeling.
- It will help us identify the right statistical model and avoid biasness.
- It will help us assess strength and direction of data between Input and output variables.
There are 4 moments of statistics to complete EDA. We will cover first 2 moments in this blog.
- First step covers Mean, Median and Mode, it is a measure of central tendency.
- Second step covers Variance, it is a measure of dispersion.
- Third step covers Skewness, it is a measure of asymmetry.
- Fourth step covers Kurtosis, it is a measure of peakness
First Statistical Moment:
Mean is the average value of datasets. It can be influenced by the outliers. It could also be a measure of central tendency.
Median is the middle most value of the sorted data set, it is partially influenced by outlier. It reflects better distribution of data. It is a better representation of central tendency as there is a lower chance of it getting influenced by outlier. If there are odd number of values in a dataset, then (N+1)/2 th value will be median whereas if there are even number of values then average of N th and (N+1) th value will be the Median value.
Mode is the value which is there most of the times in the dataset i.e., search for most frequent value used.
Second Statistical Moment:
Variance is how spread out are value from the mean. Sum of all residuals of values to Mean should be Zero.
Standard Deviation is calculated as the square root of variance. Statistically Standard deviation reflects how close are values to the mean and spread. SD helps us determines margin of error, confidence level and significance level as well.
Third Statistical Moment:
It is a measure of Skewness where it helps us depicts which side distribution of data is tapered, as against mean. There could be Positive or Negative Skewness.
Negative Skewness reflects left long tail and data is more distributed toward right, for example 3,4,5,6, 8,9
Positive Skewness reflects right long tail and data is more distributed toward left. for example, 4,6, 8,9, 10
Fourth Statistical Moment:
It is a measure of tailedness.
Positive Kurtosis defined Thin Peak with no long tail. If it needs to be explain in retail domain thin peak covers items like bread and milk are sold maximum.
Negative Kurtosis defines wider peak with long tail.
Let us now take an object having 20 values
CarsMileage <- c(12, 14, 12.5, 13.5, 15, 10, 11, 12, 12, 14, 12, 11.5, 12.5, 13.5, 15, 10.5, 15, 12, 14, 14)
Let us derive all 4 moments now:
# Get Mean value
Output will be 12.8
# Get Median value
Output will be 12.5
# Get mode value
Output will be Numeric
# Get Variance
Output will be 2.24736
# Get Standard Deviation
Output will be 1.499123
Third moment and Fourth moment:
I will cover how to derive third and fourth moment in my future blogs.
I hope you must have got an essence and an importance of four moments in the statistical analysis. This is the foundation for rest of the statistical world. 60% – 70% of the time of statistician goes in executing EDA to make sure data captured is complete and unbiased which will help one take right decision and right outcome. Getting right data be it a population or sample, it is very important for one to make a right choice. Now that basics for EDA is clear I will help one with visualization to explore advance statistical problems.
Thank you for sparing time and going through this blog I hope it helped you built sound foundation of statistics using R. Kindly share your valuable and kind opinion. Please do not forget to suggest what you would like to understand and hear from me in my future blogs.
Outstanding Outliers:: “AG”.