Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data

Hello Data Experts,

Let me continue from my last blog “Data Types and Visualization using R” where we discussed Object/Data Types like Vector, List, Factor, Data Frame, Array and Matrix. I did cover how to have a different graphical representation like Line Graph, Scatter Graph, Pie chart, Bar Graph, Histogram and Boxplot Graph.

Let us now move forward with core statistical fundamentals for any statistical problem. We will learn about Exploratory Data Analysis. Why do we need it and how to perform **Exploratory Data Analysis (EDA)**? This is the foundation for any statistical outcome and 60 % – 70% of the time Data Scientist spend to make sure base data is rightly picked and reflect correct sample.

Let me first help you understand what it is. EDA is important for Data Scientist because it will help us

- Understand key attributes about the data like Mean, Median etc.
- EDA will help us visualize if there are any anomalies like outliers.
- Data visualization help us detect if there is any pattern like direct or indirect relationship between 2 set data points.
- Identify data errors, data inconsistency (like Skewness).
- It will help us validate assumptions if data (sample/Population) is appropriate for statistical modeling.
- It will help us identify the right statistical model and avoid biasness.
- It will help us assess strength and direction of data between Input and output variables.

There are **4 moments** of statistics to complete EDA. We will cover first 2 moments in this blog.

- First step covers Mean, Median and Mode, it is a
**measure of central tendency**. - Second step covers Variance, it is a
**measure of dispersion**. - Third step covers Skewness, it is a
**measure of asymmetry**. - Fourth step covers Kurtosis, it is a
**measure of peakness**

**First Statistical Moment:**

**Mean** is the average value of datasets. It can be influenced by the outliers. It could also be a measure of central tendency.

**Median** is the middle most value of the sorted data set, it is partially influenced by outlier. It reflects better distribution of data. It is a better representation of central tendency as there is a lower chance of it getting influenced by outlier. If there are odd number of values in a dataset, then (N+1)/2 th value will be median whereas if there are even number of values then average of N th and (N+1) th value will be the Median value.

**Mode** is the value which is there most of the times in the dataset i.e., search for most frequent value used.

**Second Statistical Moment:**

**Variance **is how spread out are value from the mean. Sum of all residuals of values to Mean should be Zero.

**Standard Deviation** is calculated as the square root of variance. Statistically Standard deviation reflects how close are values to the mean and spread. SD helps us determines margin of error, confidence level and significance level as well.

**Third Statistical Moment:**

It is a measure of Skewness where it helps us depicts which side distribution of data is tapered, as against mean. There could be Positive or Negative Skewness.

**Negative Skewness **reflects left long tail and data is more distributed toward right, for example 3,4,5,6, 8,9

**Positive Skewness** reflects right long tail and data is more distributed toward left. for example, 4,6, 8,9, 10

**Fourth Statistical Moment:**

It is a measure of tailedness.

**Positive Kurtosis** defined Thin Peak with no long tail. If it needs to be explain in retail domain thin peak covers items like bread and milk are sold maximum.

**Negative Kurtosis** defines wider peak with long tail.

Let us now take an object having 20 values

**CarsMileage <- c(12, 14, 12.5, 13.5, 15, 10, 11, 12, 12, 14, 12, 11.5, 12.5, 13.5, 15, 10.5, 15, 12, 14, 14)**

Let us derive all 4 moments now:

First moment:

Mean:

**# Get Mean value**

** mean(CarsMileage)**

Output will be 12.8

Median:

**# Get Median value**

** median(CarsMileage)**

Output will be 12.5

Mode:

**# Get mode value**

** mode(CarsMileage)**

Output will be Numeric

Second Moment:

Variance:

**# Get Variance**

** var(CarsMileage)**

Output will be 2.24736

Standard Deviation:

**# Get Standard Deviation**

** sd(CarsMileage)**

Output will be 1.499123

Third moment and Fourth moment:

I will cover how to derive third and fourth moment in my future blogs.

I hope you must have got an essence and an importance of four moments in the statistical analysis. This is the foundation for rest of the statistical world. 60% – 70% of the time of statistician goes in executing EDA to make sure data captured is complete and unbiased which will help one take right decision and right outcome. Getting right data be it a population or sample, it is very important for one to make a right choice. Now that basics for EDA is clear I will help one with visualization to explore advance statistical problems.

Thank you for sparing time and going through this blog I hope it helped you built sound foundation of statistics using R. Kindly share your valuable and kind opinion. Please do not forget to suggest what you would like to understand and hear from me in my future blogs.

Thank you…

Outstanding Outliers:: “AG”.

Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data

Kurtosis does not measure peakedness – that is an outdated, and actually incorrect definition. Kurtosis measures outlier or outlier potential only, which is unrelated to the shape of the peak. See the current Wikipedia entry for correct information.

LikeLike

Thank you Peter for your inputs, I will get this refreshed after going through Wikipedia however What is wrote is from the old school of thought where definition of kurtosis was the measure of peakness. Appreciate your inputs and thank you for your inputs.

LikeLike

Right, we used to think the sun circled the earth, too! But there is no sense in repeating “old school” thoughts when they are patently wrong! It just impedes the progress of science. Here is a direct link to a clear explanation of why kurtosis is definitely NOT peakedness: https://en.wikipedia.org/wiki/Talk:Kurtosis#Why_kurtosis_should_not_be_interpreted_as_.22peakedness.22

LikeLike

Thank you so much again…I am in process of exploring more and will refine article to avoid representing old school thought. Appreciate you help and correcting the old school of thoughts. Keep this community honest and correct. appreciate it.

LikeLike