🔥Data Analytics Full Course 2024 Data Analytics For Beginners Data Analytics Course Simplilearn

hey everyone welcome to Simply learn
today we have data analytics course for
our aspiring data and business analysts
today we will learn about the complete
data analytics fundamentals tools roles
and responsibilities job profile
projects and interview questions that
said if these are the type of videos
you'd like to watch then hit that like
And subscribe buttons and the Bell icon
to get notified just for your quick info
if you want to upscale yourself master
data science and data analytics skills
and landar your dream job or growing
your career then you must explore simply
learns CAD of various data science and
analytics programs simply learn offers
various certifications and postgraduate
programs in collaboration with some of
the world's leading universities like
Caltech Purdue IIT conour and many more
through our courses you will gain
knowledge and work ready expertise in
skills like Advanced python advanced
Excel machine learning data
visualization generative Ai and over a
dozen others that's not all you also get
the opportunity to work on multiple
projects and learn from industry experts
working on Top Tire data and product
companies and academics from top
universities after completing these
courses thousands of Learners have
transitioned into a data science or data
analytics role as a fresher or moved
onto a higher paying job and profile if
you are passionate about making your
career in this field then make sure to
check out the link in the pin comment
and description box below to find a data
science and data analytics program that
fits your experience and areas of
Interest so what is data
analytics companies around the world are
generating vast volumes of data every
hour this data could be in the form of
log files web server and transactional
data as well as various customer related
data also data is being generated at a
rapid rate from social media websites
and applications
such as Facebook Instagram Twitter and
WhatsApp companies want to use this data
to derive value out of it and make
business
decisions that's where data analytics
comes into
use data analytics is the process of
exploring and analyzing large data sets
to find hidden patterns unseen Trends
discover correlations and valuable
insights to make business
predictions data analytics improves the
speed and efficiency of your
business a few years ago a business
would have gathered information manually
performed statistical and complex
analytics and Unearthed information that
could be used for future
decisions but today that business can
identify insites on the Fly for
immediate
decisions most organizations have big
data and many understand the need to
harness that data and extract value out
of it so they use a lot of modern tools
and Technologies to perform dat data
analytics some of the tools I will
discuss in detail later in this
tutorial now that we have looked at at
what data analytics really is let us
understand the ways in which you can use
data
analytics first is improved decision
making data analytics eliminates a lot
of guesswork and manual tasks from
choosing the right content planning
marketing campaigns and developing
products organizations can use the
insights they gain from data analytics
to make informed decisions leading to
better outcomes and customer
satisfaction it gives you a 360 degree
view of your customers which helps you
understand the behavior completely
enabling you to better meet their
needs second is better customer service
data analytics provides you with more
accurate insights of your customers
allowing you to tailor customer service
to their needs provide more
personalization and build stronger
relationships with them your data can
reveal information about your customers
communication preferences their
interests their concerns and more it
helps you give better recommendations
for products and
services next is efficient operations
data analytics can help you streamline
your processes save money and boost
production when you have an improved
understanding of what your audience
wants you waste less time in creating
ads and content that don't match your
audience interests this helps you
optimize your campaigns create better
content strategies and hence improve
results and finally we have effective
marketing when you understand your
audience better you can Market to them
more effectively data analytics also
gives you useful insights into how your
campaigns are performing so that you can
fine-tune them for optimal
outcomes you also find out the probable
customers who are the most likely to
interact with a Campa campign and
convert into
leads now let's discuss the various
steps involved in the data analytics
process as you can see on the screen
there are five process steps now let me
make you understand each of this one by
one so the first step is to understand
the
problem before starting with the
analysis you need to understand the
business problem and Define your goals
asking questions at the outlet is vital
because this would address issues such
as how can we reduce production costs
without sacrificing quality what are
some of the ways to increase sales
opportunities with our current
resources do customers view your brand
in a favorable way answers to these
questions will help you build a clear
road map with lucrative Solutions also
try to find out the key performance
indicators and consider the Matrix to
track along the way
the second step in the process is data
collection after you have finalized your
goals it's time to start looking for
your data data collection is the process
of gathering information on targeted
variables identified as data
requirements the emphasis is on ensuring
accurate and right data is collected
data collection starts with primary
sources which are also known as internal
sources this is typically structured
data gathered from CRM software Erp
systems marketing automation tools and
others these sources contain information
about customers finances gaps in sales
Etc under external sources you have both
structured and unstructured data so if
you're looking to perform a sentiment
analysis towards your brand you would
gather data from various review websites
or social media
apps the next step is to clean the
data the data which is is collected from
various sources is highly likely to
contain incomplete duplicate and missing
values so you need to clean these
unwanted redundant data to make it ready
for analysis so to generate accurate
results analytics professionals must
identify duplicate and animaly data and
other inconsistencies that could skew
the analysis according to a report 60%
of data scientists say most of the time
is spent cleaning the data while 57% of
data scientists say it's their least
enjoyable
task now the fourth step in the process
is data exploration and
Analysis once data is cleaned and ready
you can go ahead and explore the data
using data visualization and business
intelligence tools you can also use
various Data Mining and predictive
modeling techniques to analyze the data
and build
models you can use different supervised
and unsupervised algorithms such as
linear regression logistic regression
decision Fusion tree KNN K means
clustering and lots more to build
prediction models for making business
decisions and the final step is to
interpret the
results this part is important because
it's how a business will gain actual
value from the previous four
steps interpreting the results will help
you find unseen Trends and patterns in
the data and gain insights you can have
a validation check if the results are
answering your questions
these results can be shown to your
clients and stakeholders for better
understanding and business
collaboration now that we have looked at
the various steps involved in data
analytics let's now see the different
tools that can be used to perform the a
steps so as you can see we have seven
tools including a few programming
languages that will help you perform
analytics better now let's discuss them
one by
one first we have python
python is an objectoriented open- Source
programming language that supports a
range of libraries for data manipulation
data visualization and data modeling
python programmers have developed tons
of free and open source libraries that
you can use you can find many of them
via the python package index which is
pypi the repository of python
software python provides the default
package installer called pip or pip
Python has libraries such such as numpy
for numerical computation of data pandas
to manipulate data on numerical tables
and time series then you have scipi for
Technical and scientific computations it
also provides psyit learn which is a
machine learning library for creating
classification regression and clustering
algorithms and finally it also has P
torch and tensorflow for deep
learning up next we have
r r is an open-source programming
language majorly used for numerical and
statistical analysis it provides a range
of libraries for data analysis and
visualization some of these libraries
are ggplot tidos plotly deer and
carrot then we have tblo tblo is a
popular data visualization and analytics
tools that helps you create a range of
visualizations to interactively present
the data build reports and dashboards to
Showcase insights and Trends it can
connect with multiple data sources and
give hidden business insights and
patterns then we have a competitor of
tblo which is
powerbi powerbi is a business
intelligence tool developed by Microsoft
that has an easy drag and draw
functionality and supports multiple data
sources with features that make data
visually appealing powerbi supports
features that help you ask questions to
your data and get immediate insights you
can also forecast your data for
predicting future
trends so the next tool is Click
view click view provides interactive
analytics with inmemory storage
technology to analyze vast volumes of
data and use data discoveries to support
decision
making it provides social media
Discovery and interactive guided
analytics it can manipulate huge data
sets instantly with
accuracy up next we have Apache
spark Apache spark is an open source
data analytic engine to process data in
real time and Carry Out complex
analytics using SQL queries and machine
learning
algorithms it supports spark streaming
for realtime analytics and Spark SQL for
writing SQL
queries it also has spark MLB which is a
library that has a repository of machine
learning algorithms and then it has
graphics for graphical
computation and finally we have
SAS
SAS is a statistical analysis software
that can help you perform analytics
visualize your data write SQL queries
perform statistical analysis and build
machine learning models to make future
predictions SAS empowers our customers
to move the world forward by
transforming data into
intelligence SAS is investing a lot to
drive software Innovation for analytics
Gartner has positioned SAS as a magic
quadrant leader for data science and
machine learning
learning moving on to the applications
of data
analytics data analytics is being used
in almost every sector of business these
days let's discuss a few of
them first we have
retail customers expect retailers to
understand exactly what they need and
when they need it data analytics helps
retailers meet those
demands retailers not not only have an
in-depth understanding of their
customers but they can also predict
Trends recommend new products and boost
profitability retailers create
assortments based on customer
preferences invoke the most relevant
engagement strategy for each customer
optimize supply chain and Retail
operations at every step of the customer
Journey the second application is on
Healthcare Healthcare Industries analyze
patient data to provide life-saving
diagnosis and treatment
options they also deal with healthcare
plans and insurance information to drive
key insights using analytics they can
discover new drugs and come up with new
drug development
methods Advanced analytics allows
Healthcare companies to improve patient
outcomes and
experience cancer cells and diabetic
retinopathy can be discovered using
Medical
Imaging at number three we have
manufacturing for man manufacturers
solving problem is nothing new they
fight with difficult problems and
situations on a daily basis from complex
Supply chains to motion applications to
labor constraints and Equipment
breakdowns they deal with such problems
on a regular basis using data analytics
manufacturing sectors can discover new
cost saving and revenue
opportunities the fourth application is
related to the banking
sector Banking and financial
institutions collect vast volumes of
structured and unstructured data to
derive analytical insights and make
sound financial decisions using
analytics they can find out probable
Loan defaulters customer turnout rate
and detect fraudulent transactions
immediately the final application is
based on
Logistics logistics companies use data
analytics to develop new business models
that can ease their business and improve
productivity
they can optimize routes to ensure
delivery reaches on time in a cost
efficient manner they also focus on
improving order processing capabilities
as well as Performance
Management with that now let's look at
the companies using data analytics on a
daily
basis so we have the e-commerce giant
Amazon then we have Accenture followed
by the American healthcare service
organization
then we have the American supplier of
Health Information Technology Solutions
Services devices and Hardware serner
followed by Target and Antivirus company
macafe NeXT we have Rapido which is an
Indian bike rental company based in
Bangalore after that we have Flipkart
and the world's largest retail company
Walmart a study from new Vantage
Partners suggest 97.2% of companies are
now investing in data and its analysis
nowadays every company needs a data
expert do you also want to become a part
of this Market if yes how to become a
part of it what are the essential skills
one should
have this video will answer all these
questions but before watching this video
please subscribe to Simply learns
YouTube channel and press the Bell icon
to never miss any updates first of all
we are going to discuss who is a data
analyst working of a data analyst skills
required to become from a data analyst
tools required and companies's hiring
and finally we are going to discuss
about salary of a data analyst let me
tell you how simply learn can help you
in your journey to become a data analyst
check out the course on data analytics
in collaboration with IBM with realtime
project and business case studies you
will learn tools like scipi pandas and
programming languages like python or R2
enroll Now link is in the description
box below question for you which one of
the foll following is not a python
library for
visualization my plot lip scipi pandas
jupyter notebook please leave the answer
in the comment section below moving on
who is a data analyst a data analyst
collects analyzes and interprets data a
data analyst will convert raw data into
useful information data analysts are in
high demand because every industry uses
data analysis work of a data analyst as
a data analyst you will work closely
with with the raw data and generate
valuable insights to help companies
decide their future goal if you like
thinking out of the box you are the
perfect fit for this domain data analyst
help maximize output when it comes to
generating Revenue working closely with
both business and data nevertheless this
field boost handsome salaries for all
levels of expertise can you become a
data analyst without prior experience
yes anyone can become a data analyst if
they enjoy solving real world problems
have a strong background in statistics
and have a creative mind if you feel you
don't have it you can definitely develop
it so let us know the skills in detail
what are the basic skill sets required
for a data analyst data analyst must
know basic mathematics and statistics
programming skills machine learning and
also data visualization tools so let us
know what are the basics that you need
to learn as a data
analyst mathematics it is always is
better to know basic mathematics like
linear algebra and probability
fundamentals linear algebra is used in
data pre-processing and transformation
which is the critical process of every
data
analyst statistics a branch of
mathematics that deals with collection
analysis presentation and
implementation probability we know that
probability is the study of How likely
something will happen which is essential
for concluding both probability and
statistics are the backbone of data
analysis it is feasible to become a data
analyst with only a basic understanding
of these three areas of mathematics but
in order to remain relevant and grow as
a data analyst one's mathematical
knowledge should not be restricted
compulsorily use some of the tools as a
data analystic what are
that first is Microsoft Excel it is the
most wellknown spreadsheet software in
the world it also has computation and
graphick features that are excellent for
data analyst no matter your area of
expertise or additional software you
might want Excel is a standard in the
industry its useful built-in features
include form design tools and pivot
table it also generate a wide range of
additional features that help simplify
data
manipulation as a programming language
every data analyst should know python it
is easy to learn and has a simple syntax
python is quite adaptable and includes a
vast variety of resource libraries that
are appropriate for a wide range of
diverse data analytics activities these
libraries help in numerical and data
computation the pandas and numai
libraries for instance are excellent for
supporting standard data processing and
streamlining highly computational
operations you can also choose between
Python or r r is a well-known op Source
programming language much like python
data visualization tool as we previously
mentioned dat visualization tool is also
necessary to become a data analyst
powerbi is a userfriendly interface
makes building interactive visual
reports and dashboard simple its most
vital selling point is its superb data
integration it works flawlessly with
Cloud sources like Google and Facebook
analytics as well as text files SQL
servers and
Excel is one of the best commercial data
analysis tool available it handles huge
amounts of data better than many other
bi tools and is effortless it has a
visual drag and drop interface however
because it has no scripting layer there
is a limit to what Tableau can do for
example it could be better for
pre-processing data or building more
complex
calculations you might have heard about
MySQL a lot of time it is a standard
language for interacting with databases
and it is very helpful when working with
structure data SQL creates userfriendly
dashboards that may present in various
data ways in since it is so simple to
send complex commands to databases and
change data in seconds it has commands
like add edit delete data in Edition SQL
is an excellent tool for creating data
warehouses because of its Simplicity
Clarity and
interactivity overall I would suggest
that to become a data analyst you should
work on programming language PES like
python or R plus MySQL to work on
databases adding to that Excel plus
visualization tools like Tableau or
powerbi you now know what are the skills
are and how it is used what are you up
to in an organization as a data
analyst to create and evaluate the
report using automated tools like tblo
or
powerbi to troubleshoot the reporting
database environment and reports data
analyst you will use statistical method
to analyze data sets and spot any
valuable trends that may develop over
time evaluate company's functional and
non-functional requirement data analyst
assess data warehousing in inspecting
and Reporting needs these are all the
responsibility of a data analyst in an
organization coming to companies hiring
a data analyst IBM Accenture cap mini
TCS Facebook Amazon Amazon flip cart
meta these are the top companies hiring
a data analyst but data suggest that
every small and medium-sized company
needs a data analyst therefore demand of
a data analyst is in every company so
there is no need to worry job and salary
of a data analyst this is the final part
the salary of a data analyst is high all
over the world when it comes to the USA
the average salary for a data analyst as
a beginner is going as high as
$7,000 plus dollar per anom for
experienced professional it is going as
high as
$120,000 perom in India for a fresher it
is going as high as 8 lakh perom and for
experienced professionals it is 20 lakh
plus perom such is the demand for data
analyst now that we have covered every
important skill it's time for you to
start working on it so Sky a limit on
what you use it for let's take a look at
types of data analytics
and this can be broken up in so many
ways uh but we're going to start with
looking at the most basic questions that
you're going to be asking in data
analytics and the first one is you want
descriptive analytics what has happened
hindsight uh how many cells per call
ratio coming out of the call center if
we have 500 tourists in a forest and you
have a certain temperature how many
fires were started how many times did
the police have to show up to certain
houses um all that's descriptive the
next one is predictive Predictive
Analytics is what will happen next we
want to predict uh this is great if you
want have a ice cream store and you want
to predict how many people to work at
the ice cream store in a certain day
based on the temperature coming up in
the time of the year and then one of the
biggest growing and most important parts
of the industry is now prescriptive
analytics and you can think of that as
combining the first two we have
descriptive and we have
predictive then you get
prescriptive analytics how can we make
it happen foresight what can we change
to make this work
better in all the industries we looked
at before we can start asking questions
uh especially in City development
there's a good one if we want to have
our city generate more income and we
want that income to be commercial based
uh what kind of commercial buildings do
we need to build in that area that are
going to bring people over do we need
huge warehouse sales Costco sales
buildings or do we need little momod
joints that are going to bring in uh
people from the country to come shop
there or do you want an industrial setup
what do you need to bring that IND
industry in there is there car industry
available in that area uh if it's not a
car industry what other Industries are
in that area all those things are
prescriptive we're guessing we're
guessing what can we do to fix it what
can we do to fix crime in area with
education what kind of education are we
going to use to help people understand
what's going on so that we lower the
rate of crime and we help our
communities grow better that's all
prescriptive it's all guessing we want
foresight into how can we make it happen
how can we make this
better and we really can't not go into
enough detail on these three because a
lot of people stumble on this when they
come in and are doing analytics whether
you're the manager shareholder or the
data scientist coming in you really need
to understand the descriptive Analytics
where you're studying the total units of
furniture sold and the profit that was
made in the past uh here we go into
Predictive Analytics predicting the
total units that would sell and the
profit we can expect in the future gear
up for how many employees we need how
much money we're going to make and
prescriptive analytics finding ways to
improve the sales and the profit so we
can uh sell maybe a different kind of
furniture uh we're going to guess at
what the area is looking for and how
that marketing is going to change hello
everyone we welcome you all to this
video by simply learn in today's session
we will learn a really interesting topic
that is the top 10 skills to become a
data analyst in
2022 in today's Digital World data is
being generated by companies and
individuals every second so the role of
a data analyst holds supreme
importance so if you're looking for a
career in data analytics this video will
help you learn what a data analyst does
and the various skills you need to
process to become a data analyst in 2022
before we get started make sure you
subscribe to the simply learn Channel
and hit the Bell icon to never miss an
update from
us let's look at the agenda for this
video first we will understand who a
data analyst is then we will understand
the top 10 data analyst skills for 2022
moving on we will look at the salary of
a data analyst and finally we will look
at the company's hiring data analysts so
now let's understand who is a data
analyst a data analyst is a professional
who collects business data from various
sources interprets it and uses various
statistical tools and techniques to
extract insights and useful information
from it they acquire data from primary
or secondary data sources and maintain
databases they also recognize and
understand the organization's goal and
collaborate with different team members
such as programmers business analysts
and data scientists to build an
effective solution to a business
problem now with this basic
understanding of who a data analyst is
let's learn the top 10 data analyst
skills for
2022 at number one we have structured
query language or SQL SQL is a top skill
that every data analyst should have data
analysts use SQL commands and functions
to store process analyze and manipulate
structured data using relational and
nosql
databases they also build data models
and write complex SQL queries and
scripts to gather and extract
information from several databases and
data
warehouses some of the popular databases
a data analyst should be familiar with
are Microsoft SQL Server myql postr SQL
and IBM db2 the second important skill
for a data analyst is Microsoft
Excel Microsoft Excel is one of the most
popular and oldest spreadsheet
applications for creating reports
performing calculations and analyzing
data data analysts need to know how to
handle tabular data in Excel so they
should be aware of features like sorting
filtering conditional formatting pivot
tables Water Analysis and functions such
as sumifs and
countifs the third crucial data anal
skill for 2022 is data cleaning and
wrangling usually the data collected by
analist from various heterogeneous
sources is often messy and contains a
lot of missing
values so it is always crucial to clean
the data and remove noise missing or
erroneous
elements it is also important to format
data using tools and methods before
using it for analysis they're
responsible for data mining as well the
data mined from various sources are
organized in order to obtain new
information from it some of the tools
you need to know for data cleaning and
wrangling are Excel power query and open
refin
the fourth skill on our list is
mathematics and
statistics data andal lists often work
on data for higher Dimensions that are
greater than three in order to interpret
such data they need to be good at linear
algebra and calculus they also build
predictive models and statistical models
such as linear regression logistic
regression knif base and K means
clustering in order to understand the
working of these algorithms they must
have knowledge about statistics and
probability coming to the fifth
important skill for a data analyst in
2022 we have
programming data analysts need to master
at least one programming language
preferably python or R in order to work
with complex business problems analysts
need to write scripts and userdefined
functions to automate tedious tasks
Python and art language provide a
collection of different libraries and
packages such as numai pandas deer
matplot lib ggplot which data analysts
can use to discover Trends and patterns
from complex data
sets after this we have data
visualization as our sixth skill another
data analyst job role is to visualize
large volumes of data and prepare
summary reports and dashboards for the
leadership team and clients so that they
can make timely business
decisions to do this data analyst use
various data visualization tools such as
powerbi tblue and click View using these
tools data analysts can integrate
various data sets apply joint conditions
sort and filter data as well create
different visualizations using charts
and graphs the seventh skill for a data
analyst is industry
knowledge data analysts should have good
knowledge and understanding of the
industry or domain they are working in
for example if you're working in a
healthcare domain you need to know how
Healthcare analytics can be applied to
improve patient care you should have not
knowledge about the challenges faced in
healthcare and how you can leverage data
and analytics to solve the issues only
if you have strong industry knowledge
can you try to improve the business the
eighth skill that is important for a
data analyst in 2022 is problem
solving a business deals with several
problems on a daily basis data analyst
should be ready to face those challenges
data analysts are expected to use their
problem solving skills work with the
team troubleshoot what went wrong and
provide an effective solution via data
analysis a data analy with good problem
solving skills can help a business
identify current and potential issues
and determine a viable solution based on
the data it
collects the ninth skill on our list is
analytical thinking data analysts need
analytical thinking ability to break
down a complex problem into simple
components and resolve these components
one by one it is a must-have skill for
data and lists analytical thinking
includes deciding the parameters that
need to be considered for defining data
sets analyzing them from different
perspectives and determining variable
dependencies coming to the 10th skill
among the top 10 skills for a data
analyst in 2022 we have
communication data analysts don't just
interact with computers and programs
they also interact with team members
stakeholders and data
suppliers so good communication skills
are essential data analysts also present
their findings in front of an audience
who might not be familiar with the
analytical methods and processes so they
need to clearly translate their findings
and insights into non-technical terms so
those were the top 10 skills a data
analyst needs to possess in 2022 do you
think we missed out on any skills then
please put your answers in the comment
section below now let's look at the
salary of a data
analyst according to glaso the average
annual salary for a data analyst in the
United States is
$69,500 rupees per ROM finally let's
look at the top companies that are
hiring data analysts in
2022 here we have the consultancy and
big four Giant deoe and the
pharmaceutical company serner
Corporation then we have the the tech
giant IBM retail company Walmart and the
e-commerce leader
Amazon to achieve the goals of data
analysis we use a number of data
analysis tools companies rely on these
tools to gather and transform their data
into meaningful insights so which tool
should you choose to analyze your data
which tool should you learn if you want
to make a career in this field we will
answer that in this session after
extensive research we have come up with
these top 10 data analysis tools here we
will look at the features of each of
these tools and the companies using them
so let's start
off at number 10 we have Microsoft Excel
all of us would have used Microsoft
Excel at some point right it is easy to
use and one of the best tools for data
analysis developed by Microsoft Excel is
basically a spreadsheet program using
Excel you can create grids of numbers
text and formula it is one of the widely
used tools be it in a small or large
setup the interface of Microsoft Excel
looks like
this let's now move on to the features
of
excel firstly Excel works
with the windows version of excel
supports programming through Microsoft's
Visual Basic for applications
VBA programming with VBA allows
spreadsheet manipulation that is
difficult with standard spreadsheet
techniques in addition to this the user
can automate tasks such as formatting or
data organization in
VBA one of the biggest benefits of excel
is its ability to organize large amounts
of data into orderly logical
spreadsheets and charts by doing so it's
a lot easier to analyze data especially
while creating graphs and other visual
data
representations the visualization can be
generated from specified group of
cells those were you of the features of
Microsoft Excel let's now have a look at
the companies using it most of the
organizations today use Excel few of
them that use it for analysis are the UK
based company Ernest and Young then we
have Urban Pro Whi Pro and
Amazon moving on to our next data
analysis tool at number nine we have
rapid
Miner a data science software platform
rapid minor provides an integrated
environment for data preparation
analysis machine learning and deep
learning it is used in almost every
business and Commercial sector rapid
Miner also supports all the steps of the
machine learning
process seen on your screens is the
interface of Rapid
Miner moving on to the features of Rapid
minor firstly it offers the ability to
drag and drop it is very convenient to
just drag drop some columns as you are
exploring a data set and working on some
analysis rapid Miner allows the usage of
any data and it also gives an
opportunity to create models which are
used as a basis for decision- making and
formulation of
strategies it has data exploration
features such as graphs descriptive
statistics and visualization which
allows users to get valuable
insights it also has more than 1,500
operators for every data transformation
and Analysis
task let's now have a look at the
companies using rapid minor we have the
Caribbean Airline leward Islands Air
transport next we have the United Health
Group the American online payment
company PayPal and the Australian
Telecom company mobilecon so that was
all about rapid Miner now let's see
which tool we have at number
eight we have talent at number eight
Talent is an open source software
platform which offers data integration
and management it specializes in Big
Data integration Talent is available
both in open-source and premium versions
it is one of the best tools for cloud
computing and Big Data
integration the interface of talent is
as seen on your
screens moving on to the features of
talent firstly automation is one of the
great Boon's Talent offers it even
maintains the tasks for the users this
helps with quick deployment and
development it also offers open- Source
tools Talon lets you download these
tools for free the development costs
reduce significantly as the processes
gradually speed
up Talent provides a unified platform it
allows you to integrate with many
databases SAS and other Technologies
with the help of the data integration
platform you can build flat files
relational databases and Cloud apps 10
times S
faster those were the features of talent
the companies using Talent are Air
France L'Oreal Cap Gemini and the
American multinational pizza restaurant
chain
dominoes next on the list at seven we
have Nim constant information minor on
Nim is a free and open-source data
analytics reporting and integration
platform it can integrate various
components for machine machine learning
and data mining through its modular data
pipelining concept Nim has been used in
pharmaceutical research and other areas
like CRM customer data analysis business
intelligence text Mining and financial
data
analysis here is how the interface of n
application looks
like now coming to the nine
features nine provides an interactive
graphical user interface to create
visual workflows using the drag and drop
feature use of jdbc allows assembly of
nodes blending different data sources
including pre-processing such as ETL
that is extraction transformation
loading for modeling data analysis and
visualization with minimal
programming it supports multi-threaded
inmemory data processing Nim allows
users to visually create data flows
selectively execute some or all analysis
steps and later inspect the results
models and interactive views
Nim server automates workflow execution
and supports team- based collaboration
nime integrates various other open-
source projects such as machine learning
algorithms from Becca hed2 Caris Park
and our project nine allows analysis of
300 million custom addresses 20 million
cell images and 10 million molecular
structures some of the companies hiring
for Nim are United Health Group asml
fractal analytics atos and and LEGO
Group let's now move on to the next tool
we have SAS at number
six SAS facilitates analysis reporting
and predictive modeling with the help of
powerful visualizations and dashboards
in SAS data is extracted and categorized
which helps in identifying and analyzing
data patterns as you can see on your
screens this is how the interface looks
like moving on to the features of
s using SAS better analysis of data is
achieved by using automatic code
generation and SAS SQL SAS allows you to
access through Microsoft Office by
letting you create reports using it and
by Distributing them through
it SAS helps with an easy understanding
of complex data and allows you to create
interactive dashboards and
reports let's now have a look at the
companies using SAS we have companies
like genpact iqvia Accenture and IBM to
name a
few that was all about
SAS so for all those who joined in late
let me just quickly repeat our list at
number 10 we have Microsoft Excel then
at number nine we have rapid minor at
number eight we have talent at number
seven we have n and at number six we
have SAS so far do you all agree with
this list let us know in the comment
section below let's now move on to the
next five Tools in our
list so at number five we have both R
and python yes we have two of them in
the fifth
position R is a programming language
which is used for analysis as well it
has traditionally been used in academics
and research python is a highlevel
programming language which has a python
data analysis Library it is used for
everything starting from importing data
from Excel spreadsheets to processing
them for
analysis this is the interface of
R next up is the interface of the Python
Jupiter
notebook let's now move on to the
features of both R and
python when it comes to the availability
of R and python it is very easy both R
and python are completely free hence it
can be used without any license
R used to compute everything in memory
and hence the computations were limited
but now it has changed both R and python
have options for parallel computations
and good data handling
capabilities as mentioned earlier as
both R and python are open in nature all
the latest features are available
without any
delay moving on to the companies using R
we have Uber Google Facebook to name a
few python is used by many companies
again to name a few we have Amazon
Google and the American photo and video
sharing social networking service
Instagram that was all about RN
python at number four we have Apache
spark Apache spark is an open- Source
engine developed specifically for
handling large scale data processing and
Analytics
spark offers the ability to access data
in a variety of sources including Hadoop
distributed file system htfs openstack
Swift Amazon S3 and
Cassandra it allows you to store and
process data in real time across various
clusters of computers using simple
programming
constructs Apache spark is designed to
accelerate analytics on Hadoop while
providing a complete Suite of
complimentary tools that include a fully
featured machine learning Library a
graph processing engine and stream
processing so this is how the interface
of a parache spark looks
like now let's look at the important
features of Apache
spark spark stores data in the ram hence
it can access the data quickly and
accelerate the speed of analytics spark
helps to run an application in a Hadoop
cluster up to 100 times faster in memory
and 10 times faster when running on
disk it supports multiple languages and
allows the developers to write
applications in Java Scala r or python
Spar comes up with 80 high level
operators for interactive querying Spar
code for batch processing join stream
against historical data or run ad hoc
queries on stream
State analytics can be performed better
as spark has a rich set of SQL queries
machine learning algorithms complex
analytics Etc Apache spark provides fall
tolerance through spark rdd spark
resilient distributed data sets are
designed to handle the failure of any
worker node in the cluster thus it
ensures that the loss of data reduces to
zero conviva Netflix iqa Loy Martin and
eBay are some of the companies that use
a party spark on a daily
basis at number three we have another
important growing data analysis tool
that is click
view click view software is a product of
Click for business intelligence and data
visualization click view is a business
Discovery platform that provides
self-service bi for all business users
and
organizations with click viw you can
analyze data and use your data
discoveries to support decision making
click viw is a leading business
intelligence and analytics platform in
Gartner magic
quadrant
on the screen you can see how the
interface of Click view looks
like now talking about its features
click view provides interactive guided
analytics within memory storage
technology during the process of data
Discovery and interpretation of
collected data The Click viw software
helps the user by suggesting possible
interpretations click viw uses a new
patent in memory architecture for data
storage all the data from the different
sources is loaded in the ram of the
system and it is ready to be retrieved
from there it has the capability of
efficient social and mobile data
Discovery social data Discovery offers
to share individual Data Insights within
groups or out of it a user can add
annotations as an addition to someone
else's insights on a particular data
report click view supports mobile data
Discovery within an HTML F enabled touch
feature which lets the user search the
data and conduct data data Discovery
interactively and explore other
server-based
applications click view performs olap
and ETL features to perform analytical
operations extract data from multiple
sources transform it for usage and load
it to a data
warehouse the companies that can help
you start your career in Click view are
Mercedes-Benz Cap Gemini City Bank
cognizant and Accenture to name a
few at number two we have powerbi
powerbi is a business analytic solution
that lets you visualize your data and
share insights across your organization
or embed them in your app or
website it can connect to hundreds of
data sources and bring your data to life
with live dashboards and
reports powerbi is the collective name
for a combination of cloud-based apps
and services that help organizations
collate manage and analyze data from a
variety of sources through a
userfriendly
interface powerbi is built on the
foundation of Microsoft Excel and has
several components such as Windows
desktop application called powerbi
desktop and online software is a service
called powerbi service mobile powerbi
apps available on Windows phones and
tablets as well as for IOS and Android
devices here is how the powerbi inter
face looks like as you can see there is
a visually interactive sales report with
different charts and
graphs moving on to the features of
powerbi it has an easy drag and drop
functionality with features that make
data visually appealing you can create
reports without having the knowledge of
any programming language powerbi helps
users see not only what's happened in
the past and what's happening in the
present but also what might happen in in
the
future it offers a wide range of
detailed and attractive visualizations
to create reports and dashboards you can
select several charts and graphs from
the visualization pan powerbi has
machine learning capabilities with which
it can spot patterns in data and use
those patterns to make informed
predictions and run what of
scenarios powerbi supports multiple data
sources such as Excel Tech CSV Oracle
SQL server PDF and XML files the
platform integrates with other popular
business management tools like
SharePoint Office 365 and dynamic 365 as
well as other non-microsoft products
like spark Hadoop Google analytics sap
Salesforce and
MailChimp some of the companies using
powerbi are Adobe AXA carlsburg Capt Min
and
Nestle moving on to the next tool so any
guesses as to what we have at number one
you can comment in the chat section
below finally on the top of the pyramid
we have
tblo Gartner's magic quadrant of 2020
classifi tblo as a leader in business
intelligence and data analysis tblo
interactive data visualization software
company was founded in Jan 2003 in
Mountain View
California tblo is a data visualization
software that is used for data science
and business intelligence it can create
a wide range of different visualization
to interactively present the data and
showcase
insights the important products of tblo
are tblo desktop tblo public tblo server
tblo online and tblo
reader this is how the interface of tblo
desktop looks
like now coming to the features of
tblo data analysis is very fast with
tblo and the visualizations created are
in the form of dashboards and worksheets
tblo delivers interactive dashboards
that support insights on the
fly it can translate queries to
visualizations and import all ranges and
sizes of data writing simple SQL queries
can help join multiple data sets and
then build reports out of it you can
create transparent filters parameters
and
highlighters tableau allows you to ask
questions spot Trends and identify
opportunities with the help of tblo
online you can connect with Cloud
databases Amazon redshift and Google big
query the companies using tblo are Deo
Adobe Cisco LinkedIn and the American
e-commerce giant Amazon to name a few
and there you go those are the top 10
data analysis Tools R is a statistical
programming language and environment
that integrates statistical computing
and Graphics R is powerful and stable
software python python can also be
called as a general purpose programming
language for data analysis and
scientific Computing python can be
considered as the best player in machine
learning python is an expressive
language with many built-in function
both are open-source software and
platform
independent and they are platform
neutral and also compatible with with
all major operating systems including
Unix Windows and Mac next we will be
covering different
parameters we will be covering learning
preferability mathematical
fundamentals speed of both
languages visualization and
Graphics data handling
capacity
demand community and customer support
employment possibility in both the
languages let us cover it one by one
first one is learning preferability or
ease of
learning python is Renown for its ease
of use Python's notebooks offer
excellent tools for sharing and
documentation despite the fact that
there are currently no guis for them
programmers find are as difficult
language as a beginner this implies that
the programmers must devote a
significant amount of time to learn and
comprehending our
coding coming to mathematical
fundamentals
required coming to python understanding
descriptive analysis is very important
in layman's terms descriptive statistics
often refers to the process of
explaining using certain representative
techniques such as charts tables Excel
files
Etc python statistics is a built-in
library for descriptive
statistics if your data sets are not too
big or if you can't rely on importing
other libraries you can use
python on the other hand R requires
basic statistics from basic statist
statistics what I mean is mean mode and
median are the terms used most
frequently in basic statistics it is
referred to as measures of central
tendency
probability statistics plays an
important role in handling various types
of probability distribution it includes
binomial and normal
distribution next parameter is
speed python is an interpreted language
with Dynamic typing python always
executes slowly because the code is
executed line by
line compart to Matlab and python our is
our language is significantly
slower our packages are substantially
slower than those for other
languages now that we have covered speed
coming to data visualization and data
collection in
Python When selecting data analysis
tools bit visualization are crucial and
python has some incredible visualization
tool in Python to large and varied
skater plots using regression lines we
can use gz plot 2 and GZ plot
tools compared to draw values visualized
data is easier to comprehend therefore R
has many packages that offers
sophisticated graphic
features in R we can
use in R we can use use tools like M
plot lip saon
Etc data handling capability in both
Python and
R the new releases in Python have
resolved the issue with the python
packages for data analysis R is useful
for analysis because of the abundance of
packages accessibility of the test and
benefit of employing formulas however
simple data analysis can also be done
using it the need to install many
packages crucial part of parameter that
is tools and libraries in Python and
R as a python developer one needs to be
well wor in the best libraries because
python has a lot of libraries that have
many different
uses libraries like tensorflow pyit Lear
numai plays an important role in solving
many python related
problems libraries perform a wide
variety of task in are that are very
beneficial for data science operations
example for that is deer bioconductor
Etc community and customer support
support index offered by Python and
R compared to R python has a larger comp
Community for assistance we can contact
www.python.org
for any queries regarding Python and
help you can support uh you I repeat you
can visit support. real
python. for any help and queries R
offers you with our studio Community R
provides assistance through its official
website for queries and community
related issues we can contact
www.r
pro.or next is job opportunities in
Python and
r a recent survey from indie.com
predicts that at least 55,000 python
jobs in the USA with exponential pay
rates are available big tech companies
like Google Amazon Twitter Facebook
require as python developer to handle
massive amount of data position provided
for a python developer is software
engineer data analyst data scientist and
many
more career in R is an excellent job
opportunity for you as a beginner big
tech companies like Google Twitter
Facebook are using
R position provided by companies as a r
developer is data scientist data analyst
data visualization analyst Etc moving on
let us wrap up an important topic which
language to be used between R and python
there is no right or wrong way to study
both python or R both are in demand
skills that will enable you to complete
almost any data analytics work you come
across It ultimately depends on your
background interest and career
objectives that which one is better for
you but compared to R python is easy to
learn let's compare its strength and
weaknesses it is used to handle large
amount of data python performs
non-statistical functions and it is best
suitable for programming however python
is better when it comes to
coding whereas R is used in data
visualization
Graphics R is a widespread language in
the statistical
Community it is used to accomplish many
mathematical task so before concluding
the topic let me answer the query that I
have asked regarding R and python do you
guys remember the
question the query
was our language is superficially
related to which
language so the answer for the question
is C language
next question was which one of the
following is a python file extension it
was an easy question answer for that
question is do
py simply learn offers you with the best
courses for both Python and R
certification offered by simply learn
regarding RS data science with our
certification CES it is a skillup course
that contains 64 hours of Applied
training 10 real life industry projects
dedicated mentoring session from
industry experts and finally lifetime
access to self-based learning I will
provide the link for this course in the
description below data scientist for
python it is a best course for a
beginner it includes five courses in
collaboration with IBM at an affordable
price courses includes python for data
science after that we will be covering
applied data science with
python machine
learning as well as table
learning and data science Capstone and
finally receive a master's certificate
with individual certification for each
course I will provide the link for this
course in the description below want to
be a certified data expert then here we
have postgraduate program in data
analytics by simply learn take a glance
at the notable features and skills
offered in this course you will benefit
from exclusive IBM hackathons and ask me
anything sessions eight times more
interaction in live online classes with
industry experts capston projects across
three domains and over 14 data analytics
projects using real data sets from
Google Play Store lift World Bank and
more master classes from puu faculty and
IBM experts along with simply learns job
assist for better job prospects acquire
skills in data analytics statistical
analysis using Excel data analysis using
Python and R and data visualization
using table and powerbi enroll Now using
the course Link in the description box
since this is data analysis with python
we got to ask the question why python
for data analytics I mean there's C++
there's Java there's net from Microsoft
why do people go to python for
it so the number of reasons one it's
easy to learn with simple
syntax uh you don't have a very high
type set like you do in Java and other
coding so it allows you to kind of be a
little lazy in your programming uh that
doesn't mean that it can't be set that
way and that you don't have to be
careful it just makes means you can spin
up code much quicker in Python the same
amount of code to do something in Python
A lot of times as one two or three or
four lines where when I did the same
thing say in Java I found myself with 10
12 13 20 lines depending on what it was
it's very scalable and flexible uh so
there's our flexibility because you can
do a lot with it and you can easily
scale it up you can go from something on
your machine to using uh piie spark into
the spark environment and spread that
across hundreds if not thousands of
servers across terabytes of data or
pedabytes of data so it's very scalable
there's a huge collection of
libraries this one's always interesting
because Java has a huge collection of
libraries C has a huge collection of
libraries net does and they're always in
competition to get those libraries out
uh Scala for your spark all those have
huge collection of libraries this is
always changing uh but because Python's
open source you almost always have easy
to access libraries that anybody can use
you don't have to go check your
licensing and have special licensing
like you do in some
packages graphics and visualization they
have a really powerful package for that
so it makes it easy to create nice
displays for people to read and
community support because python is open
source it has a huge community that
supports it you can do a quick Google
and probably find a solution for almost
anything you're working
on python libraries let's bring it
together we have data analytics and we
have python so when we're talking dat
data analytics we're talking python
libraries for data analytics and the big
five players are numpy pandas matplot
Library scipi which is going to be in
the background so we're not going to
talk too much about the scientific
formulas in scipi and
CIT so numpy supports in dimensional
arrays provides numerical Computing
tools useful for linear algebra and
forear
transform um and you can think of this
is just a grid of numbers um and you can
even have uh a grid inside a grid or
data it's not even numbers because you
can also put uh words and characters and
just about anything into that array but
you can think of a grid and then you can
have a grid inside a grid and you end up
with a nice three-dimensional array if
you want to talk threedimensional array
you can think of images you have your
three channels of color four if you have
an alpha and then you have your XY
coordinates for the image we're looking
at so you can go XY and then what are
the the three channels to generate that
color and numpy isn't restricted to
three dimensions you could imagine uh
watching a movie well now you have your
movie clips and they each have their X
number of frames and each of those
frames have X number of XY coordinates
for the pictures in each frame and then
you have your three dimensions for the
colors so numpy is just a great way to
work with in dimensional
arrays now closely with numpy is pandas
uh useful for handling missing data per
mathematical operations provides
functions to manipulate data pandas is
becoming huge because it is basically a
data frame and if you're working with
big data and you're working in spark or
any of the other major packages out
there you realize that the data frame is
very Central to a lot of that and you
can look at it as a Excel spreadsheet
you have your columns you have your rows
or indexes and uh you can do all kinds
of different manipulations of the data
Within uh including filling in missling
data which is a big thing when you're
dealing with large pools or lakes of
data where they might be collected
differently from different uh
locations and Matt plot Library we did
kick over the S piie which is a lot of
mathematical computations which usually
runs in the background of the of numpy
and pandas um although you do use them
they're useful for a lot of other things
in there but the map plot Library that's
the final part that's what you want to
show people and this is your plotting
library in Python several toolkits
extend map plot Library
functionality there's like a 100
different toolkits to extend matplot
Library which range from uh how to
properly display star constellations
from astronomy there's a very specific
one built just for that all the way to
some uh very generic ones we'll actually
add Seaborn in when we do the labs in a
minute several toolkits extend met plot
Library functionality and it creates
interactive visualization
uh so there's all kinds of cool things
you can do as far as just displaying
graphs and there's even some that you
can create interactive graphs we won't
do the interactive grasp but you'll see
you'll get a a pretty good grasp of some
of the different things you can do in
matplot
library let's jump over to the demo
which is my favorite roll up our sleeves
get our hands in on what we're doing now
there's a lot of options when we're
dealing with python uh you can use py
charm is a really popular one uh and
you'll see this all over the place um so
it's one of the main ones that's out
there and there's a lot of other ones I
used to use net beans which is kind of
lost favor uh don't even have it
installed on my new
computer but the most popular one right
now for data science now pie CHMS really
popular for python General development
for data science we usually go to
Jupiter uh notebook or anaconda and
we're going to jump into Anaconda
because that's my favorite one to go to
cuz it has a lot of external tools for
for us we're not going to dig into those
but we will pop in there so you can see
what it looks like so with Anaconda we
have our Jupiter lab we have our um
notebook these are identical Jupiter lab
is an upgrade to the notebook with
multiple tabs that's all it is and we'll
be using the notebook and you can see
that py charm is so popular with um
python that we even have it highlighted
here in Anaconda as part of the setup uh
Jupiter notebook can also be a
standalone uh so we're actually going to
be running jupyter notebook and then you
have your different environments um I
have we're going to be under main Pi 36
there's a rout one and I usually label
it Pi
36 the reason is is currently as of
writing this tensor flow only works in
36 and not in 37 or 38 for doing neural
networks but you can actually have
multiple environments which is nice
they're they separate the kernel so it
helps protect your computer when you're
doing development and this is just a
great way to do a display or a demo
especially if you're looking for that
job pull up your laptop open it up or if
you're doing a meeting get it broadcast
up to the big screen so that the CEO can
see what you're looking
at and when we launch the notebook uh it
actually opens up a file browser in
whatever web browser you have this
happens to be Chrome and then you can
just go under new there's a lot of
different options depending on what you
have installed uh Python 3 and this just
creates an Untitled uh version of this
and you can see here I'm actually in a
simply learn folder for other work I've
done for simply
learn uh and that's where I save all my
stuff and I can browse through other
folders making it really easy to jump
from one project to another and under
here we'll go ahead and change the name
of this and we'll go ahead and rename
it data analytics data analytics just so
I can remember what I was
doing which is probably about 50 of the
folders in here right or files in here
right now uh so let's go ahead and jump
in there and take a look at some of
these different uh tools that we were
looking
at and as we go through the demo let's
start with the uh numpy uh the least
visually exciting and I'm going to zoom
in here so you can see what we're
doing and the first thing we want to do
is import
numpy and we'll import it as NP that is
the most common numpy
terminology and let's go and change the
view so we also have the line numbers um
I don't know why we probably won't need
them but like it for easy reference uh
and then we'll create a onedimensional
array we just call this array one and it
equals np. array and you put your array
information in here in this case we'll
spell it out uh you can actually do like
a range and other ways there's lots of
ways to generate these arrays but we'll
just do one two three so three
integers and if we
print our array
one we can go ahead and run this and you
can see right here it prints one two
three you can see why this is a really
nice interface to show other people what
you're doing uh with the jupyter
notebook uh so this is the basic we've
created an array this is a
onedimensional array and then array is
one two three one of the nice things
about the Jupiter notebook is whatever
ran in this first setup is still running
it's still in the kernel so it still has
the nump imported as NP and it still has
our variable um arr1 for array one equal
to NP array of one two three so when we
go to the next
cell we can check the type of the array
we're just going to print we say hey
what's what what what is this um setup
in here and we want
type um and then we want what is the
type of array one let's go ahead and run
that and it says class numpy in the
array so it's its own class that's all
we're doing is is checking to see what
that class
is and if you're going to look at the
array class uh probably the biggest
thing you do I don't know how many times
I find myself uh doing this uh because I
forget what I'm working on and I forget
I'm working with a three-dimensional or
four-dimensional array uh and I have to
reformat somehow so it works with
whatever other things I have and so we
do the array shape uh the array shape is
just three because it has three members
and it's a one-dimensional array that's
all that
is and with the numpy array we can
easily access um stick with the print
statement if you actually put a variable
in Jupiter notebook and it's the last
one in the cell it will the same as a
print statement so if I do this where
array one of two it's the same as doing
print array of two that's those are
identical statements in our Jupiter
notebook we'll go and stick with the
print on this one and it's three so
there's our print space two and we have
0 1 2 2 = 3
we can easily change that so we have
array one of place
two equals
5 and then if we print our array
one uh you can see right down here when
it comes out it's one two and five and
there I leftt the print statement off
because it's the last variable in the
list um and it'll always print the
variable if you just put it in like that
that's a Jupiter notebook thing don't do
that in by charm I've forgotten before
doing a demo
and we talked about multiple Dimensions
so we'll do an array um two-dimensional
array and this is again a numpy
array and in the numpy array we need um
our first Dimension we'll do one 2 3 and
our second dimension uh 3 4 five and you
can see right here that when we hit the
uh we'll do this we'll just do array two
and we can run that and there's our
array two 1 2 3 3 4 5 we can also do
array
two of uh
one and then we can do let's do zero
doesn't really matter which one actually
just do uh two there we go and if I run
this it'll print out five uh because
here we are this is zero uh 0 1 2 3 is
on our zero row 345 is on our one row
you always start with zero and then the
two 0 one two goes to the
five and then maybe we forgot what we
were working with so we'll go do array
2.
shape and if we do array two of
shape uh we'll go and run that we'll see
we have two rows and each row has three
elements a two-dimensional array 2 three
if you looked up here when we did it
before it just had three comma nothing
when you have a single entity it always
saves it as a tupal with a blank
space uh but you can see right here we
have 2 comma
3 and if you remember from up here we
just did this array two of oh let's go
what is it one comma
2 we run that we get the five you can
also count backwards this is kind of fun
and you'll see I just kind of Switched
something on you because you can also do
one comma two to the same spot um now
two is the last one 0 one two it's the
last one in there we can count backwards
and do minus one and if we run this we
get the same answer whether we count it
as uh let's go back up here whether we
count this as 0
one2 or we count backwards as minus1 -2
minus 3 and you can see that if I change
this minus1 to a minus two and run
that I get four which is going backwards
-1 - 2 so there's a lot of different
ways to reference what we're working on
inside the numpy
array it's really a cool tool it's got a
lot of things you can do with
it and we talked about the fact that it
can also hold things that are not values
and we'll call this array ask for
Strings equals uh np.
array put our setup in there brackets
and let's
go
China um
[Music]
India
USA uh
Mexico doesn't matter we can make
whatever we want on here and if we print
that
out we run this you can see that we get
a our numpy array China India USA Mexico
it even gives us our dtype of a
U6 and a lot of times when you're
messing with data we'll call this array
R for range just to kind of keep keep it
uniform np. a range so this is a command
inside numpy to create a range of
numbers and if you're testing data Maybe
you want maybe you have equal time
increments um that are spaced a certain
point apart but in this case we're just
going to do
integers and we're going to do a uh a
setup from 0 20 skipping every other one
and we'll print it out and see what that
looks
like and you can see here here we have 0
2 4 6 8 10 12 14 16 18 like you expected
it skips every one and just a quick
note there's no 20 on here uh why well
this starts at zero and counts up to 20
so if you're used to another language
where explicitly says uh less than or
less than equal to 20 like four xal 0 um
x++ uh X is less than 20 that's what
this is it just assumes X is less than
20 on
here and if we want to create a very
uniform uh set you know 0 2 4 6 what
happens if I want to create numbers uh
from 0 to 10 but I need 20 increments in
there uh we can do that with line space
so we can create um an R uh we'll call
this
L equals I don't think we'll actually
use any of this again so I don't know
why I'm creating unique um identifiers
for it but we'll do
NP Lin
space and we're going to do
0 to 10 or 0 to 9 remember it doesn't it
goes up to 10 and then we want to let's
say we have
20 different um increments in there so
we're creating a we have a data set and
we know it's over a certain time period
and we need to divide that time period
by 20 and it happens to just have 10
pieces in it um and here we go you can
see right here we have TW or it has 20
pieces in it but it's over 10 years we
got to divide it in the middle and you
can see it does goes 0.52 remember yeah
there's our 10 on the end so it goes up
to
10 uh and then we can also do random
there's np. random if you're doing
neural
networks uh usually you start it by
seating it with random
numbers and we'll just do np. random and
we'll just call this array we'll stop
giving it unique numbers we'll print
that one out and run it and you can see
we have random numbers they are zero to
one so you'll see that all these numbers
are under one and you can easily alter
that by multiplying them out or
something like that if you want to do
like 0 to 100 um you can also round them
up if it's integer 0 to 100 there's all
kinds of things you can do but it
generates a random float between zero
and one and you have a couple options
you could reshape that um or you can
just generate them uh in whatever shape
you want and so we can see here uh we
did three and four and so you can see
three rows by four variables same thing
as doing a reshape of 12 variables to
three and
four and if you're going to do that you
might need an empty data set um I have
had this come up many times where I need
to start off with zero and I don't know
you know because I'm going to be adding
stuff in there or it might be zero and
one or one is uh if you're removing the
background of an image you might want
the background is zero and then you
figure out where the image is and you
set all those boxes to one and you
create a mask so creating mask over
images is really big and doing that with
a a numpy array of
zero and we can also
uh give it a
space and we'll just do this all in one
shot this time and we'll do the same
thing like we did
before zero and in this case we'll do uh
2 comma 3 and so when we run
this forgot the asteris around it I knew
I was forgetting something there we go
so when we run this uh you can see here
we have our 10 zeros in a row and maybe
this is a mask for an image and so it
has uh two rows of three digits in it so
it's a very small image um little tiny
pixel
and maybe you're looking to do something
the opposite way uh instead of uh
creating a mask of zeros and filling in
with ones uh maybe you want to create a
mask of ones and fill them in with uh
zeros and we'll just do just like we did
before we'll do three comma four and
when we run this you'll see it's all
ones and we could even do this even U
we'll do it this way let's do
10 10 x 10 icon and then you have your
three colors you so creates quite a
large array there for doing pictures and
stuff like that when you add that third
dimension
in um if we take that off it's a little
bit easier to
see we'll do 10
again and you can easily see how we have
10 rows of 10
ones and you can also do something uh
like create an
array and we'll do 0 one 2
and then in this array um we actually
print it right out we want a repeat and
so you can actually do a repeat of the
array and maybe you need this array um
let's repeat it three
times so there's our repeat of an array
repeat three
times and if we run this you'll see we
have 0000 0
111222 and whenever I think of a repeat
I don't really think of repeating being
the first digit three times the second
digit I really always think of it as um
012 012 012 it catches me every time uh
but the actual code for that one is
going to be tile uh and again if we do
AR range
three and we run this you can see how
you can generate one 012
012012 and if you're dealing with um an
identity Matrix um we can do that also
if you're big on your doing your
matrixes and we'll just
identity I guess we'll go and spill it
out today
Matrix and the command we're looking for
is um I eye and we'll do three and then
we'll just go ahead and print this
out there we go there's our identity
Matrix and it comes out by a 3X3 array
because there's our
Matrix uh and then it puts the ones down
the middle and for doing your different
Matrix math and we can manipulate that a
little bit too um we talk
about uh
matrixes we might not want ones across
the middle in which case we now have uh
the diagonal so we can do an np.
diagonal and we do a diagonal uh let's
put in the
diagonal 1 2 3 4 5 and when we run
this again this generates a value and by
just putting that value in there is the
same as putting print around it or
putting array equals and then print
array and you can see it generates a
diagonal 1 2 3 4 5 and there's your uh
your beginning of your Matrix array for
working with
matrixes and we can actually go in
reverse uh let's create an array equals
remember our random om random. random
and we'll do a 5x5 array uh oops there
we go 5x
five and just so you can see what that
looks
like helps if I tpe don't mistype the
numbers which is in this case I just
need to take out the brackets and there
you go you have your your 5x5 array set
up in there and we can know because
we're working with matrixes we might
want to do this in reverse and extract
the diagonals which would be the 79 the
678 and so
on and we simply type in np.
diagonal we put our array in there um
and this will of course print it out
because it returns it as a variable and
you can see here here's our diagonal
going across from our
Matrix and we did talk about shape
earlier if you remember you can do um
print the shape out you can also do the
dimensions uh so so in Dimensions very
similar to shape it comes out and just
has two Dimensions we can also look at
the size so if we do size on here we can
run that and you can see has a size of
25 two dimensions and of course 5x5 and
that was from the shape from earlier
that we looked at uh there's our 5x5
shape and if you remember earlier we did
random well you can also do uh random I
talked a little bit about manipulating
zero to one and how you can get
different answers you can also straight
for the integer part and we'll do minus
uh 10 to
10
4 and so we're going to Generate random
integers between minus 10 to 10 uh we're
going to generate four of those and so
we run that we have 7 - 3 - 6 - 3
they're all be between - 10 and 10 and
there's four of
them and now we jump into some of the
functionality of arrays uh which is
really great cuz this is where they come
in here's your array and you can add 10
to it and if I run this um there takes
my original array from up
here with the integers and adds 10 to
all of those values so now we have oh
this is the decimal that's right this is
a random decimal I had stored in
Array um but this takes a random decim
the random numbers I had from 0o to one
and adds 10 to them and we can just as
easily do uh minus
10 uh we could even
do times
two and we could do divide by two and it
would it'll take that random number we
generated and cut it in half so now all
these numbers are under
0.5 uh another way you can change the
numbers to what you need on
there and as you dig deeper into numpy
we can also do exponential so as an
exponential function uh which should
generate some interesting numbers off of
the random so we're taking them to the
power I don't even remember what the
original numbers in the um array were
because we did the random numbers up
there here's our original numbers and if
you build an exponential on there uh
this is where you get e to the X on this
and just like you can do e to the X you
can also do the log so if we doing
logarithmic
functions that reinforced learning you
might be doing some kind of log setup on
there and you can see the logarithmic of
these different array
numbers and if you're working with a log
base
two you can do you can just change it in
there NP log 2 you have to look it up
because this is not log 1 2 3 four five
um it is Log and log two uh so just a
quick note that's not a variable going
in that is an actual command there's a
number of them in there and you'll have
to go look and see uh what the
documentation is but you can also do log
10 so here's log value
10 uh some other really cool functions
you can do with this is your sign so we
can take a sign value of all of our
different uh values in there and if you
have sign you of course have
cosine we can run that uh so here's a
cosine of those and if you're doing
activations in your numpy array and
you're doing a tangent activation uh
there's your tangent for that and the
tangent activation is actually uh from
uh neural networks that's one of the
ways you can activate it because it
forms a nice curve between uh from
whether you're generating one to
negative one uh with some discrepancy in
the
middle just jumping a little bit in
there into neural
networks and then we get into let me
just put the array back out there so
that we can see it uh while we're doing
this as we're getting into this you can
also sum the values so we have n p
sum and you can do a summation of all
the values in this array and you'll see
that if you added all these together
they'd equal 12.51 9 so on I don't know
what the whole setup is in
there uh but you can see right here the
the summation of this one of the things
you can also do is by axes so we could
do axes equals zero and if we run the
summation of the axis equal
zero and you can think of that uh in
numpy as the rows so that would be uh or
you can think of that in nump is being
the columns we're summing these columns
going across and you can also change
this to
one and now we're summing the
rows and so that is the summation of
this row and so forth and so forth going
down and maybe you don't need to um know
the summation Maybe maybe what you're
looking for is the
minimum uh so here's our minimal you're
looking for and this comes up a lot
because you have like your errors we
want to find the minimal error inside of
this array and just like um the other
one we can do X is equals
z and you can see here
0.645 is the smallest number in this
First Column is 0645 and so
on and if you have a minimum well you
might also want to know the max Maybe
looking for the maximum profit and here
we go you can see maximum 79 is a
maximum on this first column and just
like we did before you can change this
to a one on axis you can take the axes
out of here and just find the max value
for the whole array and the max value in
here was 8344 so on so
on and since we're talking data
analytics uh we want to go ahead and
look at the mean uh pretty much the same
as the average this is the mean across
the whole thing and just like we did
before we could also do axis equals
z and then you'll see this is the mean
of this axis and so on and we have mean
we might want to know the
median and there's our median our most
common numbers uh if we have median we
might want to know the standard
deviation or if we have the average a
lot of times you do the means and the
standard deviation um we can run that
and there's our standard deviations
along the axes we can also do it across
the whole
array uh if we're going to do standard
deviations there's also uh
variance which is your
V and there's our variance across the
different
levels and so if we looked at that we
looked at variance we looked at standard
deviation the median and the means
there's more but those are the most
common ones used with data analytics um
and then going through your data and
figuring out uh uh what you're going to
present to the
shareholders and some other things we
can do is we can actually take slices uh
you'll hear that
terminology and a slice might be um like
we have a 5x5 array but maybe we don't
want the whole array maybe we want uh
from one on we don't want the zero in
there so we got up to four and maybe on
the second part we just want two to Row
three and see this notation right here
says one to the end and if we run this
you can see how that generates a single
row to the end and then row two and
three now remember it doesn't include
three that's why we only get the one
column so if you wanted two and three
you would need to go ahead and go two to
four so it goes up to four we could also
do this in Reverse just like we learned
earlier we can go minus one
oops and when we go to minus one it's
the same thing because we have 0 1 two 3
four this is the same thing as two to
four it goes two to the last
one also very common with arrays is
you're going to want to sort them so we
still have our array up here that we
randomly generated and we might want to
um sort it and we'll go and throw an
axis back in there uh axis equals 1 if
we run this
you can see from the axes that it sorts
it uh the point 2 being the lowest value
to the highest value by the row we can
also change this of course to axis zero
if you're sorting it by columns so maybe
your values are based on columns and
then of course you can do the whole
array and we can sort that don't usually
do that but you know I guess sometimes
you might that might come
up and so you can see right here we have
a nice sorted array uh something else
let's just go ahead and reprint our
array so we can look at it again
starting to get too many boxes up there
uh something else you can do with an
array is we can take and transpose it
this comes up more than you would think
when you transpose it you'll see that um
the rows and the column are transposed
so where
7957 064 is a column now we've switched
it and we have 7 9.42 as the index
you can see this really more dramatic if
we take a
slice and we'll just do a slice of the
first couple and then we'll just do all
the other um the full rows and if we run
this you can see how it comes up a
little bit different and we'll just do
the same slice up here so you can see
how those two look next to each
other there we go there's our slice run
uh and so you can see the slice comes up
and it has H one two three four five
columns now we have 1 2 3 four five rows
and three columns versus three
rows and the original version when they
first started putting this um together
uh was a function so the original
version was transpose and this still
works you can still see it generates the
same value as just a capital T so many
times we flip this data because we'll
have an XY value or we'll have an image
or something like that and it's being
read one way into the next process and
the next one needs it the opposite uh so
this actually happens a lot you need to
know how to transpose the data really
quick and we can go ahead oh let's just
take um here's our transpose we'll just
stick with the transpose on here and
instead of uh doing it this way we might
need to do something called flattening
why would you flatten your data uh if
this is an array going into a neural
network you might want to send it in as
one set of values instead of two two
rows and you can see here is all the
values as a single array it just
flattens it down into one
array so we covered our scientific uh
means transpose median um some different
variations on here some of the other
things we want to do is what happens if
we want to Pinn to our array uh so let's
create a new array get En tired of
looking at the same set of random
numbers we generated earlier um so we'll
go and create a new array here something
a little simpler so it's easier to see
what we're
doing and four five 6 7 8 uh that's good
now we'll just do
45678 and if we print this
array there it is
45678 and we might want to pend
something to the array so we have our
array we need to extend it you got to be
very careful about appending things to
your array and there's a number of
reasons for that uh one is runtime
because of the way the numpy array is
set up a lot of times you build your
data and then push it into the numpy
array instead of continually adding on
to the array um and then it also usually
it automatically generates a copy for
protecting your data so there's a lot of
reasons to be careful about a pending
this way uh but you can certainly do it
and we can just take our array we're
going to create a new array array one
and if we print array one and we append
eight to it you'll see four five six 7
and then there's our eight appended on
to the
end and if you want to append something
to an array um you'd probably also want
to
whoops array one let's try that again
there we go now we have the eight pended
on to the end um so you can see four
five six seven eight and then we pended
another eight on
there and if you're going to appin
something you might want to um go ahead
and insert instead of appending it might
be you need to keep a certain order and
we can do the same thing we do our
array um and we're going to pin or um
insert at the beginning and let's go
ahead and insert uh one two three one
two3 and we go ahead and print our array
to we run it and you can see 1 two 3 is
inserted at the beginning uh inserts a
lot more powerful and that you can put
it anywhere in the array we can move it
to the one spot and there we go 1 2 3
three uh we can do a minus one just for
fun and you'll see it comes up uh 1 2 3
we're counting backwards by one I
imagine you can do a minus
0 and run this and it turns out that
minus 0 puts it back at the beginning
because that's why it registers a zero
just takes a minus sign
off and just like we add numbers on we
might want to delete numbers and so uh
let's do an np. delete well let's let's
keep it a little bit make it a little
easy here um to watch we'll go and
create an array three and we'll do NP
delete we were just working with array
uh
two and what we want to do is delete
zero space uh so if you look at this
here's our array two array two starts
with one and when we delete the space on
here and print that out uh we deleted
the one right out of
there and we can also do something like
this where we can do it as a slice and
we can do let's do 1 comma 3 and if we
run 1 comma 3 you'll see we've deleted
the one space and the three space out
which deleted our two and
four now keep in mind when you're
messing with um adding lines and
deleting
lines uh you have to be really careful
because there's a time element involved
um as far as where the data is coming
from and it's really easy to delete the
wrong data and corrupt what you're
working on or to insert stuff where you
don't want it um so there always a
warning when we talk about manipulating
numpy
arrays and just like anything else we're
doing uh we'll create an array C which
equals we'll just do our um our numpy
array that we just created our numpy
array three and we can do copy so you
can make a copy of it um maybe you want
to protect your original data or maybe
you're making a mask and so you copy the
array and then the new array make all
these alterations and change it from
values to 0o to one to mask over the
first one and of course we if we do um
array C since it equals a copy of uh
array three it's the same thing 1 3 5 6
78 and now we're getting into uh combine
and split arrays I end up doing a lot of
this and I don't know how many times I
end up fiddling with this and having a
mess uh so but but you do it a lot you
know you combine your arrays you split
them you might need one set of data for
one thing another set of data for the
other so let's go and create two arrays
array one array two and I want you to
note in the terminology we're going to
look for is concatenate what that means
is we're going to take um we'll call
this AR Ray cat I like a ray cat there
we go um our array cat or concatenated
array
we're taking array one and two and it's
very important to really pay attention
to your axes and your counts I can't
merge two arrays that have like if their
axes are messed up and I'm merging on
axis zero it's going to give me an error
and I'll have to reshape them so you got
to make sure that whatever you're
concatenating together works and what
that
means is you can see here we have 1 2 3
4 1 2 3 4 and then 5 6 7 8 5 5 6 7 8
along the zero axis these each are four
values um so it's a 2x4 value and if we
go ahead and switch this to one you can
see how that's that flips it a little
bit so now we have 1 2 3 4 5 6 seven
eight it's interesting that we chose
that one if I did something like
this where this is
now there we go and we can catenate it
um run this and it gives me an answer
okay because I have two by two and I'm
using axis one but if I switch this to
axis zero where now it's got three and
five it gives me an error so you got to
be really careful on that to make sure
that your whatever axes you are putting
together that they match um so like I
said this one oops axis one axis one has
two entities and since we're going on
axis one or by row you can see that it
lets it merge it right onto the end
there and you could imagine this if this
was a XY plot of value or the x value
going in and the predicted yv value
coming out and then you have another
prediction and you want to combine them
this works really easy for that we'll go
back and let's just put this back to
where we had
it oops I forgot how many changes I made
there we go U we'll just put it Oops I
messed up in my concatenation order
[Music]
here there we go
okay so you can see that we went through
the different concatenation axes is
really important when you're doing your
concatenation values on here we'll
switch this back to one just because I
like the looks of that better there we
go two
rows now there are other commands in
here um so we can do uh cap V equals
npv v stack this is nothing more than
your
concatenation uh but instead we don't
have to put the axes in there uh because
it's v stands for vertical and so if we
print out
cat V and we run this you can see we get
the 1 2 3 4 One 2 3 4 and that would be
the same as making this axis zero for
vertical stack and if you're going to
have a vertical stack uh you can also
have an H
stack so if we change this to from v
stack to oops here we go H stack and
we'll just change this from cat to cat
and I run
this it's the same as doing axis zero
the process is identical in the
background um this is like a legacy
setup uh your v stack and your hstack
most people just use concatenate and
then put the axes in there because it's
much uh has a lot more clarity and um is
more more commonly used
nowadays
the last section in numpy we're going to
cover uh
is is kind of uh data exploration um and
that'll make a little bit more sense in
just a moment sometimes they call them
set operations but let's say we have an
array 1 2 3 4 5 6 3 whatever it is uh
you we generate a nice little array here
and what I want to go ahead and do is
find the unique values in that
array uh so maybe I'm generating what
they call a one hot
encoder and so these values then all
become I need to know how long my bit
array is going to be so each word how
many how many each word is represented
by a number and then I want to know just
how many of those words are in there if
we're doing word count very popular
thing to
do um and you can see here when we do
unique uh we have 1 2 3 4 five six those
are our unique
values uh some of the things we can do
with the unique values is we can also
instead of doing just unique we can do
uniques our unique values and counts of
each unique
value and this is very similar to what
we just did up here where we uh we're
doing NP unique uh but we're going to
add a little bit more into there and
it's just part of the arguments in this
and we want to do
return counts equals true so in instead
of just returning the unique values uh
we want to know how many of those unique
values are in each one one and we'll go
ahead and
print our
uniques and
print our
counts when we run that uh you can see
here we have our unique value 1 2 3 4 5
six just like we had before and then
there's two of the first of two ones two
twos two 3es two fours one five two
sixes and so on and you can go through
and actually look at that if you want to
count them um but a quick way to find
out your um uh distribution of different
values so you might want to know how
often the word the' is used versus the
word and if each word is represented as
a unique
number and along the set variables we
might want to know um let me just put a
note up here we're going to start
looking
at uh
intersection and we might want to also
know
differentiation and uh
neither so when we're oops
neighbor neither um so what we're
looking at now is we want to know hey
where do these two arrays intersect and
we have 1 2 3 4 5 3 4 5 6 7 we might
want to know what it's common between
the two
arrays um and so when we do that we have
um
NP
intersect and it's a ond array
onedimensional array
and then we need to go ahead and put
array uh one array
two and if we run
this we can see they intersect at 3 four
five that's what they have
common uh and because we're going to go
ahead and go through these and look at a
couple different options let's change
this from intersect
1D and we'll do the same thing we'll go
ah and print this so we might want to
know the intersection uh where they have
commonalities
another uh unique word is Union of 1D uh
so instead of uh
intersect we want to know all the values
that are in both of them so here's our
Union of 1D when we run that you can see
we have 1 2 3 4 5 6 7 so it's all the
different values in
there and the last one of the last words
we have two more to go uh is we want to
know what the set difference
is uh and so that's where the you'll see
if you remember set we talked about that
being the what they call these things um
so the set
difference of a 1D array when we run
that you can see that one is only in one
array and two is only in one
array and if we want to know uh what's
in Array one but not in Array two we
might want to know what is in Array one
but not two and what's in two but not
one uh and this would be the set X or 1D
on here uh so we have the four different
options here where we can do an
intersection what do they both have in
common uh we can do a union what are all
the unique values in both arrays we can
see the difference what's in Array one
but not array two so set diff 1D and
then set X or what is not in one but is
in two and what is in not in two but in
one so we dug a lot in numpy because
we're talking um there's a lot of
different little mathematical things
going on in numpy a lot of this can also
be done in pandas although usually the
heavy lifting is left for numpy because
that's what it's designed for let's go
ahead and open up another Python
3 setup in here and so we want to
explore uh what happens when you want to
display this this is where it starts
getting in my opinion a little fun
because you're actually playing with it
and you have something to show people
and we'll go ahead and rename this we're
going to call this uh
pandas uh and pip plot so pandas pip
plot just so I can remember for next
time and we want to go ahead and import
the necessary libraries we're going to
import pandas as PD now remember this is
a data frame so we're talking rows and
columns and you'll see how uh pandas
work so nicely uh when you're actually
showing data to people and then we're
going to have numpy in the background
numpy works with pandas uh so a lot of
times you just import them by
default caborn sits on top of the map
plot l Library uh so sometimes we use
the caborn because it kind of extends
it's one of the 100 packages that
extends a map plot Library probably the
most common used because it has a lot of
built-in functionality um almost by
default I usually just put caborn in
there in case I need it and of course we
have matplot Library as pyplot as
PLT and note we have as PD as NP as snss
PLT those are pretty standard so when
you're doing your Imports I would
probably keep those just so other people
can read your code and it makes sense to
them that's pretty much a standard
nowadays and then we have the strange
line here uh it says uh amber sign
matplot Library
inline that is for Jupiter notebook only
so if you're running this in a different
package you'll have a popup when it goes
to display the matplot library um you
can with the most current version of
Jupiter usually leave that out and it
will still display it right on the page
as we go and we'll see what that looks
like and then then we're going to go
ahead and just uh do the um caborn the
SNS do set and we're going to set the
color codes equals true let them uh just
keep the default one so we don't have to
think about it too
much and we of course have to run this
um the reason we run this is because
these values are all set if we don't run
this and I access one of these um
afterward it'll it'll crash the cool
thing about Jupiter uh notebooks is if
you forgot to import one of these you
forgot to install it CU you do have to
install this under your anaconda setup
or whatever setup you're in you can flip
over to Anaconda and run your install
for these um and then just come back and
run it you don't have to close anything
out and we'll go ahead and paste this
one in here real quick where we have car
equals pd. reor CSV and then we have uh
the actual path this path of course will
vary depending on what you are working
with uh so it's wherever you saved the
file at and you can see here I have um
like my one drive documents simply Learn
Python data analytic using python slash
car CSV it's quite a long
file when we open that up what we get is
we get a CSV file and we have the make
the model the year the engine fuel type
uh engine horsepower cylinders and so on
um and this is just a comma separated
file so each row is like a row of data
think of it as a um spread sheet and
then each one is a column of data on
here and as you can see right here it
has the uh make model so it has columns
for a header on
here now your pandas just does an
excellent job of automatically pulling a
lot of this in so when you start seeing
the pandas on here you realize that you
are already like halfway done with
getting your data in uh I just love
pandas for that reason numpy also has it
you can load a CSV directly into numpy
um but we're working with pandas and
this is where it really gets cool is I
can come down here and I can print uh
you remember our print statement we can
actually get rid of it and we're just
going to do car head because it's going
to print that out the head is going to
print the top values of that data file
we just ran in and so you can see right
here it does a nice print out it's all
nice and inline because we're in jupyter
Notebook I can scroll back and forth and
look at the different data uh and just
like we expected we have our column it
brought the header right in one thing to
note is the index it automatically
created an index 0 1 2 3 4 and so on and
we're just looking at the head so we got
0 1 2 3
4 um you can change this you might want
to just look at the top two we can run
that there's our top two
BMWs um another thing we can do is
instead of head we can do
tail and look at the last three Val
values that are in that data file and uh
you can see right here it numbered them
all the way up to
11,910 oh my goodness they put a lot of
data in this file I didn't even look to
see how big the file was uh so you can
really easily get through and view the
different data in here when you're
talking about Big
Data you almost never just print out car
uh fact let's see what happens when we
do if we run this and we just run the
car it's huge uh in fact it's so big
that the pandas automatically truncates
it and just does head plus tail so you
can see the two um so we really don't
want to look at the whole thing we'll go
and go back to we'll stick with the head
displaying our data there we go so
there's a head of our data gives us a
quick look to see what's actually in
there um I can zoom out if we want so
you actually get a better
view although we'll keep it zoomed in so
you can see the code I'm working
on and then from from the uh data
standpoint we course want to look at um
data types uh what's going on with our
data what does it look like uh now this
you know you show your when you're
talking to your
shareholders they like to see these nice
easy to read charts they look like a
spreadsheet uh so it's a nice way of
displaying pieces of the
chart when we talk about the data types
now we're getting into the data science
side of it what are we working with well
we have uh make model we have integer 64
for the year uh engine fuel type is an
object if we go up here you can see that
there most of them are um like you know
it's a set manual rear wheel drive uh so
they might be very limited number of
types in
there uh and so forth and you it's
either going to be a float 64 an integer
or an object is the way it's going to
read it on
here and the next thing you going to
know is like your
columns and since it loaded the columns
automatically uh we have here the make
the model the year the engine the size
all the way up to the
MSRP and um just out of something you'll
see come up a lot is whenever you're in
pandas you type in values it converts it
from a pandas uh list to a numpy
array and that's true of any of these uh
so then you end up in a numpy array so
you'll see a little switch in there in
the way that the data is actually stored
and that's true of any of these uh in
this case so we went car.
columns you have a total list of your
car columns and like any good data um
scientist we want to start looking at
analytical summary of the data set
what's going on with our data so we can
start trying to um piece Mill it
together so we can do
car uh
describe and then we'll do is we'll do
include
equals all uh so a nice Panda command is
to describe your data if you're working
with r this should start looking
familiar uh and we come down here and
you can see um count there's uh make the
model the year um how many of each one
how many unique values of each one uh
the top value of each one what's most
common the frequency the mean um clearly
on some of these it's an object so it
really can't tell you what the um
averages you it' just be the top ones
the average I guess um the year what's
the average year on there um all this
stuff comes down here your standard
deviation your minimum value your
maximum value uh what's in the lower
quarter 50% Mark where's that line at
and what's in the upper 75% the top 25%
going into the
max now this next part is just cool uh
this is what we always wanted computers
to be back like in the9 90s instead of
5,000 lines of code to do this maybe not
5,000 all right I built my own plot uh
Library back in 95 and the amount of
code for doing a simple plot was um I
don't know probably about 100 lines of
code this is being done in one line of
code we have our car which is our pandas
we generated that it's our data frame
and we have doist for histogram that is
the power of caborn now it's still going
to generate a
graph but caborn sits on top and then we
can do the figure size this is just um
so it fits nicely on the paper on here
and we do something simple like this and
you can see here where it comes up and
it does say m plot library and does
subplots and
everything but we're looking at a
histogram of all the different pieces in
our database and we have our engine
cylinders um that's always a good one
because you can see like they have some
that are they had a null on there so
they came out at zero um maybe a couple
maybe one of them had a two-cylinder
engine way back when four is a common uh
six a little less common and then you
see the 8 cylinder uh 12 cylinder
engines boy that's got to be a Speedster
or something uh but you can see right
here it just breaks it down so now you
have uh how many cars with how many
whatever it is cylinders horsepower uh
and so on and it does a nice job
displaying it you can see if you're
working with your uh um you're going
into your uh demo it's really nice just
to be able to type that in and boom
there it is I can see it all the way
across and we might want to zero in uh
and use like a box plot and this time
we'll go ahead and call the um Seaborn
SNS box plot and we're going to go ahead
and do um vehicle size versus um engine
horsepower XY plot and the data comes
from the car so if we run this we end up
with a nice box plot
you see our midsize Compact and large
you can see the variation there's our
outlier showing up there on the compact
that must be a high-end sports car uh
large car might have a couple engines
and again we have all these outliers and
then your deviation on
them very powerful and quick way to zero
in on one small piece of data and
display it for people who need to have
it reduced to something they can see and
look at and understand and that's our
caborn box plot or SNS
boxplot and then if we're going to back
out and we want a quick look at um what
they call pair plotting uh we can run
that and you can see with the caborn it
just does all the work for you uh it
takes it just a moment for it to pull
the data in and compile
it and once it does it creates a nice
grid um and this grid if you look at uh
this one space here which is might not
be able to see the small number says
engine horsepower this this is engine
horsepower uh to the year was built and
it's just flipped so everything to the
right of the middle diagonal is just uh
the rotation of what's on the left and
as you expect um the engine horsepower
um gets bigger and bigger and bigger as
time goes on so the the year it was
built further up in the year the more
likely you are to have a heavy
horsepower
engine and you can quickly look at
Trends with our pair plot coming up uh
and look how fast that was that was it
took it a you know a moment to process
uh but right away I get a nice view of
all these different um information which
I can look at visually and and kind of
see how things group and
look now if I was doing a meeting I
probably wouldn't show all the data
um one of the things I've learned over
the years is um people myself included
love to show all our work you know we
were taught in school show all your work
prove what you know the CEO doesn't want
to see a huge U grid of of graphs I
guarantee it uh so we want to do is we
want to go ahead and
drop um the stuff that might not be
interested in and we're going to I'm not
really a car person guing the back is
obviously so you have your engine fuel
type we're going to drop that we're
going to drop Market category vehicle
style popularity number of doors vehicle
size um and we have the axes in here if
you remember from numpy we have to
include that axis to make it clear what
we're working on that's also true with
pandas and then we'll look at just what
what it looks like um from the head and
you can see that we dropped out those
categories and now we have the make
model year uh and so forth um and we
took out the engine fuel type Market
category
Etc uh and this should look familiar to
you now when you start working with
pandas I just love pandas for this
reason look how easy it is it just
displays it as a nice um uh spreadsheet
for you you can just look at it and view
it very easily uh it's also the same
kind of view you're going to get if
you're working in spark or py spark
which is python for spark across Big
Data this is the kind of thing that they
they come up with this is why pandas is
so
powerful and we might look at this and
decide we don't like these columns and
so you can go in here and we can
actually rename the
columns simple command car equals car
rename uh columns equals engine
horsepower equals horsepower power this
is just your standard python
dictionary um so it just Maps them out
and you know instead of having like the
lengthy here we had um engine horsepower
we just want horsepower we don't need to
know what the engine horsepower engine
cylinders we don't need to know that
it's for the engine because there's only
one thing we're describing if we're
talking about cars and that's
cylinders uh we'll go ahead and just run
this and again here's our car head and
you can see how that changed we have
model year in Horsepower versus model
year in engine horsepower engine
cylinders and just
cylinders again we want to keep reducing
this so it's more in more readable the
more readable you get it the better um
and of course we can also adjust the
size a little bit so that when it prints
out instead of splitting it on two lines
we get like a single line we can do that
also that's just your control mouse up
or plus sign you use in Chrome that's a
chrome
command and if you remember from numpy
we had shap well pandas works the same
way uh we can look at the shape of the
data so we now have um
11,914 rows and 10 columns uh so you see
some similarities because pandas is
built on
numpy and questions that come up just
like you did in numpy we might want to
know duplicate rows and so we can do car
and look at this switch here um we're
doing a selection this is a panda
selection with the brackets but we want
want to select it based on car.
duplicated so how many duplicates on
there so it's starting to look a little
bit different as far as how we access
some of the data on here this can be a
logical statement and we get the number
of duplicate rows we have 989 rows by 10
columns
again and this is one of those
troubleshooting things that we end up
doing uh a lot more than we really feel
like we should uh we might go ahead and
do like a car count uh just to see how
many rows we're dealing with and and
then right after that we might want to
go ahead and say hey um let's drop
duplicates so remember we did all the
duplicates on there so car equals car.
drop duplicates and then we can print
the head again we'll just do car head
here and you can see the data on there
um looks the same as
before uh and just note that we did car
equals car draw
duplicates there are commands in here
where you can do where it changes the
actual value and and it works on some of
them and not on others depending on what
you're doing but by default it always
returns a copy so when we do this we're
reassigning it to car and you can see
it's the same header but we want to go
ahead and do count and see how the count
changes let's go ahead and run this and
you can see here instead of 11914 we
have
10,925 uh so we've removed about 100
cars that were duplicated just slightly
under a 100
there and then as we're prepping our
data we might want to know um car is
null uh so it's going to count the
values of null and then we want to sum
that up and when we do that uh we do the
car is n function. suum uh we end up
with uh HP the horsepower 69 have null
values and 30 have cylinders have null
values now if you don't put the sum at
the end it's just going to return a mask
with the true false of is it null or is
it not by zero and one so you're summing
up the ones underneath each
column and this of course uh then you
have to decide what you're going to do
with the uh null values there's a lot of
different options it might be that you
need to put in the average or means uh
maybe you want to put in the median
value um there's a lot of different ways
to fill it usually when you first start
out with the data a lot of them you just
drop your null values and you can see
here car dot drop na which is equal to
all and then we're going to go ahead and
count it and you can see that we've
dropped almost another 100 values so
from
10925 to
10827 maybe 75 or so values uh so we
clean this is really a big part of
cleaning data you need to know how to
get rid of your null values or at least
count them and what to do with them and
of course if we go back to um uh
counting our null values we should now
have uh null null values there we go and
you'll see there's zero null values I
don't know how many times I've been
running a model that doesn't take null
values and it crashes and I just sit
there and look at it trying to get why
did that crash it should have worked uh
it's because I forgot to remove the null
values so we've been jumping around a
lot we're going to go back to uh finding
outliers and let's go ahead and bring
that back into our caborn and if you
remember we did a box plot earlier uh
this time we're going to do a box plot
just on the price and you can see here
um our price value and we have the
deviation with the two thinner bars on
each side of the main value and then as
we get up here we have all these
outliers um in fact we have one way out
here that's um probably a really
expensive high-end car is what we're
looking
at if you were doing um fraud analysis
you would be jumping on all over these
outliers why are these deviation from
the standard what are these people doing
again this is probably like I said
really high-end expensive car out here
that's what we're looking at and we can
also look at the um box plot for the
horsepower and we'll put that in down
here and run that and you can see again
here's our horsepower and it just jumps
and there's these really odd huge muscle
cars out here that are
outliers and we're going to jump into
making this a little bit more um as you
start displaying your data or your
information to your shareholders uh
we're going to look at plotting a
histogram for the number of cars per
brand and the first thing we want to go
ahead and do is we have with our car go
back over here here we go uh we have our
make value counts largest plot um and we
want to do a kind equals bar uh fig size
105 and right off the bat we jump up
here and we see Chevrolet it's going
against what was it it's um figure res
the value counts and we want the largest
value so here's our value counts and
compared to what the different cars are
Chevrolet puts out a lot of different
kinds of cars I didn't realize that they
made that many cars or different types
and then for readability uh let's go
ahead and add a title number of cars by
make number of cars and make if you had
looked at this the first time you would
have been like well what the heck am I
looking at well we're looking at the
number of cars by make and then you can
see here now we're talking about the
type of cars and the different ones were
put out Lotus I guess only had a few
different kinds of cars over there very
high-end
cars and then as uh doing data analytics
and as a data scientist one of the
things I am most interested in is the
relationship between the
variables uh so this is always a place
to start we want to know what's going on
with our variables and how they connect
with each other uh so the first thing
we're going to do is we're going to go
ahead and set a figure size because we
want to make sure it fits our graph um
we'll just go ahead and set this one
plot Figure Set to figure size 20110 if
you never used the map plot Library
which is sitting behind Seaborn uh
whatever is in the PLT this is what's
loaded it's like a canvas you're
painting on so the second you load that
uh pip plot as PLT anything you do to
that is affecting everything on
it uh and then we want to go ahead uh
since we're using
caborn we'll go ahead and create a
variable C for for uh relationships or
correspondence and car
docr that's a correlation in uh Seaborn
on top of pandas again one line and you
get the whole correlation on there and
because we're working with Seaborn let's
put it into a nice heat map if you're
not familiar with heat maps that means
we're just using color as part of our um
uh setup so we have a nice
visual and we can see here that the
Seaborn connected to the pandas prints
out a nice chart we'll talk a little bit
about the color here in a second it
prints out a nice chart this is a chart
I look at as a data scientist these are
the numbers I want to look at uh and
we'll just highlight one of them um
here's cylinders versus horsepower the
closer to one the higher the correlation
so 788 pretty high correlation between
the number of cylinders and how heavy
the horsepower
is I'm betting if you looked at the year
versus uh horsepower um we just look at
that one here's year and horsepower 314
not as so much but if you combine them
uh you don't actually add them but if
you combine them you'll start to see an
increase in Horsepower per year and
cylinders you could probably get a
correlation there and just like 78 is a
positive correlation uh you might notice
if we look at
cylinders and or let's look at
horsepower and mileage uh so if we go
here to horsepower to mileage you get a
nice um negative we'll do cylinders
that's a bigger number with cylinders to
the miles per gallon it's a minus. 6 so
it's a negative correlation the closer
to minus one the more the negative
correlation
is and then the chart you would actually
show people is a nice heat map this is
all our colors and it's just those
numbers put into a heat map the darker
the color the higher the correlation you
can see straight down the middle um
obviously the year correlates directly
with the year horsepower with horsepower
and and so on that's why it's a one the
closer to the one the higher the
correlation between the two pieces of
data now this is a good introduction uh
pandas goes Way Beyond this most the
functionality in numpy since pandas sits
on it is also in pandas and then even
has additional features in it and we use
caborn pretty extensively sitting on top
over our pip plot uh so keep in mind
that our pip plot has a ton of other
features in it that we didn't even touch
on in here uh we couldn't even if you
had a sole course in it uh there's just
so many things hidden in there depending
on what your domain you're working on uh
but you can see here here's our Seaborn
and here's our matplot library that's
all our Graphics that we did and then
the Seaborn worked really nicely with
the pandas uh we really like that so
that wraps up our demo part for today
let's learn about data manipulation in R
and here we will learn about deer
package and when we talk about this deer
package it is much faster and much
easier to read than base R so deer
package is used to transform and
summarize tabular data with rows and
columns you might be working on a data
frame or you might be getting in a
inbuilt our data set which can then be
converted into a data frame so we can
get this package deer by just calling in
library function and this can be used
for group in by data summarizing the
data adding new variables selecting
different set of columns filtering our
data sets sorting it selecting it
arranging it or even mutating that is
basically creating new columns using
functions on existing variables so let's
see how we work with deer now here I can
basically get the package here so I can
just say install. packages deer now we
already see the the package here which
is showing up so I will just select this
one I can do a control enter and that
will basically set up the package
package deer successfully unpacked so
that is done now you can start using
this package by just doing a library
deer and this was built it shows me my
version of R so let's also use a inbuilt
data set that is New York flights 13 so
we can do install. packages and and that
will search and get that relevant data
set I can again Call It by using Library
function now once that is done we can
look at some sample data here by just
doing view flights and that shows me the
data in a neat and a Taber format which
shows me year month day departure time
schedule departure time and so on now we
can also do a head to look at some
initial data which can help help us in
understanding the data better so what is
this data about how many columns we have
what are the data types or object types
here it shows me how many variables we
have so this is fine now we can start
using deer and in that we can use sa
filter function if we would want to look
in for specific value now here we have
the column as month so I will do a
filter now I'm creating a variable F1
I'm using the filter function on flights
which we already
have and then what we can do is we can
basically look at the month where the
month value is
07 so let's look at that and this one
you can do a view on F1 which shows me
the data wherein you have filtered out
all the data based on month being seven
so this is a simple usage of filter we
can take some other example we may want
to include multiple columns so we can
say F2 filter flights and here we will
say month is equal to 7 day is three and
then look at the value of F2 if you are
interested in seeing this and that tells
you the month is seven and day is three
you could also look into a more readable
format by using view on F2 and that
gives me my my selected result so we are
just extracting in some specific value
we can keep extending this so here we
can say flights is what we would want to
work on I'm using the filter function so
I can straight away instead of creating
a variable then then doing a view I can
also do a view in this way I can just
pass in my filter within the view and
within this I'm saying filter I would
want to look at the flights month being
09 day being two and origin being
LGA and then that shows me the value
here and obviously you can scroll and
look at all the columns and if you see
the origin column it shows the selected
value so now we have filtered out our
data based on values in three different
columns now what we can also do is we
can use and or we can use or operators
so I could have done this in a a little
different way so I could have said head
which shows me initial result I will do
a flight So within my head function I'm
passing in this and what does that
contain so you are saying flights and in
this flights data set you would want to
pick up the month being the column so we
use the dollar symbol here we given a
value and I'll say and and I'll again
say flights wherein I will select the
day being two and and and remember when
you talk about and it is going to check
if all the values are met true so then
you say flight's origin LGA and you look
at the value so in this way I can filter
out specifically multiple values by
specifying columns now we could have
done it in this way we could have
created a view or we could have assigned
this to a variable and then done a view
on that where we could have selected
month being day and origin or you can be
more specific in specifying all the
columns it makes the code more readable
so let's look at the values and here
you're are looking at head which shows
me based on month day and then you can
look for further columns for other
variables that is origin being LGA now
what we can also do is we can do some
slicing here to select rows by
particular position
so I can say slice and I would want to
look at rows one to5 and I can do this
so you can always assign or look at the
view of this I can just do here so when
I did a Sly 1 is to 5 it shows me my
entries for 1 to five now similarly we
can do a slice 5 to 10 and now you are
looking at 5 to 10
values so you can always look at the
complete data and then you can slice out
particular data now mutate is usually a
function which is used when you would
want to apply some variable on a
particular data set and then you would
want to add it to your existing data
frame or you would want to add a new
column so this is where you use mute
mutate which is mainly used to add new
variables so let's see how you you work
on mutate so it's pretty simple so you
create a variable over delay now I would
want to do a mutate so that it adds a
new column so I'm selecting my data
which is flights I will call the new
column as overall delay and then
basically I can look at overall delay
being arrival delay minus departure
DeLay So let's create this and let's
look at view of this which shows me or
which should show me my new column which
is overall delay which was not in my
original data set so you can anytime do
a head on this one to compare the value
so this one shows me arrival delay and
then there are many other variables what
you can also do is you can do a view and
you could have just look at flights if
you would want to compare so you can
look at the flight and this one would
not have any overall delay column so it
basically shows me 19 columns only what
we see here and if
you do a view on overall delay then that
basically shows me 20 column so we know
that the new column has been added to
this overall DeLay So if you would want
to work with 20 columns you will use
overall delay if you would want to work
with with your original data set you
will use flights now you can also use a
transmute function which is used to show
only the new column so we can do a
overall delay and at this time we will
say transmute we will say flights
overall delay the computation remains
same but at this time if I look at view
on overall delay it only shows me the
new column so sometimes we may want to
compute result based on two variables or
two columns and just look at the new
value and then we can decide if we would
want to add it to our existing structure
now you can also use summarize and
summarize basically helps us in getting
a
summary based on certain criteria so we
can always do a
summarize and what we can do is we can
look at our data and we can say on what
basis we would want to summarize this
particular data
so we can do a summarize function now
summarize on flights I will say average
air time and I would want to calculate
an average so for that I'm using inbuild
function called mean I will do that on
airtime
column so let's look at flights once
again and here we can see there is
arrival time not a time sorry arrival
time and we would want to do some
average on this particular data we would
want to summarize this so what I'll do
is I will use the summarize function I
will say average air time and this one I
will look at mean of air time so let's
see if there is a air time column I
might be let's look at this one arrival
delay and yes we have an air time so we
were actually looking at summarizing
based on air time not the arrival time
so air time is how much time it takes in
air for this particular fight and we
will want to use the trans summarize
function not the transmute so summarize
flights average air time and this one we
will calculate the mean of average a
time and I will also do a na removal
which is I'm saying true so let's do
this and that basically shows me the
average air time is
151 I can also do a total air time where
I'm doing a summation of values or I can
get the stand deviation or I can
basically get multiple values such as
mean I can say total air time where I'm
doing a summation and then I can look at
other values which is if you would want
to put in standard deviation here you
could do that so let's look at the
result of this summarize and this
basically allows me to get some useful
information which is summarized based on
a particular function such as mean sum
standard deviation
or all three of them now let's look at
grouping by so sometimes we may be
interested in summarizing the data by
groups and that's where we use the group
by function so we can always use the
group by Clause now here we are taking a
different data set so we will say for
example let's look at head of empt cars
and that is basically my data set on
empt cars now that shows me the model of
the car it shows me mileage cylinder p
and your horsepower and various other
characteristics or variables in this
particular data set so here we can say
let's do a grouping by gear so there is
a column called gear so I will call it
by gear I will look at my data set
and then what I'm using here which you
see with these percentage and greater
symbol is called piping so that
basically feeds your previous data frame
into next one so this is sometimes
useful and you can get this by just
saying control shift and M and you can
then use this so we are going to have
piping so I'm saying empty cars now this
is my original data set where I did a
head or I could have done a view on this
one if you would want to see it in a
more readable format and that basically
shows me the data so we are using a
different data set so I want to group it
by the gear column so I'm going to call
it by gear and this one takes my data
that is empty cars I'm using the piping
and then I'm saying group the data based
on gear column that's done now let's
look at the value of by gear or you can
always do to a view so remember whenever
you're doing a group by it is giving you
a internal object where your data is
grouped based on a particular column so
we can look at the values here you can
do a view that shows you your data
grouped based on a particular column now
I can again use the summarize function
where I would want to Now work on the
new one where it was grouped based on
GAR so I'm doing a summarize and here
I'm going to say Gear 1 which will be
having the value of summation on the
gear column and then I'm saying gear two
which is mean well you could give some
meaningful names to this and let's look
at the value of this
one where we are basically now looking
at the values which is sum and mean
values based on the
gear similarly we can use look at
different example so we can say by gear
here and I'm again using
piping but earlier we had taken gear we
had grouped the data and we called it by
gear so we took our original data set
empty cars but now within this
particular data which was grouped by
gear I will take this data set I will
use the piping and I will summarize it
where I am saying within this particular
data set I would want to get the sum or
I would want to get the mean and then
you can look at the values so what
you're doing is you are either looking
at your original data set or you're
looking at the data which was already
grouped and then you can look at the
values now here what we can do is we can
Group by cylinder say might be you are
interested in looking at data which is
summarized based on the cylinder column
you can do that and then for this byy
cylinder I'm doing a piping where I'm
using the summarize function
and summarizing will then be done based
on the mean values of the gear column or
the horsepower so let's do this and then
you can basically look at the value at
any point you may want to look at the
data set again so just do ahead and you
can look at what does the value
contain and by cylinder or by gear and
do a head and it gives you the value so
you can always do some summarizing or
grouping in these ways now here we are
going to use sample n function and
Sample
underscore fraction for creating samples
so for this let's take the flights data
set again and we would want to get 15
random values now that is done and it
shows me 15 rows with some random values
from the data what you can also do is
you can do a a portion of data by using
sample underscore fraction and here I'll
say flights I'll say point 4 which will
return 40% of the total data so this can
be useful when you are building your
machine learning where you would want to
split your data into training and test
might be you are interested in some
portion of the data so you can do this
which is very useful function and then
you can look at the value of that
now what we can also do is we can use AR
range function so like we were doing a
grouping by or we were trying to pull
out a particular column so in the same
way we can use arrange which is a
convenient way of sorting than your base
our sorting so for arrange function
let's do a view based on arrange so we
will work on the flights data set which
we have and here what we would want to
do is we would want to arrange the
flights data set which is based on year
and departure time and we are doing a
view out of it so that basically gives
me the data which is arranged based on
your year and departure time now I can
do a head to give me some highlighting
of that data now the piping operator
what we are using can be used in these
ways also so here I will say DF I will
just assign the data set mty cards to it
let's look at the DF which has basically
your different models you can obviously
look at the head or view of it to look
at useful information we can also go for
nesting options which can be useful so
we are creating a variable called result
here now that has the arrange function
so what does this arrange function do so
when then we would want to use arrange
to sort the data so I would want to sort
the data but what data would I sort so I
will use sample n which will give me
some portion of the data or some sample
data now what is that sample data so
here we are using nesting that is
earlier when we did a sample we just
said data and how many random samples we
want but instead of giving that what we
are going to do is we are going to use
filter here here now this filter will
work on
DF so filtering will happen based on the
mileage which is greater than 20 I will
say size is five and I would want to
basically arrange this in a descending
order so I'm using the desk on this
particular mileage column by default it
is always ascending so let's get the
result out of this which will basically
show me the mileage details in a
descending order so this is my data
frame and now we can look at the result
what we have created so just do a view
or do a head and look at the view so
here you see mileage where the highest
value is on the
top and we were only interested in five
values in a random sample so that's why
when you did a view it shows your five
values and it shows was in a descending
order based on mileage so we have not
only used an inbuilt function we have
not only arranged the data that is we
have sorted the data but we have sorted
the data based on a descending order on
a particular column we have said the
value should be greater than 20 and we
have also said we just need five random
samples now let's look at some other
examples so you can always do a multi
assignment so I can say filter
wherein I'm going to use DF which was
assigned empt cars I'm going to say
mileage should be greater than 20 then I
say B which is going to get a sample out
of a and I just want five random values
so let's look at that so we have B which
is going to get a set of five values
from a now I will create a result
variable which which will arrange B
which is sample data in a descending
order now let's look at the result of
this and that basically shows me what we
were seeing earlier so you can do a
multi- assignment where you can create a
variable get a sample out of it and then
basically whatever is that result you
can arrange that or sort that in a
descending or by default descending
order so same thing we can do it using
pipe operator so piping so here here I
will say result I'm passing in my DF
that's the data set I'm using piping and
which basically tells what you need to
do on this particular data set so I'm
going to filter out the data based on
mileage 50 sorry mileage 20 then I'm
going to push that or forward it to get
the random sample and whatever is this
random sample is going to be pushed so
you are arranging this in a descending
order so this is one more way of doing
it and then basically you can look at
the result so these are some simple
examples where you can use your deer
with multiple assignments or using your
nesting to filter out the data you can
also do a range which is to sort the
data you can get some random samples out
of it you can summarize the data you can
also summarize the data based on one or
two or multiple columns and you can use
some inbuilt functions to summarize the
database Bas on some functions which are
applied on the variables or on the
columns you can transmute it where you
would be interested in only looking at
one column you can mutate it where you
want to add a new column you can slice
it and you can give the conditions where
you can say and and or to filter out the
data so what we can also do is on this
particular data set which we have say
for example DF where I have my data
let's look at this one and if I just do
a DF at this point it shows me my data
set and if you would be interested only
in particular column then your deer also
allows you to either we can do a filter
or we can simply do a select now for
selecting we can
choose uh our data so for example I'll
say DF underscore I'm interested in
mileage I'm interested in Horsepower
might be I'm interested in your
cylinders in this and for this one what
I can do is when I would want to do a
select I can basically say selected DF
let's call it some name I can say
control shift M which is for piping and
then basically what you can do is you
can do a select and you can choose your
columns so I was interested in mileage I
was interested in
Horsepower I was interested in cylinder
and here what I'm doing is I'm using a
select where I can look at the new data
frame so let's do this and uh I'm sorry
here we will have to give it DF this is
where you are passing in your data yeah
now this one is done and we can look at
the value of this one by just doing a DF
or head on
DF underscore mileage horsepower
cylinder and look at the selected result
so you can be looking at selective
columns I could have done this filter
but filter will always look for a
condition say your mileage is greater
than 20 or might be your cylinders are
more than four or something else but
when you do a select you are selecting
specific columns so view always gives
you all the columns head gives you
highlight but then select can be useful
when we are interested in looking at
only specific data so this is how you
can use deer for manipulation for your
data transformation for basically
filtering out the data by selecting
particular data and then working on it
so similarly there is one more package
called tidd and we'll see how we can use
data manipulation done using your tidd
package Let's uh learn about the tidd r
package which makes it easy to tidy your
data and this basically helps you
creating a more cleaner data so which is
easy to visualize and model now this
comes with the mainly four functions so
you have gather which makes your data
wide or it makes wide data longer so
that is basically used to stack up
multiple columns you have spread
function which makes long data wider
that is stacking the data together or
stack if you would want to unstack the
data to data and you are talking about
data which has same attributes and then
your spread can spread the data across
multiple columns you have separate which
is function which splits single column
into multiple columns and to complement
that you have one more function which is
unite and that combines multiple columns
into single columns so these are four
main functions which are used in your
tidd package so let's look how we work
with this so let me bring up my R Studio
here now for this first is let me just
clean up my screen here doing a control
l so I will install the package it is
already installed but we can just do a
control enter and then I can say do you
want to restart or prior to reinstall to
install I'll say
okay and it is basically going to get
the
package now it says package t t y r that
is TDR has been successfully unpacked
let's use that package with using our
library
function and that was built under R
version 3.6 now I can basically start
using these functions so for example
here we are creating a data frame so
let's say n is
10 and then we basically would say we
will call it white now that's the
variable name I'm using the data. frame
function I'm saying ID which will be 1
to n so that will take the values from 1
to 10 and then these are the values
which
have 10 entries so this is a vector
Phase 1 Phase 2 phase three let's create
a data frame out of it now that's done
we can have a look at our data frame by
just doing a view wide and that shows me
the ID column and it has ph. 1 pH do2
and ph. 3 now we can use our function so
for example we can work with gather that
is reshaping the data from wide format
to Long format and basically you can say
stacking up multiple columns so let's
see how we do that here I'll call it
long I'm working on wide I'm using the
piping functionality and then I'm using
gather so this one I will say what will
be the data which I will use so we are
using wide as a data frame then I'm
saying response time so that will be
basically one more column and then you
have your columns which you would want
to basically stack so I'm saying from
phase one to phase three so let's do
this and and once this is done let's
have a look at our variable long so this
one shows me that I have an ID column I
have the response time column and I have
the face column which we mentioned and
that basically has all the values
stacked in so you have ph. one pH do2
and pH do3 so have all the columns are
being stacked here so all my data so now
I have totally 30 entries in this one so
this is basically using your gather
function now sometimes we may want to
use a separate function now separate
function is basically splitting a single
column into multiple columns so which we
would want to use when multiple
variables are captured in a single
variable column okay so let's look at an
example of this one so let's say long
separate that's what we will call we
will work on this long which has all the
data stacked in as the columns we
selected then I'm saying separate I want
the face column and then I would say
when I separate the columns what are my
column names now I could also give a
separator by giving a comma and then
mentioning the separator if that is
required so let's do this now once this
is done let's have a look at our long
separate so what we see here is the
column which we used so we were doing a
face column and that was to be split and
we wanted to split it into Target and
number so that's what we see here so you
have face being split into Target and
number and then you have the response
time so this is how you use the separate
function now there is also something
called as unite function which is
basically a complimenting of separate
function so it takes multiple columns
and combines the elements to a single
column so for example here we will call
it long Unite and we will take long
separate which was separating the data
we want to unite so we will take phas
Target number and we want to have a
separator between them so let's
basically do this and now let's look at
the result of this unite so you see you
have the face and Target merge together
so you have face. one the separator is
dot as we have mentioned and we have
United multiple columns so this this is
one more function of your tid R which
helps you basically uh tidy up your data
or put it in a particular way now then
you have your spread function and this
is basically for unstacking so that is
if you have if you would want to convert
a stack to data or if you would want to
unstack the data which is of same
attributes spread can be used so that
you can spread the data across multiple
columns so it will take two columns say
key and value and spread it into
multiple columns so it makes long data
wider so we can look at this one we will
say long unite I'm using the piping I
will use the spread function I'll work
on the face column and response time and
let's do this and then let's do a view
on this so it tells me our data is back
in the shape as it was in the beginning
so these are four functions which are
very helpful when we work with tidd
package so let's learn about
visualization and here we will learn
about R which can be used for your
visualization now one thing which we
need to understand is because of our
ability to see patterns which is highly
developed we can understand the data
better if we can visualize it so the
efficient way or effective way to
understand what is in our data or what
we have understood in our data we should
or we can use graphical displays that is
your data visualization so there are
actually two types of data
visualizations so you have exploratory
data visualization which helps us to
understand the data and then you have
explanatory visualization which helps us
to share our understanding with others
so when you talk about r r provides
various tools and packages to create
data
visualizations and which can be used for
both kind of data analysis or both kind
of visualizations so when you talk about
exploratory data visualization the key
is to keep all the potentially relevant
details together now the objective when
we talk about explor data analysis is to
help you see what is in your data and
the main question is how much details
can we interpret now when you talk about
different functions which we see here
such as plot which is more for a generic
uh plotting you have bar plot which is
used to plot data using rectangular bars
or you can say creating bar charts you
have histogram or hist function to
create histograms where you look at the
frequency
of uh the data or basically used to look
at the central tendency of the data you
have box plot which is used to represent
data in the form of quartiles you have
ggplot which is a package which enables
the user to create sophisticated
visualizations with the little code
using the grammar of graphics and then
you have plotly or plot ly it creates
interactive web- based graphs via the
open-source JavaScript graphing library
now before we see some examples here
let's also talk about when you talk
about plotting let's also try to
understand what kind of plots you can
have and what kind of techniques you
have so let me open up my R Studio here
now for example I can pull out a
particular data set and let's look at
this one so here I can look at
all the panes and that shows me the
information now what I can do is I can
install and get the inbuilt data sets
and then I can simply do a plot wherein
I am doing a plot on Chi weight data set
so let's see what does that show it
summarizes the relationship between four
variables in chick weight data frame
which is in ours built-in data set
package now from these plots we can see
for example weight varies systematically
over time you can also see that chicks
were assigned to four different diets
now when we talk about explanatory data
analysis or visualization that shows
others what we found in the data this
means we need to make some editorial
decisions what features we would want to
highlight for
emphasis what features are distracting
or confusing and you want them to be
eliminated right so there are different
ways of doing it now when you talk about
your graphics or visualizations you have
I would say three different types or you
can say four so you have the base
Graphics which is easiest to learn now
here we are having an example of Base
Graphics where I can use the base
Graphics I can get a uh data set using
library then I can simply create using
plot function function to a generate a
simple scatter plot of calories with
sugar from us serial data frame in the
mass package and then I can give it a
title so this is basically a simple
example of Base Graphics now you also
have what we call as grid Graphics which
is powerful set of modules for building
other tools now you also have latest
Graphics which is general purpose system
based on grid graphics and then you have
your your GG plot 2 which implements
grammar of graphics and is based on grid
Graphics so you have different ways now
here since I already have used library
and I have the data set I can just do a
x so I can assign the sugar related
values to X and calories related value
to Y then I can use one more which is
Library function and calling in Grid now
I can basically use functions such as
push view Port if I would want to create
a plot using your grid Graphics to
create the similar kind of plot which we
created using base Graphics but this
will give you much more power than base
Graphics it will have uh a steep
learning curve but it is usually useful
so I can do this where I'm saying push
view Port then I can basically say I
would want to have a data view Port I
would say different functions of your
grid package so I'm saying rectangle you
have x axis y AIS given some points here
and then basically you can add details
to the graph by giving the names to the
columns and you can basically create a
simple grid Graphics based plot here now
there are different other options which
we can use to create plots now before we
go into understanding how you create
plots let me just give you a brief on
what the different kind of plots and how
they can be used so here we will look at
these different plots now for example we
have a bar chart which is a graph which
shows comparisons across discrete
categories so you have x axis which will
show the categories being compared and Y
AIS which represents a measured value
and height of the bars are proportional
to measured values now to create
different kind of charts you can use
ggplot which is a package for creating
graphs in R it is basically method of
thinking about and decomposing complex
graphs into logical subunits and that is
a part of Tidy worse ecosystem so it
takes each component of graph axises you
can give scales you can give colors you
can give the objects and you can build
graphs on particular data you can modify
each of those components in a way that's
more flexible and user friendly you can
if you are not providing details for the
components then ggplot will use sensible
defaults and this basically makes it a
powerful and a flexible tool now here
are different options when you use your
GG plot such as you can use geom or what
we call as geometry objects to form the
basis of different type of graphs for
bar charts you have for line graphs you
have Scatter Plots that is underscore
point you have underscore box plot for
box plots you have quartile for
continuous X Vin for richer display of
distribution and Jitter for small data
so here is some simple example where I
would not go into too many details here
but you can just have a look at this one
where we are using Library function to
get the GG plot 2 package then basically
we would want to look into the mileage
data we would want to look at the
structure of it and then we can
basically get the Tidy worse package
finally we can create a bar chart using
gomore bar and we can basically also
mention what would be in xaxis now you
can also give different colors to
basically add more meaning to your data
you could also go for stacked bar charts
so here we are actually telling GG plot
to map the data in the the drive column
to fill the aesthetic so here I am
giving aesthetic access class and I'm
saying what is the data we need to have
and then we are using gomore bar so you
can also have dodged bar in your GG plot
that is not bar charts which are stacked
but next to each other and you can
create that by using your position as
position _ Dodge okay now you can
obviously use your different packages
which are inbuilt and you can create
your bar charts and you have other kind
of graphs such as line graph which is
basically a type of graph that displays
information as a series of data points
connected by straight line segments such
as this one and for this one we are
using if you see gomore line now you can
also create a scatter plot which is a
two-dimensional data data visualization
that uses points to graph the values of
two different variables one in on x axis
one on Y axis like what we saw in base
Graphics example and they are mainly
used if you would want to assess the
relationship or lack of relationship
between two variables and you also have
histogram which I mentioned is mainly to
look at the distribution of a data to
look at the central tendency of the data
basically looking at
your um large amount of data or for a
single variable you would be interested
in saying where is more data found in
terms of frequency where is lesser data
found in the graph how close the data is
towards its uh Mido or what we call as
mean median mode so you can use
histogram where you can categorize the
data in what we call as bins so these
are some Basics on different kind of
graphs now we can look at some examples
and see how that works so what we were
seeing is some quick examples of Base
Graphics or grid Graphics now here let's
do an example of pie chart for different
products and units sold so you want to
create a graph for this first let's
create a vector and pass in the value
here now I can also create labels which
I would want to assign to these values
and then basically I can plot the chart
by saying Pi so that's the kind of chart
which I would want to create and I would
say the data would be X and labels so
let's do this and that shows me a simple
piie chart now I can also give main
details here so instead of just doing a
pi x comma labels I can say what is the
main and then what kind of coloring it
should follow so this is the way you can
create a
simple uh plot now I can also find out
what is the percentage
and then basically I would be interested
in plotting the pie chart which takes X
which takes the labels which will be the
percentage which we are calculating here
by doing a round function and then you
can basically give details to your graph
you can say what color it follows you
can basically look at the legend where
it needs to be in your
chart what are the values and then
basically fill up the colors so let's
run this one and that shows me the
percentage which was calculated and it
gives me the details and we can always
have a look at our plot now if you would
want to go for a 3D pie chart then you
can get the package which is plot Trix
let's use that by calling in the library
function let's pass in some data to X
and let's give some values or labels
which will make more meaning to the data
and then let's plot the 3D graph so I'm
saying Pi 3D here where I'm using X and
labels then I'm basically doing an
explode which will basically control how
your graph looks like and basically give
the values so it also takes the title
when you say Main and by chart of
countries now let's create data for
graph so again we are having a variable
here we are create using the function
creating a vector and then let's create
a histogram for this one where I would
say xlb what would be your data around
x-axis what is the color what is the
border and here I'm creating a simple
histogram which as I discussed earlier
will always show your values on the x
axis and y AIS is more of frequency and
then you can look at the set of values
and what is their frequency
and we can basically use this histogram
for exploratory data analysis look at
the data try to understand what is the
central tendency of your data values now
we can also give some limits by using
the X Lim and Y Lim and then I can also
specify what is the limit so we have
given some values here wherein we have
said your X limit is 0 to 40 and Y limit
is 0 to 5 now if you compare this with
the previous one which we had created
this one based on the frequency had
taken the limits but we can assign
limits explicitly by giving this and
then create a histogram which makes more
meaning now let's take another data set
that is air quality let's view this to
see what does that data contain so you
have ozone solar wind temperature month
and the day so this is the kind of
information we have in the air quality
now let's use the plot function to draw
a scatter plot where as I mentioned you
would be interested in analyzing
variables and see what is the
relationship between them so to plot a
graph between ozone and wind values so
we will say plot we will say the data
which is air quality from that I would
be interested in the ozone column or
ozone field and the wind field I can
create a plot based on this now I can
also be saying what should be the color
what is the type of the data which you
would want to create and you can look at
the in information so you can create a
histogram you can create a a scatter
plot to basically understand the data
better and then infer some information
from that data so let's take the air
quality data set itself without
specifying any particular column and you
can create a plot which shows me all the
different values which you have in the
data and it basically shows you the
difference this is more of an example
like what we did for chick weight where
we did a base Graphics now you can
assign labels to the plot so that is
when you are creating a plot you can say
air quality you will say ozone and then
that's your ozone concentration you have
your ylab which is the number of
instances you have what is the title o
levels in New York City what is the
color so these are the details what we
have given with our plot function and
let's look at the data so it just tells
me that this is the ozone
concentration uh the number of instances
what you have and your looking at the
data now we could also create a
histogram by picking up a particular
column that is such as solar from your
air quality and that basically shows me
the frequency of solar value
and we can then try to find out what is
the mid what is the mean what is the
standard deviation and so on you can
also look at your histogram and try to
understand if it is left skewed and
right skewed so we can do that now here
let's get the temperature out from this
particular data set let's create a
histogram on temperature and that
basically shows me the frequency of the
temperature values and what values have
the most frequency or most occurrence
now you can create a
histogram with
labels so let's do that with the limit
and then let's also use text to
basically given the values which also
takes the values and for each set of
frequency or each set of values it gives
me the labels now you can have a
histogram with non-uniform width so you
could do that by doing a hist function
and then passing in your temperature you
can say what will be the main what is
the title what will be your
xlab it will tell you a limit around
x-axis what is the color what is the
Border what are the brakes you would
want to have for your bars and you can
simply create a histogram using this so
This basically takes the brakes which we
have given such as 55 to 60 60 to 70 70
to 75 and so on so this is basically
creating a histogram with non-uniform
width and it purely depends on the kind
of values what you have now you can also
create a box plot which sometimes helps
us in understanding the the data
quartiles also understanding our
outliers so you can create multiple box
plots based on the data from air quality
so we'll select all the data and then
we'll do some slicing on the data so
let's create a box plot which tells me
the values and if you look at these
points here like single dots these are
basically your outliers we can learn
about that more in later sections so you
can use your GG plot to library to
analyze a particular data set so for
that we will first use the install.
packages and get ggplot
2 so it says do you want to restart r
and I can say yes so let it get the
package I think the package was already
there and now let's look at using ggplot
2 so for that I have the library
function and let's do a attach where I'm
getting the data set which is empty cars
now then I will create a variable P1 I
will use GG plot I will pass in my data
I'll give the Aesthetics what is the
columns which you would be interested in
and then you using gomore boox plot to
basically create a plot which gives me
the box plot for the values here and
this is based on the cylinders which is
there in your data so we can always look
at what does our data contain and what
kind of values or features are available
in the data now let's create a box plot
we will also use the coordinate function
and that basically gives me based on the
data so I've changed the coordinates now
if you look at the previous one where we
created a plot we had mileage on the Y
AIS and cylinders on the
xaxis now I did a coordinate flip and
that's like your transpose function so
you have created the box plot but you
have just flip the coordinates you can
create a box plot and then say fill
which is the factor of cylinder so that
can be used to fill up the values in
your box plot now what we can also do is
we can create factors so we have learned
about factors earlier which is usually
used to work on categorical variables so
here let's create a factor which is
empty cars gear you have a you have
cylinder and if you look at the factors
which we have created we have passed our
data
what is the field or the column we are
interested in what is the level of
values there and what are the labels for
those values right so we have learned
about factors you can always look into
the previous section and learn more
about factors now let's create a scatter
plot by using the GG plot function again
we will use the data as empty cars I
will go for mapping option and then I
will give my Aesthetics that is what
would would be x what would be your Y
and you also would want to use what kind
of function you are using so let's go
for gome point point and that basically
helps me in creating a scatter plot now
you can create a scatter plot by factors
so here we will say GG plot so notice in
all of these cases depending on the kind
of data you have depending on the kind
of plot you are interested in you will
use the GG plot and then basically a
function with that or the inbuilt
package so here I'm saying data is empty
cars I'm going for mapping which
basically will take the values for your
X and Y what is the color and the
coloring will be done based on the
factor values now if you remember
factors will obviously have some levels
and uh those levels will basically help
you in differentiating between your
categorical variables so I'm saying as
do Factor on cylinder and then I'm using
gome point to basically create this
scatter plot so let's do this and I can
look at the values of this one so it
says must be there is an error which
says must at least one color from the
Hue pallet so let's look at that one so
the error which we were facing when we
gave color as the factor values was
because when you look at these factors
which were created with some label if we
look at the values of these it tells me
there are any values in that particular
column similarly your gar or similarly
you can completely look at the complete
data set it tells me cylinder you have
am you have gear now these have some we
have created some labels but these have
na values so what we can do is we can
create a scatter plot as we did earlier
by giving the Aesthetics and that's a
simple scatter plot wherein I'm also
using Gom point so that I can have these
points by defaults or with defaults you
can also give a color specific basically
if you would want to have different kind
of data in the same plot or I can create
Scatter Plots by different sizes by
giving a size or I can give a color and
size and that's again one way in which
you can create your Scatter Plots now
let's also see how you can visual
visualize one more data set which is
MPG so I can also do it in this way
where I said GG plot two and then pass
and look at the data set what we have
here you can just do a view on this to
see what my data contains if the fields
have any na values if that's going to
affect your plotting so now what we can
do is we can create a bar plot or a bar
chart so I'm saying GG plot the data
would be as we have given in previous
lines that is ggplot 2 MPG then I will
say what should be in my Aesthetics and
what kind of chart are you going to
create so I'm saying gomore bar so
that's my bar chart and that has
basically your class and count now you
can create a stacked bar chart where
your information is stacked in the same
bars and we are still using the same
data we are going for Aesthetics which
which is class and then when you say
goome bar which creates your stack bar
we will use fill which is drive and we
can always go back and look at our data
for example you can always look into
this so you have the drive column here
and you are also working on this
complete data set so let's go ahead and
create a stacked bar chart and that
basically gives me the information where
you have the drive information which is
stacked here here now you can do a Dodge
by giving the position as Dodge so we
are still going to go for a stack chart
but this time the bars will be next to
each other and that can also be done
which is very useful you can use this by
using geom point where you're mapping
and you're specifying what are your
Aesthetics so we were creating a scatter
plot now you can also use or give more
details where you can say color can be
based on the class and we have different
classes and based on that my points have
been colored now you can also use a plot
ly or plotly Library so let's install
this one I will say yes for example let
it basically restart so that all my
packages are
updated then I can access that package
using Library function and then create a
variable to which you are assigning your
plot unor ly plot so data is empty cars
what will be your X AIS what will be
your y AIS and details on your marker
which we have given wherein I will give
a list which is size color which is a
combination and then you have your line
what kind of color it will have and what
will be the width so this is where I'm
going to use plot
ly and let's look at this
plot so it B basically gives me some
information now we see some warnings
which are getting generated but there is
you don't need to worry about that so
you can look at the packages what you
have and what options you're using so
similarly we can create one more plot
using plot ly and look at the values of
those so that's a plot with a trend
which explains me about my data so this
is a simple small tutorial on
understanding or uh how you can have
your graphics or
visualization used to understand your
data obviously there are much more
examples much more ways in which you can
pass into your plot functions or your GG
plot and the inbuilt packages which are
available in R for your visualization
now that could be for explorat data
analysis or explanat data analysis so
try these graphs and see if you can
change these options and try or create
new
visualizations good morning and good
evening everyone so welcome to this
session where we will learn on time
series analysis using our programming
language so this is basically a mini
project where we will look at time
series data and how we can analyze it
visualize it to basically find some
important information or gather insight
sites from the data now when you talk
about time series analysis time series
is basically any data set where your
values are measured at different points
in time so when you talk about time
series data data is usually uniformly
spaced at a specific frequency for
example hourly weather measurements you
have daily counts of website visits
monthly sales total and so on so when
you talk about Time series that can also
be irregularly spaced and sporadic for
example time stamp data in computer
systems event log or history of 911
emergency calls now when we work with
time series data for example here I I'm
taking a energy data set we can see how
techniques such as time based indexing
resampling rolling windows can help us
explore variations in electricity demand
and renewable energy Supply over time
now here we will look at some aspects of
this data set which I'm considering so
there is this is open Power Systems data
set and here is the data set I have we
can look at the data set now this is in
a simple format it has time it basically
has values for consumption and then you
have data for wind and solar and wind
plus solar so in certain cases you have
only the date and the consumption but
then if we scroll down we will also find
data for wind solar wind plus solar and
so on so this is a Time series data set
which we would want to work on sometimes
you may also have the data collected
which just does not have the time but it
may also have timestamp that is it would
have say hour minutes and seconds and
that can also be worked upon so let's
consider this data set and let's work on
this project where we will Analyze This
Time series data set now here we can
work on this time series data we can
basically create some data structures
out of it such as data frames we can do
some time based indexing we can
visualize the data we can look at the
seasonality in the data look at some
frequencies and also do some Trend
detection now when you talk about this
data set it has electricity production
consumption which is reported as daily
totals in gwatt hours and here are The
Columns of the data which I was just
showing you so you have data you have
consumption you have wind you have solar
and wind plus solar so this is the data
we have and we will basically explore
say electricity consumption and
production in Germany which has varied
over time so some of the questions which
we can answer here is when is
electricity consumption typically
highest and lowest how do wind and solar
power production vary with seasons of
the year what are the long-term trends
in electricity consumption solar power
and wind power how do wind and solar
power production compare with
electricity consumption and how has this
ratio changed over time we can also do
wrangling or cleaning of this data or
pre-processing of data and create a data
frame and then we can visualize this now
let's see how do we do that so I will
open up my R studio and let's look at
the data set so here is the data set now
I'm picking it up from my machine you
can also pick it up from GitHub so all
the data sets or similar data sets can
be find in my GitHub repository and here
I can look in the data sets you will
find lot of different data sets here
there are some time series data sets
such as power I can search for power or
you have basically
coal or you have
this
opsd Germany daily data set and there
are many other data sets which you can
work on now to get the documentation on
this project you can also look in my
GitHub repository and you can search for
repositories and then basically you can
look in data science and R and here the
there is a project folder where I given
the documentation sample data set and
also your time series analysis related
document this is also the code which you
can directly Import in your R studio and
you can practice or work on this project
so let's see how does that work so first
thing is we will create a data frame
from this data set now here if you see
I'm using header as true so that it
understands the heading of each column
I'm also giving row. names and I'm
specifying date so there is this date
column in the data set as I showed you
earlier let's look at it again so you
have date consumption wind solar wind
plus solar so you can suggest that date
should become the index column which can
be useful so you can do this now let's
just create this let's look at what does
this data frame contain and here if you
see it shows me some data which has
been now as a part of this data frame
structure it starts with consumption
wind solar wind plus solar and if you
see this one is becoming my index column
so I can always do a head and look at
part of the data frame using head or
tail so look at the first records so
let's see this now that shows me the
head data I can also do a tail and look
at the
ending values so if you closely see here
we have wind solar wind do solar and
that basically has na values so there
are missing values but let's look at the
tail and that tells me that there is
some data available for wind and solar
and wind solar now we can always look in
a tabular format using View and we can
look at the data so this shows me that
there are values in these column we see
na values but if I really scroll down I
can see some values which would be
available for wind and solar and wind
solar so I can just use view now I can
look at the dimensions of this
particular object and that tells me
there are 400 uh
4,384 rows and four columns you can
always look at the structure that is
check the data type of each column which
can be very useful so if if I see here I
don't see the date column because date
column was considered as an index which
can be useful but I also look at my
other columns they are of the num types
so that's the data type for each
attribute or each column here now we
would be interested in looking at this
date column so let's look at the data
type of this date column now if I try to
do this this will show me that this is
null because date as the column does not
exist because we created it as an index
so if I look at row names and then I
search for my data show me the index
column or row. names it tells me these
are the values that's the date column
which we are seeing here now we can
access a specific row by just doing a my
data and give the index value or row
name value so let's look at that and
that shows me based on this index you're
looking at the the value you can
obviously search for a different date
something like this you can also pass in
a vector and you can give range of
values so that is
01 2006 to 4 of January and we can look
at this one so it shows me these are the
values so here actually I'm not giving a
range but I'm just selecting multiple
values from row. names now we already
know that R you have a summary function
so you can always do a summary and that
gives you for each column it gives you
minimum first quartile median mean third
quartile and maximum values so we are
looking at consumption we are looking at
wind solar and wind. solar now this is
good but then if I would want to really
visualize the data access the data do
some analysis then it would be good to
take all the The Columns and then we can
later decide to change the data type of
say date column if we want to use it so
earlier I was using date as row. names
or the name of the rows or index what
you call in any other programming
language so here I will just use my data
set and I'll say header is true I'm
calling it my data 2 let's look at the
data and this one shows me five columns
where My First Column is the date
consumption wind solar and so on now
looking at the structure so let's look
at the data type so it tells me that if
now I'm interested in looking at the
date column from my data 2 Data frame it
tells me it is a factor with 4 384
levels and these are the values so it is
not in a datetime format it's a factor
now what we we can do is we can convert
this into a date format how do we do
that so let's have a variable X and I'm
going to use as. date function and I'm
going to pass in my date column so
that's assigned to X now let's look at
the head of X and it shows me the values
we will also see what kind of class it
is and we will look at the structure of
X so class already says it is date type
and look at the structure so it shows me
the format now we have converted this
column or column related value into X
now how do I basically extract values
out of it or make it a part of data
frame so first I will use so all once it
has been converted in date format I will
go for as. numeric and here I will
create a variable called ear and I will
just do a format on X which is basically
of date type and then I'm saying
percentage y so that will get me the ear
component out of this let's look at the
values that shows me ear component now
similarly we can get the month out of
this and then basically look at the
month values we can get the day out of
it and we can get the day component now
if I look at my data 2 which we had
created earlier this basically had date
consumption wind solar wind solar so
what I can do is I can add these
extracted columns such as year month Day
to my data frame using a cbind that is
column bind and I will assign it to my
data 2 again so let's do this and now if
you look at head it shows me date so
that should be date format consumption
now this one might not be date format
but we'll see you have consumption wind
solar and we have extracted the year
month and day which can help us for
group buy we we can do some aggregations
we can do a plotting and we can do
various things by these additional
columns now let's look at first three
rows here so I'll say 1 is to three for
my data two and that shows me some data
here you can always do ahead and look at
the sample of data so that basically
shows me month day your columns and then
you have your date now what we can do is
we would want to visualize this data we
would want to basically understand the
consumption now as I said if we want to
visualize the data say for example I
want this which is consumption of data
over years and this one is in terms of
gigawatts per hour as we were mentioning
here gigawatt hours so if I would want
to create this visual to basically
understand the pattern of the data how
do we do it so we can you create a line
plot of full-time series of Germany's
electricity consumption using the plot
method now how do we do that so here one
of the option is I can straight away use
the plot method I can then say what
would be in my x-axis what would be on
my Y axis what would be the type of
graph I would want to plot what is my
name on x-axis y AIS and this is the
simplest way so I'm saying my data 2 I'm
extracting the year column and here I'm
taking the consumption so let's create a
plot and here if you see we are looking
at a plot we do see some tick times and
we see that the data has been divided
with every two years so from 2006
onwards to
2016 but then really this data does not
give me uh you know a very useful way of
looking at the data or understanding it
might be what I can do is I can use the
same way but I can give apart from xaxis
and Y AIS I can say the limits that is X
limit is 2006 to 2018 and Y limit is
from 800 to 1700 so we can do this and
let's look at this again this is a plot
but it really does not help me in
visualizing and understanding the data
so what are the better options I can go
for multiple plots in window as of now
we are just sticking to one plot in
window so if you would want to have
multiple plots you can always change the
value here and make it two or three that
will say how many rows and how many
columns so as of now we will just keep
it as it is par MF row now if I would
want to plot I can straight away give
the column name so I am interested in
getting the consumption now I can just
do a plot I'll say my data 2 and I will
choose the second column which is
consumption which we saw here from our
data so consumption was the second
column so I can just do a plot in a
straightaway way without mentioning your
x axis y AIS limits and so on and if you
look at this this one is giving me a
pattern now here I looking at um x-axis
Y axis which is not really named we do
not have a name name to this graph and
we are looking at the data it does show
me some kind of pattern but mightbe we
can make it more meaningful so I can do
it this way where I say my data second
column let's give axis as ear x-axis Y
axis as consumption now that has changed
the x axis and y axis now I can also
give some more details I can say type
should be line I have the line width I'm
saying color is blue and let's do this
so this looks more meaningful might be
shows a wavering pattern of consumption
over years I can also give a limit of x
that is 0 to 2018 and that basically
shows me the range now we can change
that and we can be more specific and
saying X limit should be 2006 to
2018 and let's look at this now this one
once you have given a proper limit it
shows the line graph and it shows what
was the consumption in 2006 and over a
period till
2018 I can then use any of these options
are fine but it depends on what and whom
you are presenting the data or what kind
of analysis you're doing so I can do a
plot I can choose column second xlab
which is x- Axis Y axis type is line
width giving X limit y limit and then
I'm giving a title to this which which
is consumption graph and then basically
you're looking at the line graph now
those are the options which you can do
either you could be very specific or you
could just give your column which you
want to plot or obviously make it more
meaningful by giving all the details now
what we can do is if we would want to
look at this data and understand it
better rather than just looking at a
simple line I can take the log values so
here I'm saying log of my data 2 second
column so I'm taking log values of
consumption and I'm taking the
difference of logs so I can say
difference and then you can basically
increase or decrease this by multiplying
it by some number so rest Remains the
Same I'm changing the color and let's
look at this plot and you see this
basically is giving me a better pattern
which makes meaning here we see the log
values so this is you are using a simple
plot
function in R you can also use ggplot
now for that we can install the ggplot
package it's already there in my machine
so I'll say no I will access this by
using the library ggplot
2 and now I can use GG plot to plot so
the way you specify here you can say my
data 2 that's the data frame I'm saying
type as o and when I'm saying line I am
basically going to use x axis which is
ear Y is consumption and let's look at
this plot so again we are back to the
one which we were doing earlier really
does not make any sense gives us some
data but then really does not give me
enough information I can in my
Aesthetics I can say x as ear Y is
consumption I can do a grouping and then
I can give line and plot so again we
have some some information but really
does not help me right now let's look at
other example so I'm just doing the same
thing here and I'm looking at line type
being tached I'm using the GG plots
other methods such as Gom line and Gom
point to give me more information and if
I look at the plot it does give me data
it tells me what are the different
values it gives me some kind of pattern
but I would still prefer the way we were
doing with plot now we can change the
color and obviously add details to it so
what we see is when you use the plot
method which I did earlier it was
choosing pretty good tick locations that
is every two years and labels the years
for the x-axis which was helpful right
but with these data points which we were
seeing here or say for example this
one or say this one or say this one we
are looking at some data but then that
really is quite crowded and it is hard
to read you can look at the values but
then it really does not give you enough
information so we can go for plot method
but then we will see how we can consider
different data now if I would want to
plot the solar and wind time series so
let's see how do we do that so wind
column is what I'm interested in so
first thing is it was always good to
find out the minimum and the maximum
values in every column so I'm saying
minimum I'm saying let's put in here my
data
2 and then let's look at the values so
we are looking at the columns we know
consumption is the second column wind is
the third column and you have solar is
the fourth and this one is the fifth so
let's say let's find out the minimum of
of each of these columns which we would
want to plot so let's say minimum of
data third column and here I'm also
saying remove the na values because we
do not want to consider the na values so
let's look at the minimum that shows me
5.7 757 what is the maximum value it is
826 so that also helps me in giving a
limit if I want to plot wind on y AIS I
can give a y limit from 5 to 850
consumption wise let's find out the
minimum from second column and maximum
and similarly for solar find the minimum
and maximum and wind plus solar minimum
and maximum so this will be helpful when
you would want to plot multiple graphs
or give some limits so that's fine now
for multiple plots as I said instead of
having one plot let's plot consumption
and wind and solar and try to see a
pattern
so I can say par function and I will say
three rows and one column so now when I
start plotting you will see you will
have multiple plots in one single window
so let's see how we do it so here let's
look at plot one so this one is
consumption as we did earlier and let's
look at the data so that gives me some
data you can always do a zoom and you
can look at the data you you can
basically expand this graph or you can
reduce this graph to see what kind of
pattern we have in consumption similarly
we can basically choose date being
x-axis my consumption being Y axis right
so this is being more specific because
here we have a range but it really does
not give me enough information so I will
basically give xaxis Y axis I will give
the name that is daily totals and then I
will basically give consumption color
and Y limit based on my minimum and
maximum limits so let's do this and now
we can look at the data here so let's
see this data makes a little more
meaning because we are looking at the
dates and let me do a zoom so it shows
me all the dates it shows me the data
points it shows me how the data pattern
is changing for consumption now this is
for consumption so what we can do is we
can also extract specific data so if you
see here I have done some testing where
I saying okay I would want to get a date
specifically I would want to extract
some values so we are looking at the
date column but if you remember we did
not change the data type we just change
the data type of date column we
extracted year mon month out of it it
would be good if we can convert a column
into date time format and put that in
our data frame now let's look at the
plot two this is mainly for your uh
column which should be consumption and
wind and solar so here I see it is solar
data and I can plot this one to see how
it looks like and that tells me from
2006 onwards we have some pattern I can
be more specific where I say I would be
giving date and then the column for
solar xaxis y AIS what is the type what
is the Y limit and what is the color it
is always good to specify your X and Y
AIS given name rather than let it
automatically pick up now this makes
more meaning because it shows me some
dates
similarly we can do for wind so either
you do it just by giving the
column or you give your X and Y AIS so
let's look at this one and this shows me
the data so we can choose plot three
this one we can choose plot two we can
choose plot one and we can put all that
data in one graph so that's when you are
putting in multi plots in one particular
graph you can always do a zoom you can
always look at the data right and this
is usually useful to look at the pattern
what kind of pattern we see what data we
have and so on now moving forward so we
have seen how you are creating these
plots all in one window let me reset
this back to one plot per window and
let's basically plot time series in a
single year so what we have seen is that
when you look at the plot method it was
quite crowded then we looked at solar
and wind and if you compare that you
will see your consumption pattern your
solar pattern your wind pattern and
basically we can see from this
particular data some kind of pattern so
electricity consumption is highest in
the
winter where we will see what is the
consumption is it highest in Winter or
is it in summer we can see that by
breaking a year further into months we
can see that but we see a pattern which
goes for every year or every two years
being highest at a particular point of
time and then it drops down so
electricity consumption is highest in
Winter and that might be due to
electrical Heating and increased
lighting usage and lowest in summer now
when you look at electricity consumption
appears to split into two clusters we
can always look at the consumption one
with oscillation centered roundly around
1,400 gaw
so you can always look at, 1400 gaws and
you see all the values here which are in
that particular consumption another with
fewer and more scattered data points
Cally roughed around 150 so if you
really expand this you can see you will
have lot of data points at this point
now we might guess that these clusters
correspond with week days and weekends
which we can see if you break that data
into yearly monthly weekly and so on now
if you look at solar production that is
highest in summer when sunlight is most
abundant and lowest in winter so
obviously when you're are making or
gathering some insights when you're
looking at the data you are also using
your domain knowledge your business
knowledge you are you know knowledge of
business to understand how this goes if
you look at wind power production that's
again highest in Winters and drops down
in summer so due to stronger winds and
more frequent storms and lowest in
summer so there is some kind of
increasing Trend in wind power
production over years which we can see
here over the years and all the time
series data what we are looking at is
referring or showing us some kind of
seasonality that is we are looking at
seasonality in which a pattern is
repeating again and again at regular
times at regular intervals so if you
look at consumption solar and wind time
series that oscillates between high and
low values on a yearly time scale which
we can break down and see I'll show you
that it corresponds with the seasonal
changes in weather over the year so
seasonality does not have to correspond
with meteorological reasons for example
if you look at retail stale sales data
uh that will show you yearly seasonality
with increased sales in particular
months so seasonality when we say can
occur on other time scales so the plots
what we are seeing here they are fine
but if you look at those plots they
might show some kind of weekly
seasonality also so in your consumption
corresponding to weekday and weekend so
let's plot for one single year now how
do I do that so first is I will look at
my data
two that shows me the structure it shows
me date which is Factor other columns
which are all numerics now like we did
earlier I'll repeat this step where I'm
going to convert the date column into
date type look at head of it look at
class of it look at the structure of it
right and then what I want to do is I
want to add
this as to my data frame so I will
create a variable called mod data and
this one will have as data and I'm
formatting the value of x which is date
time into month day and year so let's do
that and now you look at the mod data
which I created like modified data so
this is the format I have it is in date
type if you carefully see here and then
I can look at the head of it so it says
me mod data now we are what we did here
is when I said my data
3 so my data see we did a cbind and I
did a mod data which is going to add
this column to my other Columns of my
data 2 so my new data frame is my data 3
let's look at the structure of it and
you see there is this date column I can
delete it I can remove it I can let it
be right so that depends on our choice
might be we want to once our analysis
done we want to remove the mod data
right so we can keep both of them now
let's let's basically extract data for a
particular year now how do you do that
so this is some wrangling so I will say
my data 4 let's call it my data 4 and I
will use subset function so subset will
work on my data 3 that's the data and
what I'll do is I will do a subset how
do how is the subset found so I'll say
take the mod data column the value
should be greater than or equal to 2017
and should be less than 2017 December
31st so I'm getting data for one year
and I'm storing it as my data
4 let's get the head of it and you see
we are specifically looking at 2017
related data now let's do a plotting of
this where I will only create a plot for
one year so I'm saying my data 4 that's
my
new data what we got
so here I am going to take the First
Column which is mod data I'm going to
take the third column which is
consumption so I'm looking at the date
format for one year consumption values
for it and then rest of the things as we
have done earlier let's look at the plot
and this makes more meaning right so
when you look at this plot it tells me
Jan to Jan it shows me some kind of
pattern where I have divided the year
into months right and it is broken down
into say two months so Jan and March and
May and July and so on but we still see
a pattern and that gives me good
understanding of pattern where I've
broken it down into months so this is
where you have taken time series in a
single year to investigate further and
this is what we see right now we can
clearly see there are some weekly o
oscillations what one more interesting
feature is that at this level of
granularity that is when you're looking
at yearly data there is a drastic
decrease in electricity consumption in
early January and late December during
the holidays or probably we can assume
that this is holidays now I can zoom in
further and look at just Jan and Feb
data let's see how we do that and let's
see how we work by zooming in the data
further so to zoom in the data further
let's see how we do it now here we have
this my data 4 which is basically having
a subset right so let's work on this one
so I will say my data 4 which earlier I
was taking data three I was doing a
subset and I was giving the date but
this time I will make it more narrower
so I'll say my data four I will say sub
set from my data 3 and I will choose mod
data column which we have modified with
the date format I will choose the
starting date as
1701 that is Jan and then let's go till
Feb and let's create
this now let's look at the head of this
so it shows me we have the data which is
Jan and then you you can basically look
at more on this now again as I said
earlier let's find out the minimum of
this from the First Column so that is
basically your mod data so let's look
into this one and that basically will
give me minimum and maximum let's look
at the values so this one tells me Jan
17 January 1 and maximum is
your Feb 28th 2nd month 2017 so we are
actually looking at two months data here
let's look at the Y minimum so this is I
will look at column three now what is
column three consumption so let's look
at the minimum value for consumption
maximum value of consumption let's look
at the values which can be given as our
limits now this is the minimum and
maximum now let's do a plotting for this
data which has been narrowed down for
consumption based on my data so I'm
saying My First Column which is mod data
and then third column which is
consumption I'm giving some naming
convention for sorry namings for your
xaxis y AIS what is my
consumption or what is my title here
what is the color and then you see I'm
using X limit to give the minimum and
maximum limit and Y limit so let's look
at this data and if you look at this
data it is specifically for 2 months and
again I can look at the pattern here
what I can also do is I can add some
grid here so I can basically look at
this data and make more meaning out of
it so it is bi-weekly data you can see
now I can add a line here using AB line
and then I can basically choose what
lines I would want to add
horizontally so that basically allows me
to dissect the data and look at data in
a more meaningful way I can also add
vertical lines so vertical lines is I'm
saying sequence will be minimum maximum
and I'm saying an interval of seven so
let's do this and this basically has
added some lines every week and you can
see at the end of week it is dropping
and then it is starting again it Peaks
somewhere in the mid of the week and
again it drops down so this is you're
looking at your consumption data right
now what we can also do is we can create
create some box plots so when we looked
at zooming in data for Jan and Feb you
can add some data points like this so
consumption is highest on the weekday as
I showed you here and lowest on the
weekends so this is what we are seeing
when we are breaking the data or zooming
it further for couple of months so we
have vertical grid lines and we have
nicely formatted tick labels that is Jan
1st Jan 15 Feb 1st and so on so we can
easily tell which days are weekday and
weekends with use of these grid lines
and basically breaking it down so there
are many other ways to actually
visualize your time series data
depending on what patterns you're trying
to explore you can use Scatter Plots you
can use heat maps you can use histograms
and so on now moving further we would
want to explore the seasonality right so
when you further explore the seasonality
of our data we can use box plots
basically to group the data by different
time periods and display the
distribution for each group now how do
we do
that let's come here and let's see how
box plot works so I can just do a simple
box plot and I can choose my consumption
column and that gives me just the
consumption data but this really does
not give me any meaning I can look at
solar data I can look at the wind data
and we can also see some outliers here
so we can create box plots but if we
would want to do a box plot what is box
plot it is basically a visual display of
your five number summary that is you
want to look at your mean median you
want to look at your 25th percentile 50
percentile or 75th percentile so we can
use a quantile function use the
consumption column and then you
basically give a vector which shows you
five number summary so that's your
quantile
and then let's do a box plot so if you
are looking at quantile it tells me what
is the minimum what is 25th percentile
50 75th and 100 that's from my
consumption column so let's create a box
plot for consumption let's give it a
name as consumption let's give Y axis as
consumption and a limit for y axis now
that's my consumption graph so I can
look at yearly data now that will make
more meaning rather rather than just
looking at the complete consumption data
so how do we do it yearly so we will say
consumption and then I will say the ear
column so it is consumption but grouped
based on ear so here I can give xaxis y
AIS and I can give y limit so let's
create this and this makes more meaning
we can give some coloring scheme here
but now I'm looking at 2006 2007 8 9 and
so on and we can look at the data what
is the range right it gives me 5
percentile or sorry five number summary
of the data per year and it basically
allows me to look at the seasonality of
this similarly we can create box plot by
just giving consumption yearly grouped
and here I'm giving the title as
consumption y axis xaxis and y limit
wherein I can also use lass so this is
one more feature which you can do and
that basically will give me the tick
points if you compare this one to the
previous graph so when I created this
previous graph I had 2006 2008 and I had
from 600 to 18800 and if I go for the
next one I am basically seeing more
useful information now let's look at
monthly data so I would want to group it
based on month months and let's create
that so this gives me the monthly data
where I'm looking at
months and I could select a particular
year or I can just do a grouping based
on months so I can have multiple plots
to see a difference here so let's do
this now let's create a box plot for
consumption which is monthly data and
let's give it a color let's look at the
wind data which is again grouped monthly
and let's look at the solar data which
is grouped monthly now if I zoom in it
basically gives me the seasonality of
the data for your wind for your
consumption for your solar so what we
are doing is we are creating these box
plots which are giving us values now
what I can also do is I could look at
the day wise also but before we look
into this how do I infer some
information from these box plots which
are being created so this is what we
have done where we are looking at the
data for month and these box plots give
me year seasonality which we were seeing
in earlier plots but gives some
additional insights so if I look at the
data here it tells me the electricity
consumption is generally higher in
Winter now this is based on months so we
can see consumption is higher in Winters
and lower in summer so we can obviously
look at our plot we can see where it is
lower where it is higher and then we can
look at the median and lower two
quartiles are lower in December and
January compared to November and
February so that is you look at the
quartiles and you will see that the
median and lower two quartiles are lower
in December and
January here Jan and December so you can
look at from my plot now this is giving
you some idea on
seasonality now that might be due to
business being closed over holidays now
this one we were also seeing when we
looked at time series for 2017 only and
box plot basically confirms that there
is this consistent pattern throughout
the years now when you look at your
solar and wind power production both
will give you a year seasonality what we
are seeing here
and if basically I look at the data so
it depends on what parameters you are
choosing but if you look at solar it
will reflect the effect of occasional
extreme wind speeds associated with
storms and other transient and since we
are grouping it based on months we can
see this pattern is quite evident every
year
now what we can do is we can group the
data day wise so here let me again reset
this to one plot per graph now I'll say
box plot I'll say consumption which is
group based on day now we know that
there is a day column and let's give a y
limit and let's look at the data so this
is where I'm grouping the data day wise
so you look at 31 days and you look at
the box plot so this is where you are
plotting it on a daily basis so you can
look at the data you can break it down
to a particular week so here I have
given day and I have chosen all the 31
days but I can break it down to a week
and I can look at the data so if we look
at the data per week or per day we can
basically infer that electricity
consumption where I'm doing a
consumption group by day is higher on
weekday than on on
weekends so time series with strong
seasonality can often be represented
with models that can decompose signal
into seasonality and long Trend now this
is an easy way now how do we look at the
frequency of the data that could be
interesting to see so let me uh look at
say the yearly
data
which we were seeing
here now let's go further and here we
have looked at data so what we will do
is we look at the frequency now when you
look at the frequency when you talk
about frequency in your data so we have
the modified date column which gives me
a frequency and if we really look into
the data that will tell me that the data
is on a daily basis so for that let's
look at my data 3 see again which gives
me data and you can just see all the
datas data or dates are in sequence so
your 22 23 24 25 26 and so on I can look
at I can access a deer
package that is basically allowing me to
work in a better way now I can look at
the summary of this and for all my
columns I am seeing what is the minimum
five number summary date and consumption
so date does not show me anything
because this is not in a date format it
is just a factor but other things have
the fin number summary so we are looking
at wind plus solar we are looking at
year and month and day and all these
columns now what we will do is we will
want to find out the sum of each column
how many entries does it have and we
will say the value should na value
should not be considered so let's look
at this one so it tells me me for my
particular columns so let me run this
again and that shows me for each column
how many values you have and these
counts do not include the na values now
similarly I can find out specifically
for consumption I can find out is there
any na value so I'm saying is do Na and
let's find out if there is any na value
or Miss value in consumption it says
zero okay that's good if you look in
wind it tells me there are
1,463 entries which are na similarly
solar similarly wind do solar or wind
plus solar so it gives me a count of Na
values that is missing values and also
values which are not missing so to
understand frequency what we can do is
we can find out the minimum on the date
that is the First Column and I'm saying
RM na. RM is true that is get rid of Na
values and find out the minimum and
let's look at the minimum value this is
the minimum from my modified date now if
I would want to get the frequency I can
basically use sequence function so I can
say from X minimum that is the minimum
value I want to look at the frequency
that is d wise and let's just look at
five entries and see if there is a
day-by-day
frequency so let's look at the value of
this and obviously it tells me there is
day wise frequency so that allows me to
look at the frequency look at the type
of it it is an integer class is a date
so similarly we can say from X minimum
we can basically look at the frequency
month-wise
and I can again look at five records so
that shows me monthly data right so I
can extract the data for frequency
similarly yearly data and that's also
very useful now we can select data which
has any values for wind so how do I do
it I would want to find out the wind
column and I want to find out where the
values are na a so I will create a
variable and here I will say my data
three and then I give a conditional
where I say is na in the column so let's
do this now once I've done this once
I've done this I have said that my
selected win data from my data 3 where
we said na values and I will give the
names to this so name should be in my
data 3 I'm interested in mod data
consumption wind and solar so these are
the four columns I'm interested in let's
look at first 10 records here or first
10 rows so that tells me these are the
values where wind has na or missing
values I can always do a view and that
gives me the complete data so it
basically shows me
1,463
entries and here it shows me all na
values so you can look at all the way to
the end and it shows me wind has na
solar does have some value here in the
last row but then also if you see the
numbers
have a difference so you have 1461 and
then you have 2174 so there is a
difference so there is some data in
between where wind has some values so we
have found out na values now what we
will do is we will select data which
does not have any values so I will call
it cell selected wi two I'll again use
my data 3 I will say which but now I'm
saying not na from this column and I
will select the data for the columns so
I'm interested in looking at 10 records
and this shows me not any values so no
more missing values so if I really look
at this data as I saw earlier which has
Na and if I look at these values which
are not na for the wind column so
looking at these two result we will know
that in year 2011 wind
column has some missing values so let's
focus on year 2011 so how do I do that
let's call it a different variable I'll
say my data three I will say here when I
say which where we were saying na here I
will say the year should have a value of
2011 and I want all these
columns let's look at the data here
and this is showing me 2011 but we are
not seeing all the values so there are
some values but then there are some
missing values also for 2011 based on
whatever analysis we have done so let's
look at the class of this it is
basically a data frame do a view and
this one will help me in finding out
where are the na values so if you just
scroll down looking at all the data
let's search if wind column has a na or
a missing value and I will see if there
is any missing value in which column or
which row it is for the wind column so
we have all the values which are
existing I could select and search for
one specific value and I'll show you how
we can do that so here let's scroll all
the way down so it's like you're
exploring your data and seeing
is wind column having na or missing
value for a particular row and let's
scroll here and here you see there is a
missing value for one particular row so
13th December 2011 has wind value 15
December has wind value but your 14th
December does not have right similarly
we can search so there was only one
entry which was missing now that could
be for some reason mightbe it was not
calculated mightbe it was not tabulated
so we have a missing value and that can
affect my plotting that can affect my
analysis so let's look at the number of
rows in this which will tell me how many
rows we have for 2011 so it tells me 365
so that is basically the number of days
in a year now we will find out if there
were na values so we earlier checked
total number of Na values per
column that is in your row number 265 to
269 we can see here 265 to
269 so this is where we were seeing are
there any na values right so let's go
back
here and we want to find out the number
of Na values for a particular year how
do I do it so I can just do a sum I will
say is na now I'm interested in my data
three wind column and I'm saying my ear
has to be 2011 but I'm finding out the
na
values so let's do this and it tells me
one and that's right that's what we saw
when we did a view let's see how many
non na values you have and that is 364
so that
basically satisfies my logic so it's 364
+ 1 missing so there are 365 let's look
at the structure of this it tells me you
have modified date and date format you
have consumption fun wind and solar now
let's create a variable selected wind 4
I will say wind three that is which was
having all my Na and non na values for
2011 I will say let's find out the na
value because I'm interested in finding
out that particular row so I'm saying
find out where the value is na a and I
want all the columns let's look at this
one and this is my specific row which
has a na value now we know that data
follows a day wise frequency which we
have clearly seen now let's select data
which has Na and non na values so let's
say let's call it test one I will use
wind 3 which has Na and non na values
but now I will say I want the modified
date which should be greater than 12 12
2001 now remember we had when we were
doing a view we saw that one particular
day or what we see here 14th of December
there is no date so I will select a
subset of data which includes This na
and non na that is might be I can take
13th of December and 15th of December so
let's start from 12 12 so the date
should be greater than 1212 that means
13th and it should be less than 16 so
that is 15th
and the columns right so now we have
some data let's look at this so I have a
I've selected a subset of data I could
have done this using subset also so I
have Na and non na values now why are we
doing this so sometimes you might have
some data for a particular column and
you may want to find out if there are
any missing values might be you want to
fill them up or replace them with
something so that is usually useful when
you are doing doing a trend detection so
say for example you have data for every
month and might be in one one of the
months you have missed or might be you
have data for every year collected
monthly and then in one of the years for
couple of months you don't have the data
like I can say 2016 I have data for all
12 months 2017 all 12 months 2018 might
be I don't have data for March and June
2019 I don't have data for same months
so I can forward fill or backward fill
them using the previous year's same
month data so we can do that so here I
have test data where I've extracted a
subset of data I can look at the class
of this it is a data frame structure of
this it has the columns now let's use
the library and function and use the
Tidy r package and what we will do is we
will fill it up so I will use test one I
will fill the wind column which has a
missing value now once you do this if
you notice it has done a forward fill so
it has taken the previous value and it
has just filled up that so you can fill
up the data using different directions
such as up and down left and right and
so on so we can take care of missing
values in our frequency data which
allows us to basically
analyze the data in a better way now
here we will want to also look at some
more data so this is to deal with
frequencies of fill column wherein you
can take care of missing values forward
filled so filling values can be done in
different directions as I said and you
may want to First convert your time
series to specified frequency if your
data does not have a frequency but we
had now if you do not have have a
frequency might be you can convert it
into a frequency such as weekly daily
monthly as I showed you and then
basically you can do a forward fill for
the values so for example if I have my
data I can break it down into weekly and
then look at the values and if there are
any values missing for weekly data I can
use a forward fill so that can take care
of my frequency data then let's look at
the trends of the data which is the the
last part of this project so basically
let's look at the trend So when you say
Trend what does that mean so in Time
series data you always have some kind of
trend so that will exhibit some slow
gradual variability in addition to
higher frequency variability such as
seasonality and noise now to visualize
these Trends what we do is we use what
we call as rolling means so we know how
our data is spread over year or month or
day but how about looking at a rolling
average and see what is the difference
so a rolling mean will tend to smooth a
Time series by averaging out the
variations and
frequencies so this can be higher than
the window size so there is something
called as windowing where you can choose
a set of time frame you can also average
out any seasonality on a time scale
equal to window size so this will allow
you to look at lower frequency variation
in the data so when we are looking at
electricity consumption time series we
already saw there is a weekly pattern
there is a yearly seasonality which we
saw using box plots so we can also look
at the rolling means of the time scales
how do we do that so for this you can
use some package like zoo and then you
can basically use a rolling mean using
the zoo package
and you can say what is
the frequency with which you want to
calculate the rolling mean now how do we
do this let's look at this data so here
I'm going to my look at my data 3 which
we have been using so far now let's call
it a three-day test you can give it any
name I'm going to use my data 3 I'm
using the PIP in
function now I will use deer and I will
arrange the data descending in ear now
you can always break it down step by
step and you can see the result of this
so I'm going to arrange this data in
descending order of year so obviously my
last one 2017 or 2018 will be on the top
you want to group the data by year so it
depends on how many years we have we'll
see so you can group the data by year
now this data is then used to basically
mutate so mutate function is going to to
allow me to use this rolling mean so
I'll call it as says 03 day so I'm going
to calculate a rolling mean every 3 days
for my consumption column and basically
let's ungroup this so let's see how this
works sorry yeah let's look at this and
here when I'm doing a 3-day test let's
look at the result of this and then I'll
explain this so if you see here we have
have the test 3-day column now this has
the rolling average now what does that
mean so first value here what we see is
1367 is the average consumption in
2017 from the first date with the data
point on either side of it that is you
can look at this date so
1130 then you look
at you're looking at the value 1367 here
so you look at 1 13 0 1 4 41 1 53 0 if I
take a mean of these so for example if I
would just do this
part and that is giving me mean okay
because I have a comment so let's
basically add anything as comment and
then let's do this so it saves me 1367
that's what we are seeing here right so
you're got getting a rolling average
every 3 days similarly if you want every
five days it takes the five values and
it gets the mid value right so you can
always find out the mean rolling mean
for a particular frequency now let's do
that for 7 days that is weekly data and
yearly data that is 365 days so how do I
do it same logic my data test now I'm
using my data 3 I'm arranging it in a
descending order I'm grouping by year so
when you do a group by year so earlier
when we did a grouping by and when we
looked at at the data it was telling me
how many rows we had right so let's do a
grouping by year and let's say test 07
so that's a rolling average every 7 days
and I'm also saying take care of the na
values similarly I'm getting rolling
average every 365 days might be you can
do quarterly might be you can do half
yearly and let's do this so let's create
this my data test and let's look at the
result of this so I will use my data
test I will say arrange based on
modified date now we know there is a
column called modified date I want to
just look at 2017 data so I'm doing a
filter right and then I will choose what
are the columns I'm interested in so I
will look at the S and 365 day and let's
look at say first seven records so let's
do this and that basically gives me the
consumption value modified date year and
my rolling 7day average or 7day mean
which is for first 7 days and then 365
you will not see the data here but if I
do a view on this I can basically see
the values so you can always select a
particular column to see the
values these are the values for every 7
Day rolling average this is for 365 days
every 365 days so you see all the values
are missing but every 365 end you will
have basically some data now let's do a
plotting of this and basically visualize
this data which we are seeing rolling
average so let me first do a plotting
one plot per graph and let's do a
plotting I will take consumption
data xaxis y AIS color and give a title
to this so let's create this and that's
my consumption data which is spread over
a period of time and that's fair enough
but now let's add some more plot to this
so I will add the sday rolling average
to this so for second plot to be added
in the same one in R you can use points
so I will say points I will choose seven
data column type is line width x limit y
limit and color so let's do this and
that's my pattern 7day rolling X average
which basically gives me some kind of
trend similarly I can add one more here
and this time I will choose the 365 day
and look at the pattern lines so now you
see some Dots here well you could do it
in a different way so I can just add
Legend to this and I can say Legend will
be where in xaxis and Y AIS so I'm
saying it will be
2,500 and Y is 1,800 so my Legend will
come in somewhere here I'm saying my
Legend will have consumption test and
this one I can give some names I can
give what is the color I can say what
kind of Legend it explains what is for
each color and then basically a vector
so let's add a legend to this and I've
added a legend now you can do a zoom and
look at the
data and
here I see that uh my x- Axis is fine
but Y axis is going a little about of my
plotting area so I can actually change
that so here I have 1,800 how about
making it
1,600 and let's look at this one so we
can
basically uh go for this one and start
again here plot and points and line and
then add a legend right and you can
basically place your Legend anywhere in
the plot so this basically is giving me
me the trend what I'm looking at my
rolling average so similarly you can
look at the trend for wind and solar
data so what we are seeing here is when
you look at Trend this is one more way
of looking at it you can always create
plots in different ways so 7-Day rolling
mean has smoothed out all week
seasonality which we were seeing Here in
My Graph where you look at every 7 Day
preserving the yearly seasonality so
7day will tell that electricity
consumption is typically higher in
Winter and lower in summer so better is
you break it down uh yearly so here if
you look at every year you can see when
is winter when is summer what is the
seasonality what you're Trend what
you're seeing here and if there is a
decrease or increase uh for few weeks
every winter so similarly if you look at
365 now as you said as I said rolling
average basically uh reduces the
variation so if I look at 365 rolling
mean we can see longterm Trend in
electricity consumption is pretty flat
now that's what we are seeing it's kind
of pretty flat there is not much
variation over years if you really join
these dots so we can can basically see
some highs and lows and that gives me a
trend now this is how you can do a trend
detection and similarly we can do
plotting for wind and solar so this is a
small project which I demonstrated using
R now all this code which you have here
in the form of a project. r file you can
find here in my GitHub page this is the
document which explains some things feel
free to download this and you can add
details to it this is the samp example
data set which you can also find in my
repository in the data sets folder so
continue learning and continue
practicing are want to be a certified
data expert then here we have
postgraduate program in data analytics
by simply learn take a glance at the
notable features and skills offered in
this course you will benefit from
exclusive IBM hackathons and ask me
anything sessions eight times more
interaction in live online classes with
industry experts capston projects across
three domains and over 14 data analytics
projects using real data sets from
Google Play Store Lyft World Bank and
more master classes from puu faulty and
IBM experts along with simply learns job
assist for better job prospects acquire
skills in data analytics statistical
analysis using Excel data analysis using
Python and R and data visualization
using table and powerbi and enroll Now
using the course Link in the description
box Excel is a really powerful tool for
data analytics and Reporting and pivot
tables are one of the features that
Excel offers for creating tabular
reports to summarize our data let's
begin by understanding what is a pivot
table a pivote table is a tool that
summarizes and reorganizes selected
columns and rows of data in a
spreadsheet to obtain a desired
report it does not actually change the
spreadsheet data it simply pivots or
turns the data to view it in different
perspectives P tables are espcially
useful with large amounts of data that
would be timec consuming to calculate
manually now let's understand the
different components of a pivot table so
there are four main components first we
have rows when a field is chosen for the
row area it populates as the first
column in the pivot table similar to The
Columns all row labels are unique values
and duplicates are removed columns
is the second component when a field is
chosen for the column area only the
unique values of the field are listed
across the
top then we have values each value is
kept in a pivot table cell and displays
the summarized information the most
common values are sum average minimum
and maximum finally we have filters
filters apply a calculation or
restriction to the entire table so let's
jump over to Microsoft Excel and let me
show you the data set that we will use
in this demo
so with India being ready for its 16th
census in
2021 that is next year it is a good time
for us to analyze India's last census
data from 2011 and see where different
states and cities across India stood in
terms of population literacy and other
socioeconomic factors we will analyze
this data by creating different pivot
tables in Excel and explore some of its
features so let's begin first I'll show
you one of the features that Excel
offers us so suppose I click on any cell
and hit contr + Q you can see our entire
table is selected and at the right
bottom there's an option of quick
analysis now you can see by default
Excel has prompted certain features such
as formatting we have charts totals and
there's one more called tables now Excel
by default has created some pivot tables
for us now the first one you say sum of
district code by state names next we
have sum of sex ratio by state name then
we have some of child sex ratio some of
male graduates and some of female
graduates by state name and there are
others before creating our pivote table
so let's have a final look at our data
set so First Column you see is the city
column so there are different cities
from different parts of India then we
have the state code we have the state
name district code we have the total
population followed by male and female
population next you can see we have the
total literates from each City then we
have the male and female literates next
we have the sex ratio then we have the
child sex ratio next we have total
number of graduates and finally you can
see we have male and female graduates so
using this table we'll create several
vote
tables now first of all let's create a
peot table to find the total population
for each state and sort it in descent
order so you can see here we have the
problem statement so our first peot
table will have the total population for
each of the states in descending order
so to create a peot table you can click
any cell in your data go to the insert
Tab and here left you can see we have
the option to create a pivote table so
let me select pivot table now my range
is already selected the entire table and
here here I'll choose existing worksheet
because I want to place my pivote table
in the same worksheet and I'll give my
location I'll point to cell Q5 now let
me click
okay you can see the peot table Fields
appears here on the right now since we
want to find the total population for
each state so what I'll do is I'll drag
my state name onto rows so here in our P
table you can can see we have the
different state names listed now we want
the total population for each of these
states so in the field list I'll search
for total population which is this one
and drag it under
values you can see we have a sum of
total population for each of these
states by default Excel will sum any
numeric column you can always change it
to average minimum maximum anything you
want now we want to sort this column in
descending order so I right click go to
sort option and choose Zed to a that is
largest to
smallest you can see here in 2011
Maharashtra had the highest number of
population or the total population in
Maharashtra was the highest then it was
utar prades we had Andra Pradesh and if
I come down we have nagaland and andan
and Island towards the end so this is a
simple P table that we created now the
next problem we have is we want to find
the total sum of litat in each City
belonging to a certain state so let's
see how to do it I'll click on any cell
go to insert and here I can click on
pivot table my range is selected I'll
choose existing worksheet and give my
location which is Q5 I click on
okay now here we want to find the total
sum of literates so what I'll do is
first let me drag total literates column
to values you have the total sum of
literates from all the
states next I want to see the sum of
total literates based on states and
cities so let me first drag state name
onto rows and then we'll draag City onto
rows
you can see here we have our pivote
table
ready to the left of the pivote table
you can see we have the state names and
the cities per state and on the right
you can see the total number of
literates from each City if I scroll
down we have Assam then you can see we
have
Bihar and if I keep scrolling we have
all the states harana Himachal Pradesh
there Jammu and Kashmir which has now
become a union territory we have jarand
Karnataka and other states as well
moving
on okay so the next thing we want to see
is what is the average sex ratio and the
child sex ratio for each state with that
we also want to find the states that had
the highest and lowest sex ratio in
2011 so let's create a peot table for
this I'll click on any cell go to insert
choose P table click on existing
worksheet I'll select cell Q5
and click on
okay now since we want the average sex
ratio and the child sex ratio so first
I'll drag those columns either you can
manually scroll and drag it or here you
have the option to search for it so if I
look for child you can see we
have the same column listed can just
drag it from there let me delete this
and I also want the X ratio so I'll
place it on top of child sex ratio next
we want to see it based on different
states so what I'll do is I'll take
state name and put it under rows so here
you can see we have our pivot table
ready on the left you can see we have
the different state names listed and on
the right we have the values now we want
to find the average Now by default Excel
will sum the numeric columns you can see
it tells you sum of sex ratio and child
sex ratio so what you can do can click
on this drop down and go to Value field
settings and here summarize values by
you can choose average you can see the
custom name it says average of six ratio
click on
okay our entire column is now giving us
the average sex ratio similarly for this
column let me convert it into average
I'll again click on the drop down go to
Value field settings click on average
and click okay and you can see here we
have the average of child sex ratio for
each of the states now the next question
says which states had the highest and
lowest sex ratio so we'll consider this
column so we'll sort it in any order you
want you can do it either ascending or
descending let me short it in descending
order you can see we have our column
sorted
now so in
2011 Kerala had the highest sex ratio
and if I scroll down to the bottom you
can see Himachal prades had the lowest
which is around
[Music]
88 up next let's explore one more
feature of P table
so suppose you want to see the top or
bottom few rows of a pivote table you
can do that as well so here we have a
question at hand we want to find the top
three cities with the highest number of
female graduates so let's see from the
entire pivote table how we can filter
the top three cities so I'll go to
insert click on the pivote table option
go to existing worksheet click on Q5 and
hit
okay now since we want to find the top
three cities I'll drag City column onto
rows and then we want the female
graduates so in the search bar I'll look
for
female and I'll choose this column that
is female graduates and drag it here
onto values so I have the sum of female
graduates for each of the Cities now
since we want to find the highest number
of female graduates in the top three
cities so let me first sort this
column I'll sort it in descending order
now we have it sorted now from this you
can say that Delhi greater Mumbai and
bangaluru are the top three cities but
it's displaying all the cities for us so
let's filter only the top three so what
you can do is right click and go to
filter under filter you have the option
of top 10 I'll select this
here I only want the top three so either
you
can go down like this or you can
directly type three your column is
already selected let me just click on
okay there you go we have the required
pivot table ready and it only displays
us the top three cities with the highest
number of female
graduates now the next thing we want to
see is how to use a Slicer in a p table
so we have a question here What's the
total population for all the cities in
Rajasthan and Karnataka so let's create
a pivot table for this and see how you
can use a slicer to filter the
table click on existing I'll click on a
location this time
q6 click okay now since I want the total
population so I'll drag total population
onto values and then I'll select
the
city on two rows and then the state name
also I'll place it on top of City so you
have in the pivot table all the states
and their cities and on the right you
can see the total population for each of
these cities but our question is we want
to find only for Rajasthan and Karnataka
now for that what you can do is go to
insert
and create a slicer either you can
create from this option or you can go to
pivote table analyze option and here you
have the option to create or insert a
slicer I click on this and since we want
to slice the table based on state that
is Rajasthan and Karnataka I'll choose
state name as my slicer
field you can see this is my slicer here
now you only want the data for Rajasthan
and Karnataka so I'll search for these
two so here we have Karnataka so let me
select Karnataka
first and I also want for
Rajasthan so let me select Rajasthan
also you can see in our P table we only
have data for Rajasthan and
Karnataka so this pivote table shows you
different cities from Karnataka and the
total sum of population from each of the
cities and similarly we also have for
Rajasthan moving ahead now we will see
another very interesting feature of
pivot that is how you
can create percentage contribution of a
table for example we have a question
here what's the percentage contribution
of male and female literates from each
state now we want to see in terms of
percentage and not as sum or average
let's do that I'll create my pivote
table click on
existing and I'll select an empty
cell okay now here since we want to find
the percentage contribution of male and
female literates so first I'll
drag male literates onto values followed
by female litat on two values by default
it has summed up the male litat and
female litat value and also I want to
drag State
column to
rows so here you can see the sum of male
literates and female literates per state
I want to convert this as percentage
contribution so what we can do is I'll
select any cell and I'll WR right click
and I'll go to so value as and here I
have the option to select percentage of
grand total so I'll select
this you can see we have the percentage
contribution of male literates to the
total now if I sort this you will get to
know which state contributed or has the
highest percentage
contribution so we have Maharashtra for
male
literates then we had with the prades in
2011 if I come
down we had migala nagaland and Andaman
and nicobar
Islands as two states which had little
or minimal contribution to male
literates similarly let's do it for
female literates I'll go to so value as
and select percentage of grand
total so you can see here also
Maharashtra utar Pradesh then Gujarat
and all had the highest percentage
contribution to female
literates so this is another good
feature to convert your data and seat
and terms of
percentage now moving ahead let's say we
want to find the bottom three cities
from each state that had the lowest
female graduates we can do that as well
I'll go to insert click on
pivot go to existing worksheet select an
empty worksheet and click on
okay now since I I want to see based on
States as well as cities so let me drag
the state name first onto Rose and let's
drag the city column onto
rows
next we want female graduates so let me
look for female graduates in the field
list I'll drag it onto
values now we have the list of states
and the respective cities and to the
right of the peot table you can see the
sum of female graduates from each
City now first I'll sort this column
I'll right click go to sort and click on
sest to largest now we have sorted our
female graduates
from shortest or smallest to largest now
since I want to find the bottom three
cities from each state I'll come to this
side
right click go to filter and select top
10 now
I'll replace top 10 with bottom and I
want the bottom three cities from each
state I have my column selected that is
some of female graduates if I click on
okay you can see here some of the states
don't have three cities so you can see
andman and nicobar islands has only one
city that is Port player
while the remaining you can find the
bottom three cities with the lowest
number of female graduates so Andra
Pradesh had these three in Assam we had
Naga then there was tiar and ccha
similarly if I come down in harana we
have palwal kathal and
zind if I come further here you can see
for Karnataka this gangavati this Rani
benur and this scholar similarly you can
see for Kerala as
well now moving
ahead now in the next example I'll tell
you how you can create a calculated
field or a calculated column in Excel
with the help of a pivote table so in a
pivote table you can create or use
custom formulas to create calculated
fields or items calculated fields are
formulas that can refer to other fields
in the pivot table calculated Fields
appear with other value fields in the
pivot table like other value Fields a
calculated Fields name May proceed with
sum of followed by the field name so
here we have a sales table that has
columns like the items which has
different fruits and
vegetables and those have been
categorized as fruits and vegetables we
have the price per kg and this is in
terms of
rupees and we have the quantity that was
sold now let's see if you want to find
the sales for each item in the table you
can create a calculated field so your
sales Colum is going to be the product
of price per kg and quantity so let me
show you how you can do that with the
help of a p table I'll create a peot
table
first click on an empty cell hit okay
now if you see on the top under P table
analyze and under calculations we have
the option Fields items and sets if I
click on this drop down I get the option
to create a calculated field or insert a
calculated field I click on this I'll
give my field name as
sales and I'll select my
formula I'll first click on price per kg
and hit insert field I'll give a space
hit shift 8 to give the product symbol
and then I'll double click on quantity
now this is my formula for sales that is
price per kg multiplied by quantity
I'll click on ADD and I click on okay if
you see here there's a calculated field
that is present in the pivote table
fields which is Sals but it did not add
it to our original table or original
table is the same but here we have added
a calculated field which is present only
in the pivote table list now we can use
this it has already taken it under
values now let's say I want to find the
sum of sales for each item
under each category you can see it here
we have our category fruit and we have
our category vegetable and under that we
have different items like apple apricot
banana similarly in vegetables we have
broccoli the scrots corn eggplant and
others so this is how you can create a
calculated field in a pivote table now
there's one more good feature that Excel
offer for for us in pivote table is to
create a pivote chart so you can use
your pivote table and create different
charts so I'll show you how to do that
if I go to
insert here I have the option of
recommended charts I click on this Excel
gives me some default charts which you
can use let's say I'll select
this let me drag it a bit to the right
here you can see I'll close this pivote
field list this is a nice bar chart that
Excel has created this is called a
pivote chart now here you can see the
category fruits and vegetables and the
different fruits and vegetables are the
items in the y axis you can see the
total Sals if you see from the graph
guava meet the highest amount of Sals
now if I sort this let's sort this first
you can see it here fruit guava meet the
highest amount of sales now since I
sorted and changed my P table the P
chart also automatically gets updated
similarly there are other charts also
that you can create let's go to the
insert Tab and let's click on
recommended charts again let's look for
a pie chart so this is a pie chart that
you can create let me click on okay so
here is our pie chart and each P
represents a certain item and the pi
that has the highest area represents it
had the highest amount of seals in this
case you can see it is quava and
similarly we have other items as well
this is fruit
banana that's
cor and we have spinach and
others let's explore a few other charts
so first I'll click on my pivot table go
to insert and under recommended charts
let's now select a line chart if I hit
okay move it to the right so this is a
line chart you can see it starts from
guava which had the highest amount of
sales then it
drops and in the x-axis you can see the
different items similarly when it starts
with the vegetables broli made the
highest amount of sales with 2,800
rupees and our lowest was eggplant at
900 rupees for fruits papaya sold the
least at 700 rupees let's take another
chart I'll go to insert under
recommended charts let's
see this time we'll see a bar chart now
this is
a horizontal bar chart and not a
vertical one we just saw a vertical
column chart like this this is an
horizontal bar chart now you can always
increase and decrease increase the size
of these
charts let's explore a last chart let's
take the area chart for now so this is
an area chart again it looks similar to
the line chart it starts with guava
which had the highest amount of seals
similarly papaya under fruits had the
lowest amount of seals under vegetables
it was broccoli and finally eggplant
made the lowest amount of Sals and of
vegetable now let's go to our first
sheet and summarize what we have done in
this demo for pivot tables in Excel so
we had our
data this is a 2011 Census Data from
India we had the different cities the
state names and we had the total
population total literates female
literates male literates we had the sex
ratio total graduates and other
information so we began by understanding
how to create a simple pivote table
where we calculated the total population
for each state and sorted it in
descending order we found that
Maharashtra utar Pradesh had the total
population in
2011 then we saw another prevote table
where we calculated the total sum of
literates in each City belonging to a
certain state so you can see we had the
different state names and the cities
under each
state
then we saw another feature where you
could calculate the average of a certain
numerical column so here we calculated
the average sex ratio and the child sex
ratio for each state and found out which
one had the highest and lowest sex
ratio after that we saw how you could
find or filter
tables we saw how to find the top three
cities with the highest number of female
graduates we found out that Delhi
greater Mumbai and bangaluru were the
top cities with highest number of female
graduates next we saw how to use Slicer
in a p table so we sliced our table
based on Rajasthan and Karnataka State
and saw the total population for all the
cities in Rajasthan and
Karnataka in the next
sheet we explored another feature that
was to find the percentage contribution
of male and female literates from each
state then here we saw how to find out
the bottom three cities for each state
having lowest female graduates now one
thing marked that some of the states did
not have three cities for example
Andaman had only one city that was Port
player but the others we found out the
bottom three cities that had the lowest
female graduates finally we looked at
how to create a calculated field in a
pivote table so we saw how to create a
calculated field called sales and then
we explored how to create different
charts and graphs so this was an area
chart that we saw there a column chart
we also saw or looked at a bar chart
that was a horizontal bar chart
similarly we saw how to create a pie
chart as well in this video we'll be
creating two dashboards using a sample
sales data set so if you want to get the
data and the dashboard file that we'll
be creating in this demo then please put
your email IDs in the comment section of
the video
our team will share the files via
email now let's begin by understanding
what is a dashboard in Excel a dashboard
is a visual interface that provides an
overview of key measures relevant to a
particular objective with the help of
charts and graphs dashboard reports
allow managers to get a high level
overview of the business and help them
make quick decisions there are different
types of dashboards such as strategic
dashboards analytical dashboards and
operational dashboards an advantage of
dashboards is the quick detection of
outliers and
correlations with Comprehensive data
visualization it is time saving as
compared to running multiple
reports with this understanding let's
jump into our demo for creating our
dashboards we'll be using a sample sales
data set let me show you the data set
first so here is the sales data set that
we'll be using for our demo so this data
set was actually generated using a
simulator and is completely random it
was not validated though we have applied
certain transformations to the data
using power query features so this data
as you can see has thousand rows so
using the simulator we had generated
thousand rows of
data similarly if I go on top you can
see this data set has 17 columns now let
me give you a brief about each of the
columns so first we have the region
column so we have Middle East and North
Africa there North America Asia
subsaharan Africa and others similarly
we have the country names from which the
item was ordered the third column is the
item type so we have different items
Cosmetics vegetables there's baby food
cereal fruits Etc then we have the
representatives name or you can see this
as the customer name who ordered the
product then we have a sales Channel
column so this there are basically two
channels whether the item was sold
offline or online next we have the order
priority column now here C stands for
critical then we have H which is for
high priority orders then we have M for
medium priority orders and finally we
have L which is for low priority orders
you can see the order date column then
we have the order ID the ship date next
we have unit sold
which is basically the total number of
units sold for each item then we have
the unit price column this is the price
at which each product was sold then we
have the unit cost column which is
basically the production cost for each
of the items next we have the total
revenue the total revenue is actually
the product of unit sold and unit price
then we have the total cost column now
the total cost column is actually the
product of UN sold and the unit cost
similarly we have the total profit
column so total profit is the difference
between total revenue and the total cost
and finally we have created two more
columns that is order year and then we
have order month now these two columns
were actually generated using the power
query features so we Ed the order date
column which is this column and
extracted order year and Order
month so first we are going to create a
revenue dashboard where we'll focus on
generating reports for Revenue by order
year Revenue by year and region Revenue
by order priority and much more we'll
create separate po tables and po charts
and format them to make them look more
interesting and presentable We'll add
slices and timeline to our dashboards in
order to filter it based on specific
Fields now let's create our first report
to see the total revenue generated each
year so we need to create a pivot table
for this
I'll click on a cell in my data set and
then I'll go to the insert tab here we
have the option to select the pivote
table I click on this you can
see my table range is
selected next I want to place my pivote
table in a new worksheet and let's just
click on okay there you go so we have a
new sheet where I can place my P
table so first I need to find find the
total revenue generated by each year so
what I'll do is I'll drag my order year
column under rows and then I'll select
the total revenue column under
values you can see I have my P chart
ready now if you want you can sort this
so from the data you can see we
have order Year from 2010 to
2017 now based on on this data let's
create our pivote chart so I'll click on
any
cell go to insert and here you have the
option to select recommended charts I
click on this now actually I want a line
chart so I'll click on line here and
select
okay there you go so we have
successfully created our first pivote
chart now let me show you how you can
format this chart to make it more
readable so first let me delete these so
I'll right click and select hide all
field buttons on the chart so this will
delete the buttons present on the chart
now let me go ahead and edit the chart
title so the title I want is total
revenue I'll type it
down by
Year all right next let's do a few more
Transformations so if I click on this
plus sign which is actually for chart
elements we have some options like to
add access access titles chart title
data labels this grid lines Legend and
others okay so let's remove the legend
now you can see the total Legend is gone
now let me add access titles so we'll
label our x-axis and Y axis so here
under
xaxis I can write it
as ear similarly on the y axis I'll put
Revenue okay now you can move a bit all
right
now let me select this chart Style
option and go to Colors
first here I'll select yellow color okay
and then let me go back to style let's
select a new style from this
list I
want this style okay now you can also
add data labels so I'll just click on
data labels you can see we have the
revenue for each of the years now this
is not readable at all so we'll will
format this a
bit if I click on this Arrow here I have
more
options if I scroll down you can see we
have something called as number here
I'll expand this and under category I'll
select
custom now here we give a format
code which is a bit
different so this is actually a kind of
a formula so I'll write if my Revenue
value is greater than let's say 9 lakh
999,000 let me make sure that are six
nines here so 1 2 3 4 5 and six okay we
are good to go I'll close the
bracket I'll give a
hash give two commas
so if the revenue is greater than 9 lakh
99,000 I'll put it in the format of
millions So within double quotes I'll
write
M I'll give a
semicolon followed by another
hash and if the value is less than the
desired
number it should be 0 million let me
click on ADD all right you can see how
nicely we have formatted our data and
you see here we
have added the new format which is in
millions all right now if you want you
can go ahead and adjust
the boxes let me move this a bit up I'll
delete this now if you notice this line
chart you can make a few conclusions for
example if you see here in
2010 the total revenue generated was
nearly 175 million now this came down to
150 million in
2011 then the revenue constantly grew
from 2011 till
2014 it reached 195 million and after
2014 it again came down to 180 million
and the revenue dropped significantly
between 2016 and
2017 in 2017 the revenue was just 96
million now before moving ahead to my
next chart let me just rename this sheet
so I'll write it as
Revenue by Year all
right
now let's analyze the revenue generated
each year in different regions so for
this we'll create another pivote table
let me close this I'll click on any cell
go to to insert and select pivote table
I just click okay so that my pivote
table is placed on a new sheet all right
now this time we want the revenue by
each year and region so first of all
let's drag region to
columns
then let's drag the order ear column
under rows and then I'll select total
revenue onto values so here here you can
see we have the pivote table ready so
for 2010 you can see in Asia this was
the revenue generated similarly if you
see for
2013 this was the total revenue
generated in Europe and we have for
other years as well now let's create a
line chart based on this poot table so
I'll select any sell in the pivote table
I'll go to insert and I'll click on on
recommended charts from this list I'll
select my line chart and click on okay
there you go so we have our next pivote
chart
ready so on the right you see the
different regions that are present in
different colors let me just expand it
so that you can see all the regions we
have so in total we
have seven
regions and each of the regions have
been represented in different
colors so if you notice this
graph for the subsaharan African region
in
2012 subsaharan Africa me the highest
amount of
sales now from the sample data you can
also tell that the revenue for North
America has been significantly low
compared to other regions similarly if
you
see for Europe
this was the revenue Trend between 2010
and
2017 so if you see here in 2011 the
sales were at this level then it
significantly dropped in
2012 then in
2013 there was a huge Spike and then in
again came down in 2015 and so on so you
can make your own conclusions by looking
at these line charts now let's format
this chart so first of all let's delete
the field buttons present on the chart
and we'll
also delete the Legend all right now let
me just reduce the size of the
chart next we'll add a chart title
so
we'll give the title as
Revenue by
year and
region
okay you can also format the y axis in
terms of
millions so I'll right click on this
axis and I'll select format
access I'll scroll down and here we have
the number drop down let me scroll again
under category I'll select custom and
we will use this format that we created
for our previous chart there you go you
can see our access labels have been
changed in terms of millions now so
let's close this and let me save it now
you can reduce the font size or increase
the font size let me just show you
suppose you want to increase the font
size of the chart title so you just
select it and from here you can either
reduce or you can increase you can see
now it's 12 if you want you can make it
16 similarly you can also edit the
access labels also by selecting the
chart title you can also move it to left
or right or you can place it in the
center as well for the time being let me
just keep it to the left all right now
we'll see the revenue and total cost by
each region and we'll create a combo
chart for this so let me show you how to
do it I'll go to my data sheet
I have my cell
selected go to insert and click on
pivote table let me just click on okay
all right so for this I'll select my
region onto
rows and
then I'll
have two columns under values the first
one is going to be total revenue and the
next column will be the total cost
column all right
so here we have the pivote table ready
now based on this pivote table let's
create our pivote chart so I'll go to
recommended charts and if you see below
at the bottom we have combo chart so
this is the preview of the combo
chart all right now let me just click on
okay there you go so we have a nice
combo chart ready here now the way to
look at it is the bars represent the
total revenue which is this
column now the line
represents the total cost so let me go
ahead and edit this chart a bit so first
of all let's delete the field buttons
all right and let's also remove the
legend from
here next we'll add data labels so I'll
click on data labels here okay so these
are the
data labels
for the bars or the revenue column now
let's format the data labels in terms of
millions so I'll click on this Arrow go
to more options if I scroll down I have
number from here I'll select custom
and I'll choose my type that is in
millions all right so you can see we
have formatted our
data next thing we'll add a chart title
so here I'll write as
Revenue
by
region it's actually revenue and total
cost by
region before moving ahead let me rename
the sheets as well I'll write
revenue
and total
cost
similarly sheet three also I'm going to
rename it as
Revenue
by year and
region so this makes your street more
readable all right now moving
ahead next we are interested to get the
revenue generated by order priority and
for this we are going to create a pie
chart so let's go to the data sheet and
create our P table first and click on
okay now I'll select order priority
under rows and under values I'll select
total
revenue so this is a very simple pivote
table so you have your order priority so
see is for critical H is for high L is
for low and M is for medium now based on
this let's create a pie chart so I'll go
to recommended charts and here you have
pie chart I want to select this 3D type
of pie chart and I'll click on okay all
right so we have our pie chart ready let
me just resize it and from here I'll
remove the field buttons and I also
don't need the Legend So I'll delete
this as well all right now let's give a
chart title so this is going to be
Revenue
by order
priority now let's add a data labels
I'll check this
option okay now let's again format this
in terms of
millions so
here I'll click on the last option I'll
go to numbers under category I'll select
custom and my type is going to be in
terms of millions there you go let me
close
this I will just move this to the
center all
right now if you want you can change the
color of the text as well so let's have
it in white color and see how it looks
okay so this looks pretty decent
cool
now moving to our next report so this
time we are going to find the total
revenue by countries so we have multiple
countries present in our data set we
want to visualize the revenue generated
in each country so for this we are going
to create a horizontal bar chart so let
me show you how to do it but before
moving ahead let me just rename this
sheet so I'll write
Revenue
by I'll just put op which stands for
order priority all right now let's
create our horizontal bar chart I'll go
to insert click on pivote table and
select
okay so I want my Revenue based on
different countries so I'll select
country and put it under rows and then
I'll choose total revenue and place it
under values so here you have the
different country names we have
Afghanistan this Albania let me scroll
down you have
Bangladesh there
are a number of countries you have Czech
Republic there's Estonia France Gabon
similarly if you scroll down we have
India there's Jamaica Italy and all the
way to the bottom if you go we have New
Zealand there Netherlands Philippines
Portugal
we also
have
Singapore lots and lots of
countries we have the UAE United States
of America Zimbabwe and
others all right let me go up so based
on this pivote table let's create our
pivote chart so I'll go to insert and
select recommended charts from here I'm
going to select the column chart you can
see the preview here and let me click on
okay
all right so here you can
see the different country names at the
bottom and the revenue for each of the
countries let's go ahead and edit this
chart so first of all I'll
delete the field buttons okay and let me
also remove the
legend here I'll write
Revenue
by
entries this going to be my chart
title
okay let's format this chart a little
more so I'll click on this option and
we'll select a new style let's
say I'll
select style six okay and let me now go
under colors and we'll SEL the color of
the bars so let's choose this color okay
so you have a horizontal column chart
ready and at the bottom you can see the
different country names and we have
the
revenue
cool now let me go ahead and rename this
sheet so I'll write Revenue by
countries and hit enter
okay and finally we'll create another
report which is going to be part of our
Revenue dashboard and this is revenue by
items so we'll visualize our revenue for
different items present in the table so
if you see this we have Cosmetics
vegetables cereal fruits the cloth
snacks households and other products as
well so let's check the revenue for each
of these
items so we'll continue the same drill
I'll create my poot table on a new
worksheet and this time I'm going to
drag item type under rows and we'll have
the total revenue under values so here
on the left of the table you can see we
have the different item names and then
we have the total revenue Vue so let me
just short this total
revenue from largest to smallest so you
can see here office supplies meet the
highest amount of Revenue followed by
household then costmetics and fruits
meet the lowest amount of
Revenue I'll click on this go to insert
and select recommended charts this time
I'm going to create a bar chart so this
is how my bar chart is going to look
like I'll select okay all right now
let's format this chart a
bit I'll delete the field buttons and
I'll delete the legend as
well
and let's edit the chart title so this
is going to
be
Revenue by items
cool we also want to change
the color of the bars so I've have
selected all the bars I'll go to my home
Tab and here let's say I want to select
green color all
right I've edited my chart a bit
now
let's make
it
14 and I'll remove the bard
okay here if you want you can change the
font Also let's keep it in blue color
all
right finally let's rename this sheet so
I'll write Revenue by
items cool finally now it's time for us
to merge all the charts that we have
created to our dashboard so let me show
show you how you can create the
dashboard I'll create a new sheet and
first thing I'm going to do is I'll
click on The View Tab and uncheck grid
lines so this will remove the grid lines
present in the
worksheet next I'm going to insert an
image so we'll have a background image
on our dashboard so the way to do is
I'll go to the insert Tab and under
illustrations I have the option to
select pictures or insert pictures so
I'm going to insert picture present on
the device that is my
computer I'll go to desktop and here I
have a folder called Excel dashboard
files and I'll select this dashboard
background and hit
insert so this is going to insert an
image now let me just drag this image so
it covers a fof
portion so I'll hit shift and I'll drag
it all
right so you can see I have successfully
added a background image if you want you
can
still expand this background image a bit
to the right
cool now the next
thing is going to be the title of the
dashboard so I'll click on insert and
here I have the option to select a text
box so I'll click on a text box
and I'm going to place a text box in the
middle and I'm going to name this
textbox as
Excel Revenue dashboard
on
seals
I'll centrer line
it let's do some more formatting so I'll
select this text box on the top you can
see shape
format here I'm going to expand this
shape fill and I'll select no fill so my
text box is transparent now and I'll
also remove the outline all right now
let me just double click on the title of
my dashboard and I'm going to
select a font you can select whichever
font you
want let me stick to britanica
bold and I'll increase the size to let's
say
30 all right I'll just just drag the
text
box I'll make the text as white instead
of black all right so we have
our title of the dashboard ready now if
you want you can also insert some icons
to this dashboard so I'll go to insert
and I'll click on illustrations again
and select
pictures I'm going to add this two
pictures which is of a store and a
cart to make it
look visually appealing so I'll place
the
icons here and similarly let me just
copy
it and I'll place
the cart and the
store to the right as
well all
right next
the idea is
to bring in all the charts that we have
created and place it on the
dashboard so let me
copy each of the charts and place it on
the dashboard so I'll hit control V to
paste
it and we'll resize this as well
all
right similarly let me bring in all the
other charts as
well all right so now you can see I have
added all my charts and graphs to this
dashboard so you can see here we have
our line charts our column charts the
combo charts the Spy chart and others
now let me go ahead and format these
charts a little more so you can see this
looks a bit cluttered so let's adjust
the labels let me bring this down
similarly I'll bring 190 million a
little
below all right this looks fine
now one more thing we are going to do
is we'll remove the white background
from each of the charts and make it
transparent print so let me show you how
to do it so I'll select this chart then
I'll right click and go to format chart
area here on the right you see we have
an option called No fill so if I select
no fill you can see the white background
is gone now similarly let me also remove
the grid lines so I'll select the grid
lines and hit delete so you have also
removed the grid lines from here now
let's also remove the white outline that
we have so I'll select this chart go to
format and here I'll go to shape outline
and I'll select no outline you see this
so we have our total revenue by year
which is a line chart and this is
completely transparent now now what I'm
going to do is I'll place this chart
over a box so I'll go to insert and in
insert we have the option to create a
shape so I'll click on on illustrations
and here I'll choose a shape and let me
select a rectangle so I'll just create a
rectangle
here all right and now what I'll do is
I'll select this and bring this to front
I'll right click and choose bring to
front and I'll place this
shape below it all right
now the next thing is to edit the shape
so first I'll change the color of this
box so let me select this blue color and
and let me increase the transparency so
I'll right click and go to format shape
here I'll increase the transparency
let's keep it to
25% or let's say 20% all right next
thing we'll just just convert all the
font to white
color including the access labels the
chart
title will
also convert all
the access labels to white color so it
looks better now we'll just adjust
our chart over
here next thing let's just remove the
the outline so I'll go to shape outline
and I'll select no outline you see we
have now formatted our chart let's just
pull this a little
up all right now we'll add this blue
background to all the other charts so
we'll first add the background make it
transparent and then we'll convert the
font text to white color to make it more
readable and visible so for the time
being I'll just pause the video and come
back
again all right so now you can see on
your screens we have nicely formatted
our dashboard so I have added a few
logos for each of the charts you can see
the logos here so for Revenue by
countries we have a globe then if you
see here this is kind of a map or a
location similarly we have all formatted
the color of the bar bars then we have
also formatted the labels in terms of
millions if you look on the Y AIS even
the revenue for year and region are all
formatted in terms of
millions if you want this you can also
format the total year by Revenue in
terms of millions so the way to do is
you can select this graph right click
and go to format
access here if I scroll down
you have
numbers and under category I'll select
custom then I'll select my type as this
format which is in
millions and you see here we have
successfully formatted our y-axis
labels all right so the next thing is to
add slicers and timelines to a dashboard
now slicers are used to format your data
based on your a particular column
suppose if you want to see Revenue by
certain items you can add item as a
slicer and you can view the
entire dashboard similarly for timelines
you can add date columns so if you want
to see what was the amount of sales or
revenue generated on a particular year
or a particular month you can do that
using a timeline so I'll select one of
the charts and then either you can go to
the insert Tab and here you can see
under filters we have slices and
timeline or if you go to the pivote
Chart analyze tab here also you have
insert slicer and timeline option so I
select insert slicer here first now it's
giving me the list of fields present in
the data set so I'll select country
region and let's say we want to know by
by item type and sales channel so these
are going to be my four slicers I'll
click on
okay you can see it here we have our
four slicers here and these are the list
of values under region we have Asia this
Europe North America and others
similarly we have the different country
names for Country slicers and then for
item type also we have all the items
that were present in our data set
now moving ahead we need to connect all
the slicers to a dashboard so what I'll
do is I'll right
click on this option and I'll go to
report
connections okay so under report
connections you have all the pivote
tables that we
created you currently see only one of
the P table is selected so we need to
select all the p tables so let me check
all the P tables present in this
workbook and click on okay all right now
that we have connected one of our
slicers we'll now connect the other
remaining slicers so I'll right click on
this go to report connections and I'll
check all the pivote tables present in
this worksheet click on okay similarly
let's do it for the country slicer I'll
go to report connections and let me
select all the pivot
tables and finally we have the item type
so I'll right click go to report
connections and then I'll select all my
P
tables and let's hit okay all right now
let me just organize this a bit so I'll
place my P
tables to the
right I'll just reduce the
size let me scroll
down I'll
add my region slicer
here
similarly I'll add my final slicer that
is sales Channel now in our next
dashboard which is going to be the
profit dashboard I'll show you how to
add a timeline
all right now I have arranged all my
slicers so let's say you want to find
the revenue that was generated for an
item type let's say beverages so you can
just select beverages here and all your
charts show the respective revenues so
you have the total revenue by year for
beverages only similarly here you can
see the revenue by year and region only
for beverages item type if I scroll
down now this chart represents the
revenue that was generated in each of
the countries only for item type
beverages let me just uncheck
it all
right let's say you want to
see the revenue generated for a country
like India so I have selected India here
and now you can
see my graph has changed only for
country
India you can see here it is showing
only for India
now now similarly you can also filter
your revenues based on the different
regions let's say you want to know the
revenue generated based on sales channel
so we have two sales channel that is
offline and online suppose you want to
know the revenue generated offline so
I'll just select offline you can see the
values have changed so these were the
revenues generated for each of the items
only for
offline if you see here now these were
all the offline sales for the different
regions
so this is our entire
Excel Revenue dashboard on sales we
created multiple charts and graphs then
we applied different formatting we added
different icons then we formatted the
labels also next we added slicers and
finally we saw how we could filter our
data based on these
slicers likewise now we are going to
create a profit dashboard based on the
same data so so before moving ahead let
me rename this sheet as Revenue
dashboard I'll write
R dashboard
okay now we'll move to our data sheet
and start creating our pivot tables and
pivot charts for the profit
dashboard all right so let me go ahead
and create my first pivot table
so I'll create a new
worksheet this time I'm going to create
a line chart to visualize the profit for
each year so I'll drag my total profit
column to values and my order year to
rows so here you can see we have
our pivote table ready now you can sort
this data to get an idea as to
which year had the highest profit and
which year had the lowest profit so from
this pivote table you can see since I've
sorted this data in descending order so
2014 had the maximum amount of profit
and 2017 had the least amount of profit
I'll just do contrl Z to undo it all
right now based on this P table let me
go ahead and create my P chart so I'll
go to recommended charts and click on a
line chart so this is the preview of the
chart I'll click on okay let me close
this
similarly we are going
to edit this chart now so first I'll
hide all the field buttons present on
the chart and I'll rename the chart
title as
total
profit by
year next I'm going to remove the legend
so I'll delete
this let's do some more
formatting so I'll go to style
and this
time I'm going to
select my style type okay and if you
want you can choose the colors as well
for the Tim being let's have this yellow
color next let me add the data labels so
again if you see here this is not
formatted properly so let's go ahead and
format
the
data labels so I'll click on
number and I'll select custom
here and the type I'm going to select is
in millions and I'll click on close so
here you can see we have
our line chart ready which shows total
profit by year let's rename this sheet
as
profit
by year all right now let's move back to
our data sheet
again next we are going
to show the total profit by countries
for this I'm going to create a map so
let me first create my pivote table so
I'll go to insert and I'll click on
pivote
table let me click on
okay since I want the country names so
I'll select country under Rose and then
I have my total profit it under
values the next thing I'm going to do is
I'll just rename the row labels as
countries and
then I'm going to delete the grand total
which you can see at the bottom so here
we have the grand total let me just
delete the grand total so I'm going to
select this pivot table go to the design
tab here we have sub totals and Grand
totals I'll switch off the grand total
let me just verify it again I'll scroll
down you see the grand total has gone
now all right now we want to create a
map out of this the way to do is I'm
going to select my
data copy it I'll go on top and I'll
paste it here using this data I can
create my field map now so I'll go to
insert here we have the option to create
a field map there you go you can see we
have our map ready I can expand this as
you can see our map has a color scale
which comes from light gray color to
dark blue color so the countries that
are in Gray or you can say light blue
have the lowest amount of profit while
the regions or the countries that have
been shaded
in dark color or dark blue color have
highest amount of profit I will'll go
ahead and delete this
scale okay next we need to connect this
map to the original data source so what
I'll do is I'll right click on this map
and I'll go to select data here instead
of the previous range I'll give my new
Range now so my new Range will be my
original P table that I had
created go on top and click on Okay so
we have our map ready now now if you
want you can change the color of the
shade so I'll just go to colors and
let's say we'll keep green
color so the countries that are shaded
in dark green have the highest amount of
profit while those which are highlighted
in light green color are are the
countries that made least amount of
profit okay now moving
on next
we want to create a pivote table that
will show us the profit by year and
sales channel so for this we are going
to create another line chart so I'll go
to insert and click on pivote table so
I'll select new worksheet here since I
want to know the profit by year first of
all I'll drag my order year column to
rows and then I'll choose my total
profit column under values next I'm
going to select my sales Channel under
columns there you go so we have our poot
table here based on this P table let me
create my P chart so I'll go to
recommended charts and I'm going to
create a line
chart I close
this you see here based on this chart
you can tell the profit generated with
online sales were actually
lower than that of offline so here the
Blue Line represents offline profit and
the Orange Line represents Online Profit
if you Mark Clearly in year 2012 the
Online Profit was actually higher than
the offline profit so let me go ahead
and edit this chart a bit so we'll
delete the field buttons I'll also
delete the legend for now let me go
ahead and add a chart title so I'll
write
profit
by year
and sales
Channel
okay so this is my second report before
moving ahead let me just rename this
sheet so I'll write
profit
by
countries similarly let me rename this
sheet as
profit by year and let's say SC for
sales
Channel
okay moving
ahead now I want to create a pie chart
based on a pivot table that will show
The Profit by sales Channel only so this
is going to be a simple pie chart so
I'll first go to insert click on P table
and click on okay so I'll drag my sales
Channel under row and then we'll have
the total profit column under
values so this is my simple pivote table
now let's create our pivote chart which
is going to be a pie chart let me
explore the other types of pie charts we
have okay so I'm going to select a donut
chart here I'll click on okay let's edit
this chart I'll remove the field buttons
let me now remove the legend as well
I'll just resize it and this is going to
be
profit
by sales
Channel
okay let's also
add data
labels and here again I'm going going
to format this
label I'll select the category as custom
and my type will be in millions okay let
me just move
this to the
left and this to the right
okay let's also delete the lines
cool now let me just rename this sheet
so I'll write profit profit
by let's say see which stands for sales
Channel cool finally I'm going to create
a report that will show the revenue and
profit by items so I'll go ahead and
create my pivote table first this time
I'll choose my total profit under
values and we'll also have the revenue
column so I'll put my Revenue at the top
then I'm going to
select item type under rows so here is
my pivote table based on this pivote
table let me now create a combo chart so
you can see the preview of the chart the
blue bars represent the total revenue
and the Orange Line represents the total
profit I'll click on okay let me close
this first let's remove the field
buttons let's also
remove the legend
here then we'll add a chart title I'll
name it
as
revenue
and
profit by
items okay if you want you can also go
ahead and change the color of the
bars so let me just
select one of these colors
okay all right so we have our five
reports ready that we are going to use
for our profit
dashboard next let's create a new sheet
and we'll get started with building our
dashboard so I'll click on a new sheet
let me just rename this as profit
dashboard all right we'll continue with
the previous drill so first of all let's
go to the view Tab and remove the grid
lines now we'll insert a background
image like we did for our Revenue
dashboard so I'll go to insert under
illustrations I'll click on pictures and
select this device I'm going to have the
same background I'll click on
insert all right so you can see we have
a picture of a company or you can say an
organization let's just drag
this a bit to the
right we'll adjust the size
also all
right now let's copy the title of my
profit dashboard so here you can see I
have brought my Revenue
dashboard and I'll copy the title and
the logos that we used for the revenue
dashboard I'll paste it on my new
dashboard let's just align
it in the
center all right the next step let me
now go ahead and edit the title so this
is actually going to
be
Excel profit dashboard instead of
Revenue now we'll copy each of the
charts that we just created for example
the revenue and profit by items then we
had profit by sales
Channel all this we are going to copy
one by one and put it on the profit
dashboard so let me just copy a few
now I'll paste it
here and later on we can make the
adjustment copy this as
well
similarly I'll bring the other three
charts onto my sales
dashboard okay so here on my profit
dashboard I have added all the
charts and I have aligned and reshaped
it so that it looks good I've also made
some formatting for example I've have
reduced the size of the chart
title now let me go ahead and show you a
few more formatting that we also did for
the revenue dashboard first let's remove
the white background from all the charts
so I'll select the first chart I'll
right click and I'll click on format
chart area here under fill I'll select
no fill next I'm going to remove the
grid lines so I'll just delete it let me
close this now we also have a outline
line so I'll go to
design actually format and I'll remove
the
outline next I'm going to
add a blue box at the back like how we
did for the revenue dashboard so let me
select a blue box from here and I'm
going to paste it here okay let me just
select the chart and I'll bring this to
front and I'll move this
to the
back next I'm going to change the font
color all to White so that it's clearly
visible and it's more
readable I'll do it for the x-axis as
well okay so here I have my first chart
ready the same I'm going to do for the
rest of the
charts okay so now you can see here I
have formatted all my charts I have also
added a blue
background you see here I have also
formatted the V labels in terms of
millions which is actually the profit
similarly here I have added the data
labels this is for
Revenue some of the charts also have the
data Legends so here you can see the
blue color represents
offline and the red represents online
similarly here you have the Legends I've
also formatted the
map as well okay now the next thing is
to make this dashboard more interactive
so we'll add our slicers as well as
timeline first let me show you how to
add a timeline so I'll select one of the
charts and I'll go to insert under
insert I have the option to create a
timeline so I'll
just click on
timelines so timeline is actually based
on on date columns so since in our data
set we only have two date columns one is
order date and one is ship date so Excel
has only shown us two columns so I'm
going to create my timeline based on
order date so I'll select my order date
column and I'll click on okay you can
see here this is called a timeline I can
expand this now this timeline is based
on months now and if I scroll this
timeline you can see here I have my
order ear 2010 and I have all the 12
months similarly we have for 2011 then
we have for
2012 all the way till
2017 now you can filter this in terms of
years quarters months or days let me
just select year now so I have years
from 2010 till
2017 let me just squiz this and I'll
place it somewhere here on the
right now let me go ahead and create a
few slicers for my profit dashboard so I
have selected one of the charts under
insert I'll click on
slicer you can see it gives me the list
of columns from which I want to create
slicers so I'll create a region let me
also select
country let's say I
want the representatives name or the
customer's name and I'll click on okay
so here here I have created three
slicers let
me first resize it and I'll place
it on the
right similarly I'll place the country
column
also then we have the region
slicer I'll resize this and I'll bring
it here okay the next thing we need to
do is
I have to connect all the slicers and
the timeline to the P
tables for the profit dashboard so I'm
going to click
on the multiple select option and go to
report connections here I'm going to
select all the pivote tables that are
related to
profit so here I have selected four and
I need one more which is po table number
10 I click on okay okay similarly let's
create or connect my region filter to
all the pivot tables so I right click go
to report connections here I'll choose
all my pivote tables which are based on
profit I'll click on okay let's do it
for the country slicer as
well click on okay and similarly I'll
connect my timeline as as well I'll go
to report
connections and I'll
select all the pivote tables related to
profit then I'll click on okay let me
now go ahead and create another slicer
based on sales channel so I'm selecting
one of the poot charts I'll go to insert
click on slicer and I'll select sales
Channel and hit
okay so I have my sales Channel slicer
now let me connect it to all the
respective pivote
tables that are based on
profit click on
okay now let me just bring it here all
right the next thing I want to show is
how are we going to use the timeline
first so you see we have all
the years here from 2010 to
2017 now suppose you want to know the
profit that was generated in the year
2012 so I'll just click on this range
and now you can see our charts only show
information for 2012 so this dot
represents that was 51 million profit in
the year 2012 similarly you can see
here the profit by sales channel for
2012 from the map you can see the
different countries and the
profit each of these countries made in
2012 if I scroll down you can see the
revenue and profit by items now if I
select another year let's say
2013 I can just drag this to the right
and now you can
see our profit by year and sales channel
for offline and online you can see the
map or the line chart for total profit
by year so in 2012 it was 51 million and
then it went up to 54 million in
2013
similarly our map has also changed now
this is a sort of an information that we
have you can click on this
and check the information that Excel has
prompted all right so this is how you
can use a timeline now as I said we
checked by years you can also see it for
for months and quarters as
well let me just uncheck
it I'll send it back to the place where
it was and I'll reduce the
size okay now suppose you want to check
the profit made by different
Representatives you can select them one
by one let's say Adam
Churchill this is
the profit generated by ad
Churchill similarly you can select
multiple persons as well now suppose you
want to see the profit by different
countries so you can use the country
slicer let me just bring this to the
middle and let's expand our chart a
bit okay so here you have
the profit by different countries chart
I'll just bring this to the front so
that you can see it clear cly okay now
here suppose you want to see the profit
generated in let's say United Kingdom
you can select United Kingdom so this is
the map of United Kingdom and it tells
you the total profit that was generated
in United Kingdom and Below you can see
the revenue and profit for all the items
that was sold in United Kingdom so you
had beverages clothes household office
supplies you can see clearly office
supplies item meet the highest amount of
profit in United Kingdom now you can
also select multiple countries let's say
I want to know for France as well so
my map will change accordingly so now I
have United Kingdom and France selected
and the other charts present in my
dashboard change accordingly
now I have
my country selected as India you can see
the map of India
here and these were the respective
profit values now one thing to note here
is this is actually not millions that
should be in K that is Thousands so
please mark this as thousand and not in
millions even for this this is actually
K and not million all right so we have
successfully created our second
dashboard that is on profit let me just
resize this a bit and we'll place it
where it was
earlier
cool so we saw how to create different
peot tables and poot charts and then we
formatted our poo charts based on our
requirement we saw how to edit the
colors now let me show you one more
thing you can also change the look and
feel of the dashboard by going to the
page layout tab under page layout you
have themes so here you can select
different themes currently we are with
the office theme now let me just select
another theme let's say faet you see the
colors have changed and it looks really
beautiful similarly let me try out
another theme let's say organic you see
our chart has changed let me just delete
this okay so now once you change the
theme the text also change a bit you can
see the slicers are in a different
font let me explore One More theme let's
say this time I'm going to choose
depth and this is more of a green type
of
color
you can play around and select whatever
theme suits the best all
right now let me just move back to my
Revenue dashboard and see how it looks
there you go so since we changed our
theme even our Revenue dashboard is also
impacted
so this is how it looks
now you can always go ahead and play
with different different themes colors
fonts and effects all right so in this
demo we saw how to create a revenue
dashboard so we created line charts this
combo chart pie chart horizontal and
vertical bar charts and then we learned
how to add slicers and connected to
different pivote tables and we filtered
our data to see Revenue as well as
profit by items by countries by
different regions sales Channel we
learned how to create a map and lots
more let's quickly see some more
examples of doing data analysis using
Excel and for that we can use some
inbuilt addins which can be added to our
Excel sheet so for example if you would
want to do a descriptive analytics or
descriptive analysis is on your data say
for example getting your descriptive
statistics such as your mean median mode
and so on so we can do that and we can
use Excel for it so for example you can
if you are given some data say I have
temperature price of ice cream units
sold and I would want to have
descriptive statistics on this what I
can do is I can click on file and here
in file you can click on options and
within options click on addin now within
addin you have Excel addin which is
selected here so click on say go for
example and that shows what addins are
available and you can choose which ones
are you interested in so for example I
have chosen analysis tool pack and
solver addin and click on okay now that
basically should add more options to
your Excel so if you click on data so
here you see data and analysis and
solver and this is what we would want to
use to get our descriptive statistics
for these three columns so for example
let's say
temperature or you can even give the
names later once you get your
descriptive statistics so for example
let's go for data analysis and here it
says what are you interested in there is
a two factor with replication you have
correlation co-variance descriptive stat
statistics you have histogram so let's
click on descriptive statistics click on
okay now this one basically asks your
input range so while your cursor is
blinking here also it is said grouped by
so let's give it a range so for example
I will say say
temperature now if I do this and I have
selected The Heading just look at that
and now you need an output range so
let's just select this and then you can
have your cursor blinking in here let's
select say Fields here and this is where
I would want the output now it also said
what options do you want so it has
output range we can then select summary
statistics confidence level so I will
say summary statistics is what I'm
interested in say okay and this says
input input range contains non-numeric
data now why is that because we chose
temperature The Heading also so click on
okay and here we will alter the range so
this one is our range should be only the
values numeric values on which we would
want the descriptive statistics we have
output range already selected we have
summary statistics and now you can click
on okay and that basically gives your
descriptive statistics for temperature
so here I could basically given a value
for this so I can say
temperature and that's my descriptive
statistics for temperature might be I
can just do some formatting and that's
it so that gives me descriptive
statistics for the values here now
similarly we can do it for price of ice
cream so what we need is we need to BAS
basically go for
data data analysis descriptive
statistics say okay now you need to give
a range so here I will change my range
to these values output range is already
selected now we are interested in
summary statistics click on okay and
this says output range will overwrite
existing data press okay to overwrite
data in range I will say say cancel no
that's not what we want to do we need to
give a new Range so let's select our new
Range which is here and now click on
okay so now we get the values which is
for your price of ice cream so again we
can basically select this and say price
of ice
cream and we got our descriptive
statistics for price of ice cream and
like we did earlier I can select this I
can basically do a merge and center and
that gives me descriptive statistics for
price of ice cream so we could also
basically change this now I can go into
data and I again go into Data
analysis descriptive statistics so we
know that we had selected this B2 to B8
and this one which is H6 to h19 we would
want to shift it might be two columns up
so mightbe I can just say H5 and I can
manually change it to H 17 and let's say
okay and we will basically get this and
I can get rid of this so I can have it
in the same range so similarly so this
one will have to be renamed and I can
basically say price of ice cream
and that's basically my descriptive
statistics for my price of ice
cream and similarly we can do it for the
third column which is units sold so we
would want to have this now let's see we
can click on data we can click on data
analysis descriptive statistics so we
need to give the range correctly so this
time our range changes to units sold now
now we can also say labels in first row
okay if we were selecting The Heading so
let's do it in this way so in my range
in my range let me empty this I can
basically select this which we know has
non-numeric data in the first row say
for example I'll say labels in first row
I'm interested in summary statistics and
this range will now have to be changed
from H to basically something like J so
let's say J and let's select these
values so that should take care of
things and now you see you have your
units sold you did not have to manually
rename it and you have basically got the
descriptive statistics so this is how
you can simply perform analysis using
data analysis here you can basically get
your descriptive statistics for your
columns and then you can do whatever
needed formatting you need to basically
make your data look in a good way now
let's look at one more example of data
analysis where we may want to look at
the frequency of values or frequency of
values occurring in a range of values so
for example if you have been given
temperatures you have been given some
pins where you would want to identify
how many values fall into the range of 0
to 20 20 to 30 30 to 40 40 to 50 and the
easiest way to do that would be creating
histogram now histogram is usually used
for data analysis where you would want
to look at different variables or say
features for example temperature is one
such feature might be there might be one
more variable or feature such as sale of
ice scream and you would want to see if
the increase or decrease in temperature
affects the increase or decrease in sale
of ice cream might be sale of ice cream
is a is a response based on temperature
so it depends so sometimes you may want
to find a relationship between two
variables whether they are positively or
negatively related or you would want to
do different kind of analysis and in
certain cases we may want to first to
analysis on one single variable look at
the frequency of values might be also
look at the defects and for which we can
use something like par chart so we can
go for histogram and that basically
gives us the frequency of values now how
do we do that so we have already added
the addin which is data analysis earlier
so we can just use the same thing again
here we would want to create a histogram
so let's say okay now I have already
selected input range so if you see here
my input range is temperature which is
also with the headings and I have bin
range which is basically the range of
values so for example I can select this
and that's my bin range I am selecting
or the option labels because I'm using
the first row which has the heading such
as temperature and bins now we need to
give a output range so for example let's
say I would want my data here and that
becomes my output range so you can have
a sorted histogram or basically a Paro
chart so if that's what you're
interested in looking at the frequencies
for your different ranges and here I'm
also selecting chart output because I
would want to have a visual histogram
which gives us the frequency and it's as
simple as this just click on okay and
now you get your bins so it basically
tells you frequency of values which is
basically 20 but that does not mean it
is only talking about the values 20 it
is basically talking as a range of 0 to
20 so we have 0 to 20 that is two so we
can basically say there is 12 12 here
with the 20 being the maximum value and
then there is one more 20 so that's your
0 to 20 then you have 20 to 30 which
shows three values so might be in that
case I can say
26 is one thing then I can say 30 that's
the second one and then basically I can
look at 22 so basically this one does
not select 20 as the lower range but it
basically selects 30 as the higher range
so I do see 20 to 30 there are three
entries similarly we can see values for
40 and 50 and since we have selected
parto or sorted histogram that shows in
a descending order what is the highest
frequency of values within a particular
range so that shows me highest frequency
is five and then you have three and
three and then two so this is how we can
create a histogram and we can perform
analysis on a single variable now as
discussed earlier as I said sometimes we
may be interested in finding out the
correlation between different variables
such as say here we have temperature
price of ice cream and units sold and we
may want to find out the correlation
between one variable to another variable
or we would want to find out the
relationship between variables are they
linearly related are they positively
related negatively related and so on and
for that we can use the
correlation of your data analysis addin
so for example you want to find out
correlation of temperature and units
sold and what we can do is we can find
out that using a formula so for example
if I search for something like Co
relation and let's search so there is a
function called correlation which we can
use and we can use this to calculate the
correlation of temperature and units
sold so for example let's select this
and that's the function so it says give
me an first array and a second array so
we are interested in finding out
correlation of temperature and units
sold so let's select the range of values
for temperature and then I'm interested
in finding out the correlation of
temperature and units sold so let's
select this and that basically gives me
a range of values it gives me the
correlation value which is
2859 say okay and that's your value so
similarly we can do it for temperature
and price ice cream so let's go
for correlation so that's the function
we are interested in you need to give a
range of Valu so here we are interested
in temperature and price of ice cream so
let's select temperature and then the
second array or list of values is price
of ice cream let's select that let's
close our bracket and here we have the
correlation value of temperature and the
price of ice cream similarly you may be
interested in finding out temperature
and units sold like what we have done
earlier so we can do the same thing
based on function so this is same as
correlation of temperature and unit sold
so I can get rid of this one now how do
I do it using the data analysis add in
so for that what we need is we need to
go into Data we need to go click on data
analysis and here you have the option
called correlation let's select this now
that basically needs an input range so
we need the range now I might be
interested in finding out the
correlation between temperature and
price of ice cream and units sold so
I've selected all the columns here we
will say Group by columns obviously we
need to select label in first row
because that is basically taking care of
the first row is heading now output
range you can just give one simple cell
and that's where your data will start
from or you can give a new workbook so
click on okay and that basically gives
you that basically gives your
correlation of your different variables
and what are the values and we can check
these values based based on the values
what we have here so we have basically
temperature and price of ice
cream and that basically shows me
96149 you have temperature and units
sold so you have
2859 now you can also look at units sold
and say for example price of ice cream
you can look at these particular values
so if I would be interested in finding
out what is the relationship between
these variables I can easily find using
correlation so I could be basically
writing in a formula here and selecting
what are the cells so here we were
selecting A2 and C here we were
selecting A and B now might be I'm
interested in price of ice cream and
units sold and if that's what I'm
interested in then I will give a range
of B2 to B8 C2 to C8 and similarly you
can get your analysis or correlation
values so it's very simple in Excel and
you can use either the data analysis Tab
and get your correlation or you can use
formulas and do that now one more
important part of data analysis is doing
your sampling now sampling could be
periodic sampling or random sampling so
sometimes you may want to look at a
variable and you may want to get some
values based on periodic data that means
might be I'm interested in range of
values I'm interested in seeing a sample
of values
for uh a particular period which could
be
basically uh a range of values or you
could just do a random sampling so for
example if I go for periodic sampling so
out of these values which I see here
might be I want to see say periodic
sampling that is a frequency of two
values how many times we have these
values occurring here or I would go for
random sampling so basically randomly I
would want to pick up say three
temperature values now how do I do it so
for example here I have seven values now
if I go for periodic sampling the sample
or periodic sample value which I need to
give has to be lesser than the total
input
values so for example we can do this
let's go in here and let's go for data
analysis so we can go for sampling here
click on okay and that needs a range of
values so we will select A2 to A8 now I
could have selected all the values for
this one temperature and in that case I
can give labels which is going to take
care of the first row now here we can go
for number of samples which you are
interested in or giving a period so
let's go for period and say for example
I have seven values so what if I select
five so for example if I say five that
means I could just get one value so
basically when I'm saying five out of
seven so that's just giving me out of
five I want one value so I can then just
give an output range so here I can
basically select this cell I'll say okay
and now you see it just shows me one
value so out of the first range that is
I have said five it has given me the
fifth value that's your periodic
sampling so for example we want more
values so let's reduce this period to
might be two which basically gives me
every second value so I can basically
say for example two and say okay and
then say okay so that shows me
26 then you have 35 then you have 40 and
then well this one does not have any
more values so that's your periodic
sampling now if you go for random
sampling that's basically Rand randomly
picking up values and you can choose how
many values you want so go for data
analysis go for sampling I'll go for
number of samples how many you want so
for example out of seven values randomly
I want three values and I can just give
this say okay and then we will do a
cancel because we need to change that
range so let's select this and say okay
and that gives me random three values
from this is values of temperature so we
can use Excel to do a simple sampling
and we can choose whether we would want
to go for want to be a certified data
expert then here we have postgraduate
program in data analytics by simply
learn take a glance at the notable
features and skills offered in this
course you will benefit from exclusive
IBM hackathons and ask me anything
sessions eight times more interaction in
live online classes with industry
experts Capstar projects across three
domains and over 14 data analytics
projects using real data sets from
Google Play Store lyt World Bank and
more master classes from puu faculty and
IBM experts along with simply learns job
assist for better job prospects acquire
skills in data analytics statistical
analysis using Excel data analysis using
Python and R and data visualization
using tableu and powerbi enroll now us
using the course Link in the description
box hello everyone and welcome to this
interesting video tutorial by simply
loan today we're going to perform two
Hands-On projects on Co data analysis
using Python and tblo this is going to
be a really interesting and fun session
where I'll be asking you a few generic
quiz questions related to Corona virus
please make sure to answer them in the
comment section of the video we'll be
happy to hear from you covid or Corona
virus is an ongoing Global pandemic of
Corona virus disease that emerged in
2019 and was first identified vuhan
China it is defined as an illness caused
by a novel Corona virus called severe
acute respiratory syndrome Corona virus
2 commonly known as SARS Cove 2 on March
11 2020 the wh declared covid-19 a
global Health Emergency the virus has so
far infected over 22 CR people and
killed more than 4.5 million Innocents
in India there have been over 3.3 CR
confirmed cases and nearly 4 lakh 41,000
deaths have been reported so
far this data is according to official
figures released by The Union Ministry
of Health and Family
welfare as the world tries to cope up
with this deadly virus we request all
our viewers and their family members to
follow all the necessary precautions to
avoid getting
infected quiz
time now let's see our first quiz in
this
project what does Corona in Corona virus
mean here are the options is it a b here
B respiratory is it C crown or is it d
sun this is a very generic question I'm
sure a lot of you may already know some
of you may
not please let us know your answers in
the comment section of the video we'll
be glad to hear from
you now in this video we will use three
different covid-19 data sets and perform
data analysis using Python and tblo the
project will give you hands-on
experience working on real world data
sets and how you can use the different
python libraries to analyze and
visualize data and draw conclusions you
will learn how to create different plots
in Tableau and then make a dashboard
from the visuals the project will give
you an idea about the impact of Corona
virus globally in terms of confirmed
cases deaths
reported the number of recoveries as
well as Active
cases we will also see how India has
been affected since the pandemic started
and dive into the different states and
union territories to learn more about
the covid-19 influence and the
vaccination status first let me show you
the two data sets that we'll be using so
for our first project using python we'll
be using the first two data sets
covid-19 India and covid vaccine
statewise let me open the two data
sets Okay so this is the first data set
you can see here we have the date we
have the time then we have the different
state names I scroll down you can see we
have Kerala Tamil Nadu Delhi harana
Rajasthan this ladak Punjab Telangana
and other states then we have something
called as confirmed Indian national so
actually this two data sets confirmed
Indian national and confirmed foreign
National we won't be using so in the
demo itself we'll be dropping these two
columns what we are concerned about are
the last three columns the
cured cases or the recoveries the number
of deaths reported and then we have the
total number of confirmed cases let me
just sort the B column that is the date
column so that you have an idea about
the recent data we have I'll continue
with the current selection and sort it
you can see here this is till 11th of
August
2021 so so this data was collected from
Kel it has some discrepancies that we
will see in the demo the data is
available for free we will provide the
link to the data sets in the description
of the video so please go ahead and
download them the visualizations and
results that you will see in the demo
are based on the data sets that we'll be
using we haven't pre-processed the data
to remove outliers or any missing
values now before I jump into the demo
let me show you the second data set that
we are going to use okay so covid
vaccine state wise is my second data set
let me open it there you go
so this is the second data set that
we'll be using in the python project you
can see we have a column called updated
on then we have the state again you can
see here there are a few discrepancies
here uh it has taken the country name
and not the state name below you can see
there are the different state
names and
you can also see we have information
about the total doses administered we
have the sessions sites first dose
administered then we have the second
dose then we have information about male
and female doses you can see here the
different vaccines administered coine
and coov shield you have Sputnik V and
here are the different age groups as
well and finally if you see we have
male individuals vaccinated the total
number of female individuals vaccinated
for a particular day we have information
about transgender individuals vaccinated
and finally we have the total
individuals vaccinated each day all
right before we jump into the Hands-On
part let's have a look at the second
quiz in this
project so here is the second quiz
question which is the first country to
start covid vaccination for toddlers is
it a Japan B Israel C Portugal or is it
D Cuba this is a very recent development
that took place if you watch Daily News
updates on Corona virus you'll be
definitely able to answer the question
please give it a try and put your
answers in the comment section of the
video it is really important for our
viewers to know the right
answer all right so now let's begin with
our
demo so I am on my Jupiter notebook so
so the first project we are going to use
Python jupyter notebook I'll just rename
this notebook as
covid data
analysis
project I'll click on
rename all right so first and foremost
we need to import all the necessary
libraries that we are going to use so
first I'm importing pandas s PD this is
for data manipulation then we have nump
as NP
nump is used for numerical computation
then we are importing mat plot lib cbon
and plotly these three libraries will be
used for plotting our data and creating
interesting visualizations finally I'm
also
importing my date time function all
right so I'll hit shift enter to run the
first cell now it's time to load our
first data set which is related to
the covid-19 cases in India for the
different states and union territories
so I'll create a variable called coore
DF DF is for data frame I'll use my
pandas
library and
then give the read undor CSV function
since our data sets are CSV files and
inside double codes I'll pass in the
location where my dat set a present so
I'll just copy this
location I'll paste it
here we'll
change the back slash to forward
slash and after that I'm going to give
my file name followed by the extension
of the file I'll say
co9 india.
CSV let's go ahead and run it all right
now to see the first few rows of the
data frame I'm going to use the head
function I'll say head and within
brackets let's say I'll pass in 10 which
means I want to see the first 10 rows of
data if I run it there you go here you
can see from zero till 9 so we have 10
rows of information and these are the
different column names you have S number
date time state or Union territory then
you have confirmed IND National
confirmed foreign National cured cases
deaths reported and the confirmed
cases all right now moving ahead let's
use the info function to get some idea
about a data
set if I run it you can see here it
gives us the total number of columns we
have nine columns the total number of
entries or the rows we have 18,1 rows of
information starting from 0 till
18,19 you see here the different types
of variables or column names that we
have then it has information about the
memory usage as
well and this side you can see the data
types
cool now we'll use another very
important function which which is to
get some idea about statistical analysis
the basic statistics about your data set
for that I'll be using the describe
function okay so if you can see
here the describe function is for
numerical columns only and you have the
measures such as count the mean standard
deviation maximum minimum the
25th percentile 50th percentile and the
75th percentile
value okay now let's move ahead and
import the second data set which is
related to
vaccination so I'll create a variable
called vacine _ DF I'll write PD
do read CSV function which is present in
Panda's Library I'll move to the top and
I'll copy the
file location and we'll change the name
of the file copy this and let me paste
it
here and instead of coore 19 India I'm
going to say
covid
underscore
vaccine underscore
statewise okay so this is the data set
that we saw coore vaccine
statewise all right let me run
it
cool and let's display the first seven
rows of information from this data
frame I'll be using the head function
and inside the function I'll pass in
seven there you go so here you can see
we have from 0 till 6 there are total 24
columns a lot of them have null values
you can see
here all
right
now from the first data set which is
the coore DF data frame we'll be
dropping a few
unnecessary columns such as the time
column confirmed Indian national and
confirmed foreign National as well as
the s number we don't need these columns
so it's better to learn how to drop the
columns for our
analysis so I'll say
coore DF dot I'll use the drop
function and within square brackets I'll
pass in the column names the first is s
number I'll give a
comma within double Cotes I'll see the
next column is time my third column that
I want to drop is
confirmed Indian
national give another comma within
double codes I'll
say
confirmed foreign
National outside the square brackets
I'll give another comma and pass in my
next
argument that is in place equal to I'll
say true
we give another comma and say axis equal
to
1 let's run
it okay it has thrown an error let's
debug the error says confirmed Indian
okay it should be Indian national and
not Indian
Nation I'll
just mention it as National we'll run it
again okay now we have
removed these four
columns let me show you the data set
now there you go so we have only the
date column state or Union territory
cured deaths and
confirmed now let's see how you can
change the format of
the date column for that you have the
function called two date time
I'll say coore
DF I'll pass in my column name that is
date I'll say equal to PD
dot I'll use the pandas function that is
2core date
time I'll say coore DF which is my data
frame
name pass in my variable which is date
give a
comma and I'll use my argument that is
format equal
to I'll say percentage
y give a dash say percentage M give
another Dash and say percentage
D let's run it and I'll
print the head of the data
frame cool
now moving
ahead now we will see how to find the
total number of active cases so Active
cases nothing but the total number of
confirmed cases minus the sum of cured
cases plus deaths
reported so
let's find the Active cases I'll give a
comment
okay I'll first write
my data frame name that is coore
DF within square brackets I'll give my
new column which is
activecore
kees I'll say equal
to Kore
DF and My First Column would
be the confirmed cases
column
minus I'll say coore
DF
then I'll pass in my Cod column
plus I'll again
say covid uncore
DF and add my deaths
column this time let's print the
last five rows from the new data frame
that we have
created okay
let's run it okay it has thrown an error
let's debug
it says data frame object has no
attribute tals this should be
tal there you
go can see here we have added a new
column called Active cases which is the
confirmed cases minus the sum of cured
and deaths reported
column now we will learn how to create a
pivot table using the Panda's Library so
in this table we'll be summing all the
confirmed deaths and cured cases for
each of the states and union territories
so we'll be using the pcore table
function for
this I'll create a variable called
statewise and say PD
dot I'll use the pivote uncore table
function
I'll pass in
my data frame that is coore
DF and then I'll give my values
parameter inside the square brackets
I'll pass in
my columns
confirmed
deaths and then we'll have
the cured
column
we give a comma and next argument would
be
index which is going to be my state
slash the union territory
column let me bring this to the next
line so it is more
readable I'll say Union
territory I
will
give a comma
Now and pass in my last argument which
is
AG function that means aggregate
function
and this function would be
Max all
right let's run
it okay now I'm going to find out the
recovery rate so recovery rate is
basically the total number of cured
cases divided by the total number of
confirmed cases into 100 so I'll
say
statewise and within square brackets
I'll pass in my variable that I want to
create which is recovery
rate this will be equal
[Music]
to
the Cur cases
multiplied by
100 by the total number
of confirmed
cases within square brackets I'll give
my column as
confirmed let's run this okay I'll just
copy this column paste it here so this
time we are going to find out the
mortality rate so mortality rate is
nothing but the total number of deaths
divided by the total number of confirmed
cases into 100 so I'm just going
to replace the names here I'll say
mortality all right and then instead of
cured I'll say my deaths column into 100
divided by the confirmed cases let's run
it okay now we are going to sort the
values based on the confirmed cases
column and we'll sort it in descending
order so let me show you how to do it
I'll say statewise equal
to I'll use the function sore
values
so I'll pass in my variable statewise
Dot and use the short
uncore values function I'll say by I
want to sort it by my my
confirmed cases
column give a comma and I'll say
ascending equal
to
false let's run
it now we are going to plot our pivot
table using an
nice visual so for that I'm going to
use my background uncore gradient
function and inside that function we'll
pass in our cmap
parameter I'll show you how to do
it say style
dot
background underscore
gradient and inside this we'll passing a
parameter called cmap so cmap stands for
color Maps this is present inside the
Matt plot Li Library so here you can see
there's a nice documentation on choosing
color maps in matplot lib this is
provided by matplot lb.org
if I scroll down you can see there are a
number of c maps that you can use
purples Blues you have something called
Reds there are other things like magma
you you have summer autumn spring winter
cool all these you can use whichever
color map that you want and here you can
see the different
shades or the
gradients
okay so I'm going to use my color map as
cube
Helix let me run it and show you the
pivote
table there you go here you can see we
have our pivote table
ready now as I said in the beginning
there are a few discrepancies in the
data set so here you can
see there's one called Maharashtra and
there's also Maharashtra Triple
Star this you can ignore even if I
scroll down you have madhia Pradesh
followed by three asteris you can ignore
this value as well even for Bihar we
have so these have been
duplicated and here you you can see the
different state names and union
territories then on the top you have the
confirmed cases cured cases the deaths
reported and the new columns that we
created these are calculated columns
recovery rate and mortality
rate and we have ordered it
in descending order of confirmed cases
so so
far our data sees that Maharashtra has
the highest number of cases followed by
Kerala Karnataka Tamil Nadu Andra
Pradesh and utar Pradesh so these are
the top five states which have the
highest number of confirmed cases even
if you see the mortality rate is also
high for
Maharashtra
and if I scroll down the mortality rate
is also high for utak if you see
here if I scroll further we have Punjab
the mortality rate is also
High all
right so this was our
first visual that we created in the co
data analysis project now moving ahead
we'll see
the top 10 states based on the number of
active
cases so we'll
start I'll give a comment top
10
Active cases
States
okay so we are going to explore another
very important pandas function in this
which is known as Group
by so I'll
first pass in my data frame which is
coore
DF dot followed by the group by
function I'm going to group my data
based
on state SL Union territory
column then I'll say
dot Max which is to find
the maximum value from the
states that
have the highest Active cases so I have
passed in my Active cases
column and we're also going to group it
based on the date
column after that we're going to sort
the values so I'll use my function that
is
sortore
values I'll say short
by my column that
is Active
cases let me bring this to the next
line
underscore
cases I'll give a comma and
say ascending equal
to
false say dot and
then reset my
index for that I'll use reset uncore
index function okay let's check if
everything is
fine I have missed a square bracket here
let me give another square bracket here
okay and all this we are going to store
in
our variable called top underscore 10
underscore
active underscore
cases okay now let me go ahead and run
this cell okay there
is a syntax error
here now let's run it
okay now I'll create another variable
called
fig here we'll pass in the PLT which is
for matplot lib library and will'll give
the figure
size using the fig size
argument say equal to and within a tle
and passing the size let's say 16 comma
9 we'll run
it okay
and let's
give a title to our plot so here we are
going to create a bar
plot
so using PLT do title we'll pass in the
title let's say top 10
states with
most active cases in
India I'll give a comma and passing the
size of my title that's a
25 okay you can see we are
slowly being able
to create our graph the most important
thing is to pass in the x- axis and the
y
axis I'll say ax which is for axis to
pass in the axis I'm going to use the
barplot function that is present inside
the cbor
Library I'll say SNS do
barplot I'll say data equal
to my variable which is topor 10
underscore
stits
this is actually active
States all right and now I'm going to
use the iock
function and take the first 10 states so
I'm using a colon and then giving 10 as
my value IC is for index
location I'll give a comma and see my y
AIS To
Be Active
cases give another comma and say my
x-axis to
be
state
slash Union
territory give another comma we're going
to pass in the line
width to two let say line width equal to
two and we'll give an edge
color let's say the edge color
is
red okay so I have my Axis defined now
let's run
it there's an error here it says stopped
in active States not
found let's go to the stop and see the
exact okay so this is top 10 Active
cases let me change it
to cases
here now run it okay the x-axis also has
a mistake this should be State /un
territory now let me run it there you go
we have
our plot created but as you can see here
the labels of the different states and
union territories are overlapping so for
that let me first pass in the X labels
I'll say PLT
dox
label and my X label would be
States then I'll see PLT do y
label my y label will have the
total Active
cases and finally I'll write PLT do show
before I run it let's collate all the
lines of code that we have written for
our top 10 states for most active cases
in India into one cell so what I'll do
is I'll just copy
this and we'll keep
adding in this cell
itself
okay I'll go to the top next we'll copy
my figure size and
we'll paste it
here next let's take
the title and put it
here give a
space and we are going to
copy this cell and I'll paste it
here all
right now it's time to run
it there you go you see
here we have a nice bar plot ready on
the top you can see the title top 10
states with most active cases and you
see the edges are in red color for all
the bars on the x-axis you have the
different state names Maharashtra
Karnataka Kerala we also have Andra
Pradesh Gujarat West Bengal and chattis
Gad so as you can see Maharashtra has
the highest number of active cases based
on our data followed by Karnataka Kerala
in Tamil Nadu at second third and fourth
place respectively and in the ninth
place we have West Bengal in the 10th
place we have chatis
card on the y axis you can see the total
Active cases which are in
lacks okay now moving ahead now we'll
see the top 10 states based on the total
number of deaths reported so I'll give a
comment top states
with
highest
deths okay so I'll first create my
variable saying top 10
uncore
TS this will be very similar
to what we did here just that we need to
change a few column names so instead of
Active cases we'll be using
deaths
okay so I'll start with my data frame
that is coore DF followed by using the
group by function and I want to group my
data based
on the state
SL Union territory
column then I'll choose the max function
function and
within double square brackets I'll pass
in my column names deaths and
date let's make it consistent I'll be
using single
Cotes okay I'll say
dot going
to S my result therefore I'm using the
sore values function I want to S it by
deaths
column I'll give another comma and
say ascending equal to false which means
I want to order my result
in descending
order then I'll say do reset uncore
index
okay after that I'll give
my figure size I'll say PLT
figure and within brackets I'll pass in
my argument which is fig size equal to
using a tle I'll say 18 comma let's say
5 now let's give a title for that I'll
use the title function which is present
in the mat plot lab Library I'll say top
10 states with
most
depths
and let's give a size to the title
25 now it's time for us to give the
access labels I'll just scroll
down okay so I'll say ax equal
to again this will be a barplot so I'm
using my cbon library followed by the
barplot
function my data will be
top underscore 10or deaths which is this
variable
I'll
give my index location i
l I'm going to
choose 12 States the reason being there
are some discrepancies in the data I'll
show you once I plot this result so in
the y axis we'll
have deaths
column in the x-axis we'll take state
slash Union
territory okay I'll give another
comma and
say line
width equal to
2 and we'll give an edge color to our
bars like we did
here so I'll say Edge color equal to
Let's
say
black okay
now finally we'll
give the X label the Y
label so I'll say PLT do X label X label
would be
States my y
label will
be total
death
cases and I'll write PLT do
show now let's run
it there you go you can see here we have
a nice bar plot on the top we have the
title top 10 states with most deaths now
what I specifically wanted you to see
was these discrepancies in the data you
can see here Maharashtra is repeated
twice even in our data that we collected
from Keel Karnataka spelling has an
error you can see here we have a few
rows of information where Karnataka is
spelled as k a r a a n instead of k a r
n a all right so
to
remove these two results or to ignore
these two results I had given my index
location till 12 so we have Maharashtra
Karnataka Tamil Nadu the then utar
Pradesh West Bengal Kerala Punjab Andra
Pradesh and chattis
Gad with the states that have the most
number of deaths
reported okay now we'll create a line
plot to see the growth or the trend of
Active cases for top five states with
most number of confirmed cases so the
states are Maharashtra Karnataka Kerala
Tamil Nadu and uttar Pradesh I can show
you so these are the states with highest
number of active
cases okay hello everyone welcome to
today's video tutorial by simply loar in
this video we are going to perform a
really interesting data analysis using
python on the Spotify music streaming
service platform data set I'll also be
asking you a few questions related to
Spotify during our discussion please
make sure to answer them in the comment
section of the video so now let's get
started Spotify is a stest audio
streaming and media services provider
founded in April
2006 it is the world's largest music
streaming service provider and has over
381 million monthly active users which
also includes 172 million paid
subscribers the total number of
downloads on the Spotify app in the
Android store exceeded 1 billion in May
2021 so millions of people listen to
music all day even I'm hooked to music
as an analyst what's better than
exploring and quantifying data about
music and drawing valuable insights
before I move ahead I have a quiz
question for you people the name Spotify
comes from a combination of two words so
which are those two words please let us
know your thoughts in the comment
section below I would like to repeat the
question again the name Spotify comes
from a combination of two words so what
are those two words we would love to
hear from you so please put your answers
in the comment section
below now let's use python libraries and
functions to analyze and visualize our
data set first I'll show you the two
data sets that we'll be
using so here is the first data set that
we'll be using for our demo and then I
have my second data set called Spotify
features which is essentially about the
geners of the different
soundtracks now these data sets have
been downloaded from kel.com now the
Links to the data sets have been
provided in the description box please
go ahead and download them now let me
just go ahead and brief you about the
columns that are present in our first
data set which is about tracks we have
our column A which is ID this is the
unique ID for each of the songs then we
have the name column which is
essentially the name of the song then we
have a column for popularity so the
popularity ranges from 0 to 100 then we
have duration in milliseconds this is
the duration of the track in
milliseconds next we have a column
called explicit now we are not bothered
about this column because we are not
going to use it in our
analysis then we have artist so the name
of the artist who has composed or sung
the song then we have ID of the artist
then we have a column for release date
which is basically the date on which the
song was released then we have a column
for
danceability so this describes how
suitable a track is for dancing based on
a combination of musical elements such
as Tempo Rhythm stability beat strength
and overall regularity the value ranges
between 0 and
1 next we have a column for energy so
the energy is a measure between 0.0 to
1.0 and represents a perceptual measure
of intensity and activity typically the
energetic tracks feel fast loud and
noisy higher the value the more
energetic is the sound then we have a
column for key so key is the pitch notes
or scale of song that forms the basis of
a song there are 12 Keys ranging from 0
to
11 moving ahead we have loudness so the
overall loudness of the track in decb it
ranges fromus 60 to 0
DB then we have mode so songs can be
classified as mus and minor 1.0
represents major or one represents major
and zero represents
minor next we have speech so speech
recognizes the presence of spoken words
in a track more exclusive speech like
the recording example talk show audio
book or poetry the closer to 1.0 the
attribute
value then we have a column for a costic
thickness so a confidence measure of 0
to 1 of whether the track is acostic or
not so 1.0 represents high confidence
the track is acostic then we have other
information about
instrumental then we also have a column
for liveness so liveness detects the
presence of an audience in the recording
then we have a column for Valance so
Valance is a measure between 0.0 to 1.0
and describes the musical positiveness
conveyed by a track or a
song and finally we have the columns for
Tempo and time
signature now even in the second data
set we have almost the same columns just
that we have an additional column that
is about the genre of the songs present
in the data set cool now let's head over
to our jupyter notebook and we'll start
with our
analysis okay so one more thing to
remember our data has information
from 1922 onwards so all the songs from
1922 till
2021 cool okay so I'm on my jupyter
notebook
so you can see I have a few cells that
have already been filled up so we'll
start with our analysis first of all
let's go ahead and import the necessary
libraries so I'm importing numpy pandas
matplot lib and cbon for my analysis and
visualization I'll hit shift enter to
import the libraries all right and in
the next cell I'm going to load my data
set using the pandas read CSV function I
have my location already put here let me
show you the location where the data
files or the data sets are located so
this is my location under Chrome
downloads I have a folder called Spotify
data sets Okay so let's import
and check the first five rows in the
data set so for that Ive used the head
function there you go so here you can
see I have my first five rows of
information from the data set and on the
top you can see the different columns
you have ID name
popularity
artist then we have the release State
danceability energy key
loudness liveliness or
liveness balance Tempo and other
information
cool now let's check for null values in
the data set I'll just give a comment as
null
values every time when you download a
data set from an open repository there
are chances that the data set would
contain null values so it's better to
check them
beforehand so I'm going to use the isol
function present in the pandas Library
PD I'm using because I had imported
pandas as PD so PD do is null then I'm
going to use the variable name dfdore
tracks because I imported my data set
and stored it in the variable dfdore
tracks so I have my data frame under
dfdore tracks
varable and then I'll use the sum
function to check the total number of
null values present in the data set for
each of the columns if I run it there
you
go so here you can see my name column
has
71 missing values or null values and we
don't have any null values for the rest
of the
columns
okay now let's use the info method that
will give us the total number of rows
and columns in the data set and we'll
also check the data types and the memory
usage so I'll say my data frame name
that is DF TRS
doino if I run it you can see here now
if you mark there are total
58661 names or the names of the songs
present in the data set while the rest
all have 5 l
clearly there are total 71 song names or
soundtracks missing from our data
set and Below you can see the data types
we have float integer and object and
then you can see the memory
usage
cool now before I move ahead with our
next analysis I have another question
for you which artist or musician has the
most number of followers on Spotify I
repeat which artist or musician has the
most number of followers on Spotify by
please put your answer in the comment
section below we would be happy to hear
from
you now let's move ahead and do our
first major analysis in this demo we are
going to find the 10 least popular songs
present in the Spotify data set so I'll
create a variable called sorted DF that
will be equal to my data frame name that
is DF tracks dot I'm going to use the
sortore values function and
say my column name is
popularity so I'm going to sort the
values based on the popularity and then
say ascending equal to
True since I want only the least popular
songs
and then I'm going to say head of 10
which means I want the top 10 least
popular songs now let's go ahead and
print sorted DF if I run it you can see
we have the list
of 10 least popular songs on Spotify you
can see that popularity is
zero and you can see the names of the
songs some of them are
songs which are not an English
language and you can see the artist
names as well
cool now moving
ahead let's see some descriptive
statistics for the numerical variables
that are present in our column so I'll
say DF TRS do
describe which is the function to get
some descriptive statistics and I'm
going to use the transpose function
after
that if I run it there you go so we have
the statistics about count mean standard
deviation minimum value 25th percentile
50th percenti 75th percentile and the
maximum value for these columns like
popularity duration in milliseconds then
we have energy key loudness
mode cool now if you see this popularity
column the minimum value is zero and the
maximum value is 100 and you can see the
50th percentile is 27 which is
essentially the median you have the
standard deviation as
18.37% most popular songs which are
greater than 90 so we are going to check
for the top 10 songs with popularity
greater than 90 let me show you how to
do it so in this cell I'm going to
create a new variable called most
popular and I'll say
dfdore tracks dot this time I'm going to
use the query function that is part of
pandas Library again I'll use the
column that is popularity and we'll set
the condition popularity should be
greater than 90 I'll give a comma and
say in place equal to false because I
don't want to change my original data
frame and then I'll say
sortore
values and I'm going to sort it based
on
popularity
in descending order so I'll
say ascending equal to
false and then
let's take only
the top 10 popular songs so I'll say
most popular use square brackets use
square brackets and we'll then pass the
slicing operator and say colon 10 now if
I run this there you go so here you can
see the 10 most popular songs that is
present in our Spotify data set first we
have features by Justin
Bieber Daniel Cesar and gon then we also
have a song name called driver's license
astronaut in the ocean save your tears
you also have the business streets and
heartbreak anniversary so the these are
the most popular songs that are present
in our data set based on the popularity
you can see paches has the highest
popularity
with 100 all right now moving to the
next cell so here we are going to set
the index to be release date column in
the main data frame so I'm setting my
index using the setor index function I
have passed in my column name as release
and I'm saying in place equal to True
which means I want to change it in my
original data frame and then I'm
changing the value to date time format
and let's print the head of the data set
so you can see here we have successfully
changed our index now here you can see
instead of 0 1 2 3 we have the release
date column and the rest of the columns
are
intact
cool now let's move ahead
so suppose you want to check the artist
at the 18th Row in our data set you can
use the index location method for that
let me show you how to filter only
specific rows of information from the
data set I'll use my data frame DF
uncore tracks and using double square
brackets I'll
say my column name which is
artist and I'll use the index location
method and and say let's say I want to
check the artist who is
present in the 18th Row in my data set
so I'll use iock 18 if I run it the
artist name is Victor
Boucher
cool now let's move ahead we are going
to convert the duration in milliseconds
to just seconds
so if you see our data set we have a
column called duration in milliseconds
so so all our songs are present in
milliseconds let's convert them into
just seconds so for that I'm using the
Lambda function and dividing the values
in milliseconds by th so that they get
converted into just seconds I have used
in place equal to true so I want to
change it in my original data frame
let's run it and do the necessary
changes all right now we'll print the
head of the data set just to check the
duration column so I'll say
dfdore
tracks dot
duration do head if I run it there you
go
so you can see the values have now been
changed to just
seconds
cool I have the final quiz question for
you who has the most monthly listeners
on Spotify please put your answers in
the comment section below we would be
glad to hear from
you now coming to the next
cell so here we are going to create a
first visualization that is going to be
a correlation map we are going to drop
three unwanted colums and those are key
mode and explicit and then we are going
to apply Pian correlation method now I
have set my figure size to 14a 6 and
then we are using the cbon heat map
function to create our correlation map I
have put
the variable that is correlation _ DF
you can see above we had created this
and then I'm setting annotation equal to
true so this will write the data value
in each cell I have set fmt equal to do
1 G so this is a string formatting code
to use when adding annotations then I
have set my vmin and Vmax so these are
the values to Anchor the color map
otherwise they are inferred from the
data and other keyword arguments cmap
here stands for color map you can just
search for SNS C map you will get the
documentation so you can choose
whichever color palette or the color map
you want here I have used infoo
and I have set my line BDS and line
color finally I'm giving a title to my
correlation map and I have set the xtic
labels let's go ahead and run this to
get our first
visualization if I scroll down there you
go we have got
a nice correlation map so here on the
right side you can see the scale it
ranges from minus1 to + 1
minus1 means the variables have least or
negative correlation while the values
which are above
0.0 means that the variables have a
positive
correlation
now here you can see there
are values like minus. 7 for energy and
acostic which means if the energy is
high the acostic is really low again for
loudness and say speech if the song is
loud the speech is low so there is
negative correlation but if you see for
energy and loudness there is really high
correlation between these two variables
you can see the value is8 so if the song
is loud this implies the song has really
high energy and vice
versa now if you see for a few other
video there is negative correlation
between acostic and
danceability there is negative
correlation again between Valance and
acostic similarly there is positive
correlation between energy and bance
which
is4 and even for disbility and bance
that is positive correlation which is
0.5 cool so from the correlation heat
map you can note that acostic appears to
have a strong negative correlation with
energy so if you see for acostic and
energy there is a strong negative
correlation and there is a moderately
strong positive relation between
loudness and popularity so if you see
for popularity and loudness the color is
orange which means it lies in this
positive
region and there is also moderately
strong positive relation between
danceability and bance so if you check
for Dan ability and balance here there
is a moderately strong positive
correlation all
right now let's move ahead we are going
to sample our data and take just 4% of
the total data and we'll create two
regression plots using this data so let
me first sample my data so I'll create a
sample data
frame using my original data frame which
is dfdore tracks and say
sample I'm going to use the int function
and see
0.04 multiplied by the length of my
original data frame which is
dfdore
tracks all
right now let's run this
and will
print the length of my sample data
frame if I run it you can
see 4%
of my total data set
is
2346
rows
cool now we are going to create a
regression plot between loudness and
energy now in our correlation map we saw
there was a positive correlation between
loudness and energy which was point8
let's plot it in the form of a
regression line so I'll use PLT dot
figure I'll set my figure
size equal to let's say 10 comma 6
I'll say SNS
dot regression plot so I'm using the
function called reeg plot and I'll use
my data as sample
underscore
DF give a comma and
say in my Y axis I'll have my
column
loudness and in the x-axis we'll have
energy
I'll give a color
to my data variable let's say the color
is
C then I'll set my title
to
loudness versus
energy
let's say
correlation all
right let's make sure everything is
fine now we'll run and see our result
there you go so you can see here clearly
there is a very high positive
correlation between loudness and energy
on the y axis we have loudness and on
the x-axis we have energy and you can
see all the data points or the songs
are in One Direction so if the energy
increases the loudness of the song also
increases and similarly if the loudness
of the song decreases your energy of the
song or the track also decreases so
there is a very high positive
correlation and you can see the
regression line here it has gone
and is increasing
gradually
cool now similarly I'll just copy this
code and we are going to see another
regression plot this time for two
different features let's say we
have popularity in the Y AIS so I'll say
popularity and then in the x-axis we
have
let's say
acostic I'll change the color to let's
say B which stands for
blue and we set the title
to
popularity
versus I'll have acos
thickness
correlation just scroll down and we'll
run it to see the
result there you go so I have
the different points for the songs and
here you can see the regression line is
downwards which means if the acostic of
the song increases the popularity
decreases and similarly if the
popularity increases acostic decreases
you can see the downward trend of the
regression line
all
right now in the next cell we are going
to create a new column called Year from
our release date column and I have
changed this to date time format let me
just run it
cool now after
that we are going to create a
distribution plot to visualize the total
number of songs in each year since 1922
that is available on the Spotify
streaming app
so I have used my cbon library and disc
plot function now one thing to remember
you need to update your cbon library
to this
version if you haven't done it so use
this command pip install D- user cbone
and the version so here in the
distribution plot we're going to plot a
histogram so Ive used kind equal to hist
which stands for
histogram let's run and see the result
okay so here you can see I have
my distribution plot so the plot tells
us that the number of songs for each
year in the data set according to their
release date have increased in the
recent year since music became more
accessible to people globally with
technological advancements so earlier
you can see there were very few songs
available in the
1920s later on the number of songs
increased rapidly and now you can see we
have a lot more songs available for
people to listen
cool
now we are going to see the duration of
songs over the years for that again we
are going to create a bar plot so I'll
first create a variable called total
duration equal to
dfdore tracks
dot I'll use the duration column that we
created with
seconds and
then I'm going
to
set
my figure Dimensions so I'll use figure
uncore DMS for Dimensions equal to 18
comma 7 7 after that I'll have my figure
axis defined so I'll use the M plot lib
subplots
function
and I'll set
the figure
size equal to my figure
dimensions and then
I'll say figure equal to SNS dot I'll
use the bar plot
function and say my x-axis to be
yours my Y axis will be total duration
or total _ Dr that we created here I'll
set my Axis equal to ax
and then I'll set error
width equal to
false let's set the title for my plot
as title equal
to your versus
duration and finally I'll say PLT do
xtex I'll rotate it
by let's say
90° all
right let me just recheck once if
everything is fine and then we'll go
ahead and run
it my access error width
title okay let me just run
it we'll see the result in a
moment if I scroll down you can see see
we have the bar plot for the different
years and the duration of the songs in
seconds in the y axis so earlier in the
1920s you can see the duration of the
songs were less and later it increased
around late
1930s and this remained consistent until
2010 where the duration was high but
after 2010 you can see the duration of
the songs have started
decreasing now in the next cell I'm
going to create a line plot to analyze
the average duration of the songs over
the years it is going to be similar to
our bar plot just that now we are going
to visualize it in terms of a
line so I have my code ready you can see
here I've used my cbone library and the
line plot function in the x-axis I have
ears and in the y axis I have total
duration I've set my title to year
versus duration and I'm rotating
my X labels by 60° let's run it and
we'll see the
output there you go now if I scroll down
you can see we have a
nice line plot and on the x-axis you
have the
years and on the y axis we have the
duration we can see that the songs from
1920s to
1960s have comparatively shorter
duration since most of the songs tended
to be more singing based rather than
instrument
based after 1960s you can see the
duration of the songs started increasing
until I would say
2010 and in the present day the duration
of the songs have started declining
since the attention span of the average
listener is also declining all right now
let's move to our second data analysis
project which is based on genr of the
songs so I'm importing my data set using
the pandas read CSV function I have
given my location and here I have my
data set name followed by the extension
of the type so this is a CSP data set
let's go ahead and run
it okay now let's print the first five
rows of the data set I have stored my
data set in a data frame called dfdore
Jer I'll use the head function to get
the first five rows of
information I'll hit shift enter to run
it there you
go you can see see here I have my genre
column artist name track name track ID
popularity
acostic duration in milliseconds again
and we have the rest of the other
columns that we saw in our first data
set just one thing to note here key is
present in terms of c d e c minor F
minor and not in terms of numbers
between 0 to
11
now we'll
see the duration of the songs for
different genes for that I'm going to
create a bar plot so I'll start with
setting the title for my plot as
duration
of the
songs
in
different geners
I'll use the cbon
library and I'll set
my color palet to let's say
rocket you give a comma and say as cmap
equal to
true this would be color pallette
now I'll say SNS do
barplot in the y axis I'll
have
Jer and in the x-axis I'll
have my duration column which is in
milliseconds so duration underscore Ms
and then I
pass my data frame using the data
argument so I'll say data equal to DF do
Jer
next we'll set the x labels so I'll say
PLT do X
label let's say my X label is duration
in
milliseconds and then I'll say
PLT do by label as
Jers now let's go ahead and run
this there you go so here you can see we
have the different geners on the y axis
and on the xaxis you have the duration
in milliseconds and if you see the graph
for classical genre and for songs that
belong to World genre the duration of
the songs are more compared to other
geners now if you check for children's
music genre the duration is less or the
least
cool and
finally we'll move to our last demo
where we'll see the top five geners by
popularity so I'll see SNS do
setor
style I'll set
my style
to
dark
grid which will be my
background and then I'll say PLT
Dot
figure I'll set my
figure size
to 10 comma
5 then I'll create a variable called
famous since I want to take only the
most popular songs based on the JRE so I
have created a variable called femus and
I'll pass my data frame name that is
dfdore joner
and I'll sort the values based on my
popularity
column so I'll say
popularity and I want to sort it in
descending order so I'll say ascending
equal to
false and I'll take
the
first 10 values I'll tell you the reason
why I'm taking the the first 10 values
are not
five then I'll say SNS do
barplot and in the y axis I'll have
Jor in the x-axis we'll
have
popularity and I'll give a comma and
using the data argument I'll say my data
to be famous which is this variable that
we
created and then I'll set
my
title
as top five
geners by
popularity all right now the reason why
I took head of 10 is because there are a
few geners which are repetitive so if
you see this we have children's music
appearing twice so hence we have taken
10 instead of five let me just go ahead
and run it there you go so here if I
scroll down you can see I have my top
five geners based on the popularity so
we have dance pop rap hip hop and Raton
so these are the five geners which are
most popular based on the data that we
have collected from
Spotify all right
hi everyone welcome to this really
interesting video on data analysis of
the 2021 World happiness report using
python so today we will perform some
exploratory data analysis using python
libraries to analyze visualize and draw
insights from 2021 World happiness Data
before I begin make sure to subscribe to
the simplyone channel and hit the Bell
icon to never miss an update first let's
understand what the world happiness
report 2021 is all about the internet
ational happiness day is celebrated
every year since 2013 on 20th of March
to emphasize the importance of happiness
in the daily lives of people so the
United Nations sustainable development
Solutions Network published the world
happiness report on 19th of March 2021
that ranks the world's 149 countries on
how happy the citizens perceive
themselves to be based on various
indicators the happiness study ranks the
countries on the basis of questions from
the Gallup World Poll the result had
then equated with other factors such as
GDP life expectancy generosity Etc this
year it focused on the effects of the
covid-19 pandemic and how people all
over the world have managed to survive
and
prosper so using this 2021 data we will
answer critical questions such as the
top 10 most corrupt
countries we will plot a graph to
understand how the happiness score is
related to the freedom of making Life
Choices we look at the life expectancy
of 10 happiest and 10 least happy
Nations so these are a few examples but
we will explore more about the data in
detail in our demo
session let's get started so first I'll
show you the data set we'll be using in
this demo so this data has been
collected from keigle let me show you
that so this is the CSV data set that we
have downloaded from giggle.com so you
can see here World happiness report
2021
and we will share the data set link in
the description of the video you can
click on the link to download the data
set now we have information about 149
countries you can see it here count is
149 let me go to the top and I'll run
through the columns that are there in
this data set so the First Column is the
country name so we have 149 different
countries and then we have something
called as Regional indicator we can call
this as just the region so you can see
we have different regions I've applied a
filter
we have Central and Eastern Europe then
we have Commonwealth of independent
states so these include countries such
as Russia then we have East Asia Latin
America and Caribbean we also have South
Asia southeast Asia subsaharan Africa
Western Europe and other regions I just
cancel this and then we have the
happiness score column that has been
sorted in descending order so we have
Finland Denmark and Switzerland who are
the top three happiest Nations now if I
scroll down we have countries like Randa
Zimbabwe and Afghanistan which are the
least three happy
countries now there are a few columns
that we won't be using in our analysis
so we will learn how to exclude those
columns and keep only the relevant ones
so columns such as as standard error of
ladder score then we have upper whiskers
and lower whisker column so we are going
to ignore these columns we are only
concerned about the GDP column which is
this one then we have the social support
or the social status column then we have
the health life expectancy freedom to
make Life Choices generosity and
perceptions of corruption
and there are a few other columns you
can see to the right and these columns
are not of our interest so we are going
to ignore them now let's head over to
our jupyter notebook and we'll start by
importing all the necessary libraries
for analysis and data
visualization okay so I'm on my Jupiter
notebook so first step I'll just rename
this notebook to let's say
happiness report data
analysis I'll click on rename all right
now we'll start by importing our
libraries so first Library I'm going to
import is nump as
NP then I have
import pandas as
PD then I'll import two data
visualization libraries cbon and matplot
lip so I'll say
import cbon as SN s and
then
import matplot
lib dot pip plot which is the module
name as
PLT and I'll
say
percentage mat plot
lib in
line okay now let me just go ahead and
run
this all
right well now let's set the parameters
that control the general style of the
plots the style parameters control
properties like the color of the
background and whether a grid is enabled
by default or not so for that I'll say
SNS do
setor style I'll give it as dark grid
next I'll
say PLT do
RC per
amps which stands for runtime
configuration parameters I'm going to
set my font
size
to let's say
15 then I'll say
PLT
dot
RC
perams now I'm going to set the figure
size so I'll say figure
dot fix
size let's say 10 comma
7 next
I just copy
this paste it
here now we are going to set the face
color so I'll
say figure
dot face
color I want to set it to peach color so
I'm going to pass in my RGB values for
peach I'm going to set it in terms of
hex code so for peach the value is
FF
e 5 b and
4 now let me run it
okay now it's time to load our data set
so for that I'll create a variable
called data and I'll use the pandas
Library followed by the read undor CSV
function because our data set that we
saw is a CSV data set which is this one
now inside the parenthesis I'll pass in
the location of
my data file so I have my data here
World happiness report 2021 I'll just
copy this location and we paste it here
and make sure the location is within
quotes and you need to change it to
either forward slash or double back
slash so here I'm using double back
slash so let me just include one more
backs
slash and then I'm going to pass in
the file name which
is
World
hyphen
happiness hyphen report hyphen
2021 do CSV which is the extension of
the file now let me run it all right now
to display the first five rows of
information you can use the head
function so I'm writing data which is my
variable that holds the data frame so
data do head there you go you can see
here we
have printed the first five rows from
the data set you have the country name
Regional indicator happiness score
then we have information about the GDP
life expectancy generosity then we have
corruption data and these are some of
the columns that we are not bothered
about so we are going to drop these
columns from our
analysis now we going to do that so I'll
create a variable called Data columns
which
are of our
interest so I'm going to take only
specific columns I need the country name
so I have taken country name make sure
the column names are within single codes
so I have my country name next I want
the second column which
is regional
indicator give a
comma we also need the happiness
score
next I need
the logged GDP per capita data so I'll
take that column I'll say logged
GDP per
capita give a comma my next column would
be social
support give a comma
here my next
column would be Health life expectancy
so I'll write
Health life
expectancy let's give another comma and
we'll take the next column as well which
is freedom to make Life Choices so I'll
write that column name freedom to
make life
choices
and
finally we'll take
the next two columns that is generosity
and perceptions of corruption So within
single quotes I'll
say
generosity give a comma and we're going
to include the final column which is of
our interest that is perceptions of
corruption let me have a
recheck to ensure that I have put the
column names correctly otherwise it will
throw an
error now let me go ahead and run this
cell I'll hit shift enter all right so
we have
successfully taken the columns that
we'll be using for an
analysis now I'm going to say data equal
to data I'll pass in my new variable
that is data score columns and I'm going
to copy all the data so I'll say dot
copy let's run it okay this should be
data _ columns all
right
now let's rename all these columns we'll
make it more simpler
and easy to understand so I'll
say let's say my
new variable is happycore D which stands
for data frame equal to I'll say data
dot we'll use the rename function and
using a dictionary we will rename our
columns so I have used a curly
bracket I'm going to pass in
My First Column which is country name
I'll just paste it here then I'm going
to give a colon and again Within single
codes I'll say country uncore name so
this is going to be my new column name
I'll give a
comma we'll take the
next column which is regional
indicator I'll paste it here and give a
colon and the new column would be small
R Regional
underscore
indicator now similarly we'll do this
for all
the remaining columns in the data
set okay now I have renamed all my
columns you can see it here let's run
it now we're going to display the head
of the data set again so I'll say
happycore DF do
head there you go so we have only those
columns that are of our interest so I
have the country name Regional indicator
happiness score GDP social support life
expectancy freedom to make Life Choices
then I have generosity and perceptions
of
corruption
cool
now we are going to check whether any of
the columns have any null values so for
that I'll say happycore
DF
dot I'm going to use the isol
function I'll give another Dot and we
are going to find the sum for each of
the colums you can see from the data we
do not have any null values in any of
the columns in the data set it is all
zero okay now let's get started with our
first visualization that is we'll create
a plot between happiness score and the
GDP
for different
regions so for that I'll give a comment
as
plot
between
happiness and
GDP I'll just scroll down
cool first I'm going to set
the RC parameters so I'll say PLT do RC
param
within square brackets I'm going to give
my figure size so I'll say figure dot
fix
size equal to let's say my figure size
is
15 comma
7 I'll set
the title that is PLT do title of my
plot
to let's say plot
between happiness
score and
GDP next I'm going to say SNS dot let's
create a scatter plot so I'll say SNS do
scatter
plot I'm going to Define my xaxis and
the y axis for the plot let's say in the
x-axis we
have my data frame
happycore
DF
dot this should be an
underscore I'll
say a column name as happiness uncore
score give a comma and in the y axis
we'll
have my data frame name that is Happy _
DF dot we'll have the GDP column that is
logged uncore
GDP underscore peror
capita let's give a comma and we'll pass
in Hue
for the color let's say for Hue I'm
going to use the regional column or the
regional indicator column so I'll say
happy _
DF Dot
Regional underscore
indicator and then I'm going to give the
size of the dots as let's say 200 then
I'll give a
semicolon come to the next line I'll say
PLT dot let me just scroll down now we
are going to define the legend so I'll
say PLT do
Legend and in Legend we'll have
have the location let's say I want to
put the legend at upper left corner so
I'll say Lo which is for location equal
to upper
left give a comma and then say font size
of my Legend let's say B
10 make sure this is within
quotes then I'm going to pass my x-axis
labels and the y-axis labels so I'll say
p dox
Lael let's say the X label is happiness
score and my y label is GDP per capita
so I'll say PLT doy
Lael within single codes I'll say GDP
per
capita uh there's an error here this
should be plot all right so I have
written my code to create a scatter plot
let's run it and see the
result there is some error
here okay this should
be
Regional indicator and not indicatory
there's a spelling mistake let's run it
again all right so here you can see we
have a nice scatter plot on the top you
can see we have the title of the plot
that is plot between happiness score and
GDP on the x-axis we have the happiness
score from 0 to 8 and above and on the y
axis you have the GDP per capita and if
you see here in this region we
have countries from Western Europe which
have
the highest happiness score and the GDP
per capita is also the highest around
this region you can see here which is
for green and and in the legend you can
see green is for subsaharan Africa so
all these countries have low happiness
score and even the GDP per capita is
also
low and if you see
the countries for Latin America and
Caribbean you see a lot of the values
lie here so they are all within the
range of 5.5 to 7 in happiness score and
even the GDP per capita is more than
nine for most of
them now even the happiness score is
high for the countries that lie in the
North America andz region and even that
GDP per capita is also the highest I can
name a few countries such as Australia
New Zealand we have Canada and United
States of America which belong to the
north America and Enz region cool now
there is one country which you see here
this seems to be like an outlayer which
means that this is the country which has
the lowest happiness score and even the
GDP per capita is also low but it is not
the lowest because you can see here
there are a few countries up here from
the subsaharan Africa which have the
lowest GDP per capita but the happiness
score is higher than this value or this
country so we can assume that this
country is Afghanistan which has the
lowest happiness score as per the 2021
happiness report
data
cool now we'll plot a p plot to
understand the GDP by region so by this
we can know which region has the highest
percentage contribution to the world's
GDP as per our data so for that I'll
create a variable GDP uncore
region equal
to we'll use our data frame that is
happycore DF dot I'm going to use the
group by
function and after that I'm going to use
my column that is
region so we have named the column as
Regional uncore indicator so I'm going
to group it by this region
column and I'm going to sum the values
of
GDP so I'm going to use the logged
underscore GDP underscore per
capita
column and after that I'm going to use
the sum
function and let me just print GDP
uncore
region you can see it here we have
the sum of all the countries for
different regions and their GDP all
total now this data we are going to plot
it in the form of a p plot so I'll say
GDP uncore
region Dot
Plot dot pi
we are going to plot it in terms of
percentage so I'm going to use a
parameter
called autop
PCT equal to then I'm going to pass in
my format so I'll say percentage
1.1 F percentage
percentage then I'll say PLT do
title let's see the title of my P plot
is going to be GDP by
region and I'll say PLT
dot by
label which is going to be
blank let's run it okay so here you can
see we have the peach background at the
back because we
had
assigned a peach face color you can see
it here
so for the
first scatter plot also we had the peach
color at the back and now you can
see we have subsaharan Africa
contributing 20.7% to the world's GDP
the reason being we have around 34
countries in the subsaharan
Africa and we have the Western European
countries contributing to 16 .2% of the
GDP
to check the least we have North America
andz region because we only have four
countries America Australia Canada and
New Zealand so hence they are
contributing only 3.1% to the world's
GDP okay now moving
ahead let's find the total number of
countries in each region so for this we
are going to use the group by function
that is part of the pandas library and
we'll count the total number of
countries in each region
so I'll just give a
comment as total
countries all right just scroll
down so I'll create a variable called
total underscore
country I'm going to use my data frame
that is happycore DF dot I'm going to
use the group by
function I'll group the value values
based on the region column So within
single quotes I'll say Regional
uncore
indicator and then I'm going
to find the total count of country name
so I'm using the column country uncore
name and after that I'll just use do
count now let's go ahead
and print my variable that is total
score
country all right now let me hit shift
enter okay so here you can see all right
so for subsaharan Africa there are total
36 countries and not 34 as I mentioned
earlier so hence you can see because it
has the highest number of countries it
is contributing the most to the world's
GDP that is
20.7% then we have the least number of
countries in North America andz that is
only four we have six countries in East
Asia and then we have 20 countries in
Latin America and Caribbean 12 countries
in Commonwealth of independent states
then we have 17 countries in Central and
Eastern Europe
cool
now I'm going to show you how to create
a correlation map so that we can see the
relationship that exists between each of
the variables that are present in our
data set I'll just run through the code
and we'll see the
output okay so here I have my code
written for the correlation map that I
want to create so first of all I'm going
to compute the correlation Matrix so I
have used the C RR function which stands
for correlation and the method I'm going
to use is peon method now here I have a
Wikipedia page opened for peon
correlation coefficient so this is a
nice article where you can understand
what the Veron correlation is all about
and then I'm going to set up the matte
plot lib figures so I have used the
subplots function and I've have given
the figure size as 10A 5 and after that
we are going to draw the heat map with
the mask so I'm using the heat map
function present in the cbon library and
then I have passed in my variable that
is co which essentially stands for
correlation and then I have used
mask which is a Boolean array or it can
be a data frame and it is an optional
parameter if it is passed the data will
not be shown in cells where the mask is
true and the cells with missing values
are automatically masked after that I
have used the cmap parameter which
stands for color map so you can
customize the colors in your heat map
and I'm going to create a heat map which
is in squares shape so I have said
Square equal to true and the axis I'm
going to pass it as ax which I have
defined here so this is the mat plot La
axis and it is optional so axis in which
you want to draw the plot otherwise you
can use the current active access now
let me just go ahead and run this and
we'll see the heat
map there you go so if I scroll down
here we have
the correlation Matrix and since I had
given my C map as blue and here you can
see it is mostly blue and we have the
scale here now I'll tell you how to read
this correlation Matrix so wherever the
cells are in blue or dark blue color
this means that the variables have very
high correlation and all this cells
where you see light blue color or
grayish and white color uh this
indicates that the variables have very
low correlation for example there is
very low or almost negative correlation
between happiness score and perceptions
of corruption so obviously if the
citizens of a country feel that there is
a lot of corruption in the country their
happiness score would obviously be less
or low and if you see there is also low
correl ation between happiness score and
generosity there is again low
correlation
between Health life expectancy and
perceptions of corruption and if you see
these places there is really high
correlation between the happiness score
and GDP per capita again for social
support also there is very high
correlation now if you see even
for social support and healthy life EXP
sped see there is very high correlation
but there
is low correlation
between making life choices and
generosity and there is negative
correlation between corruption and
freedom of making life choices you can
see it is almost white color which means
it falls around this region so negative
correlation similarly
for logged GDP per capita and Corruption
there
is negative correlation all
right okay now we are going to visualize
a bar plot that will tell us the
corruption in different regions so I'll
just give a comment corruption
in
regions let me just scroll down okay so
I'll create a variable called
corruption which will be equal to
my data frame that is happycore DF now
first of all I'm going to use the group
by function to group all my regions so
I'll say Regional
underscore indicator which is
my column name present in the data set
and after this I'm going to find the
average of the corruption that is
perceptions of corruption so I'm going
to pass in my variable name that is
perceptions of
corruption and I'm going to use the mean
function to find the average corruption
in each of these
regions now let me just go ahead and
print
my variable that is
corruption where it you can see here I
have the values for the different
regions and from here you can see
that Central and Eastern Europe has the
highest perceptions of corruption as per
the questions answered in the poll and
if you see the
table we have Western Europe and the
North America region with the least
perceptions of corruption
now we are going to visualize this using
a bar plot now so first of all I'll set
my parameters by giving the figure size
I'll
say fig
size it's rather figure do fix
size I'll just add figure
here okay and now let's say
I'll assign it as 12a 8 now I'll give a
title to my plot so I'll say PLT do
title and my title of the plot is going
to be perception of
corruption
in
various
regions all right then I'm going to
Define my X label
so I'll say x label will
be
regions
and we'll set the font size of the X
label to let's say 13 or let it be
15 then we are going to set the Y
label in y label I'll
have corruption index as the label name
again we we are going to set the font
size to
15 now I'm going to use x tick
parameter since I want to rotate the
axis labels in the x-axis by 30° so I'll
say rotation equal to 30 and I'll say h
a which stands for horizontal alignment
equal to
right make sure this right should be
within single
Cotes and finally I'll say PLT dot bar
because I'm going to plot a bar
graph I'll say
corruption dot
index
comma and then I'll
say
corruption
dot perceptions of corruption which is
my column
name all right so I have my code ready
for the bar plot now let me just go
ahead
and print the bar plot make sure
everything is correct I'll just hit
shift
enter okay there is one mistake here it
says okay this should be x t and not X
tick let me run it again there you go so
if I scroll
down on the top you have the title which
is perception of corruption in various
regions on the x-axis you have the
regions label on the y- axis you have
the corruption index and if you see this
as per our table that we
created we have least corruption in
North America and AZ region then we have
the next least corruption in Western
Europe but we have the highest
corruption in Central and Eastern Europe
as per their citizens perception
similarly we have
the second and third highest corruption
in Latin America and Caribbean as well
as South
Asia
cool now moving ahead I'm going to show
you how you can find the life expectancy
of of the top 10 happiest countries and
bottom 10 happy countries so for that
I'm going to run you through the code
and we'll see the visualization side by
side okay so I have my code written in
these two cells so first I'm going to
find out the top
10 happiest countries and then I'm going
to find the bottom 10 happiest countries
so for that I'm using the head function
and I have passed in 10 since I want the
top 10 country names and to get the
bottom 10 countries as per their
happiness score I'm using the tail
function so let me just run it so we
have saved the result in two variables
top underscore 10 and bottomore 10 and
to create the bar plots we are using two
different codes so here you can see I
have set my figure size and axis and
then I have my X label as country name
after that I am setting my title of the
bar plot to top 10 happiest countries
life expectancy I've used my xtick
labels and I'm rotating it by 45° and I
have my horizontal alignment as right
you can see it here we have used the
barplot function I have the xaxis as
country name and by access as healthy
life expectancy column and I have set my
AIS then I am using the X labels and Y
labels as continum and life expectancy
and similarly I have my barplot for the
bottom 10 least happy countries life
expectancy let's just run it and we'll
see the result there you go now if I
scroll down you can see we have two
different bar plots the first one is for
the 10 happiest countries and then we
have the bottom 10 least happy
countries so if you
see on an average the top 10 happiest
countries life expectancy is above 70
years so if you are from one of these
countries you are expected to live for
more than 70 years now if you check the
bottom 10 least happy countries you see
here loto has less than
50 life expectancy age and most of them
are less than 60 years so if you are
from one of the top 10 happiest
countries you expected to live 10 years
more than these countries that lie in
the bottom 10 region cool now moving
ahead all right so now we are going to
to see the plot between freedom to make
life choices and the happiness score for
this I'm going to use a scatter plot so
I'll first Define my figure size so I'll
say PLT do RC p a RS which stands for
parameters I'm going to pass in
my figure size so I'll say figure dot
fig
size equal
to let's say 15 comma
7 then using the seone library and the
scatter plot
function we'll pass in
the x-axis let's say the x-axis is my
data frame name happycore DF and I'll
have the freedom to make life choices in
the
x-axis
so the column name is freedom _ 2core
meore
life and
choices I'll give a comma and we'll pass
in the y axis again I'm going to use my
data frame name Dot and I'll have the
happiness
score I'm also going
to pass in a hue parameter
to
differentiate the different
regions so I'll say DF
do Regional underscore
indicator and I'll give the size of the
dots or the Bubbles as let's say
200 now I'll
say PLT
do
Legend I'll place the legend
at upper left corner so I'm giving the
location which is loc equal to upper
left
and font
size let it be
12 I'll say PLT do X
Lael
as freedom to make
life
choices
and my y
label would be happiness
score all
right now we have the code written for
the scatter plot that I want to create
let me just run
it there you go so here we have the
legend for the different regions and we
have the different colors for each of
these regions and you can see here I
have on the x-axis freedom to make life
choices and on the y axis we have the
happiness
core so you can see it very
clearly for the countries that lie in
the Western Europe region the blue dots
the freedom to make Life Choices is more
and so is the happiness score then if
you see the values in the green region
which is for Middle East and North
Africa the freedom to make Life Choices
is lower and hence the happiness score
is also low and for all these countries
that are part of the subsaharan Africa
some of them
have decent score for freedom to make
life choices but the happiness score is
low
again
now if you focus on
the pink dots which is for the Southeast
Asian countries the happiness score is
comparatively lower but the freedom to
make life's choices is more than
point8 cool and again we have one data
point which
is lying at the bottom we can assume
this is Afghanistan so so in Afghanistan
the freedom to make life's choices is
very low and even the happiness score is
really
low
cool now moving to our next analysis we
are going to see the top 10 most corrupt
countries so first I'm going to sort the
perceptions of corruption column and
find out the top 10 countries in the
list so for that I'm going to create a
variable called
country I'll have my data frame name
that is happycore DF do
sortore
values by I'm going to sought by column
that is
perceptions
of
corruption dot head and I want to find
the top 10 most corrupt countries as per
the
poll and
then I'm going
to pass in the figure size so I'll say
PLT do
RC
perams and then I'm going to set the
figure size so I'll say figure
dot fig size
make sure there is no spelling
mistake let's say the figure size is 12
comma
6 I'll pass in the title so I'll say PLT
do title let's say the title of my plot
is going to be
countries with
most perception
of corruption
after that I'll say PLT do X
Lael in the X label I'll have country
and I'll set
the font size
to
13 then I'll say PLT
do y
label in by label I'll have the
corruption
index
next
I'll pass in the font size for the by
label again it is going to be
13 now I'll see PLT do
X
tick I'm going
to rotate it by
30 and I'll say horizontal alignment
equal to
right and then I'll have my bar function
or the bar plot function so I'll say PLT
do
bar I'm going to pass in country
dot country underscore name and then
we'll have my variable country dot the
column name that
is perceptions of
corruption all
right now if I run it you can see
here so these are the countries with the
least perceptions of corruption so
Singapore has
the lowest corruption
index and then we have Randa Denmark
Finland now if you want to see the the
countries with the highest perceptions
of corruption you need to change this
head to tail of 10 since we are sorting
it in ascending order so we would want
to know the bottom 10 countries if I run
it there you go so these are the
countries with the most perception of
corruption you have Slovakia loto Kosovo
that's Ukraine
Afghanistan bugaria Romania and Croatia
all of these countries have a corruption
index of more than
85 cool now coming to the final section
of this interesting video on happiness
report data analysis for 2021 I want to
visualize a scatter plot that will tell
us how the corruption varies in terms of
Happiness score so I'll just say a
comment as corruption
versus
happiness okay so first of all I'll set
my figure size so I'll say PLT do
RC per
amps within single quotes I'll give the
figure
size
as 15 comma
7 then I'm going to use the scatterplot
function that is part of the cbon
library in the x-axis I'll have
my column that is corruption or rather
we'll have the happiness score in the
x-axis so I'll say happycore DF dot
column name that is happiness
score let me give a comma and in the y
axis we'll
have the corruption column so I'll say
happycore DF
do perceptions of
corruption and
then in h we'll have the region that
is regional uncore indicator so I'll say
happycore DF
dot
Regional
indicator
and I'm going to give the size of the
dots so I'll say s equal to
200 then I'm going to say PLT do
Legend I'm going to put it
at lower left corner this time so I'm
giving my location as lower
left and the font size of the legend I
want
is
14 after that I'll just say PLT do X
Lael
as corruption and PLT
dot V
label as happiness score
now this should rather be the opposite
since in the xaxis we have taken
happiness score so we'll put the
happiness score label in X and I'll have
corruption in
y all right so I have my scatter plot
code ready let me just run it the you go
so here you can see on the lower leftt I
have the different regions and on the
x-axis I have the happiness score on the
y axis I have the corruption index now
you can see the general Trend as per the
scatter plot that the countries with
greater happiness score have lower
corruption index so you can see these
countries which are from the Western
European regions
have highest happiness score and their
corruption is also really low so all
these blue countries you can name a few
Finland we have Sweden we have Belgium
France Netherland lying in these regions
we also have a few countries from North
America and Enz which are essentially
Australia then we also have Canada us
and New
Zealand okay now if you focus on these
green region this region now these are
are the countries from the subsaharan
Africa and most of the countries from
these regions have a very low or less
than five happiness score but the
corruption is also really high you can
see here they have almost more than7 the
corruption index now you also have a
country here which is from the southeast
Asia we would like to give you a task it
would be really great if you can tell us
in the comment section which country is
this it has more than six happiness
score but the corruption index is really
low Which is less
than2 now if you
consider Middle East and North African
regions the darker green countries here
also if you see the happiness score is
less than five and the corruption index
is also high now there are a few
countries the Commonwealth of
independent states for
example countries such as usbekistan
then you also have kajak istan we have
Russia Armenia belonging to
these gray color dots you also have
Georgia and Ukraine as part of
Commonwealth of independent states so
here you see the corruption is below
point six for these countries and the
happiness score is more than
five all right so with that we have come
to the end of this demo session on world
happiness report 2021 deta analysis
using python hello everyone welcome to
this exciting video on Olympics data set
analysis using python by simply
loan this is going to be a very
interesting and interactive session
where I'll give give you a brief about
the history of Olympics and throw some
light into the recently held Tokyo
Olympics but we'll focus more on using
Olympics data set that is available on
kigle and perform some exploratory data
analysis you will understand how to use
different functions in Python to analyze
and extract meaningful information
during the course of our discussion I'll
be asking you a few general questions
related to the Olympics try to answer
them in the comment section of the video
so let's begin the Olympics is one of
the biggest sporting events on the
planet it is held every 4 years the
first modern Olympics took place in
Athens Greece in
1896 as per National Geographic the
original Olympics took place in 776 BC
so they began as part of an ancient
Greek festival which celebrated Jews the
Greek god of sky and weather the Rings
in the Olympics logo represent the five
continents Europe Africa Asia the
Americas and oania
from 1924 to 1992 the winter and the
Summer Olympics took place in the same
year but now they alternate every 2
years before I move on here is an
interesting question for you only two
people have ever won gold medals at both
the summer and the Winter Olympics who
are those two people please share your
answers in the comment section of the
video we would like to know from
you the Summer Olympics in Tokyo began
on the 23rd of J July and recently
concluded on the 8th of August we got to
witness some Thriller matches that went
down to the wire some amazing victories
and sadly there were a lot of
heartbreaks as well winning and losing
are part and parcel of any
game fans across the world were really
happy to see This Global event happen
this year Following last year's
postponement due to the Corona virus
pandemic with our Olympic fever High
let's take this opportunity to work
around a project to perform exploratory
data analysis using python to analyze
and visualize past Olympics data and
answer specific questions in this video
we'll be using specific python libraries
such as numpy pandas matplot lip and
cabon to make sense of our data and
extract meaningful information we are
going to visualize our results with
different charts and graphs before I
show you the data set that we are going
to use for our analysis I have another
general question for you which is the
first Olympics where all the
participating countries sent female
athletes I repeat which is the first
Olympics where all the participating
countries sent female athletes put the
year and the city name where the
Olympics was held in the comment section
of the video we'll be more than happy to
hear from
you now let me go ahead and show you the
data sets that we'll be using in this
video so here you can see I have my
Olympics data set folder that has two
data sets these are CSV files one is
called the athlete events and the the
other one is called nocore regions so
we'll be using these two data sets these
data sets can be downloaded from Kel we
will post the data set links in the
description of the video please go ahead
and download them these data sets have
been directly taken from the internet
now I'll show you the data sets so this
is my first data set you can see on the
top athlete events so I have around 15
columns you can see the count here and
there are
nearly 2H 71,1 2016 rows of
information so this is a huge data set
that we are going to use the other data
set we are going to use in this demo is
called nocore regions now here one thing
to notice no stands for National Olympic
Committee this is a three-letter code
that is given by the Olympic Committee
we have the region names so afg is for
Afghanistan you have ALB for Albania ALG
for Algeria we have also a column called
notes you find some notes about the
region for example here antiqua is
actually antiqua and
Barbuda all right let's move to the
first data set that I showed you this is
our primary data set for our
demo so these data sets have been
directly taken from the internet and
were not validated so the results that
you will see in the demo is purely based
on the data that we have
collected so the file athlete events.
CSV contains nearly 2ak 71,1 20116 rows
of information and there are 15 columns
so each row corresponds to an individual
athlete competing in an individual
Olympic event so here ID is actually a
unique number for each athlete then we
have the name which is basically the
athletes name we have the sex or the
gender which is male or f for female
then we have the age of the athlete
which is in terms of
integers we have the height in cm of the
athletes then we have the weight in
kilograms we have the team name these
are the country names you have China
Denmark Netherland there are around 200
country names then we have the nooc as I
said nooc is a 3let code that stands for
National Olympic
Committee we have the games this games
contains the year and the season you can
see here 1992 summer you also have the
Winter Olympic information all right
then we have a specific ear
column to tell you in which year this
event had
occurred now the year column is also an
integer column we have the season so
whether it was summer olympics or Winter
Olympics then we have the city name this
is the host
City then we have the sport name you can
see here we have basketball Judo
football tgof War speed skating there
are so
many then we have another column called
event which is the complete event name
as you can see the sport name is
basketball but the event name is
basketball men's basketball to be very
specific again for Speed skating you
have different categories like women's
1,000 M 500
M all right now the final column is the
metal column so it
has the information about the athlete
whether the athlete had won a medal bait
gold silver bronze and NA means the
athlete did not win any medals so we'll
use these two data sets now let's get
started with the demo we'll be using
jupyter notebook for our analysis so
I'll take you to my Jupiter notebook
right
away I have opened it on Chrome so this
is my jupyter notebook that we are going
to use you can see here Olympics data
set analysis I have
already have a few cells that have been
filled with some piece of code and you
can see there are some Commons written
as well we are going to use this data
set so let's get
started first and foremost we'll import
the data sets so we are going to use
numpy pandas matplot lib and C on let me
hit shift enter to import all the
libraries all right now the next thing
is to load the data
setes using Panda's read CSV function
so I'll create a variable called
athletes I'll say equal to PD dot I'll
use the read CSV
function and inside this function will
give the location of the files so I'll
copy this
location and here I'll paste make sure
this is
within
quotations and these are
all forward
slash I'll give another forward slash
and after that I'll give
the file name that is
athlete
underscore
events.
CSV I'll close the
quotation let's run it okay similarly we
will load the second data set I'll
create another variable called regions
so it will have my data frame I'll use
the same function PD do read
CSV I'll just copy this file path till
here we'll paste
it
and I'll give my second file name which
is
nocore
regions. CSV you can verify here the
first one is athlete _ events the second
one is nocore
Regions okay I'll close the quotation
and we'll run it
again okay there seems to be some error
I think this is
Regions
cool now moving ahead let's see
the first few rows of both the data sets
I'll use the head function for
that I'll
say
athletes. head so this will print the
first five
rows from my data set athletes you can
see here from from 0 to 4 you have five
rows of information the ID name sex or
the gender age height
weight if I go to the right you have
games year season sport event and
medal now let's see
the second data set that we imported or
loaded I'll say regions.
head it says regions
is not
defined let me cross check
it okay I have taken it as region let's
make it
regions
cool now let's run it there you go so
these are the first five rows from our
second data set you have nooc region and
some
notes all
right now the next step is to combine
both the data sets so we'll join the
data frames using the pandas merge
function let me show you how to do it so
this is going to be a horizontal join
I'll create a
variable called athletes unor
DF then I'll use my first variable which
is
athletes I'll see do merge which is my
function name I'm going to merge the
regions data frame
I'll use the how
parameter in how I'll give my value as I
want to do a left
joint and my common column in both the
data sets is actually
nooc so I'm going to use the nooc column
as my common column to merge both the
data sets now let's go ahead
and print the merged data frame I'll say
athletes DF dohe
let's run it there you go so if you see
clearly if I move to the right you can
see here we have added two more columns
that were present in the second data set
which was regions so we have regions and
notes added to my first data set that is
athletes
uncore
events
cool now one thing to note here is so
the column names are not consistent if
you see here the rest of the columns
start with a capital letter but if you
see the last two columns that we just
added now start with the lower case
letter so we'll use the rename function
to make the column names
consistent but before
that
let's see the shape of the data frame so
I'll print the shape of the data frame
to know the total number of rows and
columns
I'll
see athletes if I hit tab you'll see it
will give
me the prompt I'll select athletes
DF I'll use the shape
attribute let's run it and see the
result you can see here it gives me the
total number of rows so 2 lakh 71,1 2016
rows and earlier we had 15 columns in
the first data set and now that we have
added two more columns so the total
number of columns becomes 17 now
cool so now it's time to make
the column names
consistent so I'll show you how to do it
I'll say athletes uncore
DF dot I'll use the rename function and
then I'll say
columns equal to
I'll use curly
braces My First Column that is region I
want to change it
to region but the r should be Capital
now or
uppercase I'll give a
comma and then see my second column this
should be region
actually my second column that is notes
I want to change the first letter to
uppercase so I'll say capital
N the rest all remains same I'll give a
comma here and say in place equal to
true so it will
change the column names and will
reflect in this athletes unor DF data
frame let's run it now to verify you can
again use the head function I'll run
this again let me scroll down and move
to the right you can see the difference
there you go so we have successfully
renamed the last two columns region and
notes all
right now moving
ahead
now I'll show you how you can use the
info method to print information about a
data frame including the index data type
the column data types non-null values
and memory usage so for that you just
need to use the info function so I'll
write athletes uncore DF dot
info let's run it there you go so here
we have the data columns you can see
total 17 columns you have 2 lak 71,1
2016 rows or entries then you have
the different column names and you say
it says nonnull and the data type of the
column below you can see some
information about the memory usage all
right
next the described method is used for
calculating statistical information such
as mean standard deviation the
percentiles of the numerical values of
the data frame and much more it analyzes
both numeric and object series and also
the data frame column sets of mixed data
types so let me show you how you
using the describe function you can
display some statistical
summary I'll just write
describe and give parenthesis let's run
it okay if I scroll down you can see
here by
default the describe function will only
give information about numerical columns
so here you can see we have the total
count the the mean of each of these
columns you have the standard deviation
the minimum and maximum value and then
we have the 25th percentile the 50th
percentile and the 75th percentile value
of the columns ID age height weight and
year all right now one thing to notice
here is if you see the year column the
minimum year is
1896 so this is when Olympics started
and until recently the real Olympics
that was held in
2016 all right now moving
ahead now let's check if there are any
null values present in The Columns of
the data set so first I am going to
create a variable called Nan
values give equal to I'll see athletes
uncore DF and the function to check is
I'll say is na
I'll create
another variable called
ncore
columns and here I'll use the any
function I'll say n
underscore this variable which is
underscore
values dot I'll see
any and I'll print my NN columns
variable so this will display the result
in terms
of
Boolean values so if there are any null
or na or missing values in any of the
columns it will say true otherwise it
will say false let's display the result
there you go so if you Mark here clearly
so there
are nearly six columns where we have
missing values we have age height weight
metal region and notes columns that have
missing values so hence it has given us
true the rest of the columns do not have
any n or missing values hence they are
false all
right let me scroll
down
now let's see the total number of null
values for the above six
columns so I'll
say athletes
underscore DF dot I'll use the function
is null and again I'll use the sum
function okay this should be is null if
I run it you can see it
here so our age column has
9,474 rows where we have null values
then we have some null values for the
the height column as well for the weight
column so these are the total number
of rows where we don't have any
information regarding age height weight
or
region notes this medal is
self-explanatory because a lot of the
athletes who participate in Olympics
don't win any medals so for them the
value is na now before I move ahead I
have a question for you people
so the question is I want you to print
the column names containing null values
or missing values in the form of a list
so please answer this question and put
it in the comment section of the video
we would be happy to know your
approach so the question I'll repeat it
again I want you people to print the
column names containing null values or
missing values in the form of a list so
basically on the top we saw there
were six columns that had null values I
want you to print these six columns in
the form of a list all right now moving
ahead now let's see the data for
specific countries let's say you want to
see the athletes who have participated
in the Olympic Games from the beginning
for India for that you can filter your
result using a function called query so
let me show you how to do it I'll say
athletes for DF dot we'll use the query
function and
then I'll say team which is my column
name should be equal to equal to and the
value I'll give is
India now let's display the first five
rows of information or the details for
the athletes who are from India if I run
on
this okay there is some error here okay
so we need to make sure this entire
expression or the condition should be
within single quote I'll just delete
this single quote from here and we'll
add it in the
end now let's run it there you go so
here you can see we
have the top five rows
of the athletes who are from India you
can see here the region is India you can
see the nooc is India the team is India
and let me go to the left you can see
the name of the athlete the sex age so
for age you have some missing values you
have the weight team year and everything
all right similarly you can also check
for Japan let me just copy the above
code I'll paste it here
and instead of India I'll say
Japan I'll run it there you go so if you
see here you can see all the details are
for the athletes who are from Japan all
right moving ahead now I want to know
the top 10 countries who have
participated since the Inception of
Olympics in
1896 so so for that I'll create a
variable called topor
10or
countries I'll say equal
to I'll use my data frame that is
athletes undor
TF I'll say dot
team then I'll use the
function value
underscore
counts I'm going to
sort my count in ascending
order just Mark the way in which I'm
writing the functions any error in terms
of syntax of the flow will throw you an
error I want to display the top 10
countries now let's go ahead
and print it I'll say topor 10 uncore
countries all right let me run it
okay it has given us an error let's
debug it okay so this should be value
counts and not count let's run it again
another
error okay so this should be short
underscore values and not value let's
run it again there you go so here you
can see the top 10 countries
participating in Olympics since 1896 you
have United States then we have France
so these are the number of participants
who have taken part since
1896 so the most number of participants
have come from us then we have France
Great Britain Italy Germany Canada Japan
Sweden we also have Australia and
Hungary I'll scroll
down now we are going to convert this
table that we got or the output that we
got in the form of a graph so I'm trying
to create a bar plot to plot the top 10
countries who participated in lumex so
here I'm using
the figure function to give the figure
size so these are the dimensions 12a 6
now I'll tell you what this PLT X6 means
just wait for a while I given a title to
my plot saying overall participation by
country
and then using the cbor library and the
bar plot function I am plotting the X
and
the Y AIS so my x axis has top I should
make this top 10 countries because above
we used topor 10or countries similarly
for y axis I'll make it topor 10or
countries and then I'm using a color
palette called set two
let's run it there you go so if I scroll
down you can see here we have a nice bar
plot this is a vertical bar plot and
here you can see the bars which
represent the different country names
first we have United States which has
the highest participation since the
beginning of the Olympics then we have
France Great Britain Italy Germany
Canada and we have the rest of the
countries cool
now moving
ahead the next visualization we are
going to see is the age distribution of
the athletes so for this we are going to
create a histogram I'll be using the
matte plot lab Library specifically for
this
visualization so I'll say PLT do
figure we'll give the figure size using
the argument fig size equal to
and this will be a tle
actually my size is going to be 12a
6 I'll
say
PLT
dot
title now you can
give any title that you want I'll say
age
distribution of the athletes
then we'll also give the X labels I'll
say PLT
dot X
label this will have
age then we see PLT dot y
label and this will
have the label as number
of
participants
next I'm going to use the hist function
so I'll say PLT
doist and
I'll pass in my data frame that
is athletes undor
DF dot age which is my column name comma
I'm going to pass in
the
bins for that I'm going to use the NP
doin function so P is for
nump and inside this I'll say 10 comma
80 comma 2 so my bins will start from
the value 10 and go until 80 with a size
of
2
comma I'll give another parameter that
is going to specify the color of the
bins I want my color to be
orange and then
I'll use the edge color to separate the
bins I'll say Edge color this is
the parameter or the argument name and
the color I want is
white let's give a semicolon here and
this should be Edge color there's a
spelling mistake here now let's run it
and we'll see the output so there you go
so this is a nice histogram that shows
the distribution of the age of the
athletes you can see the title age
distribution of the athletes on the y
axis you have the number of participants
on the x-axis we have the age values
ranging from 10 to 80 with a size of
2 and if you see this we have most
number of athletes who have age between
20 to 30 you can see here so early 20s
we have maximum number of athletes
participating in the Olympics we also
have a few athletes who are Beyond 40
years of age you can see here we have a
few athletes even closer to 60 also and
similarly we have a few athletes who are
under 18 years of
age all
right now you can see here the bins we
had taken orange color so are all
the bars are represented in orange color
and the edge you can see here we have
the white color all
right now moving
ahead now in the initial slides of the
video we discussed the summer and winter
Olympic games let's look at the
different sporting events that are part
of the summer and winter Olympic Games
just to give you a heads up the Winter
Olympic Games are held once every four
years for sports practiced on snow and
ice so here I have my variable name
wintercore Sports so I'm
extracting for season equal to equal to
Winter and I am going to display only
the unique values now let me run this
cell okay so you can see here these are
the different winter sports that
are held during the winter Olympics now
similarly let's see for Summer Olympics
so what I'm going to do here is I'll
just copy this cell of code and we'll
edit the variables I'll say summer
underscore Sports athletes _ DF Remains
the Same here season we'll change it
to
Summer do sport do unique Remains the
Same and here I'll see summercore
Sports this should be small now let's
run it there you go so here you can see
we
have more number of Olympic sports that
are held during the Summer Olympics
compared
to Winter Olympics so these are the
sports that have played on snow and ice
and these are the sports that are played
during
summer moving ahead now it's time to
analyze the total number of male and
female participants who have taken part
in different games since 1896 till 2016
Rio
Olympics all right so I'll see
gender underscore counts which is my
variable name I'll use
my data frame that is athletes uncore DF
dot my variable name that is sex dot
Valore
counts and then we are going to
print gender uncore counts
let's run this there you go so since the
Inception of Olympics we have more
number of male participants than female
participants now here in this current
cell you can see we are trying to plot a
pie chart for male and female athletes
so I have given my figure size then I
have my title as gender distribution I'm
using the matplot LA library and the Pi
function I'm using the variable gender
counts that we used here then I'm giving
my labels as gender _ counts. index now
I'm using this autop PCT the autop PCT
parameter enables you to display the
percent value using python string
formatting so this is my string
formatting that I have used and I'm
initialized my start angle for the pie
chart as 150° and I'm also giving Shadow
to my pie chart so I have written sadow
equal to True let's run it all right so
you can see here this is my Pi chart
that shows you the distribution of the
male and female participation so for
male it is
72.5% for female so far it is 27.5% as
per the data set that we have we can
change this start angle to let's say
180° and it will change the pie chart to
this
direction cool now moving
ahead this time we're going to find the
total number of medals that the athletes
have won so I'll use
my data frame athletes DF Dot and then
my column that is medal I'll say dot
value uncore
counts
let's run it there you go so you can see
the gold medals the bronze medals and
the silver medals are very much similar
to each other the numbers are pretty
much the
same all right now it's time to focus on
the total female athletes who have taken
part in each
Olympics so I'll create a variable
called female
uncore participants
I'll say equal to and then use my data
frame name that is athletes
DF using square
brackets I'll
say
athletes underscore DF
dot the gender should be equal to equal
to
female and I'll use am
erson I'll
say
just copy
this paste it
here LC dot
season equal to equal to we'll check for
Summer
Olympics and I want to extract only the
gender column so here it is sex
comma my year column which is
here
okay and then I'll
say
female
underscore
participants equal
to I'll just copy
this paste it here and then I'll use the
group by
function say Group by which is part of
the pandas
Library I'll say
here and then I want to find the count I
reset the
index say reset uncore
Index this is a
function and finally we are going to
print female uncore
participants and let's say the
head all right so what we are trying to
do is I'm trying to filter my data only
for the female athletes for Summer
Olympics and here I I am
printing the participation based on each
year so I'm finding out the count so
here we need to
add one more square
bracket all right now we'll run it and
see the result there you go so you can
see here from 1900
19004 all these years you can see the
female
participation let me change it to tail
so that that we have the recent data of
the Olympics you can see it here for the
Beijing Olympics
5,816 women athletes participated in
2012 London Olympics we had
5,815 Olympics similarly for the 2016
real Olympics we had more participation
than the London Olympics so
6,223 women athletes had
participated all
right now here this is another way in
which you can filter your data for
female athletes so I'm using two
parameters my gender should be equal to
female and season is Summer Olympics let
me just run it I have stored it in a
variable called women
Olympics now here I'm trying to find a
count plot or I'm trying to create a
account plot using my cbon SNS Library
I've have set my style to dark grid then
my figure size I have given as 20 comma
10 my count plot will have the
x-axis as ear the data is women Olympics
and pallet I'm using as spectral and
then I given a title women participation
let's run and see the
output there you go if I scroll down you
can see on the top I have the title for
my count plot which says women
participation and on the y- axis I have
the count and at the bottom you can see
the different year values from 1900 till
2016 and 2016 had the highest number of
female
participation cool let me scroll down so
in this cell of code we are trying to
plot a line graph let me run it and show
you the output so this line graph shows
the trend which says plot for female
athletes over time so here you can see
the line graph so gradually the women
participation has increased in Olympics
since its Inception there was a slight
decrease here this is
around
1950s and again that you can see here
there's a decrease with the women
participation in 1980 but since then
since 1980 there has been
continuous increase in the participation
of female athlete
numbers cool now coming to the next
section of our
analysis now we are going to filter the
data to see the details of the athletes
who have won gold
medals so I'll create a variable called
gold
medals equal to I'll use
my data frame as athletes
DF
and within parenthesis I'll use my data
frame name which is
athletes
underscore DF
dot I'll use the condition medal equal
to equal
to and my value would be
gold next I'm going to
say gold medals.
head let's run it there you go so here
we have
the top five rows of the athletes who
have one a gold medal you can see all
the records have medal as
gold now we are going to use
this
subset of the data and and perform some
more analysis so
here I want to take only the values that
are different from naan so I'm going to
use is
finite function and inside that I have
set gold medals age let me just run it
cool now I want to
see the athletes who have secured a gold
medal beyond the age of 60 years which
is very
rare so we'll see the total number of
athletes with more than 60 years of age
having won a gold
medal so here I'm going to
say gold
medals I'll use
my column as
ID and
then I'll
use Maybe variable gold
medals and
then I'll say
age should
be greater
than
60 and then I'll close my square bracket
and say
count okay let me just verify if
everything is
fine yeah I'll run it and we'll see the
result so there are
total six athletes who have won a gold
medal having age more than 60 years
cool now let me
check for
which sport these six gold medals have
come so I'll
say sporting uncore event this is going
to be my variable name
equal to I'll use
the same variable name gold
medals I'll going to
choose my sport column and
then I'll give
my condition
AG greater than
60 now let's go ahead and will
print the variable sporting event if I
run this you can see it
here so for art
competitions even for archery and
suiting and this one called Rook we have
had gold medals from athletes who had an
age more than 60 years now we are going
to
plot this result of the table that we
achieved on the
top so I'll say PLT do
figure I'll give
my figure
size equal to let's say 10A
5 then I'll say PLT
dot we'll use
the tight
layout see tight underscore layout
then I'm going to use the cbon library
and create a count plot for my
variable sporting underscore
event then I'll see PLT dot
title I'll give the title as gold
medals
for athletes over 60 years of age
let's run
it and if I scroll down you can see here
we have a nice count plot I have my
title gold medals for athletes over 60
years and since for archery we had three
players who secured a gold medal having
an age more than 60
years you can see here archery is three
art competition one Rook is one and
shooting we had one medal
all
right now we'll see the total gold
medals from each country so for
that I'll
say
gold
medals dot we'll use
the region column and followed by that
I'll use the value underscore counts
function and I'll reset the
index with
name equal to I'll provide the medal
column then we are going to print the
head for the top
five countries let me run it you can see
it
here
so
USA has secured the most number of gold
medals then we have Russia Germany UK
and then we have
Italy
cool now in this cell I'm going to
create a plot
to visualize the table that I got above
so I've used my labels my titles I'm
using the cat plot function that is
present in the seone library I have my
xais my y AIS the data which is total
gold
medals let me just verify
okay and we'll run
it if I scroll
down you can see this is a different
pallet that I had used which is
rocket and you can see USA has got the
most number of gold medals then we have
Russia Germany UK Italy and France now
we also have included France here which
was not present in this table the reason
being I have used head as six so will
give me the top six
countries all right now we will
analyze and look at the data for the
most recent Olympic event that is
present in our data set I'm talking
about the 2016 Rio Summer
Olympics so first of all I'll create a
variable called Max
year which will be equal to my data
frame name athletes unor
DF my column name year and I'll use the
max
function then let's go ahead and print
my Max
here I'll just show you the output you
can see here my Max year present in the
data set is 2016 the r
Olympics
next we will create another variable
called teamore
names that will be equal to
my data frame name athletes DF
within square brackets I'll give
parenthesis and then see athletes
underscore DF dot
year should be equal to equal
to my output that is Max here which is
actually
2016 and then I'll
say Amberson
I'll just copy
this we'll paste it
here this time I'll use
the medal column and say equal to equal
to
Gold I'll say dot
team then I'll say team underscore names
dot
value
uncore
counts do head and we display the top
10
countries let me run this there you go
so in Rio
Olympics United States secured the most
number of gold medals now the reason
this is 137 is we have all also counted
the team events for example
basketball similarly Great Britain had
64 gold medals in total Russia 50 we
have Brazil 34 Argentina 21 France 20
and Japan
17 all
right
now using the above result we are going
to create a bar plot I'll just change
this teamore list to teamore names
similarly here also I'll make the
corrections as teamore
names cool now we'll run it and you can
see here we have a nice horizontal bar
plot so we have United States at the top
Great Britain Russia Germany China
Brazil so here I'm displaying the top 20
Nations since I have used head as 20 so
these are the top 20 Nations to secure
the most number of gold medals in the vi
Olympics all
right now in the final section of this
video we will create a scatter plot to
visualize the height and weight of male
and female athletes who have won a medal
now it can be a gold medal a silver
medal or a bronze medal but before that
we need to filter the data only for
athletes who have won a medal now if
you had noticed here
are medals column also had a few null
values so we are not going to consider
those null values here we are only going
to consider for the athletes who have v
a medal so for that I'll create
a variable called notore
nullcore
medals
equal to I'll
say athletes underscore
DF
then I'll give
my let's make it
DF I'll give my
condition let me just copy
this I'll paste it
here and then
I'll give my condition
height and use
the not null
function and then I'll say
Amberson create another parenthesis I'll
see athletes uncore
DF this time we'll consider the weight
column again I'll use the function as
not
null
and I'll close the
parenthesis okay I had missed one s here
cool let me just run it now all
right now we are going to create our
final plot I'll say PLT do
figure I'll give my
figure
size equal
to I'll say 12 comma
10 I'll say
access equal to SNS dot we'll be
creating a scatter plot to plot
the height and weight of the athletes
who have won a
medal I'll say scatter plot
in my x-axis I'll
have
height comma in my Y axis I'll have
V
comma my data should
be not underscore null underscore
medals so this is what we created here
not under _ nullcore medals comma I'll
use H equal to my sex
column cool finally I'll say PLT dot
I'll give a title to my plot
saying height
versus
weight
of
Olympic
medalist
cool now let's see our scatter plot this
might take some time to give the result
because there are so many athletes
present in a data set who have won a
medal and also we are
filtering in terms of hue which is for
male and female there you go so if I
scroll down you can see here we have a
nice scatter plot on the top we have the
title of the plot saying height versus
weight of Olympic medalist you can see
our age as the Hue so blue points are
for the male athletes and orange points
are for the female
athletes and on the y- axis we have the
weight in terms
of kilog and you have the height in
terms of
CM
cool so that brings us to the end of
this demo session on Olympics data set
analysis so we used two data sets
to carry out this explorat data analysis
and we created some visualizations to
analyze and visualize the data all right
want to be a certified data expert then
here we have postgraduate program in
data analytics by simply learn take a
glance at the notable features and
skills offered in this course you will
benefit from exclusive IBM hackathons
and ask me anything sessions eight times
more interaction in live online classes
with industry experts capston projects
across three domains and over 14 data
analytics projects using real data sets
from Google Play Store lift World Bank
and more master classes from puu faculty
and IBM experts along with simply learns
job assist for better job prospects
acquire skills in data analytics
statistical analysis using Exel data
analysis using Python and R and data
visualization using table and powerbi
enroll Now using the course Link in the
description
box do you know what's common between
International Tech giants like meta
Google and a Smalls size d2c startup
that is running from Jakarta is it the
product their services or their work
culture not really the biggest common
denominator between all tech companies
globally today's data and AI the
overarching need of of utilizing data
for decision making and implementing the
almost necessary capabilities of AI in a
product or service has become the demand
of the All Things Considered wouldn't it
be amazing if you can utilize AI to
perform some of the most complex
processing and analytics task on data I
think your answer would be yes with that
being said hello everyone and welcome to
this extremely important video in data
analytics using AI in this video we will
discuss how you can utilize AI for
performing data analytics which is the
very first skill you must learn to make
a career in the field of data analytics
business analytics and even becoming a
data scientist but before we begin with
that make sure to subscribe to our
Channel and hit the Bell icon for
regular updates the fear is in there
whether it's marketing automating
building driving medical practice or
anything you name AI can do it coming
into data analytics which is believed to
be one of the promising career options
AI is getting integrated
and it is leveraging analytics by
enhancing and revolutionizing the way
data is analyzed interpreted and used to
derive meaningful insights here are some
key ways in which AI is leveraging
analytics number one data processing and
preparation AI algorithms automate and
streamline data processing tasks
including data cleaning data integration
and data transformation this enables
faster and more efficient data
preparation ensuring that high quality
data is available for analysis the next
is pattern recognition and anomal
detection AI powered analytics systems
excel at recognizing complex patterns in
large data sets machine learning
algorithms can identify hidden patterns
Trends and relationships that may not be
apparent to human analyst additionally
AI algorithms can detect anomalies or
outliers in data helping to ID identify
potential issues or fraudlent activities
and the next is Predictive Analytics AI
techniques such as predictive modeling
and forecasting enable organizations to
make accurate predictions about future
events or outcomes based on historical
data by leveraging AI algorithms
business can anticipate customer
Behavior demand Trends Market changes
and more allowing for proactive decision
making and the next we have is natural
language processing and text analytics
AI powered NLP techniques enable the
analysis of unstructured data such as
text documents social media posts
customer reviews and emails by
extracting and analyzing textual
information organizations can gain
valuable insights into customer
sentiment market trends and emerging
topics and the next we have is automated
insights and Reporting AI power
analytics systems can automatically
generate meaningful insights and reports
Based on data analysis this reduces the
time and effort required for manual
analysis and Reporting allowing analyst
to focus on higher value tasks like
interpretation and strategy development
and the next we have is personalization
and recommendation systems AI algorithms
are employed in recommendation systems
that provide personalized suggestions to
users based on their preferences
Behavior and historical data this is
particularly prevalent in e-commerce
streaming services and content platforms
where air driven recommendations enhance
user experiences and drive customer
engagement and next we have is
optimization and decision support a
algorithms can optimize complex
processes and decision making by
evaluating multiple variables
constraints and scenarios for example AI
power optimization models can ass assist
in Supply Chain management resource
allocation inventory planning and
pricing strategies that helps
organizations achieve better operational
efficiency and cost savings and the next
we have is continuous learning and
Improvement AI systems have the ability
to continuously learn and adapt based on
new data and feedback this allows
analytics models to improve their
accuracy and performance over time
making them more effective in generating
value able insights and predictions and
by leveraging AI capabilities in
analytics organizations can unlock the
full potential of their data gain a
competitive advantage and drive
datadriven decision making across
various domains the Synergy between Ai
and analytics is revolutionizing how
businesses extract insights from data
leading to transformative outcomes and
Innovations if you are watching this I
believe you are either an aspiring data
analyst or a Prof professional already
now the question of the r is Will AI
replace you should I even care to
proceed with a career in data analytics
don't worry we are here to answer your
questions and make things less
complicated or we believe we can sort it
out now ai is designed to assist humans
for example CH jity B and B these smart
AI language models can automate the time
consuming task such as writing down
emails performing data extraction
transformation and loading and
exploratory analysis and Reporting
however in the real time every
organization has specifically customized
needs and requirements a special set of
kras and kpis to address you may have to
understand the business requirements and
build a custom dashboards you may have
to understand the real-time streaming of
data integrate AWS with powerbi not just
these there are a gazillion task
that need human intervention and
Logics a still lacks the human cognition
and the ability acquiring the domain
knowledge which only a human can you as
a data analyst may need to understand
that AI can perform a redundant and
straightforward task but not the
critical operations which completely
rely on a
human only a human has the caliber to
decode the intentions of the client and
simplify the technical Concepts to non
technical client has a strong focus on
the business outcomes most importantly
try to unravel the potential of AI and
leverage it to improvise and gete if you
are someone that not only performs
analysis but also is a torch Bearer the
organization with the right business
strategies in very short time then you
are someone which all major Tech Giants
need a study from new Vantage Partners
suggest 97.2% of companies are now
investing in data and its analysis
nowadays every company needs a data
expert do you also want to become a part
of this Market if yes how to become a
part of it what are the essential skills
one should
have this video will answer all these
questions but before watching this video
please subscribe to Simply learn's
YouTube channel and press the Bell icon
to never miss any updates question for
you which one of the following is not a
python library for
visualization M lot lip scipi Panda
jupyter notebook please leave the answer
in the comment section below moving on
who is a data analyst a data analyst
collects analyzes and interprets data a
data analyst will convert raw data into
useful information data analyst are in
high demand because every industry uses
data analysis work of a data analyst as
a data analyst you will work closely
with the raw data and generate valuable
insights to help companies decide their
future goal if you like thinking out of
the box you are the perfect fit for this
domain data analyst help maximize output
when it comes to generating Revenue
working closely with both business and
data nevertheless this field boost
handsome salaries for all levels of
expertise can you become a data analyst
without prior experience yes anyone can
become a data analyst if they enjoy
solving real world problems have a
strong background in statistics and have
a creative mind if you feel you don't
have it you can definitely develop it so
let us know the skills in detail what
are the basic skill sets required for a
data analyst data analyst must know
basic mathematics and statistics
programming skills machine learning and
also data visualization tools so let us
know what are the basics that you need
to learn as a data
analyst mathematics it is always better
to know basic mathematics like linear
algebra and probability fundamentals
linear algebra is used in data
pre-processing and transformation which
is the critical process of every data
analyst statistics a branch of
mathematics that deals with collection
analysis presentation and
implementation probability we know that
probability is the study of How likely
something will happen which is essential
for concluding both probability and
statistics are the backbones of data
analysis it is phys to become a data
analyst with only a basic understanding
of these three areas of mathematics but
in order to remain relevant and grow as
a data analyst one's mathematical
knowledge should not be restricted
compulsorily use some of the tools as a
data analystic what are
that first is Microsoft Excel it is the
most well-known spreadsheet software in
the world it also has computation and
graphick features that are excellent for
data analyst no matter your area of
expertise or additional software you
might want Excel is a standard in the
IND industry its useful built-in
features include form design tools and
pivot table it also generate a wide
range of additional features that help
simplify data
manipulation as a programming language
every data analyst should know python it
is easy to learn and has a simple syntax
python is quite adaptable and includes a
vast variety of resource libraries that
are appropriate for a wide range of
diverse data analytics activities these
libraries help in numerical and data
computation the pandas and numai
libraries for instance are excellent for
supporting standard data processing and
streamlining highly computational
operations you can also choose between
Python or r r is a well-known open
source programming language much like
python data visualization tool as we
previously mentioned data visualization
tool is also necessary to become a data
analyst powerbi is a userfriendly
interface makes building interactive
visual reports and dashboard simple its
most vital selling point is its superb
data integration it works flawlessly
with Cloud sources like Google and
Facebook analytics as well as text files
SQL servers and
Excel is one of the best commercial data
analysis tool available it handles huge
amounts of data better than many other
bi tools and is effortless it has a
visual drag and drop interface
however because it has no scripting
layer there is a limit to what Tableau
can do for example it could be better
for pre-processing data or building more
complex
calculations you might have heard about
MySQL a lot of time it is a standard
language for interacting with databases
and it is very helpful when working with
structur data SQL creates userfriendly
Das ports that may present in various
data ways in since it is so simple to
send complex commands to databases and
change data in seconds it has commands
like add edit delete data in addition
SQL is an excellent tool for creating
data warehouses because of its
Simplicity Clarity and
interactivity overall I would suggest
that to become a data analyst you should
work on programming languages like
python or R plus MySQL to work on
databases adding to that Excel plus
visualization to like Tablo or powerbi
you now know what are the skills are and
how it is used what are you up to in an
organization as a data
analyst to create and evaluate the
report using automated tools like
Tableau or
powerbi to troubleshoot the reporting
database environment and reports data
analyst you will use statistical method
to analyze data sets and spot any
valuable trends that may develop over
time
evaluate company's functional and
non-functional requirement data analyst
assess data warehousing in inspecting
and Reporting needs these are all the
responsibility of a data analyst in an
organization coming to companies hiring
a data analyst IBM accentor capam mini
TCS Facebook Amazon flip cart meta these
are the top companies hiring a data
analyst but data suggest that every
small and mediumsized company needs a
data analyst therefore demand of a data
analyst is in every company so there is
no need to worry job and salary of a
data analyst this is the final part the
salary of a data analyst is high all
over the world when it comes to the USA
the average salary for a data analyst as
a beginner is going as high as 70,000
plus per anom for experienced
professional it is going as high as
$120,000 perom in India for a fresher it
is going as high as 8 lakh perom and for
experienced professionals it is 20 lakh
plus per anom such is the demand for
data analyst now that we have covered
every important skill it's time for you
to start working on it hello and welcome
to data analytics interview questions my
name is Richard kersner with the simply
learn team that's www.s simplylearn
today we're going to jump into some
common questions you might see on numpy
arrays and Panda's data frames in the
python along with some Excel tblo and
SQL let's start with our first question
what is the difference between Data
Mining and data
profiling it's really important to note
that data mining is a process of finding
relevant information which has not been
found before it is a way in which raw
data is turned into valuable information
you can think of this as anything from
the
cells uh stats and from their SQL Server
all the way to web scraping and Census
Bureau information where the heck do you
mine it from where do you get all this
data and
information then we look at data
profiling is usually done to assess a
data set for its uniqueness consistency
and logic it cannot identify incorrect
or inaccurate data values so if somebody
has a statistical analysis on one side
and they doing there you might the wrong
data to then program your data setup so
you got to be aware of that when you're
talking about data mining you need to
look at the Integrity of what you're
bringing in where it's coming from data
profiling is looking at it and saying
hey how is this going to work what's the
logic what's the consistency is it
related to what I'm working with find
the term data wrangling and data
analytics data wrangling is a process of
cleaning structuring and enriching the
raw data into a desired usable format
for better decision decision making and
you can see a nice chart here with our
Discover it we structure the data how we
want it we clean it up get rid of all
those null values we enrich it so we
might take and reformat some of the
settings instead of having uh five
different terms for height of somebody
you know an American English or whatever
who clean some of that up and we might
do a calculation and bring some of them
together and validate I was just talking
about that in the last one need to
validate your data make sure you have a
solid data source and and then of course
it goes into the anal is very important
to notice here in data wrangling 80% of
data analytics is usually in this whole
part of wrangling the data getting it to
fit correctly and don't confuse that
with data cooking which is actually when
you're going into neural networks
cooking the data so it's all be between
zero and one
values what are common problems that
data analysts encounter during
analysis handling duplicate and missing
values collecting the meaningful right
data the right time making data secure
and dealing with compliance issues
handling data purging and storage
problems again we're talking about data
wrangling here 80% of most jobs are en
wrangling that data and getting it in
the right format making sure it's good
data to
use number four what are the various
steps involved in any analytics
project understand the problem we might
spend 80% doing wrangling but you better
be ready to understand the problem
because if you can't you're going to
spend all your time in the wrong
direction this is probably uh the most
important part of the process everything
after it falls in and then you can come
back to it two data collection data
cleaning number three four data
exploration analysis and five interpret
the results number five is a close
second for being the most important if
you can't interpret what you bring to
the table to your clients you're in
trouble so when this question comes up
you probably want to focus on those two
noting that the rest of it does 80% of
the work is in two three and four while
one and five are the most important
parts which technical tools have you
used for analysis and presentation
purposes being a data analyst you are
expected to have knowledge of the below
tools for analysis and presentation
purposes there's a wide variety out
there uh SQL Server MySQL you have your
Excel your SPSS which is the IBM
platform tblo python uh you have all
these different Tools in here now
certainly a lot of jobs are going to be
narrowed in on just a few of these tools
like you're not going to have a
Microsoft SQL Server MySQL server but
you better understand how to do basic
SQL polls and also understanding Excel
and how the different formats um for
column and how to get those set
up number six what are the best best
practices for data cleaning this is
really important to remember to go
through this in detail these always come
up because 80% of uh most data
analysises in cleaning the data make a
data cleaning plan by understanding
where the common errors take place and
keep Communications open identify and
remove duplicates before working with
the data this will lead to an effective
data analysis process focus on the
accuracy of the data maintain the value
types of data provide mandatory
constraints and set Cross Field
validation standardize the data at the
point of entry so that is less chaotic
and you will be able to ensure that all
the information is standardized leading
to fewer errors on
Entry number seven how can you handle
missing values in a data set listwise
deletion and listwise deletion method
entire record is excluded from analysis
if any single value is missing sometimes
we're talking about records remember
this could be a single line in a
database base so if you have uh your SQL
comes back and you have 15 different
columns every one of those that has a
missing value you might just drop it
just to make it easy because you already
have enough data to do the processing
average imputation use the average value
of the responses from the other
participants to fill in the missing
value this is really useful uh and
they'll ask you why these are useful I
guarantee it uh if you have a whole
group of data that's collected and it
doesn't have that information in it at
that point you might average it in there
regression substitution you can use
multiple regression analysis to estimate
a missing value I kind of goes with the
average imputation input uh regression
model means you're just going to get
you're going to actually generate a
prediction as to what you think that
value should be for those people based
on the ones you do have multiple
imputation so we talk about multiple
inputs uh it creates plausible values
based on the correlations for the
missing data and then average the
simulated data sets by incorporating
ating random errors in your
predictions what do you understand by
the term normal distribution and the
second you hear the word normal
distribution should be think in a bell
curve like we see here normal
distribution is a type of continuous
probability distribution that is
symmetric about the mean and in the
graph normal distribution will appear as
a bell curve the mean median and mode
are equal that's a quick way to know if
you have normal distribution is you can
compute mean median and mode all of them
are located at the center of the
distribution 68% of the data lies within
one standard deviation of the mean 95%
of the data Falls within two standard
deviations of the mean 99.7% of the data
lies within three standard deviations of
the
mean what is time series analysis time
series analysis is a statistical method
that deals with ordered sequence of
values of a variable of equally spaced
time intervals time series data on a
covid-19 cases and you can see we're
looking at by days so our space is of
days and each day goes by if we take and
graph it you can see a Time series graph
always looks really nice if you have
like two different in this case we have
what the United States going over there
I'd have to look at the other setup in
there but they picked a couple different
countries uh and it is it's time
sensitive you know the next result is
based on what the last one was Co is an
excellent example of this uh any you do
any word analytics where you're figuring
out what someone's saying what they said
before makes a huge difference as what
they're going to say next another form
of Time series
analysis 10 how is joining different
from blending in Tabo so now we're going
to jump into the tblue package data
joining data joining can only be done
when the data comes from the same Source
combining two tables from the same
database or two or more worksheets from
the same Excel file all the combined
tables or sheets contains common set of
dimensions and
measures data blending data blending is
used when the data is from two or more
different sources combining the Oracle
table with the SQL server or two sheets
from Excel or combining Excel sheet and
Oracle table in data blending each data
source contains its own set of
dimensions and
measures how is overfitting different
from underfitting
always a good one uh overfitting
probably the biggest uh danger in data
analytics today is overfitting model
trains from the data too well using the
training set the performance drops
significantly over the test set happens
when the model learns the noise and
random fluctuations in the training data
set in detail and again the performance
drops way below what the test set has
the model neither trains a data well nor
can generalize to new data performs
poorly both on train and the test set
happens when there is less data to build
and an accurate model and also when we
try to build a linear model with a
nonlinear
data in Microsoft Excel a numeric value
can be treated as a text value if it
proceeds with an apostrophe definitely
not an exclamation uh if you're used to
programming in Python you'll look for
that hash code and not an Amber
sign and we can see here uh if you enter
the value 10 into a fi but you put the
apostrophe in front of it it will read
that as a text not as a
number what is the difference between
count count a count blank and count if
in
Excel we can see here when we run in
just count D1 through
d23 we get 19 and you'll notice that
there is 19 numbers coming down here
and so it doesn't count the cost of each
which is a top bracket it doesn't count
the blank spaces either with the
straight count when you do a count a
you'll get the answer is 20 so now when
you do count a it counts all of them
even the title cost of
each when you do count blank we'll get
three why there's three blank
fields and finally the count if if we do
count if of E1 to E2 three is greater
than 10 there's 11 values in there basic
counting of whatever is in your column
pretty solid on the table
there explain how vlookup Works in
Excel vlookup is used when you need to
find things in a table or a range by
Row the syntax has four different parts
to it uh we have our lookup value that's
a value you want to look up we have our
table
array uh the range where the lookup
value is
located column index number the column
number and range that contains the
return value and the range lookup
specify true if you want an approximate
match or false if you want an exact
match of the return
value so here we see V lookup F3 A2 to
C8 2 comma 0 for prints now they don't
show the F3 F3 is the actual um cell
that prints is in that's what we're
looking at is F3 so there's your prints
he pulls in from F3 A2 to C8 is the the
data we're looking into and then number
two is a column in that data so in this
case we're looking for a uh age and we
count name is one age is two keep in
mind this is Excel versus a lot of your
Python and programming languages where
you start at zero in Excel we always
look at the cells as one two three so
two represents the age zero is uh false
for having an exact matchup versus one
we don't actually need to worry about
that too much in this zero or one would
work with this example and you can see
with the Angela lookup again her name
would be in the F column number four
that's what the F4 stands for is where
where they pulled Angela from and then
you have A1 to C8 and then we're looking
at uh number three so number three is
height name being one age two and then
height three and you'll see here pulls
in her height 5.8
so we're going to run jump over to uh
SQL how do you subset or filter data in
SQL to subset or filter data in SQL we
use where and having clause and you can
see we have a nice table on the left
where we have the title the director the
year the duration we want to filter the
table for movies that were directed by
Brad Bird um why just because we want to
know who what Brad Bird did so we're
going to do select star you should know
that the star refers to all in this case
where what are we going to return we're
going to return all title directory year
and duration that's what we mean by all
from movies movies being our table where
director equals Brad Bird and you can
see um he comes back and he did the
incredible and Rit
twoe to subset or filter data SQL we can
also use the wear and having claws so
we're going to take a closer look at the
um different ways we can filter it here
filter the table for directors whose
movies have an average duration greater
than 115 minutes so there's a lot of
really cool things into this SQL query
and these SQL queries can get pretty
crazy select director sum duration as
total duration average duration as
average duration from movies Group by
director having average duration greater
than
115 uh so again what are we going to
return we're going to return whatever we
put in our select which in this case is
director we're going to have total
duration and that's going to be the sum
of the duration we're going to have the
average duration average underscore
duration which is going to be the
average duration on there and then we of
course go ahead and group by director
and we want to make sure we group them
by uh anyone that has an having an
average duration greater than
115 these SQL queries are so important I
don't know how many times your the SQL
comes up and there's so many different
languages not just MySQL and not
Microsoft SQL but in addition to that
where the SQL language comes in
especially with Hadoop and other areas
so you really should know your basic
SQL doesn't hurt to get that little um
cheat sheet and glance over it and
double check some of the different
features in
SQL what is the difference between where
and having clause in SQL where wear
Clause works on row data in wear Clause
the filter occurs before any grouping
are made aggregate functions cannot be
used uh so the syntax is select your
columns from table where what the
condition is having Clause works on
aggregated data having is used to filter
values from a group aggregate functions
can be used in the syntaxes select
column names from table where the
condition is grouped by having a
condition ordered by column names
what is the correct Syntax for reshaped
function in numpy so we're going to jump
to the numpy array program and what you
come up with is you have uh in this case
it be numpy do reshape a lot of times
you do an import numpy as NP reshape and
then your array and the new
shape and you can see here as we uh as
the actual um example comes in the
reshape is a and we're going to reshape
it in two comma five uh setups and you
can see the print out in there that
prints in two rows with five values in
each
one what are the different ways to
create a data frame in
pandas well we can do it by initializing
a list so you can import your pandas as
PD very common data equals Tom 30 Jerry
20 Angela 35 we'll go ahead and create
the data frame and we'll say uh pd.
dataframe is the data columns equals
name and age so you can designate your
columns you can also designate index in
there should always remember that the
index uh in this case maybe you want the
index instead of one two to be um the
date they signed up or who knows you
know whatever and you can see right
there it just generates a nice pandas
data frame with Tom Jerry and Angela
another way you can initialize a uh data
frame is from dictionary you can see
here we have a dictionary where the date
equals name Tom Jerry Angela Mary age is
20 21 19 18 and if we do a dfpd
. dataframe on the data you'll get a
nice the same kind of setup you get your
name age Tom Jerry Angela and
Mary write the python code to create an
employees data frame from the empcs v
file and display the head and summary of
it to create a data frame in Python you
need to import the Panda's library and
use the read CSV function to load the
CSV file and here you can see where we
have import pandas as PD employees or
the data frame employees equals pd. read
CSV and then you have your path to that
CSV file there's a number of settings in
the read CSV where you can tell it how
many rows are the top index uh you can
set the columns in there you can have uh
skip rows there's all kinds of things
you can also go in there and double
check with your read CSV but the most
basic one is just to read a basic
CSV how will you select the department
and age column from an employees data
frame so we have import pandas as pedd
you can see we have created our data uh
we will go ahead and create our
employees PD data frame on the left and
then on the right to select department
and age from the data frame uh we just
do employees and you put the brackets
around it now if you're just doing one
column you could do just department but
if you're doing multiple columns you got
to have those in a second set of bracket
so it's got to be a reference with a
list within the
reference what is the criteria to say
whether a developed data model is good
or not a good model should be intuitive
insightful and
self-explanatory follow the old saying
kiss keep it simple the model developed
should be able to easily consumed by the
clients for actionable and profitable
results so if they can't read it what
good is it a good model should easily
adapt to changes according to business
requirements we live in quite a dynamic
world nowadays so that's pretty
self-evident and if the data gets
updated the model should be able to
scale accordingly to the new data so you
have a nice data pipeline going where
when something when you get new data
coming in you don't have to go and
rewrite the whole
code what is the significance of
exploratory data
analysis exploratory data analysis is an
important step in any data analysis
process exploratory data analysis Ed
helps to understand the data better it
helps you obtain confidence in your data
to a point where you're ready to engage
a machine learning algorithm it allows
you to refine your selection of feature
variables that will be used later for
model building you can discover hidden
Trends and insights from the data how do
you treat outliers in a data set an
outlier is a data point that is distant
from other similar points they may be
due to variability in the measurement or
may indicate experimentally
errors uh one you can drop the outlier
records pretty straightforward you can
cap your outlier data so it doesn't go
past a certain value you can assign it a
new value you can also try a new
transformation to see if those outliers
come in if you transform it slightly
differently explain descriptive
predictive and prescriptive analytics
descriptive provides insights into the
past to answer what has happened uses
data aggregation and data mining
techniques examples an ice cream company
can analyze how much ice cream was sold
which flavors were sold and whether more
or less ice cream was sold than
before predictive understands the future
to the answer what could happen uses
statistical models and forecasting
techniques example predicts the sale of
ice creams during the summer spring and
rainy days uh so this is always
interesting because you have your
descriptive which comes in and your
businesses are always looking to know
what happened hey did we have good sales
last uh quarter what are we expecting
next quarter in sales and we have a huge
jump when we do uh
prescriptive suggest various courses of
action to answer what should you do uses
optimization and simulation algorithms
to advise possible outcomes example
lower prices to increase sell of ice
creams produce more or less quantities
of certain flavor of ice cream and we
can certainly uh today's world with the
covid virus because we had that on our
earlier graph you could see that as a
descriptive what's happened how many
people have been infected how many
people have died in an area predictive
where do we predict that to go um do we
see it going to get worse is it going to
get better what do we predict that we're
going to need in hospital beds in
prescriptive what can we change in our
uh setup to have a better outcome uh
maybe if we did more social distancing
if we track the virus how do these
different things directly affect the end
and can we create a better ending by
changing some underlying uh
criteria what are the different types of
sampling techniques used by data
analysists sampling is a statistical
method to select a subset of data from
an entire data set population to
estimate the characteristics of the
whole population one we can do a simple
random sampling so we can just pick out
500 random people in the United States
to sample them they call it a population
in regular data we also call that a
population just because that's where it
came from was mainly from doing
census systematic sampling cluster
sampling stratified sampling and
judgment or purposive sampling then we
have our systematic sampling that's
where you're doing like uh uh using one
five 10 15 20 using very systematic
approach for pulling samples uh from the
setup cluster sampling uh that's where
we look at it and we say hey some of
these things just naturally group
together if you were talking about
population which is the really a nice
way of looking at this cluster sampling
would be maybe by ZIP code we're going
to do everybody's zip code and just
naturally cluster it that way stratified
sampling would be more uh looking for
shared things the group has like income
uh so if you're studying something on
poverty you might look at their
naturally group people uh based on
income to begin with and then study
those individuals in the income to find
out what kind of traits they have and
then judgmental uh that is where the uh
researcher very carefully selects each
member of their own group uh so it's
very much um based on their personal
knowledge jumping on the 26 what are the
different types of hypothesis testing
hypothesis testing is a procedure used
by staticians and scientists to accept
or reject statistical hypothesis we
start with the hypothesis testing we
have null hypothesis and alternative
hypothesis on the null hypothesis it
states that there is no relation between
the predictor and the outcome variables
in the population it is denoted by H
notot example there is no association
between patient
BMI and
diabetes alternative hypothesis it
states there is some relation between
the predictor and outcome variables in
the population it is denoted by H1
example there could be an association
between patients BMI and
diabetes and that's the body mass index
if you didn't catch the BMI and you're
not
MediCal describe univariate bivariate
and multivariate Analysis a univariate
analysis it is the simplest form of data
analysis where the data being analyzed
contains only one variable an example of
studying the heights of players in the
NBA because it's so simple it can be
described using Central Tendencies
dispersion quartiles bar charts
histograms pie charts frequency
distribution tables the B variate
analysis it involves analysis of two
variables to find causes relationship
ships and correlations between the
variables example analyzing sale of ice
creams based on the temperature out side
byari analysis can be explained using
correlation coefficients linear
regression logistic regression Scatter
Plots and box
plots and multivariate Analysis it
involves analysis of three or more
variables to understand the relationship
of each variable with the other
variables
example analyzing Revenue based on
expenditure so if we have our TV ads we
have our newspaper ads our social media
ads and a revenue we can now compare all
those together the multi verted analysis
can be performed using multiple
regression factor analysis
classification and regression trees
cluster analysis principal component
analysis clustering bar chart dual AIS
chart what function would you use to get
the current date and time in Excel in
Excel you can use the today and now
function to get the current date and
time and you can see down here with the
two examples we just equals today or
equals
Now using the sum ifs function in Excel
find the total quantity sold by celles
Representatives whose names start with a
and the cost of each item they have sold
is greater than
10 and you can see here on the left we
have our actual table
and then we want to go ahead and sum ifs
so we want the uh E2 through E20 B2
through B20 greater than 10 and this
basically is just saying hey we're going
to take everything in the uh e column
and we're going to sum it up but only
those objects where the D column is
greater than 10 that's what that means
there is the below query correct if not
how will you Rectify it
select customer ID year order date as
order Year from order where order year
is greater than or equal to
2016 and hopefully you caught it right
there uh it's in the devils in the
details we can't not use the Alias name
while filtering data using the wear
Clause so the correct format is all the
same except for where it says where the
year order date is greater than or equal
to 16 versus using the order year which
we assign under the select
setup how are union intersect and accept
used in
SQL the union operator is used to
combine the results of two or more
select
statements and you can see here we have
select star from region one and we're
going to make a union with select star
from region two and it basically takes
both these SQL tables and combines them
to form a full new table on there so
that's your union as we bring everything
together when we look at the intersect
operator Returns the common records that
are the result of the two or more select
statements so you can see here we select
star from region one intersect select
star from region two and we come up with
only those records that are shared that
have the same data in them and hopefully
you jumped uh ahead to the accept the
accept operator returns The Uncommon
records that are the result of two or
more select statements so these are the
two records or the records that are not
shared between the two
databases using the product price table
write an SQL query to find the record
with the fourth highest market
price so here we have a little bit of a
brain
teaser uh they're always fun and the
first thing we want to do is we're going
to go ahead and uh I'm going to if you
look at the uh script on the left we
really want the fourth one down so we're
going to select the top four from
product price but we're going to order
it by market price descending SP order
by market price ascending so we do is we
take the top four of the market price
ascending and that's going to give us
the four greatest values and then we're
going to reverse that order and do
descending and we're going to take the
top one of that which is going to give
us the lowest value which will be the
fourth greatest one in the
list from the product price table find
the total and average market price for
each currency where the average market
price is greater than 100 and currency
is in the INR or the
AUD so um INR or AUD India ruple or
Australia dollar you can see over here
the SQL query if you had trouble putting
this together uh you might actually do
some of in reverse and you can see right
here where the average market price is
greater than 50 remember we use having
not where at the end because it's part
of the group so Group by currency
because we want those two currencies and
we want the currency India R the INR or
the AUD and um as you keep going
backwards we're actually going to be
selecting the currency the Su of the
market price as total price and the
average market price as average price so
there's our select it's going to come
from the product price which is just our
table over there and then we have where
our currency is in uh and like I said
you can put it together however you want
but hopefully you got to the end
there so this question will test your
knowledge in tblo exploring the
different features of tblo and creating
a suitable graph to solve a business
problem and of course tblo is very
visual in its use so it's very hard to
test it without actually just getting
your hands on and if you can't visualize
some of this and how to do it and you
should go back and refresh yourself
using the sample superstor data set
Create A View to analyze the cells
profice and quantity sold across
different subcategories of items present
under each category so the first step is
to go ahead and load the sample
superstor data set so make sure you know
how to load the sample the superstore
data set that's underneath either the
connect button in the upper left um or
the um table icon up there and be able
to pull in the data set and then once
you've done that you just drag the
category and subcategory on row and
salaries onto columns it will result in
a horizontal bar
chart so in this one we're just going to
drag profit onto color and quantity onto
label sort the celles axes in descending
order of sum and cells within each sub
category and if you're at home doing
this you'll see that chairs under
Furniture Cate atory had the highest
sales and profit while tables had the
lowest profit for office supplies
subcategory binders made the highest
profit even though storage had the
highest sales under technology category
copers made the highest profit though it
was the least amount of
sales let's work to create a dual axis
chart in Tabo to present sales and
profits across different years using
sample Superstore data set load the
orders sheet from the sample Superstore
data
set drag the ordered data field from the
dimensions onto columns and convert it
into continuous
month drag sells onto rows and profits
to the right corner of the view until
you see a light green rectangle one of
those things if you haven't done this
Hands-On you don't know what you're
doing you're you're going to run into a
bank so you're going to be just kind of
dropping it and wondering what happened
synchronize the right axes by right
clicking on the profit axes
and then let's finalize it by going
under the marks card change some celles
to bar and some profit to line and
adjust the size and then we have a nice
display that we can either print out or
save and send off to the uh
shareholders let's go and do one more
table uh design a view in table to show
statewise sales and profits using the
sample Superstore data set in here you
go ahead and drag the country field onto
the view section and expand it to see
the states drag the states fi onto size
and profit onto
color increase the size of the bubbles
add a border and a Halo color States
like Washington California and New York
have the highest sales and profits while
Texas Pennsylvania and Ohio have a good
amount of sales but the least amount of
profits we'll go ahead and Skip back to
python numpy Suppose there is an array
number equals NP or numpy if you're
using numpy depending on how you set it
up do array and we just have one to nine
broken up into three groups extract the
value eight using 2D indexing so you can
see on the left we have our import nump
as NP number equals our NP array if we
print the number we have 1 2 3 4 5 6 7 8
nine since the value8 is present in the
second row and First Column we use the
same index position and pass it to to
the array and you just have number 2
comma 1 and you get eight and remember
we're in Python so you start at zero not
one like you do in Excel always gets me
if I'm working between Excel and python
where I just kind of flip and usually
it's the Excel that messes up because I
do a lot more
programming suppose there's an array
that has values 01 all the way up to
nine how will you display the following
values from the array 1 3 5 7
nine uh so first of all we go and create
the array uh np. a range of 10 which
goes from 0o to nine because there's 10
numbers in it but we don't include the
10 we print it out the first thing you
want to do is what's going on here with
1357 9 well if we divide by two there's
going to be a remainder equal to one and
then from python remember that if you
use the percentage sign you get the uh
remainder on there so the remainder one
and then you have the your numpy array
and then we just want to do um a logical
statement of all values that have a
remainder of one and that generates our
nice
13579 there are two arrays A and B stack
the arrays A and B
horizontally boy these horizontal
vertical questions will get you every
time and in numpy we go ahead and we've
created uh two different arrays over
here A and B uh the first one is your
concatenate np. concatenate A and B on
axes equal
one that is the same as
hstack and in the back end they're still
identical they run the same that's all
hstack is a concat it axis equals
one how can you add a column to a
panda's data frame suppose there's an
imp data frame that has information
about few employees let's add address
column to that data frame and you can
see in the left we have our basic data
frame uh you should know your data
frames very well uh basically looks like
an Excel spreadsheet as you come over
here it's really simple you just do um
DF of address equals the address once
you've assigned values to the
address using the below given data
create a pivot table to find the total
cells made by each cells represented for
each item display the cells as a
percentage of the grand total so we're
back in uh table select the entire table
range click on insert Tab and choose
pivot
table select the table range and the
worksheet where you want to place the
pivot table it will return a pivot table
where you can analyze your
data uh drag the cell total on the
values and cells rep an item onto row
labels it'll give the sum of the cells
made by each representative for each
item they have
sold and finally right click on sum of
sell total and expand show values as to
select percentage of grand
total uh real important just to
understand what a pivot table is we're
just pivoting it from rows and columns
and switching this direction on there
and finally uh we have our final pivot
table and you can see the values rolls
and sum of total
sale so we're going to go ahead and take
a product table this is off of of an SQL
so we're going to do some SQL here and
we're going to use the product and sales
order detail table find the products
that have total units sold greater than
1.5 million and here's our sales order
detail table so we have a product table
and a sales order detail table two
separate tables in the database and
we're going to do is put together the
SQL query we want to select PP name sum
sod unit price as set Sals and then we
have our pp. product ID from production
product as PP inter jooin sales. sales
order detail as sod on PP product ID
equals s. product ID Group by pp. name
comma pp. product ID having a sum of s.
unit price greater than uh the 150
million there that's a mouthful and
again these SQL queries they start
looking really crazy until you just
break them apart and do them step by
step and what we're looking for is the
in join and how did you do the group by
this really want they know how do you do
this in join this comes up so much in
SQL uh how do you pull in the ID from
one chart and the information from
another chart and the sum totals on that
chart how do you write a stored
procedure in SQL let's create a storage
procedure to find the sum the squares of
the first in natural number numbers so
here we have our formula n * n + 1 * 2 n
+ 1 / 6 and you can see from the command
prompt uh or the setup you have
depending on what your login is the
command is create procedure Square sum
one declare a variable at n of integer
as begin then we're going to declare the
sum of integer set sum equals n * n + 1
+ 2 * n + 1 um over 6 and of course we
can go ahead and print those out print
First cast um Ember sign n or our
variable as a variable character 20
natural numbers parth the sum of the
square is cast the at sum as variable
character 40 in then we do the output
display the sum of the square for first
four natural numbers we have execute
Square sum one and then we're going to
put in four and you can see here where
brings up the first four natural number
sum of square is
30 write a store procedure to find the
total even number between two user given
numbers couple things to note here first
we go and create our procedure you have
your two different variables the N1 N2
and we go ahead and begin we're going to
declare our variable count as an integer
we're going to set count equal to zero
and then we have while n is less than N2
we're going to begin and if N1 remainder
two equal 0 so we're going to divide it
by two even number begin we're going to
set the count equal to count plus one
we're going to print even number plus
cast N is a variable character 10 for
printing count is plus cast variable
count as variable character 10 and else
print odd number plus cast variable
number one as variable character 10 and
then we go ahead and set the um
increment our variable one up one so
they go from N1 all the way to N2 and
it'll print the total number of even
numbers and you can see here we went
ahead and executed it we're going to
count the even numbers between 30 and 45
and you just see it goes all the way
down to
eight what is the difference between
tree maps and heat maps in
Tabo now if you've worked in Python
other
programmingsoftware
use Dimensions to define the structure
of the tree map and measure to define
the size or color of individual
rectangles tree maps are a relatively
simple data visualization that can
provide Insight in a visually attractive
format and again you can see the squares
over here this is our tree map over here
with the each block also has this
information inside of its different
blocks a heat map helps to visualize
measures against Dimensions with the
help of colors and size to compare one
or more dimensions and up to two
measures the layout is similar to a text
table with variations and values encoded
as colors in heat map you can quickly
see a wide array of information and in
this one uh you can see they use the
colors to denote one thing and the size
of the little square to denote something
else a lot of times you can even graph
this into a three-dimensional graph with
other data uh so it pops out but again a
heat map is the color and the size
using the sample superstor data set
display the top five and bottom five
customers based on their profit so you
start by dragging the customer name fi
onto rows and profit on columns
rightclick on the customer name column
to create a set give a name to the set
and select top tab to choose top five
customers by some profit similarly
create a set for the bottom five
customers by some profit
select both the sets right click to
create a combined set give a name to the
set and choose all members in both
sets and then you can drag top and
bottom customer sets onto the filters
and profit field onto color to get the
desired
results as we get down to the end of our
list we're going to try to keep you uh
on your toes so we're going to skip back
to numpy how to print four random
integers between 1 and 15 using numy to
Generate random numbers using numpy we
use the random random integer function
you can see here we did the import numpy
is NP random Arrangement equals np.
random. random integer 1 through 15 of
four from the below data frame going
jump again on you now we're into pandas
how will you find the unique values for
each column and subset the data for age
less than 35 and height greater than six
to find the unique values and the number
of unique elements use the unique and
the in unique
function you can see here we just did DF
height so we're selecting just the
height column and we want to look for
The Unique that returns an array where
in unique if we do that on the height or
the age we'll return just the number of
unique values and then we can do a
subset the data for ages less than 35
and height greater than six so if you
look over here we have a new DF uh
remember this is going to be taking
slices of our original data frame it
doesn't actually change the data frame
so our new DF equals the data frame or
DF the data frame where age is less than
35 and the height is greater than
six and in case you're not using uh TBL
blue which has a lot of its own uh
different mapping programs in there make
sure you understand how to use the
basics of map plot Library Library plot
assign graph using numpy and matplot
library and Python and the way we did
this is we went ahead and generate an X
we know our y equals np. sin ofx if you
print out X you'll see a whole value
here our Matt plot Library P plot as PT
if you are working in Jupiter notebook
make sure you understand the matap plot
Library inline that little percentage
sign met plot Library inline that prints
it on the page in the Jupiter notebook
the newer version of jupyter notebook or
U Jupiter Labs automatically does that
for you but I usually put it in there
just in case I end up on an older
version if you print y you can see here
we have our different y values and our
different X
values you simply put in pt. plot XY and
do a plot
show and before we go let's get one more
in we're going to do a pandas uh using
the below pandas data frame find the
company with the highest average cells
derive the summary statistics for this
CES column and transpose these
statistics that's a mouthful and just
like any of these computer problems
break it apart uh so first of all we're
looking for the highest average cells so
group the company column and use the
mean function to find the average sales
you can see here buy company equals DF
do group by
company once we've done that using the
describe function we can now go ahead
and look at the summary of statistics
ICS on here use the describe function to
find the summary uh so by company those
are groups we're just going to describe
them and you could actually bundle those
together if you wanted and just do them
all in one line uh so here we go bu
company. discr you can see we have a
nice breakout always good to remember uh
whether you're using any of the packages
whether it's tblo or uh pandas in python
or even r or some other package being
able to quick look and describe your
data is very very important and then we
can go ahead and just do a basic apply a
transpose function over the describ
method to transpose the statistics all
we've done here is flip the index with
the column names but if you're following
the numbers a lot of times it's easier
to follow across one line or maybe you
want to average out the count or there
all kinds of different reasons to do
that and with that we have reached to
the end of the session on data analytics
course should you need any assistance PP
or any other resources used in this
session please let us know in the
comment section below and our team of
experts we'll be happy to help you as
soon as possible until next time thank
you and keep learning stay tuned for
more from Simply learn staying ahead in
your career requires continuous learning
and upskilling whether you're a student
aiming to learn today's top skills or a
working professional looking to advance
your career we've got you covered
explore our impressive catalog of
certification programs in cutting Edge
domains including data science cloud
computing cyber security AI machine
learning or digital marketing designed
in collaboration with leading
universities and top corporations and
delivered by industry experts choose any
of our programs and set yourself on the
path to Career Success click the link in
the description to know
more hi there if you like this video
subscribe to the simply learn YouTube
channel and click here to watch similar
videos to nerd up and get certified
click
here