Understanding Data

Written and reviewed by Rohit Singh, Computer Science & Commerce faculty at UniDrill (MCA, Osmania University · 6 years teaching). Based on NCERT Computer Science Class XII, Chapter 7.

Why does a college brochure not just print its raw placement list?

Imagine a college has kept the records of about 2,000 placed students over three years: company, package, branch. Reading that list tells you almost nothing. The brochure instead says things like "average package" and "most students were placed in this range". The data had to be gathered, processed and summarised before anyone could decide anything. That is the whole chapter in one idea: data is only useful after it is collected, stored, processed and summarised.

What is data?

Data is a collection of characters, numbers and other symbols that represent values of some situations or variables. The word "data" is plural, and its singular is datum. Raw data does not tell you much by itself, so we process it into information: a table, a chart or a sentence that supports a decision. When you withdraw cash, the bank must update your account; a weather office watches satellite data for a cyclone; an airline sets ticket prices by looking at demand and supply. In each case the decision is only as good as the data behind it.

Data comes from many places: personal details such as name, age and phone number, bank and ticket transactions, images, audio and video, documents and web pages, online posts and messages, sensor signals and satellite readings. Storing it electronically makes processing much faster than doing it by hand.

Structured and unstructured data

Data is classified into two broad types on the basis of its format.

Worked example: a table is structured data. A shop keeps the following kitchen items in a table with the attributes ModelNo, ProductName, Unit Price, Discount(%) and Items_in_Inventory.

The table has 5 attributes (columns) and 4 observations (rows). Because it is structured, the owner can sum the Items_in_Inventory column to count all items, or multiply Unit Price by Items_in_Inventory for each row and add the results to get the total value of the stock. Such calculations are impossible on a pile of loose emails.

Metadata is data about data. It is used to describe unstructured data. For an email, the subject, recipient, main body and attachment details are metadata. For an image file, the size (in KB or MB), the type (JPEG or PNG) and the resolution are metadata. The pixels themselves are the data; the facts about the file are the metadata.

Beyond the textbook. For the exam: a photograph is unstructured, and its size, type and resolution are metadata. In reality: the same photo file usually carries more metadata than that, such as the date it was taken and the camera model, which is why sharing original photos can reveal more than you intended. Metadata helps computers organise unstructured data, such as sorting a gallery by date.

Data collection and storage

Data collection means identifying data that is already available or gathering it from appropriate sources. Take a grocery shop's sales data. It can exist in three states, and each needs a different first step.

Data storage is storing data on a device so it can be retrieved later. Common devices are the hard disk drive (HDD), solid state drive (SSD), CD/DVD, tape drive, pen drive and memory card. Falling prices of storage have made it practical to keep the huge amount of data being produced.

Data is usually kept as files, and programs are written to add, change or delete data in them. This file processing has limits that a database management system (DBMS) overcomes, which is the subject of the next chapter.

Data processing

Data processing converts raw data (numbers, text, images) into information (tables, charts, text). The data process cycle has three stages.

The easy way to remember where an activity belongs is to ask what it does to the data. If it brings data in, it is input. If it works on data already inside, it is processing. If it hands something back to the user, it is output.

Worked example: the same cycle in three real tasks. Each task starts from a problem, takes some inputs, processes them and gives an output.

Pause and predict: In the ATM example, "deducting the withdrawn amount from the account" belongs to which stage of the data process cycle?

Check your answer: Processing. It acts on data already stored inside the bank's system (the balance) and updates it. Typing the PIN is input, and the printed slip is output.

Statistical techniques: summarising data

A table with 2,000 rows cannot be read at a glance, so we summarise it with a few numbers. Two families are used.

Two different data sets can have the same mean but a very different spread, so you often need one from each family.

We will use one data set for all five techniques. The heights (in cm) of nine students are 90, 102, 110, 115, 85, 90, 100, 110, 110.

Mean: the balance point

The mean (average) is the sum of the values divided by how many there are.

$$\bar{x} = \dfrac{x_1 + x_2 + \dots + x_n}{n} = \dfrac{\sum_{i=1}^{n} x_i}{n}$$

For the heights the sum is 912 and $n = 9$, so the mean is $912 / 9 = 101.33$ cm. The mean uses every value, which is its strength and also its weakness: one extreme value drags it along. An exceptionally large or small value compared with the rest is called an outlier. Outliers are usually treated as errors, and they should be removed before the mean is calculated.

Median: the middle value

Sort the data in ascending or descending order. The median is the middle value, so half the data lies on each side. With an odd number of values, it is the value at the middle position. With an even number, it is the average of the two middle values.

Sorted heights: 85, 90, 90, 100, 102, 110, 110, 110, 115. There are 9 values, so the median is the value at position 5, which is 102 cm.

Mode: the most frequent value

The mode is the value that occurs most often. A data set has no mode if every value occurs only once, and it can have more than one mode if several values tie for the highest frequency. Mode is the only one of the three that also works on non-numeric data, such as the most popular colour of cars.

In the heights, 110 occurs three times, more than any other value, so the mode is 110 cm.

Range: the widest gap

The range is the maximum minus the minimum, so it works only for numerical data. For the heights it is $115 - 85 = 30$ cm. Because it uses only the two extreme values, an outlier distorts it badly.

Standard deviation: the typical distance from the mean

The standard deviation $\sigma$ (sigma) looks at every value. It is the positive square root of the average of the squared differences between each value and the mean.

$$\sigma = \sqrt{\dfrac{\sum_{i=1}^{n}(x_i - \bar{x})^2}{n}}$$

A small $\sigma$ means the data is packed close to the mean; a large $\sigma$ means it is spread out. The squaring is there because deviations above and below the mean would otherwise cancel to zero. Taking the square root at the end brings the answer back to the original unit (cm, not cm squared).

Worked example: standard deviation of the heights. The mean is $\bar{x} = 101.33$.

Average of the squares: $938 / 9 = 104.22$. Square root: $\sigma = \sqrt{104.22} \approx 10.2$ cm. A common wrong step is to divide by 8 instead of 9, or to forget the square root and quote 104.22. In this chapter you divide by $n$, the number of values.

Worked example: a simple one you can do in your head. Take 2, 4, 4, 4, 5, 5, 7, 9.

Why the mean fails and the median doesn't

Worked example: one outlier. Monthly pay (in thousands of rupees) of five employees is 18, 20, 22, 24, 26. A sixth employee joins at 200.

The mean more than doubled and the range exploded, yet the median barely moved. That is why the median is the safer summary for incomes and house prices, and why the mean should be used only after removing outliers.

Pause and predict: Two classes each have three students. Class A scored 48, 50, 52 and Class B scored 30, 50, 70. Which statistic is identical for both classes, and which statistic tells them apart?

Check your answer: The mean and median are both 50 for each class. The range (4 versus 40) and the standard deviation (about 1.6 versus about 16.3) separate them. The averages hide the fact that Class B is far more varied.

Beyond the textbook. For the exam: standard deviation divides the squared deviations by $n$. In reality: statisticians distinguish a whole population from a sample taken from it, and for a sample they divide by $n - 1$. When you reach the Pandas chapters you will see that a Pandas std() uses $n - 1$ by default, while NumPy's std() divides by $n$, so the two can give slightly different numbers for the same data.

Which technique should I use?

Formula sheet

Exam traps

60-second revision

Where this chapter connects

• Previous: Searching shows how to locate one value in a list; this chapter summarises all the values.
• Next: Database Concepts explains the DBMS that overcomes the limits of file processing mentioned here.
• Later use: Data Handling using Pandas I and Plotting Data using Matplotlib apply these statistics to tables of data in Python.