Understanding Data

Written and reviewed by Rohit Singh, Computer Science & Commerce faculty at UniDrill (MCA, Osmania University · 6 years teaching). Based on NCERT Informatics Practices Class XI, Chapter 5.

Can 2,000 salary figures tell you which college to join?

A college has a list of 2,000 students placed in the last three years, each with a company and a salary. Staring at that list tells you nothing. You cannot say whether placements are good. Now suppose the college summarises it: the average package, the middle package, the lowest and highest. Suddenly you can decide. That jump from a pile of numbers to a decision is what this chapter is about: collecting data, storing it, processing it, and using a few statistics to understand it.

What is data, and why does it matter?

Data is a collection of characters, numbers and other symbols that represent values of some situation or variable. The word is plural, and the singular is datum. Data are stored in electronic form because computers process them much faster and more easily than people can by hand. The growth of computers, mobiles and the Internet (the ICT revolution) now produces data in huge volume and at high speed.

Computers process data to generate information, and analysing and interpreting information gives knowledge. Data is the raw material, and information is what you get after processing it.

Why data matters. Decisions work better when they rest on data. People use data to choose a college, governments use census data to plan policies, sports coaches study opponents' performances, and banks keep customer records so they can update a balance after an ATM withdrawal. Processing large data on a computer can also show hidden patterns that humans cannot see directly.

Types of data: structured and unstructured

Data come in different formats. An image is a collection of pixels, a video is made of frames, a fee slip has a few numeric and non numeric entries, and a chat message may contain text, emoticons and pictures. On the basis of format, data falls into two broad categories.

Why a newspaper is unstructured. One day a page may have three images and five stories, and the next day one big image and three stories. There is no fixed pattern. An email is similar: nothing fixes how many lines or attachments it holds.

Structured data in action. A shop keeps this inventory table.

Because the data is in columns, a spreadsheet or a program can sum or multiply a whole column. Here each column is an attribute and each row is one item.

Worked example: totals from a table. Find the total number of items, and the total value of the stock (unit price multiplied by items in inventory).

The common wrong step is to add the Unit Price column and the Items column together. Adding prices does not give a value. You must multiply each row's price by that row's count first, and only then add.

The same data in code, using a list of pairs (price, items) and the loop ideas from the Python chapters:

Metadata. Metadata is data about data. Unstructured data is often described using metadata. An email has a subject, recipient, main body and attachment. An image file has a size (KB or MB), a type (JPEG, PNG) and a resolution.

Beyond the textbook. For the exam: metadata means data about data, with email subject and image size as the examples. In reality: a photograph taken on a phone usually carries extra metadata such as the date and time, and often the device and the location, stored inside the file. Why it matters: metadata can reveal more than the picture itself, which is why privacy settings often offer to remove it before sharing.

Pause and predict: a college stores student records with columns RollNo, Name and Marks. Is this structured or unstructured, and what is "Marks" called?

Check your answer: structured, because it is organised in rows and columns. Marks is an attribute (also called a variable or characteristic), and each student is one row.

Data collection

To process data, you first collect it. Data collection means identifying data that is already available or gathering it from the right sources, and then storing it in a file or a database. Take a grocery store with sales data. There are three situations.

Data is continuously being generated: hospitals collect patient data to improve services, and shopping malls record what people buy. If analysis shows that bedsheets and groceries are often bought together, the owner can display bedsheets near the grocery section to increase sales. A political analyst may study social media posts to see public opinion, and organisations such as the World Bank and the IMF collect economic data from countries for forecasting.

Data storage

Once data is gathered and processed, we usually keep it for future use. Data storage is storing data on storage devices so that it can be retrieved later. Large volumes arrive at a high rate, so storage is a challenge, but falling prices of digital storage have helped.

Data like images, documents and audio or video are stored as files, and school or hospital data is kept in data files. File processing has limitations, which a Database Management System (DBMS) is designed to overcome.

Data processing

Looking at a large pile of raw data does not give a conclusion. Data must be processed to get results, and after analysing those results we arrive at conclusions or decisions. Processing turns raw data (numbers, text, images) into information (in the form of tables, charts or text).

The data process cycle in one line: Input (data collection, data preparation, data entry) → Processing (store, retrieve, classify, update) → Output (reports, results).

Many everyday services follow this cycle: online bill payment, registering a complaint, booking tickets.

Why this table is useful. In any such question, ask three things: what goes in, what is checked or calculated, and what comes out. That is the entire cycle.

Pause and predict: in the ATM example, which part of the cycle is "checking for a valid PIN and balance"?

Check your answer: processing. The PIN and amount are input, the checks and the deduction are processing, and the notes and slip are output.

Statistical techniques: summarising data

To understand data quickly, we summarise it using statistics. There are two families. Measures of central tendency give one value that represents the data (mean, median, mode). Measures of variability (also called measures of dispersion) tell how spread out the values are (range, standard deviation).

We will use one data set throughout: the heights, in cm, of nine students.

$$\text{Heights} = [90, 102, 110, 115, 85, 90, 100, 110, 110]$$

Mean: the average

The mean (average) is the sum of the values divided by how many there are.

$$\bar{x} = \dfrac{\sum_{i=1}^{n} x_i}{n}$$

$$\text{Mean} = \dfrac{90 + 102 + 110 + 115 + 85 + 90 + 100 + 110 + 110}{9} = \dfrac{912}{9} \approx 101.33\ \text{cm}$$

The mean works on numerical data. It is sensitive to outliers. An outlier is an exceptionally large or small value compared with the rest, and it can pull the average away from where most of the data sits.

Median: the middle value

To find the median, sort the values and take the middle one. If there is an odd number of values, the median is the value at the middle position. If there is an even number, it is the average of the two middle values. It splits the data into two equal halves.

Sorted heights: $[85, 90, 90, 100, 102, 110, 110, 110, 115]$. There are 9 values, so the median is the 5th value, which is 102 cm, whether you count from the left or from the right.

Pause and predict: the marks 12, 15, 11, 18, 14, 20 are given. What is the median?

Check your answer: 14.5. Sorted, they are 11, 12, 14, 15, 18, 20. There are six values, so take the average of the two middle values 14 and 15, which is $(14 + 15)/2 = 14.5$.

Mode: the most frequent value

The mode is the value that appears the most times. It depends on how often each distinct value occurs. A data set has no mode if every value occurs only once, and it can have more than one mode if several values tie for the highest frequency. Mode can be found for numeric and non numeric data. In the heights, 110 appears 3 times, more than any other value, so the mode is 110 cm.

Why three measures? They answer different questions. A single "average" can mislead. If four people earn ₹20, ₹22, ₹25 and ₹24 thousand and a fifth earns ₹300 thousand, the mean is ₹78.2 thousand, a figure that nobody in the group is near. The median is ₹24 thousand, which describes the typical person much better.

Beyond the textbook. For the exam: outliers are treated as errors and are removed before calculating the mean. In reality: an outlier is not always a mistake. A genuinely very high salary is real data, and deleting it hides a true feature of the group. Analysts therefore often keep it and report the median alongside the mean. Why it matters: this is why income is usually quoted as a median, and why "which measure should I use?" depends on the data.

Range: the simplest spread

The range is the difference between the maximum and minimum values. It works only for numerical data. Because it uses only the two extreme values, an outlier can badly distort it.

$$\text{Range} = \text{Maximum} - \text{Minimum} = 115 - 85 = 30\ \text{cm}$$

Standard deviation: spread using every value

The standard deviation, written $\sigma$, uses every value, not just the extremes. It is the positive square root of the average of the squared differences of each value from the mean.

$$\sigma = \sqrt{\dfrac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n}}$$

A small standard deviation means the data is closely packed around the mean. A large one means it is widely spread.

Why squares? Deviations above the mean are positive and below the mean are negative, and they cancel out. For the heights, the deviations add up to almost exactly zero. Squaring makes every deviation positive, so they add instead of cancelling. The square root at the end brings the answer back to the original unit (cm).

Worked example: standard deviation of the heights. The mean is 101.33 cm.

The common wrong step is to stop at 104.22. That number is the average squared difference, in cm squared. The standard deviation is its square root.

Worked example: a simple standard deviation by hand. Data: 2, 4, 4, 4, 5, 5, 7, 9. Find the mean and standard deviation.

Beyond the textbook. For the exam: divide by $n$, the total number of values. In reality: when the data is only a sample taken from a larger group, statisticians divide by $n - 1$ instead, which is the usual sample standard deviation. Why it matters: the two formulas differ slightly, so use the one the question gives. In this chapter the divisor is $n$.

Putting it together in Python

The same measures can be computed with the list ideas from the earlier chapters.

The second print finds the middle element of the sorted list: for 9 values the middle index is 9 // 2, which is 4, and heights[4] is 102. That trick works only when the number of values is odd. For an even count you average the two middle values. The last line counts how many times 110 occurs, which gives its frequency.

Worked example: weekly temperatures. The temperature (in degrees Celsius) over seven days was 34, 34, 27, 28, 27, 34, 34. Find the mean, median, mode and range.

For the standard deviation of these temperatures, the squared differences from 31.14 add up to about 76.86, so $\sigma = \sqrt{76.86/7} \approx 3.31$.

Choosing the right technique

Why it matters. The question word is the clue: "average" suggests mean, "middle" suggests median, "most common" or "popular" suggests mode, and "spread", "variation" or "difference" suggests range or standard deviation.

Exam traps

60-second revision

Where this chapter connects

• Previous: Working with Lists and Dictionaries gives the list handling used for the Python versions of mean, median and mode.
• Earlier hardware link: Computer System introduced data capturing, storage and retrieval, which this chapter extends into processing and statistics.
• Class XII link: Understanding Data and Data Handling using Pandas I take these statistics into real tables of data.