1: Introduction to Data Science

Gather Data

The first step is to locate the data and load it into the computer memory so we can use the Python programming language to work with it.

import requests

# download the entire book from the Project Gutenberg website
url = 'https://www.gutenberg.org/cache/epub/113/pg113.txt'
garden = requests.get(url).text

# separate into chapters
garden_chapters = garden.split('CHAPTER ')[1:]
print(garden_chapters[0][:200])  # print the first 200 characters of the first chapter to check

Output?

I.
THERE IS NO ONE LEFT

When Mary Lennox was sent to Misselthwaite Manor to live with her uncle
everybody said she was the most disagreeable-looking child ever seen.
It was true, too. She had 

Now that we have the book chapters stored in memory at a location or variable named garden_chapters, we can start the data analysis. One of the most common starting points is to look at the size of the data. Below we find how many characters and how many words the book has.

# Find the number of characters and number of words in the book

# initialize counters at 0, since there are 0 characters and 0 words at the start
total_characters = 0
total_words = 0

# go through the chapters and count all printable characters and words
for chapter in garden_chapters:
    total_characters += len(chapter) - chapter.count(' ')
    words = chapter.split()
    total_words += len(words)

print(f'Number of characters: {total_characters:,d}')
print(f'Number of words: {total_words:,d}')

Output?

Number of characters: 379,495
Number of words: 83,346

In the output above we see that the character dataset and the word dataset are both large, and each dataset is likely diverse because it has a variety of characters or words, as most books do.

Analyze Data

Next we will analyze specific words in the text and see whether the data tells us something about the book.

First, we take the list of all chapters of the book and store them in a special container that's commonly used for data analysis: a DataFrame. A DataFrame is a table consisting of rows and columns. We'll put each chapter in a row of the DataFrame, so the DataFrame has one column which contains the chapters of the book.

# Store the chapters into a table or a DataFrame and name it chapters

import pandas as pd

# store the chapters into a table where each chapter is one row of the table
chapters = pd.DataFrame({'chapters': garden_chapters})

# print the first 5 rows of the table to check
chapters.head()

Output?

From the output above, we can see the first five rows of the DataFrame, numbered 0-4. Each row shown is the beginning part of a chapter.

The book The Secret Garden has several major characters and their names are: Mary, Colin, Dickon, Craven, Martha, Ben, and Medlock. Without reading the book, we want to use the dataset of the words in the book to predict which of these characters is the main character of the story.

To do this, we'll count the number of times each name appears in the book. Counting the number of occurrences of a data value in a dataset is a basic statistics concept. From the count of each name we can infer who the main character is by choosing the name with the highest count.

# Find the number of times the names of the characters appear

# identify the names of the characters to count
names = ['Mary', 'Colin', 'Dickon', 'Craven', 'Martha', 'Ben', 'Medlock']

# create room in memory to store the counts of each name
name_counts = pd.DataFrame(columns=['Name', 'Count'])
counts = {}

# go through each name and count how many times it appears in the chapters
for name in names:
    count = 0
    for chapter in chapters['chapters']:
        count += chapter.lower().count(name.lower())
    counts[name] = count

# store the name counts into a DataFrame for easier viewing
for name, count in counts.items():
    name_counts = pd.concat([name_counts, pd.DataFrame({'Name': [name], 'Count': [count]})], ignore_index=True)

name_counts

Output?

From the output above, we see that Mary has the highest count, so we can conclude or infer that Mary is most likely the main character. When we infer something, we reach a conclusion about it by combining facts with reasoning, rather than by directly observing it. In this case the fact (Mary has the highest name count) and the reasoning (the most important character will appear the most times in the book) are used to infer that Mary is the main character, without actually reading the book.

Visualize Data

Even though the name count for Mary is already very convincing, often in data science we display the resulting data in an appropriate plot to clearly show the pattern in the data. This is called data visualization.

Let's plot the counts of all the names and see if data visualization really shows who the main character is.

# Plot the count of each name

import matplotlib.pyplot as plt

name_counts.plot(x='Name', y='Count', kind='barh', legend=False)
plt.xlabel("Character Name")
plt.ylabel("Number of Occurrences")
plt.title("Count of Each Character's Name in 'The Secret Garden'")
plt.show()

It's pretty clear that Mary's name appears the most in the book, so we can infer that Mary is the main character of the story. If you've read the book The Secret Garden, you can confirm whether our data science exercise reaches a correct conclusion, and if you haven't read the book, a quick online search can also confirm our conclusion.

Through this quick tour, we see a few key components of a data science application:

• A large and diverse dataset. If we only looked at one chapter of the book (a small dataset), the count of each character name in the chapter may not be the same as the count in the entire book and we might arrive at the wrong main character. This is why choosing a large and diverse dataset is important.
• Using the appropriate statistical methods and visualization methods, we can see different patterns in the data, such as how large the dataset is.
• With a good dataset and appropriate statistical methods, we can come to a correct conclusion about a characteristic of the book (such as who the main character is) without personally reading the book.

It is the extraction of meaningful patterns or insights from data that makes data science an important tool for many industries, and it also makes data science the basis for machine learning and artificial intelligence.