< Previous Page | Home Page | Next Page >

Introduction of Series and DataFrame Objects¶

At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy structured arrays in which the rows and columns are identified with labels rather than simple integer indices. As we will see during the course of this chapter, Pandas provides a host of useful tools, methods, and functionality on top of the basic data structures, but nearly everything that follows will require an understanding of what these structures are. Thus, before we go any further, let's introduce these two fundamental Pandas data structures: the Series and DataFrame.

We will start our code sessions with the standard NumPy and Pandas imports:

import numpy as np
import pandas as pd

The Pandas Series Object¶

In pandas, 1-D arrays are referred to a series. You can think of it a cross between a list and a dictionary. The items are all stored in an order and there's labels (so-called indices) with which you can retrieve them.

Combining many individual Series objects creates a DataFrame. That's why it's a good idea to learn about the Series before moving on to the DataFrame.

A Series object can be created from an array of data.

data = pd.Series([0.25, 0.5, 0.75, 1.0])
data

0    0.25
1    0.50
2    0.75
3    1.00
dtype: float64

As we see in the output, the Series wraps both a sequence of values and a sequence of indices (labels of the individual "rows"), which we can access with the values and index attributes. The values are simply a familiar NumPy array:

data.values

array([0.25, 0.5 , 0.75, 1.  ])

Like with a NumPy array, data can be accessed by the associated index via the familiar Python square-bracket notation:

data[1]

0.5

data[1:3]

1    0.50
2    0.75
dtype: float64

As we will see, though, the Pandas Series is much more general and flexible than the one-dimensional NumPy array that it emulates.

`Series` as generalized NumPy array¶

From what we've seen so far, it may look like the Series object is basically interchangeable with a one-dimensional NumPy array. The essential difference is the presence of the index: while the Numpy Array has an implicitly defined integer index used to access the values, the Pandas Series has an explicitly defined index associated with the values.

This explicit index definition gives the Series object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type. For example, if we wish, we can use strings as an index:

data = pd.Series([0.25, 0.5, 0.75, 1.0],
                 index=['a', 'b', 'c', 'd'])
data

a    0.25
b    0.50
c    0.75
d    1.00
dtype: float64

And the item access works as expected:

data['b']

0.5

We can even use non-sequential indices:

data = pd.Series([0.25, 0.5, 0.75, 1.0],
                 index=[2, 5, 3, 7])
data

2    0.25
5    0.50
3    0.75
7    1.00
dtype: float64

data[5]

0.5

Series as specialized dictionary¶

In this way, you can think of a Pandas Series a bit like a specialization of a Python dictionary. A dictionary is a structure that maps arbitrary keys to a set of arbitrary values, and a Series is a structure which maps typed keys to a set of typed values.

The Series-as-dictionary analogy can be made even more clear by constructing a Series object directly from a Python dictionary:

population_dict = {'California': 38332521,
                   'Texas': 26448193,
                   'New York': 19651127,
                   'Florida': 19552860,
                   'Illinois': 12882135}
population = pd.Series(population_dict)
population

California    38332521
Texas         26448193
New York      19651127
Florida       19552860
Illinois      12882135
dtype: int64

By default, a Series will be created where the index is drawn from the sorted keys. From here, typical dictionary-style item access can be performed:

population['California']

38332521

Unlike a dictionary, though, the Series also supports array-style operations such as slicing:

population['California':'Illinois']

California    38332521
Florida       19552860
Illinois      12882135
dtype: int64

Constructing Series objects¶

We've already seen a few ways of constructing a Pandas Series from scratch; all of them are some version of the following:

>>> pd.Series(data, index=index)

where index is an optional argument, and data can be one of many entities.

A series is created through the pd.Series constructor, which has a lot of optional arguments. Let's explore it a bit more by typing in pd.Series?

pd.Series?

For example, data can be a list or NumPy array, in which case index defaults to an integer sequence:

pd.Series([2, 4, 6])

0    2
1    4
2    6
dtype: int64

data can be a scalar, which is repeated to fill the specified index:

pd.Series(5, index=[100, 200, 300])

100    5
200    5
300    5
dtype: int64

If we want our index to be more descriptive, we can also create a Series using a dictionary.

sports = {'Football': 'Germany',
          'Golf': 'Scotland',
          'Sumo': 'Japan',
          'Taekwondo': 'South Korea'}
s = pd.Series(data=sports) 
s

Football         Germany
Golf            Scotland
Sumo               Japan
Taekwondo    South Korea
dtype: object

We can also separate the index creation from the data by passing in the index as a list explicitly to the series.

s = pd.Series(['Tiger', 'Bear', 'Moose'], index=['India', 'America', 'Canada'])
s

India      Tiger
America     Bear
Canada     Moose
dtype: object

The Pandas DataFrame Object¶

The next fundamental structure in Pandas is the DataFrame. A dataFframe is a table or a two-dimensional array-like structure in which each column contains values of one variable and each row contains one set of values from each column. As pandas allows us to do a lot of stuff with this table, you can imagine a dataframe as a sort of excel spreadsheet on steroids.

DataFrame as a generalized NumPy array¶

If a Series is an analog of a one-dimensional array with flexible indices, a DataFrame is an analog of a two-dimensional array with both flexible row indices and flexible column names.

You can think of a DataFrame as a sequence of aligned Series objects. Here, by "aligned" we mean that they share the same index.

To demonstrate this, let's first construct a new Series listing the area of each of five American states:

area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,
             'Florida': 170312, 'Illinois': 149995}
area = pd.Series(area_dict)
area

California    423967
Texas         695662
New York      141297
Florida       170312
Illinois      149995
dtype: int64

Now that we have this along with the population Series from before, we can use a dictionary to construct a single two-dimensional object containing this information:

states = pd.DataFrame({'population': population,
                       'area': area}) # note that each value in the dictionary is a Series object
states

Like the Series object, the DataFrame has an index attribute that gives access to the index labels:

states.index

Index(['California', 'Florida', 'Illinois', 'New York', 'Texas'], dtype='object')

Additionally, the DataFrame has a columns attribute, which is an Index object holding the column labels:

states.columns

Index(['area', 'population'], dtype='object')

Thus the DataFrame can be thought of as a generalization of a two-dimensional NumPy array, where both the rows and columns have a generalized index for accessing the data.

DataFrame as specialized dictionary¶

Similarly, we can also think of a DataFrame as a specialization of a dictionary. Where a dictionary maps a key to a value, a DataFrame maps a column name to a Series of column data. For example, asking for the 'area' attribute returns the Series object containing the areas we saw earlier:

states['area']

California    423967
Florida       170312
Illinois      149995
New York      141297
Texas         695662
Name: area, dtype: int64

Constructing DataFrame objects¶

You can create a DataFrame in many different ways, some of which you might expect. For instance, you can use a group of series, where each series represents a row of data. Or you could use a group of dictionaries, where each dictionary represents a row of data.

From a single Series object¶

A DataFrame is a collection of Series objects, and a single-column DataFrame can be constructed from a single Series:

pd.DataFrame(population, columns=['population'])

From a list of dicts¶

Any list of dictionaries can be made into a DataFrame. We'll use a simple list comprehension to create some data:

data = [{'a': i, 'b': 2 * i}
        for i in range(3)]
pd.DataFrame(data)

Even if some keys in the dictionary are missing, Pandas will fill them in with NaN (i.e., "not a number") values:

pd.DataFrame([{'First Column': 1, 'Second Column': 2}, {'Second Column': 3, 'Third Column': 4}])

From a dictionary of Series objects¶

As we saw before, a DataFrame can be constructed from a dictionary of Series objects as well:

pd.DataFrame({'population': population,
              'area': area})

From a two-dimensional NumPy array¶

Given a two-dimensional array of data, we can create a DataFrame with any specified column and index names. If omitted, an integer index will be used for each:

pd.DataFrame(np.random.rand(3, 2), # this creates a two dimensional array with random values 
             columns=['foo', 'bar'],
             index=['a', 'b', 'c'])

From an imported file¶

In case we have several hundreds or thousands of data entries, it would be infeasible to create a DataFrame manually. Therefore, pandas allows us to import several types of existing data sets.

You can load a csv file via pd.read_csv("filename"), an excel file via pd.read_excel("filename") or read a sql database via pd.read_sql(). While this is not further elaborated here, a great tutorial can be found here.