Pandas drop_level(), pivot(), pivot_table(), reorder_levels(), sort_values(), sort_index()

by
5/5 - (4 votes)

The Pandas DataFrame/Series has several methods to handle Missing Data. When applied to a DataFrame/Series, these methods evaluate and modify the missing elements.

Preparation

Before any data manipulation can occur, two (2) new libraries will require installation.

  • The Pandas library enables access to/from a DataFrame.
  • The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.

To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.

$ pip install pandas

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install numpy

Hit the <Enter> key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.

Feel free to view the PyCharm installation guide for the required libraries.

Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import pandas as pd
import numpy as np

DataFrame drop_level()

The drop_level() method removes the specified index or column from a DataFrame/Series. This method returns a DataFrame/Series with the said level/column removed.

Python Pandas - Pivot and Pivot Table

The syntax for this method is as follows:

DataFrame.droplevel(level, axis=0)
ParameterDescription
levelIf the level is a string, this level must exist. If a list, the elements must exist and be a level name/position of the index.
axisIf zero (0) or index is selected, apply to each column. Default is 0 (column). If zero (1) or columns, apply to each row.

For this example, we generate random stock prices and then drop (remove) level Stock-B from the DataFrame.

nums = np.random.uniform(low=0.5, high=13.3, size=(3,4))
df_stocks = pd.DataFrame(nums).set_index([0, 1]).rename_axis(['Stock-A', 'Stock-B'])
print(df_stocks)

result = df_stocks.droplevel('Stock-B')
print(result)
  • Line [1] generates random numbers for three (3) lists within the specified range. Each list contains four (4) elements (size=3,4). The output saves to nums.
  • Line [2] creates a DataFrame, sets the index, and renames the axis. This output saves to df_stocks.
  • Line [3] outputs the DataFrame to the terminal.
  • Line [4] drops (removes) Stock-B from the DataFrame and saves it to the result variable.
  • Line [5] outputs the result to the terminal.

Output

df_stocks

  23
Stock-AStock-B  
12.32771010.862572  7.105198 8.295885
11.4748721.563040   5.915501 6.102915

result

 23
Stock-A  
12.3277107.105198 8.295885
11.4748725.915501 6.102915

DataFrame pivot()

The pivot() method reshapes a DataFrame/Series and produces/returns a pivot table based on column values.

Python Pandas - Pivot and Pivot Table

The syntax for this method is as follows:

DataFrame.pivot(index=None, columns=None, values=None)
Parameter Description
indexThis parameter can be a string, object, or a list of strings and is optional. This option makes up the new DataFrame/Series index. If None, the existing index is selected.
columnsThis parameter can be a string, object, or a list of strings and is optional. Makes up the new DataFrame/Series column(s).
valuesThis parameter can be a string, object, or a list of the previous and is optional.

For this example, we generate 3-day sample stock prices for Rivers Clothing. The column headings display the following characters.

  • A (for Opening Price)
  • B (for Midday Price)
  • C (for Opening Price)
cdate_idx = ['01/15/2022', '01/16/2022', '01/17/2022'] * 3
group_lst = list('AAABBBCCC')
vals_lst  = np.random.uniform(low=0.5, high=13.3, size=(9))

df = pd.DataFrame({'dates':  cdate_idx,
                                    'group':  group_lst,
                                   'value':  vals_lst})
print(df)

result = df.pivot(index='dates', columns='group', values='value')
print(result)
  • Line [1] creates a list of dates and multiplies this by three (3). The output is three (3) entries for each date. This output saves to cdate_idx.
  • Line [2] creates a list of headings for the columns (see above for definitions). Three (3) of each character are required (9 characters). This output saves to group_lst.
  • Line [3] uses np.random.uniform to create a random list of nine (9) numbers between the set range. The output saves to vals_lst.
  • Line [4] creates a DataFrame using all the variables created on lines [1-3]. The output saves to df.
  • Line [5] outputs the DataFrame to the terminal.
  • Line [6] creates a pivot from the DataFrame and groups the data by dates. The output saves to result.
  • Line [7] outputs the result to the terminal.

Output

df

 datesgroupvalue
001/15/2022A9.627767
101/16/2022    A11.528057
201/17/2022    A13.296501
301/15/2022B2.933748
401/16/2022    B2.236752
501/17/2022    B7.652414
601/15/2022C11.813549
701/16/2022    C11.015920
801/17/2022    C0.527554

result

groupABC
dates   
01/15/2022  8.051752 9.571285  6.196394
01/16/2022 6.511448 8.158878 12.865944
01/17/2022 8.421245 1.746941 12.896975

DataFrame pivot_table()

The pivot_table() method streamlines a DataFrame to contain only specific data (columns). For example, say we have a list of countries with associated details. We only want to display one or two columns. This method can accomplish this task.

The syntax for this method is as follows:

DataFrame.pivot_table(values=None, index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All', observed=False, sort=True)
ParameterDescription
valuesThis parameter is the column to aggregate and is optional.
indexIf the parameter is an array, it must be the same length as the data. It may contain any other data types (but not a list).
columnsIf an array, it must be the same length as the data. It may contain any other data types (but not a list).
aggfuncThis parameter can be a list of functions. These name(s) will display at the top of the relevant column names (see Example 2).
fill_valueThis parameter is the value used to replace missing values in the table after the aggregation has occurred.
marginsIf set to True, this parameter will add the row/column data to create subtotal(s) or total(s). False, by default.
dropnaThis parameter will not include any columns where the value(s) are NaN. True by default.
margins_nameThis parameter is the name of the row/column containing the totals if margins parameter is True.
observedIf True, display observed values. If False, display all observed values.
sortBy default, sort is True. The values automatically sort. If False, no sort is applied.

For this example, a comma-delimited CSV file is read in. Then, a pivot table is created based on selected parameters.

Code – Example 1

df = pd.read_csv('countries.csv')
df = df.head(5)
print(df)

result = pd.pivot_table(df, values='Population', columns='Capital')
print(result)
  • Line [1] reads in a CSV file and saves to a DataFrame (df).
  • Line [2] saves the first five (5) rows of the CSV file to df (over-writing df).
  • Line [3] outputs the DataFrame to the terminal.
  • Line [4] creates a pivot table from the DataFrame based on the Population and Capital columns. The output saves to result.
  • Line [5] outputs the result to the terminal.

Output

df

 CountryCapitalPopulationArea
0GermanyBerlin   83783942 357021
1France  Paris   67081000 551695
2Spain Madrid   47431256 498511
3Italy   Rome   60317116 301338
4Poland Warsaw   38383000 312685

result

CapitalBerlinMadridParisRomeWarsaw
Population83783942 47431256 67081000 60317116 38383000

For this example, a comma-delimited CSV file is read in. A pivot table is created based on selected parameters. Notice the max function.

Code – Example 2

df = pd.read_csv('countries.csv')
df = df.head(5)

result = pd.pivot_table(df, values='Population', columns='Capital', aggfunc=[max])
print(result)
  • Line [1] reads in a comma-separated CSV file and saves to a DataFrame (df).
  • Line [2] saves the first five (5) rows of the CSV file to df (over-writing df).
  • Line [3] creates a pivot table from the DataFrame based on the Population and Capital columns. The max population is a parameter of aggfunc. The output saves to result.
  • Line [4] outputs the result to the terminal.

Output

result

 max    
CapitalBerlinMadridParisRomeWarsaw
Population83783942 47431256 67081000 60317116 38383000

DataFrame reorder_levels()

The reorder_levels() method re-arranges the index of a DataFrame/Series. This method can not contain any duplicate level(s) or drop level(s).

The syntax for this method is as follows:

DataFrame.reorder_levels(order, axis=0)
ParameterDescription
orderThis parameter is a list containing the new order levels. These levels can be a position or a label.
axisIf zero (0) or index is selected, apply to each column. Default is 0 (column). If zero (1) or columns, apply to each row.

For this example, there are five (5) students. Each student has some associated data with it. Grades generate by using np.random.randint().

index = [(1001, 'Micah Smith', 14), (1001, 'Philip Jones', 15), 
         	(1002, 'Ben Grimes', 16), (1002, 'Alicia Heath', 17), (1002, 'Arch Nelson', 18)]
m_index = pd.MultiIndex.from_tuples(index)
grades_lst = np.random.randint(45,100,size=5)
df = pd.DataFrame({"Grades": grades_lst}, index=m_index)
print(df)

result = df.reorder_levels([1,2,0])
print(result)
  • Line [1] creates a List of tuples. Each tuple contains three (3) values. The output saves to index.
  • Line [2] creates a MultiIndex from the List of Tuples created on line [1] and saves to m_index.
  • Line [3] generates five (5) random grades between the specified range and saves to grades_lst.
  • Line [4] creates a DataFrame from the variables on lines [1-3] and saves to df.
  • Line [5] outputs the DataFrame to the terminal.
  • Line [6] re-orders the levels as specified. The output saves to result.
  • Line [7] outputs the result to the terminal.

Output

df

   Grades
1001Micah Smith1452
 Philip Jones1565
1002Ben Grimes1683
 Alicia Heath1799
 Arch Nelson 1878

result

   Grades
Micah Smith14100152
Philip Jones15100165
Ben Grimes16100283
Alicia Heath17100299
Arch Nelson 18100278

DataFrame sort_values()

The sort_values() method sorts (re-arranges) the elements of a DataFrame.

The syntax for this method is as follows:

DataFrame.sort_values(by, axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last', ignore_index=False, key=None)
ParameterDescription
byThis parameter is a string or a list of strings. These comprise the index levels/columns to sort. Dependent on the selected axis.
axisIf zero (0) or index is selected, apply to each column. Default is 0 (column). If zero (1) or columns, apply to each row.
ascendingBy default, True. Sort is conducted in ascending order. If False, descending order.
inplaceIf False, create a copy of the object. If True, the original object updates. By default, False.
kindAvailable options are quicksort, mergesort, heapsort, or stable. By default, quicksort. See numpy.sort for additional details.
na_positionAvailable options are first and last (default). If the option is first, all NaN values move to the beginning, last to the end.
ignore_indexIf True, the axis numbering is 0, 1, 2, etc. By default, False.
keyThis parameter applies the function to the values before a sort. The data must be in a Series format and applies to each column.

For this example, a comma-delimited CSV file is read in. This DataFrame sorts on the Capital column in descending order.

df = pd.read_csv('countries.csv')
result = df.sort_values(by=['Capital'], ascending=False)
print(result)
  • Line [1] reads in a comma-delimited CSV file and saves to df.
  • Line [2] sorts the DataFrame on the Capital column in descending order. The output saves to result.
  • Line [3] outputs the result to the terminal.

Output

 CountryCapitalPopulationArea
6USA Washington  328239523  9833520
4Poland     Warsaw   38383000   312685
3Italy       Rome   60317116   301338
1France      Paris   67081000   551695
5Russia     Moscow  146748590 17098246
2Spain     Madrid   47431256   498511
8India      Dheli 1352642280  3287263
0GermanyBerlin   83783942   357021
7IndiaBeijing 1400050000  9596961

DataFrame sort_index()

The sort_index() method sorts the DataFrame.

The syntax for this method is as follows:

DataFrame.sort_index(axis=0, level=None, ascending=True, inplace=False, kind='quicksort', na_position='last', sort_remaining=True, ignore_index=False, key=None)
ParameterDescription
axisIf zero (0) or index is selected, apply to each column. Default is 0 (column). If zero (1) or columns, apply to each row.
levelThis parameter is an integer, level name, or a list of integers/level name(s). If not empty, a sort is performed on values in the selected index level(s).
ascendingBy default, True. Sort is conducted in ascending order. If False, descending order.
inplaceIf False, create a copy of the object. If True, the original object updates. By default, False.
kindAvailable options are quicksort, mergesort, heapsort, or stable. By default, quicksort. See numpy.sort for additional details.
na_positionAvailable options are first and last (default). If the option is first, all NaN values move to the beginning, last to the end.
ignore_indexIf True, the axis numbering is 0, 1, 2, etc. By default, False.
keyThis parameter applies the function to the values before a sort. The data must be in a Series format and applies to each column.

For this example, a comma-delimited CSV file is read into a DataFrame. This DataFrame sorts on the index Country column.

df = pd.read_csv('countries.csv')
df = df.set_index('Country')
result = df.sort_index()
print(result)
  • Line [1] reads in a comma-delimited CSV file and saves to df.
  • Line [2] sets the index of the DataFrame to Country. The output saves to df (over-writing original df).
  • Line [3] sorts the DataFrame (df) on the indexed column (Country) in ascending order (default). The output saves to result.
  • Line [4] outputs the result to the terminal.

Output

 CountryPopulationArea
ChinaBeijing 1400050000  9596961
FranceParis   67081000   551695
GermanyBerlin   83783942   357021
IndiaDheli 1352642280  3287263
ItalyRome   60317116   301338
PolandWarsaw   38383000   312685
RussiaMoscow  146748590 17098246
SpainMadrid   47431256   498511
USAWashington  328239523  9833520

Further Learning Resources

This is Part 13 of the DataFrame method series.

  • Part 1 focuses on the DataFrame methods abs(), all(), any(), clip(), corr(), and corrwith().
  • Part 2 focuses on the DataFrame methods count(), cov(), cummax(), cummin(), cumprod(), cumsum().
  • Part 3 focuses on the DataFrame methods describe(), diff(), eval(), kurtosis().
  • Part 4 focuses on the DataFrame methods mad(), min(), max(), mean(), median(), and mode().
  • Part 5 focuses on the DataFrame methods pct_change(), quantile(), rank(), round(), prod(), and product().
  • Part 6 focuses on the DataFrame methods add_prefix(), add_suffix(), and align().
  • Part 7 focuses on the DataFrame methods at_time(), between_time(), drop(), drop_duplicates() and duplicated().
  • Part 8 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 9 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 10 focuses on the DataFrame methods reset_index(), sample(), set_axis(), set_index(), take(), and truncate()
  • Part 11 focuses on the DataFrame methods backfill(), bfill(), fillna(), dropna(), and interpolate()
  • Part 12 focuses on the DataFrame methods isna(), isnull(), notna(), notnull(), pad() and replace()
  • Part 13 focuses on the DataFrame methods drop_level(), pivot(), pivot_table(), reorder_levels(), sort_values() and sort_index()
  • Part 14 focuses on the DataFrame methods nlargest(), nsmallest(), swap_level(), stack(), unstack() and swap_axes()
  • Part 15 focuses on the DataFrame methods melt(), explode(), squeeze(), to_xarray(), t() and transpose()
  • Part 16 focuses on the DataFrame methods append(), assign(), compare(), join(), merge() and update()
  • Part 17 focuses on the DataFrame methods asfreq(), asof(), shift(), slice_shift(), tshift(), first_valid_index(), and last_valid_index()
  • Part 18 focuses on the DataFrame methods resample(), to_period(), to_timestamp(), tz_localize(), and tz_convert()
  • Part 19 focuses on the visualization aspect of DataFrames and Series via plotting, such as plot(), and plot.area().
  • Part 20 focuses on continuing the visualization aspect of DataFrames and Series via plotting such as hexbin, hist, pie, and scatter plots.
  • Part 21 focuses on the serialization and conversion methods from_dict(), to_dict(), from_records(), to_records(), to_json(), and to_pickles().
  • Part 22 focuses on the serialization and conversion methods to_clipboard(), to_html(), to_sql(), to_csv(), and to_excel().
  • Part 23 focuses on the serialization and conversion methods to_markdown(), to_stata(), to_hdf(), to_latex(), to_xml().
  • Part 24 focuses on the serialization and conversion methods to_parquet(), to_feather(), to_string(), Styler.
  • Part 25 focuses on the serialization and conversion methods to_bgq() and to_coo().

Also, have a look at the Pandas DataFrame methods cheat sheet!

Related Tutorials