Pandas DataFrame to_coo() Method – Be on the Right Side of Change

Preparation

Before any data manipulation can occur, four (4) new libraries will require installation.

The Pandas library enables access to/from a DataFrame.
The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.
The pandas_gbq allows access to Google Big Query (GBQ)
The google.auth authentication.

To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.

$ pip install pandas

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install pandas_gbq

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install numpy

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install google.auth

Hit the <Enter> key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.

Feel free to view the PyCharm installation guide for the required libraries.

Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import pandas as pd
import numpy as np 
from google.cloud import bigquery
import google.auth

DataFrame Sparse to_coo()

The sparse to_coo() method creates a scipy.sparse.coo_matrix from a Series containing a MultiIndex. The row_levels and column_levels determine the row/column coordinates.

The syntax for this method is as follows:

Series.sparse.to_coo(row_levels=(0,), column_levels=(1,), sort_labels=False)

Parameter	Description
`row_levels`	This parameter is a tuple or a list.
`column_levels`	This parameter is a tuple or a list.
`sort_labels`	The sort is performed before creating the sparse matrix if this parameter is True.

This example has random and missing data. This data is re-sampled and converted into a tuple format using to_coo().

stats = pd.Series([1.0080, 4.00260, 7.0, 9.012183, np.nan, np.nan])
stats.index = pd.MultiIndex.from_tuples(
    [(np.nan, 2, "a", 0),
     (1, 2, "a", 1),
     (np.nan, 1, "b", 0),
     (1, 1, "b", 1),
     (2, 1, "b", 0),
     (np.nan, 1, "b", 1)],
    names=["HYD", "HEL", "LIT", "BER"])

new_stats = stats.astype("Sparse")

A, rows, columns = new_stats.sparse.to_coo(
    row_levels=["HYD", "HEL"], 
    column_levels=["LIT", "BER"], sort_labels=True)

print(A)

Output

(0, 0) 1.008

(1, 1) 4.0026

(2, 2) 7.0

(3, 3) 9.012183

If we applied the todense() method to the above data, the output would be as follows:

stats = pd.Series([1.0080, 4.00260, 7.0, 9.012183, np.nan, np.nan])
stats.index = pd.MultiIndex.from_tuples(
    [(np.nan, 2, "a", 0),
     (1, 2, "a", 1),
     (np.nan, 1, "b", 0),
     (1, 1, "b", 1),
     (2, 1, "b", 0),
     (np.nan, 1, "b", 1)],
    names=["HYD", "HEL", "LIT", "BER"])

new_stats = stats.astype("Sparse")

A, rows, columns = new_stats.sparse.to_coo(
    row_levels=["HYD", "HEL"], 
    column_levels=["LIT", "BER"], sort_labels=True)

print(A.todense())

Output

[[1.008 0. 0. 0. ]
[0. 4.0026 0. 0. ]
[0. 0. 7. 0. ]
[0. 0. 0. 9.012183]
[0. 0. 0. 0. ]]

More Pandas DataFrame Methods

Feel free to learn more about the previous and next pandas DataFrame methods (alphabetically) here:

👈 df.to_clipboard()

df.to_csv() 👉

Also, check out the full cheat sheet overview of all Pandas DataFrame methods.