Pandas Time-Series DataFrame Methods – Part 17

Rate this post

The Pandas DataFrame/Series has several methods related to time series.

This is Part 17 of the DataFrame methods series:

  • Part 1 focuses on the DataFrame methods abs(), all(), any(), clip(), corr(), and corrwith().
  • Part 2 focuses on the DataFrame methods count(), cov(), cummax(), cummin(), cumprod(), cumsum().
  • Part 3 focuses on the DataFrame methods describe(), diff(), eval(), kurtosis().
  • Part 4 focuses on the DataFrame methods mad(), min(), max(), mean(), median(), and mode().
  • Part 5 focuses on the DataFrame methods pct_change(), quantile(), rank(), round(), prod(), and product().
  • Part 6 focuses on the DataFrame methods add_prefix(), add_suffix(), and align().
  • Part 7 focuses on the DataFrame methods at_time(), between_time(), drop(), drop_duplicates() and duplicated().
  • Part 8 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 9 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 10 focuses on the DataFrame methods reset_index(), sample(), set_axis(), set_index(), take(), and truncate()
  • Part 11 focuses on the DataFrame methods backfill(), bfill(), fillna(), dropna(), and interpolate()
  • Part 12 focuses on the DataFrame methods isna(), isnull(), notna(), notnull(), pad() and replace()
  • Part 13 focuses on the DataFrame methods drop_level(), pivot(), pivot_table(), reorder_levels(), sort_values() and sort_index()
  • Part 14 focuses on the DataFrame methods nlargest(), nsmallest(), swap_level(), stack(), unstack() and swap_axes()
  • Part 15 focuses on the DataFrame methods melt(), explode(), squeeze(), to_xarray(), t() and transpose()
  • Part 16 focuses on the DataFrame methods append(), assign(), compare(), join(), merge() and update()
  • Part 17 focuses on the DataFrame methods asfreq(), asof(), shift(), slice_shift(), tshift(), first_valid_index(), and last_valid_index()
  • Part 18 focuses on the DataFrame methods resample(), to_period(), to_timestamp(), tz_localize(), and tz_convert()
  • Part 19 focuses on the visualization aspect of DataFrames and Series via plotting, such as plot(), and plot.area().
  • Part 20 focuses on continuing the visualization aspect of DataFrames and Series via plotting such as hexbin, hist, pie, and scatter plots.
  • Part 21 focuses on converting one data type to/from another data type.
  • Part 22 focuses on converting one data type to/from another data type.
  • Part 23 focuses on converting one data type to/from another data type.
  • Part 24 focuses on converting one data type to/from another data type.
  • Part 25 focuses on converting one data type to/from another data type.

Preparation

Before any data manipulation can occur, two (2) new libraries will require installation.

  • The Pandas library enables access to/from a DataFrame.
  • The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.

To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.

$ pip install pandas

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install numpy

Hit the <Enter> key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.


FeFeel free to view the PyCharm installation guide for the required libraries.


Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import pandas as pd
import numpy

DataFrame asfreq()

The asfreq() method converts a time series to a specified frequency. To view a list of available frequencies, click here.

The syntax for this method is as follows:

DataFrame.asfreq(freq, method=None, how=None, normalize=False, fill_value=None)
ParameterDescription
freqClick here to view the frequencies, or navigate to an IDE and run: print(pd.tseries.offsets.__all__)
methodThis parameter completes missing values in an indexed Series (non-NaN). The available options are:
backfill/bfill: last valid observation to the following valid observation.
pad/ffill: use the following valid observation to fill.
howThe available options are start and end. The default is end.
normalizeDetermines whether to reset the output index to midnight.
fill_valueThis parameter is the fill value(s) to apply to missing values (not NaN values).

For this example, five (5) random integers generate and display on sequential (Daily Frequency) days and business (Business Day Frequency) days.

Code – Example 1

lst = np.random.randint(10,60, size=5)
idx = pd.date_range('1/16/2022', periods=5, freq='D')
series = pd.Series(lst, index= idx)
df = pd.DataFrame({'Series': series})
print(df)

result = df.asfreq(freq='B')
print(result)
  • Line [1] generates five (5) random integers between the specified range and saves them to lst.
  • Line [2] does the following:
    • An index creates based on the start date for five (5) days.
    • The frequency changes to 'D' (Daily Frequency).
    • The output saves to idx.
  • Line [3] creates a Series based on the lst and idx variables. This output saves to series.
  • Line [4] creates a DataFrame from the series variable and saves it to df.
  • Line [5] outputs the DataFrame to the terminal.
  • Line [6] uses the asfreq() method to set the frequency to 'B' (Business Day Frequency). This output saves to result.
  • Line [7] outputs the result to the terminal.

Output

df (5 consecutive days)

 Series
2022-01-16     13
2022-01-17     15
2022-01-18     19
2022-01-1942
2022-01-20    26

result (5 business days – M-F)

 Series
2022-01-17     15
2022-01-18     19
2022-01-1942
2022-01-20    26

January 16, 2022, does not display in the result table as it falls on Sunday.

Selecting 'B' as a frequency will ignore any date that does not fall between Monday-Friday.


DataFrame asof()

The asof() method retrieves and returns the last row(s) of a DataFrame/Series (non-NaN values) based on the date(s) entered in the where parameter.

The syntax for this method is as follows:

DataFrame.asof(where, subset=None)
ParameterDescription
whereThis parameter is a single date or array of dates before the last row(s) return.
subsetDataFrame columns to check for NaN values.

For this example, we pass a single date. One date matches, and the appropriate value returns.

Code – Example 1

nums = np.random.randint(1,50, size=7)
idx = pd.date_range('1/24/2022', periods=7, freq='D')
series = pd.Series(nums, index=idx)
print(series)

result = series.asof('1/27/2022')
print(result)
  • Line [1] generates seven (7) random integers between the specified range and saves them to nums.
  • Line [2] does the following:
    • An index creates based on the start date for five (5) days.
    • The frequency changes to ‘D’ (Daily Frequency).
    • The output saves to idx.
  • Line [3] creates a series based on the nums and idx variables. The output saves to series.
  • Line [4] outputs the series to the terminal.
  • Line [5] retrieves a single value associated with the specified date and saves it to result.
  • Line [6] outputs the result to the terminal.

Output

df (7 consecutive days)

 Series
2022-01-24     10
2022-01-25     34
2022-01-26    31
2022-01-2725
2022-01-28    35
2022-01-2941
2022-01-30    49

result (2022-01-27)

Freq: D, dtype: int32
25

A CSV file containing five (5) rows reads in and saves to a DataFrame.

Code – Example 2

df = pd.read_csv('data.csv', parse_dates=['date'])
df.set_index('date', inplace=True)
print(df)

result = df['price'].asof(pd.DatetimeIndex(['2022-02-27 09:03:30', '2022-02-27 09:04:30']))
print(result)
  • Line [1] creates a DataFrame from the CSV file and parses the date field using parse_dates(). This output saves to df.
  • Line [2] sets the index for the DataFrame on the date field and inplace=True.
  • Line [3] outputs the DataFrame to the terminal.
  • Line [4] retrieves the price(s) based on the specified date range. The output saves to result.
  • Line [5] outputs the result to the terminal.

Output

df

 price
date     
2022-02-27 09:01:00  8.12
2022-02-27 09:02:00  8.33
2022-02-27 09:03:008.36
2022-02-27 09:04:008.29
2022-02-27 09:05:008.13

result

2022-02-27 09:03:30   8.36
2022-02-27 09:04:30   8.29
Name: price, dtype: float64

DataFrame shift()

The shift() moves the index by a select number of period(s) with an option of setting the time-frequency.

The syntax for this method is as follows:

DataFrame.shift(periods=1, freq=None, axis=0, fill_value=NoDefault.no_default)
periodsThis parameter is the number of periods to shift (positive/negative).
freqClick here to view the frequencies, or navigate to an IDE and run: print(pd.tseries.offsets.__all__)
axisIf zero (0) or index is selected, apply to each column. Default is 0 (column). If zero (1) or columns, apply to each row.
fill_valueThis parameter is the fill value of new missing values. The default value depends on dtype.
– Numeric: np.nan.
Datetime/timedelta/period: NaT.
– Extension dtypes: self.dtype.na_value.

This example generates seven (5) random numbers for three (3) daily samples. Running this code shifts the data by one (1) index. The shifted data replaces with the NaN value.

df = pd.DataFrame({'Sample-1':  list(np.random.randint(0,100,size=5)),
                   'Sample-2':  list(np.random.randint(0,100,size=5)),
                   'Sample-3':  list(np.random.randint(0,100,size=5))},
                   index=pd.date_range('2020-01-01', '2020-01-05'))
print(df)

result1 = df.shift(periods=1)
print(result1)

result2 = df.shift(periods=1, fill_value=0)
print(result2)
  • Line [1] does the following:
    • An index creates based on the start date for five (5) days.
    • The frequency changes to 'D' (Daily Frequency).
    • The output saves to idx.
    • Create a DataFrame with five (5) random integers for three (3) samples.
    • The index creates based on a specified date range.
    • The output saves to df.
  • Line [2] outputs the DataFrame to the terminal.
  • Line [3] shifts the data by one (1) period. The first-row data replaces with NaN values. The output saves to result1.
  • Line [4] outputs result1 to the terminal.
  • Line [5] shifts the data by one (1) period and sets the fill value to zero (0). The output saves to result2.
  • Line [6] outputs result2 to the terminal.

Output

df

 Sample-1Sample-2Sample-3
2020-01-01    18       85       15
2020-01-02    27       66        4
2020-01-03    78       68        5
2020-01-04     6       77       18
2020-01-05     94       20       82

result1

 Sample-1Sample-2Sample-3
2020-01-01    NaNNaNNaN
2020-01-02     18 .0      85.0       15.0
2020-01-03     27 .0   66.0        4.0
2020-01-04     78.0       68 .0       5.0
2020-01-05      6 .0      77.0       18.0

The values in the first row now display NaN values.

The last row from the original DataFrame (df) does not display.

result2

 Sample-1Sample-2Sample-3
2020-01-01    000
2020-01-02     18 .0      85.0       15.0
2020-01-03     27 .0   66.0        4.0
2020-01-04     78.0       68 .0       5.0
2020-01-05      6 .0      77.0       18.0

The NaN values from result1 change to zero (0).

The last row from the original DataFrame (df) does not display.


DataFrame slice_shift() & tshift()

These methods are no longer in use (deprecated since v1.2.0). Use the shift() method shown above instead.


DataFrame first_valid_index()

The first_valid_index() method returns the index for the first non-NA value or None if no NA value exists.

The syntax for this method is as follows:

DataFrame.first_valid_index()

This method contains no parameters.

Rivers Clothing has an issue with its pricing table. Therefore, they want to locate the first index (Small, Medium, or Large) that contains a valid price. To do this, run the following code.

idx = ['Small', 'Mediun', 'Large']

df = pd.DataFrame({'Tops':     [np.nan, np.nan, np.nan],
                   'Tanks':    [np.nan, 13.45, 14.98],
                   'Pants':    [np.nan, 56.99, 94.87]}, index=idx)
print(df)

result = df.first_valid_index()
print(result)
  • Line [1] creates an index for the DataFrame and saves it to idx.
  • Line [2] creates a DataFrame of incomplete inventory pricing, sets the index, and saves it to df.
  • Line [3] outputs the DataFrame to the terminal.
  • Line [4] retrieves the first valid (non-NA) value from the DataFrame and saves the index to result.
  • Line [5] outputs the result to the terminal.

Output

df

 TopsTanksPants
SmallNaN   NaN   NaN   
MediumNaN   13.45 56.99
LargeNaN   14.98 94.87

result: Medium

The first non-NA value occurs in the Medium index under the Tanks category.


DataFrame last_valid_index()

The last_valid_index() method returns the index for the last non-NA value or None if no NA value exists.

The syntax for this method is as follows:

DataFrame.last_valid_index()

This method contains no parameters.

For this example, Rivers Clothing has an issue with its pricing table. Therefore, they want to locate the last index (Small, Medium, or Large) that contains a valid price.

To do this, run the following code.

idx = ['Small', 'Mediun', 'Large']

df = pd.DataFrame({'Tops':     [np.nan, np.nan, np.nan],
                   'Tanks':    [np.nan, 13.45, 14.98],
                   'Pants':    [np.nan, 56.99, 94.87]}, index=idx)
print(df)

result = df.last_valid_index()
print(result)
  • Line [1] creates an index for the DataFrame and saves it to idx.
  • Line [2] creates a DataFrame of incomplete inventory pricing, sets the index, and saves it to df.
  • Line [3] outputs the DataFrame to the terminal.
  • Line [4] retrieves the last valid (non-NA) value from the DataFrame and saves the index to result.
  • Line [5] outputs the result to the terminal.

Output

df

 TopsTanksPants
SmallNaN   NaN   NaN   
MediumNaN   13.45 56.99
LargeNaN   14.98 94.87

result Large

The last non-NA value occurs in the Large index under the Pants category.