5 Best Ways to Add a New Column to an Existing DataFrame in Pandas in Python

by

💡 Problem Formulation: When working with datasets in Python, it’s often necessary to alter the structure of your DataFrame to include additional information. Suppose you have a DataFrame containing product information and you want to add a new column representing the tax for each product. This article illustrates different ways to add this new ‘tax’ column to your existing DataFrame using the pandas library.

Method 1: Using Direct Assignment

Direct assignment is the simplest method to add a new column. You specify the new column name and assign a value or list of values to it. The length of the list must match the DataFrame’s number of rows or you can assign a single value to be repeated for all rows.

Here’s an example:

import pandas as pd

# Sample DataFrame
df = pd.DataFrame({'Product': ['Apple', 'Banana', 'Cherry'], 'Price': [0.95, 0.65, 1.20]})
# Adding tax column through direct assignment
df['Tax'] = [0.07, 0.07, 0.07]

print(df)

Output:

  Product  Price   Tax
0   Apple   0.95  0.07
1  Banana   0.65  0.07
2  Cherry   1.20  0.07

This straightforward approach of direct assignment is the most intuitive, and it works well when you need to add a statically calculated column or a fixed value to each row.

Method 2: Using the assign() Method

The assign() method allows you to create a new DataFrame with added columns. It’s useful for chaining commands and creating a functional programming style. The added column can be an existing Series, a list of values, or a function applied to the DataFrame.

Here’s an example:

# Using the previous df DataFrame
# Adding a 'Tax' column using the assign method
df_with_tax = df.assign(Tax=lambda x: x['Price'] * 0.07)

print(df_with_tax)

Output:

  Product  Price   Tax
0   Apple   0.95  0.0665
1  Banana   0.65  0.0455
2  Cherry   1.20  0.0840

The assign() method is non-destructive and returns a new DataFrame. This is beneficial when you want to keep the original DataFrame unchanged. It enables inline operations and is more compatible with a functional programming style.

Method 3: Using the insert() Method

This method inserts a new column into the DataFrame at a specified index, allowing more control over the column order. It’s favorable when the position of the new column is essential. The new column can be a list, a Series, or a scalar value to be repeated in each row.

Here’s an example:

# Using the previous df DataFrame
# Inserting a 'Tax' column before the 'Price' column
df.insert(loc=1, column='Tax', value=[0.07, 0.07, 0.07])

print(df)

Output:

  Product   Tax  Price
0   Apple  0.07   0.95
1  Banana  0.07   0.65
2  Cherry  0.07   1.20

With the insert() method, the position of the new ‘Tax’ column is explicitly set to the second column (index 1). It is a straightforward way to organize your DataFrame columns as needed, although it does modify the original DataFrame in place.

Method 4: Using loc[]

With pandas loc[] functionality, you can not only access data but also modify your DataFrame. It allows you to add a new column by specifying the name of the new column and optionally applying functions to each row if needed.

Here’s an example:

# Using the previous df DataFrame
# Adding the 'Tax' column using loc[]
df.loc[:, 'Tax'] = df['Price'] * 0.07

print(df)

Output:

  Product  Price   Tax
0   Apple   0.95  0.0665
1  Banana   0.65  0.0455
2  Cherry   1.20  0.0840

The loc[] method resembles direct assignment but is more versatile, allowing for complex row-wise operations and condition-based assignments. This operation modifies the original DataFrame.

Bonus One-Liner Method 5: Using a Dictionary with **

For Python enthusiasts, you can use dictionary unpacking with ** to quickly add multiple new columns. This is a compact and Pythonic way to add columns, although it might be less clear to those unfamiliar with dictionary unpacking.

Here’s an example:

# Using the previous df DataFrame without the 'Tax' column
# Adding the 'Tax' column using dictionary unpacking
df = pd.DataFrame({**df, **{'Tax': pd.Series([0.07, 0.07, 0.07], index=df.index)}})

print(df)

Output:

  Product  Price   Tax
0   Apple   0.95  0.07
1  Banana   0.65  0.07
2  Cherry   1.20  0.07

This one-liner method merges the original DataFrame with a new ‘Tax’ column and simultaneously introduces the dictionary unpacking concept. It’s concise and efficient but may be less readable for some.

Summary/Discussion

  • Method 1: Direct Assignment. Simplest and most intuitive. Does not require creating a new DataFrame. However, it operates in place and can overwrite existing data if not used carefully.
  • Method 2: Using assign(). Enables functional programming and operation chaining. It creates a new DataFrame, leaving the original unchanged, but may be less efficient with memory for large DataFrames.
  • Method 3: Using insert(). Provides control over the column position. Alters the original DataFrame directly, which might be undesirable in some workflows.
  • Method 4: Using loc[]. Offers flexibility for complex row-wise operations. Modifies the original DataFrame and may be less intuitive for beginners.
  • Bonus Method 5: Dictionary Unpacking with **. Quick and Pythonic. Best for one-liners and when adding multiple columns at once. Could be confusing due to the use of advanced Python features and less explicit behavior.