Append Pandas DataFrames Using for Loop

When dealing with large datasets, we often need to combine dataframes into single dataframe. Usually concat() is used along with the for loop to append the dataframes. Let us consider an example:

import pandas as pd import numpy as np # Create some example DataFrames dataframes = [pd.DataFrame(np.random.rand(10, 5)) for _ in range(100)] # Efficient way: collect in a list and concatenate once combined_df = pd.concat(dataframes, ignore_index=True) # Display the result print(combined_df) 

Output:

Here we are generating 100 dataframes. Each dataframe comprises of 10 rows and 5 columns. Now using a for loop, we are iterating over the list of dataframes and finally using the concat method to append the dataframes. This is much more memory efficient.

Let us consider an another example: here we have 10 dataframes which are appended to the list with the help of list comprehension. Then using concat() we are concatenating all the dataframes.

import pandas as pd # Example DataFrames (Creating 10 DataFrames with simple values) dfs = [pd.DataFrame({'A': [i, i+1], 'B': [i+2, i+3]}) for i in range(0, 10)] # Concatenate all DataFrames in the list result = pd.concat(df_list, ignore_index=False) print(result) 

Output:

From the output we can see that the dataframes have been stacked one over the other. This technique is used for large datasets as it does not create dataframes in each iteration. Hence it is much more memory efficient.

Appending dataframes but with different columns

There can be scenarios when we need to append dataframes but each of them having different column names. So we need to preprocess the columns and append the dataframes using for loop and concat method.

Let us consider a scenario. Here we have three dataframes and each of them have different column names. Now we will first collect all the column names and use reindex in the for loop to ensure each dataframes has all the columns and append them to the list. Finally use concat to concatenate all the dataframes.

import pandas as pd # Creating 10 DataFrames with different columns df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]}) df2 = pd.DataFrame({'A': [5, 6], 'C': [7, 8]}) df3 = pd.DataFrame({'A': [9, 10], 'D': [11, 12]}) # List of DataFrames dfs = [df1, df2, df3] # List to store DataFrames for concatenation df_list = [] # Get all columns across the DataFrames all_columns = list(set(df1.columns).union(set(df2.columns), set(df3.columns))) # For loop to append DataFrames, reindexing them to the same column set for df in dfs: df = df.reindex(columns=all_columns) # Reindex with all columns df_list.append(df) # Concatenate all DataFrames result = pd.concat(df_list, ignore_index=True) print(result) 

Output:

From the output we can see that for those dataframes that do not have the particular column, it generates NaN value.

Append Pandas DataFrames Using for Loop - Examples

Example 1: Let us consider that we have list of dataframes. We will iterate over the list and for each iteration we will use concat method to concatenate the dataframes one by one.

import pandas as pd # Create sample DataFrames with different columns df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]}) df2 = pd.DataFrame({'B': [5, 6], 'C': [7, 8]}) # List of DataFrames to concatenate dfs = [df1, df2] # Initialize an empty DataFrame to concatenate into result = pd.DataFrame() # For loop to concatenate DataFrames for df in dfs: result = pd.concat([result, df], ignore_index=True, sort=False) print(result) 

Output:

From the output we can see that all the columns are present in the final dataframe. The values which does not exist in a particular column are assigned NaN. This method is useful for small datasets since concat() creates a new dataframe in every iteration and consumes much more memory . So we can also use reindex() to preprocess the dataframes and concat at one go as well.

Example 2: Here we have three dataframes. So we will iterate and append the dataframes to the list. Lastly we will use concat() to combine all the dataframes that are present in the list.

import pandas as pd # Create sample DataFrames df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]}) df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]}) df3 = pd.DataFrame({'A': [9, 10], 'B': [11, 12]}) # Append DataFrames to a list df_list = [] for i in range(1,4): df_list.append(eval(f'df{i}')) # Concatenate all DataFrames in the list result = pd.concat(df_list, ignore_index=True) print(result) 

Output:

So here we have appended all the dataframes to a list using append method and then use concat() to combine the dataframes.