How to add a constant column in a PySpark DataFrame?

Adding a constant column to a PySpark DataFrame is quite straightforward. You can achieve this by using the lit function from the pyspark.sql.functions module, which is used to create a column expression with a constant value.

Here��s a step-by-step guide to add a constant column to a PySpark DataFrame:

Step 1: Import Necessary Libraries

First, make sure you have PySpark installed. If not, you can install it via pip:

pip install pyspark 

Then, import the necessary modules in your Python script:

from pyspark.sql import SparkSession from pyspark.sql.functions import lit 

Step 2: Initialize Spark Session

Create a Spark session, which is the entry point to using PySpark:

spark = SparkSession.builder \ .appName("AddConstantColumn") \ .getOrCreate() 

Step 3: Create a DataFrame

Let��s assume you already have a DataFrame. If not, you can create one for demonstration:

data = [("John", 28), ("Jane", 25), ("Mike", 23)] df = spark.createDataFrame(data, ["Name", "Age"]) df.show() 

Step 4: Add a Constant Column

Use the lit function to add a new column with a constant value:

constant_value = 1 # Example constant value df = df.withColumn("NewConstantColumn", lit(constant_value)) df.show() 

This will add a new column named "NewConstantColumn" with the constant value 1 to your DataFrame.

Full Example Code

Here��s the full example code put together:

from pyspark.sql import SparkSession from pyspark.sql.functions import lit # Initialize Spark Session spark = SparkSession.builder \ .appName("AddConstantColumn") \ .getOrCreate() # Sample DataFrame data = [("John", 28), ("Jane", 25), ("Mike", 23)] df = spark.createDataFrame(data, ["Name", "Age"]) # Show initial DataFrame print("Initial DataFrame:") df.show() # Add a constant column constant_value = 1 df = df.withColumn("NewConstantColumn", lit(constant_value)) # Show DataFrame with new constant column print("DataFrame with Constant Column:") df.show() # Stop the SparkSession spark.stop() 

Notes

• The lit function is versatile and can be used to add columns with various data types (string, integer, boolean, etc.).
• Remember to stop the SparkSession (spark.stop()) at the end of your script to release resources.
• PySpark DataFrames are immutable. When you use withColumn, it returns a new DataFrame with the added column, and the original DataFrame remains unchanged.

---

More Tags

control-statements savechanges figsize hcatalog backbone.js rselenium kafka-python sample embedded-tomcat-8 polling

More Programming Guides

• How Neural Networks are used for Regression in R?
• Escaping and Non-Escaping Closures in Swift
• Spring - REST XML Response
• Spring with Castor Example
• Similarities And Differences Between Error And Exception In Java
• How to calculate the difference in months between two dates in C#
• How to extract numbers from a string in Python
• Mac OS Install Python Environment

Other Guides

• MOQ - LINQ Predicates in Setup Method
• Default implementation for ListView OwnerDraw in C#
• How to do parallel programming in Python?
• Python nested dictionaries or tuples for key?
• Default or Defender methods in Java

More Programming Examples

• java - Required request body content is missing: org.springframework.web.method.HandlerMethod$HandlerMethodParameter
• c# - .net core - How to properly kill started process?
• c# - Most efficient way to find all modes in a List of randomly generated integers and how often they occured
• javascript - Is there any way to count number of tabs are opened in chrome?
• java - How to verify text color in selenium webdriver?
• Python asyncio - Loop exits with Task was destroyed but it is pending