Python pyspark : drop (column 삭제하기)

 

 

 

 

Spark dataframe에 drop method를 적용하면 특정 column을 제거(drop)할 수 있습니다.

 

from pyspark.sql import SparkSession
from pyspark.sql.functions import col
import pandas as pd

spark = SparkSession.builder.getOrCreate()

df_test = pd.DataFrame({
    'a': [1, 2, 3],
    'b': [10.0, 3.5, 7.315],
    'c': ['apple', 'banana', 'tomato']
})
df_spark = spark.createDataFrame(df_test)
df_spark.show()

df_spark_new = df_spark.drop(df_spark.a)
df_spark_new.show()



-- Result
+---+------+-------+
|  a|     b|      c|
+---+------+-------+
|  1|  10.0|  apple|
|  2|   3.5| banana|
|  3| 7.315| tomato|
+---+------+-------+

+------+-------+
|     b|      c|
+------+-------+
|  10.0|  apple|
|   3.5| banana|
| 7.315| tomato|
+------+-------+

 

dataframe.drop(dataframe.column_name)

위처럼 drop을 적용할 dataframe에다가

drop method의 parameter로서 drop할 column을 적어주면 됩니다.