목록Python (384)

달나라 노트

Python pyspark : drop (column 삭제하기)

Spark dataframe에 drop method를 적용하면 특정 column을 제거(drop)할 수 있습니다. from pyspark.sql import SparkSession from pyspark.sql.functions import col import pandas as pd spark = SparkSession.builder.getOrCreate() df_test = pd.DataFrame({ 'a': [1, 2, 3], 'b': [10.0, 3.5, 7.315], 'c': ['apple', 'banana', 'tomato'] }) df_spark = spark.createDataFrame(df_test) df_spark.show() df_spark_new = df_spark.drop(df_sp..
Python/Python pyspark 2021. 5. 31. 15:06
Python pyspark : lit (literal value로 column 만들기)

pyspark의 lit method는 literal value로 column을 생성할 때 사용됩니다. from pyspark.sql import SparkSession from pyspark.sql.functions import col, lit import pandas as pd spark = SparkSession.builder.getOrCreate() df_test = pd.DataFrame({ 'a': [1, 2, 3], 'b': [10.0, 3.5, 7.315], 'c': ['apple', 'banana', 'tomato'] }) df_spark = spark.createDataFrame(df_test) df_new = df_spark.select( col('a'), col('b'), lit('z..
Python/Python pyspark 2021. 5. 28. 20:24
Python pyspark : udf (사용자 정의 함수 User Defined Functions 사용하기)

udf method를 이용하면 만든 function을 column에 적용시킬 수 있습니다. from pyspark.sql import SparkSession from pyspark.sql.functions import udf from pyspark.sql.function import col import pandas as pd spark = SparkSession.builder.getOrCreate() df_test = pd.DataFrame({ 'a': [1, 2, 3], 'b': [10.0, 3.5, 7.315], 'c': ['apple', 'banana', 'tomato'] }) df_spark = spark.createDataFrame(df_test) def myfunc(x): # 1 return ..
Python/Python pyspark 2021. 5. 28. 20:13
Python pyspark : alias (컬럼 이름 변경하기)

alias method를 이용해서 spark dataframe의 컬럼에 새로운 이름을 붙여줄 수 있습니다. from pyspark.sql import SparkSession from pyspark.sql.functions import col import pandas as pd spark = SparkSession.builder.getOrCreate() df_test = pd.DataFrame({ 'a': [1, 2, 3], 'b': [10.0, 3.5, 7.315], 'c': ['apple', 'banana', 'tomato'] }) df_spark = spark.createDataFrame(df_test) df_spark = df_spark.select( col('a'), col('b'), col('c..
Python/Python pyspark 2021. 5. 28. 19:10
Prev 1 ··· 66 67 68 69 70 71 72 ··· 96 Next