如何用category其中的不同值填充這一列空值？+---++--------+----------+| id||category| Date|+---+---------+----------+| A1| Null|2010-01-02|| A1| Null|2010-01-03|| A1| Nixon|2010-01-04|| A1| Null|2010-01-05|| A9| Null|2010-05-02|| A9| Leonard|2010-05-03|| A9| Null|2010-05-04|| A9| Null|2010-05-05|+---+---------+----------+所需的數(shù)據(jù)框：+---++--------+----------+| id||category| Date|+---+---------+----------+| A1| Nixon|2010-01-02|| A1| Nixon|2010-01-03|| A1| Nixon|2010-01-04|| A1| Nixon|2010-01-05|| A9| Leonard|2010-05-02|| A9| Leonard|2010-05-03|| A9| Leonard|2010-05-04|| A9| Leonard|2010-05-05|+---+---------+----------+我試過(guò)：w = Window().partitionBy("ID").orderBy("Date")df = df.withColumn("category", F.when(col("category").isNull(), col("category")\.distinct().over(w))\.otherwise(col("category")))我也嘗試過(guò)：df = df.fillna({'category': col('category').distinct()})我也嘗試過(guò)：df = df.withColumn('category', when(df.category.isNull(), df.category.distinct()).otherwise(df.category))
1 回答
斯蒂芬大帝
TA貢獻(xiàn)1827條經(jīng)驗(yàn) 獲得超8個(gè)贊
您可以使用first()參數(shù)ignorenullsas True。
另外，請(qǐng)rowsBetween(-sys.maxsize, sys.maxsize)在窗戶上使用。
from pyspark.sql import functions as F
from pyspark.sql.functions import *
from pyspark.sql.window import Window
import sys
w = Window().partitionBy("id").orderBy("Date")
df.withColumn("new", F.first('category', True).over(w.rowsBetween(-sys.maxsize, sys.maxsize)))\
.orderBy("id", "Date").show()
+---+--------+----------+
| id|category| Date|
+---+--------+----------+
| A1| Nixon|2010-01-02|
| A1| Nixon|2010-01-03|
| A1| Nixon|2010-01-04|
| A1| Nixon|2010-01-05|
| A9| Leonard|2010-05-02|
| A9| Leonard|2010-05-03|
| A9| Leonard|2010-05-04|
| A9| Leonard|2010-05-05|
+---+--------+----------+
添加回答
舉報(bào)
0/150
提交
取消