我正在對(duì)地址進(jìn)行分組，其中有多種類型的地址，但我需要對(duì)它們進(jìn)行分組并應(yīng)用計(jì)數(shù)和排序。輸入Address1 Address2 Address3
a1 b1 c1
b2 a2 e2輸出要求Address4
a1
b2
b1
a2
c1
e2
3 回答
滄海一幻覺
TA貢獻(xiàn)1824條經(jīng)驗(yàn) 獲得超5個(gè)贊
給出的實(shí)現(xiàn)相同，但使用更高級(jí)別的數(shù)據(jù)集 api。
Dataset<Row> Ad1 = df.select(functions.col("Address1").as("Address4"));
Dataset<Row> Ad2 = df.select("Address2");
Dataset<Row> Ad3 = df.select("Address3");
Dataset<Row> Union_DS = Ad1.union(Ad2).union(Ad3);
Union_DS.show();
Dataset<Row> Union_Sorted = Union_DS
.groupBy("Address4")
.agg(functions.count(functions.col("Address4")).as("Count"))
.sort(functions.desc("Count"))
;
Union_Sorted.show();
慕娘9325324
TA貢獻(xiàn)1783條經(jīng)驗(yàn) 獲得超4個(gè)贊
您應(yīng)該能夠使用UNIONSpark SQL 來解決這個(gè)問題：
spark.sql(
"""
|SELECT Address4
|FROM (
| SELECT Address1 FROM table
| UNION
| SELECT Address2 FROM table
| UNION
| SELECT Address3 FROM table
| )
""".stripMargin).show()
慕的地8271018
TA貢獻(xiàn)1796條經(jīng)驗(yàn) 獲得超4個(gè)贊
#You can be able to do it with the below approach
val input_rdd = spark.sparkContext.parallelize(List(("a1", "b1", "c1"), ("a1", "b2", "c1"), ("a1", "b1", "c2"), ("a2", "b2", "c2")))
val input_df = input_rdd.toDF("Address1", "Address2", "Address3")
input_df.show()
+--------+--------+--------+
|Address1|Address2|Address3|
+--------+--------+--------+
| a1| b1| c1|
| a1| b2| c1|
| a1| b1| c2|
| a2| b2| c2|
+--------+--------+--------+
val out_address1_df = input_df.groupBy("Address1").agg(count(input_df("Address1")).as("count_address1")).
select(input_df("Address1").as("ADDRESS"), col("count_address1").as("COUNT"))
//out_address1_df.show()
val out_address2_df = input_df.groupBy("Address2").agg(count(input_df("Address2")).as("count_address2")).
select(input_df("Address2").as("ADDRESS"), col("count_address2").as("COUNT"))
//out_address2_df.show()
val out_address3_df = input_df.groupBy("Address3").agg(count(input_df("Address3")).as("count_address3")).
select(input_df("Address3").as("ADDRESS"), col("count_address3").as("COUNT"))
val output_df = out_address1_df.unionAll(out_address2_df).unionAll(out_address3_df)
output_df.show()
+-------+-----+
|ADDRESS|COUNT|
+-------+-----+
| a2| 1|
| a1| 3|
| b2| 2|
| b1| 2|
| c1| 2|
| c2| 2|
+-------+-----+
添加回答
舉報(bào)
0/150
提交
取消