在這個問題之后，我現(xiàn)在運行這個代碼：List<StructField> fields = new ArrayList<>();fields.add(DataTypes.createStructField("A",DataTypes.LongType,true));fields.add(DataTypes.createStructField("B",DataTypes.DoubleType,true));StructType schema1 = DataTypes.createStructType(fields);Dataset<Row> df1 = spark.sql("select 1 as A, 2.2 as B");Dataset<Row> finalDf1 = spark.createDataFrame(df1.javaRDD(), schema1);fields = new ArrayList<>();fields.add(DataTypes.createStructField("B",DataTypes.DoubleType,true));fields.add(DataTypes.createStructField("A",DataTypes.LongType,true));StructType schema2 = DataTypes.createStructType(fields);Dataset<Row> df2 = spark.sql("select 2.2 as B, 1 as A");Dataset<Row> finalDf2 = spark.createDataFrame(df2.javaRDD(), schema2);finalDf1.printSchema();finalDf2.printSchema();System.out.println(finalDf1.schema());System.out.println(finalDf2.schema());System.out.println(finalDf1.schema().equals(finalDf2.schema()));這是輸出：root |-- A: long (nullable = true) |-- B: double (nullable = true)root |-- B: double (nullable = true) |-- A: long (nullable = true)StructType(StructField(A,LongType,true), StructField(B,DoubleType,true))StructType(StructField(B,DoubleType,true), StructField(A,LongType,true))false雖然列的排列順序不同，但這兩個數(shù)據(jù)集具有完全相同的列和列類型。這里需要什么比較才能得到true？
3 回答
慕沐林林
TA貢獻2016條經(jīng)驗 獲得超9個贊
假設訂單 cols 不匹配并且相同的名稱具有相同的語義并且需要相同數(shù)量的列。
一個使用 SCALA 的例子，你應該能夠適應 JAVA：
import spark.implicits._
val df = sc.parallelize(Seq(
("A", "X", 2, 100), ("A", "X", 7, 100), ("B", "X", 10, 100),
("C", "X", 1, 100), ("D", "X", 50, 100), ("E", "X", 30, 100)
)).toDF("c1", "c2", "Val1", "Val2")
val names = df.columns
val df2 = sc.parallelize(Seq(
("A", "X", 2, 1))).toDF("c1", "c2", "Val1", "Val2")
val names2 = df2.columns
names.sortWith(_ < _) sameElements names2.sortWith(_ < _)
返回真或假，試驗輸入。
慕標5832272
TA貢獻1966條經(jīng)驗 獲得超4個贊
如果它們的順序不同，則它們不相同。即使它們都具有相同的列數(shù)和相同的名稱。如果您想查看兩個架構是否具有相同的列名，請從兩個數(shù)據(jù)幀的列表中獲取架構，然后編寫代碼來比較它們。見下面的java示例
public static void main(String[] args)
{
List<String> firstSchema =Arrays.asList(DataTypes.createStructType(ConfigConstants.firstSchemaFields).fieldNames());
List<String> secondSchema = Arrays.asList(DataTypes.createStructType(ConfigConstants.secondSchemaFields).fieldNames());
if(schemasHaveTheSameColumnNames(firstSchema,secondSchema))
{
System.out.println("Yes, schemas have the same column names");
}else
{
System.out.println("No, schemas do not have the same column names");
}
}
private static boolean schemasHaveTheSameColumnNames(List<String> firstSchema, List<String> secondSchema)
{
if(firstSchema.size() != secondSchema.size())
{
return false;
}else
{
for (String column : secondSchema)
{
if(!firstSchema.contains(column))
return false;
}
}
return true;
}
夢里花落0921
TA貢獻1772條經(jīng)驗 獲得超6個贊
按照前面的答案，似乎是比較StructFields（列和類型）而不僅僅是名稱的最快方法如下：
Set<StructField> set1 = new HashSet<>(Arrays.asList(schema1.fields()));
Set<StructField> set2 = new HashSet<>(Arrays.asList(schema2.fields()));
boolean result = set1.equals(set2);
添加回答
舉報
0/150
提交
取消