我正在運(yùn)行 Spark 2.3。我想將features以下 DataFrame 中的列從ArrayType轉(zhuǎn)換為DenseVector. 我在 Java 中使用 Spark。+---+--------------------+| id| features|+---+--------------------+| 0|[4.191401, -1.793...|| 10|[-0.5674514, -1.3...|| 20|[0.735613, -0.026...|| 30|[-0.030161237, 0....|| 40|[-0.038345724, -0...|+---+--------------------+root |-- id: integer (nullable = false) |-- features: array (nullable = true) | |-- element: float (containsNull = false)我寫了以下內(nèi)容，UDF但似乎不起作用：private static UDF1 toVector = new UDF1<Float[], Vector>() { private static final long serialVersionUID = 1L; @Override public Vector call(Float[] t1) throws Exception { double[] DoubleArray = new double[t1.length]; for (int i = 0 ; i < t1.length; i++) { DoubleArray[i] = (double) t1[i]; } Vector vector = (org.apache.spark.mllib.linalg.Vector) Vectors.dense(DoubleArray); return vector; }}我希望將以下特征提取為向量，以便對其進(jìn)行聚類。我也在注冊 UDF，然后繼續(xù)調(diào)用它，如下所示：spark.udf().register("toVector", (UserDefinedAggregateFunction) toVector);df3 = df3.withColumn("featuresnew", callUDF("toVector", df3.col("feautres")));df3.show(); 在運(yùn)行此代碼段時，我面臨以下錯誤：ReadProcessData$1 不能轉(zhuǎn)換為 org.apache.spark.sql.expressions。用戶定義的聚合函數(shù)
1 回答
慕尼黑的夜晚無繁華
TA貢獻(xiàn)1864條經(jīng)驗(yàn) 獲得超6個贊
問題在于您如何udf在 Spark中注冊。您不應(yīng)將UserDefinedAggregateFunctionwhich is not an udfbut an udafused 用于聚合。相反，您應(yīng)該做的是：
spark.udf().register("toVector", toVector, new VectorUDT());
然后要使用注冊的函數(shù)，請使用：
df3.withColumn("featuresnew", callUDF("toVector",df3.col("feautres")));
在udf本身應(yīng)稍微調(diào)整如下：
UDF1 toVector = new UDF1<Seq<Float>, Vector>(){
public Vector call(Seq<Float> t1) throws Exception {
List<Float> L = scala.collection.JavaConversions.seqAsJavaList(t1);
double[] DoubleArray = new double[t1.length()];
for (int i = 0 ; i < L.size(); i++) {
DoubleArray[i]=L.get(i);
}
return Vectors.dense(DoubleArray);
}
};
請注意，在Spark 2.3+ 中，您可以創(chuàng)建udf可直接調(diào)用的 Scala 樣式。從這個答案：
UserDefinedFunction toVector = udf(
(Seq<Float> array) -> /* udf code or method to call */, new VectorUDT()
);
df3.withColumn("featuresnew", toVector.apply(col("feautres")));
添加回答
舉報(bào)
0/150
提交
取消