我想使用 Java 將 Google BigQuery 中的表中的數(shù)據(jù)讀取到 Spark 中。我如何在 Java 中做到這一點(diǎn)，我需要什么依賴項(xiàng)以及生成的數(shù)據(jù)類型是什么？我能找到的所有東西都在 Scala 中，但我需要 Java 中的。
1 回答
動漫人物
TA貢獻(xiàn)1815條經(jīng)驗(yàn) 獲得超10個贊
下面是 Scala Shakespeare 示例的 Java 等效項(xiàng)：
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
public class JavaShakespeare {
public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("spark-bigquery-demo")
.getOrCreate();
// Use the Cloud Storage bucket for temporary BigQuery export data used
// by the connector. This assumes the Cloud Storage connector for
// Hadoop is configured.
String bucket = spark.sparkContext().hadoopConfiguration().get("fs.gs.system.bucket");
spark.conf().set("temporaryGcsBucket", bucket);
// Load data in from BigQuery.
Dataset<Row> wordsDF = spark.read().format("bigquery")
.option("table", "publicdata.samples.shakespeare").load().cache();
wordsDF.show();
wordsDF.printSchema();
wordsDF.createOrReplaceTempView("words");
// Perform word count.
Dataset<Row> wordCountDF = spark.sql(
"SELECT word, SUM(word_count) AS word_count FROM words GROUP BY word");
// Saving the data to BigQuery
wordCountDF.write().format("bigquery").option("table", "wordcount_dataset.wordcount_output")
.save();
}
}
添加回答
舉報
0/150
提交
取消