Sur la base de la DataFrame
suivante:
val client = Seq((1,"A",10),(2,"A",5),(3,"B",56)).toDF("ID","Categ","Amnt")
+---+-----+----+
| ID|Categ|Amnt|
+---+-----+----+
| 1| A| 10|
| 2| A| 5|
| 3| B| 56|
+---+-----+----+
Je voudrais obtenir le numéro d'identification et le montant total par catégorie:
+-----+-----+---------+
|Categ|count|sum(Amnt)|
+-----+-----+---------+
| B| 1| 56|
| A| 2| 15|
+-----+-----+---------+
Est-il possible de faire le compte et la somme sans avoir à faire une jointure?
client.groupBy("Categ").count
.join(client.withColumnRenamed("Categ","cat")
.groupBy("cat")
.sum("Amnt"), 'Categ === 'cat)
.drop("cat")
Peut-être quelque chose comme ça:
client.createOrReplaceTempView("client")
spark.sql("SELECT Categ count(Categ) sum(Amnt) FROM client GROUP BY Categ").show()
Je donne un exemple différent du vôtre
Plusieurs fonctions de groupe sont possibles comme ceci. essayez en conséquence
// In 1.3.x, in order for the grouping column "department" to show up,
// it must be included explicitly as part of the agg function call.
df.groupBy("department").agg($"department", max("age"), sum("expense"))
// In 1.4+, grouping column "department" is included automatically.
df.groupBy("department").agg(max("age"), sum("expense"))
import org.Apache.spark.sql.{DataFrame, SparkSession}
import org.Apache.spark.sql.functions._
val spark: SparkSession = SparkSession
.builder.master("local")
.appName("MyGroup")
.getOrCreate()
import spark.implicits._
val client: DataFrame = spark.sparkContext.parallelize(
Seq((1,"A",10),(2,"A",5),(3,"B",56))
).toDF("ID","Categ","Amnt")
client.groupBy("Categ").agg(sum("Amnt"),count("ID")).show()
+-----+---------+---------+
|Categ|sum(Amnt)|count(ID)|
+-----+---------+---------+
| B| 56| 1|
| A| 15| 2|
+-----+---------+---------+
Vous pouvez faire une agrégation comme ci-dessous sur un tableau donné:
client.groupBy("Categ").agg(sum("Amnt"),count("ID")).show()
+-----+---------+---------+
|Categ|sum(Amnt)|count(ID)|
+-----+---------+---------+
| A| 15| 2|
| B| 56| 1|
+-----+---------+---------+
Il y a plusieurs façons de faire des fonctions d'agrégation dans spark,
val client = Seq((1,"A",10),(2,"A",5),(3,"B",56)).toDF("ID","Categ","Amnt")
1.
val aggdf = client.groupBy('Categ).agg(Map("ID"->"count","Amnt"->"sum"))
+-----+---------+---------+
|Categ|count(ID)|sum(Amnt)|
+-----+---------+---------+
|B |1 |56 |
|A |2 |15 |
+-----+---------+---------+
//Rename and sort as needed.
aggdf.sort('Categ).withColumnRenamed("count(ID)","Count").withColumnRenamed("sum(Amnt)","sum")
+-----+-----+---+
|Categ|Count|sum|
+-----+-----+---+
|A |2 |15 |
|B |1 |56 |
+-----+-----+---+
2.
import org.Apache.spark.sql.functions._
client.groupBy('Categ).agg(count("ID").as("count"),sum("Amnt").as("sum"))
+-----+-----+---+
|Categ|count|sum|
+-----+-----+---+
|B |1 |56 |
|A |2 |15 |
+-----+-----+---+
3.
import com.google.common.collect.ImmutableMap;
client.groupBy('Categ).agg(ImmutableMap.of("ID", "count", "Amnt", "sum"))
+-----+---------+---------+
|Categ|count(ID)|sum(Amnt)|
+-----+---------+---------+
|B |1 |56 |
|A |2 |15 |
+-----+---------+---------+
//Use column rename is required.
4. Si vous êtes un expert en SQL, vous pouvez le faire aussi
client.createOrReplaceTempView("df")
val aggdf = spark.sql("select Categ, count(ID),sum(Amnt) from df group by Categ")
aggdf.show()
+-----+---------+---------+
|Categ|count(ID)|sum(Amnt)|
+-----+---------+---------+
| B| 1| 56|
| A| 2| 15|
+-----+---------+---------+