web-dev-qa-db-fra.com

Comment utiliser la clause NOT IN en condition de filtre en spark

Je veux filtrer une colonne d'une source RDD:

val source = sql("SELECT * from sample.source").rdd.map(_.mkString(","))
val destination = sql("select * from sample.destination").rdd.map(_.mkString(","))

val source_primary_key = source.map(rec => (rec.split(",")(0)))
val destination_primary_key = destination.map(rec => (rec.split(",")(0)))

val src = source_primary_key.subtractByKey(destination_primary_key)

Je veux utiliser la clause IN dans la condition de filtre pour filtrer uniquement les valeurs présentes dans src depuis la source, comme ci-dessous (EDITED):

val source = spark.read.csv(inputPath + "/source").rdd.map(_.mkString(","))
val destination = spark.read.csv(inputPath + "/destination").rdd.map(_.mkString(","))

val source_primary_key = source.map(rec => (rec.split(",")(0)))
val destination_primary_key = destination.map(rec => (rec.split(",")(0)))

val extra_in_source = source_primary_key.filter(rec._1 != destination_primary_key._1)

code SQL équivalent est 

SELECT * FROM SOURCE WHERE ID IN (select ID from src)

Je vous remercie

scalaapache-sparkapache-spark-sql
7
Vignesh

Puisque votre code n'est pas reproductible, voici un petit exemple utilisant spark-sql pour savoir comment select * from t where id in (...):

// create a DataFrame for a range 'id' from 1 to 9.
scala> val df = spark.range(1,10).toDF
df: org.Apache.spark.sql.DataFrame = [id: bigint]

// values to exclude
scala> val f = Seq(5,6,7)
f: Seq[Int] = List(5, 6, 7)

// select * from df where id is not in the values to exclude
scala> df.filter(!col("id").isin(f  : _*)).show
+---+                                                                           
| id|
+---+
|  1|
|  2|
|  3|
|  4|
|  8|
|  9|
+---+

// select * from df where id is in the values to exclude
scala> df.filter(col("id").isin(f  : _*)).show

Voici la version RDD du not isin:

scala> val rdd = sc.parallelize(1 to 10)
rdd: org.Apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> val f = Seq(5,6,7)
f: Seq[Int] = List(5, 6, 7)

scala> val rdd2 = rdd.filter(x => !f.contains(x))
rdd2: org.Apache.spark.rdd.RDD[Int] = MapPartitionsRDD[3] at filter at <console>:28

Néanmoins, je crois toujours que c'est exagéré, car vous utilisez déjà spark-sql.

Il semble que dans votre cas, vous utilisez réellement des DataFrames, les solutions mentionnées ci-dessus ne fonctionnent donc pas.

Vous pouvez utiliser l'approche anti-jointure gauche:

scala> val source = spark.read.format("csv").load("source.file")
source: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields]

scala> val destination = spark.read.format("csv").load("destination.file")
destination: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields]

scala> source.show
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|_c0|               _c1|     _c2|       _c3|            _c4|_c5|_c6|       _c7|  _c8|      _c9|        _c10|
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|  1|        Ravi kumar|   Ravi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  2|Shekhar shudhanshu| Shekhar|shudhanshu|      Manulife |  2|  M|18-01-1994|76.34|   250000|     Alaska |
|  3|Preethi Narasingam| Preethi|Narasingam|        Retail |  3|  F|19-01-1994|77.45|270000.01|    Arizona |
|  4|     Abhishek Nair|Abhishek|      Nair|       Banking |  4|  M|20-01-1994|78.65|   345000|   Arkansas |
|  5|        Ram Sharma|     Ram|    Sharma|Infrastructure |  5|  M|21-01-1994|79.12|    45000| California |
|  6|   Chandani Kumari|Chandani|    Kumari|          BNFS |  6|  F|22-01-1994|80.13| 43000.02|   Colorado |
|  7|      Balaji Kumar|  Balaji|     Kumar|           MSO |  1|  M|23-01-1994|81.33|  1234678|Connecticut |
|  8|  Naveen Shekrappa|  Naveen| Shekrappa|      Manulife |  2|  M|24-01-1994|  100|   789414|   Delaware |
|  9|     Milind Chavan|  Milind|    Chavan|        Retail |  3|  M|25-01-1994|83.66|   245555|    Florida |
| 10|      Raghu Rajeev|   Raghu|    Rajeev|       Banking |  4|  M|26-01-1994|87.65|   235468|     Georgia|
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+


scala> destination.show
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|_c0|                _c1|     _c2|       _c3|            _c4|_c5|_c6|       _c7|  _c8|      _c9|        _c10|
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|  1|         Ravi kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  1|        Ravi1 kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  1|        Ravi2 kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  2| Shekhar shudhanshu| Shekhar|shudhanshu|      Manulife |  2|  M|18-01-1994|76.34|   250000|     Alaska |
|  3|Preethi Narasingam1| Preethi|Narasingam|        Retail |  3|  F|19-01-1994|77.45|270000.01|    Arizona |
|  4|     Abhishek Nair1|Abhishek|      Nair|       Banking |  4|  M|20-01-1994|78.65|   345000|   Arkansas |
|  5|         Ram Sharma|     Ram|    Sharma|Infrastructure |  5|  M|21-01-1994|79.12|    45000| California |
|  6|    Chandani Kumari|Chandani|    Kumari|          BNFS |  6|  F|22-01-1994|80.13| 43000.02|   Colorado |
|  7|       Balaji Kumar|  Balaji|     Kumar|           MSO |  1|  M|23-01-1994|81.33|  1234678|Connecticut |
|  8|   Naveen Shekrappa|  Naveen| Shekrappa|      Manulife |  2|  M|24-01-1994|  100|   789414|   Delaware |
|  9|      Milind Chavan|  Milind|    Chavan|        Retail |  3|  M|25-01-1994|83.66|   245555|    Florida |
| 10|       Raghu Rajeev|   Raghu|    Rajeev|       Banking |  4|  M|26-01-1994|87.65|   235468|     Georgia|
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+

Vous aurez juste besoin de faire ce qui suit:

scala> val res1 = source.join(destination, Seq("_c0"), "leftanti")

scala> val res2 = destination.join(source, Seq("_c0"), "leftanti")

C'est la même logique que j'ai mentionné dans ma réponse ici .

14
eliasah

Vous pouvez essayer comme--

df.filter(~df.Dept.isin("30","20")).show()

// Ceci listera toutes les colonnes de df où Dept NOT IN 30 ou 20

1
Blue Bird