Je souhaite supprimer des chaînes de col1
présents dans col2
:
val df = spark.createDataFrame(Seq(
("Hi I heard about Spark", "Spark"),
("I wish Java could use case classes", "Java"),
("Logistic regression models are neat", "models")
)).toDF("sentence", "label")
en utilisant regexp_replace
ou translate
ref: fonctions spark api
val res = df.withColumn("sentence_without_label", regexp_replace
(col("sentence") , "(?????)", "" ))
de sorte que res
ressemble à ci-dessous:
Vous pouvez simplement utiliser regexp_replace
df5.withColumn("sentence_without_label", regexp_replace($"sentence" , lit($"label"), lit("" )))
ou vous pouvez utiliser la fonction udf simple comme ci-dessous
val df5 = spark.createDataFrame(Seq(
("Hi I heard about Spark", "Spark"),
("I wish Java could use case classes", "Java"),
("Logistic regression models are neat", "models")
)).toDF("sentence", "label")
val replace = udf((data: String , rep : String)=>data.replaceAll(rep, ""))
val res = df5.withColumn("sentence_without_label", replace($"sentence" , $"label"))
res.show()
Production:
+-----------------------------------+------+------------------------------+
|sentence |label |sentence_without_label |
+-----------------------------------+------+------------------------------+
|Hi I heard about Spark |Spark |Hi I heard about |
|I wish Java could use case classes |Java |I wish could use case classes|
|Logistic regression models are neat|models|Logistic regression are neat |
+-----------------------------------+------+------------------------------+
Si label
c'est juste un littéral c'est assez simple:
import org.Apache.spark.sql.functions._
df.withColumn("sentence_without_label",
regexp_replace(col("sentence"), col("label"), lit(""))).show(false)
+-----------------------------------+------+------------------------------+
|sentence |label |sentence_without_label |
+-----------------------------------+------+------------------------------+
|Hi I heard about Spark |Spark |Hi I heard about |
|I wish Java could use case classes |Java |I wish could use case classes|
|Logistic regression models are neat|models|Logistic regression are neat |
+-----------------------------------+------+------------------------------+
Dans Spark 1.6, vous pouvez faire de même avec expr
:
df.withColumn(
"sentence_without_label",
expr("regexp_replace(sentence, label, '')"))