web-dev-qa-db-fra.com

Exception de classe introuvable dans le travail Mapreduce wordcount

j'essaie d'exécuter un travail wordcount dans hadoop.but de toujours obtenir une classe non trouvée exception. Je poste la classe que j'ai écrite et la commande que j'utilise pour exécuter le travail.

import Java.io.IOException;
import Java.util.*;
import org.Apache.hadoop.fs.Path;
import org.Apache.hadoop.conf.*;
import org.Apache.hadoop.io.*;
import org.Apache.hadoop.mapreduce.*;
import org.Apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.Apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.Apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.Apache.hadoop.mapreduce.lib.output.TextOutputFormat;

 public class WordCount {

 public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text Word = new Text();

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            Word.set(tokenizer.nextToken());
            context.write(Word, one);
        }
    }
 } 

 public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

    public void reduce(Text key, Iterable<IntWritable> values, Context context) 
      throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable val : values) {
            sum += val.get();
        }
        context.write(key, new IntWritable(sum));
    }
 }

 public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();

    Job job = new Job(conf, "WordCount");

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);
    job.setJarByClass(WordCount.class);
 }

}

le wordcount.jar est exporté vers mon dossier de téléchargements . C'est la commande que j'utilise pour exécuter le travail.

   jeet@jeet-Vostro-2520:~/Downloads$ hadoop jar wordcount.jar org.gamma.WordCount /user/jeet/getty/gettysburg.txt /user/jeet/getty/out

dans ce cas, mon travail mapreduce est démarré mais il se termine au milieu du processus. Impression de l'arbre des exceptions.

 14/01/27 13:16:02 WARN mapred.JobClient: Use GenericOptionsParser for parsing the   arguments. Applications should implement Tool for the same.
14/01/27 13:16:02 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/01/27 13:16:02 INFO input.FileInputFormat: Total input paths to process : 1
14/01/27 13:16:02 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/01/27 13:16:02 WARN snappy.LoadSnappy: Snappy native library not loaded
14/01/27 13:16:03 INFO mapred.JobClient: Running job: job_201401271247_0001
14/01/27 13:16:04 INFO mapred.JobClient:  map 0% reduce 0%
14/01/27 13:16:11 INFO mapred.JobClient: Task Id : attempt_201401271247_0001_m_000000_0,     Status : FAILED
    Java.lang.RuntimeException: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:849)
at org.Apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.Java:199)
at org.Apache.hadoop.mapred.MapTask.runNewMapper(MapTask.Java:719)
at org.Apache.hadoop.mapred.MapTask.run(MapTask.Java:370)
at org.Apache.hadoop.mapred.Child$4.run(Child.Java:255)
at Java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.Java:415)
at org.Apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.Java:1149)
at org.Apache.hadoop.mapred.Child.main(Child.Java:249)
    Caused by: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:366)
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:355)
at Java.security.AccessController.doPrivileged(Native Method)
at Java.net.URLClassLoader.findClass(URLClassLoader.Java:354)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:425)
at Sun.misc.Launcher$AppClassLoader.loadClass(Launcher.Java:308)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:358)
at Java.lang.Class.forName0(Native Method)
at Java.lang.Class.forName(Class.Java:270)
at org.Apache.hadoop.conf.Configuration.getClassByName(Configuration.Java:802)
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:847)
... 8 more

14/01/27 13:16:16 INFO mapred.JobClient: Task Id : attempt_201401271247_0001_m_000000_1,   Status : FAILED
Java.lang.RuntimeException: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:849)
at org.Apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.Java:199)
at org.Apache.hadoop.mapred.MapTask.runNewMapper(MapTask.Java:719)
at org.Apache.hadoop.mapred.MapTask.run(MapTask.Java:370)
at org.Apache.hadoop.mapred.Child$4.run(Child.Java:255)
at Java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.Java:415)
at   org.Apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.Java:1149)
at org.Apache.hadoop.mapred.Child.main(Child.Java:249)
    Caused by: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:366)
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:355)
at Java.security.AccessController.doPrivileged(Native Method)
at Java.net.URLClassLoader.findClass(URLClassLoader.Java:354)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:425)
at Sun.misc.Launcher$AppClassLoader.loadClass(Launcher.Java:308)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:358)
at Java.lang.Class.forName0(Native Method)
at Java.lang.Class.forName(Class.Java:270)
at org.Apache.hadoop.conf.Configuration.getClassByName(Configuration.Java:802)
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:847)
... 8 more

  14/01/27 13:16:20 INFO mapred.JobClient: Task Id : attempt_201401271247_0001_m_000000_2, Status : FAILED
  Java.lang.RuntimeException: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:849)
at org.Apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.Java:199)
at org.Apache.hadoop.mapred.MapTask.runNewMapper(MapTask.Java:719)
at org.Apache.hadoop.mapred.MapTask.run(MapTask.Java:370)
at org.Apache.hadoop.mapred.Child$4.run(Child.Java:255)
at Java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.Java:415)
at org.Apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.Java:1149)
at org.Apache.hadoop.mapred.Child.main(Child.Java:249)
    Caused by: Java.lang.ClassNotFoundException: org.gamma.WordCount$Map
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:366)
at Java.net.URLClassLoader$1.run(URLClassLoader.Java:355)
at Java.security.AccessController.doPrivileged(Native Method)
at Java.net.URLClassLoader.findClass(URLClassLoader.Java:354)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:425)
at Sun.misc.Launcher$AppClassLoader.loadClass(Launcher.Java:308)
at Java.lang.ClassLoader.loadClass(ClassLoader.Java:358)
at Java.lang.Class.forName0(Native Method)
at Java.lang.Class.forName(Class.Java:270)
at org.Apache.hadoop.conf.Configuration.getClassByName(Configuration.Java:802)
at org.Apache.hadoop.conf.Configuration.getClass(Configuration.Java:847)
... 8 more

   14/01/27 13:16:26 INFO mapred.JobClient: Job complete: job_201401271247_0001
   14/01/27 13:16:26 INFO mapred.JobClient: Counters: 7
   14/01/27 13:16:26 INFO mapred.JobClient:   Job Counters 
   14/01/27 13:16:26 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=20953
   14/01/27 13:16:26 INFO mapred.JobClient:     Total time spent by all reduces waiting    after reserving slots (ms)=0
   14/01/27 13:16:26 INFO mapred.JobClient:     Total time spent by all maps waiting    after reserving slots (ms)=0
   14/01/27 13:16:26 INFO mapred.JobClient:     Launched map tasks=4
   14/01/27 13:16:26 INFO mapred.JobClient:     Data-local map tasks=4
   14/01/27 13:16:26 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
   14/01/27 13:16:26 INFO mapred.JobClient:     Failed map tasks=1

somebody please please help i think i am very close of it
javadebugginghadoop
11
lucifer

Essayez d'ajouter ceci

Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
22
KT_admin

J'ai également eu le même problème et résolu en supprimant le même fichier WordCount.class dans le même répertoire à partir duquel j'exécute mon fichier jar. On dirait qu'il prend la classe à côté du bocal. Essayer 

4
Nagaraj Vittal

Je soupçonne ceci:

14/01/27 13:16:02 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

J'ai eu la même erreur lors de l'utilisation de CDH4.6 et cela a été résolu après avoir résolu l'avertissement ci-dessus.

3
Thamme Gowda

Vous devez ajouter cette méthode

job.setJarByClass (WordCount.class);

avant d'appeler la méthode

job.waitForCompletion (true);

Comme suit:

job.setJarByClass (WordCount.class); 

job.waitForCompletion (true);

1
Kalu

Utilisez le code ci-dessous pour résoudre ce problème. Job.setJarByClass (DriverClass.class);

0
Kumar Basapuram

job.setJarByClass (WordCount.class); job.waitForCompletion (true);

0
Victor

try job.setJar("wordcount.jar");, où wordcount.jar est le fichier jar que vous allez empaqueter. Cette méthode fonctionne pour moi, mais PAS setJarByClass!

0
Think Recursively

Bien que le programme MapReduce soit un traitement parallèle. Les classes Mapper, Combiner et Reducer ont un flux de séquence. Attendre la fin de chaque flux dépend de l’autre classe. job.waitForCompletion(true); Mais il faut définir le chemin d’entrée et de sortie avant de lancer les classes Mapper, Combiner et Reducer. Référence

Changez votre code comme ceci: 

 public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();

    Job job = new Job(conf, "WordCount");

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.setJarByClass(WordCount.class);
    job.waitForCompletion(true);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);
 }

J'espère que ça va marcher.

0
ǨÅVËĔŊ RĀǞĴĄŅ