hadoop - Datanode starts but not namenode -

after bit of struggling had managed use hadoop in pseudo-distributed node, namenode , jobtracker working (at http://localhost:50070 , http://localhost:50030)

yesterday tried restart namenode, datanode, etc with:

$hadoop namenode -format $start-all.sh 

and jps gives me following output:

17148 datanode 17295 secondarynamenode 17419 jobtracker 17669 jps 

namenode doesn't seem willing start anymore ... , jobtracker dies few seconds later.

mark hadn't restarted computer , i've tried solution given in following thread namenode not getting started didn't help.

here log of namenode, bunch of errors. don't know how solve issue @ all

    2013-08-16 09:02:21,647 info org.apache.hadoop.hdfs.server.namenode.namenode: startup_msg:  /************************************************************ startup_msg: starting namenode startup_msg:   host = localhost.lan/192.168.1.94 startup_msg:   args = [] startup_msg:   version = 1.2.1 startup_msg:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled 'mattf' on mon jul 22 15:23:09 pdt 2013 startup_msg:   java = 1.7.0_25 ************************************************************/ 2013-08-16 09:02:21,839 info org.apache.hadoop.metrics2.impl.metricsconfig: loaded properties hadoop-metrics2.properties 2013-08-16 09:02:21,868 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source metricssystem,sub=stats registered. 2013-08-16 09:02:21,871 info org.apache.hadoop.metrics2.impl.metricssystemimpl: scheduled snapshot period @ 10 second(s). 2013-08-16 09:02:21,871 info org.apache.hadoop.metrics2.impl.metricssystemimpl: namenode metrics system started 2013-08-16 09:02:22,098 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source ugi registered. 2013-08-16 09:02:22,103 warn org.apache.hadoop.metrics2.impl.metricssystemimpl: source name ugi exists! 2013-08-16 09:02:22,110 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source jvm registered. 2013-08-16 09:02:22,111 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source namenode registered. 2013-08-16 09:02:22,140 info org.apache.hadoop.hdfs.util.gset: computing capacity map blocksmap 2013-08-16 09:02:22,140 info org.apache.hadoop.hdfs.util.gset: vm type       = 64-bit 2013-08-16 09:02:22,140 info org.apache.hadoop.hdfs.util.gset: 2.0% max memory = 932118528 2013-08-16 09:02:22,140 info org.apache.hadoop.hdfs.util.gset: capacity      = 2^21 = 2097152 entries 2013-08-16 09:02:22,140 info org.apache.hadoop.hdfs.util.gset: recommended=2097152, actual=2097152 2013-08-16 09:02:22,174 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: fsowner=rlk 2013-08-16 09:02:22,174 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: supergroup=supergroup 2013-08-16 09:02:22,174 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: ispermissionenabled=true 2013-08-16 09:02:22,189 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: dfs.block.invalidate.limit=100 2013-08-16 09:02:22,189 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: isaccesstokenenabled=false accesskeyupdateinterval=0 min(s), accesstokenlifetime=0 min(s) 2013-08-16 09:02:22,271 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: registered fsnamesystemstatembean , namenodemxbean 2013-08-16 09:02:22,320 info org.apache.hadoop.hdfs.server.namenode.fseditlog: dfs.namenode.edits.toleration.length = 0 2013-08-16 09:02:22,321 info org.apache.hadoop.hdfs.server.namenode.namenode: caching file names occuring more 10 times  2013-08-16 09:02:22,363 info org.apache.hadoop.hdfs.server.common.storage: start loading image file /home/rlk/hduser/dfs/name/current/fsimage 2013-08-16 09:02:22,364 info org.apache.hadoop.hdfs.server.common.storage: number of files = 1 2013-08-16 09:02:22,372 info org.apache.hadoop.hdfs.server.common.storage: number of files under construction = 0 2013-08-16 09:02:22,375 info org.apache.hadoop.hdfs.server.common.storage: image file /home/rlk/hduser/dfs/name/current/fsimage of size 109 bytes loaded in 0 seconds. 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog: start loading edits file /home/rlk/hduser/dfs/name/current/edits 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog: eof of /home/rlk/hduser/dfs/name/current/edits, reached end of edit log number of transactions found: 0.  bytes read: 4 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog: start checking end of edit log (/home/rlk/hduser/dfs/name/current/edits) ... 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog: checked bytes after end of edit log (/home/rlk/hduser/dfs/name/current/edits): 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog:   padding position  = -1 (-1 means padding not found) 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog:   edit log length   = 4 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog:   read length       = 4 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog:   corruption length = 0 2013-08-16 09:02:22,376 info org.apache.hadoop.hdfs.server.namenode.fseditlog:   toleration length = 0 (= dfs.namenode.edits.toleration.length) 2013-08-16 09:02:22,382 info org.apache.hadoop.hdfs.server.namenode.fseditlog: summary: |---------- read=4 ----------|-- corrupt=0 --|-- pad=0 --| 2013-08-16 09:02:22,382 info org.apache.hadoop.hdfs.server.namenode.fseditlog: edits file /home/rlk/hduser/dfs/name/current/edits of size 4 edits # 0 loaded in 0 seconds. 2013-08-16 09:02:22,387 info org.apache.hadoop.hdfs.server.common.storage: image file /home/rlk/hduser/dfs/name/current/fsimage of size 109 bytes saved in 0 seconds. 2013-08-16 09:02:22,553 info org.apache.hadoop.hdfs.server.namenode.fseditlog: closing edit log: position=4, editlog=/home/rlk/hduser/dfs/name/current/edits 2013-08-16 09:02:22,553 info org.apache.hadoop.hdfs.server.namenode.fseditlog: close success: truncate 4, editlog=/home/rlk/hduser/dfs/name/current/edits 2013-08-16 09:02:22,933 info org.apache.hadoop.hdfs.server.namenode.namecache: initialized 0 entries 0 lookups 2013-08-16 09:02:22,933 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: finished loading fsimage in 776 msecs 2013-08-16 09:02:22,935 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: dfs.safemode.threshold.pct          = 0.9990000128746033 2013-08-16 09:02:22,935 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: dfs.namenode.safemode.min.datanodes = 0 2013-08-16 09:02:22,935 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: dfs.safemode.extension              = 30000 2013-08-16 09:02:22,935 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: number of blocks excluded safe block count: 0 total blocks: 0 , safe blocks: 0 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: total number of blocks = 0 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: number of invalid blocks = 0 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: number of under-replicated blocks = 0 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: number of  over-replicated blocks = 0 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.statechange: state* safe mode termination scan invalid, over- , under-replicated blocks completed in 21 msec 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.statechange: state* leaving safe mode after 0 secs 2013-08-16 09:02:22,956 info org.apache.hadoop.hdfs.statechange: state* network topology has 0 racks , 0 datanodes 2013-08-16 09:02:22,962 info org.apache.hadoop.hdfs.statechange: state* underreplicatedblocks has 0 blocks 2013-08-16 09:02:22,972 info org.apache.hadoop.util.hostsfilereader: refreshing hosts (include/exclude) list 2013-08-16 09:02:22,974 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: replicatequeue queueprocessingstatistics: first cycle completed 0 blocks in 1 msec 2013-08-16 09:02:22,974 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: replicatequeue queueprocessingstatistics: queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles 2013-08-16 09:02:22,974 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: invalidatequeue queueprocessingstatistics: first cycle completed 0 blocks in 0 msec 2013-08-16 09:02:22,974 info org.apache.hadoop.hdfs.server.namenode.fsnamesystem: invalidatequeue queueprocessingstatistics: queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles 2013-08-16 09:02:22,983 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source fsnamesystemmetrics registered. 2013-08-16 09:02:23,026 info org.apache.hadoop.ipc.server: starting socketreader 2013-08-16 09:02:23,029 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source rpcdetailedactivityforport8020 registered. 2013-08-16 09:02:23,030 info org.apache.hadoop.metrics2.impl.metricssourceadapter: mbean source rpcactivityforport8020 registered. 2013-08-16 09:02:23,037 info org.apache.hadoop.hdfs.server.namenode.namenode: namenode at: localhost.localdomain/127.0.0.1:8020 2013-08-16 09:02:23,195 info org.mortbay.log: logging org.slf4j.impl.log4jloggeradapter(org.mortbay.log) via org.mortbay.log.slf4jlog 2013-08-16 09:02:23,306 info org.apache.hadoop.http.httpserver: added global filtersafety (class=org.apache.hadoop.http.httpserver$quotinginputfilter) 2013-08-16 09:02:23,318 info org.apache.hadoop.http.httpserver: dfs.webhdfs.enabled = false 2013-08-16 09:02:23,329 info org.apache.hadoop.http.httpserver: port returned webserver.getconnectors()[0].getlocalport() before open() -1. opening listener on 50070 2013-08-16 09:02:23,331 info org.apache.hadoop.http.httpserver: listener.getlocalport() returned 50070 webserver.getconnectors()[0].getlocalport() returned 50070 2013-08-16 09:02:23,331 info org.apache.hadoop.http.httpserver: jetty bound port 50070 2013-08-16 09:02:23,331 info org.mortbay.log: jetty-6.1.26 2013-08-16 09:02:23,386 info org.mortbay.log: extract jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/lib/ext/hadoop-core-1.2.1.jar!/webapps/hdfs /tmp/jetty_0_0_0_0_50070_hdfs____w2cu08/webapp 2013-08-16 09:02:25,171 warn org.mortbay.log: failed jsp: java.lang.noclassdeffounderror: javax/servlet/jsp/jspfactory 2013-08-16 09:02:25,215 warn org.mortbay.log: failed org.mortbay.jetty.webapp.webappcontext@12305d34{/,jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/lib/ext/hadoop-core-1.2.1.jar!/webapps/hdfs}: java.lang.noclassdeffounderror: javax/servlet/jsp/jspfactory 2013-08-16 09:02:25,225 warn org.mortbay.log: failed contexthandlercollection@25370a40: java.lang.noclassdeffounderror: javax/servlet/jsp/jspfactory 2013-08-16 09:02:25,226 error org.mortbay.log: error starting handlers java.lang.noclassdeffounderror: javax/servlet/jsp/jspfactory     @ org.apache.jasper.servlet.jspservlet.init(jspservlet.java:99)     @ org.mortbay.jetty.servlet.servletholder.initservlet(servletholder.java:440)     @ org.mortbay.jetty.servlet.servletholder.dostart(servletholder.java:263)     @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50)     @ org.mortbay.jetty.servlet.servlethandler.initialize(servlethandler.java:736)     @ org.mortbay.jetty.servlet.context.startcontext(context.java:140)     @ org.mortbay.jetty.webapp.webappcontext.startcontext(webappcontext.java:1282)     @ org.mortbay.jetty.handler.contexthandler.dostart(contexthandler.java:518)     @ org.mortbay.jetty.webapp.webappcontext.dostart(webappcontext.java:499)     @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50)     @ org.mortbay.jetty.handler.handlercollection.dostart(handlercollection.java:152)     @ org.mortbay.jetty.handler.contexthandlercollection.dostart(contexthandlercollection.java:156)     @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50)     @ org.mortbay.jetty.handler.handlerwrapper.dostart(handlerwrapper.java:130)     @ org.mortbay.jetty.server.dostart(server.java:224)     @ org.mortbay.component.abstractlifecycle.start(abstractlifecycle.java:50)     @ org.apache.hadoop.http.httpserver.start(httpserver.java:638)     @ org.apache.hadoop.hdfs.server.namenode.namenode$1.run(namenode.java:517)     @ org.apache.hadoop.hdfs.server.namenode.namenode$1.run(namenode.java:395)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1190)     @ org.apache.hadoop.hdfs.server.namenode.namenode.starthttpserver(namenode.java:395)     @ org.apache.hadoop.hdfs.server.namenode.namenode.initialize(namenode.java:337)     @ org.apache.hadoop.hdfs.server.namenode.namenode.<init>(namenode.java:569)     @ org.apache.hadoop.hdfs.server.namenode.namenode.createnamenode(namenode.java:1479)     @ org.apache.hadoop.hdfs.server.namenode.namenode.main(namenode.java:1488) caused by: java.lang.classnotfoundexception: javax.servlet.jsp.jspfactory     @ java.net.urlclassloader$1.run(urlclassloader.java:366)     @ java.net.urlclassloader$1.run(urlclassloader.java:355)     @ java.security.accesscontroller.doprivileged(native method)     @ java.net.urlclassloader.findclass(urlclassloader.java:354)     @ java.lang.classloader.loadclass(classloader.java:424)     @ java.lang.classloader.loadclass(classloader.java:357)     ... 27 more 2013-08-16 09:02:25,307 info org.mortbay.log: started selectchannelconnector@0.0.0.0:50070 2013-08-16 09:02:25,307 error org.apache.hadoop.security.usergroupinformation: priviledgedactionexception as:rlk cause:java.io.ioexception: problem in starting http server. server handlers failed 2013-08-16 09:02:25,308 info org.mortbay.log: stopped selectchannelconnector@0.0.0.0:50070 2013-08-16 09:02:25,308 error org.mortbay.log: exception  java.lang.nullpointerexception     @ org.apache.jasper.servlet.jspservlet.destroy(jspservlet.java:282)     @ org.mortbay.jetty.servlet.servletholder.destroyinstance(servletholder.java:318)     @ org.mortbay.jetty.servlet.servletholder.dostop(servletholder.java:289)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.servlet.servlethandler.dostop(servlethandler.java:185)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.handler.handlerwrapper.dostop(handlerwrapper.java:142)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.handler.handlerwrapper.dostop(handlerwrapper.java:142)     @ org.mortbay.jetty.servlet.sessionhandler.dostop(sessionhandler.java:125)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.handler.handlerwrapper.dostop(handlerwrapper.java:142)     @ org.mortbay.jetty.handler.contexthandler.dostop(contexthandler.java:592)     @ org.mortbay.jetty.webapp.webappcontext.dostop(webappcontext.java:537)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.handler.handlercollection.dostop(handlercollection.java:169)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.mortbay.jetty.handler.handlerwrapper.dostop(handlerwrapper.java:142)     @ org.mortbay.jetty.server.dostop(server.java:283)     @ org.mortbay.component.abstractlifecycle.stop(abstractlifecycle.java:76)     @ org.apache.hadoop.http.httpserver.stop(httpserver.java:688)     @ org.apache.hadoop.hdfs.server.namenode.namenode.stop(namenode.java:604)     @ org.apache.hadoop.hdfs.server.namenode.namenode.<init>(namenode.java:571)     @ org.apache.hadoop.hdfs.server.namenode.namenode.createnamenode(namenode.java:1479)     @ org.apache.hadoop.hdfs.server.namenode.namenode.main(namenode.java:1488) 2013-08-16 09:02:25,336 warn org.apache.hadoop.hdfs.server.namenode.fsnamesystem: replicationmonitor thread received interruptedexceptionjava.lang.interruptedexception: sleep interrupted 2013-08-16 09:02:25,337 info org.apache.hadoop.hdfs.server.namenode.decommissionmanager: interrupted monitor java.lang.interruptedexception: sleep interrupted     @ java.lang.thread.sleep(native method)     @ org.apache.hadoop.hdfs.server.namenode.decommissionmanager$monitor.run(decommissionmanager.java:65)     @ java.lang.thread.run(thread.java:724) 2013-08-16 09:02:25,339 info org.apache.hadoop.hdfs.server.namenode.fseditlog: number of transactions: 0 total time transactions(ms): 0 number of transactions batched in syncs: 0 number of syncs: 0 synctimes(ms): 0  2013-08-16 09:02:25,375 info org.apache.hadoop.hdfs.server.namenode.fseditlog: closing edit log: position=4, editlog=/home/rlk/hduser/dfs/name/current/edits 2013-08-16 09:02:25,375 info org.apache.hadoop.hdfs.server.namenode.fseditlog: close success: truncate 4, editlog=/home/rlk/hduser/dfs/name/current/edits 2013-08-16 09:02:25,403 info org.apache.hadoop.ipc.server: stopping server on 8020 2013-08-16 09:02:25,411 info org.apache.hadoop.ipc.metrics.rpcinstrumentation: shut down 2013-08-16 09:02:25,412 error org.apache.hadoop.hdfs.server.namenode.namenode: java.io.ioexception: problem in starting http server. server handlers failed     @ org.apache.hadoop.http.httpserver.start(httpserver.java:662)     @ org.apache.hadoop.hdfs.server.namenode.namenode$1.run(namenode.java:517)     @ org.apache.hadoop.hdfs.server.namenode.namenode$1.run(namenode.java:395)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1190)     @ org.apache.hadoop.hdfs.server.namenode.namenode.starthttpserver(namenode.java:395)     @ org.apache.hadoop.hdfs.server.namenode.namenode.initialize(namenode.java:337)     @ org.apache.hadoop.hdfs.server.namenode.namenode.<init>(namenode.java:569)     @ org.apache.hadoop.hdfs.server.namenode.namenode.createnamenode(namenode.java:1479)     @ org.apache.hadoop.hdfs.server.namenode.namenode.main(namenode.java:1488)  2013-08-16 09:02:25,413 info org.apache.hadoop.hdfs.server.namenode.namenode: shutdown_msg:  /************************************************************ shutdown_msg: shutting down namenode @ localhost.lan/192.168.1.94 ************************************************************/ 

i give hadoop configuration (i'm using hadoop-1.2.1) :

core-site.xml :

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- core-site.xml --> <configuration>   <property>     <name>hadoop.tmp.dir</name>     <value>/home/rlk/hduser</value>   </property>   <property>     <name>fs.default.name</name>     <value>hdfs://localhost/</value>   </property> </configuration> 

hdfs-site.xml :

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- hdfs-site.xml --> <configuration>   <property>     <name>dfs.replication</name>     <value>1</value>   </property> </configuration> 

mapred-site.xml :

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- mapred-site.xml --> <configuration>   <property>     <name>mapred.job.tracker</name>     <value>localhost:8021</value>   </property> </configuration> 

i found solution : jar collisions. had duplicate jar files both in hadoop-x.y.z/ , hadoop-x.y.z/lib , in path-to-java/jre/lib/ext/.

i removed formers , works fine again.