拓冰网站定制 
1、记录一下错误修改配置后重新格式化namenode后机器上的datanode没有启动起来。2026-10-04 21:08:56,865 WARN hdfs.DataStreamer: DataStreamer Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/QuasiMonteCarlo_1791119335675_932816193/in/part0 could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation.提醒要删除hadoop下的tmpData目录名称根据实际情况变动后重新格式化namenode。2、mapred的一些配置2026-10-04 21:17:47,500 INFO mapreduce.Job: Job job_1791119580394_0001 failed with state FAILED due to: Application application_1791119580394_0001 failed 2 times due to AM Container for appattempt_1791119580394_0001_000002 exited with exitCode: 1Failing this attempt.Diagnostics: [2026-10-04 21:17:47.305]Exception from container-launch.Container id: container_1791119580394_0001_02_000001Exit code: 1[2026-10-04 21:17:47.309]Container exited with a non-zero exit code 1. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster[2026-10-04 21:17:47.311]Container exited with a non-zero exit code 1. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMasterFor more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1791119580394_0001 Then click on links to logs of each attempt.. Failing the application.2026-10-04 21:17:47,524 INFO mapreduce.Job: Counters: 0Job job_1791119580394_0001 failed!解决方法打开mapred-site.xml文件添加下面代码propertynameyarn.app.mapreduce.am.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/propertypropertynamemapreduce.map.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/propertypropertynamemapreduce.reduce.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/property3、mapreduce_shuffle要配置。2026-10-04 21:43:50,739 INFO mapreduce.Job: Task Id : attempt_1791121357322_0001_m_000000_2, Status : FAILEDContainer launch failed for container_1791121357322_0001_01_000019 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not existat sun.reflect.GeneratedConstructorAccessor48.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:163)at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:394)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)42026-10-04 21:53:03,023 INFO mapreduce.Job: Task Id : attempt_1791121853079_0001_m_000000_2, Status : FAILED[2026-10-04 21:53:00.316]Container [pid27514,containerIDcontainer_1791121853079_0001_01_000069] is running 437209600B beyond the VIRTUAL memory limit. Current usage: 62.2 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1791121853079_0001_01_000069 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|- 27514 27512 27514 27514 (bash) 0 0 115834880 308 /bin/bash -c /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stacktrue -Dhadoop.metrics.log.levelWARN -Xmx820m -Djava.io.tmpdir/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configurationcontainer-log4j.properties -Dyarn.app.container.log.dir/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize0 -Dhadoop.root.loggerINFO,CLA -Dhadoop.root.logfilesyslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69 1/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stdout 2/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stderr|- 27528 27514 27514 27514 (java) 84 4 2576232448 15611 /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stacktrue -Dhadoop.metrics.log.levelWARN -Xmx820m -Djava.io.tmpdir/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configurationcontainer-log4j.properties -Dyarn.app.container.log.dir/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize0 -Dhadoop.root.loggerINFO,CLA -Dhadoop.root.logfilesyslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69[2026-10-04 21:53:00.327]Container killed on request. Exit code is 143[2026-10-04 21:53:00.755]Container exited with a non-zero exit code 143.配置yarn-site.xml中添加下面内容propertynameyarn.nodemanager.resource.memory-mb/namevalue22528/valuediscription每个节点可用内存,单位MB/discription/propertypropertynameyarn.scheduler.minimum-allocation-mb/namevalue1500/valuediscription单个任务可申请最少内存默认1024MB/discription/propertypropertynameyarn.scheduler.maximum-allocation-mb/namevalue16384/valuediscription单个任务可申请最大内存默认8192MB/discription/propertymapred-site.xmlpropertynamemapreduce.map.memory.mb/namevalue1500/valuedescription每个Map任务的物理内存限制/description/propertypropertynamemapreduce.reduce.memory.mb/namevalue3000/valuedescription每个Reduce任务的物理内存限制/description/propertypropertynamemapreduce.map.java.opts/namevalue-Xmx1200m/value/propertypropertynamemapreduce.reduce.java.opts/namevalue-Xmx2600m/value/propertypropertynamemapreduce.framework.name/namevalueyarn/value/property参考csdn:https://blog.csdn.net/qq_40059532/article/details/103493928/ 几个hadoop中运行mapreduce的错误记录一下。
1、记录一下错误修改配置后重新格式化namenode后机器上的datanode没有启动起来。2026-10-04 21:08:56,865 WARN hdfs.DataStreamer: DataStreamer Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/QuasiMonteCarlo_1791119335675_932816193/in/part0 could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation.提醒要删除hadoop下的tmpData目录名称根据实际情况变动后重新格式化namenode。2、mapred的一些配置2026-10-04 21:17:47,500 INFO mapreduce.Job: Job job_1791119580394_0001 failed with state FAILED due to: Application application_1791119580394_0001 failed 2 times due to AM Container for appattempt_1791119580394_0001_000002 exited with exitCode: 1Failing this attempt.Diagnostics: [2026-10-04 21:17:47.305]Exception from container-launch.Container id: container_1791119580394_0001_02_000001Exit code: 1[2026-10-04 21:17:47.309]Container exited with a non-zero exit code 1. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster[2026-10-04 21:17:47.311]Container exited with a non-zero exit code 1. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMasterFor more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1791119580394_0001 Then click on links to logs of each attempt.. Failing the application.2026-10-04 21:17:47,524 INFO mapreduce.Job: Counters: 0Job job_1791119580394_0001 failed!解决方法打开mapred-site.xml文件添加下面代码propertynameyarn.app.mapreduce.am.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/propertypropertynamemapreduce.map.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/propertypropertynamemapreduce.reduce.env/namevalueHADOOP_MAPRED_HOME${HADOOP_HOME}/value/property3、mapreduce_shuffle要配置。2026-10-04 21:43:50,739 INFO mapreduce.Job: Task Id : attempt_1791121357322_0001_m_000000_2, Status : FAILEDContainer launch failed for container_1791121357322_0001_01_000019 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not existat sun.reflect.GeneratedConstructorAccessor48.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:163)at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:394)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)42026-10-04 21:53:03,023 INFO mapreduce.Job: Task Id : attempt_1791121853079_0001_m_000000_2, Status : FAILED[2026-10-04 21:53:00.316]Container [pid27514,containerIDcontainer_1791121853079_0001_01_000069] is running 437209600B beyond the VIRTUAL memory limit. Current usage: 62.2 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1791121853079_0001_01_000069 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|- 27514 27512 27514 27514 (bash) 0 0 115834880 308 /bin/bash -c /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stacktrue -Dhadoop.metrics.log.levelWARN -Xmx820m -Djava.io.tmpdir/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configurationcontainer-log4j.properties -Dyarn.app.container.log.dir/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize0 -Dhadoop.root.loggerINFO,CLA -Dhadoop.root.logfilesyslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69 1/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stdout 2/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stderr|- 27528 27514 27514 27514 (java) 84 4 2576232448 15611 /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stacktrue -Dhadoop.metrics.log.levelWARN -Xmx820m -Djava.io.tmpdir/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configurationcontainer-log4j.properties -Dyarn.app.container.log.dir/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize0 -Dhadoop.root.loggerINFO,CLA -Dhadoop.root.logfilesyslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69[2026-10-04 21:53:00.327]Container killed on request. Exit code is 143[2026-10-04 21:53:00.755]Container exited with a non-zero exit code 143.配置yarn-site.xml中添加下面内容propertynameyarn.nodemanager.resource.memory-mb/namevalue22528/valuediscription每个节点可用内存,单位MB/discription/propertypropertynameyarn.scheduler.minimum-allocation-mb/namevalue1500/valuediscription单个任务可申请最少内存默认1024MB/discription/propertypropertynameyarn.scheduler.maximum-allocation-mb/namevalue16384/valuediscription单个任务可申请最大内存默认8192MB/discription/propertymapred-site.xmlpropertynamemapreduce.map.memory.mb/namevalue1500/valuedescription每个Map任务的物理内存限制/description/propertypropertynamemapreduce.reduce.memory.mb/namevalue3000/valuedescription每个Reduce任务的物理内存限制/description/propertypropertynamemapreduce.map.java.opts/namevalue-Xmx1200m/value/propertypropertynamemapreduce.reduce.java.opts/namevalue-Xmx2600m/value/propertypropertynamemapreduce.framework.name/namevalueyarn/value/property参考csdn:https://blog.csdn.net/qq_40059532/article/details/103493928/