hive--map数量过小导致hive运行缓慢_hive在map阶段特别的慢-程序员宅基地

技术标签： 错误集锦  hive  

map数量过小导致hive运行缓慢

  • 网页显示
    在这里插入图片描述
  • 日志
    • Not uberizing job_1607937945101_247633 because: not enabled; too much RAM;
    • nput size for job job_1607937945101_246670 = 38627311. Number of splits = 1
    • Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0这条一直出现。。。。。。。
  • 思路
    • 数据量过大,map数过少
    • 调整 参数
      • set mapred.max.split.size=25000000;
      • set mapred.min.split.size.per.node=25000000;
      • set mapred.min.split.size.per.rack=25000000;
    • 再次查看日志 Number of splits = 4，速度大大加快
    • 如果不行（则应该是压缩不可切分造成）
  • 完整日志可以参考下面

Logged in as: yarn
Application
About
Jobs
Tools
Log Type: syslog

Log Upload Time: 星期一 三月 01 18:18:41 +0800 2021

Log Length: 169813

2021-03-01 16:57:51,172 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1607937945101_246670_000001
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33bc72d1)
2021-03-01 16:57:51,604 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:51,606 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:52,081 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-01 16:57:52,215 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2021-03-01 16:57:52,222 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2021-03-01 16:57:52,257 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,277 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,293 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,306 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2021-03-01 16:57:52,340 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2021-03-01 16:57:52,536 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2021-03-01 16:57:52,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1607937945101_246670 to jobTokenSecretManager
2021-03-01 16:57:52,709 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1607937945101_246670 because: not enabled; too much RAM;
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1607937945101_246670 = 38627311. Number of splits = 1
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1607937945101_246670 = 0
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from NEW to INITED
2021-03-01 16:57:52,730 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1607937945101_246670.
2021-03-01 16:57:52,753 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2021-03-01 16:57:52,762 INFO [Socket Reader #1 for port 36866] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36866
2021-03-01 16:57:52,793 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2021-03-01 16:57:52,826 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: IPC Server listener on 36866: starting
2021-03-01 16:57:52,826 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:52,828 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdhuatdn62-209.com/10.9.62.209:36866
2021-03-01 16:57:52,883 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2021-03-01 16:57:52,889 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2021-03-01 16:57:52,893 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2021-03-01 16:57:52,902 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2021-03-01 16:57:52,941 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 46085
2021-03-01 16:57:52,941 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2021-03-01 16:57:52,964 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/hadoop-yarn-common-2.6.0-cdh5.13.3.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_46085_mapreduce____.frd8sy/webapp
2021-03-01 16:57:53,196 INFO [main] org.mortbay.log: Started [email protected]:46085
2021-03-01 16:57:53,196 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 46085
2021-03-01 16:57:53,435 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2021-03-01 16:57:53,439 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1607937945101_246670
2021-03-01 16:57:53,440 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2021-03-01 16:57:53,441 INFO [Socket Reader #1 for port 43229] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43229
2021-03-01 16:57:53,481 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:53,481 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: IPC Server listener on 43229: starting
2021-03-01 16:57:53,501 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2021-03-01 16:57:53,564 INFO [main] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider: Failing over to rm709
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:131072, vCores:32>
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.w_hanzhiying
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2021-03-01 16:57:53,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from INITED to SETUP
2021-03-01 16:57:53,615 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2021-03-01 16:57:53,617 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from SETUP to RUNNING
2021-03-01 16:57:53,752 INFO [Socket Reader #1 for port 36866] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hive (auth:SIMPLE) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
2021-03-01 16:57:53,772 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1607937945101_246670, File: hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist
2021-03-01 16:57:53,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from NEW to SCHEDULED
2021-03-01 16:57:53,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2021-03-01 16:57:53,793 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1>
2021-03-01 16:57:53,904 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP   job_1607937945101_246670. Report-size will be 1
2021-03-01 16:57:53,964 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE   job_1607937945101_246670. Report-size will be 0
2021-03-01 16:57:53,983 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:54,603 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2021-03-01 16:57:54,632 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:179712, vCores:199> knownNMs=7
2021-03-01 16:57:55,642 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2021-03-01 16:57:55,671 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e64_1607937945101_246670_01_000002 to attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,673 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 16:57:55,743 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://lhService01/user/hive/.staging/job_1607937945101_246670/job.jar
2021-03-01 16:57:55,746 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1607937945101_246670/job.xml
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2021-03-01 16:57:55,784 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx1639m
2021-03-01 16:57:55,786 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2021-03-01 16:57:55,791 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e64_1607937945101_246670_01_000002 taskAttempt attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,794 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,855 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1607937945101_246670_m_000000_0 : 13562
2021-03-01 16:57:55,858 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1607937945101_246670_m_000000_0] using containerId: [container_e64_1607937945101_246670_01_000002 on NM: [cdhuatdn62-209.com:8041]
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from SCHEDULED to RUNNING
2021-03-01 16:57:56,675 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:177664, vCores:194> knownNMs=7
2021-03-01 16:57:57,308 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1607937945101_246670 (auth:SIMPLE)
2021-03-01 16:57:57,317 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1607937945101_246670 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1607937945101_246670_m_70368744177666 asked for a task
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1607937945101_246670_m_70368744177666 given task: attempt_1607937945101_246670_m_000000_0
2021-03-01 16:58:10,060 INFO [IPC Server handler 6 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 16:58:16,148 INFO [IPC Server handler 12 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0


此处省略

org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 18:18:33,300 INFO [IPC Server handler 15 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 1.0
2021-03-01 18:18:33,305 INFO [IPC Server handler 26 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,309 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER
2021-03-01 18:18:33,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2021-03-01 18:18:33,351 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2021-03-01 18:18:33,352 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from RUNNING to COMMITTING
2021-03-01 18:18:33,362 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2021-03-01 18:18:33,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
2021-03-01 18:18:33,406 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from COMMITTING to SUCCEEDED
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2021-03-01 18:18:33,426 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2021-03-01 18:18:33,794 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:34,090 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,215 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,222 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1_conf.xml to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,307 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,317 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary
2021-03-01 18:18:34,319 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml
2021-03-01 18:18:34,321 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist
2021-03-01 18:18:34,321 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2021-03-01 18:18:34,322 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:34,359 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2021-03-01 18:18:34,364 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
2021-03-01 18:18:34,365 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://cdhuatmaster62-201.com:19888/jobhistory/job/job_1607937945101_246670
2021-03-01 18:18:34,374 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2021-03-01 18:18:35,375 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:35,376 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://lhService01 /user/hive/.staging/job_1607937945101_246670
2021-03-01 18:18:35,388 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 43229
2021-03-01 18:18:35,392 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 43229
2021-03-01 18:18:35,392 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:35,394 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2021-03-01 18:18:35,396 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2021-03-01 18:18:40,394 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 36866
2021-03-01 18:18:40,394 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 36866
2021-03-01 18:18:40,394 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:40,396 INFO [Thread-298] org.mortbay.log: Stopped [email protected]:0
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_46893497/article/details/113846400

智能推荐

分布式光纤传感器的全球与中国市场2022-2028年：技术、参与者、趋势、市场规模及占有率研究报告_预计2026年中国分布式传感器市场规模有多大-程序员宅基地

文章浏览阅读3.2k次。本文研究全球与中国市场分布式光纤传感器的发展现状及未来发展趋势，分别从生产和消费的角度分析分布式光纤传感器的主要生产地区、主要消费地区以及主要的生产商。重点分析全球与中国市场的主要厂商产品特点、产品规格、不同规格产品的价格、产量、产值及全球和中国市场主要生产商的市场份额。主要生产商包括：FISO TechnologiesBrugg KabelSensor HighwayOmnisensAFL GlobalQinetiQ GroupLockheed MartinOSENSA Innovati_预计2026年中国分布式传感器市场规模有多大

07_08 常用组合逻辑电路结构——为IC设计的延时估计铺垫_基4布斯算法代码-程序员宅基地

文章浏览阅读1.1k次，点赞2次，收藏12次。常用组合逻辑电路结构——为IC设计的延时估计铺垫学习目的：估计模块间的delay，确保写的代码的timing 综合能给到多少HZ，以满足需求！_基4布斯算法代码

OpenAI Manager助手（基于SpringBoot和Vue）_chatgpt网页版-程序员宅基地

文章浏览阅读3.3k次，点赞3次，收藏5次。OpenAI Manager助手（基于SpringBoot和Vue）_chatgpt网页版

关于美国计算机奥赛USACO，你想知道的都在这_usaco可以多次提交吗-程序员宅基地

文章浏览阅读2.2k次。USACO自1992年举办，到目前为止已经举办了27届，目的是为了帮助美国信息学国家队选拔IOI的队员，目前逐渐发展为全球热门的线上赛事，成为美国大学申请条件下，含金量相当高的官方竞赛。USACO的比赛成绩可以助力计算机专业留学，越来越多的学生进入了康奈尔，麻省理工，普林斯顿，哈佛和耶鲁等大学，这些同学的共同点是他们都参加了美国计算机科学竞赛（USACO)，并且取得过非常好的成绩。适合参赛人群USACO适合国内在读学生有意向申请美国大学的或者想锻炼自己编程能力的同学，高三学生也可以参加12月的第_usaco可以多次提交吗

MySQL存储过程和自定义函数_mysql自定义函数和存储过程-程序员宅基地

文章浏览阅读394次。1.1 存储程序1.2 创建存储过程1.3 创建自定义函数1.3.1 示例1.4 自定义函数和存储过程的区别1.5 变量的使用1.6 定义条件和处理程序1.6.1 定义条件1.6.1.1 示例1.6.2 定义处理程序1.6.2.1 示例1.7 光标的使用1.7.1 声明光标1.7.2 打开光标1.7.3 使用光标1.7.4 关闭光标1.8 流程控制的使用1.8.1 IF语句1.8.2 CASE语句1.8.3 LOOP语句1.8.4 LEAVE语句1.8.5 ITERATE语句1.8.6 REPEAT语句。_mysql自定义函数和存储过程

半导体基础知识与PN结_本征半导体电流为0-程序员宅基地

文章浏览阅读188次。半导体二极管——集成电路最小组成单元。_本征半导体电流为0

随便推点

【Unity3d Shader】水面和岩浆效果_unity 岩浆shader-程序员宅基地

文章浏览阅读2.8k次，点赞3次，收藏18次。游戏水面特效实现方式太多。咱们这边介绍的是一最简单的UV动画（无顶点位移），整个mesh由4个顶点构成。实现了水面效果（左图），不动代码稍微修改下参数和贴图可以实现岩浆效果（右图）。有要思路是1，uv按时间去做正弦波移动2，在1的基础上加个凹凸图混合uv3，在1、2的基础上加个水流方向4，加上对雾效的支持，如没必要请自行删除雾效代码(把包含fog的几行代码删除)S..._unity 岩浆shader

广义线性模型——Logistic回归模型（1）_广义线性回归模型-程序员宅基地

文章浏览阅读5k次。广义线性模型是线性模型的扩展，它通过连接函数建立响应变量的数学期望值与线性组合的预测变量之间的关系。广义线性模型拟合的形式为：其中g(μY)是条件均值的函数（称为连接函数）。另外，你可放松Y为正态分布的假设，改为Y 服从指数分布族中的一种分布即可。设定好连接函数和概率分布后，便可以通过最大似然估计的多次迭代推导出各参数值。在大部分情况下，线性模型就可以通过一系列连续型或类别型预测变量来预测正态分布的响应变量的工作。但是，有时候我们要进行非正态因变量的分析，例如：（1）类别型.._广义线性回归模型

HTML+CSS大作业 环境网页设计与实现(垃圾分类) web前端开发技术 web课程设计 网页规划与设计_垃圾分类网页设计目标怎么写-程序员宅基地

文章浏览阅读69次。环境保护、 保护地球、 校园环保、垃圾分类、绿色家园、等网站的设计与制作。 总结了一些学生网页制作的经验：一般的网页需要融入以下知识点：div+css布局、浮动、定位、高级css、表格、表单及验证、js轮播图、音频 视频 Flash的应用、ul li、下拉导航栏、鼠标划过效果等知识点，网页的风格主题也很全面：如爱好、风景、校园、美食、动漫、游戏、咖啡、音乐、家乡、电影、名人、商城以及个人主页等主题，学生、新手可参考下方页面的布局和设计和HTML源码（有用点赞△） 一套A+的网_垃圾分类网页设计目标怎么写

C# .Net 发布后,把dll全部放在一个文件夹中,让软件目录更整洁_.net dll 全局目录-程序员宅基地

文章浏览阅读614次，点赞7次，收藏11次。之前找到一个修改 exe 中 DLL地址 的方法, 不太好使,虽然能正确启动, 但无法改变 exe 的工作目录,这就影响了.Net 中很多获取 exe 执行目录来拼接的地址 ( 相对路径 ),比如 wwwroot 和 代码中相对目录还有一些复制到目录的普通文件 等等,它们的地址都会指向原来 exe 的目录, 而不是自定义的 “lib” 目录,根本原因就是没有修改 exe 的工作目录这次来搞一个启动程序,把 .net 的所有东西都放在一个文件夹,在文件夹同级的目录制作一个 exe._.net dll 全局目录

BRIEF特征点描述算法_breif description calculation 特征点-程序员宅基地

文章浏览阅读1.5k次。本文为转载，原博客地址：http://blog.csdn.net/hujingshuang/article/details/46910259简介 BRIEF是2010年的一篇名为《BRIEF:Binary Robust Independent Elementary Features》的文章中提出，BRIEF是对已检测到的特征点进行描述，它是一种二进制编码的描述子，摈弃了利用区域灰度..._breif description calculation 特征点

房屋租赁管理系统的设计和实现，SpringBoot计算机毕业设计论文_基于spring boot的房屋租赁系统论文-程序员宅基地

文章浏览阅读4.1k次，点赞21次，收藏79次。本文是《基于SpringBoot的房屋租赁管理系统》的配套原创说明文档，可以给应届毕业生提供格式撰写参考，也可以给开发类似系统的朋友们提供功能业务设计思路。_基于spring boot的房屋租赁系统论文

推荐文章

热门文章

相关标签