Recent advances in reinforcement learning (RL) and multi-agent reinforcement learning (MARL) have led to a great deal of progress in creating artificial agents which can cooperate to solve tasks: DeepMind’s AlphaStar surpassed professional-level performance in the StarCraft II [35], OpenAI Five defeated the world-champion in Dota II [4], and OpenAI demonstrated the emergence of human-like tool-use agent behaviors via multi-agent learning [2]. These notable successes were driven largely by on-policy RL algorithms such as IMPALA [10] and PPO [30, 4] which were often coupled with distributed training systems to utilize massive amounts of parallelism and compute. In the aforementioned works, tens of thousands of CPU cores and hundreds of GPUs were utilized to collect and train on an extraordinary volume of training samples. This is in contrast to recent academic progress and literature in MARL which has largely focused developing off-policy learning frameworks such as MADDPG [22] and value-decomposed Q-learning [32, 27]; methods in these frameworks have yielded state-of-the-art results on a wide range of multi-agent benchmarks [36, 37].
        强化学习 （RL） 和多智能体强化学习 （MARL） 的最新进展导致在创建可以协作解决任务的人工智能体方面取得了很大进展：DeepMind 的 AlphaStar 在《星际争霸 II》中超越了专业级性能 [ 35]，OpenAI Five 在 Dota II 中击败了世界冠军 [ 4]，OpenAI 通过多智能体学习展示了类人工具使用智能体行为的出现 [ 2]。这些显著的成功很大程度上是由策略RL算法推动的，如IMPALA [ 10]和PPO [ 30， 4]，这些算法通常与分布式训练系统相结合，以利用大量的并行性和计算。在上述工作中，数以万计的 CPU 内核和数百个 GPU 被用于收集和训练大量训练样本。这与MARL最近的学术进展和文献形成鲜明对比，后者主要侧重于开发非政策学习框架，如MADDPG [ 22]和价值分解Q学习 [ 32， 27];这些框架中的方法在广泛的多智能体基准上产生了最先进的结果[36,37]。

        In this work, we revisit the use of Proximal Policy Optimization (PPO) – an on-policy algorithm1 popular in single-agent RL but under-utilized in recent MARL literature – in multi-agent settings. We hypothesize that the relative lack of PPO in multi-agent settings can be attributed to two related factors: first, the belief that PPO is less sample-efficient than off-policy methods and is correspondingly less useful in resource-constrained settings, and second, the fact that common implementation and hyperparameter tuning practices when using PPO in single-agent settings often do not yield strong performance when transferred to multi-agent settings.
        在这项工作中，我们重新审视了近端策略优化 （PPO） 的使用——一种策略算法 1。在单 RL 中很受欢迎，但在最近的 MARL 文献中未得到充分利用——在多药环境中。我们假设，在多智能体环境中相对缺乏 PPO 可归因于两个相关因素：首先，认为 PPO 的采样效率低于非策略方法，并且在资源受限的环境中相应地不太有用，其次，在单智能体环境中使用 PPO 时的常见实现和超参数调整实践在转移到多智能体设置时通常不会产生强大的性能。

        We conduct a comprehensive empirical study to examine the performance of PPO on four popular cooperative multi-agent benchmarks: the multi-agent particle world environments (MPE) [22], the StarCraft multi-agent challenge (SMAC)  [28], Google Research Football (GRF) [19] and the Hanabi challenge [3]. We first show that when compared to off-policy baselines, PPO achieves strong task performance and competitive sample-efficiency. We then identify five implementation factors and hyperparameters which are particularly important for PPO’s performance, offer concrete suggestions about these configuring factors, and provide intuition as to why these suggestions hold.
        我们进行了一项全面的实证研究，以检验PPO在四个流行的合作多智能体基准上的性能：多智能体粒子世界环境（MPE）[22]、星际争霸多智能体挑战（SMAC）[28]、Google Research Football（GRF）[19]和Hanabi挑战[3]。我们首先表明，与非政策基线相比，PPO实现了强大的任务绩效和有竞争力的样本效率。然后，我们确定了对 PPO 性能特别重要的五个实现因素和超参数，就这些配置因素提供了具体建议，并提供了这些建议成立原因的直觉。

        Our aim in this work is not to propose a novel MARL algorithm, but instead to empirically demonstrate that with simple modifications, PPO can achieve strong performance in a wide variety of cooperative multi-agent settings. We additionally believe that our suggestions will assis