【vbers】ibv_poll_cq()|RDMA_work request flushed error-程序员宅基地

技术标签: RDMA  

目录

描述

参数

返回值

例子

常见问题


原文:https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/  (强烈建议去看原文)

描述


ibv_poll_cq()从完成队列(CQ)轮询WC(工作完成),非阻塞函数。

[工作完成] 表示 WQ(工作队列)中的WR(工作请求) 以及与CQ相关联的所有已发布到该工作队列的未发出信号的WR(工作请求)均已完成。

(A Work Completion indicates that a Work Request in a Work Queue, and all of the outstanding unsignaled Work Requests that posted to that Work Queue, associated with the CQ are done.)

任何发送和接收请求 以及以错误结束的发送请求,将在处理结束后生成WC(工作完成)。

(Any Receive Requests, signaled Send Requests and Send Requests that ended with an error will generate a Work Completion after their processing end.)

当[工作请求]结束时,会将[工作完成]添加到与此工作队列相关联的CQ的尾部。 ibv_poll_cq()检查CQ中是否存在[工作完成],并按其输入顺序从CQ的顶部弹出(FIFO)。从CQ弹出[工作完成]后,无法将其返回给CQ。

应该以比产生WC更快的速度消费CQ中的WC,防止CQ队列溢出,如果发生CQ溢出,将触发异步事件IBV_EVENT_CQ_ERR,并且无法再使用CQ。

参数

Name Direction Description
cq in

从ibv_create_cq()返回的CQ

num_entries in

期望从CQ读取的WC数

wc out

从CQ读取的WC数num_entries的数组。(Array of size num_entries of the Work Completions that will be read from the CQ)

结构ibv_wc描述了wc的属性。

struct ibv_wc {
    uint64_t        wr_id;
    enum ibv_wc_status    status;
    enum ibv_wc_opcode    opcode;
    uint32_t        vendor_err;
    uint32_t        byte_len;
    uint32_t        imm_data;
    uint32_t        qp_num;
    uint32_t        src_qp;
    int            wc_flags;
    uint16_t        pkey_index;
    uint16_t        slid;
    uint8_t            sl;
    uint8_t            dlid_path_bits;
};

这是struct ibv_wc的完整说明:

wr_id

与WR对应关联的64 bit值 (The 64 bits value that was associated with the corresponding Work Request)

status

Status of the operation. The value can be one of the following enumerated values and their numeric value:

操作状态。该值可以是下列枚举值及其数字值之一:

  • IBV_WC_SUCCESS (0) - Operation completed successfully: this means that the corresponding Work Request (and all of the unsignaled Work Requests that were posted previous to it) ended and the memory buffers that this Work Request refers to are ready to be (re)used.
  • IBV_WC_LOC_LEN_ERR (1) - Local Length Error: this happens if a Work Request that was posted in a local Send Queue contains a message that is greater than the maximum message size that is supported by the RDMA device port that should send the message or an Atomic operation which its size is different than 8 bytes was sent. This also may happen if a Work Request that was posted in a local Receive Queue isn't big enough for holding the incoming message or if the incoming message size if greater the maximum message size supported by the RDMA device port that received the message.
  • IBV_WC_LOC_QP_OP_ERR (2) - Local QP Operation Error: an internal QP consistency error was detected while processing this Work Request: this happens if a Work Request that was posted in a local Send Queue of a UD QP contains an Address Handle that is associated with a Protection Domain to a QP which is associated with a different Protection Domain or an opcode which isn't supported by the transport type of the QP isn't supported (for example: RDMA Write over a UD QP).
  • IBV_WC_LOC_EEC_OP_ERR (3) - Local EE Context Operation Error: an internal EE Context consistency error was detected while processing this Work Request (unused, since its relevant only to RD QPs or EE Context, which aren’t supported).
  • IBV_WC_LOC_PROT_ERR (4) - Local Protection Error: the locally posted Work Request’s buffers in the scatter/gather list does not reference a Memory Region that is valid for the requested operation.
  • IBV_WC_WR_FLUSH_ERR (5) - Work Request Flushed Error: A Work Request was in process or outstanding when the QP transitioned into the Error State.
  • IBV_WC_MW_BIND_ERR (6) - Memory Window Binding Error: A failure happened when tried to bind a MW to a MR.
  • IBV_WC_BAD_RESP_ERR (7) - Bad Response Error: an unexpected transport layer opcode was returned by the responder. Relevant for RC QPs.
  • IBV_WC_LOC_ACCESS_ERR (8) - Local Access Error: a protection error occurred on a local data buffer during the processing of a RDMA Write with Immediate operation sent from the remote node. Relevant for RC QPs.
  • IBV_WC_REM_INV_REQ_ERR (9) - Remote Invalid Request Error: The responder detected an invalid message on the channel. Possible causes include the operation is not supported by this receive queue (qp_access_flags in remote QP wasn't configured to support this operation), insufficient buffering to receive a new RDMA or Atomic Operation request, or the length specified in a RDMA request is greater than 2^{31} bytes. Relevant for RC QPs.
  • IBV_WC_REM_ACCESS_ERR (10) - Remote Access Error: a protection error occurred on a remote data buffer to be read by an RDMA Read, written by an RDMA Write or accessed by an atomic operation. This error is reported only on RDMA operations or atomic operations. Relevant for RC QPs.
  • IBV_WC_REM_OP_ERR (11) - Remote Operation Error: the operation could not be completed successfully by the responder. Possible causes include a responder QP related error that prevented the responder from completing the request or a malformed WQE on the Receive Queue. Relevant for RC QPs.
  • IBV_WC_RETRY_EXC_ERR (12) - Transport Retry Counter Exceeded: The local transport timeout retry counter was exceeded while trying to send this message. This means that the remote side didn't send any Ack or Nack. If this happens when sending the first message, usually this mean that the connection attributes are wrong or the remote side isn't in a state that it can respond to messages. If this happens after sending the first message, usually it means that the remote QP isn't available anymore. Relevant for RC QPs.
  • IBV_WC_RNR_RETRY_EXC_ERR (13) - RNR Retry Counter Exceeded: The RNR NAK retry count was exceeded. This usually means that the remote side didn't post any WR to its Receive Queue. Relevant for RC QPs.
  • IBV_WC_LOC_RDD_VIOL_ERR (14) - Local RDD Violation Error: The RDD associated with the QP does not match the RDD associated with the EE Context (unused, since its relevant only to RD QPs or EE Context, which aren't supported).
  • IBV_WC_REM_INV_RD_REQ_ERR (15) - Remote Invalid RD Request: The responder detected an invalid incoming RD message. Causes include a Q_Key or RDD violation (unused, since its relevant only to RD QPs or EE Context, which aren't supported)
  • IBV_WC_REM_ABORT_ERR (16) - Remote Aborted Error: For UD or UC QPs associated with a SRQ, the responder aborted the operation.
  • IBV_WC_INV_EECN_ERR (17) - Invalid EE Context Number: An invalid EE Context number was detected (unused, since its relevant only to RD QPs or EE Context, which aren't supported).
  • IBV_WC_INV_EEC_STATE_ERR (18) - Invalid EE Context State Error: Operation is not legal for the specified EE Context state (unused, since its relevant only to RD QPs or EE Context, which aren't supported).
  • IBV_WC_FATAL_ERR (19) - Fatal Error.
  • IBV_WC_RESP_TIMEOUT_ERR (20) - Response Timeout Error.
  • IBV_WC_GENERAL_ERR (21) - General Error: other error which isn't one of the above errors.
opcode The operation that the corresponding Work Request performed. This value controls the way that data was sent, the direction of the data flow and the valid attributes in the Work Completion. The value can be one of the following enumerated values:

相应的工作请求执行的操作。此值控制数据的发送方式、数据流的方向以及“工作完成”中的有效属性。该值可以是下列枚举值之一:

  • IBV_WC_SEND - 已发布到“发送队列”中的WR的发送操作(Send operation for a WR that was posted to the Send Queue) 
  • IBV_WC_RDMA_WRITE - 已发布到发送队列的WR的RDMA写操作(RDMA Write operation for a WR that was posted to the Send Queue)
  • IBV_WC_RDMA_READ - 已发布到发送队列的WR的RDMA读取操作(RDMA Read operation for a WR that was posted to the Send Queue)
  • IBV_WC_COMP_SWAP - 已发布到“发送队列”中的WR的比较和交换操作(Compare and Swap operation for a WR that was posted to the Send Queue)
  • IBV_WC_FETCH_ADD - 已发布到发送队列的WR的获取和添加操作(Fetch and Add operation for a WR that was posted to the Send Queue)
  • IBV_WC_BIND_MW - 已发布到“发送队列”中的WR的“内存窗口”绑定操作(Memory Window bind operation for a WR that was posted to the Send Queue)
  • IBV_WC_RECV - 发送已发布到(QP或SRQ的)接收队列的WR的数据操作(Send data operation for a WR that was posted to a Receive Queue (of a QP or to an SRQ))
  • IBV_WC_RECV_RDMA_WITH_IMM -  发送到(QP或SRQ)的接收队列的WR的带立即数的RDMA。对于此操作码,仅消耗一个接收请求,并且未使用此RR的sg_list(RDMA with immediate for a WR that was posted to a Receive Queue (of a QP or to an SRQ). For this opcode, only a Receive Request was consumed and the sg_list of this RR wasn't used)
vendor_err

供应商特定的错误,如果completion 因错误而结束,则会提供更多信息。一旦WC以错误结束,该值向RDMA设备的供应商提供有关失败原因的提示。

byte_len

传输的字节数。与incoming Send或 RDMA Write with immediate operations的接收队列有关。该值不包括立即数的长度(如果存在)。与RDMA读取和原子操作的“发送队列”相关。对于不与SRQ关联的UD QP的接收队列,或者对于与UD QP关联的SRQ,该值等于消息的有效负载加上为GRH保留的40个字节。传输的字节数是消息的有效负载加上为GRH保留的40个字节(无论是否存在GRH)

(The number of bytes transferred. Relevant if the Receive Queue for incoming Send or RDMA Write with immediate operations. This value doesn't include the length of the immediate data, if such exists. Relevant in the Send Queue for RDMA Read and Atomic operations.For the Receive Queue of a UD QP that is not associated with an SRQ or for an SRQ that is associated with a UD QP this value equals to the payload of the message plus the 40 bytes reserved for the GRH.The number of bytes transferred is the payload of the message plus the 40 bytes reserved for the GRH, whether or not the GRH is present)

imm_data

(可选)以网络顺序的SEND或RDMA WRITE操作码中的32 bit 数字,与有效载荷一起发送到远程端,并放置在[接收工作完]( Receive Work Completion)成中,而不是在远端内存缓冲区中。如果设置了IBV_WC_WITH_IMM,则此值有效

qp_num

已完成的WR的本地QP的号码。与和SRQ相关的[接收工作完成](Receive Work Completions)相关.

(Local QP number of completed WR. Relevant for Receive Work Completions that are associated with an SRQ)

src_qp Source QP number (remote QP number) of completed WR. Relevant for Receive Work Completions of a UD QP
wc_flags Flags of the Work Completion. It is either 0 or the bitwise OR of one or more of the following flags:

  • IBV_WC_GRH - Indicator that GRH is present for a Receive Work Completions of a UD QP. If this bit is set, the first 40 bytes of the buffered that were referred to in the Receive request will contain the GRH of the incoming message. If this bit is cleared, the content of those first 40 bytes is undefined
  • IBV_WC_WITH_IMM - Indicator that imm_data is valid. Relevant for Receive Work Completions
pkey_index P_Key index. Relevant for GSI QPs
slid Source LID (the base LID that this message was sent from). Relevant for Receive Work Completions of a UD QP
sl Service Level (the SL LID that this message was sent with). Relevant for Receive Work Completions of a UD QP
dlid_path_bits Destination LID path bits. Relevant for Receive Work Completions of a UD QP (not applicable for multicast messages)

以下测试(opcode & IBV_WC_RECV)将指示 来自接收队列的完成状态。(The following test (opcode & IBV_WC_RECV) will indicate that the status of a completion is from the Receive Queue.)
对于UD QP的[接收工作完成](receive Work Completions),无论是否设置了IBV_WC_GRH位,数据均从已发布的接收缓冲区的偏移量40开始。
并非所有wc属性始终有效。如果完成状态不是IBV_WC_SUCCESS,则仅以下属性有效:

  • wr_id
  • status
  • qp_num
  • vendor_err

返回值

Value Description
正数

从CQ读取的WC数及其值在wc中返回。如果该值小于num_entries,则表示CQ中没有更多的工作完成。如果此值等于num_entries,则CQ中可能会有更多的工作完成

0 CQ为空
负数

尝试从CQ读取WC(工作完成)时发生故障

例子

从CQ轮询WC(工作完成)(在轮询模式下):

struct ibv_wc wc;

int num_comp;

 
do {
    num_comp = ibv_poll_cq(cq, 1, &wc);
} while (num_comp == 0);
 
if (num_comp < 0) {
    fprintf(stderr, "ibv_poll_cq() failed\n");
    return -1;
}
 
/* verify the completion status */
if (wc.status != IBV_WC_SUCCESS) {
    fprintf(stderr, "Failed status %s (%d) for wr_id %d\n", 
        ibv_wc_status_str(wc.status),
        wc.status, (int)wc.wr_id);
    return -1;
}

常见问题

那工作完成(WC)到底是什么?

工作完成意味着相应的工作请求已结束,缓冲区可以(重新)用于读取,写入或释放。

ibv_poll_cq()是否引起上下文切换?

否。Work Completions的轮询根本不会导致上下文切换;它不会导致上下文切换。这就是为什么RDMA技术可以实现极低的延迟(低于1 usc)的原因。

Is there a limit to the number of Work Completions that can we polled when calling ibv_poll_cq()?

调用ibv_poll_cq()时可以轮询的[工作完成](Work Completions 数量是否有限制?

没有,你想读多少都行。

我调用了ibv_poll_cq(),它填充了我提供给它的所有数组。我能否知道CQ中还有多少工作完成?

不,你不能。

我从UD QP的接收队列中获得了工作完成(WC),并且进展顺利。我从内存缓冲区中读取了数据,但数据不正确。为什么?

也许您查看了数据的起始偏移量0。对于UD QP的任何工作完成,无论是否存在GRH,数据都将放置在相关内存缓冲区的偏移量40中。


什么是GRH,为什么我需要它?
全局路由头(GRH)提供的信息对于将消息发回给此消息的发件人(如果来自其他子网或来自多播组)最有用。

I've got completion with error status. Can I read all of the Work Completion fields?


否。如果“工作完成”状态表明存在错误,则仅以下属性有效:wr_id,status,qp_num和vendor_err。其余属性未定义。

我从CQ上Read了一个WC,但我不需要,我可以将其退还给CQ吗?

不,你不能。

我可以阅读属于特定工作队列的工作完成吗?(Can I read Work Completion that belongs to a specific Work Queue?

不,你不能。


如果添加的工作完成(WC)数量超过CQ的size,将会发生什么情况?


CQ将超限,并且CQ(以及与之关联的所有QP)将进入错误状态。

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/bandaoyu/article/details/116493052

智能推荐

远程桌面服务器连接失败,Windows服务器远程桌面连接失败是什么原因-程序员宅基地

文章浏览阅读6.1k次。1、先通过显示器直接连接的方式连接到那台连接出错的服务器上。登陆以后,在桌面上的此电脑图标上面点右键,选择属性。2、然后在属性页面中我们可以看到对应的系统版本是Windows Server 2016。CPU是intel xeon型号,内存是16g。这个时候我们点击它左上角的远程设置。3、在远程设置界面,我们可以看到,它默认的设置是允许远程到这台服务器上,但是它下面勾选了”仅允许运行使用网络级别身份..._远程桌面服务无法加入服务器 win-th0hfpjn9vr 上的连接代理。 错误: 当前异步消息

ApacheCN 翻译活动进度公告 2019.6.15-程序员宅基地

文章浏览阅读109次。Special Sponsors我们组织了一个开源互助平台,方便开源组织和大 V 互相认识,互相帮助,整合资源。请回复这个帖子并注明组织/个人信息来申请加入。请回复这个帖子来推荐希望翻译的内容。如果大家遇到了做得不错的教程或翻译项目,也可以推荐给我们。我们会联系项目的维护者,一起把...

Python培训课程深圳,群年轻人正在追捧Python-程序员宅基地

文章浏览阅读182次。记者 | 伍洋宇 袁伟腾编辑 | 文姝琪1李楠打算年底换份新工作,Python方向的、纯软件岗位,发挥空间更大的全栈开发工程师就很不错,目标月薪一万二。这使得他在今年下半年开始系统学习Python。因为本科是计算机专业,期间也自学过Python这门语言,李楠选择了继续自学。学Python真的有用吗?“当然有用啦,没用谁去学它啊。”今年24岁、刚刚毕业一年的李楠这么说。目前他在一家智能硬件公司做嵌入式开发软件工程师,月薪一万,工作是“往硬件里面写软件”,他觉得太枯燥了。“代码都是写好的,基..

Ubuntu下安装R,升级R版本,安装Rstudio,安装Rstudio Server以及安装Shiny Server_marutter-ubuntu-rrutter-focal.list-程序员宅基地

文章浏览阅读2.9k次。一、安装R只需要一步命令:sudo apt-get install r-base二、升级R版本第一步给Ubuntu指定PPA:sudo add-apt-repository ppa:marutter/rrutter第二步:sudo apt-get update第三步:sudo apt-get upgrade三、安装Rstudio直接去Rstudio官网下载最新版的Rst..._marutter-ubuntu-rrutter-focal.list

Redis5.0集群搭建(Redis Cluster)_rediscluster搭建 5.0-程序员宅基地

文章浏览阅读9.1k次。Redis5.0集群搭建RedisCluster_rediscluster搭建 5.0

题目-java基础_面向过程的程序设计是把计算机程序视为一系列的命令集合-程序员宅基地

文章浏览阅读405次。多线程和单线程线程不是越多越好,假如你的业务逻辑全部是计算型的(CPU密集型),不涉及到IO,并且只有一个核心。那肯定一个线程最好,多一个线程就多一点线程切换的计算,CPU不能完完全全的把计算能力放在业务计算上面,线程越多就会造成CPU利用率(用在业务计算的时间/总的时间)下降。但是在WEB场景下,业务并不是CPU密集型任务,而是IO密集型的任务,一个线程是不合适,如果一个线程在等待数据时,把CPU的计算能力交给其他线程,这样也能充分的利用CPU资源。但是线程数量也要有个限度,一般线程数有一个公式:最佳启_面向过程的程序设计是把计算机程序视为一系列的命令集合

随便推点

储能8串电池用140W DCDC电路2 USB_A 2个 TYPE-C 2A2C_pl56002-程序员宅基地

文章浏览阅读78次。储能8串电池,输出是2个C口,2个USBA口,功率是C1:140W,C2:100W,A1:18W,A2:18W.A1,A2不降功率,使用IP2736,IP2723T,IP2163,_pl56002

python3.8.1手机版下载-Python官方下载|Python最新版 V3.8.1 -推背图下载站-程序员宅基地

文章浏览阅读2k次。Python最新版是一款功能强大脚本编程软件。Python最新版它可以帮助编程人员更加便捷的进行代码编写,适合完成各种高层任务,兼容所有的操作系统中使用,因为它的便捷性,在程序员中得到广泛的应用,新入门的编程学习者可以使用它快速学习,欢迎前来下载!功能特点1、简单易学Python极其容易上手,因为Python有极其简单的说明文档 。2、免费开源Python是FLOSS(自由/开放源码软件)之一。3..._手机版python官网下载

Unity3D学习之(坦克大战解析)-程序员宅基地

文章浏览阅读3.9k次。欢迎大家光临我的博客!对坦克大战项目的解析:一、游戏模块主要是:注册模块、登录模块、我方模块、和敌方模块。①注册模块:可以跳转到登录界面!②登录模块:可以跳转到游戏界面!③我方模块:可以前后左右移动,可以发射子弹,可以死亡销毁。④敌方模块:可以可以发射子弹,追踪我方的位置,也可以随机出现,可以死亡销毁。 二、所用到的技术①键盘事件 //敌我双方通过键盘上下左右键的移动②位移 ...

【linux】进程和线程的几种状态及状态切换_linux线程状态-程序员宅基地

文章浏览阅读3.6k次,点赞46次,收藏54次。进程和线程的状态_linux线程状态

Java/Mysql数据库+SSM+学生信息管理系统 11578(免费领源码)计算机毕业设计项目推荐上万套实战教程JAVA、PHP,node.js,C++、python、大屏可视化等-程序员宅基地

文章浏览阅读1.1k次,点赞22次,收藏20次。免费领取项目源码,请关注●点赞●收藏并私信博主,谢谢~本系统以实际运用为开发背景,通过系统管理员可以对所有的学生和教师等人员以及学生相关联的一些学生管理、分配任务、完成任务、打卡签到、师生交流等数据信息进行统一的管理,方便资料的保留。教师和学生可以通过注册,然后登录到系统当中,对分配任务、完成任务、打卡签到以及师生交流这些信息进行查询管理。总的来说,系统的前台是通过Java页面展示,后台使用SSM这个框架,数据库采用目前流行的开源关系型数据库MYSQL。

如何在群辉NAS系统下安装cpolar套件,并使用cpolar内网穿透?_在群晖nas安装cpolar套件-程序员宅基地

文章浏览阅读1.2k次,点赞39次,收藏34次。群晖作为大容量存储系统,既可以作为个人的私有存储设备,也可以放在小型企业中作为数据中心使用。其强大的数据存储和管理功能,让其还能够胜任更多任务。但由于群晖的应用场景所限,这些功能通常只能在局域网内实现,想要让群晖NAS存储的数据能在公网访问到,我们可以借助cpolar的辅助,轻松实现在公共互联网访问内网群晖NAS上的数据。在这之前,我们还是需要了解下cpolar的基本操作方式。_在群晖nas安装cpolar套件