Where’s my message? Durability and you-程序员宅基地

技术标签: RabbitMQ  

There’s a dirty secret about creating queues and exchanges in Rabbit: by default they don’t survive reboot. That’s right; restart your RabbitMQ server and watch those queues and exchanges go poof (along with the messages inside). 

The reason is because of a property on every queue and exchange called  durable. It defaults to false, and tells RabbitMQ whether the queue (or exchange) should be re-created after a crash or restart of Rabbit. Set it to true and you won’t have to re-create those queues and exchanges when the power supply in your server dies. You might also think that setting durable to true on the exchanges and queues is all you need to do to make your messages survive a reboot, but you’d be wrong. Whereas queues and exchanges must be durable to allow messages to survive reboot, it isn’t enough on its own. 

 A message that can survive a crash of the AMQP broker is called persistent. You flag a message as persistent by setting the  delivery mode option of the message to 2 (your AMQP client may use a human-friendly constant instead) before publishing it. At this  point, the message is indicated as persistent, but it must be published to an exchange  that’s durable and arrive in a queue that’s durable to survive. If this weren’t the case,  the queue (or exchange) a persistent message was sitting in when Rabbit crashed  wouldn’t exist when Rabbit restarted, thereby orphaning the message. So, for a mes sage that’s in flight inside Rabbit to survive a crash, the message must 

Have its delivery mode option set to 2 (persistent) 
 Be published into a durable exchange 
 Arrive in a durable queue 

Do these three things and you won't have to play Where’s Waldo with your critical
messages. 
 The way that RabbitMQ ensures persistent messages survive a restart is by writing them to the disk inside of a persistency log file. When you publish a persistent message to a durable exchange, Rabbit won’t send the response until the message is committed to the log file. Keep in mind, though, that if it gets routed to a non durable queue after that, it’s automatically removed from the persistency log and won’t survive a restart. When you use persistent messages it’s crucial that you make sure  all three elements required for a message to persist are in place (we can’t stress this enough).
Once you consume a persistent message from a durable queue (and acknowledge it), RabbitMQ flags it in the persistency log for garbage collection. If Rabbit restarts any-time before you consume a persistent message, it’ll automatically re-create theexchanges and queues (and bindings) and replay any messages in the persistency log
into the appropriate queues or exchanges (depending on where in the routing process the messages were when Rabbit died). 


 You might be thinking that you should use persistent messaging for all of your messages. You could do that, but you’d pay a price for ensuring your messages survive Rabbit restarts: performance. The act of writing messages to disk is much slower than just storing them in RAM, and will significantly decrease the number of messages per second your RabbitMQ server can process. It’s not uncommon to see a 10x or more decrease in message throughput when using persistency.

There’s also the issue thatpersistent messages don’t play well with RabbitMQ’s built-in clustering. Though
RabbitMQ clustering allows you to talk to any queue present in the cluster from any
node, those queues are actually evenly distributed among the nodes  without redun-
dancy (there’s no backup copy of any queue on a second node in the cluster). If the
cluster node hosting your seed_bin queue crashes, the queue disappears from the
cluster until the node is restored … if the queue was durable. More important, while
the node is down its queues aren’t available and the durable ones can’t be re-created.
This can lead to black-holing of messages. We’ll cover the behavior in more detail and
show alternate clustering approaches to get around this in chapter 5. 
 Given the trade-offs, when should you use persistent/durable messaging? First, you
need to analyze (and test) your performance needs. Do you need to process 100,000

messages per second on a single Rabbit server? If so, you should probably look at
other ways of ensuring message delivery (or get a very fast storage system). For exam-
ple, your producer could listen to a reply queue on a separate channel. Every time it
publishes a message, it includes the name of the reply queue so that the consumer can
send a reply back to confirm receipt. If a message isn’t replied to within a reasonable
amount of time, the producer can republish the message. That said, the critical
nature of messages requiring guaranteed delivery generally means they’re lower in
volume than other types of messages (such as logging messages). So if persistent mes-
saging meets your performance needs, it’s an excellent way to help ensure delivery.
We use it a lot for critical messages. We’re just selective about what types of content
use persistent messaging. For example, we run two types of Rabbit clusters: traditional
RabbitMQ clustering for nonpersistent messaging, and pairs of active/hot-standby
nonclustered Rabbit servers for persistent messaging (using load balancers). This
ensures the processing load for persistent messaging doesn’t slow down nonpersistent
messages. It also means Rabbit’s built-in clustering won’t black-hole persistent mes-
sages when a node dies. Do keep in mind that while Rabbit can help ensure delivery, it
can never absolutely guarantee it. Hard drive corruption, buggy behavior by a con-
sumer, or other extreme events can trash/black-hole persistent messages. It’s ulti-
mately up to you to ensure your messages arrive where they need to go, and persistent
messaging is a great tool to help you get there. 
 A concept that’s related to the durability of a message is the AMQP transaction. So
far we’ve talked about marking messages, queues, and exchanges as durable. That’s all
well and good for keeping a message safe once RabbitMQ has it in its custody, but
since a publish operation returns no response to the producer, how do you know if
the broker has persisted the durable message to disk? Should the broker die before it
can write the message to disk, the message would be lost and you wouldn’t know.
That’s where transactions come in. When you absolutely need to be sure the broker
has the message in custody (and has routed the message to all matching subscribed
queues) before you move on to another t奋斗ask, you need to wrap it in a transaction. If 
you come from a database background, it’s important not to confuse AMQP transactions with what “transaction” means in most databases. In  AMQP, after you place a channel into transaction mode, you send it the publish you want to confirm, followed

by zero or more other AMQP commands that should be executed or ignored depend-
ing on whether the initial publish succeeded. Once you’ve sent all of the commands,
you commit the transaction. If the transaction’s initial publish succeeds, then the chan-
nel will complete the other AMQP commands in the transaction. If the publish fails,
none of the other  AMQP commands will be executed. Transactions close the “last
mile” gap between producers publishing messages and RabbitMQ committing them
to disk, but there’s a better way to close that gap. 
 Though transactions are a part of the formal AMQP 0-9-1 specification, they have
an Achilles heel in that they’re huge drains on Rabbit performance. Not only can
using transactions drop your message throughput by a factor of 2–10x, but they also
make your producer app synchronous, which is one of the things you’re trying to get 
rid of with messaging. Knowing all of this, the guys at RabbitMQ decided to come up with a better way to ensure message delivery: publisher confirms.

2
 Similar to transactions,
you have to tell Rabbit to place the channel into confirm mode, and you can’t turn it
off without re-creating the channel. Once a channel is in confirm mode, every mes-
sage published on the channel will be assigned a unique  ID number (starting at 1).
Once the message has been delivered to all queues that have bindings matching the
message’s routing key, the channel will issue a publisher confirm to the producer app
(containing the message’s unique  ID). This lets the producer know the message has
been safely queued at all of its destinations. If the message and the queues are dura-
ble, the confirm is issued only after the queues have written the message to disk. The
major benefit of publisher confirms is that they’re asynchronous. Once a message has
been published, the producer app can go on to the next message while waiting for the
confirm. When the confirm for that message is finally received, a callback function in
the producer app will be fired so it can wake up and handle the confirmation
. If an
internal error occurs inside Rabbit that causes a message to be lost, Rabbit will send a
message nack (not acknowledged) that’s like a publisher confirm (it has the message’s
unique ID) but indicates the message was lost. Also, since there’s no concept of mes-
sage rollback (as with transactions), publisher confirms are much lighter weight and
have an almost negligible performance hit on the Rabbit broker. 
 Now you have the individual parts of RabbitMQ down, from consumers and pro-
ducers to durable messaging, but how do they all fit together? What does the lifecycle
of an actual message look like from beginning to end? The best way to answer that is
to look at the life of a message in code. 

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/hanruikai/article/details/49024411

智能推荐

攻防世界_难度8_happy_puzzle_攻防世界困难模式攻略图文-程序员宅基地

文章浏览阅读645次。这个肯定是末尾的IDAT了,因为IDAT必须要满了才会开始一下个IDAT,这个明显就是末尾的IDAT了。,对应下面的create_head()代码。,对应下面的create_tail()代码。不要考虑爆破,我已经试了一下,太多情况了。题目来源:UNCTF。_攻防世界困难模式攻略图文

达梦数据库的导出(备份)、导入_达梦数据库导入导出-程序员宅基地

文章浏览阅读2.9k次,点赞3次,收藏10次。偶尔会用到,记录、分享。1. 数据库导出1.1 切换到dmdba用户su - dmdba1.2 进入达梦数据库安装路径的bin目录,执行导库操作  导出语句:./dexp cwy_init/[email protected]:5236 file=cwy_init.dmp log=cwy_init_exp.log 注释:   cwy_init/init_123..._达梦数据库导入导出

js引入kindeditor富文本编辑器的使用_kindeditor.js-程序员宅基地

文章浏览阅读1.9k次。1. 在官网上下载KindEditor文件,可以删掉不需要要到的jsp,asp,asp.net和php文件夹。接着把文件夹放到项目文件目录下。2. 修改html文件,在页面引入js文件:<script type="text/javascript" src="./kindeditor/kindeditor-all.js"></script><script type="text/javascript" src="./kindeditor/lang/zh-CN.js"_kindeditor.js

STM32学习过程记录11——基于STM32G431CBU6硬件SPI+DMA的高效WS2812B控制方法-程序员宅基地

文章浏览阅读2.3k次,点赞6次,收藏14次。SPI的详情简介不必赘述。假设我们通过SPI发送0xAA,我们的数据线就会变为10101010,通过修改不同的内容,即可修改SPI中0和1的持续时间。比如0xF0即为前半周期为高电平,后半周期为低电平的状态。在SPI的通信模式中,CPHA配置会影响该实验,下图展示了不同采样位置的SPI时序图[1]。CPOL = 0,CPHA = 1:CLK空闲状态 = 低电平,数据在下降沿采样,并在上升沿移出CPOL = 0,CPHA = 0:CLK空闲状态 = 低电平,数据在上升沿采样,并在下降沿移出。_stm32g431cbu6

计算机网络-数据链路层_接收方收到链路层数据后,使用crc检验后,余数为0,说明链路层的传输时可靠传输-程序员宅基地

文章浏览阅读1.2k次,点赞2次,收藏8次。数据链路层习题自测问题1.数据链路(即逻辑链路)与链路(即物理链路)有何区别?“电路接通了”与”数据链路接通了”的区别何在?2.数据链路层中的链路控制包括哪些功能?试讨论数据链路层做成可靠的链路层有哪些优点和缺点。3.网络适配器的作用是什么?网络适配器工作在哪一层?4.数据链路层的三个基本问题(帧定界、透明传输和差错检测)为什么都必须加以解决?5.如果在数据链路层不进行帧定界,会发生什么问题?6.PPP协议的主要特点是什么?为什么PPP不使用帧的编号?PPP适用于什么情况?为什么PPP协议不_接收方收到链路层数据后,使用crc检验后,余数为0,说明链路层的传输时可靠传输

软件测试工程师移民加拿大_无证移民,未受过软件工程师的教育(第1部分)-程序员宅基地

文章浏览阅读587次。软件测试工程师移民加拿大 无证移民,未受过软件工程师的教育(第1部分) (Undocumented Immigrant With No Education to Software Engineer(Part 1))Before I start, I want you to please bear with me on the way I write, I have very little gen...

随便推点

Thinkpad X250 secure boot failed 启动失败问题解决_安装完系统提示secureboot failure-程序员宅基地

文章浏览阅读304次。Thinkpad X250笔记本电脑,装的是FreeBSD,进入BIOS修改虚拟化配置(其后可能是误设置了安全开机),保存退出后系统无法启动,显示:secure boot failed ,把自己惊出一身冷汗,因为这台笔记本刚好还没开始做备份.....根据错误提示,到bios里面去找相关配置,在Security里面找到了Secure Boot选项,发现果然被设置为Enabled,将其修改为Disabled ,再开机,终于正常启动了。_安装完系统提示secureboot failure

C++如何做字符串分割(5种方法)_c++ 字符串分割-程序员宅基地

文章浏览阅读10w+次,点赞93次,收藏352次。1、用strtok函数进行字符串分割原型: char *strtok(char *str, const char *delim);功能:分解字符串为一组字符串。参数说明:str为要分解的字符串,delim为分隔符字符串。返回值:从str开头开始的一个个被分割的串。当没有被分割的串时则返回NULL。其它:strtok函数线程不安全,可以使用strtok_r替代。示例://借助strtok实现split#include <string.h>#include <stdio.h&_c++ 字符串分割

2013第四届蓝桥杯 C/C++本科A组 真题答案解析_2013年第四届c a组蓝桥杯省赛真题解答-程序员宅基地

文章浏览阅读2.3k次。1 .高斯日记 大数学家高斯有个好习惯:无论如何都要记日记。他的日记有个与众不同的地方,他从不注明年月日,而是用一个整数代替,比如:4210后来人们知道,那个整数就是日期,它表示那一天是高斯出生后的第几天。这或许也是个好习惯,它时时刻刻提醒着主人:日子又过去一天,还有多少时光可以用于浪费呢?高斯出生于:1777年4月30日。在高斯发现的一个重要定理的日记_2013年第四届c a组蓝桥杯省赛真题解答

基于供需算法优化的核极限学习机(KELM)分类算法-程序员宅基地

文章浏览阅读851次,点赞17次,收藏22次。摘要:本文利用供需算法对核极限学习机(KELM)进行优化,并用于分类。

metasploitable2渗透测试_metasploitable2怎么进入-程序员宅基地

文章浏览阅读1.1k次。一、系统弱密码登录1、在kali上执行命令行telnet 192.168.26.1292、Login和password都输入msfadmin3、登录成功,进入系统4、测试如下:二、MySQL弱密码登录:1、在kali上执行mysql –h 192.168.26.129 –u root2、登录成功,进入MySQL系统3、测试效果:三、PostgreSQL弱密码登录1、在Kali上执行psql -h 192.168.26.129 –U post..._metasploitable2怎么进入

Python学习之路:从入门到精通的指南_python人工智能开发从入门到精通pdf-程序员宅基地

文章浏览阅读257次。本文将为初学者提供Python学习的详细指南,从Python的历史、基础语法和数据类型到面向对象编程、模块和库的使用。通过本文,您将能够掌握Python编程的核心概念,为今后的编程学习和实践打下坚实基础。_python人工智能开发从入门到精通pdf