当我们使用python爬取网页时，运行以下代码：
import urllib.request
baseURL = "https://www.bilibili.com/anime/index/#season_version=-1&area=-1&is_finish=-1©right=-1&season_status=-1&season_month=-1&year=-1&style_id=-1&order=3&st=1&sort=0&page=1"
response = urllib.request.urlopen(baseURL)
print(response.read())得到运行结果如下：
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xbd[ks\xd4F\x16\xfd\xbe\xbfb\xa2\xaa\xad\xe0\xda\x9e\x19I\xf3\xb4\xb1\xbd\xc5\x12\xb2\xa1j!\xd9\x84\xa4v7\x95r\xc93\x9a\x19\x81F\x9aH\x1a\x1b/a\x0b\x82\x01\x1b\x08\x8f\x04\x02\xc4\x06\xc2k\x81\x80\xc1\t\x8e\xcb\xc6
原因分析：
得到以上运行结果的原因是该网页使用了gzip压缩，这里我们只需要使用gzip解压即可。
解决办法如下：
import urllib.request
import gzip # 引入gzip库
baseURL = "https://www.bilibili.com/anime/index/#season_version=-1&area=-1&is_finish=-1©right=-1&season_status=-1&season_month=-1&year=-1&style_id=-1&order=3&st=1&sort=0&page=1"
response = urllib.request.urlopen(baseURL)
result = response.read()
print(gzip.decompress(result).decode('utf-8')) # 使用gzip来解压文件运行结果如下：
<!DOCTYPE html><html lang="zh-Hans"><head><meta charset="utf-8"><title>番剧索引 - 哔哩哔哩 (゜-゜)つロ 干杯~-bilibili</title><meta name="description" content="bilibili是国内知名的视频弹幕网站，这里有最及时的动漫新番，最棒的ACG氛围，最有创意的Up主。大家可以在这里找到许多欢乐。"><meta name="keywords" content="B站,弹幕,字幕,AMV,MAD,MTV,ANIME,动漫,动漫音乐,游戏,游戏解说,ACG,galgame,动画,番组,新番,初音,洛天依,vocaloid"><meta name="renderer" content="webkit"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="baidu-site-verification" content="gbRdPloQBZ"><meta name="spm_prefix" content="666.14"><link rel="dns-prefetch" href="//s1.hdslb.com"><link rel="dns-prefetch" href="//s2.hdslb.com"><link rel="dns-prefetch" href="//s3.hdslb.com"><link rel="dns-prefetch" href="//i0.hdslb.com"><link rel="dns-prefetch" href="//i1.hdslb.com"><link rel="dns-prefetch" href="//i2.hdslb.com"><link rel="dns-prefetch" href="//static.hdslb.com"><link rel="shortcut icon" href="//static.hdslb.com/images/favicon.ico"><link rel="search" type="application/opensearchdescription+xml" href="//static.hdslb.com/opensearch.xml" title="哔哩哔哩"><script type="text/javascript" src="//static.hdslb.com/js/jquery.min.js"></script><script type="text/javascript" src="//s1.hdslb.com/bfs/seed/jinkela/header-v2/header.js"></script><script type="text/javascript" src="//static.hdslb.com/js/promise.auto.min.js"></script><script type="text/javascript" src="//www.bilibili.com/gentleman/polyfill.js?features=Promise%2CObject.assign%2CString.prototype.includes%2CNumber.isNaN"></script><script type="text/javascript" src="//s1.hdslb.com/bfs/static/ogv/fe/iris.min.js"></script><script type="text/javascript" src="//s1.hdslb.com/bfs/static/jinkela/long/js/sentry/sentry-5.2.1.min.js"></script><script type="text/javascript" src="//s1.hdslb.com/bfs/static/jinkela/long/js/sentry/sentry.vue.js"></script><link rel="stylesheet" href="//s1.hdslb.com/bfs/static/bangumi-index/css/bangumi-index.0.17133a0d63c411c79a833857f5870231042107b7.css"></head><body><div id="biliMainHeader" type="all" style="height:263px"></div><script type="text/javascript">0</script><div id="app" data-server-rendered="true" class="bangumi-index-wrapper"><div class="bangumi-index-crumb"><ul class="clearfix"><li><a href="//www.bilibili.com/anime/">全部</a></li> <li><a href="//www.bilibili.com/v/anime/serial/">连载动画</a></li> <li><a href="//www.bilibili.com/v/anime/finish/">完结动画</a></li> <li><a href="//www.bilibili.com/v/anime/information/">资讯</a></li> <li><a href="//www.bilibili.com/v/anime/offical/">官方延伸</a></li> <li><a href="//www.bilibili.com/anime/timeline/">新番时间表</a></li> <li class="on"><a href="javascript:;">番剧索引</a></li></ul> <!----> <!----> <!----> <!----></div> <div class="bangumi-index-body clearfix"><div class="filter-body"><ul class="sort-banner clearfix"><li class="sort-item"><i class="up"></i><span>追番人数</span><i class="down active"></i></li><li class="sort-item"><i class="up"></i><span>更新时间</span><i class="down active"></i></li><li class="sort-item lock-desc"><i class="up"></i><span>最高评分</span><i class="down active"></i></li><li class="sort-item"><i class="up"></i><span>播放数量</span><i class="down active"></i></li><li class="sort-item"><i class="up"></i><span>开播时间</span><i class="down active"></i></li></ul> <ul class="bangumi-list clearfix"> <div class="load-state"><span class="loading">正在加载...</span> <!----> <!----></div> <!----></ul> <!----></div> <div class="filter-wrapper"><div class="filter-title">筛选</div> <div class="filter-list"><div class="filter-block"><div class="filter-name">类型</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="正片" class="filter-item">正片</li><li title="电影" class="filter-item">电影</li><li title="其他" class="filter-item">其他</li> <!----></ul></div><div class="filter-block"><div class="filter-name">地区</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="日本" class="filter-item">日本</li><li title="美国" class="filter-item">美国</li><li title="其他" class="filter-item">其他</li> <!----></ul></div><div class="filter-block"><div class="filter-name">状态</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="完结" class="filter-item">完结</li><li title="连载" class="filter-item">连载</li> <!----></ul></div><div class="filter-block"><div class="filter-name">版权</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="独家" class="filter-item">独家</li><li title="其他" class="filter-item">其他</li> <!----></ul></div><div class="filter-block"><div class="filter-name">付费</div> <ul class="filter-item-wrapper free"><li title="全部" class="filter-item on">全部</li><li title="免费" class="filter-item">免费</li><li title="付费" class="filter-item">付费</li><li title="大会员" class="filter-item">大会员</li> <!----></ul></div><div class="filter-block"><div class="filter-name">季度</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="1月" class="filter-item">1月</li><li title="4月" class="filter-item">4月</li><li title="7月" class="filter-item">7月</li><li title="10月" class="filter-item">10月</li> <!----></ul></div><div class="filter-block"><div class="filter-name">时间</div> <ul class="filter-item-wrapper free"><li title="全部" class="filter-item on">全部</li><li title="2021" class="filter-item">2021</li><li title="2020" class="filter-item">2020</li><li title="2019" class="filter-item">2019</li><li title="2018" class="filter-item">2018</li><li title="2017" class="filter-item">2017</li><li title="2016" class="filter-item">2016</li><li title="2015" class="filter-item">2015</li><li title="2014-2010" class="filter-item">2014-2010</li><li title="2009-2005" class="filter-item">2009-2005</li><li title="2004-2000" class="filter-item">2004-2000</li><li title="90年代" class="filter-item">90年代</li><li title="80年代" class="filter-item">80年代</li><li title="更早" class="filter-item">更早</li> <!----></ul></div><div class="filter-block"><div class="filter-name">风格</div> <ul class="filter-item-wrapper"><li title="全部" class="filter-item on">全部</li><li title="原创" class="filter-item">原创</li><li title="漫画改" class="filter-item">漫画改</li><li title="小说改" class="filter-item">小说改</li><li title="游戏改" class="filter-item">游戏改</li><li title="特摄" class="filter-item">特摄</li><li title="布袋戏" class="filter-item">布袋戏</li><li title="热血" class="filter-item">热血</li><li title="穿越" class="filter-item">穿越</li><li title="奇幻" class="filter-item">奇幻</li><li title="战斗" class="filter-item">战斗</li><li title="搞笑" class="filter-item">搞笑</li><li title="日常" class="filter-item">日常</li><li title="科幻" class="filter-item">科幻</li><li title="萌系" class="filter-item">萌系</li><li title="治愈" class="filter-item">治愈</li><li title="校园" class="filter-item">校园</li><li title="少儿" class="filter-item">少儿</li><li title="泡面" class="filter-item">泡面</li><li title="恋爱" class="filter-item">恋爱</li><li title="少女" class="filter-item">少女</li><li title="魔法" class="filter-item">魔法</li><li title="冒险" class="filter-item">冒险</li><li title="历史" class="filter-item">历史</li><li title="架空" class="filter-item">架空</li><li title="机战" class="filter-item">机战</li><li title="神魔" class="filter-item">神魔</li><li title="声控" class="filter-item">声控</li><li title="运动" class="filter-item">运动</li><li title="励志" class="filter-item">励志</li><li title="音乐" class="filter-item">音乐</li><li title="推理" class="filter-item">推理</li><li title="社团" class="filter-item">社团</li><li title="智斗" class="filter-item">智斗</li><li title="催泪" class="filter-item">催泪</li><li title="美食" class="filter-item">美食</li><li title="偶像" class="filter-item">偶像</li><li title="乙女" class="filter-item">乙女</li><li title="职场" class="filter-item">职场</li> <!----></ul></div></div></div></div></div><script>window.__INITIAL_STATE__={"indexType":"anime","ssType":1,"tid":13,"filters":[{"title":"类型","key":"season_version","list":[{"value":-1,"name":"全部"},{"value":1,"name":"正片"},{"value":2,"name":"电影"},{"value":3,"name":"其他"}]},{"title":"地区","key":"area","list":[{"value":-1,"name":"全部"},{"value":2,"name":"日本"},{"value":3,"name":"美国"},{"value":"1,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55","name":"其他"}]},{"title":"状态","key":"is_finish","list":[{"value":-1,"name":"全部"},{"value":1,"name":"完结"},{"value":0,"name":"连载"}]},{"title":"版权","key":"copyright","list":[{"value":-1,"name":"全部"},{"value":3,"name":"独家"},{"value":"1,2,4","name":"其他"}]},{"title":"付费","key":"season_status","list":[{"value":-1,"name":"全部"},{"value":1,"name":"免费"},{"value":"2,6","name":"付费"},{"value":"4,6","name":"大会员"}],"free":true},{"title":"季度","key":"season_month","list":[{"value":-1,"name":"全部"},{"value":1,"name":"1月"},{"value":4,"name":"4月"},{"value":7,"name":"7月"},{"value":10,"name":"10月"}]},{"title":"时间","key":"year","list":[{"value":-1,"name":"全部"},{"value":"[2021,2022)","name":"2021"},{"value":"[2020,2021)","name":"2020"},{"value":"[2019,2020)","name":"2019"},{"value":"[2018,2019)","name":"2018"},{"value":"[2017,2018)","name":"2017"},{"value":"[2016,2017)","name":"2016"},{"value":"[2015,2016)","name":"2015"},{"value":"[2010,2015)","name":"2014-2010"},{"value":"[2005,2010)","name":"2009-2005"},{"value":"[2000,2005)","name":"2004-2000"},{"value":"[1990,2000)","name":"90年代"},{"value":"[1980,1990)","name":"80年代"},{"value":"[,1980)","name":"更早"}],"free":true},{"title":"风格","key":"style_id","list":[{"value":-1,"name":"全部"},{"value":10010,"name":"原创"},{"value":10011,"name":"漫画改"},{"value":10012,"name":"小说改"},{"value":10013,"name":"游戏改"},{"value":10102,"name":"特摄"},{"value":10015,"name":"布袋戏"},{"value":10016,"name":"热血"},{"value":10017,"name":"穿越"},{"value":10018,"name":"奇幻"},{"value":10020,"name":"战斗"},{"value":10021,"name":"搞笑"},{"value":10022,"name":"日常"},{"value":10023,"name":"科幻"},{"value":10024,"name":"萌系"},{"value":10025,"name":"治愈"},{"value":10026,"name":"校园"},{"value":10027,"name":"少儿"},{"value":10028,"name":"泡面"},{"value":10029,"name":"恋爱"},{"value":10030,"name":"少女"},{"value":10031,"name":"魔法"},{"value":10032,"name":"冒险"},{"value":10033,"name":"历史"},{"value":10034,"name":"架空"},{"value":10035,"name":"机战"},{"value":10036,"name":"神魔"},{"value":10037,"name":"声控"},{"value":10038,"name":"运动"},{"value":10039,"name":"励志"},{"value":10040,"name":"音乐"},{"value":10041,"name":"推理"},{"value":10042,"name":"社团"},{"value":10043,"name":"智斗"},{"value":10044,"name":"催泪"},{"value":10045,"name":"美食"},{"value":10046,"name":"偶像"},{"value":10047,"name":"乙女"},{"value":10048,"name":"职场"}],"line":10}],"orders":[{"title":"追番人数","key":"3","sort":"0,1"},{"title":"更新时间","key":"0","sort":"0,1"},{"title":"最高评分","key":"4","sort":"0"},{"title":"播放数量","key":"2","sort":"0,1"},{"title":"开播时间","key":"5","sort":"0,1"}],"total":0,"pageData":[],"pages":0};(function(){var s;(s=document.currentScript||document.scripts[document.scripts.length-1]).parentNode.removeChild(s);}());</script><script src="//s1.hdslb.com/bfs/static/bangumi-index/1.bangumi-index.17133a0d63c411c79a833857f5870231042107b7.js" crossorigin defer></script><script src="//s1.hdslb.com/bfs/static/bangumi-index/bangumi-index.17133a0d63c411c79a833857f5870231042107b7.js" crossorigin defer></script><div id="biliMainFooter"></div><script type="text/javascript" src="//s1.hdslb.com/bfs/seed/jinkela/footer-v2/footer.js" defer="defer"></script><script type="text/javascript">window.bid = 13;
window.spmReportData = {}
window.reportConfig = {
sample : 1,
scrollTracker: true,
msgObjects : 'spmReportData',
errorTracker: true
}
window.isForceNarrow = true;</script></body></html>这样就可以爬取到完整的网页内容了~
文章浏览阅读1k次。通过使用ajax方法跨域请求是浏览器所不允许的，浏览器出于安全考虑是禁止的。警告信息如下：不过jQuery对跨域问题也有解决方案，使用jsonp的方式解决，方法如下：$.ajax({ async:false, url: 'http://www.mysite.com/demo.do', // 跨域URL ty..._nginx不停的xhr
文章浏览阅读2k次。关于在 Oracle 中配置 extproc 以访问 ST_Geometry，也就是我们所说的 使用空间SQL 的方法，官方文档链接如下。http://desktop.arcgis.com/zh-cn/arcmap/latest/manage-data/gdbs-in-oracle/configure-oracle-extproc.htm其实简单总结一下，主要就分为以下几个步骤。..._extproc
文章浏览阅读1.5w次。linux下没有上面的两个函数，需要使用函数 mbstowcs和wcstombsmbstowcs将多字节编码转换为宽字节编码wcstombs将宽字节编码转换为多字节编码这两个函数，转换过程中受到系统编码类型的影响，需要通过设置来设定转换前和转换后的编码类型。通过函数setlocale进行系统编码的设置。linux下输入命名locale -a查看系统支持的编码_linux c++ gbk->utf8
文章浏览阅读750次。今天准备从生产库向测试库进行数据导入，结果在imp导入的时候遇到“ IMP-00009:导出文件异常结束” 错误，google一下，发现可能有如下原因导致imp的数据太大，没有写buffer和commit两个数据库字符集不同从低版本exp的dmp文件，向高版本imp导出的dmp文件出错传输dmp文件时，文件损坏解决办法：imp时指定..._imp-00009导出文件异常结束
文章浏览阅读143次。当下是一个大数据的时代，各个行业都离不开数据的支持。因此，网络爬虫就应运而生。网络爬虫当下最为火热的是Python，Python开发爬虫相对简单，而且功能库相当完善，力压众多开发语言。本次教程我们爬取前程无忧的招聘信息来分析Python程序员需要掌握那些编程技术。首先在谷歌浏览器打开前程无忧的首页，按F12打开浏览器的开发者工具。浏览器开发者工具是用于捕捉网站的请求信息，通过分析请求信息可以了解请..._初级python程序员能力要求
文章浏览阅读7.6k次，点赞2次，收藏6次。@Service标注的bean，类名：ABDemoService查看源码后发现，原来是经过一个特殊处理：当类的名字是以两个或以上的大写字母开头的话，bean的名字会与类名保持一致public class AnnotationBeanNameGenerator implements BeanNameGenerator { private static final String C..._@service beanname
文章浏览阅读6.9w次，点赞73次，收藏463次。1.前序创建#include<stdio.h>#include<string.h>#include<stdlib.h>#include<malloc.h>#include<iostream>#include<stack>#include<queue>using namespace std;typed_二叉树的建立
文章浏览阅读7.1k次。在Asp.net上使用Excel导出功能，如果文件名出现中文，便会以乱码视之。 解决方法： fileName = HttpUtility.UrlEncode(fileName, System.Text.Encoding.UTF8);_asp.net utf8 导出中文字符乱码
文章浏览阅读2.1k次，点赞4次，收藏23次。第一次实验 词法分析实验报告设计思想词法分析的主要任务是根据文法的词汇表以及对应约定的编码进行一定的识别，找出文件中所有的合法的单词，并给出一定的信息作为最后的结果，用于后续语法分析程序的使用；本实验针对 PL/0 语言 的文法、词汇表编写一个词法分析程序，对于每个单词根据词汇表输出： (单词种类, 单词的值) 二元对。词汇表：种别编码单词符号助记符0beginb..._对pl/0作以下修改扩充。增加单词
文章浏览阅读773次。我在使用adb.exe时遇到了麻烦.我想使用与bash相同的adb.exe shell提示符,所以我决定更改默认的bash二进制文件(当然二进制文件是交叉编译的,一切都很完美)更改bash二进制文件遵循以下顺序> adb remount> adb push bash / system / bin /> adb shell> cd / system / bin> chm..._adb shell mv 权限
文章浏览阅读6.8k次，点赞12次，收藏125次。1. 单目相机标定引言相机标定已经研究多年，标定的算法可以分为基于摄影测量的标定和自标定。其中，应用最为广泛的还是张正友标定法。这是一种简单灵活、高鲁棒性、低成本的相机标定算法。仅需要一台相机和一块平面标定板构建相机标定系统，在标定过程中，相机拍摄多个角度下（至少两个角度，推荐10~20个角度）的标定板图像（相机和标定板都可以移动），即可对相机的内外参数进行标定。下面介绍张氏标定法（以下也这么称呼）的原理。原理相机模型和单应矩阵相机标定，就是对相机的内外参数进行计算的过程，从而得到物体到图像的投影_相机-投影仪标定
文章浏览阅读2.2k次。文章目录Wayland 架构Wayland 渲染Wayland的 硬件支持简 述： 翻译一篇关于和 wayland 有关的技术文章, 其英文标题为Wayland Architecture .Wayland 架构若是想要更好的理解 Wayland 架构及其与 X (X11 or X Window System) 结构；一种很好的方法是将事件从输入设备就开始跟踪, 查看期间所有的屏幕上出现的变化。这就是我们现在对 X 的理解。 内核是从一个输入设备中获取一个事件，并通过 evdev 输入_wayland