調(diào)用urlopen時(shí)要加timeout, 否則網(wǎng)速不好的時(shí)候容易卡死.

html_downloader.py:

#?coding:?UTF-8
import?urllib2

class?HtmlDownloader(object):
????def?__init__(self):
????????pass
????????
????def?download(self,?url):
????????if?url?is?None:
????????????return?None
????????response?=?urllib2.urlopen(url,timeout=5)
????????
????????if?response.getcode()?!=?200:
????????????print?'???-->?Can?not?open?url:',?url
????????????return?None
????????????
????????html_doc?=?response.read()
????????#?print?html_doc
???????????????
????????return?html_doc