notebook/2-2-beautifulsoup-css.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beautifulsoup: find by CSS class \n",
    "**First we still need to open a [page](https://mofanpy.com/static/scraping/list.html), then we can apply beautifulsoup on this page's html.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n<html lang=\"cn\">\n<head>\n\t<meta charset=\"UTF-8\">\n\t<title>爬虫练习 列表 class | 莫烦 Python</title>\n\t<style>\n\t.jan {\n\t\tbackground-color: yellow;\n\t}\n\t.feb {\n\t\tfont-size: 25px;\n\t}\n\t.month {\n\t\tcolor: red;\n\t}\n\t</style>\n</head>\n\n<body>\n\n<h1>列表 爬虫练习</h1>\n\n<p>这是一个在 <a href=\"https://mofanpy.com/\" >莫烦 Python</a> 的 <a href=\"https://mofanpy.com/tutorials/scraping\" >爬虫教程</a>\n\t里无敌简单的网页, 所有的 code 让你一目了然, 清晰无比.</p>\n\n<ul>\n\t<li class=\"month\">一月</li>\n\t<ul class=\"jan\">\n\t\t<li>一月一号</li>\n\t\t<li>一月二号</li>\n\t\t<li>一月三号</li>\n\t</ul>\n\t<li class=\"feb month\">二月</li>\n\t<li class=\"month\">三月</li>\n\t<li class=\"month\">四月</li>\n\t<li class=\"month\">五月</li>\n</ul>\n\n</body>\n</html>\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "\n",
    "# if has Chinese, apply decode()\n",
    "html = urlopen(\"https://mofanpy.com/static/scraping/list.html\").read().decode('utf-8')\n",
    "print(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Parsing this html using a method called lxml, create a soup object. Find all \"li\" tag which has a class=month.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "一月\n二月\n三月\n四月\n五月\n"
     ]
    }
   ],
   "source": [
    "soup = BeautifulSoup(html, features='lxml')\n",
    "\n",
    "# use class to narrow search\n",
    "month = soup.find_all('li', {\"class\": \"month\"})\n",
    "for m in month:\n",
    "    print(m.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Or using some helpful functions to find tags. Access the attribute of found tags using a key just like what you do in a python dictionary.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "一月一号\n一月二号\n一月三号\n"
     ]
    }
   ],
   "source": [
    "\n",
    "jan = soup.find('ul', {\"class\": 'jan'})\n",
    "d_jan = jan.find_all('li')              # use jan as a parent\n",
    "for d in d_jan:\n",
    "    print(d.get_text())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}