05-data-harvesting-for-digital-marketing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9093092c-2f43-49d3-9bec-74eacb3408b3",
   "metadata": {},
   "source": [
    "# Data Harvesting for Digital Marketing\n",
    "\n",
    "While working with digital markeing data, you will come across hetrogenous data types which can be cumbersome to work with. this is the reason why data engineering is a specific filed where people are only working thier best to clean, and pre-process the data to bring them in a homgenous format. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7f0b53-2c48-4c38-b4fa-e76eeddf50c9",
   "metadata": {},
   "source": [
    "## What are Different Data Types\n",
    "\n",
    "- Text\n",
    "    - Whois record\n",
    "        - https://www.ip2location.io/ip2whois-documentation\n",
    "        - whois application\n",
    "        - whois python package\n",
    "- Images\n",
    "- Spatial\n",
    "    - Lat/Long\n",
    "        - https://www.ip2location.io/ip2location-documentation\n",
    "- Temporal\n",
    "    - Time\n",
    "    - Date\n",
    "    - Date and Time\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72768229-c62a-428b-9895-8c200135d48e",
   "metadata": {},
   "source": [
    "## Data Source Types\n",
    "\n",
    "Nevertheseless, you will come acrss following formats. We have highlited them for your conveneice and then exaplain them in details. All prcocessing tool are discussed are usually available as Python package."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f62eecd-5f8d-4f85-8dda-2501f2bdedc8",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "### Data as Files\n",
    "\n",
    "- Tools\n",
    "    - Standard File Operatios\n",
    "        - Context Manager (with)\n",
    "        - Open Command\n",
    "    - Remote Files\n",
    "        - urllib (urlsparse, urlopen)\n",
    "        - requests (fetch, sitemap)\n",
    "        - wget https://pypi.org/project/wget/\n",
    "- location:\n",
    "    - Local Path\n",
    "        - C:/Users/Admin/tmp/robots.txt\n",
    "        - /urs/home/admin/tmp/robots.txt\n",
    "    - Remote Server \n",
    "        - https://www.seowings.org/robots.txt\n",
    "        - ftp://192.168.0.1/robots.txt\n",
    "- format\n",
    "    - txt\n",
    "    - data\n",
    "    - json\n",
    "    - zip files\n",
    "    - csv (comma seperated values)\n",
    "    - ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "af56b10b-14ba-4240-b939-0cfe81164f35",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/test.dat'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[25], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m file_path \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdata/test.dat\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m----> 3\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m \u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mr\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m file:\n\u001b[0;32m      4\u001b[0m     f\u001b[38;5;241m.\u001b[39mreadlines()\n",
      "File \u001b[1;32m~\\Documents\\GitHub\\data-science-for-digital-marketers\\.venv\\lib\\site-packages\\IPython\\core\\interactiveshell.py:284\u001b[0m, in \u001b[0;36m_modified_open\u001b[1;34m(file, *args, **kwargs)\u001b[0m\n\u001b[0;32m    277\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[0;32m    278\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[0;32m    279\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    280\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    281\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    282\u001b[0m     )\n\u001b[1;32m--> 284\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m io_open(file, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/test.dat'"
     ]
    }
   ],
   "source": [
    "file_path = \"data/test.dat\"\n",
    "\n",
    "with open (file_path, 'r') as file:\n",
    "    f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c0765ce-faf0-4e9c-93ed-ed2d124b7050",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Collecting/Scraping data from a Website\n",
    "- Tools\n",
    "    - python standard library\n",
    "        - urllib\n",
    "    - 3rd party\n",
    "        - requests\n",
    "    - Heavy lifting\n",
    "        - selenium \n",
    "            https://selenium-python.readthedocs.io/\n",
    "        - pyqt annd pyqtweb \n",
    "            https://www.pythonguis.com/examples/python-web-browser/\n",
    "\n",
    "- Location\n",
    "    - Remote (Web server)\n",
    "    - local (Web server)\n",
    "- format\n",
    "    - HTML\n",
    "    - CSS \n",
    "    - JS \n",
    "    \n",
    "#### HTTP Status Codes\n",
    "\n",
    "- After 399 status code, you will get errors\n",
    "- python requests libray returns a status object\n",
    "    - request object histroy (r.history will show request histroy as reponse )\n",
    "\n",
    "#### Types of Status Codes\n",
    "- Informational responses (100 – 199)\n",
    "- Successful responses (200 – 299)\n",
    "    - 200 response \n",
    "- Redirection messages (300 – 399)\n",
    "    - 301 permanent redirect\n",
    "    - 302 temporay redirect\n",
    "- Client error responses (400 – 499)\n",
    "    - 404 error - page not found\n",
    "    - 410 error - Page is permenatly gone\n",
    "- Server error responses (500 – 599)\n",
    "    - 500 error - server not avialble\n",
    "\n",
    "referece: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fe3077d-6d8f-4bf3-9814-2f24104b8ae6",
   "metadata": {},
   "source": [
    "### Downloading data via API Interface\n",
    "- Tools for data collection\n",
    "    - requests\n",
    "    - urllib (standard library)\n",
    "\n",
    "- Location\n",
    "    - Remote (Web server)\n",
    "    - local (Web server) \n",
    "- format\n",
    "    - XML (older websites/legacy)\n",
    "    - JSON (mostly new website)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e46791dd-8b6e-4fed-a68d-998476b55a1f",
   "metadata": {},
   "source": [
    "### Getting DAta from Databases\n",
    "- Tools:\n",
    "    - Specialized handlers for indvidual libraries\n",
    "        https://www.freecodecamp.org/news/connect-python-with-sql/\n",
    "    - Sqlalchemy\n",
    "        https://www.sqlalchemy.org/\n",
    "- Location\n",
    "    - Remote \n",
    "        - Oracle\n",
    "        - Postgres\n",
    "        - mysql\n",
    "        - sqlserver\n",
    "        - nosql\n",
    "    - local/Desktop \n",
    "        - (SQlite- https://sqlite.org/index.html)\n",
    "- format\n",
    "    - SQL Query"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1afa8a00-69f2-4aee-8b8e-073484b20b53",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Common Data Sources/Website for Digital Marketing\n",
    "\n",
    "- Mostly available via API (99%)\n",
    "    - official\n",
    "    - unofficial\n",
    "\n",
    "- Data Source\n",
    "    1. Twitter Data\n",
    "    2. Facebook data\n",
    "    3. linkedin data\n",
    "    4. instagram\n",
    "        - https://github.com/InstaPy/InstaPy\n",
    "    5. pinterest\n",
    "    6. Google search\n",
    "    7. Gmail Data\n",
    "    8. Analytics\n",
    "    9. Amazon data\n",
    "        - PPC\n",
    "    10. Wallmart\n",
    "    11. Ebay\n",
    "    12. Youtube\n",
    "        - https://developers.google.com/youtube/v3/quickstart/python\n",
    "        - https://github.com/ytdl-org/youtube-dl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fb3e122-7c4f-48b1-aa18-72fe5eafa616",
   "metadata": {
    "tags": []
   },
   "source": [
    "## How to Read Data from Local Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01d4d9e-70fe-44e3-8728-27571dcaec04",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "16da618b-a6bf-4304-812a-5204ec90a71c",
   "metadata": {},
   "source": [
    "## Before you get started with Data Scraping\n",
    "\n",
    "- Websites\n",
    "Most websites forbid to harvesting data via automated bots. Make sure you first read Terms and Condition of indvidual websites and see if automated web scraping is allowed or not.\n",
    "\n",
    "- Platforms\n",
    "If you have hosted your python code on a 3rd party sever, you might not be allowed to perform web scraping. Please always read terms and condiition of your serverice providers if you are allowed to reun bots or not.\n",
    "\n",
    "- All examples shown in these notes are for educational purpose and are governed by the license mention in LICENSE file. In short, we do not take any responislibty of the damage cause by instruction in this book and examples shown here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d6d7b0b-06c7-4903-9359-9982b3cc7b4c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Tips to imporve your webscraping\n",
    "- try to randomize yout request\n",
    "    - time e.g. with sleep (1)\n",
    "    - random\n",
    "- User User agent /Header\n",
    "- User Cache e.g. lru_cache\n",
    "    - if you have acceessed website1.com then next time the same website will not be crawled but get the value from cach.\n",
    "- vpn - ip shuffling\n",
    "- selinium python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c49fbe70-928d-4cbe-ba50-8cec276ebf90",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "from functools import lru_cache\n",
    "\n",
    "@lru_cache\n",
    "def check_archive_data(domain):\n",
    "    url = f\"http://archive.org/wayback/available?url={domain}&timestamp=19900101\"\n",
    "    r = requests.get(url)\n",
    "    archive_data = json.loads(r.text)\n",
    "    return archive_data['archived_snapshots']['closest']['timestamp']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e95d0f76-f07c-4c38-a5b3-304e4e3f17cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "check_archive_data(domain=\"https://www.python.org\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e61117-6af0-43ca-8a2a-a7c5c19dbdd2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import random\n",
    "for domain in [\"https://www.seowings.org\", \"https://www.seowings.org\"]:\n",
    "    time.sleep(3*random.random())\n",
    "    print (check_archive_data(domain))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ff4a6d-e4ae-4abe-8d03-284ff4686866",
   "metadata": {
    "tags": []
   },
   "source": [
    "## How to Store Data?\n",
    "\n",
    "There are differnt ways to save or represent yor data in python and several datatypes are avialble for that. \n",
    "\n",
    "For example if you have list of urls/website then you are good to go with list data type.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ecd9fb0d-8025-437f-80bd-81f8c61ace20",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "web_links = [\"website1.com\", \"website2.com\", \"website3.com\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325713ef-c3dd-4c8a-9595-9b67700fdba3",
   "metadata": {
    "tags": []
   },
   "source": [
    "However, if you need to hash a value (key,value pair) then you are good to use dictionaires.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c8454237-26ae-4d2d-9bfc-8c0a1fd246a4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "websites = {\n",
    "    \"website1\" : [1,2,3], \n",
    "    \"website2\" : [3.2,4.9, 4],\n",
    "    \"website3\" : [1.9,2.1,3.3]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a0dbda5-ad67-4dfb-9da5-a07ced05f5b3",
   "metadata": {
    "tags": []
   },
   "source": [
    "## How to Check/Parse URL?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3857a04-1a6f-484e-9da4-6b9df3f37a42",
   "metadata": {},
   "source": [
    "You basically need to check the netloc and scheme to make sure that a url is valid. You can also used python packages which are specialized in checking the validity of a url. This has itself became a complicated topic to to large availabiltiy of custom domain names, handshake domains and etc. \n",
    "\n",
    "As a bruteforce check, you can parse the url and look for its netloc to quickly check the validity of a url. This approach is very handy if you know that input url is probaly correct and a simple check is required to make sure it doesnt break your algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0e13e3d8-79d6-4549-bcd3-d947e376e29f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ParseResult(scheme='https', netloc='serpwings.com', path='/software/simply-static-netlify-callback/', params='', query='', fragment='')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Standard Library\n",
    "from urllib import parse\n",
    "parse.urlparse(\"https://serpwings.com/software/simply-static-netlify-callback/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "10a933c9-84a0-457d-987f-ddbc74c203e7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ParseResult(scheme='https', netloc='www.amazon.com', path='/deal/4ab5cff9', params='', query='showVariations=true', fragment='')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parse.urlparse(\"https://www.amazon.com/deal/4ab5cff9?showVariations=true\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d96736d3-a840-4e74-83e3-d5308fb2d682",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ParseResult(scheme='https', netloc='www.googletagmanager.com', path='/gtag/js', params='', query='id=G-R87TYPSWPH', fragment='')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parse.urlparse(\"https://www.googletagmanager.com/gtag/js?id=G-R87TYPSWPH\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "831f5332-521d-41fa-8dc3-b1fd27e6cc84",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "valid\n"
     ]
    }
   ],
   "source": [
    "# Scheme\n",
    "# Netloc\n",
    "\n",
    "parsed_value = parse.urlparse(\"https://serpwings.com\")\n",
    "\n",
    "if parsed_value.scheme:\n",
    "    print (\"valid\")\n",
    "else:\n",
    "    print (\"not valid\")\n",
    "    \n",
    "is_valid = True if parsed_value.scheme else False\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "631ac0c1-f40d-45f7-afb1-1a4f801d4810",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'valid'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def is_url_valid(url):\n",
    "    parsed_value = parse.urlparse(url)\n",
    "    return \"valid\" if parsed_value.scheme and parsed_value.netloc else \"not valid\"\n",
    "    \n",
    "url = \"https://google.com\"\n",
    "url_validity = is_url_valid(url)\n",
    "url_validity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "88f5dbb1-07fc-43c1-b7f8-a1fc6a7bb68d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://archive.org/wayback/available?url=https://seowings.org&timestamp=19900101'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain = \"https://seowings.org\"\n",
    "\n",
    "url = f\"http://archive.org/wayback/available?url={domain}&timestamp=19900101\"\n",
    "\n",
    "url \n",
    "#is_url_valid(url) # helper functions - utility functions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "90beba45-b128-474e-837c-3f318186daf5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (2.30.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests) (2.0.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests) (3.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests) (2023.5.7)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0.1 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "# 3rd party packages \n",
    "# pip install requests\n",
    "\n",
    "!pip install requests\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "49100409-6648-4346-a3b3-e5f74c30d903",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "r = requests.get('https://requests.readthedocs.io/')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2816aa99-7f7a-4061-b0e6-f40769fb2cbd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r\n",
    "\n",
    "# 200 ok\n",
    "# 301 302 redirect\n",
    "# 404 page doenst not exit\n",
    "# 410 permanently deleted\n",
    "# 500 server error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ca571911-7fbb-425d-a051-8158d49588c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Date': 'Wed, 17 May 2023 17:36:03 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'x-amz-id-2': 'ZBFWdzKoSl8WXLUGcchqGnLq4f/c2n1FiWK+YZjWey0PO21giqtJLOjTB3uZ999fbgujdLhMf8A=', 'x-amz-request-id': '7SRWZAHDWZFGQEP6', 'Last-Modified': 'Mon, 15 May 2023 15:05:40 GMT', 'ETag': 'W/\"3bc64e8dfb9ad78da7dfc4dd7f82af3f\"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-meta-mtime': '1684163110.360453993', 'X-Served': 'Nginx-Proxito-Sendfile', 'X-Backend': 'web-i-09882d230536e047f', 'X-RTD-Project': 'requests', 'X-RTD-Version': 'latest', 'X-RTD-Path': '/proxito/html/requests/latest/index.html', 'X-RTD-Domain': 'requests.readthedocs.io', 'X-RTD-Version-Method': 'path', 'X-RTD-Project-Method': 'public_domain', 'Referrer-Policy': 'no-referrer-when-downgrade', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'CDN-Cache-Control': 'public', 'Content-Encoding': 'gzip', 'Cache-Control': 'max-age=1200', 'CF-Cache-Status': 'HIT', 'Age': '6832', 'Server': 'cloudflare', 'CF-RAY': '7c8d9ed88ce13649-FRA', 'alt-svc': 'h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.headers\n",
    "# dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "13f464ce-3d1e-4e5d-9a60-d6ff984837ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.timedelta(microseconds=252987)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "r = requests.get('https://seowings.org/safdsafsafd')\n",
    "\n",
    "r.history\n",
    "r.elapsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "df2733d2-7912-45c1-857c-7951c762dc70",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "404"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# http with www\n",
    "# http without www\n",
    "# https with www\n",
    "# https without www\n",
    "\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4156863e-5ec6-4183-a3f4-08a3d6544932",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.seowings.org/safdsafsafd'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f6ac79f-f10f-407a-b5eb-c873429cf0bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "r.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9d51c20e-f4d5-4a65-ab8c-a10058bcb69d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"url\": \"https://seowings.org\", \"archived_snapshots\": {\"closest\": {\"status\": \"200\", \"available\": true, \"url\": \"http://web.archive.org/web/20210802110250/https://www.seowings.org/\", \"timestamp\": \"20210802110250\"}}, \"timestamp\": \"19900101\"}'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "domain = \"https://seowings.org\"\n",
    "url = f\"http://archive.org/wayback/available?url={domain}&timestamp=19900101\"\n",
    "r = requests.get(url)\n",
    "r.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6a96b217-d583-4384-b393-5f5e5e39084f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "archive_data = json.loads(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9cd57cd7-d993-481e-9b6b-346df01d05a3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'url': 'https://seowings.org',\n",
       " 'archived_snapshots': {'closest': {'status': '200',\n",
       "   'available': True,\n",
       "   'url': 'http://web.archive.org/web/20210802110250/https://www.seowings.org/',\n",
       "   'timestamp': '20210802110250'}},\n",
       " 'timestamp': '19900101'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "archive_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "aba7f391-140f-468d-b072-ef9f2b6307a7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'20210802110250'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "archive_data['archived_snapshots']['closest']['timestamp']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "04924ef5-151c-46a3-9ad6-0028f9013884",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "def check_archive_data(domain):\n",
    "    url = f\"http://archive.org/wayback/available?url={domain}&timestamp=19900101\"\n",
    "    r = requests.get(url)\n",
    "    archive_data = json.loads(r.text)\n",
    "    return archive_data['archived_snapshots']['closest']['timestamp']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f0368574-546b-4829-8dd6-e85b5b0a154c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'20150831144010'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_archive_data(domain = \"https://www.islamicfinanceguru.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "14753024-bded-4ff0-9123-4b236614b125",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://stackoverflow.com 20040203071020\n",
      "https://dw.de 20110324045535\n",
      "https://seowings.org 20210802110250\n"
     ]
    }
   ],
   "source": [
    "\n",
    "list_of_competitors = [\"https://stackoverflow.com\", \"https://dw.de\", \"https://seowings.org\"]\n",
    "\n",
    "for competitor in list_of_competitors:\n",
    "    print (competitor, check_archive_data(domain = competitor))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ff75864b-94e2-4eaf-8dad-994853f65563",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'whois_checker' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[1], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43mwhois_checker\u001b[49m\n\u001b[0;32m      2\u001b[0m word_counter\n\u001b[0;32m      3\u001b[0m url_extractors\n",
      "\u001b[1;31mNameError\u001b[0m: name 'whois_checker' is not defined"
     ]
    }
   ],
   "source": [
    "whois_checker\n",
    "word_counter\n",
    "url_extractors\n",
    "external\n",
    "internal\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acd4cba3-737b-4dae-a3dd-f89cbf793e0d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3dbe4e3f-b82d-480e-892b-a7204ba1ea1b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "zameean_url = \"https://www.zameen.com/Homes/Lahore_Shadman-15803-1.html\"\n",
    "r = requests.get(zameen_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "79b028a8-1db4-4cb2-9b49-16e5f1e6500e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6175b0bf-0527-4a3d-8c3b-b9d3b8728fd5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zameen.com%2FProperty%2Fshadman_shadman_2_old_house_at_only_plot_price-45204154-8950-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_in_stunning_shadman_is_available_for_sale-45203936-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_get_in_touch_now_to_buy_a_house_in_shadman-45203934-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_available_for_sale_in_shadman-45203931-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_well_constructed_house_available_for_sale_in_shadman-45203930-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_in_shadman_is_available-45203929-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_in_shadman_for_sale-45203924-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_is_available_for_sale-45203923-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_ready_to_buy_a_house_in_shadman_shadman-45203921-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_get_in_touch_now_to_buy_a_house_in_shadman_shadman-45203920-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_buying_a_house_in_shadman_shadman_-45203919-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_a_good_option_for_sale_is_the_house_available_in_shadman_in_shadman-45203918-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_75_kanal_house_for_sale_in_shadman-45203917-15803-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_buying_a_house_in_shadman_-45203916-15803-1.html\n",
      "zameen.com%2FProperty%2Fshadman_shadman_1_10_marla_house_for_sale_shadman_ii_lahore-36073783-497-1.html\n",
      "zameen.com%2FProperty%2Fshadman_shadman_2_1_kanal_7_marla_house_for_sale_shadman_2_lahore-36073804-8950-1.html\n",
      "zameen.com%2FProperty%2Fshadman_shadman_2_at_canal_road_25_marla_semi_commercial_house-36768114-8950-1.html\n",
      "zameen.com%2FProperty%2Flahore_shadman_1_bedroom_luxurious_apartment_on_easy_installments-43790016-15803-1.html\n",
      "zameen.com%2FProperty%2Fshadman_shadman_2_2_kanal_house_for_sale_shadman_2-36073817-8950-1.html\n",
      "zameen.com%2FProperty%2Fshadman_shadman_1_6_kanal_house_for_sale_main_jail_road_shadman_lahore-36073800-497-1.html\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from urllib import parse\n",
    "\n",
    "soup = BeautifulSoup(r.text, 'html.parser')\n",
    "\n",
    "for link in soup.find_all('a'):\n",
    "    url = link.get(\"href\")\n",
    "    if url:\n",
    "        if url.startswith(\"https://wa.me\"):\n",
    "            url_query = parse.urlparse(url).query\n",
    "            print (url_query.split(\"www.\")[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f5f23e83-11d6-4dc5-83be-dfb16a152868",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2+2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57f012e0-0e85-4cfa-8f54-505e7e9657e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "504854de-8d2f-4197-87fb-8c13a6688962",
   "metadata": {
    "tags": []
   },
   "source": [
    "## How to Prase HTML data\n",
    "\n",
    "There are several options available to parse html data. You can use Python Standard Library Package (minidom) or any 3rd party package available on PiPy. In most of the cases you are good to go with [**Beautifulsoup4**](https://pypi.org/project/beautifulsoup4/), which is an old and [well maintinaded and documented Python project](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). You can install it using following command on commandline.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "206dbf46-2611-4e01-9f2b-69a32d587b48",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "^C\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (4.12.2)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from beautifulsoup4) (2.4.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "53b2043e-d234-4071-956c-bf1326807955",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "ua = \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36\"\n",
    "\n",
    "r = requests.get('https://seowings.org/', headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "c729b28a-a916-470a-88da-7d094c1aa961",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.timedelta(microseconds=212813)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.elapsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "50812b32-29ce-405f-b489-bddba3429f7b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (4.12.2)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from beautifulsoup4) (2.4.1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0.1 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "!pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d82d6794-f228-4781-be2a-c8af83e61054",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "3cabb429-c7b9-4439-9ca3-f9c43b0c8bd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(r.text, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77775ddf-2f2a-4382-b4c8-5515e339ed80",
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "e608b782-8e4c-4560-94a9-e4a490e897ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>Digital Marketing with Data Science: SEO Wings</title>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "4c723e6d-d3d4-4c01-8e1b-030c2bd70e0a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Digital Marketing with Data Science: SEO Wings'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "14043a6c-42ba-4e4c-bea6-45e625b9823c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.seowings.org/\n",
      "/\n",
      "/simply-static-tutorial/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/download/\n",
      "https://www.seowings.org/simply-static-tutorial/\n",
      "https://www.seowings.org/simply-static-tutorial/\n",
      "https://www.seowings.org/wordpress/\n",
      "/terms-of-services/\n",
      "/privacy-policy/\n",
      "/unsubscribe/\n",
      "https://twitter.com/seowings_org\n",
      "https://www.facebook.com/seowings.org\n",
      "https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA\n",
      "https://github.com/seowings\n",
      "/wordpress/\n",
      "/web-domains/\n",
      "/software/\n",
      "/search-engine-optimization/\n",
      "/static-websites/\n",
      "/research-work/\n",
      "/static-web-hosting/\n",
      "/static-web-hosting/\n",
      "/static-websites/\n",
      "/best-static-website-hosting-services/\n",
      "/best-static-website-hosting-services/\n",
      "/static-websites/\n",
      "/sitemap-seo/\n",
      "/sitemap-seo/\n",
      "/search-engine-optimization/\n",
      "/what-is-a-domain-name/\n",
      "/what-is-a-domain-name/\n",
      "/web-domains/\n",
      "/website-architecture-and-seo/\n",
      "/website-architecture-and-seo/\n",
      "/search-engine-optimization/\n",
      "/seoquake-tutorial/\n",
      "/seoquake-tutorial/\n",
      "/search-engine-optimization/\n",
      "/web-server-log-files/\n",
      "/web-server-log-files/\n",
      "/static-websites/\n",
      "/robots-txt/\n",
      "/robots-txt/\n",
      "/search-engine-optimization/\n",
      "/best-software-for-installing-wordpress-on-localhost/\n",
      "/best-software-for-installing-wordpress-on-localhost/\n",
      "/wordpress/\n",
      "/seo-minion-tutorial/\n",
      "/seo-minion-tutorial/\n",
      "/search-engine-optimization/\n",
      "/seo-and-web-hosting/\n",
      "/seo-and-web-hosting/\n",
      "/search-engine-optimization/\n",
      "/local-wordpress-hosting-on-virtual-machines/\n",
      "/local-wordpress-hosting-on-virtual-machines/\n",
      "/wordpress/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/wordpress/\n",
      "/competitor-analysis-using-sitemaps/\n",
      "/competitor-analysis-using-sitemaps/\n",
      "/search-engine-optimization/\n",
      "/static-wordpress/\n",
      "/static-wordpress/\n",
      "/wordpress/\n",
      "/advantages-of-local-wordpress-hosting/\n",
      "/advantages-of-local-wordpress-hosting/\n",
      "/wordpress/\n",
      "https://twitter.com/seowings_org\n",
      "https://www.facebook.com/seowings.org\n",
      "https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA\n",
      "https://github.com/seowings\n",
      "/privacy-policy/\n",
      "/terms-of-services/\n",
      "/contact-us/\n",
      "/download/\n",
      "/privacy-policy/\n",
      "/terms-of-services/\n"
     ]
    }
   ],
   "source": [
    "for link in soup.find_all('a'):\n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "bb3b6d88-3436-4d1e-a5b9-1a54bf60678d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "79"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(soup.find_all('a'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "777997d1-e067-4a62-9913-a8e9cd50eeac",
   "metadata": {},
   "source": [
    "## Types of Links\n",
    "1. internal \n",
    "2. external\n",
    "3. anchors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "2aa0a65f-24df-4509-bfc6-a8f9e29df7d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from urllib import parse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "6ebe681c-1517-4f57-b299-3f45324d2c1e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "seowings_url = parse.urlparse(\"https://www.seowings.org\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "64c54735-a9aa-4c8d-8bee-cf9bcda32c02",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'www.seowings.org'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seowings_url.netloc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "67ee8592-6b48-4ca1-a922-a11b886c1e94",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "external_links = set()\n",
    "\n",
    "for link in soup.find_all('a'): #extract all urls <a href=\"https://www.seowings.org/test\" > text </a>\n",
    "    \n",
    "    link_href = link.get('href') # https://www.seowings.org/test\n",
    "    \n",
    "    if seowings_url.netloc not in link_href and link_href.startswith(\"https\"):\n",
    "        external_links.add(link_href)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "6d676fdb-28fb-45bd-8a5b-69049cb5dd4a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'https://github.com/seowings',\n",
       " 'https://twitter.com/seowings_org',\n",
       " 'https://www.facebook.com/seowings.org',\n",
       " 'https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA'}"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1134a7e4-0c1d-4e38-8c26-2dec71748e99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# list comprehensions, lambda functions , decorators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "917c731a-a682-4015-ae5b-c4fbf82109a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WordPress Hosting on Netlify Using Simply Static 7\n",
      "Newsletter Sign Up 3\n",
      "When and Why to Use Static Web Hosting? 8\n",
      "11 Best JAMStack Web Hosting Providers (2023 update) 8\n",
      "How Sitemaps can be Used for Search Engine Optimization? 9\n",
      "What is a Domain Name - Basic Tutorial for Newbies 10\n",
      "Website Architecture and Its Importance for Technical SEO? 8\n",
      "SEOquake-Best Web Extension for Search Engine Optimization? 7\n",
      "What Are Web Server Log Files? 6\n",
      "What is Robots.txt File and How to Manage Crawlers? 9\n",
      "10 Best Software for Installing WordPress on Localhost 8\n",
      "SEO Minion-Web extension for Content Creators and SEO 8\n",
      "Importance of Web Hosting for Search Engine Optimization? 8\n",
      "How To Host WordPress on Virtual Machines? 7\n",
      "WordPress on Netlify - Step by Step Complete Guide (2022) 10\n",
      "How to use Website Sitemaps for Competitor Analysis? 8\n",
      "Static WordPress-An Introductory Tutorial [2023] 5\n",
      "What Are The Advantage Of Insalling WordPress on Localhost? 9\n"
     ]
    }
   ],
   "source": [
    "for h2 in soup.find_all('h2'): #extract all urls <a href=\"https://www.seowings.org/test\" > text </a>\n",
    "    h2_string = h2.string\n",
    "    h2_string.split(' ')\n",
    "    print (h2_string, len(h2_string.split(' ')) )\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f7dc09f7-b30c-494c-bc86-1c7a90412964",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "date = '20150831144010'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e62dcf0e-5dee-47e0-b473-ff52872e2549",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'str' object cannot be interpreted as an integer",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[8], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mdatetime\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m datetime\n\u001b[1;32m----> 3\u001b[0m \u001b[43mdatetime\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfromtimestamp\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdate\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[1;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "datetime.fromtimestamp(date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f8505f9-10ad-4dbd-a4bc-fdf3bfe63a1a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e005f7a3-2b9a-4f52-af6c-4cd036445c9c",
   "metadata": {},
   "source": [
    "# Beautiful Soup and Data Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "97e35d89-4034-4ad6-afde-b1f691075981",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Standard Libaries\n",
    "from functools import lru_cache\n",
    "\n",
    "# 3rd Party Libraries\n",
    "import requests\n",
    "from requests.adapters import HTTPAdapter\n",
    "\n",
    "# Own libraries\n",
    "\n",
    "@lru_cache\n",
    "def download_url(url):\n",
    "    headers =  {\n",
    "        \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36\"\n",
    "    }\n",
    "\n",
    "    max_retires = 5\n",
    "    s = requests.Session()\n",
    "    s.mount(url, HTTPAdapter(max_retries=max_retires))\n",
    "    return s.get(url, headers=headers)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "13f5e273-e8ad-47e7-a3f3-47adc733cf16",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url = \"https://www.jang.com.pk\"\n",
    "r = download_url(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c68bb0fa-10ba-49e5-a00a-d2ca58e63ba4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bf2821d-8f19-47b3-80ac-6b5eaae4ca6a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Response [301]>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b2e83c8a-bc14-4e9f-9215-1391980d4c28",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://jang.com.pk/'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6d79f905-88b4-434a-a2bb-1967449e3b4b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Date': 'Tue, 06 Jun 2023 16:53:42 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'X-Powered-By': 'PHP/7.1.33', 'Expires': 'Tue, 06 Jun 2023 16:56:42 GMT', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'public, max-age=180', 'X-Varnish': '97540791 97857681', 'Via': '1.1 varnish (Varnish/6.0)', 'X-Cache': 'HIT', 'X-Cache-Hits': '99', 'CF-Cache-Status': 'EXPIRED', 'Last-Modified': 'Tue, 06 Jun 2023 16:48:20 GMT', 'Server': 'cloudflare', 'CF-RAY': '7d322c4d7ab237f7-FRA', 'Content-Encoding': 'gzip', 'alt-svc': 'h3=\":443\"; ma=86400'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9a4580b6-3920-44f1-891e-52e5531100d8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.timedelta(microseconds=440207)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.elapsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "90f50cba-f2d0-49a4-a8cc-8ee37cbd458b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.is_redirect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b2072c37-1b7c-4549-9080-1b29f3e64aeb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6c7ca643-474c-435d-b7ba-3eafe6537eda",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.jang.com.pk/\n"
     ]
    }
   ],
   "source": [
    "for h in r.history:\n",
    "    print (h.url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "13dfb328-b87d-4661-a64e-68c1595cfa07",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (4.12.2)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\serpwings\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from beautifulsoup4) (2.4.1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0.1 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "!pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3717d1-cba9-494e-b502-33cd88bb2131",
   "metadata": {},
   "source": [
    "# Beautiful Soup installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8a8cc198-bc53-42f0-8691-36696835697c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "90931987-cf42-45e5-9c0d-9d0baee33cc4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url = \"https://www.seowings.org\"\n",
    "r = download_url(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "562bd7f9-74d7-4a4d-8f46-6f6202da6548",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "seowings = BeautifulSoup(r.text,  'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ff0617c0-03be-4cee-afa4-73e1e41590fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>Digital Marketing with Data Science: SEO Wings</title>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seowings.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a16e7b07-064f-44b1-a1a0-f1b8879e1920",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Digital Marketing with Data Science: SEO Wings'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seowings.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "81600f78-0b00-4fea-8f14-3b7793eb1f97",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.seowings.org/\n",
      "/\n",
      "/simply-static-tutorial/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/download/\n",
      "https://www.seowings.org/simply-static-tutorial/\n",
      "https://www.seowings.org/simply-static-tutorial/\n",
      "https://www.seowings.org/wordpress/\n",
      "/terms-of-services/\n",
      "/privacy-policy/\n",
      "/unsubscribe/\n",
      "https://twitter.com/seowings_org\n",
      "https://www.facebook.com/seowings.org\n",
      "https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA\n",
      "https://github.com/seowings\n",
      "/wordpress/\n",
      "/web-domains/\n",
      "/software/\n",
      "/search-engine-optimization/\n",
      "/static-websites/\n",
      "/research-work/\n",
      "/static-web-hosting/\n",
      "/static-web-hosting/\n",
      "/static-websites/\n",
      "/best-static-website-hosting-services/\n",
      "/best-static-website-hosting-services/\n",
      "/static-websites/\n",
      "/sitemap-seo/\n",
      "/sitemap-seo/\n",
      "/search-engine-optimization/\n",
      "/what-is-a-domain-name/\n",
      "/what-is-a-domain-name/\n",
      "/web-domains/\n",
      "/website-architecture-and-seo/\n",
      "/website-architecture-and-seo/\n",
      "/search-engine-optimization/\n",
      "/seoquake-tutorial/\n",
      "/seoquake-tutorial/\n",
      "/search-engine-optimization/\n",
      "/web-server-log-files/\n",
      "/web-server-log-files/\n",
      "/static-websites/\n",
      "/robots-txt/\n",
      "/robots-txt/\n",
      "/search-engine-optimization/\n",
      "/best-software-for-installing-wordpress-on-localhost/\n",
      "/best-software-for-installing-wordpress-on-localhost/\n",
      "/wordpress/\n",
      "/seo-minion-tutorial/\n",
      "/seo-minion-tutorial/\n",
      "/search-engine-optimization/\n",
      "/seo-and-web-hosting/\n",
      "/seo-and-web-hosting/\n",
      "/search-engine-optimization/\n",
      "/local-wordpress-hosting-on-virtual-machines/\n",
      "/local-wordpress-hosting-on-virtual-machines/\n",
      "/wordpress/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/wordpress/\n",
      "/competitor-analysis-using-sitemaps/\n",
      "/competitor-analysis-using-sitemaps/\n",
      "/search-engine-optimization/\n",
      "/static-wordpress/\n",
      "/static-wordpress/\n",
      "/wordpress/\n",
      "/advantages-of-local-wordpress-hosting/\n",
      "/advantages-of-local-wordpress-hosting/\n",
      "/wordpress/\n",
      "https://twitter.com/seowings_org\n",
      "https://www.facebook.com/seowings.org\n",
      "https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA\n",
      "https://github.com/seowings\n",
      "/privacy-policy/\n",
      "/terms-of-services/\n",
      "/contact-us/\n",
      "/download/\n",
      "/privacy-policy/\n",
      "/terms-of-services/\n"
     ]
    }
   ],
   "source": [
    "from time import sleep\n",
    "\n",
    "for anchor in seowings.find_all(\"a\"):\n",
    "    url = anchor.get(\"href\")\n",
    "    sleep(1)\n",
    "    response = download_url(url)\n",
    "    print (url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d2941f00-c678-458a-ad10-4a0981856a6d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#list comprehensions\n",
    "\n",
    "from time import sleep\n",
    "\n",
    "all_urls = []\n",
    "for anchor in seowings.find_all(\"a\"):\n",
    "    all_urls.append(anchor.get(\"href\"))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f0946157-1a8e-4b04-8e0e-083336405cb2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.seowings.org/',\n",
       " '/',\n",
       " '/simply-static-tutorial/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/download/',\n",
       " 'https://www.seowings.org/simply-static-tutorial/',\n",
       " 'https://www.seowings.org/simply-static-tutorial/',\n",
       " 'https://www.seowings.org/wordpress/',\n",
       " '/terms-of-services/',\n",
       " '/privacy-policy/',\n",
       " '/unsubscribe/',\n",
       " 'https://twitter.com/seowings_org',\n",
       " 'https://www.facebook.com/seowings.org',\n",
       " 'https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA',\n",
       " 'https://github.com/seowings',\n",
       " '/wordpress/',\n",
       " '/web-domains/',\n",
       " '/software/',\n",
       " '/search-engine-optimization/',\n",
       " '/static-websites/',\n",
       " '/research-work/',\n",
       " '/static-web-hosting/',\n",
       " '/static-web-hosting/',\n",
       " '/static-websites/',\n",
       " '/best-static-website-hosting-services/',\n",
       " '/best-static-website-hosting-services/',\n",
       " '/static-websites/',\n",
       " '/sitemap-seo/',\n",
       " '/sitemap-seo/',\n",
       " '/search-engine-optimization/',\n",
       " '/what-is-a-domain-name/',\n",
       " '/what-is-a-domain-name/',\n",
       " '/web-domains/',\n",
       " '/website-architecture-and-seo/',\n",
       " '/website-architecture-and-seo/',\n",
       " '/search-engine-optimization/',\n",
       " '/seoquake-tutorial/',\n",
       " '/seoquake-tutorial/',\n",
       " '/search-engine-optimization/',\n",
       " '/web-server-log-files/',\n",
       " '/web-server-log-files/',\n",
       " '/static-websites/',\n",
       " '/robots-txt/',\n",
       " '/robots-txt/',\n",
       " '/search-engine-optimization/',\n",
       " '/best-software-for-installing-wordpress-on-localhost/',\n",
       " '/best-software-for-installing-wordpress-on-localhost/',\n",
       " '/wordpress/',\n",
       " '/seo-minion-tutorial/',\n",
       " '/seo-minion-tutorial/',\n",
       " '/search-engine-optimization/',\n",
       " '/seo-and-web-hosting/',\n",
       " '/seo-and-web-hosting/',\n",
       " '/search-engine-optimization/',\n",
       " '/local-wordpress-hosting-on-virtual-machines/',\n",
       " '/local-wordpress-hosting-on-virtual-machines/',\n",
       " '/wordpress/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/wordpress/',\n",
       " '/competitor-analysis-using-sitemaps/',\n",
       " '/competitor-analysis-using-sitemaps/',\n",
       " '/search-engine-optimization/',\n",
       " '/static-wordpress/',\n",
       " '/static-wordpress/',\n",
       " '/wordpress/',\n",
       " '/advantages-of-local-wordpress-hosting/',\n",
       " '/advantages-of-local-wordpress-hosting/',\n",
       " '/wordpress/',\n",
       " 'https://twitter.com/seowings_org',\n",
       " 'https://www.facebook.com/seowings.org',\n",
       " 'https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA',\n",
       " 'https://github.com/seowings',\n",
       " '/privacy-policy/',\n",
       " '/terms-of-services/',\n",
       " '/contact-us/',\n",
       " '/download/',\n",
       " '/privacy-policy/',\n",
       " '/terms-of-services/']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "63a15d4a-7513-4849-9b29-b4e4fe7d0d62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#list comprehensions\n",
    "\n",
    "all_urls = set([anchor.get(\"href\") for anchor in seowings.find_all(\"a\")])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e7745f16-3e22-4d51-822b-9c395098d639",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'/',\n",
       " '/advantages-of-local-wordpress-hosting/',\n",
       " '/best-software-for-installing-wordpress-on-localhost/',\n",
       " '/best-static-website-hosting-services/',\n",
       " '/competitor-analysis-using-sitemaps/',\n",
       " '/contact-us/',\n",
       " '/download/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/local-wordpress-hosting-on-virtual-machines/',\n",
       " '/privacy-policy/',\n",
       " '/research-work/',\n",
       " '/robots-txt/',\n",
       " '/search-engine-optimization/',\n",
       " '/seo-and-web-hosting/',\n",
       " '/seo-minion-tutorial/',\n",
       " '/seoquake-tutorial/',\n",
       " '/simply-static-tutorial/',\n",
       " '/sitemap-seo/',\n",
       " '/software/',\n",
       " '/static-web-hosting/',\n",
       " '/static-websites/',\n",
       " '/static-wordpress/',\n",
       " '/terms-of-services/',\n",
       " '/unsubscribe/',\n",
       " '/web-domains/',\n",
       " '/web-server-log-files/',\n",
       " '/website-architecture-and-seo/',\n",
       " '/what-is-a-domain-name/',\n",
       " '/wordpress/',\n",
       " 'https://github.com/seowings',\n",
       " 'https://twitter.com/seowings_org',\n",
       " 'https://www.facebook.com/seowings.org',\n",
       " 'https://www.seowings.org/',\n",
       " 'https://www.seowings.org/simply-static-tutorial/',\n",
       " 'https://www.seowings.org/wordpress/',\n",
       " 'https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA'}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "95cf9458-c3c3-4fe5-826f-28d1ca887b97",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "github.com\n",
      "youtube.com\n",
      "facebook.com\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "035370fb-48ac-4bf9-baae-e7ddc88736fe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "b5e0a392-b140-4b32-a8bf-06461e0fcafb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4e3d11b-9065-4b79-9675-d3d8c7e456ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "triple = \"\"\" This is a string \"\"\" # comments, documentation or multiline \n",
    "doub = \" string \" # standards \n",
    "sing = 'string' # standard but is not used that often in modren python\n",
    "\n",
    "f\"This is my {'t'}\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b274459-3943-4396-8473-869344c3feb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Text\n",
    "    - CSS\n",
    "        - text\n",
    "    - HTML\n",
    "        - Beuatifulsoup\n",
    "    - xml\n",
    "        - Beuatifulsoup\n",
    "    - JS\n",
    "        text\n",
    "    - json\n",
    "        - JSON libary\n",
    "    - ----\n",
    "# Images\n",
    "    - webp\n",
    "        - PIL \n",
    "    - png\n",
    "    --\n",
    "# Binary -fonts\n",
    "    - Google/fonts\n",
    "        - r.content (not r.text)\n",
    "# Zip\n",
    "    - zip\n",
    "        use from zipfile import ZipFile\n",
    "\n",
    "with ZipFile('spam.zip') as myzip:\n",
    "    with myzip.open('eggs.txt') as myfile:\n",
    "        print(myfile.read())\n",
    "\n",
    "#PDF\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "fc951dda-0f56-438a-82ae-3ff88917c0ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bbc_xml_newsfeed = \"http://feeds.bbci.co.uk/news/rss.xml\"\n",
    "r = download_url(bbc_xml_newsfeed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "5c4b2d68-04d4-47da-8f3c-40b8b5d13d3e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Server': 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips', 'Content-Type': 'text/xml; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Cache-Control': 'max-age=181', 'Date': 'Tue, 06 Jun 2023 17:35:05 GMT', 'Content-Length': '6372', 'Connection': 'keep-alive', 'Timing-Allow-Origin': 'https://www.bbc.co.uk, https://www.bbc.com', 'Server-Timing': 'ak_p; desc=\"1686072905238_388358823_89668702_16_587_26_0_-\";dur=1'}"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "e64d25b0-93f3-488e-8537-7444e3b44463",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bbc_feed = BeautifulSoup(r.text,   'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "304c13c8-ba93-4469-ac05-ff8c0ad85638",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
       "<?xml-stylesheet title=\"XSL_formatting\" type=\"text/xsl\" href=\"/shared/bsp/xsl/rss/nolsol.xsl\"?>\n",
       "<rss version=\"2.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:media=\"http://search.yahoo.com/mrss/\">\n",
       "<channel>\n",
       "<title><![CDATA[BBC News - Home]]></title>\n",
       "<description><![CDATA[BBC News - Home]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/\n",
       "        <image/>\n",
       "<url>https://news.bbcimg.co.uk/nol/shared/img/bbc_news_120x60.gif</url>\n",
       "<title>BBC News - Home</title>\n",
       "<link/>https://www.bbc.co.uk/news/\n",
       "        \n",
       "        <generator>RSS for Node</generator>\n",
       "<lastbuilddate>Tue, 06 Jun 2023 17:32:47 GMT</lastbuilddate>\n",
       "<copyright><![CDATA[Copyright: (C) British Broadcasting Corporation, see http://news.bbc.co.uk/2/hi/help/rss/4498287.stm for terms and conditions of reuse.]]></copyright>\n",
       "<language><![CDATA[en-gb]]></language>\n",
       "<ttl>15</ttl>\n",
       "<item>\n",
       "<title><![CDATA[Prince Harry: I couldn't trust anybody due to phone hacking]]></title>\n",
       "<description><![CDATA[The Duke of Sussex is giving evidence in his case against Mirror Group Newspapers.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-65818521?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-65818521</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:39:34 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine dam: Thousands flee floods after dam collapse near Nova Kakhovka]]></title>\n",
       "<description><![CDATA[Around 40,000 people need to be evacuated, says Ukraine, after a collapse released a torrent of water.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65819591?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65819591</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 17:03:39 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Criminal investigation launched over royal escort crash]]></title>\n",
       "<description><![CDATA[Helen Holland, 81, died after being hit by a police motorcycle escorting the Duchess of Edinburgh]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-england-london-65826561?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-england-london-65826561</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 17:32:01 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Bournemouth beach boat operations suspended after deaths]]></title>\n",
       "<description><![CDATA[Sunnah Khan and Joe Abbess died after being pulled from the sea near Bournemouth Pier.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-england-dorset-65823704?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-england-dorset-65823704</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:20:17 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[PGA Tour & DP World Tour agree shock merger with LIV Golf to end split in golf]]></title>\n",
       "<description><![CDATA[The PGA Tour and DP World Tour agree to merge with Saudi Arabian-backed circuit LIV Golf in a deal that ends the acrimonious split in the game.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/golf/65825327?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/golf/65825327</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 17:16:19 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA['Ducking hell' to disappear from Apple autocorrect]]></title>\n",
       "<description><![CDATA[One of the most common swear words will no longer be automatically changed when you type out a message.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/technology-65823482?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/technology-65823482</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:23:33 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ferrier suspended from Commons over Covid rule breach]]></title>\n",
       "<description><![CDATA[The move will almost certainly lead to a by-election in Margaret Ferrier's constituency.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-scotland-65702252?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-scotland-65702252</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 14:12:12 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ian Blackford to stand down as SNP MP at next election]]></title>\n",
       "<description><![CDATA[The SNP's former Westminster leader says he will not stand in next year's general election.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-scotland-scotland-politics-65827165?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-scotland-scotland-politics-65827165</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:09:39 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Daniel Allen pleads guilty to killing Gossett family in house fire]]></title>\n",
       "<description><![CDATA[Denise Gossett, her children Roman and Sabrina and granddaughter Morgana were killed in the blaze.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-northern-ireland-65820447?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-northern-ireland-65820447</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 12:55:41 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Astrud Gilberto: The Girl from Ipanema singer dies at 83]]></title>\n",
       "<description><![CDATA[She was only in the studio as a translator, but ended up singing on the hit song and becoming a star.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65818566?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65818566</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 12:31:04 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Cuba Gooding Jr settles rape lawsuit ahead of civil trial]]></title>\n",
       "<description><![CDATA[The actor, who has faced multiple misconduct allegations, has said the encounter was consensual.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65825715?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65825715</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:37:46 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[DJ Michael Bibi, 32,  diagnosed with rare cancer]]></title>\n",
       "<description><![CDATA[The dance music DJ was due to play at the Parklife festival in Manchester and Glastonbury this summer.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/newsbeat-65820353?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/newsbeat-65820353</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 13:31:52 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Watch: Building floats along flooded Dnipro river]]></title>\n",
       "<description><![CDATA[The roof of a building is seen floating along the swollen river.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65825444?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65825444</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 14:47:31 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA['Thicko, cheat, underage drinker' - Key extracts from Prince Harry's statement ]]></title>\n",
       "<description><![CDATA['I couldn't trust anybody' - Harry on the impact of alleged unlawful information gathering on his life.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-65819707?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-65819707</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 15:32:14 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Robert Hanssen: The fake job that snared FBI agent who spied for Moscow]]></title>\n",
       "<description><![CDATA[Agents who helped to bring down one of the most notorious Cold War spies tell us how they did it.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-us-canada-65820220?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-us-canada-65820220</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 12:20:08 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[MoveIt hack: What action can data-breach victims take?]]></title>\n",
       "<description><![CDATA[Experts give advice to the more than 100,000 warned their personal data is in the hands of cyber-criminals.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/technology-65820603?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/technology-65820603</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 11:58:28 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA['I still speak to my brother Garvey Gayle after he killed our dad']]></title>\n",
       "<description><![CDATA[Essence Gayle still speaks regularly to her brother Garvey, two years after he killed their father.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-wales-65787917?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-wales-65787917</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 06:18:56 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Can UK’s Storm Shadow missiles change Ukraine fight against Russia?]]></title>\n",
       "<description><![CDATA[The UK is sending missiles to Ukraine – but how could Storm Shadows change its fight against Russia?]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65813770?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65813770</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 00:44:55 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine war: Holy Trinity painting on display in Moscow]]></title>\n",
       "<description><![CDATA[A fragile historic painting was moved from a museum to a church. What does it say about the war?]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65813549?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65813549</guid>\n",
       "<pubdate>Mon, 05 Jun 2023 16:29:01 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Jason Derulo: My 'most unsexy' business investment in a car wash]]></title>\n",
       "<description><![CDATA[The Savage Love singer says when wants to invest he looks for firms in sectors he understands.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/business-65645982?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/business-65645982</guid>\n",
       "<pubdate>Mon, 05 Jun 2023 23:03:21 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[India train crash: Haunting images of a deadly 2002 disaster]]></title>\n",
       "<description><![CDATA[Saibal Das survived a harrowing crash in Bihar and captured the first pictures of the tragedy.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-asia-india-65807458?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-asia-india-65807458</guid>\n",
       "<pubdate>Mon, 05 Jun 2023 21:07:56 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Chris Mason: Sunak gives prominence to vast migration challenge]]></title>\n",
       "<description><![CDATA[The prime minister visits Dover to highlight progress on a problem some say is impossible to solve.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-politics-65814270?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-politics-65814270</guid>\n",
       "<pubdate>Mon, 05 Jun 2023 16:32:20 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Love Island 2023 summer launch loses a million TV viewers]]></title>\n",
       "<description><![CDATA[The ratings decline comes amid suggestions that the ITV dating show has lost some of its appeal.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65818567?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65818567</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 15:40:58 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Phillip Schofield: ITV defends duty of care to staff at This Morning]]></title>\n",
       "<description><![CDATA[But an executive says remarks about aubergines by the show's editor were \"extremely ill-judged\".]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65818564?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65818564</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 11:23:21 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA['Now I finally know how my daughter sees this world']]></title>\n",
       "<description><![CDATA[A mum says a virtual reality headset gives her a new understanding of her girl's visual impairment.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-northern-ireland-65811076?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-northern-ireland-65811076</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 07:54:35 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Gonorrhoea and syphilis sex infections reach record levels in England]]></title>\n",
       "<description><![CDATA[There were 82,592 cases of gonorrhoea in 2022 - up 50% on the year before, officials say.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/health-65810160?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/health-65810160</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 09:39:55 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Diablo 4 character creator under the spotlight]]></title>\n",
       "<description><![CDATA[The new game's character creator mode has got people talking, but does it hit the mark for players?]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/newsbeat-65820043?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/newsbeat-65820043</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 10:47:33 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[CBI: Scandal-hit business group wins survival vote]]></title>\n",
       "<description><![CDATA[Members have backed the lobbying giant but it is not clear if the government will renew engagement.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/business-65809069?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/business-65809069</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 17:25:41 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Pat Nevin: My son's autism wasn't a secret]]></title>\n",
       "<description><![CDATA[Former footballer Pat Nevin has spoken for the first time about his grown-up son's autism diagnosis.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-scotland-65820771?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-scotland-65820771</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 12:16:50 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Starmer vows to stop oil and gas communities withering]]></title>\n",
       "<description><![CDATA[The Labour leader addressed the GMB after union leaders criticised his green energy plans.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-politics-65822211?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-politics-65822211</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:56:06 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[D-Day veteran Les Budding, 97, tells of first-wave role]]></title>\n",
       "<description><![CDATA[Former Royal Marine Les Budding tells of his role on D-Day on the 79th anniversary of the landings.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-england-lincolnshire-65815672?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-england-lincolnshire-65815672</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 05:08:32 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Dads' race at school sports day put me in A&E]]></title>\n",
       "<description><![CDATA[Two competitive dads issue a warning about the dangers of the sports day parents' race.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-scotland-edinburgh-east-fife-65791936?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-scotland-edinburgh-east-fife-65791936</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 07:39:49 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[The Full Monty's striking writers picket Sheffield premiere of Disney+ reboot]]></title>\n",
       "<description><![CDATA[Sheffield hosts the launch of a Disney+ series that reunites the characters from the hit 1997 film.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65808382?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65808382</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 07:38:58 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Charles Wotten: Liverpool race riot victim to get headstone]]></title>\n",
       "<description><![CDATA[Seaman Charles Wotten died in 1919 during some of the UK's worst racial violence.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-england-merseyside-65813499?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-england-merseyside-65813499</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 09:18:20 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Nursing: Strikes 'because patients are dying', says nurse]]></title>\n",
       "<description><![CDATA[A Cardiff-based nurse on the picket line says there simply is not enough staff to cope.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-wales-65815004?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-wales-65815004</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 13:57:36 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Snooker match-fixing scandal: Liang Wenbo and Li Hang handed lifetime bans]]></title>\n",
       "<description><![CDATA[Chinese players Liang Wenbo and Li Hang receive lifetime bans for match-fixing in snooker's biggest corruption scandal.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/snooker/65641122?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/snooker/65641122</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:54:20 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[French Open 2023 results: Novak Djokovic beats Karen Khachanov to reach semi-finals]]></title>\n",
       "<description><![CDATA[Novak Djokovic survives a French Open quarter-final scare from Karen Khachanov to keep his bid for a record 23rd men's Grand Slam title alive.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/tennis/65825658?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/tennis/65825658</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 17:31:51 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[French Open 2023: Aryna Sabalenka 'does not support' Belarus president Alexander Lukashenko]]></title>\n",
       "<description><![CDATA[Aryna Sabalenka says she does not support Belarusian president Alexander Lukashenko \"right now\" because of her nation's support of Russia's war in Ukraine.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/tennis/65826192?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/tennis/65826192</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:33:00 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA['He makes you feel invincible' - what will Postecoglou bring to Spurs?]]></title>\n",
       "<description><![CDATA[Ange Postecoglou will demand front-foot attacking football at Tottenham and those who don't follow will discover his ruthless streak.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/football/65822800?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/football/65822800</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 14:54:44 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Alexis Mac Allister: Liverpool medical for midfielder before move from Brighton]]></title>\n",
       "<description><![CDATA[Brighton's World Cup-winning midfielder Alexis Mac Allister is set for a medical before a £55m move to Liverpool.]]></description>\n",
       "<link/>https://www.bbc.co.uk/sport/football/65764964?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/sport/football/65764964</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 13:03:59 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Beaver explores flooded Kherson district after dam breach]]></title>\n",
       "<description><![CDATA[The streets in the district of Neftehavan have been flooded by rising water.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65827355?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65827355</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:50:14 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Why is the Kakhovka dam important?]]></title>\n",
       "<description><![CDATA[A flood of water has been released after a major dam in the Russian-controlled region of Kherson was hit.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65818705?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65818705</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 11:45:11 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine war: Wagner boss rubbishes Russian claims of Ukrainian casualties]]></title>\n",
       "<description><![CDATA[Moscow says it inflicted 3,700 casualties on Kyiv on Monday, but Yevgeny Prigozhin called the claim \"absurd\".]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65825304?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65825304</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 16:35:08 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine dam: Swans seen swimming through Nova Kakhovka]]></title>\n",
       "<description><![CDATA[The city was flooded after Ukraine accused Russia of destroying a huge dam nearby.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65818722?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65818722</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 10:31:48 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine war: Kyiv says troops advance on eastern front]]></title>\n",
       "<description><![CDATA[Ukraine says its has gained ground near Bakhmut, as Russia claims to have thwarted a new attack.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65813560?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65813560</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 03:24:40 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine war: 'It's better to die at home than abroad']]></title>\n",
       "<description><![CDATA[Thousands of Ukrainians are moving back to towns close to the front line, despite the dangers.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65778235?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65778235</guid>\n",
       "<pubdate>Sun, 04 Jun 2023 23:00:03 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Ukraine war: Anti-Kremlin fighters say Russian soldiers 'captured']]></title>\n",
       "<description><![CDATA[They say they will hand the men over to Ukraine, after accusing a Russian official of failing to meet them.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/world-europe-65804249?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/world-europe-65804249</guid>\n",
       "<pubdate>Sun, 04 Jun 2023 19:03:18 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Prince Harry: British press and government at rock bottom]]></title>\n",
       "<description><![CDATA[The Duke of Sussex says the press have \"got into bed\" with the government to ensure the status quo.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-politics-65822218?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-politics-65822218</guid>\n",
       "<pubdate>Tue, 06 Jun 2023 15:09:38 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Harry blames press intrusion for Chelsy break-up]]></title>\n",
       "<description><![CDATA[The prince said Ms Davy decided \"a royal life was not for her\", as his High Court case against Mirror Group Newspapers begins.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-65550021?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-65550021</guid>\n",
       "<pubdate>Wed, 10 May 2023 20:31:08 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Piers Morgan says he was not aware of phone hacking at Daily Mirror]]></title>\n",
       "<description><![CDATA[The former newspaper editor tells the BBC phone hacking is wrong and \"shouldn't have been happening\".]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/entertainment-arts-65531864?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/entertainment-arts-65531864</guid>\n",
       "<pubdate>Wed, 10 May 2023 17:55:48 GMT</pubdate>\n",
       "</item>\n",
       "<item>\n",
       "<title><![CDATA[Prince Harry surrounded by 'web of unlawful activity', court hears]]></title>\n",
       "<description><![CDATA[Prince Harry's lawyers claim his privacy was invaded by Mirror Group newspapers throughout his life.]]></description>\n",
       "<link/>https://www.bbc.co.uk/news/uk-65808672?at_medium=RSS&amp;at_campaign=KARANGA\n",
       "            <guid ispermalink=\"false\">https://www.bbc.co.uk/news/uk-65808672</guid>\n",
       "<pubdate>Mon, 05 Jun 2023 21:59:12 GMT</pubdate>\n",
       "</item>\n",
       "</channel>\n",
       "</rss>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bbc_feed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a7f9ad-7279-42e6-ad6a-a70b343bcf1e",
   "metadata": {},
   "source": [
    "robots_txt = \"http://seowings.org/robots.txt\"\n",
    "\n",
    "# to get robots.txt please use \n",
    "# https://docs.python.org/3/library/urllib.robotparser.html\n",
    "# or use any specific package e.g. https://github.com/serpwings/pyrobotstxt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "1a283731-89ee-4edc-bef8-a272144d9fd9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "github.com\n",
      "youtube.com\n",
      "facebook.com\n"
     ]
    }
   ],
   "source": [
    "social_media_websites = [\"github.com\", \"youtube.com\", \"facebook.com\"]\n",
    "\n",
    "for social in social_media_websites:\n",
    "    \n",
    "    print (social)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "60586b8b-61ea-4175-8cd2-16c367772601",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'/',\n",
       " '/advantages-of-local-wordpress-hosting/',\n",
       " '/best-software-for-installing-wordpress-on-localhost/',\n",
       " '/best-static-website-hosting-services/',\n",
       " '/competitor-analysis-using-sitemaps/',\n",
       " '/contact-us/',\n",
       " '/download/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/local-wordpress-hosting-on-virtual-machines/',\n",
       " '/privacy-policy/',\n",
       " '/research-work/',\n",
       " '/robots-txt/',\n",
       " '/search-engine-optimization/',\n",
       " '/seo-and-web-hosting/',\n",
       " '/seo-minion-tutorial/',\n",
       " '/seoquake-tutorial/',\n",
       " '/simply-static-tutorial/',\n",
       " '/sitemap-seo/',\n",
       " '/software/',\n",
       " '/static-web-hosting/',\n",
       " '/static-websites/',\n",
       " '/static-wordpress/',\n",
       " '/terms-of-services/',\n",
       " '/unsubscribe/',\n",
       " '/web-domains/',\n",
       " '/web-server-log-files/',\n",
       " '/website-architecture-and-seo/',\n",
       " '/what-is-a-domain-name/',\n",
       " '/wordpress/',\n",
       " 'https://github.com/seowings',\n",
       " 'https://twitter.com/seowings_org',\n",
       " 'https://www.facebook.com/seowings.org',\n",
       " 'https://www.seowings.org/',\n",
       " 'https://www.seowings.org/simply-static-tutorial/',\n",
       " 'https://www.seowings.org/wordpress/',\n",
       " 'https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA'}"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "637073b1-9f84-49b3-a94b-406f60952ecc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.youtube.com/channel/UCkVCjD48ge5zm1NN715IdLA\n",
      "https://github.com/seowings\n",
      "https://www.facebook.com/seowings.org\n",
      "https://twitter.com/seowings_org\n"
     ]
    }
   ],
   "source": [
    "social_media_websites = [\"github.com\", \"youtube.com\", \"facebook.com\", \"twitter.com\"]\n",
    "# this is not optimal\n",
    "for url in all_urls:\n",
    "    for social_url in social_media_websites:\n",
    "        if social_url in url:\n",
    "            print (url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "8f776a56-3c74-4cbd-8252-c97a1f5199be",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.seowings.org/wordpress/\n",
      "https://www.seowings.org/simply-static-tutorial/\n",
      "https://www.seowings.org/\n"
     ]
    }
   ],
   "source": [
    "internal_link = [\"www.seowings.org\"]\n",
    "\n",
    "# this is not optimal\n",
    "for url in all_urls:\n",
    "    for social_url in internal_link:\n",
    "        if social_url in url:\n",
    "            print (url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "249a192f-e9b1-44ff-abe6-7995841e674b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/web-server-log-files/\n",
      "/seo-and-web-hosting/\n",
      "/website-architecture-and-seo/\n",
      "/web-domains/\n",
      "/wordpress/\n",
      "/what-is-a-domain-name/\n",
      "/sitemap-seo/\n",
      "/best-software-for-installing-wordpress-on-localhost/\n",
      "/local-wordpress-hosting-on-virtual-machines/\n",
      "/search-engine-optimization/\n",
      "/unsubscribe/\n",
      "/software/\n",
      "/seoquake-tutorial/\n",
      "/host-wordpress-website-on-netlify/\n",
      "/static-web-hosting/\n",
      "/competitor-analysis-using-sitemaps/\n",
      "/best-static-website-hosting-services/\n",
      "/static-websites/\n",
      "/advantages-of-local-wordpress-hosting/\n",
      "/contact-us/\n",
      "/privacy-policy/\n",
      "/simply-static-tutorial/\n",
      "/seo-minion-tutorial/\n",
      "/\n",
      "/robots-txt/\n",
      "/download/\n",
      "/terms-of-services/\n",
      "/static-wordpress/\n",
      "/research-work/\n"
     ]
    }
   ],
   "source": [
    "for url in all_urls:\n",
    "    if url.startswith(\"/\"):\n",
    "        print (url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "d4adb3f1-f80a-4cbd-b6a7-d306681fa97e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'/',\n",
       " '/advantages-of-local-wordpress-hosting/',\n",
       " '/best-software-for-installing-wordpress-on-localhost/',\n",
       " '/best-static-website-hosting-services/',\n",
       " '/competitor-analysis-using-sitemaps/',\n",
       " '/contact-us/',\n",
       " '/download/',\n",
       " '/host-wordpress-website-on-netlify/',\n",
       " '/local-wordpress-hosting-on-virtual-machines/',\n",
       " '/privacy-policy/',\n",
       " '/research-work/',\n",
       " '/robots-txt/',\n",
       " '/search-engine-optimization/',\n",
       " '/seo-and-web-hosting/',\n",
       " '/seo-minion-tutorial/',\n",
       " '/seoquake-tutorial/',\n",
       " '/simply-static-tutorial/',\n",
       " '/sitemap-seo/',\n",
       " '/software/',\n",
       " '/static-web-hosting/',\n",
       " '/static-websites/',\n",
       " '/static-wordpress/',\n",
       " '/terms-of-services/',\n",
       " '/unsubscribe/',\n",
       " '/web-domains/',\n",
       " '/web-server-log-files/',\n",
       " '/website-architecture-and-seo/',\n",
       " '/what-is-a-domain-name/',\n",
       " '/wordpress/'}"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set([url for url in all_urls if url.startswith(\"/\")])       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "34d238ac-42ca-41d0-95f5-19e4e9ad1e9b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mInit signature:\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0miterable\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m/\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
       "\u001b[1;31mDocstring:\u001b[0m     \n",
       "Built-in mutable sequence.\n",
       "\n",
       "If no argument is given, the constructor creates a new empty list.\n",
       "The argument must be an iterable if specified.\n",
       "\u001b[1;31mType:\u001b[0m           type\n",
       "\u001b[1;31mSubclasses:\u001b[0m     _HashedSeq, StackSummary, _Threads, ConvertingList, DeferredConfigList, _ymd, _Accumulator, DeprecatedList, SList, _ImmutableLineList, ..."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "8111660f-af96-4aae-ace5-f1b1d582dbdb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8, 10]\n"
     ]
    }
   ],
   "source": [
    "[i for i in range(11) if i % 2 == 0]\n",
    "print(list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "c8593a2f-a16a-4375-91f6-a3217406decd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "del list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "7c204380-bc23-47d4-9c70-af4306fc4c7d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mInit signature:\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0miterable\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m/\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
       "\u001b[1;31mDocstring:\u001b[0m     \n",
       "Built-in mutable sequence.\n",
       "\n",
       "If no argument is given, the constructor creates a new empty list.\n",
       "The argument must be an iterable if specified.\n",
       "\u001b[1;31mType:\u001b[0m           type\n",
       "\u001b[1;31mSubclasses:\u001b[0m     _HashedSeq, StackSummary, _Threads, ConvertingList, DeferredConfigList, _ymd, _Accumulator, DeprecatedList, SList, _ImmutableLineList, ..."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "acb22490-81f2-48f7-a5ec-cda70a6aa0fa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for i in range(10):\n",
    "    if i%2 == 0:\n",
    "        print (i)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "4c2fffdb-f5dc-420b-bc8c-44d7fe898277",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list_of_even_numbers = [i for i in range(10) if i%2 == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "b482af29-c325-40ac-9226-d064c368c0f3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 2, 4, 6, 8]"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_even_numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "aa5757f4-b213-403b-a37f-0c5131561bb2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "pages_ctr = [1.2, 1.5, 2.5]\n",
    "\n",
    "list_of_better_ctrs = [pc for pc in pages_ctr if pc > 2]\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea89ec0a-db2a-4c0d-bc95-94335bd98037",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "6af5af8a-a63e-4dde-937a-5e8bc3eaccbe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2.5]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_better_ctrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "105d0a91-d73b-448e-bd50-1c3b1f2cbddd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable                Type                          Data/Info\n",
      "---------------------------------------------------------------\n",
      "BeautifulSoup           type                          <class 'bs4.BeautifulSoup'>\n",
      "HTTPAdapter             type                          <class 'requests.adapters.HTTPAdapter'>\n",
      "a                       dict                          n=2\n",
      "all_urls                set                           {'https://www.youtube.com<...>witter.com/seowings_org'}\n",
      "anchor                  Tag                           <a class=\"text-white\" hre<...>nk\">Terms of Services</a>\n",
      "bbc_feed                BeautifulSoup                 <?xml version=\"1.0\" encod<...>item>\\n</channel>\\n</rss>\n",
      "bbc_xml_newsfeed        str                           http://feeds.bbci.co.uk/news/rss.xml\n",
      "download_url            _lru_cache_wrapper            <functools._lru_cache_wra<...>ct at 0x000002DCA405CE00>\n",
      "h                       Response                      <Response [301]>\n",
      "headers                 dict                          n=1\n",
      "i                       int                           9\n",
      "internal_link           list                          n=1\n",
      "jang_news               BeautifulSoup                 <!DOCTYPE html>\\n\\n<html <...>script>\\n</body>\\n</html>\n",
      "list_of_better_ctrs     list                          n=1\n",
      "list_of_even_numbers    list                          n=5\n",
      "lru_cache               function                      <function lru_cache at 0x000002DC9E0E36D0>\n",
      "max_retires             int                           5\n",
      "pages_ctr               list                          n=3\n",
      "pc                      float                         2.4\n",
      "pprint                  function                      <function pprint at 0x000002DCA0077D00>\n",
      "r                       Response                      <Response [200]>\n",
      "requests                module                        <module 'requests' from '<...>\\\\requests\\\\__init__.py'>\n",
      "s                       Session                       <requests.sessions.Sessio<...>ct at 0x000002DCA3C10DF0>\n",
      "seowings                BeautifulSoup                 <!DOCTYPE html>\\n\\n<html <...>script>\\n</body>\\n</html>\n",
      "sleep                   builtin_function_or_method    <built-in function sleep>\n",
      "social                  str                           facebook.com\n",
      "social_media_websites   list                          n=4\n",
      "social_url              str                           www.seowings.org\n",
      "soup                    BeautifulSoup                 \\n<!DOCTYPE html>\\n\\n<htm<...></script></body>\\n</html>\n",
      "url                     str                           https://twitter.com/seowings_org\n"
     ]
    }
   ],
   "source": [
    "%whos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f2db228-59ac-4313-8057-29cfbec98dc5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}