02-3-how-to-use-3rd-party-python-packages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "954f40fe-440f-4d80-8043-4168ee51ab32",
   "metadata": {},
   "source": [
    "# Plyaing with Python Code\n",
    "\n",
    "Since, you have already aquired basic understanding of python and its core concepts for data handling and basic maniupluation. You might be wondering that \n",
    "\n",
    "- What is Python Standard Library?\n",
    "- How to include 3rd party packages in your python projects\n",
    "- what are the best practaices to organize your python projects\n",
    "- How to distribute you Python code to others.\n",
    "\n",
    "Answers to these questiions are elaborated in the following sections."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12756103-41d7-4d7b-a9b9-ecfa9172b77a",
   "metadata": {},
   "source": [
    "## How to import Python Packages?\n",
    "\n",
    "Python code can be organized into modules, Packages or indvidual Scripts. This code can be written by your, 3rd party or may be the part of Python standard library. When importing these modules you have sevreal options\n",
    "\n",
    "- Full import\n",
    "    - import pysitemap\n",
    "\n",
    "- Import a desired Sub-Modules\n",
    "    - from pysitemap import Sitemap\n",
    "    \n",
    "- Import all Sub Modueles\n",
    "    - from pysitemap import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986bc997-ce6b-447b-9d40-31da808df1f1",
   "metadata": {},
   "source": [
    "## Python Standard Library\n",
    "\n",
    "Standard Python, also know as Vanila Python only contains basic functionalities to keep it foot print minimum. However, this minimum installation also contians tons of useful functions, and pcakges which are know as python standard library. This library includes functionalities for data handling, basic manipulation and tons of other algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71786ae6-99c7-46b5-a2b7-2eb0db1b788f",
   "metadata": {},
   "source": [
    "## 3rd Party Packages\n",
    "\n",
    "This standard library has limited functionalities and if you are looking for a function which is not part of standard library then you can install aditional 3rd party library. \n",
    "\n",
    "Python has a rich collection of 3rd party packages, application interfaces, and modules which allows you to perform custom marketing taks without even writing those functionalities by yourself. For example, if you are interested in numerical calculations, then you can use numpy and pandas python packages. For most of the scintific works with extenstive algorithms, you are good to go with scipy and scikit-* packages. \n",
    "\n",
    "One of the reason for Python popularity is the wide availablity of such packages. In most of the cases, you will always find out packages for any of your taks.\n",
    "\n",
    "\n",
    "You can isntall 3rd party packages using pip commands.  For example if you want to install pysitemaps library which is available at pipy, then you can simply isntall it using following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "63b15256-967f-4882-ba71-9cf1661a1dc3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pysitemaps in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (0.0.6)\n",
      "Requirement already satisfied: requests in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from pysitemaps) (2.28.2)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from pysitemaps) (4.12.2)\n",
      "Requirement already satisfied: lxml in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from pysitemaps) (4.9.2)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from beautifulsoup4->pysitemaps) (2.4.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests->pysitemaps) (3.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests->pysitemaps) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests->pysitemaps) (1.26.15)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\admin\\documents\\github\\data-science-for-digital-marketers\\.venv\\lib\\site-packages (from requests->pysitemaps) (2022.12.7)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.1 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "!pip install pysitemaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "acab8206-80cf-4240-bb92-4717161aaee1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'parent': {'loc': 'https://www.seowings.org/sitemap.xml', 'lastmod': '2023-05-06', 'urls': [{'loc': 'https://www.seowings.org/static-wordpress/', 'lastmod': '2022-11-18', 'images': []}, {'loc': 'https://www.seowings.org/simply-static-tutorial/', 'lastmod': '2023-03-05', 'images': []}, {'loc': 'https://www.seowings.org/host-wordpress-website-on-netlify/', 'lastmod': '2022-11-19', 'images': []}, {'loc': 'https://www.seowings.org/web-server-log-files/', 'lastmod': '2022-11-27', 'images': []}, {'loc': 'https://www.seowings.org/local-wordpress-hosting-on-virtual-machines/', 'lastmod': '2022-11-20', 'images': []}, {'loc': 'https://www.seowings.org/seoquake-tutorial/', 'lastmod': '2022-11-29', 'images': []}, {'loc': 'https://www.seowings.org/robots-txt/', 'lastmod': '2022-11-27', 'images': []}, {'loc': 'https://www.seowings.org/best-static-website-hosting-services/', 'lastmod': '2023-01-01', 'images': []}, {'loc': 'https://www.seowings.org/static-web-hosting/', 'lastmod': '2023-01-02', 'images': []}, {'loc': 'https://www.seowings.org/competitor-analysis-using-sitemaps/', 'lastmod': '2022-11-19', 'images': []}, {'loc': 'https://www.seowings.org/seo-minion-tutorial/', 'lastmod': '2022-11-22', 'images': []}, {'loc': 'https://www.seowings.org/best-software-for-installing-wordpress-on-localhost/', 'lastmod': '2022-11-24', 'images': []}, {'loc': 'https://www.seowings.org/sitemap-seo/', 'lastmod': '2022-12-17', 'images': []}, {'loc': 'https://www.seowings.org/seo-and-web-hosting/', 'lastmod': '2022-11-21', 'images': []}, {'loc': 'https://www.seowings.org/what-is-a-domain-name/', 'lastmod': '2022-12-02', 'images': []}, {'loc': 'https://www.seowings.org/advantages-of-local-wordpress-hosting/', 'lastmod': '2022-11-15', 'images': []}, {'loc': 'https://www.seowings.org/website-architecture-and-seo/', 'lastmod': '2022-12-01', 'images': []}, {'loc': 'https://www.seowings.org/wordpress/', 'lastmod': '2022-12-14', 'images': []}, {'loc': 'https://www.seowings.org/web-domains/', 'lastmod': '2022-12-13', 'images': []}, {'loc': 'https://www.seowings.org/software/', 'lastmod': '2022-11-19', 'images': []}, {'loc': 'https://www.seowings.org/search-engine-optimization/', 'lastmod': '2022-11-13', 'images': []}, {'loc': 'https://www.seowings.org/static-websites/', 'lastmod': '2022-12-14', 'images': []}, {'loc': 'https://www.seowings.org/research-work/', 'lastmod': '2022-11-28', 'images': []}]}, 'xsl-file': '', 'sub_sitemaps': []}\n"
     ]
    }
   ],
   "source": [
    "# https://serpwings.com/software/python-sitemaps/\n",
    "from pysitemaps import Sitemap, Url, XmlDocument \n",
    "\n",
    "smp = Sitemap(website_name=\"https://www.seowings.org/\")\n",
    "smp.fetch(include_urls=True)\n",
    "print(smp.as_dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c328949b-f6f3-4b33-9a5d-5423e86b1263",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['parent', 'xsl-file', 'sub_sitemaps'])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sitemap.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8df0537f-3220-4f06-9d55-e84edd1541c4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loc', 'lastmod', 'urls'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sitemap['parent'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "13ee2c8d-d501-4d54-b520-5026572bffa1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(sitemap['parent']['urls'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d309b0ca-0a0b-4bd5-8e2c-d4ce59420449",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/robots-txt/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/schema-checker/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/user-agent/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/user-details/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/webpage-metadata/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/webpage-urls/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/whois-checker/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/web-tools/wp-details/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/sitemaps/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/sitemaps/sitemap-finder/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/sitemaps/sitemap-details/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/text-processing/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/text-processing/url-cleaner/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/text-processing/url-extractor/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/convertors/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/convertors/json-to-xml/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/convertors/xml-to-json/', 'lastmod': '2023-03-19', 'images': []}\n",
      "{'loc': 'https://serpwings.com/online-seo-tools/convertors/xml-to-tree/', 'lastmod': '2023-03-19', 'images': []}\n"
     ]
    }
   ],
   "source": [
    "smp = Sitemap(website_name=\"https://www.serpwings.com/\")\n",
    "smp.fetch(include_urls=True)\n",
    "sitemap = smp.as_dict()\n",
    "\n",
    "all_urls = sitemap['parent']['urls']\n",
    "for url in all_urls:\n",
    "    # print (extract_keywrod(url['loc']))\n",
    "    print (url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2df4ea74-3de4-4a23-959a-2b0e57912ce5",
   "metadata": {},
   "source": [
    "## How to Plan Your Python Project\n",
    "\n",
    "So, you have written small scripts or functions and now wondering how to distbruted it to your ppers. As you write your python code and your code base grows, then it becomes cumbersome to handle them. \n",
    "\n",
    "In almost all cases, it is desired that Python code should be able to reproduce their results. However, this become difficut for data science porjects and data is udually not well planed as part of Python Packages (most developers only care of Python source code).\n",
    "\n",
    "For you project, you have different options avaiable\n",
    "\n",
    "- for small scripts inside your organizatoin, you can distribute them as it is (either .py file or notebooks)\n",
    "- For codes performing bunch of tasks and distrbution to broarder audiance (both in your organizations or external world, you can create python packages\n",
    "    - Organize them as src module\n",
    "    - organize them as script module\n",
    "- Use cookie cutters to organiz your projects (python code and data) for data science projects\n",
    "- For most of the cause where you have simple scipts then orgnaiiung htem inside bunch of jupyter note books will also work well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "856dc6b0-c059-4d65-9f81-4811e8422373",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}