Web scraping application using python

Web scraping website with pyqt5

Those who bored of learning basics of python and who want to do something interesting, this blog will be definitely useful. Okay, let’s go with the title. What is web scraping? Scrap is nothing but a left over from a product. Same way web scraping is nothing but extracting data from a website. Like metal scrap we can extract useful things from the website which ever we are looking for. I used to check everyday for price drop of ipad mini 5 in amazon.in website(unfortunately it never drops 😅). But in-order to check price, i need to open browser, enter the web url, then look for the product and then check price. But what if, we have a application coded to check for price as soon as we click on it?! Exactly! that is what we are going to do it now. We are going to build a .exe application, which could extract price of a product from amazon.in and show it in the pop up window. Let’s get started!!

Requirements:

You need to have the following python libraries to proceed further,

pip install beautifulsoup4
pip install requests
pip install pyqt5
pip install pyqt5-tools
pip install auto-py-to-exe

Creating web scraper code:

First let us create code to do web scraping,

In the User-Agent key, instead of ‘#’ paste your user agent name. You can find your user agent name by googling as “my user agent”. Just paste the same in the single quote. This user agent can make sure not to get blocked when we are doing web scraping. Also paste the url link of the website that you want to do web scraping in url variable. Try to execute the script and check if it is working.

Creating pyqt5 GUI:

In order to create GUI, we can make user of Qt Designer. QtDesigner is present at ‘site-packages/pyqt5_tools’. You can find the location of the designer by using the following command in python,

import site
site.getsitepackages()

Now go to the location and look for pyqt5designer. Then double click on it, to open the designer. Drag and drop the label and push button. Then, save the file. The file will be in .ui format. We need to convert it into .py file, in-order to do some changes in the code.

Convert .ui file to .py:

To convert it, you need to go to sitepackages in command prompt or terminal

>>> cd “C:\\Users\\……\\Programs\\Python\\Python36-32\\lib\\site-packages” [Location of sitepackages]

Use the below command to convert it into .py file,

>>> pyuic5 “C:\Users\……\FILENAME.ui”[Exact location of .ui  file] -o ” C:\Users\…….\FILENAME.py” [Location where want to put .py  file]

Now edit the file and copy paste the code,

Credits: Geeks for Geeks

Check the file by executing it in command prompt or in terminal.

You can find the complete blog for pyqt5 in Geeks of Geeks.

Now lets combine both the code and create a combined program as below,

combined code of web scraping and pyqt5

Converting .py into .exe file:

Hope you have already downloaded auto-py-to-exe convertor. If you still have any issue, please refer the link - auto-py-to-ext.

Select the script location and output location, then click on convert .py to .exe button.

auto-py-to-exe convertor

That’s it! It is done. Now, go to the output location and then click on the .exe application which is inside the folder as below,

project1.exe is my file name

When the window pops up, just click on the button to get the application price. You will have the output as below,

Python application

Yes we did it!! Happy coding 😊