Metadata-Version: 2.1
Name: 9gag-scraper
Version: 0.0.5
Summary: A simple 9GAG scraper
Project-URL: Homepage, https://github.com/federlizer-strypes/9gag-scraper
Project-URL: Issues, https://github.com/federlizer-strypes/9gag-scraper/issues
Author-email: Federlizer <federlizer@protonmail.com>
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 2
Requires-Python: >=2.7
Requires-Dist: requests==2.27.1
Requires-Dist: selenium==3.141.0
Description-Content-Type: text/markdown

# 9GAG Scraper

A simple 9GAG scraper that uses selenium's Chromium webdriver. It allows you to scrape the images given a particular
search term. The application is written in Python 2.7 with Tkinter 8.6.

## Installation

#### Tkinter
First, you will need to manually install Tkinter. To install it, follow the instructions from the
[tkdocs.com installation tutorial](https://tkdocs.com/tutorial/install.html) webpage. Remember to follow the correct
instructions to install Tkinter for Python 2.7

#### Chromium webdriver

The scraper also uses selenium's Chromium webdriver. I suspect that using a normal Chrome webdriver is going to work
just the same, but I was unable to test it out, since I cannot install pure Chrome webdriver on Linux (whoops). Follow
the instructions from [this webpage](https://chromedriver.chromium.org/getting-started).

#### Pip packages

Finally, you need to install the pip packages that are required for the application to run. It's recommended to create
a virtual environment, so you can separate the installed packages from your global packages:

```bash
# Create a virtual environment
virtualenv ./venv

# Activate the virtual environment for your current shell
source ./venv/bin/activate

# Finally, install the packages from the requirements.txt file
pip install -r ./requirements.txt

# When you're done using the application, you can deactivate the virtual environment
deactivate
```

### Usage

To start the application, you just need to execute the `main.py` script:

```bash
python ./src/main.py
```

This will open up the GUI application, where you'll be able to define the search term you want to scrape with. Hitting
the "Scrape!" button (or enter) will start the scraping process - the webdriver will fetch the webpage needed, scroll
down the required amount (twice by default) and fetch all images that have been found. The images that have been found
will be displayed as thumbnails and you will have the opportunity to save the images on your computer.

### Known issues

##### Cloudflare - are you a robot?!

9GAG is protected by CloudFlare or some other proxy. Sometimes, I've had issues with it trying to verify that I'm human.
The webdriver is obviously not human, so you cannot simply click the button to go around the verification. The way that
I went around it is to open the same page with my own Chrome browser and going through the verification step.

### Next steps

- Get the webdriver to minimize
- Have an entry where the user can define the download folder (otherwise, use the default)
- Have an entry field that allows the user to select the number of times the webdriver will scroll down
- Try and find a way to use the `with` keyword with the scraper and gui classes, so the browser can be closed
  nicely without the try/finally clause
- Have some CLI arguments about logging level, logging location, more?



### Jorkata feedback

- [ ] Error handling
- [x] 2x newline between imports and the rest of the code
- [ ] newline at the end of the files
- [x] avoid newlines in functions
- [x] main.py - config_logger function - not the best name
- [x] import whole library instead of each small component
- [x] Docstrings
    - [?] // Read the PEP8 document for docstrings, get familiar with it
    - [x] Newline after docstrings
    - [x] If a docstring can be on a single line, it's better for it to be that way
    - [x] Docstring everything
    - [x] One liner description for multi-line docstrings
    - [x] Docstrings must end on with a dot
    - [x] Imperative form for docstrings
    - [x] Some comments that I have are useless and don't look good ("WHAT THE FAK")
- [x] Use constants for literals
- [x] if statements - continue or return, try to avoid using else
- [x] scrape.py -> _get_images or whatever it is called - make into a generator (yield)
- [x] scraper.py#79 - too many new lines (javascripty)
- [x] initialize fields in __init__ even if that means that they need to be set to None - clearer definition of
      what the class contains as fields
- [ ] Make into a PyPi package - python wheel
- [ ] UNITTEST!
- [ ] Make it python3 compatible
- [x] Make into MVC





### Unittest

- Test `ScrapedImage` somehow?
- Test `NineGagScraper` somehow?
    - We'll need to mock the selenium driver somehow
