RTG/scrapper/README.md

# credit-scrapper-poc

## Server setup

####Step 1 – Installing Nginx
```
sudo apt update
sudo apt install nginx
```


####Step 2 – Adjusting the Firewall
```
sudo ufw allow 'Nginx HTTP'
sudo ufw allow 'OpenSSH'
sudo ufw enable
sudo ufw status
```


####Step 3 – Checking your Web Server
```
sudo systemctl status nginx
curl -4 icanhazip.com
```
When you have your server’s IP address, enter it into your browser’s address bar:


####Step 4 – Installing the Components from the Ubuntu Repositories
```
sudo apt update
sudo apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools
```


####Step 5 – Clone the repository
```
git clone https://gitlab.com/sunflowerlab/Repositories/TechnologyPOCs/credit-scrapper-poc.git
```


####Step 6 – Creating a Python Virtual Environment
```
sudo apt install python3-venv
cd credit-scrapper-poc
python3.6 -m venv venv
source venv/bin/activate
```


####Step 7 – Setting Up an Application
```
pip install wheel
pip install uwsgi
pip install -r requirements.txt
```


####Step 8 – Test the setup
```
python app.py
uwsgi --socket 0.0.0.0:5000 --protocol=http -w wsgi:app
uwsgi --socket 0.0.0.0:8080 --protocol=http -w wsgi:app
```


####Step 9 – Deactivate python environment
```
deactivate
```


####Step 10 – Creating a systemd Unit File
```
sudo nano /etc/systemd/system/app.service
```

Add the following contain

```
[Unit]
Description=uWSGI instance to serve app
After=network.target

[Service]
User=ubuntu
Group=www-data
WorkingDirectory=/home/ubuntu/credit-scrapper-poc
Environment="PATH=/home/ubuntu/credit-scrapper-poc/venv/bin"
ExecStart=/home/ubuntu/credit-scrapper-poc/venv/bin/uwsgi --ini app.ini

[Install]
WantedBy=multi-user.target

```

Update the directory paths.

We can now start the uWSGI service we created and enable it so that it starts at boot:
```
sudo systemctl start app
sudo systemctl enable app
sudo systemctl status app
```

####Step 11 – Configuring Nginx to Proxy Requests

```
sudo nano /etc/nginx/sites-available/app
```


###Step 12  -  Install Javascript  library

source install_phantomjs.sh




Add the following contain
```
server {
    listen 80;
    server_name 54.68.236.191;

    location / {
	proxy_read_timeout 1200s;
	proxy_connect_timeout 1200s;
        include uwsgi_params;
        uwsgi_pass unix:/home/ubuntu/credit-scrapper-poc/app.sock;

	# when a client closes the connection then keep the channel to uwsgi open. Otherwise uwsgi throws an IOError
	uwsgi_ignore_client_abort on;

	uwsgi_buffer_size 60M;
	uwsgi_buffers 8 60M;
	uwsgi_busy_buffers_size 60M;

	uwsgi_read_timeout 1200;
	uwsgi_send_timeout 1200;

	uwsgi_connect_timeout 1200;
    }
}
```
Update the directory paths and IP or domain address.

```
sudo nano /etc/nginx/conf.d/timeout.conf 
```

Add the following contain
```
proxy_connect_timeout 1200;
proxy_send_timeout 1200;
proxy_read_timeout 1200;
send_timeout 1200;
```

```
sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled
sudo nginx -t
sudo systemctl restart nginx
sudo systemctl restart app
```

When you have your server’s IP address, enter it into your browser’s address bar.


## API Details

1. Smart Credit - Get Credit Report 3B
    
    Method - `POST`
    
    Endpoint - `/smart-credit-credit-report-3b`
    
    Content-Type - `application/json`
    
    Payload - `{
        "email": "email",
        "password": "password"
    }`
    
    Response - `HTML CONTENT` or `ERROR`
    
 2. IdentityIQ - Get most recent Credit Report
    
    Method - `POST`

    Endpoint - `/identity-iq-credit-report-html`

    Content-Type - `application/json`
    
    Payload - `{
        "email": "email",
        "password": "password",
        "ssn": "last-four-digit"
    }`

    Response - `HTML CONTENT` or `ERROR` or `TIME_OUT`

 3. IdentityIQ - Get refresh date and button type
    
    Method - `POST`

    Endpoint - `/identity-iq-report-refresh-date`
    
    Content-Type - `application/json`
    
    Payload - `{
        "email": "email",
        "password": "password",
        "ssn": "last -four-digit"
    }`

    Response - `{
                'refresh_date': "MM/DD/YYYY",
                'button_type': 1 or 2
            }`
      or 
      `ERROR`
      or
      `TIME_OUT`

    button_type - `1` for Purchase AND `2` for Refresh

 4. IdentityIQ - To click on the refresh button.
    
    Method - `POST`
    
    Endpoint - `/identity-iq-click-refresh-btn`

    Content-Type - `application/json`

    Payload - `{
        "email": "email",
        "password": "password",
        "ssn": "last -four-digit"
    }`

    Response - `DONE` or `ERROR` or `TIME_OUT` or `BUTTON_NOT_FOUND`

 5. SmartCredit - Get refresh date and button type

    Method - `POST`
    
    Endpoint - `/smart-credit-credit-refresh-status`
    
    Content-Type - `application/json`

    Payload - `{
        "email": "email",
        "password": "password"
    }`

    Response - `{
                'button_type': 0 or 1 or 2.,
                'refresh_date': TEXT
            }`
      or 
      `ERROR`
      or
      `TIME_OUT`

    button_type - `0` for no button AND `1` for Purchase AND `2` for Refresh

 6. SmartCredit - To click on the refresh button.
    
    Method - `POST`
    
    Endpoint - `/smart-credit-credit-click-refresh-btn`
    
    Content-Type - `application/json`

    Payload - `{
        "email": "email",
        "password": "password"
    }`

    Response - `DONE` or `ERROR` or `TIME_OUT` or `BUTTON_NOT_FOUND`


***Reference Link*** https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uswgi-and-nginx-on-ubuntu-18-04