RTG/scrapper/README.md

credit-scrapper-poc

Server setup

####Step 1 – Installing Nginx

sudo apt update
sudo apt install nginx

####Step 2 – Adjusting the Firewall

sudo ufw allow 'Nginx HTTP'
sudo ufw allow 'OpenSSH'
sudo ufw enable
sudo ufw status

####Step 3 – Checking your Web Server

sudo systemctl status nginx
curl -4 icanhazip.com

When you have your server’s IP address, enter it into your browser’s address bar:

####Step 4 – Installing the Components from the Ubuntu Repositories

sudo apt update
sudo apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools

####Step 5 – Clone the repository

git clone https://gitlab.com/sunflowerlab/Repositories/TechnologyPOCs/credit-scrapper-poc.git

####Step 6 – Creating a Python Virtual Environment

sudo apt install python3-venv
cd credit-scrapper-poc
python3.6 -m venv venv
source venv/bin/activate

####Step 7 – Setting Up an Application

pip install wheel
pip install uwsgi
pip install -r requirements.txt

####Step 8 – Test the setup

python app.py
uwsgi --socket 0.0.0.0:5000 --protocol=http -w wsgi:app
uwsgi --socket 0.0.0.0:8080 --protocol=http -w wsgi:app

####Step 9 – Deactivate python environment

deactivate

####Step 10 – Creating a systemd Unit File

sudo nano /etc/systemd/system/app.service

Add the following contain

[Unit]
Description=uWSGI instance to serve app
After=network.target

[Service]
User=ubuntu
Group=www-data
WorkingDirectory=/home/ubuntu/credit-scrapper-poc
Environment="PATH=/home/ubuntu/credit-scrapper-poc/venv/bin"
ExecStart=/home/ubuntu/credit-scrapper-poc/venv/bin/uwsgi --ini app.ini

[Install]
WantedBy=multi-user.target

Update the directory paths.

We can now start the uWSGI service we created and enable it so that it starts at boot:

sudo systemctl start app
sudo systemctl enable app
sudo systemctl status app

####Step 11 – Configuring Nginx to Proxy Requests

sudo nano /etc/nginx/sites-available/app

###Step 12 - Install Javascript library

source install_phantomjs.sh

Add the following contain

server {
    listen 80;
    server_name 54.68.236.191;

    location / {
	proxy_read_timeout 1200s;
	proxy_connect_timeout 1200s;
        include uwsgi_params;
        uwsgi_pass unix:/home/ubuntu/credit-scrapper-poc/app.sock;

	# when a client closes the connection then keep the channel to uwsgi open. Otherwise uwsgi throws an IOError
	uwsgi_ignore_client_abort on;

	uwsgi_buffer_size 60M;
	uwsgi_buffers 8 60M;
	uwsgi_busy_buffers_size 60M;

	uwsgi_read_timeout 1200;
	uwsgi_send_timeout 1200;

	uwsgi_connect_timeout 1200;
    }
}

Update the directory paths and IP or domain address.

sudo nano /etc/nginx/conf.d/timeout.conf 

Add the following contain

proxy_connect_timeout 1200;
proxy_send_timeout 1200;
proxy_read_timeout 1200;
send_timeout 1200;

sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled
sudo nginx -t
sudo systemctl restart nginx
sudo systemctl restart app

When you have your server’s IP address, enter it into your browser’s address bar.

API Details

1. Smart Credit - Get Credit Report 3B

   Method - POST

   Endpoint - /smart-credit-credit-report-3b

   Content-Type - application/json

   Payload - { "email": "email", "password": "password" }

   Response - HTML CONTENT or ERROR

2. IdentityIQ - Get most recent Credit Report

   Method - POST

   Endpoint - /identity-iq-credit-report-html

   Content-Type - application/json

   Payload - { "email": "email", "password": "password", "ssn": "last-four-digit" }

   Response - HTML CONTENT or ERROR or TIME_OUT

3. IdentityIQ - Get refresh date and button type

   Method - POST

   Endpoint - /identity-iq-report-refresh-date

   Content-Type - application/json

   Payload - { "email": "email", "password": "password", "ssn": "last -four-digit" }

   Response - { 'refresh_date': "MM/DD/YYYY", 'button_type': 1 or 2 } or ERROR or TIME_OUT

   button_type - 1 for Purchase AND 2 for Refresh

4. IdentityIQ - To click on the refresh button.

   Method - POST

   Endpoint - /identity-iq-click-refresh-btn

   Content-Type - application/json

   Payload - { "email": "email", "password": "password", "ssn": "last -four-digit" }

   Response - DONE or ERROR or TIME_OUT or BUTTON_NOT_FOUND

5. SmartCredit - Get refresh date and button type

   Method - POST

   Endpoint - /smart-credit-credit-refresh-status

   Content-Type - application/json

   Payload - { "email": "email", "password": "password" }

   Response - { 'button_type': 0 or 1 or 2., 'refresh_date': TEXT } or ERROR or TIME_OUT

   button_type - 0 for no button AND 1 for Purchase AND 2 for Refresh

6. SmartCredit - To click on the refresh button.

   Method - POST

   Endpoint - /smart-credit-credit-click-refresh-btn

   Content-Type - application/json

   Payload - { "email": "email", "password": "password" }

   Response - DONE or ERROR or TIME_OUT or BUTTON_NOT_FOUND

Reference Link https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uswgi-and-nginx-on-ubuntu-18-04