inital commit

scrapper/README.md

@@ -0,0 +1,284 @@
+# credit-scrapper-poc
+
+## Server setup
+
+####Step 1 – Installing Nginx
+```
+sudo apt update
+sudo apt install nginx
+```
+
+
+####Step 2 – Adjusting the Firewall
+```
+sudo ufw allow 'Nginx HTTP'
+sudo ufw allow 'OpenSSH'
+sudo ufw enable
+sudo ufw status
+```
+
+
+####Step 3 – Checking your Web Server
+```
+sudo systemctl status nginx
+curl -4 icanhazip.com
+```
+When you have your server’s IP address, enter it into your browser’s address bar:
+
+
+####Step 4 – Installing the Components from the Ubuntu Repositories
+```
+sudo apt update
+sudo apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools
+```
+
+
+####Step 5 – Clone the repository
+```
+git clone https://gitlab.com/sunflowerlab/Repositories/TechnologyPOCs/credit-scrapper-poc.git
+```
+
+
+####Step 6 – Creating a Python Virtual Environment
+```
+sudo apt install python3-venv
+cd credit-scrapper-poc
+python3.6 -m venv venv
+source venv/bin/activate
+```
+
+
+####Step 7 – Setting Up an Application
+```
+pip install wheel
+pip install uwsgi
+pip install -r requirements.txt
+```
+
+
+####Step 8 – Test the setup
+```
+python app.py
+uwsgi --socket 0.0.0.0:5000 --protocol=http -w wsgi:app
+uwsgi --socket 0.0.0.0:8080 --protocol=http -w wsgi:app
+```
+
+
+####Step 9 – Deactivate python environment
+```
+deactivate
+```
+
+
+####Step 10 – Creating a systemd Unit File
+```
+sudo nano /etc/systemd/system/app.service
+```
+
+Add the following contain
+
+```
+[Unit]
+Description=uWSGI instance to serve app
+After=network.target
+
+[Service]
+User=ubuntu
+Group=www-data
+WorkingDirectory=/home/ubuntu/credit-scrapper-poc
+Environment="PATH=/home/ubuntu/credit-scrapper-poc/venv/bin"
+ExecStart=/home/ubuntu/credit-scrapper-poc/venv/bin/uwsgi --ini app.ini
+
+[Install]
+WantedBy=multi-user.target
+
+```
+
+Update the directory paths.
+
+We can now start the uWSGI service we created and enable it so that it starts at boot:
+```
+sudo systemctl start app
+sudo systemctl enable app
+sudo systemctl status app
+```
+
+####Step 11 – Configuring Nginx to Proxy Requests
+
+```
+sudo nano /etc/nginx/sites-available/app
+```
+
+
+###Step 12  -  Install Javascript  library
+
+source install_phantomjs.sh
+
+
+
+
+Add the following contain
+```
+server {
+    listen 80;
+    server_name 54.68.236.191;
+
+    location / {
+	proxy_read_timeout 1200s;
+	proxy_connect_timeout 1200s;
+        include uwsgi_params;
+        uwsgi_pass unix:/home/ubuntu/credit-scrapper-poc/app.sock;
+
+	# when a client closes the connection then keep the channel to uwsgi open. Otherwise uwsgi throws an IOError
+	uwsgi_ignore_client_abort on;
+
+	uwsgi_buffer_size 60M;
+	uwsgi_buffers 8 60M;
+	uwsgi_busy_buffers_size 60M;
+
+	uwsgi_read_timeout 1200;
+	uwsgi_send_timeout 1200;
+
+	uwsgi_connect_timeout 1200;
+    }
+}
+```
+Update the directory paths and IP or domain address.
+
+```
+sudo nano /etc/nginx/conf.d/timeout.conf 
+```
+
+Add the following contain
+```
+proxy_connect_timeout 1200;
+proxy_send_timeout 1200;
+proxy_read_timeout 1200;
+send_timeout 1200;
+```
+
+```
+sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled
+sudo nginx -t
+sudo systemctl restart nginx
+sudo systemctl restart app
+```
+
+When you have your server’s IP address, enter it into your browser’s address bar.
+
+
+## API Details
+
+1. Smart Credit - Get Credit Report 3B
+    
+    Method - `POST`
+    
+    Endpoint - `/smart-credit-credit-report-3b`
+    
+    Content-Type - `application/json`
+    
+    Payload - `{
+        "email": "email",
+        "password": "password"
+    }`
+    
+    Response - `HTML CONTENT` or `ERROR`
+    
+ 2. IdentityIQ - Get most recent Credit Report
+    
+    Method - `POST`
+
+    Endpoint - `/identity-iq-credit-report-html`
+
+    Content-Type - `application/json`
+    
+    Payload - `{
+        "email": "email",
+        "password": "password",
+        "ssn": "last-four-digit"
+    }`
+
+    Response - `HTML CONTENT` or `ERROR` or `TIME_OUT`
+
+ 3. IdentityIQ - Get refresh date and button type
+    
+    Method - `POST`
+
+    Endpoint - `/identity-iq-report-refresh-date`
+    
+    Content-Type - `application/json`
+    
+    Payload - `{
+        "email": "email",
+        "password": "password",
+        "ssn": "last -four-digit"
+    }`
+
+    Response - `{
+                'refresh_date': "MM/DD/YYYY",
+                'button_type': 1 or 2
+            }`
+      or 
+      `ERROR`
+      or
+      `TIME_OUT`
+
+    button_type - `1` for Purchase AND `2` for Refresh
+
+ 4. IdentityIQ - To click on the refresh button.
+    
+    Method - `POST`
+    
+    Endpoint - `/identity-iq-click-refresh-btn`
+
+    Content-Type - `application/json`
+
+    Payload - `{
+        "email": "email",
+        "password": "password",
+        "ssn": "last -four-digit"
+    }`
+
+    Response - `DONE` or `ERROR` or `TIME_OUT` or `BUTTON_NOT_FOUND`
+
+ 5. SmartCredit - Get refresh date and button type
+
+    Method - `POST`
+    
+    Endpoint - `/smart-credit-credit-refresh-status`
+    
+    Content-Type - `application/json`
+
+    Payload - `{
+        "email": "email",
+        "password": "password"
+    }`
+
+    Response - `{
+                'button_type': 0 or 1 or 2.,
+                'refresh_date': TEXT
+            }`
+      or 
+      `ERROR`
+      or
+      `TIME_OUT`
+
+    button_type - `0` for no button AND `1` for Purchase AND `2` for Refresh
+
+ 6. SmartCredit - To click on the refresh button.
+    
+    Method - `POST`
+    
+    Endpoint - `/smart-credit-credit-click-refresh-btn`
+    
+    Content-Type - `application/json`
+
+    Payload - `{
+        "email": "email",
+        "password": "password"
+    }`
+
+    Response - `DONE` or `ERROR` or `TIME_OUT` or `BUTTON_NOT_FOUND`
+
+
+***Reference Link*** https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uswgi-and-nginx-on-ubuntu-18-04