How this was clobbeled together

Navigate elsewhere

 This site was primarily organized in two parts. First, my personal site's flask app was cloned, HTML and CSS was copied from this twitter bootstrap theme. Heroku's addon PostgreSQL was added; a python script was written to scrape and store data, data can then be queried and graphed.

Cloning previous site:

 Make sure python is upto date, along with any libraries. Same with Heroku toolbelt. I'm fairly sure that my library management with virtual env is a little out of wack. I believe, that when I originally setup the venv file for my personal site; I was accidently installing lots more packages on that python interpreter. My python distro may need a complete wipe and re-set to get better organized.

cd dekstop
brew install heroku
heroku login
heroku git:clone -a myapp

 Rename the folder, cd inside and start venv.

cd my_project_folder
source venv/bin/activate

 At this point, the cloned python code had many moldule dependancies, which are not necessarily downloaded. Running the app instantly fails since these libaries are missing. Modules are installed with pip until failures stop.

pip install requests
pip install flask
pip install flask-mail

 Once the debugger starts, navigate over to http://127.0.0.1:5000 to see the local version. This exact copy is pushed to a new heroku:

heroku create my_application
git init
heroku apps:create example
git add .
git commit -m "pushing first time"
git push 'enter the heroku git address here' master

 I have two applications on heroku, so the destiantion application within heroku needs to be specified. Push to github also.

git remote add origin remote repository URL
git remote -v
git push -u origin master

Adding new HTML and CSS theme:

 Next, HTML and CSS was downloaded from twitter bootstrap theme. This theme will be the base, and all extra text will be contained within the genera content area. The theme's html, renamed base_2.html is placed in flask's template folder, its associated CSS files are placed in the static folder. CSS files are linked to the base_2.html by adding the following few lines in the header of the HTML file:

<link href="../../../static/simple-sidebar.css" rel="stylesheet">

 This template uses the same JS (boostrap.min) I had already been using, so no additional JS needed to be added. Base_2.html then has block and end block content added so that flask builds on it.

{% block content %}
{% endblock %} 

 Future HTML to extend this base have one line added to the top:

{% extends "base_2.html" %} 

 The HTML for the selections within the side bar were changed and added to. HTML under the toggle button is also changed. Two non template pages are made, the landing page (index.html was changed)and this page by simply making an html file of the format:

{% extends "base_2.html" %}
{% block content %}
//Insert all html for the content of the page
{% endblock %}

 Routes within the hello.py file had to be changed to link to new pages:

@app.route("/base_2")
@app.route("/home")
@app.route("/index")
@app.route("/")
def base_2():
    return render_template('index.html')
#   return render_template('hello.html')
#
#

@app.route("/how_this_was_made")
def how_this_was_made():
    return render_template('how_this_was_made.html')

 So far, I've made a base template, and two pages, the landing page and this page. Each page extends the base template so all pages have the same togglebar / links. Next I deleted all my extra leftover code from my previous website to clean up.

Clean up local venv and make a .gitignore:

 Thanks to help from a redditor, my intial handling of venv we done very poorly. Venv should be completely ignored when pushing and pulling. This is likely the root of all my initial dependancy errors. I'm currently in a situation where my requirements.txt file is correct, and the install of venv is very off. Best practice is to re-create venv (delete and re-initialize) then re-build dependancies from requirements.txt. So, I delete the venev folder, re-install a new venv and then use the correct requirements.txt to re-populate the modules:

rm -r venv
python -m virtualenv venv
venv/bin/pip install -r requirements.txt

  A .gitigore is added to hide any files which may in the future contain login credentials. The venv folder ought to also be ignored, since there is no reason to keep track of it (the proper venv can be re-created from a requirements.txt as above). Also complied python doesn't need to be pushed to git. Cd over to the root level and:

touch .gitignore
open .gitignore
 Add following lines:
venv/
*.pyc
*~
.DS_Store
 Purge the ignored files from all previous commits:
git rm --cached -r .
git add .

Installing PostgreSQL locally, adding and graphing data:

 Next, I install PostgreSQL on my machine. I start by loading data into a local database before putting it on the cloud. Homebrew, PostgrSQL and the python package Psycopg2 are installed. Psycopg2 installed within the venv python interpreter. This was a very helpful tutorial.

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew update  #Just to make sure
brew install postgres

 Also, configure postgreSQL to start on boot. Create database: from within postgreSQL:

psql
create database rmv_scraping;

 Make a small script to test things are setup:

#!/usr/bin/python2.4
# Small script to show PostgreSQL and Pyscopg together
import psycopg2
try:
    conn = psycopg2.connect("dbname='rmv_scraping' user='patrick' host='localhost' password=sadf")
    print '123'
except:
    print "I am unable to connect to the database"

 Now, its (finally) time to start playing with some data. Quite a bit of debugging was done within the terminal ($ psql); once commands determined they would be added to python. First, two tables were created (Current_Data and Current_Data_1). Date + time formatting needs to be adjusted. This first set of code is a proof of concept, very messy, simply trying to see if data moves correctly.

Code here: Its a little too large to put in the page.

 Two slightly different tables. Date and time will be stored in the first or first two columns, then each column will contain the current wait time. Each location has two data points (registration and licensing). Once the table is created we can add some fake data from a list. After running this script, used the terminal to navigate to psql and verify that data was recieved.

Code here: Its a little too large to put in the page.

 Next, pretty large step: Combine the ability to inject data with previously created webscraping. The script is run every 5 minutes with a cron job; webscraped data is then stored in the database!

Code here: Its a little too large to put in the page.

 After a weekday has passed, the data is retrieved and graphed:

Code here: Its a little too large to put in the page.

 It isn't pretty, but its working.

Creating and connecting to heroku psql Addon:

 So far, testing psql was done on a local psql. Time to make a heroku addon and start using it, both from the deployed app and during local development. General heroku instructions here. First thing is to install the needed modules within the local python. The local version of python needs psycopg2 and pandas installed: requirements.txt is updated and the app is pushed just to test if additions were added correctly.

pip install pandas
pip install psycopg2
pip freeze > requirements.txt
git add .
git push -m "testing correct pip config"
git push heroku master

 Create a postgresql database on Heroku:

heroku addons:create heroku-postgresql:hobby-dev

 The DATABASE_URL is created on Heroku and can be accessed in the database interface or from the command line:

heroku config:get DATABASE_URL -a rmv-scraping

 Database credentials must be accessed from within the running app. The app must be able to access the database both when run from a local environment ($python hello.py) and once deployed on Heroku. The general scheme to achieve this is to have a try except branch at the top of the file; one path gathers the credentials for a local execution while the other gathers credentials for the deployed app. Logic for determining whether an execution is local or deployed is done by creating a config.py file in the root folder which contains initially credentials for debugging [eventually unused]. Config.py is then added to the gitignore so the config.py file exists on the local deployment but not the deployed. So geneally:

try:
    print 'Config imported:  Local build'
    from config import * 
    #   gather DB credentials for local 
except:
    print 'no config imported: this is a deployed build'
    #   gather DB credentials for local 

 Prints will eventually be removed, but left for the moment to make sure things are running correctly. Prints appear in the command line when run locally ($python hello.py). Alternatively, when deployed, the heroku website has a "view log" option within the dashboard. Locally, the DATABASE_URL is fetched from Heroku ($heroku config:get DATABASE_URL -a rmv-scraping) [Note: As with other heroku commands, you must make sure this application is the promoted application, if you have multiple apps on Heroku]. When deployed, the application fetches the global varible. The specific code (Github) looks like:

try:
    print 'Config imported:  Local build'
    from config import * 
    stdout = Popen('heroku config:get DATABASE_URL -a rmv-scraping', shell=True, stdout=PIPE).stdout
    DATABASE_URL = stdout.read()
    urlparse.uses_netloc.append("postgres")
    url=urlparse.urlparse(DATABASE_URL)
    DB_name = url.path[1:]
    DB_user = url.username
    DB_password = url.password
    DB_host = url.hostname
    DB_port = url.port
    DB_name=DB_name[:-1]    #not even a little sure why this is needed, but there was a phantom /n
    print ' DB_name: ', DB_name
    print ' DB_user: ', DB_user
    print ' DB_password: ', DB_password
    print ' DB_host: ', DB_host
    print ' DB_port: ', DB_port    
    print 'got config file this must be a local build'
except:
    print 'no config imported: this is a deployed build'
    urlparse.uses_netloc.append("postgres")
    url = urlparse.urlparse(os.environ["DATABASE_URL"])
    DB_name = url.path[1:]
    DB_user = url.username
    DB_password = url.password
    DB_host = url.hostname
    DB_port = url.port
    print ' DB_name: ', DB_name
    print ' DB_user: ', DB_user
    print ' DB_password: ', DB_password
    print ' DB_host: ', DB_host
    print ' DB_port: ', DB_port
    print 'database type : ', type(url.path[1:]), ' Database name: ', str(url.path[1:])
    print 'user type : ', type(url.username), ' User:  ', str(url.username)
    print 'password type: ', type(url.password), ' Password: ', str(url.password)
    print 'host type : ', type (url.hostname), ' Host: ', str(url.hostname)
    print 'port type: ', type(url.port), ' Port: ', str(url.port)

 Once credentials are gathered, connections are made with psycopg2 and we can start tinkering:

conn = psycopg2.connect(
    database=DB_name,
    user=DB_user,
    password=DB_password,
    host=DB_host,
    port=DB_port,
    slmode='require'
    )
cur = conn.cursor()
cur.execute("CREATE TABLE test_1 (id serial PRIMARY KEY, num integer, data varchar);")