Python Development Setup
October 2, 2016
python | git | django |
Notes, Documentation, Code Snippets
Table of Contents
- Initial Setup
- Homebrew
- Python Project Setup
- pip
- Git and GitHub
- Markdown
- Jekyll
- Sass
- Heroku
- VirtualEnv
- Vagrant
- Databases
Getting Set Up
I basically just followed this guide for the most part. Essentially, install Homebrew and XCode and create a .bash_profile.
Next I followed this guide
kinda. Install both 2.x and 3.x pythons using Homebrew, and install pip,
virtualenv, virtualenvwrapper if not already installed and add a line to
.bash_profile
to only allow running pip
in virtual environments (but also
create an alias gpip
that allows you to install / upgrade globally).
Homebrew
Python, Git and other packages are best installed via Homebrew.
# Update Homebrew
brew update
# See which packages are outdated
brew outdated
# Upgrade all packages
brew upgrade
# Install a package
brew install git
# Check for problems
brew doctor
Python Project Setup
Before starting a project, I would recommend:
- Updating Homebrew and any packages that are outdated
- Updating Pip and any packages that are outdated
# Create virtualenv project with Python3
mkproject -p python3 test_project
# or if the project folder already exists
mkvirtualenv -p python3 test_project
# Install Django into virtualenv
pip install django
Pip
{% highlight sh %}
Upgrade
pip install -U pip gpip install -U pip # in my case due to my .bash_profile
Search packages
pip search django
Install
pip install django pip install django==1.8.1
Upgrade
pip install -U django
Uninstall
pip uninstall django
Show info about package
pip show django
Show what’s installed in current virtualenv
pip list
Show what’s outdated
pip list –outdated
Dump current set up to file
pip freeze > requirements.txt
Install based on a requirements file
pip install -r requirements.txt {% endhighlight %}
Git and GitHub
Resources
- http://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository
- https://www.codecademy.com/learn/learn-git
- https://www.atlassian.com/git/tutorials/using-branches/git-branch
- http://learngitbranching.js.org/
- http://stackoverflow.com/questions/2621610/what-git-branching-models-work-for-you
Basic Commands
{% highlight sh %}
To install
brew install git
Pull down and clone an existing repository
git clone https://github.com/getup8/basic.git
Initialize a local folder as a repository
Add files to staging
Commit the files or changes with a useful description message
git init git add [filenames or –all] git commit -m “Useful commit description”
Skip the staging area and just commit new changes (files need to all be
trackd already)
git commit -a -m “added new benchmarks”
Add a remote repository and push local commit to it
git remote add origin https://github.com/getup8/nano-project-1.git git push -u origin master
Get the status of files in the repository
git status
See the changes made since last commit that are not yet staged
git diff
See the changes that are staged
git diff –staged
Forget what this does differently
git diff –cached
Remove file from tracking:
git rm PROJECTS.md
Remove file from staging but not from filesystem:
git rm –cached README git mv README.md README
See commit history of repository:
git log
See commit diffs and limit to last two entries
git log -p -2
Other commands
git log –stat git log –pretty=oneline|short|full|fuller git log –pretty=format:“%h %s” –graph git log –since=2.weeks
Add a file you forgot
git commit –amend
Remove from staging
git reset HEAD CONTRIBUTING.md
Totally reset everything (throw away local commits) like you’re cloning
from remote
git reset –hard [HEAD]
Revert back to initial version of file before changes
git checkout – CONTRIBUTING.md
Revert all files
git checkout – .
Remove files that are not tracked (I think)
git clean -df
See list of current remote repositories
git remote -v
Add a remote repository
git remote add [shortname] [url] git remote add pb https://github.com/paulboone/ticgit
Need more documentation here
git fetch pb git fetch origin
Sync local repo with remote
get pull
Another way to sync with remote I believe
git fetch origin git merge origin/master
Push to remote (what does -u do?)
git push origin master
Get more info on remote
git remote show origin {% endhighlight %}
Markdown
Jekyll
I’m using Jekyll to make writing notes and documentation easier for my GitHub projects and other ventures. Below are some quick resources.
Resources
- https://help.github.com/articles/using-jekyll-with-pages/
- http://jekyllrb.com/
- http://pygments.org/docs/lexers/
Quick Commands
{% highlight sh %}
After making a change in the project run this to build into _site directory
jekyll build
Build the project into _site and watch and auto-build after changes
jekyll build –watch
Firing up the local server http://localhost:4000
bundle exec jekyll serve {% endhighlight %}
More to come..
Sass
Resources
- http://sass-lang.com
- http://sass-lang.com/guide
- http://sass-lang.com/documentation/file.SASS_REFERENCE.html
Installation is as easy as:
sudo gem install sass
Example Usage
Some simple documentation of key concepts below. The code is written in SCSS not SASS, the main difference being SASS doesn’t use semi-colons or parens.
Variables
{% highlight scss %} $font-stack: Arial, sans-serif; $color: #CCCCCC; {% endhighlight %}
Nesting
{% highlight scss %} nav { ul { } } {% endhighlight %}
Partials
Write snippets in separate files to include in other Sass files: _partial.scss
Used with the @import directive: @import ‘partial’;
Mixins
Kinda like functions, useful especially for vendor prefixes.
{% highlight scss %}
@mixin border-radius($radius) {
-webkit-border-radius: $radius;
-moz-border-radius: $radius;
-ms-border-radius: $radius;
border-radius: $radius;
}
.box { @include border-radius(10px); } {% endhighlight %}
Extend/Inheritance
Share CSS between selectors. Inherit from other selectors. {% highlight scss %} .message { border: 1px solid #ccc; padding: 10px; color: #333; }
.success { @extend .message; border-color: green; } {% endhighlight %}
Operators / Math
Multiply, divide, add, subtract, modulus. {% highlight scss %} .container { width: 100%; }
article[role=“main”] { float: left; width: 600px / 960px * 100%; }
aside[role=“complimentary”] { float: right; width: 300px / 960px * 100%; } {% endhighlight %}
Running Sass
To run Sass from the command line, just use:
sass input.scss output.css
You can also tell Sass to watch the file and update the CSS every time the
Sass file changes:
sass --watch input.scss:output.css
If you have a directory with many Sass files, you can also tell Sass to watch
the entire directory:
sass --watch app/sass:public/stylesheets
Further Reading
Getting Started on Heroku
Following the steps outlined here.
Initial assumptions
- Python installed
- virtualenv, setuptools, pip installed
- Heroku account
- Heroku toolkit installed (Heroku client, git, Foreman - an easy way to run apps locally)
Steps
Clone the getting started repository and move into the directory. This initializes git and sets up some boilerplate structure and files. {% highlight sh %} git clone https://github.com/heroku/python-getting-started.git cd python-getting-started {% endhighlight %}
Login to Heroku and create an app on Heroku, which also creates a git “remote” which is associated with the local git repository. {% highlight sh %} heroku login heroku create {% endhighlight %}
Deploy the code to Heroku. {% highlight sh %} git push heroku master {% endhighlight %}
Allocate one dyno (lightweight linux “container”) to the app. {% highlight sh %} heroku ps:scale web=1 {% endhighlight %}
Open the app to check it out. {% highlight sh %} heroku open
https://aqueous-cove-8924.herokuapp.com/
{% endhighlight %}
View server logs. {% highlight sh %} heroku logs –tail {% endhighlight %}
Procfiles define the processes that run and the command that is executed to start your app.
To add more containers/dynos, you can scale up to make the app persistently “up”. You only get 750 free dyno hours though so this would make our app not free after a short time. {% highlight sh %} heroku ps:scale web=2 {% endhighlight %}
Declaring app dependencies is done via requirements.txt. Heroku reads this file and installs the proper dependencies via pip. To do this locally, we should use a virtual env. {% highlight sh %} virtualenv venv source venv/bin/activate {% endhighlight %}
Run pip to install these dependencies locally (Heroku does the same). Doing this lets us run the app locally, which is really what we want to do in development. {% highlight sh %} pip install -r requirements.txt –allow-all-external {% endhighlight %}
Running foreman reads the Procfile (similar to Heroku) and runs the app: {% highlight sh %} foreman start web
http://0.0.0.0:5000
{% endhighlight %}
In order to modify anything (let’s modify the Procfile for instance to require another plugin), re-run these commands. I’m guessing pip isn’t necessary unless you need something else installed. {% highlight sh %} pip install -r requirements.txt –allow-all-external foreman start {% endhighlight %}
Now we deploy these changes to Heroku. First add the modified files to git and then commit the changes to the local repository. {% highlight sh %} git add . git commit -m “Demo” {% endhighlight %}
Then just deploy to Heroku again. {% highlight sh %} git push heroku master heroku open {% endhighlight %}
There are tons of addons for Heroku as well, should be checked out at some point.
To run a REPL (read eval print loop) / shell interface, use the run command. {% highlight sh %} heroku run python manage.py shell heroku run bash
Type “exit” to exit the shell and release the dyno.
{% endhighlight %}
Check out config vars later, maybe good for dev/prod and other things?
Databases are addons, you can see the current ones by just running: {% highlight sh %} heroku addons {% endhighlight %} I think they set up postgres automatically.
You can also run the following to see the current config vars set up (relevant to set up database): {% highlight sh %} heroku config # General config heroku pg # Specifically for Postgres {% endhighlight %}
I guess this app is already configured with a database and you can visit (what?
i think just a view) by just appending /db after the URL:
https://aqueous-cove-8924.herokuapp.com/db
But initially it’ll give an error b/c the tables don’t exist. Need to initialize it first, the Django way. {% highlight sh %} heroku run python manage.py syncdb {% endhighlight %} It creates the tables and now visiting that URL/view works.
You can access the DB directly (assuming postgres is installed locally) via: {% highlight sh %} heroku pg:psql \dt # to show tables in postgres {% endhighlight %}
Rename the app
{% highlight sh %}
heroku apps:rename
Further Reading:
- https://devcenter.heroku.com/articles/heroku-postgresql
- https://devcenter.heroku.com/articles/getting-started-with-python#next-steps
- https://devcenter.heroku.com/articles/getting-started-with-django
VirtualEnv
Resources
I installed all of those.
Before doing anything anymore: {% highlight sh %}
Create the project, virtualenv, move into the dir and activate
mkproject dt
Delete
rmvirtualenv dt
Show current environments
lsvirtualenv
To get out of the venv
deactivate
To move to directory and activate an existing virtualenv
workon dt {% endhighlight %}
Vagrant
Resources
If you’d like to see what was installed in the VM, look in /vagrant/pg_config.sh.
{% highlight sh %} vagrant up vagrant ssh exit # exit the command line vagrant halt {% endhighlight %}
Random:
To connect psql to the forum database for Lesson 3, type psql forum
at the
command line. To exit psql, type \q
or Control-D (^D).
Postgres
Resources
Commands
{% highlight sh %} – Connect to database psql [db] {% endhighlight %}
{% highlight psql %} – Show databases \l \list \l+
– Connect to database \c [database_name]
– Show tables \dt
– Desc table \d [tablename]
– Quit \q
– Help - get all these commands \h
– Connectinon info \conninfo
– Change database name ALTER DATABASE name RENAME TO new_name
{% endhighlight %}
DB Link
http://www.postgresql.org/docs/current/interactive/dblink.html
{% highlight psql %}
CREATE EXTENSION dblink;
– Verify SELECT pg_namespace.nspname, pg_proc.proname FROM pg_proc, pg_namespace WHERE pg_proc.pronamespace=pg_namespace.oid AND pg_proc.proname LIKE ‘%dblink%’;
– Test connection SELECT dblink_connect(‘host=localhost user=alexscott dbname=wewander_db’);
– Add back the social login set up info UPDATE django_site SET domain = ‘localhost:8000’, name = ‘localhost’ where id = 1;
INSERT INTO socialaccount_socialapp SELECT * FROM dblink(‘dbname=wewander_db2’, ‘SELECT * FROM socialaccount_socialapp’) AS t1(id integer, provider character varying(30), name character varying(40), client_id character varying(191), secret character varying(191), key character varying(191));
INSERT INTO socialaccount_socialapp_sites (socialapp_id, site_id) VALUES (1, 1), (2, 1), (3, 1);
{% endhighlight %}
{% highlight psql %} CREATE DATABASE db DROP DATABASE db DROP TABLE table CREATE VIEW view AS SELECT … – Useful for aggregations {% endhighlight %}
Database Concepts
- Atomicity: a command executes as a whole or not at all (commit!)
- Normalized vs Denormalized: Normalized databases have relationships between
tables that are really there among the data.
- Every row has the same number of columns
- There is a unique key for every row, every column describes the key
- Facts that don’t relate to the key belong in their own table
- Tables shouldn’t imply relationships that don’t exist
- http://en.wikipedia.org/wiki/Database_normalization
- http://www.bkent.net/Doc/simple5.htm
SQLAlchemy
Resources
Protocols
Protocols are the “gramatical rules” that ensure that clients and servers (and really any computer on the internet) are communicating in the same way.
- Transmission Control Protocol (TCP) Enables info to be broken into small packets and sent. Counterpart is UDP (User Data Protocol) which is good for streaming
- Internet Protocol (IP) Like a postal address. Can be static or dynamic. Computer looks up IP in a DNS (Domain Name Server) Uses the address to initiate communication Operating systems use Ports to designate channels of communication on the same IP address. Can range from 0 to 65,536. First 10K often reserved by OS. 80 is most common. 8080 is often used for web communication. When client and server on the same machine, we use “localhost” which has the IP address 127.0.0.1
- Hypertext Transfer Protocol (HTTP, HTTP2?) Clients tell servers what they want using HTTP “verbs” Most common are GET and POST requests GETs are sometimes called safe methods Server replies to these requests with a Status Code. Can also supply requested resources like HTML, CSS, JS, images, etc 200: Successful GET 301: Successful POST 404: File not found