Development environment
for Django project - my way
!
Robert Lujo, 2014

about me
• software
• professionally 17 y.
• python since 2.0, django since 0.96
• freelancer
• more info -> linkedin

“Software Development
Environment” is …?
I found no unique / unified answer …

Software Development
Environment (1)
refers to a server tier designated to a specific stage in a
release process.
List of tiers: Local, Development, Integration, Test/QA, UAT,
Stage/Pre-production, Production/Live
Hm…!
https://en.wikipedia.org/wiki/Development_environment_
%28software_development_process%29

Software Development
Environment (2)
a collection of procedures and tools for developing, testing
and debugging an application or program.
Getting closer …!
http://www.techopedia.com/definition/16376/development-environment

Integrated development
environment - IDE
!
software application that provides comprehensive facilities to
computer programmers for software development.
… source code editor, build automation tools and a debugger...
Intelligent code completion features ... compiler, interpreter, ...
version control system and various tools are integrated to simplify
the construction of a GUI ... class browser, an object browser,
and a class hierarchy diagram.
Closer and closer …!
https://en.wikipedia.org/wiki/Integrated_development_environment

Software Development
Environment (3)
contains everything required by a team to build and
deploy software-intensive systems (where software is an
essential and indispensable element)
I approve!!
http://www.ibm.com/developerworks/rational/library/define-scope-development-
environment/index.html

What is “everything”?
Software development lifecycle (SDLC)
• Planning
• Designing
• Implementation, testing and documenting
• Deployment and maintenance
https://en.wikipedia.org/wiki/Software_development
!

What about “a team”?
Interacts with:
• Software project management - organisation,
coordination, covers: risk, change, software configuration
and release management
https://en.wikipedia.org/wiki/Software_project_management
• Software development process a.k.a. software
development methodology - e.g. waterfall, agile …
https://en.wikipedia.org/wiki/Software_development_process

My priorities
1. implement at least 90% of the most important
requested features
2. shortest possible time needed for “from request
to deploy” cycle
3. low-cost scaling - human/machines - initial
setup of DevEnv or running system
4. ensure high quality deliverables

!
now, the real stuff
!
btw. where has Django disappeared?

Initial setup - proj layout
~/env
├── proj-1
├── cli-1
├── …
├── biz
├── …
└── proj-N

Initial setup - proj layout
~/env/biz
├── archive
├── biz-web (GIT)
│ ├── ... (to be continued)
├── biz-data(GIT)
├── biz-ie (GIT)
├── doc -> ln -s ~/Dropbox/…/biz/doc
└── temp

Initial setup - python proj layout
~/env/biz/biz-web (GIT)
├── bin
├── doc (sphinx,rst: readme,todo,
│ install, ...)
├── pybiz (venv python installation)
├── reqs (base.txt, dev.txt)
├── src
│ ├── (to be continued)
│ ...
└── var
├── biz.sqlite
├── biz-app.log
├── biz-jobs.log
└── bulk-data

Initial setup - django proj layout
~/env/biz/biz-web/src
├── bizproj (global project folder - formats, settings)
├── bizjobs
│ ├── ...
├── bizapp
│ ├── management / commands
│ ├── migrations
│ ├── scripts
│ ├── templates
│ │ ├── biz
│ │ │ ├── components
│ │ │ ├── backoffice
│ │ │ ├── protected
│ │ │ ├── public
│ │ │ └── trial
│ │ ├── admin
│ │ ├── rosetta
│ └── templatetags
├── locale (de/en)
├── media
└── static (js, css, images, libs, fonts) # to bizapp?

Initial setup - django settings
~/env/biz/biz-web/src/bizproj/settings
!
base.py
from .secret import *
!
dev.py
dev_demo.py
!
test.py
!
secret.py # not in git repo
EMAIL_HOST_USER = "..."
DATABASES = {
!
secrete.py.template # is in git repo
!
dbrouter.py
testrunner.py

Initial setup - .gitignore
!
~/env/biz/biz-web/.gitignore
…
pybiz
var
src/media/*
secret.py
fabfile_settings.py
*.pyc
*.swp
*.pickle
*.sqlite
*.bak
…

File and directory naming
• very careful in naming !
• naming convention - inherit existing +
adaptations!
• very careful in choosing folder/structure!
• bigger files better than many files!
• less files/folders is better than more!

Project dependencies
• as small as possible - more components -> more complex -> more integration/
upgrade/other issues
• choose very carefully
• proven, stable, I/team is familiar with
• easily installable/upgradeable
• adaptable (when possible)
• when choosing new
• read documentation/read reviews/choose several/experiment/make
decision
• bigger community -> better

Project dependencies
• solutions
1. make a script to setup everything from the
scratch
2. install manually/reuse existing images and
then freeze state in VM, Vagrant, Docker
image
3. combine both

Project dependencies
• make a shell script
• two types of dependencies
• pip installable
pip install -r reqs/dev.txt
• pip non-installable
• set of shell commands to install everything that is missing
• Problem: different OS-es, e.g. OSX, Linux CentOS?
if [ $(uname) == 'Linux' ] …

Project deps - pip requirements
!
~/env/biz/biz-web/reqs
!
base.txt :
django>=1.5,<1.6
pillow
reportlab
johnny-cache
markdown
elasticsearch
beautifulsoup4
lxml
requests
celery[redis]
…
!
dev.txt :
-r base.txt
south
django_extensions
django-debug-toolbar
django-rosetta # for translation
mock
ipdb
…

Project deps - shell script
!
non-PIP installables
!
~/env/biz/biz-web/bin/biz # chmod u+x
!
case "$1" in
env)
if [ ! -d $BIZ_HOME/pybiz ]; then
cd $BIZ_HOME
sudo apt-get install python-devel
sudo apt-get install python-pip
sudo apt-get install python-xml
sudo apt-get install elasticsearch
...
pip install virtualenv
virtualenv pybiz
fi
!
(to be continued …)

Project deps - shell script
!
PIP installables
!
(continued …)
!
if [ -f $BIZ_HOME/pybiz/bin/activate ]; then
source $BIZ_HOME/pybiz/bin/activate
echo "pip=`which pip`, python=`which python`”
!
pip install -q --requirement=$BIZ_HOME/reqs/dev.txt
!
cd $BIZ_HOME/src/bizapp
export PYTHONPATH=$PYTHONPATH:$BIZ_HOME/src
echo "== env 'BIZ' initialized, checking base packages"
python -c"import django;
print(' Django - ok v%s' % django.get_version())"
python -c"import redis ;print(' redis - ok')"
python -c"import celery ;print(' celery - ok')"
python -c"import bizappp ;print(' biz app - ok’)"
…
fi
!
(to be continued …)

Project deps - shell script
!
check services
!
echo "== Is redis running?"
ps ax | grep -v grep | grep redis
!!
echo "== Is elasticsearch running?"
ps ax | grep -v grep | grep elasticsearch | awk "{print $1, $2,
$3, $4, $5, "..."}"
!
…

Project management shell script
• why?
• to have one central place where to do project management
• what? everything that needs automation
• setup initial environment, update environment
• open editor/IDE
• run tests
• deployment, remote management

Project management shell script
!
# biz
!
Usage:
!
*env - will initialize dev environment
!
redis - start redis in background
elas - start elasticsearch in background
cel - start celery worker in background
!
doc-update/du - will update doc/build/html wiki doc
doc - will open doc/build/html/index.html - home of wiki
!
log2 - will tail app log (if located in local var directory)
!
run/n - django-admin runserver
django/d - django-admin
sp - django-admin shell_plus
!
test-ping/tp - ping production
test/t - django-admin test --settings=test_settings
trans-make|tm - TRANSLATION - calls makemessages
!
r/remote - will call fabric script for deployment and remote operations
!
migrate|mig - do south migration - only dev system
mig-auto - create new south auto-migration - only dev system
!
browse|b - browse to localhost:8000
issues - browse to opened issues on bitbucket
!
* - should be called with source/.

Editor: ViM
• although steep learning curve, is:
• very adaptable
• unique features philosophy
• not in the way, very effective
• personal preference:
• I don’t feel comfortable in IDE
• I like to be close to “bare metal”
• to use editor designed for source code editing

• personal preference:
Editor: ViM
• I don’t like plugins too much, only .vimrc
• when working on the server to have the same possibilities
• configuration
• server
scp ~/.vimrc user@server:/home/user/
• local development machine
ln -s ~/Dropbox/env/.vimrc ~
• flavors
• gVim / MacVim - GUI, ViM - on every server

Development on the server
• with project shell script one can do directly on
the server
• devenv setup/upgrade
• webapp setup/upgrade
• vim & shell-script -> do “live” development
• run dev server

File browser
• personal preference:
• I like side by side file browsers
• usual file operations: create folders, check files,
search for files, zip etc.
• easy integration with editor (F4)
• not found “one-to-rule-them-all”
• TotalCommander, FreeCommander, DoubleCmd …

Idea evaluation
• evaluate a new idea or to test concept
• ipython / django shell_plus
• use reload:
from biz import models
models.test_something()
# change models and save
reload(models)
models.test_something()
• next step is - create a module/function

• I use a lot
Debugging
• plain-old pdb or sometimes ipdb
• conditional breakpoints:
if i==500:
import pdb; pdb.set_trace()
• conditional breakpoints (2):
try:
bad_bad_code()
except:
import pdb; pdb.set_trace()
• server side - logging, send emails on error
logger.error(“this will be send by email)
logger.error(“in try/except - will be send by email with traceback”)

• Unit tests
• integration tests
Testing
• “click-testing” - local development server
• automating http requests to staging/production
servers (from curl, nagios to dedicated services)
• “click-testing” - staging/production servers

Deployment
• must be simple, automated, fast and reliable
• usually:
• setup environment
• git push+pull / collectstatic / restart server
• analyse logs
• run integration tests on server instance
• example:
git ci -am”very important fix”
biz r test upgrade prod upgrade

Philosophy
• “Unix philosophy by Mike Gancarz” filtered:
Make each program do one thing well.
Use software leverage to your advantage.
Use shell scripts to increase leverage and
portability.
(btw. Store data in flat text files)
https://en.wikipedia.org/wiki/Unix_philosophy

Thanks for the patience!
Questions?
!
robert.lujo@gmail.com
@trebor74hr