1. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Re-inventing Python Packaging &
Testing (0.9)
Holger Krekel, http://twitter.com/hpk42 (http://twitter.com/hpk42)
Pycon Russia, Ekaterinburg, 25th Feb 2013
(XKCD of course)
1 of 41 03/01/2013 12:04 PM

2. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Very happy to be here!
2 of 41 03/01/2013 12:04 PM

3. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
My background
started games programming 1986
studied computer science in the 1990ties
lots of Assembler, C, C++, distributed programming
came to Python around 2001
co-founded PyPy project in 2003, core dev for many years
author of pytest, tox, execnet libraries
3 of 41 03/01/2013 12:04 PM

4. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
And now for something completely
different
4 of 41 03/01/2013 12:04 PM

5. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Remember
Perl:
There is more than one way to do it.
Python:
There should be one -- and preferably only one -- obvious way to do it.
5 of 41 03/01/2013 12:04 PM

6. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Second version syndrome
Python3 may have beaten Perl6
(http://www.theregister.co.uk/2008/12
/04/python_3point0_release/)
(from theregister.co.uk)
6 of 41 03/01/2013 12:04 PM

7. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
but Perl is coming back to haunt us ...
7 of 41 03/01/2013 12:04 PM

8. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Comprehensive Perl Archive network
(CPAN)
cpan.pm comes with base Perl installs:
downloads/builds/installs and tests packages and deps
cpan index has hundreds of public mirrors
uses declarative metadata since 2003
can verify cryptographic signing of packages
8 of 41 03/01/2013 12:04 PM

9. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
CPAN server / uploading
PAUSE is the upload server for perl packages:
supports/can enforce versioning semantics
development releases are mirrored but not indexed
policy for taking over maintenance of abandoned packages
provides a mirrored un-indexed "home" area for authors/uploaders for patches, little
docs
9 of 41 03/01/2013 12:04 PM

10. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Python Installer situation Feb 2013
no download+install tool coming with core Python
two ways to download/install packages (pip, easy_install):
pip installs only source distributions
easy_install also binary distributions
slow: dependency discovery requires client-side crawling for packages in all
download and homepage sites
no reliable way to test before install
no cryptographic signature verification
10 of 41 03/01/2013 12:04 PM

11. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
upload/PYPI server interaction today
pypi.python.org server code not easily deployable on your laptop
no enforced versioning semantics
no policy for taking over maintenance of abandoned packages
brittle protocol: network and server failures of any download/homepage server
lead to aborted installs or installation of wrong versions
metadata through code (setup.py) has tons of problems, has been hard so far
to move away from.
and as to mirrors ...
11 of 41 03/01/2013 12:04 PM

12. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
CPAN Mirror status
(http://mirrors.cpan.org/)
(http://mirrors.cpan.org/ (http://mirrors.cpan.org/) )
12 of 41 03/01/2013 12:04 PM

13. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
PyPI Mirror status
(http://www.pypi-
mirrors.org)
(http://www.pypi-mirrors.org (http://www.pypi-mirrors.org) )
13 of 41 03/01/2013 12:04 PM

14. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
We can stop the comparison already ...
(from www.cavstheblog.com)
14 of 41 03/01/2013 12:04 PM

15. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Three Lessons learned
15 of 41 03/01/2013 12:04 PM

16. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Comparing with others can quickly
make you unhappy
16 of 41 03/01/2013 12:04 PM

17. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Perl and Python both not living up to
their mantras
17 of 41 03/01/2013 12:04 PM

18. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
There is a lot to improve on
18 of 41 03/01/2013 12:04 PM

19. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Python Tools and PEPs to improve
setuptools/distribute projects
pip/distlib/pkglib/... projects
PEP386 new version comparison for distutils
PEP345 Packaging Metadata 1.2
PEP376 infrastructure for managing distributions locally
PEP405 virtual environments
PEP426 Packaging Metadata Version 1.3 (finishing)
PEP427 Wheel binary package format (APPROVED)
19 of 41 03/01/2013 12:04 PM

20. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
beware: the standardization trap
(one of the many great XKCD comics)
20 of 41 03/01/2013 12:04 PM

21. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
To avoid the standardization trap:
Don't demand that the world changes first
before your tool/idea can be used
(was also my motto when starting PyPy)
21 of 41 03/01/2013 12:04 PM

22. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Rather: write meta tools
configure and invoke existing tools
make it work for most use cases
enable new facilities/standards from there
22 of 41 03/01/2013 12:04 PM

23. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
example from the testing arena
(you do know the goat is Python's testing mascott, right?)
23 of 41 03/01/2013 12:04 PM

24. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Python Testing tools
nose: popular unittest-runner
py.test: popular generic test runner
unittest/2: standard library TestCases
zope.testing: layered-setup unittest-runner
twisted.trial: running twisted/async test cases
many other methods, including "make test"
setuptools has "setup.py test"
Would inventing a new test runner help to standardize testing?
24 of 41 03/01/2013 12:04 PM

25. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
tox: a "meta" test running tool
mission: standardize testing in Python
configure multiple dependency configs
install packages in virtualenv environments
run tests with your tools of choice
act as a frontend to CI servers
See http://tox.testrun.org (http://tox.testrun.org) for details.
25 of 41 03/01/2013 12:04 PM

26. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
But that's about testing ...
what about packaging/pypi?
26 of 41 03/01/2013 12:04 PM

27. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
I did a little prototyping ...
27 of 41 03/01/2013 12:04 PM

28. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
shown first here at Pycon Russia :)
28 of 41 03/01/2013 12:04 PM

29. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
devpi: a meta tool for packaging
activities
devpi serves two purposes:
it's a new compatible index and upload server
subcommands for managing release and QA workflows
29 of 41 03/01/2013 12:04 PM

30. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
why a new approach to index serving?
there is no good existing pypi server to build these features:
multiple indexes and staging (dev->QA->PROD)
merging of upstream (pypi.python.org) indexes
server-side crawling of download/homepages
recording of automated test results with each package
seemlessly allow offline usage (e. g. on a laptop)
30 of 41 03/01/2013 12:04 PM

31. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Open Source principle:
talk is cheap, show working code!
31 of 41 03/01/2013 12:04 PM

32. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
$ devpi server
multiple indexes, organized as ~USERNAME/INDEXNAME
each index has an upstream index whose packages are seemlessly merged
each index is "crawl-clean" for clients
uploading a package to an index will not push it to upstream
compatible for pip, easy_install and setup.py upload
enough theory - on to a a little demo :)
32 of 41 03/01/2013 12:04 PM

33. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
summary of devpi server demo insights
upstream merging makes a private overlay index simple
crawling at server-side makes it easy and fast for installers
devpi maintains a nginx-servable static view
but:
typing in index server urls, maintaining .pypirc config is burdensome with
multiple indexes ... therefore ...
33 of 41 03/01/2013 12:04 PM

34. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
devpi also provides "workflow"
subcommands
use set current pypi index
install download and install one or more packages
upload build and upload packages from a checkout
test download and test a package
push push a package to another index
34 of 41 03/01/2013 12:04 PM

35. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
$ devpi test
example:
$ devpi test -e py25,py33 mypkg
download and unpack a package from current index
call tox for provisioning and performing tests
stream test results from a test run (currently pytest only, nose/unittest to follow)
35 of 41 03/01/2013 12:04 PM

36. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
$ devpi push
example:
$ devpi push mypkg https://pypi.python.org
pushes uploaded packages to another index
currently specified by "PKGNAME-VER"
in the future this could be done to constraints like "all tests on windows need to
pass"
36 of 41 03/01/2013 12:04 PM

37. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
devpi: typical Open Source work flow
1. start a localhost index server:
$ devpi server
2. use a user-specific index:
$ devpi use http://localhost:3141/~hpk42/dev/ (http://localhost:3141/~hpk42
/dev/)
3. upload a package from a checkout:
$ devpi upload
4. perform tests on multiple machines from multiple users:
$ devpi test PKGNAME
5. test results at http://localhost:3141/~hpk42/dev/tests (http://localhost:3141/~hpk42
/dev/tests)
6. when release is ready:
$ devpi push PKG https://pypi.python.org
37 of 41 03/01/2013 12:04 PM

38. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
devpi implementation plan
to be MIT licensed
test-driven development
get early adopters and streamline processes according to real-life usage
don't aim for standard lib or PEPs (for now)
38 of 41 03/01/2013 12:04 PM

39. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
my priorities
use cases from contracting companies
use cases from my and others open source work
39 of 41 03/01/2013 12:04 PM

40. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
Open areas / future work
use VCS for index persistence?
plugin system for upload/serve/test/install
implement mirroring of index as static files
support declarative metadata / Wheel format /...
support server-side PGP signing and verification
a public, redundant,global web service
CI: per-commit automatic integration, testing and packaging
40 of 41 03/01/2013 12:04 PM

41. file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
devpi going public mid 2013 ...
aiming for a <1.0 public release, to involve early adopters and contributors for finalizing
processes and good defaults.
Approach me here at Pycon Russia 2013
and thanks for listening!
No public channels yet, but am going to announce them via:
http://twitter.com/hpk42 (http://twitter.com/hpk42)
http://holgerkrekel.net (http://holgerkrekel.net)
41 of 41 03/01/2013 12:04 PM