Slides from Plone Conf 2008 (Washington D.C.)

1. delivering applications
with zc.buildout
and a distributed model
Tarek Ziadé <tarek.ziade@ingeniweb.com>
1

2. Who am I ?
New to Plone (~1 year) -> I worked on peripheral matters
Used to Zope (CPS core developer)
CTO at Ingeniweb - a Plone company (>20 developers)
Python fan - wrote some books about it
(even in english, because you don’t hear my accent in a book)
involved in plone.org migration, PSC, zc.buildout
2

3. It will look good on your desk
besides Martin Aspeli’s book
Special promotion during the PloneConf :
buy one book == get a big hug
3

4. Thanks !
* Andreas Jung
* Veda Williams
* David Glick
* Jeff Kowalczyk
* Youenn Boussard
* Christian Klinger
* Jesse Snyder
* Alec Mitchell
* John Habermann
* Maurits van Rees
* Jean-François Roche
* Martin Aspeli
* Alain Meurant
* Aleksandr Vladimirskiy
* Jon Stahl
* Alexander Limi
* Stephen McMahon
4

5. Part 1 - working with packages
Part 2 - working with zc.buildout
Part 3 - application lifecycle
5

6. Part 1 - working with packages
6

7. distutils
builds and distributes a package
registers and uploads it to PyPI
7
7

8. get the screencast at http://ziade.org/ploneconf
8
8

9. +1
distutils
is
the standard
in the whole Python community.
9
9

10. -1
distutils
is
broken
(but there’s hope)
10
10

11. setuptools
simple dependencies management, namespaced package
egg distribution
provides easy_install
11
11

12. “python setup.py bdist_egg”
one single .egg file
12
12

13. quot;Eggs are to Pythons as Jars are to Java...quot;
egg
=
deployment format
in a zip archive
13
13

14. “sdist” is enough, though
(and should be mandatory)
14
14

15. get the screencast at http://ziade.org/ploneconf
15
15

16. easy_install my_package
=
will get and install “my_package”
from “PyPI”
16
16

17. +1
setuptools
is
the de facto standard
in the Plone community.
17
17

18. -1
setuptools
is
broken
too (pje says)
(but there’s hope)
18
18

19. packaging future in Python
?
19
19

20. PJE Debian packagers BSD packagers Us !
Ian Zope 3
guys
Repoze Discussions
guys
Python
PEPs core
TG2 guys developers
Pylons
guys Guido
Heaven distutils 2 20
20

21. Packaging future in Python
hard to know.
but community-driven
come sprint with us saturday !
21
21

22. It’s OK, fellows
we will eventually
make it through.
22
22

23. Problems with packaging ?
#1 PyPI == SPOF
#2 packages need privacy sometimes
#3 plone.org/products is *dying*
23
23

24. Solutions ?
#1 PyPI == SPOF
Make a PyPI mirror
#2 packages need privacy sometimes
Run you own private PyPI
#3 plone.org/products is *dying*
Make plone.org/products PyPI compatible
24
24

25. #1 Make a PyPI mirror : Smart mirroring
easy_install collective.eggproxy
25
25

26. 26
26

27. In zc.buildout : the index option
[buildout]
index = http://my.mirror:8888
all calls will go through the proxy
the mirror is filled on-demand
27
27

28. #2 Run your own private PyPI
PloneSoftwareCenter !
28
28

29. get the screencast on http://ziade.org/ploneconf
29
29

30. collective.dist
python 2.6 new “register” and “upload” commands
“mregister” and “mupload” in 2.4 and 2.5
$ python setup.py mregister sdist mupload
$ python setup.py mregister sdist mupload -r plone.org
$ python setup.py mregister sdist mupload -r my-company
...
30
30

31. Developer side : multiple servers with .pypirc
[distutils]
index-servers =
pypi
plone.org
my-company
[pypi]
username:tarek
password:hahaha
[plone.org]
username:tarek
password:hahaha...
repository:http://plone.org/products
[my-company]
username:tarek
password:hahaha..hahaha.haha...ha..pfiiiuuuuuu..and..hahahaahah.i..am..dying..
repository:http://my-company.com/products
31
31

32. Let’s try our new PyPI
with collective.dist
32
32

33. get the screencast on http://ziade.org/ploneconf
33
33

34. #3 Make plone.org/products
PyPI compatible
plone.org was
suppose
to switch to that for months.
Lazy guys
34
34

35. Let’s try new.plone.org
35
35

36. get the screencast on http://ziade.org/ploneconf
36
36

37. The big picture
37
37

38. Yes !!!
38
developer #1
38

39. Yepeee !!!
Releasing everywhere
let’s spam PyPI and plone.org
tchak ! tchak ! tchak !
developer #2
39
39

40. Hahaha !
We have our package all over
the place.
developer #3
40
40

41. end of part 1
questions ?
#1 Make mirror
#2 Run your own PyPI
#3 push to several servers
#3 use “mregister” and “mupload”
41
41

42. Part 2 - working with zc.buildout
(reminders)
42
42

43. 5 hours in 2006
43
43

44. 5 hours in 2006
install python extra packages
get zope
install zope
create an instance
get extra products
read extra products doc
install extra products dependency
install extra products
doesn’t work
ahhh right, install python-ldap
checkout products in development
doesn’t work
ahhh right, wrong python-ldap version
start to work 44
44

45. Me in 2006
45
45

46. 5 minutes in 2008
46
46

47. 5 minutes in 2008
get the “buildout”
$ python bootstrap.py
$ bin/buildout
start to work
47
47

48. But that was not the main reason
48
48

49. 2005 Now
zope.interface
zope.schemas
zope.stuff
Zope
one big-O zope.whatsthat
package. “eggification”
zope.makecoffee
zope.dodishes
...
over 100 packages
49
49

50. 2005 Now
plone.ista
plone.portlets
plone.stuff
Plone
products plone.whatsthat
“eggification”
plone.makecoffee
plone.enolp
...
50
50

51. Your app.site
Killer
Plone app.skin
App
...
51
51

52. zope.stuff
...
Your plone.stuff
Distribution
...
app.site
app.skin
52
52

53. Python
site-packages
+100 packages
to run Plone
something is Increase conflict
going wrong here
probabilities
53

54. zc.buildout
=
isolated environment
54
54

55. zc.buildout best practices
#1 use the same layout for all your projects
#2 make sure all developers have the same environment
#3 use one cfg per target
55
55

56. #1 same layout for all projects
project1
docs
buildout
packages
releases
project2
docs
buildout
packages
releases
....
56
56

57. easy_install collective.releaser
57
57

58. get the screencast on http://ziade.org/ploneconf
58
58

59. #2 make sure all developers have the same environment
Warning
Plone buildouts are source based
Windows developers
Get my Windows installer : python2.4.4-win32.zip
Google “An installer for a buildout-ready Windows”
59
59

60. #3 use one cfg per target
Typical buildout layout uses the extends feature
buildout.cfg
dev.cfg (extends buildout.cfg)
prod.cfg (extends buildout.cfg)
+ bootstrap.py
60
60

61. buildout.cfg :
[buildout]
parts =
one
two
[buildout]
parts =
dev.cfg : one
two
[buildout]
three
extends = buildout.cfg
develop =
parts =
...
three
develop =
...
61
61

62. demo
creating a fresh Plone 3 buildout (Paste)
adding the dev.cfg
hooking a new development package
adding a prod.cfg
62
62

63. get the screencast on http://ziade.org/ploneconf
63
63

64. end of part 2
questions ?
#1 use the same layout
for all projects
#2 make sure all developers
have the same environment
#3 use one cfg per target
64
64

65. Part 3 - application lifecycle
65

66. 66

67. Releasing packages
Releasing the buildout
67

68. Releasing packages
for package in packages:
raise the version
edit CHANGES.txt
create a branch (svn)
go to that branch
remove the dev tag (setup.cfg)
release it with “mregister sdist mupload -r somehwere”
release it with “mregister sdist mupload -r somehwereelse”
68
68

69. Releasing packages with collective.releaser:
for package in packages:
python setup.py release
69
69

70. Defining targets in .pypirc
[distutils]
index-servers =
pypi
plone.org
private
glob-style = true
[pypi]
...
release-command = mregister sdist mupload
release-packages =
iw.*
[plone.org]
...
release-command = mregister sdist mupload
release-packages =
plone.*
collective.*
[private] 70
...
70

71. get the screencast on http://ziade.org/ploneconf
71
71

72. collective.releaser makes a lot of assumptions
it’s svn-oriented (the first use case)
72
72

73. Release the buildout
What packages should be frozen ?
- recipes
- your released packages
- exceptions (security fixes, major bug fixes)
73
73

74. Release the buildout
authentication: use lovely.buildouthttp
[buildout]
...
extensions=lovely.buildouthttp
...
repository:http://my-company.com/products
HOME/.buildout/.httpauth
pypi,http://my-company.com/products,tarek,hahaha
pypi,http://plone.org/products,tarek,hahaha
74
74

75. Release the buildout -> project layout
project
...
buildout
packages
release/0.1 <- tag for the buildout
$ cd buildout
$ svn cp . http://somewhere/releases/0.1
75
75

76. Release the buildout with collective.releaser :
with project_release
$ cd buildout
$ project_release
What version you are releasing ? 0.1
Added version file.
76
76

77. Build your release distribution
- bin/buildout on target system
- remove some stuff
- offline mode to ‘true’
- tar -czvf release-0.1.tgz release/0.1
77
77

78. Build your distribution with collective.releaser
with project_deploy
$ svn co http://somesvn/my_projet/releases/0.1 project
$ cd project
$ project_deploy prod.cfg
78
78

79. Upgrading the application
79
79

80. Upgrading the application : 3 strategies
#1 full release
#2 diff release, using collective.releaser
#3 egg pack release, using collective.releaser
80
80

81. collective.releaser :
the releaser_project template
project_release
project_deploy
project_eggs
project_diff
81
81

82. That’s it !
Thanks !
questions ?
82
82

83. http://tarekziade.wordpress.com
zc.buildout @ PyPI
collective.dist @ PyPI
collective.releaser @ PyPI
http://alterway.fr
83
83