1. Automate Yo' Self!
SeaGL 2016
Seattle, yo
John SJ Anderson
@genehack
if anybody has any questions or i'm going too fast, please throw up a hand and ask -- or grab me on the hallway
track
3. @genehack
i go by genehack most places on line.
today i'm going to talk to you today about some tools and tricks i have for being productive while developing. but
before i do that, i need to explain a bit about why i needed it. my life is busy.
17. Don't
Make
Me
Think
none of this automation stuff should require me to think -- if i have to think about it, it's not really saving me any time
18. This is my "you made me think" face.
the whole point is not having to think.
19. Consistency is Good.
so, in those terms, one thing that's essential: make everything the same. have a standard directory layout under $HOME. you shouldn't have to think about
what system you're on, what shell, what project. things should just be the same -- or at least *correct* -- all the time
20. Idempotence is better!
Idempotence: the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result
beyond the initial application
21. App::MiseEnPlace
so, here's an example of idempotence, a utility i wrote to manage symlinks and directories in my home directory
22. App::MiseEnPlace
"everything in its place"
"mise en place" is a French phrase meaning "everythng in its place" -- it comes from cooking, and the principle that you should have all your ingredients
prepped and ready to go before you start cooking. i wanted something to make sure i always had my standard directory layout under $HOME as well as
setting up various symlinks
23. % cat ~/.mise
---
manage:
- doc
- etc
- private
- proj/*
- src/*
create:
directories:
- bin
- proj
- proj/go/src/github.com/genehack
- src
- var
links:
- Desktop: var/tmp
- Desktop: tmp
here's what the top level config for mise looks like -- it goes in .mise in your home directory. we have a list of
directories we want to manage (more on that in a minute) and a set of directories and symlinks to create. links are
source:target
24. % cat proj/emacs/.mise
---
create:
links:
- DIR: ~/.emacs.d
- bin/build-most-recent-emacs: BIN
- bin/e: BIN
- bin/ec: BIN
- bin/git-blame-from-line-num: BIN
- bin/map-test-lib: BIN
this is a per-project config file. mise has a couple special keywords DIR and BIN, that refer to the directory
containing the .mise file and ~/bin, respectively
the advantage of this is you don't need to have a huge gnarly $PATH with a bunch of project directories in it,
everything is just symlinked into ~/bin
25. % mise
[LINK] created ~/proj/emacs -> ~/.emacs.d
[LINK] created ~/proj/emacs/bin/build-most-recent-emacs -> ~/bin/build-most-recent-emacs
[LINK] created ~/proj/emacs/bin/e -> ~/bin/e
[LINK] created ~/proj/emacs/bin/ec -> ~/bin/ec
[LINK] created ~/proj/emacs/bin/git-blame-from-line-num -> ~/bin/git-blame-from-line-num
[LINK] created ~/proj/emacs/bin/map-test-lib -> ~/bin/map-test-lib
% mise
% rm ~/bin/e
% mise
[LINK] created ~/proj/emacs/bin/e -> ~/bin/e
when we run mise for the first time, you can see it creates all those links. if we run it again, it does NOTHING.
Idempotency for the win! If you remove a link and run it again, it creates _just_ that link
26. App::MiseEnPlace
get it from your favorite CPAN mirror
available on CPAN, minimal dependencies, works on any perl from this decade.
27. smartcd
ok, so that handles getting a consistent directory structure, and linking project binaries. what if you have other per-project stuff that you want to set up?
environment variables or other stuff?
28. smartcd
Automatically run code when entering or
leaving directories or sub-directories
enter smartcd, which hooks the 'cd' command and then runs scripts when you enter or leave a directory (or even a subdirectory)
29. % smartcd show enter
/Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists
-------------------------------------------------------------------------
########################################################################
# smartcd enter - /Users/genehack/fake-node-proj
#
# This is a smartcd script. Commands you type will be run when you
# enter this directory. The string __PATH__ will be replaced with
# the current path. Some examples are editing your $PATH or creating
# a temporary alias:
#
# autostash PATH=__PATH__/bin:$PATH
# autostash alias restart="service stop; sleep 1; service start"
#
# See http://smartcd.org for more ideas about what can be put here
########################################################################
autostash NODE=4.2.3
autostash PATH=$PATH:__PATH__/node_modules/.bin/
nvm use $NODE
-------------------------------------------------------------------------
smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script
for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you
leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use
30. % smartcd show enter
/Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists
-------------------------------------------------------------------------
########################################################################
# smartcd enter - /Users/genehack/fake-node-proj
#
# This is a smartcd script. Commands you type will be run when you
# enter this directory. The string __PATH__ will be replaced with
# the current path. Some examples are editing your $PATH or creating
# a temporary alias:
#
# autostash PATH=__PATH__/bin:$PATH
# autostash alias restart="service stop; sleep 1; service start"
#
# See http://smartcd.org for more ideas about what can be put here
########################################################################
autostash NODE=4.2.3
autostash PATH=$PATH:__PATH__/node_modules/.bin/
nvm use $NODE
-------------------------------------------------------------------------
smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script
for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you
leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use
31. % smartcd show enter
/Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists
-------------------------------------------------------------------------
########################################################################
# smartcd enter - /Users/genehack/fake-node-proj
#
# This is a smartcd script. Commands you type will be run when you
# enter this directory. The string __PATH__ will be replaced with
# the current path. Some examples are editing your $PATH or creating
# a temporary alias:
#
# autostash PATH=__PATH__/bin:$PATH
# autostash alias restart="service stop; sleep 1; service start"
#
# See http://smartcd.org for more ideas about what can be put here
########################################################################
autostash NODE=4.2.3
autostash PATH=$PATH:__PATH__/node_modules/.bin/
nvm use $NODE
-------------------------------------------------------------------------
smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script
for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you
leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use
32. % cd ~/fake-node-proc
Now using node v4.2.3 (npm v2.14.7)
% echo xx$NODE
xx4.2.3
% cd ..
% echo xx$NODE
xx
here's what it looks like when you cd into that directory. and you can see the env var is set. if we change back out of
the directory, the environment variable is unset.
33. smartcd
https://github.com/cxreg/smartcd
pretty cool, available on github. works with bash and zsh. really easy to install.
i can pay this the highest possible compliment: i haven't forked it or needed to patch it in any way, i just use a checkout from the upstream repo
42. perlbrew
plenv
rubyenv
nvm
App::GitGitr
build-most-
recent-emacs
here are some of the ones i've used or use now. perlbrew and plenv let you have multiple perls. nvm does the same thing for node. similar tools exist for
python, ruby, etc.
then there's GitGitr, which I wrote while maintaining Git wrapper library. I needed to be able to quickly install arbitrary Git versions while responding to bug
reports -- so I wrote a little tool that does that.
Similarly, I'm an Emacs user. If a new version is released, I want to upgrade, across all my systems - so I scripted that.
43. Consistency Corollary:
Don't trust system binaries
Back at the beginning, I said "consistency is good". Now, if you're developing on MacOS and deploying to Linux (or dev-ing on Ubuntu and deploying to
Debian), you're probably not going to have the same version of tool from the OS (and if you do now, it's not going to last). Even if the versions are the
same, there may have been vendor patches applied.
44. Automate building your
critical tools.
No, if you really want to be consistent, the best approach is to build the tools that are most critical for your project. ("Build" in this case may just mean
automating the install; it doesn't have to mean "build from source".) Note: only do this for the *important* stuff. (include examples)
45. The Silver Searcher
Speaking of tools, here's a tool that has literally improved my entire development life -- the silver searcher. Anybody
here using this?
46. grep?
ok, so, grep -- everybody knows grep, right? let's you search for text inside files, which is something you do a lot
while developing code.
47. powerful, speedy,
indiscriminate
so, grep is super powerful in terms of what you can search for, and it's pretty quick, but it's not very selective. you
can list all the files you want to search, or search whole dir trees, but you quickly realize this sucks, because of things
like .git directories. anybody ever do a recursive grep on a big git checkout? yeah.
49. powerful, selective,
slow
it's just as powerful as grep in terms of what you can search for, but it's recursive by default (which is what you want)
and it's smart about ignoring .git and SVN stuff. the problem is, it's pretty slow, particularly to start up (because
Perl)
56. Why bother?
no, but seriously. it's a little bit of a pain to develop the discipline but once you get used to having _everything_
under revision control, it's nice. you don't have to worry about experimenting with anything, backing stuff up, or
dealing with cross-machine variation in your environments
57. say
automation
again
originally i had series of kludgy shell scripts to manage repo updates and checkouts
super ugly and not worth sharing
and then, inspiration: Ingy döt Net talking about App::AYCABTU @ PPW2010
58. Ingy döt Net
this is ingy - he's a crazy awesome open source hacker guy who has done a whole bunch of stuff. probably best
known for being one of the inventors of YAML
59. Things I wanted to steal
the thing ingy had developed had a bunch of stuff i wanted to steal:
The basic idea
The interface
Info about repositories in config file
Flexible ways of selecting repos for operations – by #, by name, by tag
60. Things I wanted to add
Support for more than just Git
Locate repositories in arbitrary locations
Easily add and remove repositories
Ability to easily extend with more subcommands
Most importantly: better name!
62. GitGot
Thus was born GitGot
http://search.cpan.org/dist/App-GitGot/
Installs a ‘got’ command
Uses Moo and App::Cmd under the covers
Add new subcommands by writing a single class!
64. got add
you tell got about repos using the 'add' command, from inside the git repo
65. % got add
Name: foo
URL:
Path: /Users/genehack/foo
Tags: bar
it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)
66. got add -D
or you can just add the '-D' switch and it'll automatically use the defaults
67. got clone <REPO URL>
you can also clone a remote repo, which will check it out into the working directory and add it to got, prompting
you for details
68. % got clone git@github.com:genehack/app-gitgot.git
Name: [app-gitgot]:
Path: [/Users/genehack/app-gitgot]:
Tags:
Cloning into '/Users/genehack/app-gitgot'...
it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)
69. got clone -D <REPO URL>
got clone also respects the '-D' switch
70. got fork <GITHUB URL>
finally, you can give got a github url, and it will fork that project under your github id, then check it out into the
current directory and add it to got
71. ~/.gitgot
all the info about the repos managed by got lives in this .gitgot file in your home directory
72. - name: App-Amylase
path: /Users/genehack/proj/App-Amylase
repo: git@github.com:genehack/App-Amylase.git
type: git
- name: Git-Wrapper
path: /Users/genehack/proj/Git-Wrapper
repo: git@github.com:genehack/Git-Wrapper.git
tags: git
type: git
- name: HiD
path: /Users/genehack/proj/HiD
repo: git@github.com:genehack/HiD.git
type: git
- name: Perl-Build
path: /opt/plenv/plugins/perl-build
repo: git://github.com/tokuhirom/Perl-Build.git
type: git
it's just a simple YAML formatted line, totally hand-editable (although you shouldn't _need_ to do that, you can)
note that repos can be located anywhere on the disk, don't have to under a common dir or in your home or
whatever. anywhere you can write to is fair game
73. But now what?
ok, so you've added all your git repositories to got. what now?
77. got ls -q
if you don't want to see the upstream repo info, you can use the '-q' or '--quiet' switch
78. 1) App-Amylase
2) Git-Wrapper
3) HiD
4) Perl-Build
5) Perl-Critic
6) STAMPS
7) advanced-moose-class
8) app-gitgitr
9) app-gitgot
and that'll get you this output.
note the numbers - those will give you a way to select repos for other commands
79. got ls [repos]
easiest way to demo that is with an example. you can restrict the listing
88. got ls 5-8 HiD 21 -t git
finally, you can combine all of these selection methods together. here we're asking for repos 5 thru 8, the repo
named HiD, repo 21, and all repos tagged with git
89. 2) Git-Wrapper
3) HiD
5) Perl-Critic
6) STAMPS
7) advanced-moose-class
8) app-gitgitr
9) app-gitgot
21) etc
and this is what we get
note that most commands operate on all repos, and any that do, you can use these techniques to restrict the
command to a subset.
93. got st
or if you're into the whole brevity thing...
94. 1) App-Amylase : OK
2) Git-Wrapper : OK
3) HiD : OK
4) Perl-Build : OK
5) Perl-Critic : OK
6) STAMPS : OK
7) advanced-moose-class : OK
8) app-gitgitr : OK
9) app-gitgot : OK
that'll get you output like this.
again, note the use of color to give quick visual cues
95. 1) App-Amylase : OK
2) Git-Wrapper : OK
3) HiD : Dirty
4) Perl-Build : OK
5) Perl-Critic : OK
6) STAMPS : OK
7) advanced-moose-class : OK
8) app-gitgitr : OK
9) app-gitgot : OK
Dirty
got status will let you know if a repo has uncommitted changes.
super handy if, for example, working on one machine and are going to move to another one and want to see what
hasn't been committed.
96. 1) App-Amylase : OK
2) Git-Wrapper : OK
3) HiD : OK
4) Perl-Build : OK
5) Perl-Critic : OK
6) STAMPS : OK Ahead by 1
7) advanced-moose-class : OK
8) app-gitgitr : OK
9) app-gitgot : OK
1) App-Amylase : OK
2) Git-Wrapper : OK
3) HiD : OK
4) Perl-Build : Dirty
5) Perl-Critic : OK
6) STAMPS : OK
7) advanced-moose-class : OK
8) app-gitgitr : OK
9) app-gitgot : OK
Dirty
1) App-Amylase : OK
2) Git-Wrapper : OK
3) HiD : OK
4) Perl-Build : OK
5) Perl-Critic : OK
6) STAMPS : OK Ahead by 1
7) advanced-moose-class : OK
8) app-gitgitr : OK
9) app-gitgot : OK
Ahead by 1
it'll also tell you if you have local commits that haven't been pushed to the remote yet
97. got st -q
finally, you can use the '-q' switch to hide the "uninteresting" stuff
98. got st -q
3) HiD : Dirty
6) STAMPS : OK Ahead by 1
Dirty
Ahead by 1
which in this case is all the repos that don't have changes and are up to date
100. got up
which abbreviates to 'up'
and yeah, i should have called it pull but i'm a dummy and we're stuck with it now.
101. 1) App-Amylase : Up to date
2) Git-Wrapper : Up to date
3) HiD : Up to date
4) Perl-Build : Updated
Updating 7f25f89..72587c8
Fast-forward
lib/Perl/Build.pm | 14 +++++++++++++-
script/perl-build | 14 +++++++++++++-
2 files changed, 26 insertions(+), 2 deletions(-)
5) Perl-Critic : Up to date
Updated
it'll do pretty much what you expect (and it supports '-q' too)
102. got update_status
finally, there's a command that combines both those, because it's something i do pretty frequent -- update
everything, then look at the status of everything
114. got chdir
finally, there are a number of commands that help you jump to the directory of a project. got chdir
115. got cd
also spelled 'got cd', will change your current working directory to the given repo (note that this is one of the few
got commands that requires a single repo)
116. got tmux
we also have tmux integration -- 'got tmux' will open a new tmux window with the working directory in your repo.
this _can_ be done with multiple repos. better, the tux window is persistent; as long as it's open 'got tmux' will just
select the already open window, not open a new one
117. got tmux -s
you can also spawn whole new tmux sessions if you prefer those to windows -- and again, those will be re-used as
long as they're around
125. package App::GitGot::Command::chdir;
# ABSTRACT: open a subshell in a selected project
use 5.014;
use App::GitGot -command;
use Moo;
extends 'App::GitGot::Command';
use namespace::autoclean;
sub command_names { qw/ chdir cd / }
sub _execute {
my( $self, $opt, $args ) = @_;
unless ( $self->active_repos and $self->active_repos == 1 ) {
say STDERR 'ERROR: You need to select a single repo';
exit(1);
}
my( $repo ) = $self->active_repos;
chdir $repo->path
or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1);
exec $ENV{SHELL};
}
1;
128. sub _execute {
my( $self, $opt, $args ) = @_;
unless ( $self->active_repos and $self->active_repos == 1 ) {
say STDERR 'ERROR: You need to select a single repo';
exit(1);
}
my( $repo ) = $self->active_repos;
chdir $repo->path
or say STDERR "ERROR: Failed to chdir to repo ($!)"
and exit(1);
exec $ENV{SHELL};
}
129. package App::GitGot::Command::chdir;
# ABSTRACT: open a subshell in a selected project
use 5.014;
use App::GitGot -command;
use Moo;
extends 'App::GitGot::Command';
use namespace::autoclean;
sub command_names { qw/ chdir cd / }
sub _execute {
my( $self, $opt, $args ) = @_;
unless ( $self->active_repos and $self->active_repos == 1 ) {
say STDERR 'ERROR: You need to select a single repo';
exit(1);
}
my( $repo ) = $self->active_repos;
chdir $repo->path
or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1);
exec $ENV{SHELL};
}
1;
130. Suggestions welcome!
areas for improvement:
support for other VCSen
better config management tools
any other crazy workflow improvement you can think of!
135. Thanks
SeaGL organizers
Ingy döt Net
Yanick Champoux
Michael Greb
Rolando Pereira
Chris Prather
photo credits:
all photos by speaker except
Ingy döt Net photo - https://www.flickr.com/photos/bulknews/389986053/
and pug - https://upload.wikimedia.org/wikipedia/commons/d/d7/Sad-pug.jpg
and automate yo'self - somewhere on the net
136. thanks to the company i work for for allowing me to (occasionally) mess with this stuff and sending me out to give these talks. holler at me on the hallway track if you're
looking for some Open Source consulting or staff augmentation
137. Questions?
as i said, i *love* to talk to people about this productivity type stuff, so grab me on the hallway track. i'm friendly.