My attempt to create wundercharts.com without actually learning anything new.

1. No Hugging, No Learning
Olaf Alders
Toronto Perl Mongers
Aug 25, 2016
@olafalders (Twitter)
oalders (GitHub)
OALDERS (PAUSE)
https://github.com/oalders/no-hugging-no-learning

2. The Problem
Building an app that can track and chart (almost)
anything that is available via 3rd party APIs
No screen scraping
No ToS violations
Doing this in my limited spare time

3. The Solution
Build an app based solely on what I already know:
"No hugging, no learning"
Of the three programmer virtues (laziness,
impatience and hubris), I would let laziness be my
guiding light
I tried to self-impose a real lack of intellectual
curiosity

4. The Evolution

5. The App
Before
Some command line scripts as a proof of concept
This became a Dancer (1) app run via Plack

6. Pros
Dancer apps are easy to get up and running
Cons
I didn't really love the Dancer (1) syntax
Dancer2 was still quite new and the plugin support
wasn't there yet

7. After
I moved to a Mojolicous Lite app run via Plack
I then transitioned to a full Mojo app
Run via morbo in development
Run via hypnotoad in production

8. morbo
By default watches your lesystem for changes
and then reloads your app

9. hypnotoad
Zero downtime restarts
You don't need to provide your own init script
If you don't care about zero downtime or just want
the prefork server, you have that option
perl script/myapp prefork
same con g args as morbo/daemon

10. Did I learn?
I partly became familiar with Mojo via my day job,
but I did have to do a fair bit of learning on my
own.
I was not familiar with Hypnotoad , but luckily that
was a pretty easy transition. The docs are really
good.

11. Authentication

12. Before
Mozilla's Persona

13. Pros
Pretty easy to con gure
All anyone really needs is an email address
Cons
Issues getting it to work of ine
No widespread adoption
Felt like the rst 80% was easy but the last 20%
was eating into my time

14. After
OAuth via any of the available integrations
I needed this anyway in order to get user data, so
it keeps things simpler to have Persona out of the
mix
With Mozilla no longer 100% behind Persona, it
becomes a less attractive solution
most non-technical users won't be familiar with
it at this point anyway

15. SSL Everywhere
Looked brie y into Let's Encrypt via Ansible
Ended up spending USD 10 for a RapidSSL cert

16. Relational Database

17. Before
Application data stored in MySQL database

18. Pros
I'm very familiar with MySQL
I prefer many MySQL tools, like the mysql command
line interface and phpMyAdmin
MySQL replication is dead easy to con gure
Cons
"Foreign keys that point to the renamed table are
not automatically updated. In such cases, you must
drop and re-create the foreign keys in order for
them to function properly."

19. After
3 Postgres databases
Minion
Application data
Test suite xtures

20. Did I learn?
Since we use Postgres at $work , I didn't have to
learn a whole lot to make the switch.
The database does a litle bit more than the bare
minimum -- I'm not taking advantage of all that
Postgres has to offer
No jsonb columns just yet

21. Job Management

22. Before
cron scripts

23. After
Minion

24. Did I learn?
Had basically zero knowledge of Minion
implementation
But, there's not much you need to learn in order to
get up and running
Minimized a lot of really convoluted cron job logic
This probably saved me time in the long run

25. Minion How To
You don't need to have a full-blown Mojo app to
use Minion
Just create a bare bones app to get started
Use this with your Catalyst, Dancer, [insert
framework here] application
Let's look at how MetaCPAN uses Minion

26. Create an App
https://github.com/metacpan/metacpan-
api/blob/master/lib/MetaCPAN/Queue.pm
Sets up a tiny Mojo app
Adds Minion tasks in the startup() method
You can see how we trap warnings and log failed
jobs in the _gen_index_task_sub() method

27. Optionally Create Your Minion
Backend Elsewhere
We've abstracted it out to
https://github.com/metacpan/metacpan-
api/blob/master/lib/MetaCPAN/Queue/Helper.pm
We do this so that we can use an SQLite database
when running under the test harness and a
Postgres database in all other cases
Using SQLite for testing makes our Travis
con guration much easier
This will also allow us more easily to run tests in
parallel

28. Start Up Your Queue
In development, you can create a 4-5 line script to
start up your queue via morbo
https://github.com/metacpan/metacpan-
api/blob/master/bin/queue.pl
In production we use Daemon::Control to manage
starting and stopping a daemon
https://github.com/metacpan/metacpan-
puppet/blob/master/modules/minion_queue/te
mplates/init.pl.erb

29. Add Tasks
A task is essentially an anonymous sub which you
can later refer to by name
A job is an instance of a task
$minion->add_task( add_things => sub {
# $job is a Minion::Job object
my ($job, $first, $second) = @_;
$job->finish($first + $second);
});

30. More Complete Example
$minion->add_task( add_things => sub {
my ($job, $first, $second) = @_;
$job->finish({
message => 'Great!',
total => $first + $second,
});
});
my $id = $minion->enqueue(add_things => [1, 1]);
my $result = $minion->job($id)->info->{result};
$result is now
{ message => 'Great!', total => 2 }

31. Storing Your Job Result
$job->finish;
$job->finish('Fantastic!');
$job->finish({ foo => 'bar' });
Later, you can nd this arbitrary data in $job->result

32. Populate Your Queue
enqueue() means "add job to queue"
$app->minion->enqueue( my_task_name => ['foo', 'bar'] );
$self->minion->enqueue(
track_service_resource =>
[{ service_resource_id => $service_resource->id }] =>
{ priority => 1 },
);
priority ranges from 0-X, where X is a positive
integer
jobs with priority 10 will run before jobs with
priority 9 etc

33. enqueue() options
attempts => 10 (# of times to attempt job)
delay => 3600 (delay this job for 3600 seconds)
parents => [$id1, $id2] (One or more existing jobs
this job depends on, and that need to have
transitioned to the state finished before it can be
processed)
priority => 3 (see previous slide)
queue => 'dexter' (some arbitrary name, defaults
to default )

34. Minion::Job
https://metacpan.org/pod/Minion::Job
You can set up events to be red on failed and
finished states
Get information via $job->info
$job->is_finished
$job->remove
Much, much more

35. Inspect Your Queue
On the MetaCPAN Vagrant box:
$ vagrant ssh
$ cd /home/vagrant/metacpan-api/
$ ./bin/run bin/queue.pl minion job -s
{
"active_jobs" => 0,
"active_workers" => 0,
"delayed_jobs" => 0,
"failed_jobs" => 0,
"finished_jobs" => 0,
"inactive_jobs" => 0,
"inactive_workers" => 1
}

36. Test Your Queue
https://github.com/metacpan/metacpan-
api/blob/master/t/queue.t
use MetaCPAN::Queue;
use Test::More;
use Test::RequiresInternet ( 'cpan.metacpan.org' => 443 );
my $app = MetaCPAN::Queue->new;
my $release = 'https://cpan.metacpan.org/authors/id/O/OA/OALDERS
$app->minion->enqueue(
index_release => [ '--latest', $release ]
);
$app->minion->perform_jobs;
done_testing();

37. Advanced: Behaviour on Events
https://github.com/jberger/Minion-
Noti er/blob/master/lib/Minion/Noti er.pm#L39-
L56

38. sub setup_worker {
my $self = shift;
my $dequeue = sub {
my ($worker, $job) = @_;
my $id = $job->id;
$self->transport->send($id, 'dequeue');
$job->on(finished => sub { $self->transport->send($id,
$job->on(failed => sub { $self->transport->send($id,
};
$self->minion->on(worker => sub {
my ($minion, $worker) = @_;
$worker->on(dequeue => $dequeue);
});
return $self
}

39. Code Context
The preceding code is called like this:
https://github.com/jberger/Minion-
Noti er/blob/master/lib/Mojolicious/Plugin/Minion/
Noti er.pm#L37

40. Pros
We can replace a failure-prone, forking, long
running process with a series of jobs
If a job fails in an unexpected way, it doesn't
prevent all the other jobs from running
We can check the status of all jobs at any point to
gauge how things are progressing
Jobs which fail can be retried at intervals
We can start an arbitrary number of worker apps
Using Postgres replication MetaCPAN can now
start X workers on 3 different boxes, which gives
us greater scaling when needed

41. Cons
Minion doesn't come with a handy daemon
handler like hypnotoad , but you can set this stuff all
up yourself pretty easily
There aren't yet a lot of tools to visualize what's
going on with the queue, but again this can be
done pretty easily if you need it.

42. Database Migrations

43. Before
Database schema managed by Database::Migrator

44. Pros
Database::Migrator is what we use at $work
It's pretty simple
In addition to SQL, you can also run Perl scripts as
part of a migration
Cons
Database::Migrator needs to be subclassed, which
means you have to provide a fair bit of
functionality just to get up and running

45. After
Application and xture databases managed via
App::Sqitch
Minion has its own schema deployment logic

46. Pros
sqitch handles rollbacks as well as deployment
Optionally can use custom assertions to ensure
that deployments were successful
Cons
There's a lot more going on here, so you probably
have to do a bit of reading before just jumping in
I had to use the CLI rather than the modules
because of some weirdness between Moo and
Moose when I tried to use the modules directly

47. Did I learn?
Yes, I did have to learn sqitch from scratch
The author (David Wheeler) was extremely helpful
I was able to get it working with my xtures

48. See Also
Mojo::Pg::Migrations
Mojo::mysql::Migrations
Mojo::SQLite::Migrations

49. Automation of Deployment

50. Before
The plan was to use Puppet

51. Pros
I was already familiar with Puppet
Cons
I was already familiar with Puppet
Even simple things seemed hard
Tired of dealing with hard to debug certi cate
issues
There may have been obvious and easy xes to
my problems, but I couldn't easily nd them
I'm not saying that Puppet isn't a wonderful tool,
but it felt like a bigger hammer than I needed

52. After
Ansible

53. Pros
Excellent integration with Vagrant
No need for bootstrapping on the target
deployment machines
It's easily installed via Homebrew on OS X
Has good handling of git submodule
User contributed plugins are trivial to install

54. Cons
I actually don't have a lot of complaints
Moving from v1 to v2 addressed the few issues
that I ran into

55. Ansible in Vagrant le
config.vm.provision "ansible" do |ansible|
ansible.verbose = "v"
ansible.playbook = "ansible/site.yml"
ansible.sudo = true
ansible.groups = {
"development" => ["default"],
"webservers" => ["default"],
}
end

56. Installing Packages
---
- gather_facts: no
hosts: webservers
tasks:
- apt: 'pkg={{item}} state=installed'
name: Install misc apt packages
with_items:
- curl
- ntp
- tree
- unzip
- apt: update_cache=yes cache_valid_time=3600
name: Run the equivalent of "apt-get update"
- apt: upgrade=dist autoremove=true
name: Update all packages to the latest version

57. Did I learn?
Yes, I had to learn all about Ansible , since I had never
used it before. In the meantime $work has also
switched from puppet to ansible , so it turned out to
be knowledge that I can apply there as well.
Keeping in mind how many hours I've spent battling
Puppet on past projects, I think learning about
Ansible was actually the better, faster choice.

58. Predictions Which Did Not Change

59. JS-Driven Charts
Highcharts turned out to be an excellent choice
Excellent docs
Responsive support
I'm also quite happy with the product itself

60. Twitter Bootstrap
Still a solid choice for a CSS framework

61. Travis CI
Private repository support is not within my budget
(starts at USD 129/month)
Open Source option works excellently for
https://github.com/oalders/wundercharts-plugin

62. Carton for Dependency Pinning
This suits my needs ne at this point
https://metacpan.org/pod/Carmel is still listed as
experimental

63. Vagrant
Initially I just ran the app inside OS X directly
I realized pretty quickly that I wanted to have a
sandbox
I've even got a Vagrantfile for the plugin system
https://github.com/oalders/wundercharts-
plugin/blob/master/Vagrant le
So, anyone who wants to contribute can have a
fully operational, sandboxed system with all of
the modules installed within a matter of
minutes

64. Code::TidyAll
I use this to tidy as much code as possible
It's enforced via a Git pre-commit hook so that I
don't have to think about it

65. LWP::ConsoleLogger
I use this a lot when debugging interactions with
3rd party APIs
This only really works when you can get at the UA
which a module is using
I wish more module authors would expose their
user agents
The Mojo equivalent is setting
MOJO_USERAGENT_DEBUG=1

66. DBIx::Class
Use foreign keys and the schema dumper
automatically sets up relationships for you
Use this trick to enable perltidy to tidy your
schema les:
https://metacpan.org/pod/release/ILMARI/DBIx-
Class-Schema-Loader-
0.07045/lib/DBIx/Class/Schema/Loader/Base.pm
# lter_generated_code

67. Mojolicious::Plugin::AssetPack
Compress and convert css, less, sass, javascript
and coffeescript les
In development mode you get the individual les
in your template (for easier debugging)

68. Strict Constructors
MooseX::StrictConstructor
MooX::StrictConstructor
MetaCPAN::Moose

69. etc
nginx
Ubuntu 14.04.5 LTS
Perl v5.18.2

70. Things I Hadn't Planned on Using

71. Wercker
I didn't know this existed
I'm not good about running my test suite after
every change, so I realized I had to automate this
I didn't want to run my own Jenkins/TeamCity/etc
service
I've seen how much con guration can go into
these things and I didn't have the time to mess
with it
Free, concurrent builds for private repositories

72. Wercker (cont'd)
I didn't want to give it access to all of my private
repositories
I created a BitBucket account (free private
repositories)
Added a new remote to my repository
"bitbucket"
Every time I push to the "bitbucket" remote the
tests are run and I get an email about success or
failure
Protip: if weird errors happen, sometimes you
have to clear your Wercker cache

73. wrapbootstrap.com
I hadn't even thought about the UI when I started
this project
I bought the "Inspina" theme for USD 18
This gave me solid options for both logged in
and logged out UI
Was trivial to implement and immediately made
my terrible design much better
Since I was already using Bootstrap there
weren't a lot of changes to be made

74. Date Range Picker
http://www.daterangepicker.com
Easy to use and makes it easy to use sensible
defaults when selecting dates or ranges of dates

75. Open Source That Came Out of
This
Mojolicious::Plugin::Web::Auth::Site::Spotify
Mojolicious::Plugin::Web::Auth::Site::Runkeeper
WebService::HealthGraph
Code::TidyAll::Plugin::YAML
WWW::Spotify (co-maint)
Patch to WebService::Instagram
I open sourced all of my plugins

76. WunderCharts Plugins
https://github.com/oalders/wundercharts-plugin
100% open source
I'm in a position to integrate user contributed
patches without open sourcing the entire code
base

77. For example, if we want to integrate FitBit
Need to create
Mojolicious::Plugin::Web::Auth::Site::FitBit
Need to send a pull request for
WunderCharts::Plugin::FitBit

78. Third Party Integrations
I didn't have a rm idea of how many I wanted
Got a mixture of OAuth, OAuth2 and no auth

79. OAuth
Twitter
OAuth 2
Facebook
Instagram
Spotify
GitHub
No authentication required
Hacker News

80. Don't Count Your Chickens
Instagram apps (unlike the other sites) have a
formal review process
I didn't realize until after my integration was
nished that my use case may not qualify
I did get accepted at the end of the day, but it's a
bit of a coin toss
You need to show them the integration via a
screen cast before you get approval, so it's
possible to do the heavy lifting and still be denied

81. Success?
Among other things, I ended up having to learn (or
learn more about) the following technologies:
Ansible
BitBucket
Date Range Picker
hypnotoad
Minion
Mojo
Mojolicious::Plugin::AssetPack

82. Persona
sqitch
Wercker
[also various 3rd party APIs for web services]

83. Thanks!
To Joel Berger for proofreading the Mojo bits of
an earlier version of this talk
All remaining (or since introduced) errors are my
own

84. The "Finished" Product
https://wundercharts.com