The document summarizes security practices at Greenhouse, a company that provides recruiting optimization software. It discusses Greenhouse's use of a bug bounty program, third-party security audits, and processes to instill a culture of security. Greenhouse stores sensitive user data and aims to demonstrate its security to customers through these measures. The document provides examples of security issues found through the bug bounty program and audit and recommends security practices for Rails applications.
2. Software that optimizes your company’s entire recruiting process:
• Sourcing – spend your money & time effectively
• Interviewing – perform structured, purposeful interviews
• Decision making – support your hiring decision with data;
see what worked, what didn’t, and refine your process
What is Greenhouse?
4. • We store sensitive data (PII, salary negotiations, etc.)
• Customers need to trust us with that data
• “We’re secure” isn’t quite good enough. We have
to be able to demonstrate it.
Security is important
5. • Invite others to “hack” on the product
• Undergo third party audits
• Instill a culture of security
How do we do that?
7. • We chose HackerOne: https://hackerone.com/greenhouse
• Security researchers from all over try to find exploits
• Pay out a small bounty for verifiable exploits
• Hundreds of man-hours for very little payout
Start a Bug Bounty Program
8. • Cross-site issues (XSS / CSRF)
• Clickjacking (embed your site in an iframe elsewhere)
• Reflected File Download (JSONP vulnerability)
• Best practices: missing security headers, DNS
configuration not optimal, etc.
• 2 CVEs found: Solr, and Rails itself
What bug reports did we see?
9. The attacker was able to determine if a file exists outside of the
Rails root (but not retrieve the file).
How? Simply visit:
“Arbitrary File Disclosure” found in Rails core
http://yoursite.com/..%2F..%2F..%2Fbin/bash
This results in a special 404 response, indicating the file exists.
10. • Triage: prepare to be overwhelmed in the beginning
• Too many fake bug reports
Downsides to a Bug Bounty Program
11. • Find security holes
• Low cost, low barrier to entry
• Gain exposure to a wide array of attack vectors
• Show people you care about security
Upsides to a Bug Bounty Program
13. • We’re not security experts ourselves
• Customers need assurance that our product is secure
• Some companies won’t sign on to Greenhouse without it
Call in the experts
14. They come on-site and have complete access to our code and
test environment.
• Penetration testing (blackbox and whitebox)
• Code review
• Design review
iSEC Partners
17. • Use 1Password to store all your account passwords
• Don’t send API keys, etc. to each other over email in plaintext:
everyone needs a PGP key
• Enable 2FA on Github / Heroku / AWS
• Background checks for anyone with access to production
• Tech leads review all code
Processes we follow
18. A few things you can
be doing to secure your
Rails app…
20. If you use CanCan, put this in your base controller:
Ensure all controllers do authorization
check_authorization
Now if you don’t call authorize! in a controller action, an
AuthorizationNotPerformed error is raised.
Tip: Start with a “reporting” mode before flipping it live. Catch
this error and log it, then fix the offending controller actions.
21. • SymmetricEncryption gem (github: reidmorrison)
• We created an ActiveRecord keyword to indicate which
columns should be encrypted/decrypted.
Encrypt sensitive data in your database
class User < ActiveRecord::Base
encrypt_columns :api_key
end
user = User.new
user.api_key = ‘abc123’ # encrypted automatically
user.api_key # decrypted on the fly
22. • DOS attack: open a lot of connections, send partial
requests, but never complete them.
• Rails servers are susceptible to this attack, e.g. unicorn
• Solution: Put nginx in front of Rails, bump up
worker_connections quite a bit.
• On Heroku? Use a buildpack to run nginx.
Mitigate slowloris attack
*** TALK SLOW ***
My name is Mike O’Neil, I’m a tech lead at Greenhouse.
I’m going to talk a little bit about our approach to security, and a few of the things we do to make our Rails app more secure.
We do that in a few ways.
Sourcing – which is how you find candidates – we help you spend your money & time effectively on only the sources that are actually working, and we empower your entire company to get involved in finding great candidates, since recruiting is not just the job of the recruiter.
Interviewing – help you perform structured, purposeful interviews, where you entire team is completely prepared, and because of that you deliver an amazing candidate experience.
Decision making – we help you support your hiring decisions with actual data about the candidate, and we give you powerful reports about all your recruiting activity, so you can find ways to continually improve your process.
Greenhouse powers the careers pages of hundreds of companies, probably many that a lot you have applied to, or work at.
Here's a few things we do to improve our security, and to provide some visibility to our customers.
When they join the program, they agree not to reveal anything that they find to the public.
The idea is, they report something. You decide if it’s legitimate and you’ll fix it or not. If so, you fix it and then pay them a bounty. After that, the researcher is free to reveal the exploit if they want.
The bounty paid depends on the severity of the issue, and is at our discretion. A typical bounty is $100 for something “interesting”, and $1000 or more for something “severe”, which means they are able to access customer data.
They tend to be lower severity issues that are difficult to exploit or have minimal impact. The ones we could verify were usually worth fixing.
However 2 CVEs were found outside of our code: in major frameworks. 1 was Solr, 1 in the Rails core.
CVE stands for Common Vulnerabilities and Exposures system. A system for rating vulnerabilities, and disclosing them to the public after they are fixed so everyone can upgrade.
For Solr, it was in one of the XML engines Solr used to parse Word documents, and it enabled the attacker to exfiltrate data.
For Rails…
What could a hacker do with that? They could find what users are on the system, they could iterate through process ids in the proc filesystem, etc.
We reported it to the Rails team. The fix was part of ActionDispatch, in the code which serves static assets.
Triage:
When you launch, you’ll have a ton of reports come in. You will spend hours just to triage them. A lot of the reports are duplicates, or unclear how to reproduce, or just clearly false.
Some people will spam you with a bunch of common security issues, hoping something sticks and they get a bounty. But that’s against the rules. They need to prove that it’s a reproducible issue. These were usually easy to find and reject, but it still wastes your time.
About gaining exposure: e.g. Reflected File Download, actually a pretty newly discovered vulnerability, we had not heard of that before.
We need someone with expertise to look over our shoulder, and validate what we’ve built is secure.
At the end they write up their findings. They come back quarterly, where they confirm we fixed the things we think we did, and look at new features and parts of the system which need to be reviewed.
*** Also, these guys went back to their company and recommended considering us for their recruiting software.
----- Meeting Notes (4/14/15 11:25) -----
This means having security on the forefront of everyone's mind, especially on the tech team, but ideally the entire company.
And we don’t just think about the engineering team, we consider security an issue that the whole company needs to deal with.
Some of these are things that came out of our security audit, some came from the bug bounty program, some that we did on our own.
API keys, SMTP passwords. Things you need to be able to read the plaintext value of.
One issue is, you lose flexibility in querying the data, e.g. a LIKE or range query.
That’s me, if you want to get in touch.
Also, we’re hiring. So check out our careers page or come see me afterwards if you want to chat about that.