Slides for the April 2017 pfSense Hangout video

1. Let’s Encrypt
ACME Package
April 2017 Hangout
Jim Pingle

2. About this Hangout
● Project News
● Let’s Encrypt Basics
● How it Works
● Why use Let’s Encrypt?
● Limitations
● Challenge Validation
Methods
– HTTP-01 / TLS-SNI-01
– DNS-01
● ACME Package
– General Info
– Account Keys
– Creating Certificates
– Validation Methods
● DNS Update Server/Domain
Setup
● Using Let’s Encrypt Certificates
– For the firewall GUI
– With the HAProxy Package
– In other places on the firewall
– With HA / XMLRPC

3. Project News
●
2.3.4-RELEASE coming, in testing now, probably next week
– Security/Errata fixes, a couple new features.
– Chrome 58 certificate fix
●
Translations are still progressing, more added to 2.4 that crossed the 75% barrier. Now includes:
– Chinese (Simplified/China and Taiwan), Norwegian Bokmål, Spanish, Russian, and German
– Free pfSense Gold or SG-1000 for assisting with translations, depending on percentage contributed. See
https://www.netgate.com/blog/bite-the-wax-tadpole.html for details
● Suricata updated to 3.2.1 with Hyperscan support
● DIN Rail mount for SG-1000
– https://store.netgate.com/SG-1000-DIN.aspx
●
Building a Behemoth Router – Working towards 40 Gbit/s IPsec
– https://www.netgate.com/blog/building-a-behemoth-router.html
– Not an April Fools joke!
– Teasing pfSense 3.0 IPsec results, 32.4Gbit/s IPsec!
●
https://twitter.com/NetgateUSA/status/853694461456646144
● Teasing a new platform, R-1, Armada 38x
– https://twitter.com/gonzopancho/status/855173807312654337
●
Training in Paris, London, Frankfurt and Saint-Petersburg
– https://www.netgate.com/training/
●
New AWS pricing, including a t2.nano instance for $0.01/hr
– https://aws.amazon.com/marketplace/pp/B00G6P8CVW
●
Azure includes a free tier (A1/Standard single core has no software cost, infrastructure costs still apply)
– https://azuremarketplace.microsoft.com/en-us/marketplace/apps/netgate.netgate-pfsense-appliance?tab=PlansAndPrice

4. Let’s Encrypt Basics
● Let’s Encrypt is an open, free, and automated Certificate
Authority from the Internet Security Research Group
● The goal of Let's Encrypt is to encrypt the web by removing
the cost barrier and some of the technical barriers that
discourage server administrators and organizations from
obtaining certificates for use on Internet servers, primarily
web servers
● Most browsers trust certificates from Let's Encrypt
● These certificates can be used for web servers (TLS/HTTPS),
SMTP servers, IMAP/POP3 servers, and other similar roles
which utilize the same type of certificates.

5. Let’s Encrypt Basics
● Let’s Encrypt certificates are domain validated, and this validation ensures that the
system requesting the certificate has authority over the server in question
– Validated by proving ownership of the domain's DNS records or hosting a file on a web server for
the domain
● Let’s Encrypt certificates are valid for 90 days
– Must be renewed periodically, every 60 days is the recommended time frame
– Automation helps significantly
● Updates are performed using the ACME protocol
– Automatic Certificate Management Environment
● One certificate can cover many domains, subdomains, or hostnames
– Up to 100 per certificate
● Two signing structures available:
– Staging / Testing CA – Use this when testing a client or when getting familiar with the package
●
Not trusted by browsers, but has no rate limits
– Production CA – Use for normal purposes
● Trusted by browsers, but has rate limits

6. How it Works
● A client, such as our ACME package or certbot, generates a public/private key pair
and registers this key pair with Let’s Encrypt
● The client attempts to prove ownership of a domain using one of the available
validation methods
● Let’s Encrypt provides a set of challenges and a nonce that must be signed by the
client’s private key to prove ownership of the key
– Nonce is just a fancy word for a single-use randomized number used for various cryto tasks
● The client sets up the appropriate resources to respond (e.g. DNS TXT record
updates, or directories/files)
● Let’s Encrypt verifies the challenges have been properly completed
● If everything checks out, the client is authorized to perform certificate operations for
the hostname(s) in question
● With this authorization, the client can issue a new certificate for that domain, renew
an existing certificate, or revoke a certificate

7. Why use Let’s Encrypt?
● That sounds difficult, why would I want to do that?
– It is free, easily automated, and useful
– After setup, it is simple to manage
● Using HTTPS is more secure than HTTP, of course
●
Browsers are complaining more about self-signed certificates, making it increasingly difficult (or
annoying) to use them, and HTTP warnings are even more prominent
– Chrome’s click-through warning for each session, plus autocomplete gets disabled
– Firefox displays warnings inside form fields for sites that it does not consider secure
– Chrome 58 / Firefox 48 require stricter formatting (ignore CN, use SAN) which may require a new self-signed cert
to work around
●
If end users are conditioned to allow self-signed certificates and to click through warnings, they would
not easily recognize if their connection was made to the correct server and not to an attacker
● Cheap certificates can be obtained from other sources but you’d still have to prove domain ownership
in various ways to issue or renew, and each firewall or service could need its own certificate or SAN,
which adds to costs
●
Wildcard certificates are expensive and do not work with subdomains
● You could self-sign a CA, but that must be imported into every client browser/OS/device/etc – a lot of
management overhead

8. Limitations
● No support for Organization Validation (OV) or Extended Validation (EV)
● No wildcard certificates
● Will not allow or validate bare IP addresses or short hostnames, FQDN only
● Certificates are not usable for e-mail content encryption or code signing
– They can be used for TLS mail access and transport (SMTPS/STARTTLS, IMAPS, POP3S), but not for message
body encryption (S/MIME)
● Rate limits
– 20 new certificates per Registered Domain per week
● Renewals are exempt
● Rolling dates, last 7 days are checked
● Grow list of subdomains/hosts in domain over time
● Use SANs to work around limit in some cases
– 100 names (SANs) per certificate
● Between certs and SANs, 2,000 FQDNs could be covered by certificates in a given week
– 5 duplicate certificates (identical name list) per week
– 5 validation failures per account, per hostname, per hour
– More rare/unusual limits at https://letsencrypt.org/docs/rate-limits/
– Use staging server to test until your validation settings are correct, to avoid hitting limits
● Let’s Encrypt servers are not known beforehand, so it is not possible to craft limited or dynamic rules

9. Challenge Validation Methods
● Challenge Validation Methods enable the client to prove it controls a given domain by
hosting a file or changing a record that proves the relationship
● HTTP-01
– Let’s Encrypt requests a file from /.well-known/acme-validation on the target servers
● The targets are determined by resolving the hostname(s) in the certificate request
● The name of the file and its contents are part of the challenge negotiated between the client and Let’s Encrypt
– Each hostname in the certificate request will be challenged and they all must be able to respond to
the request
– Requires each SAN in the cert to have a valid, public DNS record
– Requires external inbound connectivity to allow Let’s Encrypt servers to request the file via HTTP
● Can be redirected so long as the requesting client can place files where Let’s Encrypt can reach them (e.g. sftp)
– HTTP validation must use port 80, no exceptions!
● Prevents secondary/user servers from being setup on arbitrary/high ports
● TLS-SNI-02
– Similar to HTTP, but uses TLS (e.g. HTTPS)
– Requires the server to support SNI to ensure that only the correct instance/vhost may validate for
shared environments
– TLS validation must use port 443, no exceptions!

10. Challenge Validation Methods
● DNS-01
– Uses a TXT record in DNS to handle the challenge response
● _acme-challenge.<hostname>
● TXT record can be safely removed after the challenge is validated
● The content of the record is part of the challenge
– Does not require direct inbound external connectivity
– Does not require a publicly resolvable A/AAAA record for the
hostname, only TXT
– Best for firewalls, private networks, lab use, etc.
– Automated updates are possible using RFC 2136 or a variety of
provider-dependent APIs
– Manual updates are supported
● Slower, cannot be automated, but can be used as a last resort if there is no
other option

11. ACME Package
● ACME is a package for pfSense that handles certificate
management through Let’s Encrypt
● It retains cert settings and makes the process straightforward
● Automates renewal process so it does not require ongoing
maintenance
– Can renew certs and restart services automatically when the time
comes.
● System > Package Manager, Available Packages tab: acme
● Once installed, navigate to Services > Acme Certificates

12. ACME Package
Account Keys
● Services > ACME Certificates, Account Keys Tab, + to add new key
● Account Keys are used as part of the authentication process when proving ownership
– NOT a certificate public/private key!
– Used to sign the whole CSR or revocation requests, separate from proving ownership (authentication, not
validation)
● Can’t make or sign a certificate without a registered key!
● Account Keys are associated with either the Let’s Encrypt staging system or production system
– Reminder: Use Staging/Testing for new configs, when getting used to the package with your chosen validation
method. No rate limits so safe to test until the config is dialed in
● The same key can be used for multiple certificates, so only one is necessary (or two, one for staging
testing and one for production)
● Fill in the Name and Description fields
● Click Create New Account Key
● Save, then edit entry again
– This step is optional but I prefer to see the key saved before registration, to be sure it has been stored properly
● Click Register ACME Account Key
– This registers the new key with the selected Let’s Encrypt system

13. ACME Package
Create Certificate
●
Create/Register account key first!
●
Services > ACME Certificates, Certificates Tab, + to add new Certificate
●
Fill in the Name of the cert
– This is also used as its name under System > Cert Manager
●
Status – Controls if this certificate is processed as part of the automatic renewal
process
●
ACME Account – The registered account key to use when attempting to issue or
renew this certificate
●
Key Size – The size of the private key for this certificate
– Usually 2048 is sufficient, larger keys are stronger but take more processing time/power
●
Domain SAN List – Names to include in the certificate
– Domains must be able to validate using one of the chosen validation methods (e.g. DNS,
HTTP)
– Must be valid, registered domains
– Will cover specific validation methods in a couple slides

14. ACME Package
Create Certificate
● DNS-Sleep – For DNS-based methods, this defines how long the update process
will sleep to allow DNS records to propagate to your DNS servers
– 2 minute default is usually OK, may take longer if it has to propagate to multiple secondary
servers, etc
● Actions List – Actions to take automatically when the certificate is renewed
– Shell Command: Full path to a shell command to run, including any required parameters
– Restart Service: The name of a service to restart (restart process must also rewrite
config/certificate! Not all do)
– PHP Command Script: PHP code to run directly
● Certificate Renewal After – Number of days to wait before attempting a renewal
– 60 is the best practice, don’t wait longer as it will not help, and could lead to the cert expiring
at 90 days without proper time to fix it.
● Save
● Click Issue/Renew
● Wait for the process to complete, inspect output for errors or further instructions

15. ACME Package
HTTP-01 Validation Methods
● Hostnames in certificate must resolve to this firewall or wherever the traffic is forwarded!
●
Files must be available to the Internet at /.well-known/acme-challenge/ on each host in certificate
● Firewall must allow access to whichever port/server is responding to the challenge
●
HTTP validation must use port 80 from public perspective
● Webroot Local Folder – Files in a directory on this firewall
– Requires a web server running on the firewall, exposed to the Internet (so don’t use the GUI server!)
– Can be used with HAProxy in some cases
●
Webroot FTP – Copies files to another server via SFTP
– Requires username/password to use for SFTP, plus server address, and full path to the challenge files
– User must have write permission for the target directory!
– Could be used with port forwarding/1:1 NAT/HAProxy/etc
● Standalone HTTP Server – Runs a simple server on the firewall itself
– Must use port 80 from the Internet
– Can be forwarded to another port for this server to avoid collisions
– Firewall must allow access, but server is only run during issue/renew process
– May eventually add/remove a temporary firewall rule, but that feature is not yet present
●
Standalone TLS (HTTPS) Server – Technically TLS-SNI-02, not HTTP-01
– Same as standalone HTTP, but must use port 443
– Likely to conflict with the GUI unless using 443 port forward + alternate listen port

16. ACME Package
DNS-01 Validation Methods
● Does not require inbound/public access, only DNS (good for private systems/labs)
●
DNS only requires special temporary TXT records (_acme-challenge.<hostname>), not public
A/AAAA records
● Higher upstream setup barrier, requires a supported provider/DNS structure
●
DNS-Manual – Manually create TXT records as needed
– Will print required record contents when the issue or renew is performed, then re-run after manually
adding/changing DNS record(s)
●
DNS-NSUpdate
– RFC 2136 style updates to your own personal BIND or similar server, or provider that supports RFC 2136 updates
– Requires update key for TXT record _acme-challenge.<hostname>
– Multiple key types/algorithms to match server setup
● DNS-PowerDNS
– Similar to NSUpdate style but for your own PowerDNS server
– Enter settings specific to your PowerDNS setup
● DNS-<providername>
– Numerous provider-specific DNS validation options
– Contact your provider to get the required keys/credentials, then fill them in

17. DNS Validation
Server/Domain Setup
● RFC 2136 dynamic zone or supported provider API
● Server setup for RFC2136 at
https://doc.pfsense.org/index.php/RFC2136_Dynamic_DNS
– A couple changes vs that doc:
– “update-policy” for the zone must allow TXT record updates
– Server must have defined keys for _acme-challenge.<hostname>
– A/AAAA records for the hostnames themselves are optional, but can be useful for Dynamic DNS
● For provider-specific setup, check type for requirements (e.g. API key + secret) and
obtain that information from the DNS provider
● CAA Record on domain to allow “letsencrypt.org” as issuer
– List of CA entities allowed to issue certificates for this domain
– Issuers must check for and respect CAA records September 8, 2017
● Except for their own hosted customers
– End users do not need it yet (no record = anyone can issue), but it is more secure
– example.org. CAA 0 issue "letsencrypt.org"

18. GUI Certificate
●
After generating the cert, select it under System > Advanced on the Admin Access
tab
●
Client browser must access the firewall by hostname!
– Add host override if there is no public DNS entry
– May still want a local host override so you hit the local/LAN address
– Can still access by IP address but the browser will give a cert error as usual
– I keep bookmarks for both hostname and IP address, in case I need to access the firewall when
DNS is not working
● New hostname must be recognized by the firewall
– Set the hostname/domain to match on System > General
– Or set hostname in Alternate Hostnames under System > Advanced on the Admin Access
tab
– Without this, HTTP_REFERER enforcement will reject attempts to reach the GUI by default
●
You could also disable that protection, but we do not recommend doing so
●
ACME package cert entry should have a shell command action for
/etc/rc.restart_webgui

19. HAProxy
● Works for sites hosted behind the firewall
● SSL offloading – certs are on the firewall, backend could be HTTP
● Several potential update methods
– DNS methods (of course)
– Have ACME place files on actual server via SFTP (Webroot FTP method)
– Use HAProxy ACME update plugin to run it on the same port used by
other services handled by HAProxy:
● https://forum.pfsense.org/index.php?topic=101186.msg690924#msg690924
● Add action to ACME Cert entry to restart HAProxy
– /usr/local/etc/rc.d/haproxy.sh restart

20. Other Uses
●
Could be used with other areas on the firewall, but your mileage may vary
– General guideline is:
●
Identifying to the public? Maybe.
●
Identifying to peers? Probably not.
●
Captive Portal – Yes!
– Use for HTTPS portal / redirects
– Enable HTTPS Login, enter hostname for the cert, pick the right cert, and make sure
clients can resolve the hostname
● OpenVPN – NO
– Always use self-signed, internal/private CA/Cert structure
●
IPsec – Maybe, but probably not
– Mutual RSA: Self-generated is best, technically possible but does not offer any
tangible benefits
– IKEv2: May work with some clients, but Let’s Encrypt does not allow the IKE
Intermediate EKU or IP Address SAN entries which some clients require

21. Other Uses
● Squid – Varies
– SquidGuard error redirects: Yes!
● Set redirect to “Ext URL Found” and use
https://host.example.com/sgerror.php?url=403&a=%a&n=%n&i=%i&s=%s&t=%t&
u=%u
– Reverse: Yes, though automating reload is tricky (check forum)
– Forward / MITM – No. Impossible to use in this context.
● FreeRADIUS / EAP – Maybe
– Self-signed is better overall, when possible
– Maybe if only used to identify the server with 802.1X/PEAP
– Can help work with clients where you can’t import/bypass your self-signed
cert structure
● NET-SNMP – No, use self-generated structure
● Ntopng – Yes, and the cert will be used automatically

22. Using with HA / XMLRPC
● Only install on primary node
● Create certificate w/SANs for each node + CARP VIP
hostname
● Use a DNS validation method since the primary must
answer for all nodes + CARP VIP
● Allow primary to create/validate/issue certificate, will
then sync to secondary
● Select as GUI cert on both units and for other services
● Will need to restart GUI on secondary after renewals

23. Conclusion
● Questions?
● Ideas for hangout topics? Post on forum,
comment on the blog posts, Reddit, etc