Mais conteúdo relacionado Semelhante a Meetup callback (20) Meetup callback1. It’s More Ops Than You Think
June 2018
Bob Balaban, CA/Veracode
2. Copyright © 2018 CA. All rights reserved.
Agenda
• The “promise” of serverless
• Veracode Dynamic Analysis
• Replacing a “server” with a “service”
– Advantages
– Dev and DevOps processes
– Demos
– Dev effort summary
– Ops effort summary
• Lessons Learned (including…security!)
• Q & A
3. Copyright © 2018 CA. All rights reserved.
Advantages of Serverless (from Wayne)
Automated Scalability
Built-in Fault Tolerance
LessOps – No infrastructure to manage
Lower Cost - Pay only for what you use
Faster development - Small functions easier to code and business logic focused
Lower Risk – responsibility transferred to provider
Integrated security – No patching, remote into O/S (SSH/RDP)
Automatic logging
We will come back to this later
4. Copyright © 2018 CA. All rights reserved.
Veracode Dynamic Analysis
• Scans a “target app” (website) looking for security
vulnerabilities
– Cross-site scripting (XSS)
– SQL Injection (SQLi)
– XML External Entity (XXE)
– Remote File Inclusion (RFI)
– OS Command Injection
– Server-side Request Forgery (SSRF)
– Directory Traversal
– …and many more
• See OWASP: http://www.owasp.org
5. Copyright © 2018 CA. All rights reserved.
How Do We Find Vulnerabilities?
• “Ethical hacking”
• We “attack” the target (in ways that don’t damage data or
functionality)
• If an attack succeeds, we report a vulnerability
– And help you fix it!
• Example: SQL Injection
– “Select (sleep(15)) where true”
– We can tell if it works because the response is sloooowwww
6. Copyright © 2018 CA. All rights reserved.
What a “callback” server does
1. Scan engine attacks a target app with (ethically)
malicious XML, Javascript, etc.
2. If successful, the attack causes the app to make a call to
the Veracode “callback server”
3. The server records the successful attack
4. The scan engine polls the server later to fetch data on
successful attacks, and reports vulnerabilities
7. Copyright © 2018 CA. All rights reserved.
Callback Server (“Before”)
Callback
Server
Target App
Scan Engine
1. Inject attack,
e.g., “include remote
PHP file from callback
service”
3. Get results
2. “I was
hacked!”
8. Copyright © 2018 CA. All rights reserved.
Existing Callback Server
• AWS EC2, golang
• Simple “GET” response to target apps
– Serves static file
– Records successful attacks
• Simple REST API for scanner engine
– Register/unregister scanners
– Fetch successful attack data
– Clean up attack data
• Postgres RDB for attack data
– Auto-purges after 7 days
9. Copyright © 2018 CA. All rights reserved.
New Callback SERVICE Objectives
Re-architect as “serverless”:
1. Lambda functions only run when invoked
2. AWS API Gateway offloads authorization, firewall and load
balancing
3. DynamoDB nosql database easy to use from lambdas
4. Both Lambda and DynamoDB usage can auto-scale
5. Simplify REST API, otherwise maintain the same interfaces
6. Add some metrics tracking (count of invocations)
10. Copyright © 2018 CA. All rights reserved.
New Callback Service (“After”)
Callback
Service
Target App
Scan Engine
1. Inject attack
3. Get results
2. “I was
hacked!”
11. Copyright © 2018 CA. All rights reserved.
Callback Service Components (AWS)
DynamoDb
Query attack info
Capture attacks,
Serve text
A
P
I
G
a
t
e
w
a
y
From target app
From scanner
λ
λ
13. Copyright © 2018 CA. All rights reserved.
Advantages of New Architecture
• DynamoDB TTL feature means we can simplify the
service’s REST API
– No need for cleanup calls
– No need for Postgres
• Lambdas do not get billed for idle time
• API Gateway will auto-scale lambda invocations when
needed (no need for auto-scaling groups)
• Logging calls go automatically to CloudWatch
14. Copyright © 2018 CA. All rights reserved.
Development Process Changes
• Smaller, independent functions (good!)
• No runtime debugging (bad!) – didn’t try Cloud9
– Fallback to “Printf debugging”
– Also required to see what your function is being called with (e.g.,
http event data structure not documented)
• Create AWS resources manually, the first time
– API Gateway (can start with a Swagger definition)
– Lambdas
– IAM
– DynamoDB tables
– S3 (backup Terraform db file)
15. Copyright © 2018 CA. All rights reserved.
Development Process Changes - 2
• Dev/test cycle
– Make code changes
– Manually (or semi-manually) deploy to AWS
• Need to figure out how to get code from IDE to AWS
• We use gitlab and Terraform
• Terraform can be used locally to push changes to AWS
• (Have to zip .js files first!)
– Test (Postman/curl)
– Commit changes to repo
– Repeat
16. Copyright © 2018 CA. All rights reserved.
Devops!
• Manual deployment is not sustainable long-term
• Must automate!
• Chose Terraform to script AWS resource
creation/configuration
– Injectable “variables” for AWS keys, config parameters,
environment variables
– Resource dependencies (e.g., create lambdas before api
gateway)
– Incremental changes! (if you keep a copy of the tf db)
• Chose Gitlab for CI/CD scripting
– Triggered on merge to Master branch
– Build zip files, manage Terraform invocation
17. Copyright © 2018 CA. All rights reserved.
Devops - 2
• Ongoing re-factoring of the project
– When and how to supply production account AWS keys
– Coordinate Gitlab deploys and “legacy” (Bitbucket) builds
– Who decides on gateway “stages”?
• Who monitors the logs?
– Some teams ship their logs to Sumo (more lambdas)
• Who watches scalability of the gateway and lambdas?
• Oh, and who watches cost?
18. Copyright © 2018 CA. All rights reserved.
Time Commitment – Dev and DevOps
• Dev was about 20% of “time to beta”
• DevOps (pipeline automation, config automation…) was
80%
• Why?
– The pipeline automation tools were new to us, we had to learn
them
– Some AWS features were new
19. Copyright © 2018 CA. All rights reserved.
Stuff We Had to Learn From Scratch
• API Gateway setup/config
• Lambda deployment/config
– Permissions, environment variables
• Querying DynamoDB efficiently
• Terraform – create/update entire AWS infrastructure
• Gitlab CI tools
20. Copyright © 2018 CA. All rights reserved.
Skills We Had to Improve
• Node.js
– Coding
– Debugging (via console.log() output, ew)
• Cloudwatch
• IAM for inter-component authorization
– E.g., API Gateway calling Lambda
– Lambda calling Cloudwatch and DynamoDB
• PostMan/curl
– Create http POST and GET tests
21. Copyright © 2018 CA. All rights reserved.
Recap: Advantages of Serverless
Automated Scalability √
Built-in Fault Tolerance √
LessOps – No infrastructure to manage meh
Lower Cost - Pay only for what you use √
Faster development - Small functions easier to code and business logic focused √
Lower Risk – responsibility transferred to provider meh
Integrated security – No patching, remote into O/S (SSH/RDP) √
Automatic logging meh
22. Copyright © 2018 CA. All rights reserved.
Lessons Learned
• Serverless is great, where appropriate
– Keep it simple
• Figure out the DevOps as you go
– Assume you will automate everything:
• Build
• Unit test
• Deploy
• Integration test
• You will re-factor the project at least as often as you
refactor your code
23. Copyright © 2018 CA. All rights reserved.
And Oh Yeah, Security!
“Security is like performance, you bolt it on at the end of the project”
24. Copyright © 2018 CA. All rights reserved.
“Security is like performance, you bolt it on at the end of the project”
(Said no one we want to listen to)
25. Copyright © 2018 CA. All rights reserved.
Severless Security: Application Dependencies
Application dependencies are going to become one of the main vectors for attackers
to compromise your FaaS infrastructure. It is crucial that you scan third party libraries
for vulnerabilities.
To drive home that point - a researcher recently found a way to get push rights to over 14% of
NPM Packages and mainstream ones at that, e.g. : debug, request, react, gulp... How? It was a
combination of brute force and credential leaks from GitHub.
In an evaluation of 1,000 open-source serverless projects, a threat research team revealed that
21% of them contained one or more critical vulnerabilities or misconfigurations, allowing
attackers to manipulate applications and perform various malicious actions.
Mitigations
Maintaining an inventory list of software packages and other dependencies and their versions
Scanning libraries for known vulnerable dependencies. Vulnerability scanning should be done as
part of your ongoing CI/CD process
26. Copyright © 2018 CA. All rights reserved.
Serverless Security: At-Rest and In-Transit equally vulnerable!
Serverless exudes a false sense of security due to secure API calls and encrypted
databases.
1. OWASP top 10 is still as relevant as ever!
2. SQL injection and other forms of injection attacks are still possible in the
serverless world, as are cross-site scripting attacks.
3. Keep your secrets in an Key Management System, not code.
4. Ensure your services have the least amount of privilege to read/write to your
databases.
5. Use direct/private connections between your VPCs and Databases wherever
possible.
Mitigations
Always use safe APIs that either sanitize or validate user input, or APIs which provide a
mechanism for binding variables or parameterizing them for underlying infrastructure (e.g.
stored procedures or prepared statements in the case of SQL queries)
Always use safe APIs that either sanitize or validate user input, or APIs which provide a
mechanism for binding variables or parameterizing them for underlying infrastructure.
27. Copyright © 2018 CA. All rights reserved.
Severless Security: Monitoring becomes extremely important
The one thing that the advent of serverless has made more difficult is monitoring and the
lack of understanding of security implications for various serverless assets.
Logging is key. Ensure you have CloudTrail enabled on all accounts for all regions. Don’t
assume a developer or malicious actor won’t create an asset outside of us-east-1.
The days of agents are gone! You now have to use platform-specific tools to monitor…
…using Cloudwatch Metrics (with alarms) to monitor resources and set limits on either calls or
spend. When alerted, investigate when/why those limits are being exceeded.
If you have a SIEM Tool or a 3rd-party monitoring service, send CloudWatch/Cloudtrail logs to
their services to be monitored. You’ll be amazed at what is already actively scanning your
public assets.
RYOM – (rolling your own monitoring) – Lambda’s written in Python using the Boto SDK are
extremely powerful to monitor/modify your assets and alert based on customized parameters.
e.g. - A developer adds their home ip address to connect to an asset for debugging
from home, but a metric was defined to only allow connections of a specific CIDR range and
kicked off the Lambda written with Boto to remove that home ip entry automatically from
accessing that asset.
Notas do Editor So what are some of the advantages? This is not a complete list.
Less operations because AWS is taking over IaaS.
Lower cost in many cases. We will review.
Small functions are easier to code, test, and maintain.
No more need to ssh or rdp into servers. So what are some of the advantages? This is not a complete list.
Less operations because AWS is taking over IaaS.
Lower cost in many cases. We will review.
Small functions are easier to code, test, and maintain.
No more need to ssh or rdp into servers.