Packet Capture on AWS. Simple explanation of why security people like to capture packets, how it can be done, potential architectures, and a POC using a WatchGuard Firebox Cloud, the CLI, a bucket, bucket policy, etc. and a lambda function to show that packet capture is possible. Next steps for an actual production solution.
Caveat: these slides were written in about one hour. Please refer to the paper for details.
Scaling API-first – The story of a global engineering organization
Packet Capture on AWS
1. Packet Capture on AWS
Teri Radichel | @teriradichel
https://www.sans.org/reading-room/whitepapers/detection/packet-capture-aws-37905
(A SANS Gold Paper)
3. What’s a packet?
• Data is split into packets or small pieces to send over the network
• Follows a protocol – the devices on both ends must understand
• Packets are reassembled at the destination, if protocol requires
• Some protocols resend failed packets (TCP) others do not (UDP)
Source DestinationNetwork
4. What Is Packet Capture?
• Capture the details about packets as they traverse the network.
• For example, if you have tcpdump installed can run this on mac/linux:
sudo tcpdump –x -n
• You’ll get a number of packets rolling by on your screen like this:
23:56:50.183837 ARP, Reply 192.168.1.95 is-at f4:5c:89:b6:7b:7f, length 28
0x0000: 1491 829a 0b62 f45c 89b6 7b7f 0806 0001
0x0010: 0800 0604 0002 f45c 89b6 7b7f c0a8 015f
0x0020: 1491 829a 0b62 c0a8 0101
• That’s a packet displayed in hex format (-x), and with IPs instead of
domain names (-n)
• http://www.tcpdump.org/tcpdump_man.html
5. Network Layers and Headers
• OSI or TCP model – network layers
• Different headers for each layer
• Example – IPv4 header
• Devices at source wrap the packet
in a headers
• Devices at the destination unwrap
the headers
http://websitenotebook.blogspot.com/2014/05/decoding-ip-header-example.html
6. Decoding Packets
• Packets are sent over the network in hex
• Decoding a packet header:
http://websitenotebook.blogspot.com/2014/05/decoding-ip-header-
example.html
• Hex to Binary Cheat Sheet:
• http://websitenotebook.blogspot.com/2014/05/hexadecimal-to-
binary-to-decimal-cheat.html
• Note that on AWS there are some AWS specific headers you don’t see
• https://www.youtube.com/watch?v=St3SE4LWhKo
7. Why do we care about packets?
• VPC Flow Logs only capture source, destination, protocol, timestamp
and a few other things; Web traffic logs only capture certain details
pertaining to web requests.
• A full packet has much more data that a other types of network logs
• Sometimes the full packet is required to resolve network issues
• Attackers obfuscate packet data to pass through security appliances
undetected – some of these attacks are only visible at packet level
• Data hidden in unused packet fields can exfiltrate data on channels in
undetected manner only visible when looking at raw packets
8. Captured packets can be stored and analyzed
• Many programs store packets in a format call “pcap” files
• Programs such as Wireshark can read these files and help security
professionals analyze the data
• An IDS/IPS can alert on or block packets that match nefarious
patterns (Snort, Bro, Suricata)
• Yara files can be used to match packet capture patterns and alert or
block malicious traffic
• Some devices like a WatchGuard Firebox with security services
provide this service so you don’t have to do it yourself
9. Typical Way to Capture Packets
• Span port on switch
“mirrors” the data
• Send packets to IDS/IPS
and/or logs
• So…how do we do this on
AWS?
• Can’t plug into the
switch…
https://wiki.wireshark.org/CaptureSetup/Ethernet
11. A better way…
• Route All the Packets Through Packet Capture Host(s)
• Capture host(s) must be able to receive the traffic, log it, and forward
the traffic to the destination
• Network configuration must force all packets through the capture
host(s)
• Any extraneous routes let packets bypass capture host(s)
13. NAT Architecture
Security groups
and route tables
can force all
traffic through
the NAT. NAT
instance can be
modified to
handle all traffic,
in and out.
14. Proxy Architecture with Load Balancing
This shows
architecture for
Squid proxy.
Squid proxy
would only
handle layer 7
traffic but we
could modify
these instances
to capture all
traffic. Load
balancing
provides scaling.
Would need to
handle inbound
and outbound. https://aws.amazon.com/articles/6463473546098546
15. Proxy Architecture – Peered VPCs
Instances can only talk to other instances in a directly peered VPC. A proxy will allow traffic to pass
through one VPC to another. This can be used for nefarious purposes but in our case we can
leverage this to capture as it crosses network boundaries, which could be useful. More on that in
an upcoming slide.
16. Packet Capture Via Security Appliance
A security appliance
designed to capture and
pass through network
traffic after analyzing it
might be a good way to
capture packets as they
traverse network
boundaries. Noticed the
network design which
separates the packet
capture functionality and
storage from the rest of
the network.
17. Transit VPC
We could use a transit
VPC to capture all traffic
coming into and leaving
AWS, or traversing the
network from one
account to another. This is
a very interesting idea
because not only does it
make managing
connections between
different networks easier,
it provides a way to
ensure all packets must
pass through our packet
capture architecture. https://aws.amazon.com/blogs/aws/aws-solution-transit-vpc/
18. POC – Capture Packets in Lambda Function
https://github.com/tradichel/PacketCaptureAWS
Here’s the POC using
WatchGuard Firebox Cloud’s
CLI command to capture
packets. Clone the code,
execute the run file and it will
ask a couple questions. You
will get this…
Note the protection of the key
in S3 with stringent bucket
policy. Could make it even
more stringent by creating the
key on the fly in AWS with no
access to it outside the AWS
network.
19. Packets!
Please read the paper for details. Must have packets flowing when you run the lambda to capture.
20. What Next? Suggestions?
• This was a POC Only!
• Lambda is short lived – would want a source that stays online
• Need to test performance over time of this solution and others
• Scalability
• Storage
• Potentially set up an open source IDS/IPS to analyze packets
• Store as PCAP
• Or store to CloudWatch and use some other type of parser to pull the
data out into PCAP or desired format
