AdvancedMD Customer Presentation
•
2 gostaram•807 visualizações
SplunkLive! Salt Lake City AdvancedMD Customer Presentation
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a AdvancedMD Customer Presentation
Semelhante a AdvancedMD Customer Presentation (20)
Mais de Splunk
Mais de Splunk (20)
Último
Último (20)
AdvancedMD Customer Presentation
- 1. Copyright © 2014 Splunk Inc. AdvancedMD
- 2. 2 Tyler Germer Manager, Platform Operations AdvancedMD
- 3. 3 ADP AdvancedMD AdvancedMD • Acquired by ADP 4 years ago • Practice Management • EHR • Mobile offering • AdvancedInsight™ Reporting ADP (parent company) • $11B revenue • 620,000 clients
- 4. 4 About Tyler Manager - Platform Operations Team ADP employee for 3.5 years Experience in all areas of system administration / networking Evangelist / Believer in Splunk Enjoy the gym, racquetball, and good food Originally from Canada …love watching Hockey. Go AVALANCHE!!
- 5. 5 Operations Team Charter Managing our data center, production server, infrastructure Patching our online medical applications Reduce client impacting incidents Ensure application uptime Work with support teams on customer issues
- 6. 6 Previous Monitoring Solutions: Key Challenges “Splunk was going to do everything our monitoring solutions couldn't.” • Broad infrastructure – HP blade servers, F5 load balancers, Juniper/Cisco switches – everything is generating logs • Monitoring solutions told us different things (Nagios, Event Sentry, Cacti) • Cumbersome to find answers Hard to troubleshoot quickly and find root cause
- 7. 7 Splunk Phase 1 - Visibility across the Stack (single pane of glass) – application, networking, etc. - Configure saved reports / dashboards - Utilize flexibility of Splunk All Logs “Phase 1 – Operational Intelligence – find out the things we weren't seeing.”
- 8. 8 Splunk Architecture 5 x Indexers (HP DL380) – physical 10 x Search Heads – virtual 4 x Heavy Forwarders – virtual 4 x SyslogNG Servers – Virtual 620WindowsUniversalForwarders(abouttodouble)–Virtual/Physical Approx. 100 networking devices 1 Deployment Server (Corp) / License Server and 1 Cluster Master 1 Deployment Server (Prod)
- 9. 9 Splunk: Cluster Topology Data sources • Windows event logs, IIS logs • Syslogs from firewall, switches, SAN storage devices, Aruba wireless, VPN, HP OA • Custom logs Indexing • Set up for 300-350 GB • Current: 20.0 GB Applications • Splunk for Juniper / Cisco, Splunk App for Stream, Prelert
- 11. 11 Fast Time to Value with Splunk "Before Splunk, it took days if not weeks to find source of a problem, now it only takes minutes or hours." Software release planned Within 30 min of software release we saw trend of huge errors Reached out to Engineering immediately to point them to errors and so they could fix the code before customer impact Splunk came to the rescue – data is so visible we were able to react quickly
- 12. 12 Splunk in the Operations Team Everyone MUST use Splunk – Mandatory goal of taking Using Splunk Training Currently 10 users, users from other Departments underway 2 actively working on providing more value through Splunk Teams look at Splunk on patch nights – Look at potential Errors Splunk User Group @ Salt Lake City – Email: Tyler Germer tgermer@advancedmd.com
- 13. 13 Lessons Learned Managers … LISTEN to your employees It’s very easy to get data into Splunk Real art is getting valuable data out / asking the right questions Splunk Deployment Server is your friend Docs.splunk.com is FANTASTIC Splunk IRC … nerds helping nerds Great book – Big Data Analytics Using Splunk
- 14. 14 What’s Next – Long Term Vision "Splunk IS a game changer for ADP AdvancedMD." MORE SPLUNK - Splunk for Sales, Engineering, Security, etc. Using more Splunk Apps to get quicker return on investment – Splunk Apps for F5, SQL, Exchange, Active Directory, NetApp Embrace DB-Connect (underway) Upgrading to Splunk Enterprise 6.2.2 Replace Event Sentry with Splunk (underway) Splunk architectural review Proactive customer service and business analytics
- 15. 15 Going above and beyond …
- 16. Thank You
Notas do Editor
- ADP’s flagship venture into the Medical industry ADP highly vested in helping AdvancedMD be successful PM: running the financial / business part of the company, EHR (Electronic Health Records) – running the medical side of the business, as well as some other offerings like ODBC and SFTP connectivity for secure transfer of data AdvancedInsight: advanced business analytics for our customers LOTS of HIPAA data. Security, Operation Intelligence, and data awareness very important
- Manager for coming up on 2 years. Worked as a Sys Admin under the team before that 12 years experience working in all areas of System Administration and Networking Splunk project took roughly 1.5 years to get approved, purchased, installed / configured
- My team’s responsibility is to ensure functionality, uptime, and maintenance of our Production Data Center, as well as the server, networking, and SAN infrastructure at our corporate office We also maintain and update our Practice Management and EHR applications Our goal is always to reduce / remove client impacting incidents by being very aware of our environment, how it is running, and what needs to be improved We also work heavily with our Client Support teams, to help resolve any client issues related to our infrastructure We invest heavily in monitoring, so that we can hear and feel the ‘heartbeat’ of our infrastructure at all times We have actually had vendors tell us that they appreciate the level of monitoring that we have, because it helps them know that THEY have a problem … Splunk was the obvious next step in taking our monitoring to even greater levels
- Before Splunk, even with the thoroughness of our monitoring, there were still challenges Our infrastructure was broad, including things like HP Blade Servers, load balancers, firewalls / switches, IIS logs Benefits were that our monitoring gave us a reasonable picture of what was happening in our environment. Things like Nagios and Event Sentry gave us insight into issues that were happening, but it was very much a reactive type monitoring Problem was that logs were spread out all over the place, but no central location to view them all when needed Makes it hard to troubleshoot a problem quickly, or become more proactive Then along came the introduction to Splunk …
- How we were going to implement Splunk Phase 1 – Operation Intelligence Get basic Windows and Syslogs into Splunk <go through green area to right>
- When we designed / built Splunk, went big from the start, for easy growth, as we knew how vital Splunk would become Production has own HF and Deploy Server, and CORP have own HF and Deploy Server
- Current sources of data include … Have hardware capacity to handle roughly 300 – 350 GB of data ingested per day, but only at 20 GB per day so far. Amazing considering how many sources of data are already included
- Outline each source Each source has unique logs with unique requirements Encourage Engineering to write code that leaves the ‘bread crumbs’ behind
- We quickly found value in Splunk Goal: All Event Logs into Splunk before Release night Night of release, Splunk saw huge errors Engineering was able to diagnose issue and get new code out in record time Data was so visible, so fast. Splunk did all the work, and helped us respond much faster
- Vision and goal for Splunk: a ‘way of life’ for the Platform Operations Team Splunk dashboards used for our Patch Nights, to help minimize client AdvancedMD housed the first Splunk User Group in SLC, and are large supporters of the value that Splunk provides
- Art of Splunk: Question to be asked … ‘I have all this data in one spot … now what?’ Be creative about what data you get out of Splunk Put on the business / bigger picture hat, decide what you’d like to know, then figure out how to make it happen with Splunk Docs.splunk.com is fantastic. Documentation easy to read, and thorough. About 15 of their documents are key / critical Quick answer / quick fix … Splunk IRC. It’s Splunk Nerds helping Splunk Nerds
- Phase 1 is Operational Intelligence, long term is all areas of company using Splunk Want to utilize Splunk Apps more, for quick value add Have used Splunk Professional Services in the past to help work through some roadblocks, as well as help us configure our systems to match best practices DB Connect: More SQL Visibility Want Splunk to analyze our data, to provide more insight into customer patterns, as well as business analytics for both us AND our customers
- Make mention of both Tony Bolander and Scott Smith Also make mention of Corporate Culture and the company’s passion for the product and what it can do.