WEBINAR: HOW URBAN OUTFITTERS MADE A 7X IMPROVEMENT IN THEIR MOBILE APP PERFORMANCE
•Transferir como PPTX, PDF•
1 gostou•1,304 visualizações
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a WEBINAR: HOW URBAN OUTFITTERS MADE A 7X IMPROVEMENT IN THEIR MOBILE APP PERFORMANCE
Semelhante a WEBINAR: HOW URBAN OUTFITTERS MADE A 7X IMPROVEMENT IN THEIR MOBILE APP PERFORMANCE (20)
Mais de Apteligent
Mais de Apteligent (14)
Último
Último (20)
WEBINAR: HOW URBAN OUTFITTERS MADE A 7X IMPROVEMENT IN THEIR MOBILE APP PERFORMANCE
- 1. Webinar: How Urban Outfitters Improved Mobile Performance 7x October 15, 2013
- 2. Agenda About URBN & URBN Mobile URBN Dev Process Challenges Rewind Use cases for mobile app performance management (mAPM) @ URBN Crittercism mAPM
- 3. Crittercism Background Leader in Mobile Application Performance Management Mobile is our DNA We Operate at Scale We started with mobile and built the mAPM solution from the ground up • Monitor 1B+ app instances and 600M+ devices • Analyzed over 100 billion app launches • Manage thousands of customer accounts Our Investors “Crittercism and its technology are widely adopted and provide value to those struggling with the emerging mobile application market.” July 22, 2013 Crittercism Solution Overview 3
- 4. Mobile is Changing How Companies Like URBN Interact With Customers • 87% of U.S. consumers spend the majority of their time on a mobile app rather than a mobile browser (Neilson) • “10 percent of in-store transactions in the United States are made via mobile phone”, Starbucks • “55% in-store customers have a smartphone. Those with our app spend 40% more and visit the store 2x as much”, Walmart • Ebay expects to generate $20 billion each from mobile commerce and mobile payments this year. June 13, 2013 Crittercism Solution Overview 4
- 5. About Urban Outfitters • Over 400 retail locations in 7 countries • Five retail brands: Urban Outfitters, Anthropologie, Free People, Terrain, and BHLDN. • Hip clothing, footwear, accessories, h ousewares products • Over 25,000 Employees June 13, 2013 Crittercism Solution Overview 5
- 6. URBN’s External Facing Mobile Apps • • Free People Brand • June 13, 2013 Urban Outfitters Brand Anthropologie coming next year Crittercism Solution Overview 6
- 7. URBN Internal Facing Mobile Apps • • • • • June 13, 2013 Crittercism Solution Overview MPOS Restocking Reticketing IPOS 2 new apps coming 7
- 8. Process • • Continuous Integration • June 13, 2013 Agile Continuous Delivery Crittercism Solution Overview 8
- 9. Issues No visibility into crashes Mobile Service No metrics on crash frequency No symbolication of stack traces No high level view of overall stability No visibility into network issues Mobile App
- 10. Pilot Issues • ~6% crash rate with lots of crashes unexplained • Hard to reproduce, often need up to 2 weeks of troubleshooting • Store associates = testers • App was nearly pulled from devices! June 13, 2013 Crittercism Solution Overview 10
- 11. Crittercism Provides • • • • • • • • • Crash breadcrumbs Symbolicated stack traces Crash statistics Upload API / dSYMs Handled exception logging Network monitoring Wicked WWDC parties Awesome hoodies Marshmallow guns June 13, 2013 Crittercism Solution Overview 11
- 12. Usage Scenario 1: Automated deploy of new dev build via API Development build crash • Receive email alert of new crash / view report • Create Jira ticket via Crittercism dashboard • Mark crash report as known • Fix issue • Mark as resolved June 13, 2013 Crittercism Solution Overview 12
- 13. Usage Scenario 1: Automated deploy of new dev build via API Development build crash • Receive email alert of new crash • Create Jira ticket via Crittercism dashboard • Mark crash report as known • Fix issue • Mark as resolved June 13, 2013 Crittercism Solution Overview 13
- 14. Usage Scenario 1: Automated deploy of new dev build via API Development build crash • Receive email alert of new crash • Create Jira ticket via crittercism dashboard • Mark crash report as known • Fix issue • Mark as resolved June 13, 2013 Crittercism Solution Overview 14
- 15. Usage Scenario 2: Handled Exceptions • Notice large numbers of handled exceptions • Look further into cause • Log Jira Ticket • Fix Issue June 13, 2013 Crittercism Solution Overview 15
- 16. Usage Scenario 3: Network Monitoring (1) • Notice may 400+ level HTTP failures – E.g. many image URLs failing • Report to Network / API team • App improvement June 13, 2013 Crittercism Solution Overview 16
- 17. Usage Scenario 3: Network Monitoring (2) • Notice may 400+ level HTTP failures images.acme.com/activity/images/xalakd.png – E.g. many image URLs failing • Report to Network / API team • App improvement images.acme June 13, 2013 Crittercism Solution Overview 17
- 18. Usage Scenario 4: Using Breadcrumbs to track user flow • Allows us to understand the users context • Can more easily duplicate crash case • Log a defect • Fix / Deploy defect June 13, 2013 Crittercism Solution Overview 18
- 19. URBN Benefits • Lower crash rates – Gone down from 6% to .8% currently – 7x improvement! • Better visibility of network issues • Better visibility of caught exceptions • 2 weeks of troubleshooting minutes • More comfortable because of Hoodie warmth June 13, 2013 Crittercism Solution Overview 19
- 20. Crittercism Solutions Unified mAPM solution for Dev/QA & Ops Crash Performance Error monitoring, reporting & troubleshooting Cloud Performance Cloud service & network performance management Crittercism Platform Run APPS FASTER, BETTER & SMARTER Full Mobile Device & OS support Real-time Big Data Engine Dashboards & Reports Security, RBAC, d ata security Availability & perf management Data & System API
- 22. Summary & Next Steps • Mobile platforms introduce new complexity in device, apps, OS, network, & carriers that require a new set of tools to monitor effectively • Urban Outfitters has been able to deliver dramatically more stable and responsive mobile apps using Crittercism’s mAPM solution • Start your free trial! Questions? sales@crittercism.com June 13, 2013 Crittercism Solution Overview 22
Notas do Editor
- Josh:Hi. Thanks for joining us today for our webinar, How Urban Outfitters Improved Mobile Performance Seven Times. Today, we’re going to be discussing how Urban Outfitters used our mobile application performance monitoring solution to give their users faster, more reliable apps.
- Josh:We’ll cover how they increased work productivity with more stable employee-facing apps. We’ll also look at how Urban Outfitters targeted performance issues in their consumer apps and really to do that, we’re actually going to be joined by a couple of guests from Urban Outfitters. Chris Hunter is the mobile engineering manager at Urban Outfitters and he leads a crack team of mobile engineers that develop both these consumer and employee-facing mobile applications. Chris has over 10 years of experience in roles as a software engineer, DevOps as well as an independent consultant. Now along with Chris, we have Jason Grandelli. Jason is one of those crack mobile engineers working on Chris’s team. He focuses on both iOS and Android development and Chris is going to be leading most of the presentation but Jason is going to be chiming in on some of the technical aspects.Let me give you a quick preview how we’re spending the next a little about 45 minutes. What we’ll be talking about is Urban Outfitters and the mobile applications that they have. We’ll spend a bit of time talking about development processes and then Chris will dive into and discuss some of the challenges that made it hard for Urban Outfitters to get the performance that they wanted.We’ll dive into the tools, mobile application performance monitoring that Chris’s team use and actually take a look at the specific use cases at Urban Outfitters and then we’ll close out with just a moment, I’ll give you broad brush strokes on what Crittercism’s mobile APM solution is and then we’ll reserve a bit of time at the end to take some of your questions so as we’re going through this presentation, please put your questions in the Chatbox and we’ll tackle those at the end of the presentation.
- Josh:Now at the highest level, before we go too far, I’d like to just spend one minute on who Crittercism is. Crittercism provides the world’s first mobile application performance monitoring solution. It provides granular data that enables developers, IT ops, product managers and businesses generally to run faster, better, smarter mobile apps. What makes this unique is we have a massively scalable platform that delivers a real time global view of diagnostics across iOS, Android, Windows phone, HTML5 and hybrid apps. We are on over 600 million devices and we capture over two billion transactions daily across the world. Along with Urban Outfitters, there are enterprises like BIM, Yahoo, Home Depot and Lowe’s that are using us to improve their mobile apps performance. Now, we are also funded by some of the top VC firms that includes like folks like Google Ventures and Kleiner Perkins. At the end of the day, what this all means is we are really specifically built from the ground up to deal with the complexity of mobile to help companies like Urban Outfitters deliver faster, better, smarter mobile applications.
- Josh:Now, to set a little bit of context around mobile and how it’s changing the game for companies like Urban Outfitters in terms of how they interact with their customers and their employees, let’s just spend a moment looking at the market.What we know is that the sales of mobile devices have overtaken PCs and therefore, the time that people spend on those devices reflects that so Nielson reports that 87% of people spend their time using a mobile app rather than a mobile browser and companies are prioritizing investments towards mobile to better engage with these customers.To give you an example, Starbucks reported that they have … they said that 10% of all their revenue in stores is now transacted through a mobile phone showing that mobile is really driving revenue for Starbucks. From Walmart’s perspective, we know that half of their in-store customers have a smartphone close at hand and more importantly, those with an app will spend 40% more and consequently spend double in the store than other customers. Mobile is driving loyalty and retention and then looking at eBay, they expect to have over 20 billion in mobile related transactions and payments this year so simply put, mobile is a critical vehicle for driving your company’s revenue, customer loyalty and satisfaction but in order to do that, your apps must really delight and they must deliver. You’re competing for very limited app real estate.The average user has about 41 apps on their smartphone. If you just look at the Apple app store, they have about 900,000 apps in their app store so you’re competing for prime real estate and that it is really critical to be able to deliver on that high performing experience for your customers in order to compete for that real estate.With that as a backdrop, let me pass it over to Chris. Chris is going to begin to walk through the Urban Outfitters context, a little bit about Urban Outfitters themselves and then setting the context of their mobile apps and how they’re approaching those. Chris, let me hand it over to you to begin setting the context of Urban Outfitters in your mobile footprint.
- Chris:Thanks, Josh. To start, we have about 523 locations. I was just told so we’re actually much higher than the 400 that’s on this slide. We represent five brands and a couple incubator brands so you see Urban Outfitters being the largest followed by Anthropologie and Free People, Terrain and BHLDN. These are all very different markets.Our age demographic ranges from about 13 to 45, 50 years old depending on the brand. Urban Outfitters, Anthropologie and Free People are very sort of complimentary in the clothing market. We market hip clothing, footwear, housewares and landscaping type products and we have about 25,000 employees which is actually a very huge opportunity for us to offer mobile applications to help run a large enterprise of this size. We have these external facing applications that service the customer and the brand and then we have these internal facing applications which allow us to streamline different operations within the company.Josh: I mentioned to my wife that I was doing this webinar with you guys and she was quite excited and of course her first comment was, “How do I get some coupons?” so definitely good footprint and growing quickly so we’re already updating the 400. Can I go to the next slide here?
- Chris: Yeah, go ahead. Here’s just an example of one of the latest apps we shipped as a team. My team is actually a very new team. We typically build applications using third party consultants but we’ve now started to drive a lot of the software engineering in-house to sort of promote quality and sort of ownership of these applications and this My UO screen is the home screen of the $5 version of the Urban Outfitters app that we built as an internal team. We shipped that around September 23rd or so.Then this on the right is an example of the, I believe this is the Android application which we shipped a couple of years ago so you can kind of see the progression in terms of color and style on that one and such. We also have a Free People application that’s out and that application is pretty slick. You should take a look at that and then possibly going to be doing an Anthropologie app in the next fiscal year. There’s other apps too but I really can’t speak too many of them unfortunately.Josh:Can you speak just a little bit to some of the capabilities and functionalities within the Urban Outfitters app itself? I understand that there might be some loyalty, social aspect, some music app. Can you tell us a little bit about that.Chris: Yeah. The previous version was primarily a shopping experience with a couple of other sort of ancillary features like the UO music feature and something like that but primarily, it was focused around using the product catalogue and driving revenue through conversion of shopping but with this latest version, we kind of wanted to focus more on the gamification aspects of it so kind of allowing people to maybe buy out of a particular category and upon completion of the checkout, you would be rewarded with maybe a 10 or 20% off on your next purchase or if you check the Urban Outfitters store and you bought something, that’s like a two-way kind of challenge and if you did that, you will get another type of promotion and then you get badges as well. There’s a badges tab on that screen would drive you into like a trophy case which we have these sort of trophies for completing different challenges. That kind of drives the retail experience. It makes it more than just your standard three-step checkout kind of flow.Josh: Yeah, more sort of immersive … the actual …Chris: Yeah, and we’re integrating with Twitter and we’re integrating with Foursquare and we’re shipping a new Instagram feature soon as well.Josh: Cool. What’s the Instagram feature?Chris: I’m not going to disclose that. You’ll have to …Josh: Fair enough. Cool. I’ll move to the …
- Chris: I could be honest. Some of the internal applications that we run are the mobile point of sale which is the MPOS. That’s our sort of iPad in-store checkout so the sales associates can checkout people. There’s the restocking and reticketing applications. Those are more around bringing products onto the store or managing inventory or managing returns and that kind of stuff and there’s the iPad point of sale which is the sort of stationary, larger featured mobile point of sale which is typically only at the kiosk and then we’re working on a couple of new and exciting applications which maybe if we do this next year, I can talk about.Josh: I think it’d be helpful if you could give the listeners just a sense of the day in the life before and now with these new apps you’re releasing with. What were they using before potentially getting access to these and then what kind of devices and what it will look like once they have these on their hands?Chris: A lot of the stuff was run by your standard sort of inventory guns for different things especially like the restocking bits and then the point of sale was typically just like an IBM kiosk style cash tool or the IBM Java checkout. This simplifies the setup quite a bit. It also reduces cost quite a bit on each one of those.Josh: What devices are you guys thinking of using for the store associates once they have the mobile app?Chris: We already have a ton of iPod Touches out on the field. That’s typically like standard issued equipment for a sales associate. We’re going to be tracking for the new iPod Touch soon and we’re trying to get everybody on iOS7 so we’re trying to get on the latest generation iPod Touch with the … I forgot the name of the company that makes it but we had this scanning cradle, this upgraded scanning cradle that we’re moving to so we’re going to be operating quite a bit of devices this coming year.Josh: Sort of in the new world, a store associate could take this app, they could potentially scan a tag on something and order additional items or determine if it’s also in a different color at a different store just straight from the iPod essentially?Chris: Yeah. They can also, when cartons of things come in from the truck, they use this to kind of bring them into the inventory. When they want to scan like a number of … there’s few out on the floor, they would scan each one and then we give them sort of a count of size and color and that would sort of drive what needed to come from the stockroom to the floor and things like that.
- Josh: Very cool. Can you maybe talk to us a little bit about the process for how you do it?Chris: Yeah. We practice agile here although we kind of call it light agile because we try to stay nimble and not have too many meetings but typical JIRA ticketing, and print planning process along with a continuous integration and delivery schedule. Actually, some people will take every single dev build that we do. Some of the business stakeholders will only take the data before their shift but we’re actually using Crittercism on every single build that we do so that when people test out something and they experience a crash, we immediately get alerted of what version that was. It will have the build number integrated with the crash report that we get so we know exactly when it was introduced or the latest version that it’s in before it ever goes to a data and especially before it goes to production.I could probably sit here and talk for 30 minutes about what continuous integration is and how we settle that up but I mean it’s pretty standard practice to do that in the software engineering shop at this point but …Josh: Just to give folks a sense of what’s your cadence like how frequently are you releasing or pushing stuff out?Chris: It can vary quite a bit. Our developers do featured branches for every ticket that we work on so our developer may be committing on their featured branch for multiple days and they won’t actually do a full request into our dev branch for a number of days or they could be doing it really easy then they could be doing multiple tickets a day and produce a build each time so every time you do a commitment to go, you get a build so that can be zero times. It could be 10 times in a day. It really depends on what they’re working on but we had slightly delivered the applications to our testers and our stakeholders.Josh: Got it. Okay, perfect.
- Chris: Before we use Crittercism, we were kind of a very new team and we didn’t have all the tooling that we have now so we were trying out this restocking pilot and some of the issues that came in too were like we have no visibility into crashes, we have no symbolication of crashes which means you don’t necessarily know where in your code that crash originated from without having a deep sense to link it up and then there was obviously no visibility into whether there was a correlation to network related issues. There’s obviously no metrics involving quantitatively how many of these crashes we’ve seen, what percentage of our overall crash rate was this particular crash and you’re kind of flying blind without any of that stuff.Josh: There may be some folks who aren’t familiar with the sort of symbolication side of things. What does that mean?Chris: If you see a stack trace without symbolication, it looks like a lot of memory references and there’s really … you basically have no visibility as to where in the source that particular crash originated from. You only know some of the binary level of details which is not really human readable. When you have symbolication, you can actually follow a stack trace using lines in your source code so you can say, “Okay, well. In this view controller outline 131, the last thing that executed successfully was this and then right at that point, that was the last thing you had contact with and you crashed.” Jason, you want to add to that?Jason: Yeah. I mean to kind of break it down over in layman’s terms, the symbolication for crash details is pretty much just providing whatever process servers you’re using even if you’re using XCode to evaluate the crashes with the blueprints of your app so that way, instead of getting memory addresses for objects, it will actually tell you the name of the object as you named it in your app and then it will also evaluate the individual lines of code. Otherwise, you just get some generic memory references which no human I’ve met understands those.Chris: I’ve met one. Yeah, you can read up on symbolication but for the most parts, it just allows a human to sort of trace back where that crash sort of originated from, the last point known.Josh: Perfect. Now that’s perfect. The net of it is it essentially allows your development team to translate something that’s very machine readable into something that’s more human readable so you can troubleshoot more quickly. Then in terms of just put it in sort of the context of your end user, what some of this would look like then is if you’re using an Urban Outfitters app and it blows up, you as a development team didn’t essentially have an idea of what was causing that on the end user side, was it a network issue meaning latencies, driving slower performance or was the app just purely blowing up. That’s essentially the challenge you were having. Is that right?Chris: Yeah. You can even have maybe a network call that just completely failed with a 500 level failure which didn’t deliver a certain payload we were expecting and maybe we weren’t handling it correctly but that caused the app to crash so without this level of visibility and considering their unmanaged devices, I mean you’re powerless to do anything although I explained kind of what we did without it but it’s hideous at best so go ahead.
- Josh: We’ll get a little bit more into that as we move forward. It sounds like …Chris: Yes. We did this pilot of the restocking app as it kind of eluded too and we put the app out to a couple of stores in Philadelphia here and kind of saw what the experience was. The app worked but we started getting these reports of, “Oh, the app keeps crashing in the middle of my job and it’s very frustrating.” We were trying to figure out what to do so we were trying to dump crash logs off the device and decipher something from those. We were trying to train the testers on to mentally keep track of the steps that you use within the application and then sort of report back. “I did this, this and this,” and then kind of giving them a standard JIRA type format on what their test case was with that crash.Now, stores associates and we’re talking about pretty young kids essentially and we’re asking them to do this sort of highly analytical thing which is probably good for a resume but it’s not really like what they do. It was very tough to get what we needed without this tooling so it was taking us weeks to fix certain things that should have only taken … it takes so long just to sort of do the archaeology of getting what you needed where the actual fix only took maybe a couple of hours so it was just a huge time suck and really inaccurate.Josh: I mean definitely not a sort of professional QA team per se that you’re forced to rely on in a way.Chris: No, and these types of store associates, they’re in the field. They’re dealing with real data on these web services where your QA team, they don’t necessarily use the app the same way that a stores associate would use in the field so a lot of times that we ship an app, we think it’s stable but we don’t really know until it’s out in the wild. I’m sure anybody on this call who’s ever shipped an app in the production will tell you that actual users are the best QA you’re ever going to get.Josh: This was a pilot but can you give us a sense of kind of what impact an app not working has to store associates? Imagine the associates see crashes happen.Chris: I think you’re pulling them down. They’d get frustrated. They don’t want to use the application. Not only that, it also affects the organization’s trust of doing this type of thing. You want to instill confidence in your senior management that mobile applications are the future and that they’re going to drive business so when you’re having like a simple pilot like this falling over, it’s not very good.Josh: Got it.
- Chris: We just saw actually on the previous slide, we used a 6% metric there which we didn’t know that until we actually implemented Crittercism to see, get a view of sort of as is how we’re doing and then we’ve shipped another build out and it really exposed kind of what was actually happening. It was sort of an epiphany of sorts when you see all that data coming in for the first time.We started using crash breadcrumbs and we’ll talk about that. You get symbolicated stack traces, crash statistics, you get the upload API and for dSYMS which is a crucial part of our continuous integration flow so that you can automatically deliver bills without having a release manager collating a bill then release notes and uploading a dSYM to Crittercism and uploading the binary, the template along with the dSYM so you need this API. That’s like a key feature for us. Then you have handled exception logging, network monitoring and then these guys are pretty cool. You get to hang out in WWSC and have a good time. You get some marshmallow guns that we almost got fired over and it’s been a pretty good relationship I would say. Josh: Hopefully, the marshmallow guns weren’t too much trouble. Chris: They hang up on the wall now. They won’t be using that much.Josh: Trophies. Chris: Yes.Josh: Should we talk a little bit about the different usage scenarios that you guys are finding yourself using Crittercism to help out in terms of your mobile performance?
- Chris: Sure. In this example, one of the main scenarios that really drives our app development here is catching crashes in development. A lot of times, we’ll do a build on each committed dev. People start using it so we have certain people that we deliver that app to so maybe the QA or UX engineers that we have or different product owners. They get all the builds and they might try out one or all of them and it might even happen on a build on the developer’s workstation where you get to see a crash. All the development team is immediately alerted of that crash so you’ll kind of go in and you’ll check out what that was and then you just fix it right there. You don’t really wait for it to necessarily to go out in a data or go out and then master it, build this. That’s one of our main use cases I would say. This particular … go ahead. It’s fine.
- Chris: When that happens, typically we would just create a JIRA ticket right from the interface so that’s one of the other integration points with Crittercism back end that’s useful for us. We use a hosted JIRA, an Atlassian JIRA and we just go ahead and create the ticket right from that interface. You don’t have to copy and paste anything and then you can see here on this slide, this is one of our agile boards from one our previous prints and you can see that pics of the crash is already there so you can just pick it off and do the work.
- Josh: This sort of helps you in a way accelerate the discovery and troubleshooting process because the crash information seems to be percolated through your process very quickly.Chris: Yeah. There’s no lags on … there’s not a lot of paper-pushing at all. You just push one button. It’s in the back log and then I’m constantly grooming this print anyway then it’s just in the flow. I go every morning. I take a look at the back log. I take a look at what’s active and if we have bandwidth to take on, if there’s no ticket, I’ll move it right in. If not, I’ll just put it in the next print but typically, if it’s a serious crash or something that is severe, it will get escalated at the blocker.
- Josh: Perfect. We’ll talk about the second use case scenario?Chris: Go ahead. Handled exceptions, similar to the Try-catch. It’s basically a Try-catch that gives you some sort of history of exactly where it happened so you may know like a particular area of your code where maybe you’re relying on a buggy service or something and you might want to wrap that call in a Try-catch that would catch a certain exception or you may be operating on a particular object where you may get a null back or something that you may have to handle. It’s good to use this type of handled exceptions so that you get logging on how often that’s happening and if we become sort of this … I can oftentimes like put user in an awkward state. You might handle it but they didn’t necessarily get where they wanted to go or it doesn’t look exactly the way it should which may not crash the application but it’s still not good so it’s good to put those types of things in handled exceptions so that you can see the frequency in which they occur and then sort of escalate maybe a refactoring or tell a team that you rely on that the service they’re delivering you is causing the situation for you and maybe they should look into their code.Josh: With the prior use case, it was very focused on more fatal areas. This is more along the lines of non-fatal areas that impact the performance of the app.Chris: Yeah, non-fatal. It’s a little more proactive I guess in terms of you know this might happen. You know [inaudible 00:28:28] so you know how often it will happen and you use it as a way to prioritize some work you might want to do.Josh: How are you guys prioritizing these? Are there impacts to the number of people impacted or how do you think about that?Chris: We try to pick off a few, from the list of handled exceptions in each print so we just kind of just go from the top to the bottom based on our availability.
- Josh: Got it. Perfect. The third scenario which is more around the monitoring kind of the connection of the app here, various internal and external services. Chris: Yeah. This is kind of a newer feature and we’ve started using it in the last few months. I find that this is a pretty useful view to kind of get a view how your network is performing out in the field and if you do have failing endpoints, if they’re failing all the time or part of the time, if you have a lot of latency in certain endpoints that might be causing a weird user experience, we had a ton of broken image URL on some products so we were able to use this information to alert the e-commerce team of that. It gives you just a snapshot into your network traffic and how it’s doing. It has a nice interface in terms of you can filter by carrier so you can now if certain carriers are having problems with your network or you can filter by a particular endpoint and like which base endpoint meaning like maybe it’s our ATG catalogue for instance. We can drill down all the endpoints from that particular domain for instance.Josh: What kind of services are you relying on? You mentioned earlier your gamification aspects such as Foursquare and Twitter but some external services and social media type interactions, what are some of the things that your apps are relying upon internal and external?Chris: We have obviously quite a few custom services which we run internally and then we also use ATG as our platform for e-commerce and then we’re also integrating with Twitter’s API. We integrated with Foursquare. We integrated with Instagram and we use this vendor called Bunchball. It has its Nitro API for our gamification sort of rules engine. I think, Jason, am I forgetting any?Jason: We’re introducing a couple of others. We have a store’s API that gets us all of our data for all of our stores.Josh: It’s kind of like all included in the internal. Any other external?Jason: External, no other than Foursquare and Twitter and coming up to Instagram, future most likely Facebook.Chris: Yeah. Facebook is one that’s on our radar.Josh: Got it. I appreciate that. I think that sort of lines up with what we’re seeing. The averages tend to be six, seven, eight different API integrations per app so definitely, I think you guys are on the top end of that spectrum.
- Chris: Cool. This is just like kind of more filter in here. You can see …Josh: You’re mentioning how you would kind of use different drill downs to take a look …Chris: Yeah. Here’s an example of an image URL. It looks like in this particular example, it’s really intermittently failing or maybe app servers that are out of sync and maybe it will expose a load balancer kind of related issue.Josh: Again, putting in the context of the end user, what they would be seeing is potentially just a blank image on a screen and you guys might not have had awareness of that in the past so it helps kind of shine a microscope on those things that might have been a little blind.Chris: Yeah. That wouldn’t trigger a crash most likely and it also most likely wouldn’t be a handled exception either so this gives you a nice view of that.Josh: Great.
- Chris: The breadcrumbs are a way for us to sort of like what I was explaining earlier when we’re trying to get the store associates to explain what they’re doing, this gives us a way to mark different aspects of the application like in every view or anything that can really get loaded in order so you can see the path through the application through the crash so you can very easily see when a user we know was at the home and then they went to shopping and then they went to the category and then they went to whatever and then they went to checkout and then it crashed. You can use that information to reproduce the crash most likely and then maybe set some break points in in a snapshot build. Do you want to talk to it, Jason?Jason: Yeah. What’s also really great is there’s been a few times where we’ve used it and it actually logs date and time for each breadcrumb as well which is excellent because we know that some of our crashes happen if the user within the app last week and they’re on a particular product view let’s say and they come back one week later, two weeks later, how many days later and maybe by that point, that product is sold out but they tried to add it to their cart and we realized that it’s crashing because that product was there last week. They were on in the app and since the app’s stateful unlike a website where a website would refresh when they came back. The app won’t so then we’re seeing crashes. We actually identified not only what happened in what order but also the time in which it takes to lead events to occur.Josh: Interesting. I’m going to try and sort of simplify essentially what is going on as this allows you to follow the user flow surrounding some kind of crash or poor user experience. It gives you kind of a trail of what the user was doing to get to that point so that’s especially how you’re using this.Chris: Yes, exactly. It’s almost as if we’re watching the users use the app so we can kind of see the steps to reproduce if you will. Josh: Perfect. I like that example of the shopping cart. That’s exactly a kind of tricky thing you wouldn’t know. It’s a very specific scenario and would become extremely hard to kind of duplicate that or even know what was going on. Did you have any way of looking at that before?Chris: [Crosstalk 00:35:43] I’m sorry. What did you say Josh?Josh: How would you approximate that in the past like before Crittercism? What other ways you would have approached that?Jason: Not really. It didn’t even occur to us that products were essentially being sold out while the user’s app was in the background and then they were coming back to it. That didn’t even occur to us until we saw it and then we’re wondering why these crashes would happen and I looked at the actual breadcrumbs. I said, “Wait a minute. This person put this in their cart in September and it was a sale item but obviously sold out now. I can’t even find that product on the web anymore.” That’s how it helped us get to that point.Chris: Yes. You might just handle that in the quality controller that would have opened the product detail and not allow them to get into that state had the product been sold out when they tried to drag there but once they’re already there, if you open up the app and then you might, say, make a network call on [a good 00:36:41] show or whatever and then you would call to get that data again. If you got nulled back and you weren’t checking for that, the app would crash so that’s kind of the scenario they would be in.Josh: In the context of the end user, we’re talking fairly technical right now from your perspective but in the end user, if something like that happens, how do they get the info back to you? How do you know that there are issues occurring? Is it through the app store reviews? Is it through … how do you …Jason: Yeah. One star app review, there you go. Chris: Yeah. We also have a call center so we do get calls when we have maybe like an outage or there’s some unexpected event that happened to the user so that’s typically an expensive form of feedback.Jason: Those are also [inaudible 00:37:33] too. If that happens to a handful of people, we usually won’t even hear about that feedback until it comes in at volume but if we get one call about that, the call center is not going to call our team and say, “Hey, we got this one singular call. Let’s talk about this anomaly,” but we don’t get that until our crash rate spikes and then it’s agreed, “Now, why is that happening?” We have to wait for enough users to be affected essentially.Chris: Typically, that feedback comes on the weekend. They really know it has happen.Josh: Exactly, right. I know some of our customers, I don’t know if you guys are doing this but some folks will actually give an interface to Crittercism support team so that they can actually potentially reproduce some of the particular issues using breadcrumbs or something like that to kind of rewind the issue potentially but that’s exactly the kind of challenge here is just the lack of visibility into the ... cool. We’ll see. I don’t remember if we have another slide. I think we transitioned at the benefits.
- Josh: We went through four different scenarios of how you guys are using the tool. There are obviously different ways you can use it and where it potentially can help out your process, your development and the ability to sort of deploy these apps so talk to us a little bit about where it’s added value for you guys.Chris: We’ve been talking about it for a while, all these things. We have much lower crash rates in the application. We’re able to be much more proactive about crashes and it’s integrated in our development flow. Our ability to deliver more fixes and higher quality code is fundamental to using the tool and we have a lot of cool proximities that we work. I don’t know. I mean the previous slides really speak to themselves.Josh: I really appreciate that. Just getting a view in how you’re using the tool. It’s an incredibly complicated ecosystem now between different carriers, the number of different platforms you have to deploy on, the different wireless connectivity options a customer has, app versions, all of that. It leads to incredibly complicated ecosystem trying to navigate it.
- Josh: Let me talk a minute so excuse just a little bit. I have just a couple of slides specific to what you guys have been talking about and what our solutions are and then we’ll just take a few minutes to look at some of the questions. There have been a few rolling in here so after a couple of slides, I’m going to take a look at those questions that the folks watching this have mostly for you guys and then we’ll wrap things up.You heard Chris and Jason talked a bit about the platform that they’re using from Crittercism to help monitor and improve the performance of their app. There are essentially three pieces to the mobile application performance, monitoring, solution that is being used here so there is a platform underpinning everything that provides a scalable and reliable big data computing storage platform and that’s one on the cloud. On top of that, we have two components, one that focuses more on the fatal and non-fatal errors around crashes so this is things around error, error monitoring, reporting, troubleshooting along the lines of app stability and quality. Then we have another aspect that’s more aligned around the server and network aspects and carrier aspects of the app so as it relates to cloud and network performance so those two components are built in the platform and that’s essentially what Jason and Chris were talking about in the Crittercism solution.
- Josh: We’re used by a lot of different customers including Urban Outfitters, folks like eBay and PayPal and LinkedIn, media folks, Hearst, Yahoo, NetFlix and then also Fortune 500 customers like Home Depot and Nike and Lowe’s. All are using Crittercism in ways similar to what you heard Chris and Jason talking about both for customer facing as well as employee facing mobile applications and how you can deliver faster and better and smarter apps along those lines.
- Josh: Let me take a look at the questions. We’ll do a few of these while we can so let me take a look at these. I got actually a couple of compliments on Crittercism so I appreciate that. Can you talk a little bit about the network monitoring side of things and how that works from an iOS perspective? How does that line up against information you gather on the device itself versus what you can gather on the server-side? Can you guys maybe talk a little bit about that and maybe you need to into a bit about how your ops and dev team are structured. Chris, Jason, could you tackle that?Chris: I’m sorry. I haven’t quite understood the question. How …Josh: Sure. Let me try it again. I’m trying to sort of paraphrase the question here. Essentially the question is around how do you do network monitoring from the perspective of the device itself? I mean I think people are aware that there are other tools for server-side so how are you using this in more of the client side perspective?Chris: We have a couple of different server-side monitoring pieces. We can’t really use Crittercism for that. The best we can do is use Crittercism for the in-app stuff and actually for some of our in-house services, our app is actually the only thing that consumes those services so we’re really using the old one, actually looking at it. Our DevOps team is a lot more concerned with our e-commerce stuff and our larger customer databases so as long as that’s working, for them the mobile [inaudible 00:44:14] isn’t nearly as important and that ... I t’s also probably because we keep a rather close eye on it so really what use Crittercism for, we end up actually kind of being the network monitor for the services. Like I said, no one else here really uses those services except for us and our app.Jason: It’s quite easy to integrate with the Crittercism network monitoring. You just basically include it just like you normally would and you’ll just get it so there’s no special coding for it or anything like that. It just snoops in on your network calls and we get the data.Josh: Got it. Then there was a question more, I don’t know if you guys can answer this but feel free just to say, “We can’t take this one,” but I’ll throw it out there. The question was a little bit about what’s the difference between your mobile app strategies across the different brands? Do you have different brands and the other strategy is a little bit different for each of those whether it’s Urban Outfitters or Free People or BHLDN?Chris: Can you give us a second on that one?Josh: Yeah, sure. While they’re thinking about that, obviously I have a slide up here. Certainly if you folks have questions more specific to Crittercism itself, there’s a way to get hold of us so just send us a note at sales@crittercism and we’ll certainly try and answer those after the webinar again. If you’re interested in a trial of this service, you can certainly sign up at our website.Chris: To answer the one about the strategy, the brands have different demographics so really the main strategy is to deliver the most unique experience for that demographic and that’s why we have UX specialists and PMs that really research their particular customer and really know what they should show when, what features are important, prioritizing what we built but in terms of technology, we try to standardize parts as much as possible like you wouldn’t know it but under the covers, the app that we would deliver for Anthropologie may use some of the same components even if you won’t see it. It’s using some of the same codes so that if we do develop a feature that’s new for Anthropologie, we can just pull it into the UO code base and that’s sort of gives us some extra velocity.Jason: It allows the brands to share knowledge too because having each brand have their own technology stack and working on silo, they get to share, “Oh, this is how we use Crittercism to build our report. This is how we use this tool.” It actually requires a lot less on the mobile team to inform and train the brand people using this. You can just say, “Hey, how about you sit with this person at the other brand who’s using it in a unique way and they can share that experience with you?”Chris: I guess beyond technology, it’s also a methodology as well.Josh: Interesting. It kind of gives you a common language for discussion.Chris: Yes.Josh: Perfect. I’m going to ask you not specific to the solutions but kind of any, and again, if this is something you can’t answer, feel free to say so but given we’re getting near the holiday season, are you guys thinking differently about things, preparations you’re making, does the tool fit into that in any way?Chris: Yeah. The tool is going to be in there for sure. We’re not taking it out for the holidays but we do have a standard code freeze here so typically we’ll do a code freeze before the holiday shopping season starts which is coming right up here. It may vary by a week or two depending on the product but for the most part, early November I’d say would be the latest we could ship an app and then we kind of code freeze through the rest of the holiday but we’ll continue to work on new stuff. It’s just we won’t be able to ship anything to the customer but we’re obviously still monitoring the reports of what’s in the field and creating tickets and sort of addressing things as we go. I guess you could make a special exception if there was a massive sort of failure that all of a sudden we’re noticing and maybe there was a back end change that caused us to have to do something and do a delivery but for the most part, no, we wouldn’t ship any apps during that time.Jason: At the same time too, we’re using these weeks and a little over a month now, leading up to that code freeze to really narrow down on our most significant process that we’re seeing and try to tighten up for one more build to get out there so that way we can put out the most stable products that we get comfortable with before shopping season hits.Chris: Yeah. We actually did a couple of sales recently which allowed us to simulate some higher volume and zero in on the most prominent issues.Josh: Awesome. I really appreciate you taking into that. I’m sure there are a number of other folks that share the type of challenge you’re up against with the holiday season approaching.Well guys, we’re getting to the top of the hour. First of all, I really wanted to thank both of you for taking the time to share your experience and your knowledge with everyone on the line. It’s very interesting for me to hear it and I’m sure folks joining us appreciate your perspective on that so thank you very much.Also, there are a couple of questions that we don’t have time to get to. We’ll try to reach out to folks separately after the webinar with the ones that we couldn’t get to so I do appreciate everyone taking the time to send in some questions and sorry we didn’t have enough time to get to them.With that, certainly like I mentioned, we have information on the last slide here so please feel free to reach out to us or try a product trial at crittercism.com and again, Chris, Jason, thank you so much for your time and for everyone joining us, I appreciate you taking the time to join us today. Thank you very much.Chris: Thanks, Josh. Jason: All right, have a good one.