Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (19)
Semelhante a Serverless testing-serverless-sydney-20181018
Semelhante a Serverless testing-serverless-sydney-20181018 (20)
Último
Último (20)
Serverless testing-serverless-sydney-20181018
- 1. Sydney Serverless Meetup October 18th 2018
- 2. Serverless Testing Rowan Udell, October 18th 2018
- 3. What I Wish Everyone Knew About Serverless Testing (Including Past Rowan) Rowan Udell, October 18th 2018
- 5. Why test?
- 6. Why test? • Feedback loop • Discover problems (bugs) before your users do Don’t make your users do your testing • Time • It’s good for writing software locally It’s even better in the cloud
- 7. Why test serverless? • Running on someone else’s computer • Blurred lines between code and infrastructure • I’ll have two DevOps please Are you more Dev or more Ops? • Limited surface area • Scale • Cost
- 8. Test ROI • The most important thing! • Some things that impact ROI: • Speed • Brittleness • Complexity • Maybe don’t write tests? Heresy! *BLAM* www.bennorthrop.com
- 9. Working Effectively with Unit Tests wewut.com
- 11. Unit Tests • White box • Many • Small • Fast! Serverless • Isolate functionality • Local (code)
- 12. Example: Naive
- 13. Example: Better
- 14. Integration Tests • Higher level than unit tests • Cross responsibility boundaries Serverless • Harder, but also more valuable • Per request • Mocks?
- 15. Don’t mock what you don’t own • Don’t mock the cloud You’ll hurt it’s feelings • Don’t build the cloud locally • Beware brittle tests!
- 16. Integration Example • SAM CLI Includes API GW • Dynalite https://github.com/mhart/dynalite Not the official DDB version • Pact https://docs.pact.io/ It’s not a mock
- 17. E2E Tests • aka UI Tests • Black box • Lots of things can cause failures Limit exposure to external systems • Non-functional tests Serverless • Do it in production Only production is production
- 18. E2E Examples • Automated user testing • Monitoring services* • Test environments • Code paths Warming functions
- 19. Manual Testing • Not part of the pyramid?! • You already doing it • Inconsistent • Doesn’t scale • Don’t forget the ROI!
- 20. Testing is hard! …but you should still do it Testing is a skill Serverless is just software (with different boundaries)
Notas do Editor
- <Versent introduction and sponsorship blurb> TBC AWS only, AWS Premier, awards, etc We’re hiring! AWS focused talk, because Versent.
- So first I want to go in to a bit of “why test?” in generally, and then specifically for serverless. If you’re a software developer, hopefully this is not a tough sell. If you’re not familiar with the benefits of testing software, hopefully I can convince you to consider it. Even if you’re an experienced developer, I think we can sometimes benefit from being reminded these things, so hopefully you get something from this talk too. Don’t test just because some guy at a meetup says you should. Do it because it makes sense.
- Developer feedback is one of the main reason “A stitch in time saves 9” Time is the most important reason to test, and one of the more counter-intuitive ones. The reality is tests are extra code, and extra code take extra time to write. You also have to maintain that extra code. The trick is that (hopefully) little bit of extra code you write makes it exponentially easer to maintain the rest of your code It results in a faster development speed over the lifetime of your codebase. “Time is money.” By ”better” I mean “more valuable” It’s much easier to see and quantify the value.
- At the end of the day your functions are running on someone else’s computer – that’s the whole point!!!!1! This has several implications… Many people experimenting with serverless these days don’t necessarily have a software development background. There are no more “sysadmins” only “devops”. That’s a good thing, but I think there are some fundamental lessons from software development that can be applied to operations in a infrastructure as code world. There’s only a limited surface area you can test, so I think it becomes more important – and valuable – to do so. Your code can go from your machine to all of these locations in a matter of minutes. I could’ve included a much more detailed diagram of locations, but I don’t want to annoy AWS, like I said we’re AWS only, so that would bad idea! That’s amazing – who would’ve thought it just 5 years ago - but also scary. This power means it’s even more important As I tell our customers: if you automate without testing, all you’ve done is enabled yourself to fail really really fast. There are plenty of horror stories out there of people creating event loops and racking up major bills. Every execution (theoretically) costs you money, and I don’t know about you but I’d rather not pay for my bugs to run. I just don’t like stack traces that much. This is one of the reasons why the Lambda free tier makes so much sense; It lets you develop and test you application for free. Developer time is going to be the biggest cost of almost any application, but particularly a serverless one! Because you only pay for the actual running time of your application, many applications cost orders of magnitude less to run when serverless, which makes the developer cost (as a proportion of total cost) even higher.
- I wish more people spoke and thought with the term “Test ROI” in mind. If you take nothing else from this talk, take this point. This is the idea that the tests you write must have good return on investment. ”Investment” means a number of things: the time it takes to write the tests, the time it takes to run them, and the amount of maintenance they require. We often talk about avoiding “brittle” tests, and that’s because brittle or fragile tests cost more to maintain, reducing their ROI. Test speed is another thing that matters a lot, and effects the ROI – as anyone who’s spent their days waiting for a slow test suite to return. ROI happens at different levels: at a specific test level (which we’ll go in to with the pyramid), but also at an application level. This idea is also how I justify not writing tests (gasp! Heresy!). If your application is just a PoC and is definitely going to get thrown out, then writing tests that you know will get thrown out probably doesn’t make much sense. The probable is always when a PoC makes it way in to production… This is just one of the reasons I think you should always throw out the first version of an application, but that probably needs a different talk to cover off properly.
- Credit to this book for introducing me to test ROI as a concept. Highly recommended if you’re after a practical guide. It’s python based, but the ideas are applicable. It’s also been a while since I read it, so if I misquote something that’s my fault.
- Hopefully you’ve all seen this diagram before. The test pyramid is a nice visual way to communicate a good approach to writing tests, which compliments the idea of testing ROI nicely. I stole this image from that website because it was the nicest test pyramid I could find out there – most of them look absolutely horrible, with strange gradients, etc. Seriously, if you want to see some horrible clipart do a Google Image Search for the testing pyramid.
- TBC Pyramid Called “white box” because they interact directly with your code. You almost (almost!) can’t have too many. They shouldn’t be very big, hence “unit”. Test one thing, and one thing only – rarely should you have more than one assertion. I’m not going to give you a hard number here, but they shouldn’t be much over a single-digit number of lines. Long Unit Tests are a code smell - there better be a good (and obvious) reason for them. They need to be fast. If you’re going to have lots of them, it makes sense that you wouldn’t want to be waiting around for them to run. In the case of serverless, in order to keep them fast you will need to run them locally. Note that I’m not talking about running them in a container locally (we’ll get to that). These should just be pure code execution. A well written will be isolated. They should test one thing, and test it well. Avoid “cascading failures” – if too many unit tests fail at once, then your tests are probably overlapping (makes maintenance harder). I like to think in terms of “happy path” and “sad path” – testing positive and negative scenarios. This is often where I see developers (or operations) struggling – you will need to change how you might intuitively write your code. Basically, this is how you should be writing unit tests anyway, so keep doing that. All this to say they can be relatively low value, because they are low cost (to maintain, run, etc), which means they still have a good testing ROI.
- When I talk about ”pure code” I mean something without any external dependencies
- As the name suggests they test the integration of components. This is one of the areas where serverless gets difficult – the things you’re integrating with are not part of your application any more (e.g. storage). Where you draw the line is up to you, but I usually aim for the endpoint level in a serverless API. I find integration tests to be harder to write in a serverless world, but also more valuable (otherwise I wouldn’t do them). This is because you’re inherently integration with external systems. Inherently stateless. Want to store some state? That needs an external system. Unlikely to be able to make assumptions about the environment. In practice, I generally scope integration tests down to specific requests. This means it doesn’t include things like complicated application flows, they go to E2E The difference between integration testing and E2E is a blurred one in serverless, so if you’ve got a good (simple) rule of differentiating then I’d like to hear it.
- TBC Sad cloud Some teams have been using localstack with success, so you might use it. I tried to do it a while back, and got burnt enough that I’ve avoided it since. Mocking other people’s code is one of the most brittle things you can do!
- Pact is an example of an approach I like. You’re not mocking the downstream system, you’re just calling out explicitly what you expect the response to be. If the response does not match, it’s obvious where the issue is. This makes isolating the cause of test failures as easy as possible
- TBC Neddy No No Picture
- TBC Pyramid Sometimes called “UI Tests” Usually “black box” tests, because they don’t care (or know) about the code being tested – they happen at a higher level. These tests are designed to be sensitive to failures anywhere in your system. This might include external systems, which can be really annoying to be notified about, but are still valuable. Ideally you would limit your exposure to those external systems (so you don’t just turn your tests in to a monitoring tool for their system), and as apriority you would want to make it clear what the cause of the test failure is. Remember that the aim of testing is not just to have passing tests (which are nice) – it’s about finding problems before your users do. I said “limit” because it’s likely you can avoid some exposure, just keep in mind testing ROI (which might mean you remove flakey tests). Limit false negatives where possible (but it’s probably impossible). Avoid brittle tests, which will probably take more time up front. E2E tests are also where I would include non-functional tests, but obviously you could include non-functional testing at the other test levels of the pyramid. I’m just not convinced that there’s the right testing ROI for non-functional tests at those levels. We’re testing for user experience, not testing’s sake. If there is, it’s unlikely to be one of the first things you do with an application, maybe you’d come back and add them for a particularly important or mature application (which are usually the same thing!). Especially in a serverless world, where scaling is not usually an issue. Finally, I think you should be running these test in production! This might have been a bit controversial just a few years back, but I’m seeing the idea gain traction and acceptance. Even in a cloud environment where I can reproduce my entire production stack (using infrastructure as code) in a matter of minutes, it’s still not easy (or maybe possible) to reproduce the exact data (which might have compliance issues), load, and behaviour. Who really knows what users are going to do to your system!? TBC Example scenario Testing in production: You already do it Charity Majors: “Only production is production”
- One way of doing this is by creating specific code paths for testing in your application. I see a bit of this out there, but not a whole lot.
- Even if you don’t have automated tests, you’re still doing manual testing. No code was written without some manual testing. Just keep in mind the ROI.