Baixado 21 vezes
![-50%
0%
50%
100%
150%
% Growth in Active Accounts (by Services)
Build Active Accounts
ELS Active Accounts
AI Active Accounts
Monaco Active Websites
VSO Account Growth Over Time
Launch
MoM Growth in [sub]Service Adoption](https://image.slidesharecdn.com/abtesting-devtest-fy15-140811061550-phpapp01/85/ABTesting-Nao-ha-nada-como-testar-em-producao-37-320.jpg)
Carregado porDaniel Semedo
ABTesting: Não há nada como testar em produção!
O documento discute práticas ágeis de desenvolvimento de aplicações, incluindo padrões de implantação em produção como canary, A/B testing e blue-green. Também aborda o papel da equipe de QA, métricas de serviço, crescimento de contas, conversão de usuários e a importância de monitoramento e métricas para melhoria contínua.
Mais conteúdo relacionado
![[ServiceNow] Governança das Instâncias (4ª edição)](https://cdn.slidesharecdn.com/ss_thumbnails/2021-06-29servicenowgovernanadasinstncias-v4-210629151923-thumbnail.jpg?width=640&height=640&fit=bounds)
![[ServiceNow] Visão geral da plataforma](https://cdn.slidesharecdn.com/ss_thumbnails/2022-02-10servicenowvisogeraldaplataforma-220211002310-thumbnail.jpg?width=640&height=640&fit=bounds)
Semelhante a ABTesting: Não há nada como testar em produção!
ABTesting: Não há nada como testar em produção!
- 1. Daniel Semedo http://semedo.azurewebsites.net V-daseme@microsoft.com A/B Testing:Não há nada como testar em produção!!
- 2. *GARTNER Agenda Overviewfor Application Development, 2014, Ian Finley, Thomas E. Murphy, Jim Duggan, Mark Driver, Gordon Van Huizen, Nathan Wilson, David Mitchell Smith, Matthew Hotle, Joe Bugajski, Brian Prentice, Ray Valdes, Ross Altman, Richard Marshall, Van L. Baker, David Norton, 9 January 2014
- 3. “It has beensaid that the only sustainable advantage in business is the ability for a company to learn faster and respond more effectively than its competitors (also known as business agility).” – CIO magazine, 2012
- 8. Apps deNegócios estão em uma cadência ágil de entrega devido a competitividade. Impedimentos exarcebados já existentes. Deployment complexo devido a aplicações multi-plataforma. Complexidade em suportar diferentes dispositivos e garantir serviços always “on”. 40%Releases quadrimestrais ou menos frequentes Source: Continuous Delivery Is Reshaping The Future Of ALM, Forrester Research, Inc., July 22, 2013
- 9. Padrões de Deployem Produção
- 10.
- 11. Stop and Go– Multi Servers
- 12.
- 13.
- 14.
- 15. Blue-Green Após Aprovação StagingIP ganha status Prod Em Falha: Rollback é feito em Prod
- 16. Blue-Green Após Successo Rollback éreaberto como staging
- 21. Papel do QA notime de desenvolvimento do Visual Studio
- 22. 24 | Un-ManagingAgile Teams Times de 8-12 Multi-disciplinar 3 perfis distintos Estrutura
- 23. 25 | Un-ManagingAgile Teams Nossas Regras Program Manager – Responsável por assegurar que estamos construindo a coisa certa. Desenvolvimento – Responsável por assegurar que estamos construindo produtos que sejam rápidos, confiáveis e bem construidos. QA – Responsável por assegurar que estamos construindo um produto de alta qualidade e que atenda as expectativas e necessidades dos usuários.
- 24. 26 | Un-ManagingAgile Teams Estrutura Organizacional Program Manager Desenvolvimento QA
- 25. 27 | Un-ManagingAgile Teams Nossos Times Program Manager Desenvolvimento QA
- 26. 28 | Un-ManagingAgile Teams Nossos Times QADev QADev QADev QADevPM PM PM PM Collaboration Version Control Build Work Item Tracking
- 27. -RBT – Requirementsbased Tests em QA para garantia de regressão em novas features SLA em cada teste automatizado...
- 34.
- 36. Service Availability& OperationalMetrics • TFS switched to weekday deployments S61 • Build availability impacted by external dependencies (Networking/Service Bus). • App Insights Usage feature had multiple incidents due issues IaaS VM with SQL installed. Working to migrate to more resilient and operable design. • App Insights lacks impact telemetry for customer data flows which reduces time to detect, accuracy of root cause and limits ability to asses incident severity . • App Insights Usage currently supported by Dev team & working to migrate to Service Delivery for more consistent Live Site Management 38
- 37. -50% 0% 50% 100% 150% % Growth inActive Accounts (by Services) Build Active Accounts ELS Active Accounts AI Active Accounts Monaco Active Websites VSO Account Growth Over Time Launch MoM Growth in [sub]Service Adoption
- 38. Created an Account Createda Project Check-In / Push Build Deployed Alimentar/medir apenas a boca do funil pode ser uma métrica sem valor, pense em alimentar/medir toda a experiência do serviço, garantindo o resultado de uso completo. Pense em cada progressão e planeje a expansão da eficiência do funil Experimente! August September October November December January* Accounts Created Projects Created Code Checked-In Builds Executed Deployments to Azure August September October November December January* Account -> Project Project -> Check-In Check-In -> Build Build -> Deploy E2E Conversion
- 39. tinhamos… … Decidimos quepodiamos fazer melhor 30%Criavam novos Team Projects no primeiro acesso
- 40.
- 41.
- 42. Configure o dashboard com métricasque mais lhe interessam Identifique issues e comportamentos monitorando suas aplicações Verifique a “pulsação de suas aplicações” • Tenha uma visão 360 para monitorar e detectar issues em produção com Application Insights.
- 43. • Testes websão executados com frequência (5m-10m) para validar disponibilidade, performance e comportamento da aplicação em produção Global service monitoring
- 44. • Identifique errossistêmicos e de mal comportamento da aplicação para priorizar/prevenir falhas • Priorize melhorias, bugs e demais correções com métricas de uso da aplicação de forma quantitativa e qualitativa Dados valiosos para definir o futuro de suas aplicações Valide seus investimentos
- 50. • Identifique errossistêmicos e de mal comportamento da aplicação para priorizar/prevenir falhas • Priorize melhorias, bugs e demais correções com métricas de uso da aplicação de forma quantitativa e qualitativa Dados valiosos para definir o futuro de suas aplicações Valide seus investimentos
- 53. Resumindo Assuma o realpapel de QA Regra clara entre dev (qa) x ops Automação não é nada sem controle Priorize Métricas vitais para o negócio
- 55.
Notas do Editor
- #11 Today, every company is a software company, “whether you know it or not, you are almost certainly in the software business.” Software runs the critical infrastructure of nearly every modern business, and for the best companies, it provides a key competitive differentiator. As companies start embracing agility and taking advantage of the elasticity and economics of consuming cloud services the complexity increases, creating more points of failures and making more complicated to ensure those applications that run critical parts of your company and engage with your customer are optimally working and when they don’t you can restore a good state as soon as possible. So how do you increase responsiveness without creating unnecessary risk? How can you stay flexible and still keep a strategy and goals in view? How do you juggle agility with all of the other new demands on the success of your business? And ensure you continue investing in solutions that will give you a differentiator advantage?
- #20 Release Management Server provides valuable insights into the entire release process. Armed with relevant and timely information, managers can achieve greater efficiencies by monitoring for continuous improvement. For example, you can keep track of the trend in releases, the amount of traffic in the release pipeline, any possible bottlenecks, and even get performance insights.
- #35 Release Management Server provides granular control for the entire release process that includes many stakeholders, including Development, QA and Ops. You can limit which group can view, edit, or create a given release template. You can also decide who can edit approvers for each release stage. For example, the development group can modify the approvers in the Dev stage but cannot modify approvers for the Production stage. Another example is how you can limit which group can edit the configuration values for each release stage. For example, the development group can modify the values in the Dev and Integration stages but cannot modify Production. Finally, you can limit the stage types a given environment can support. For example, you can limit which production servers can be used.
- #39 Nov: restated # TFS tickets. Team did problem management and broke down general LSI tickets to specific events Availability model: Outside in views not shown due to noise in current tests. Outside of our control – customers network bandwidth, working towards measuring to the Time to First byte sent Vs., total time. Total time is usually bad if customer network bandwidth is poor AI Availability Model: Multi-step synthetic transaction test covers more surface area, includes more steps and is more sensitive to customer experience. 10 E2E scenario tests cover full surface area, new model aggregates availability across all these tests. Example sensitive measures: Single PoP node failures, data latency issues all count against availability model – these were not part of Gomez before. Availability model continues to be refined as new scenarios are uncovered.
- #40 ELS Active accounts based on 60 days activities are: Sep-13 to Jan 14: 194, 263, 343, 400, 459
- #42 For the success of features we’re investing in a “Build, Measure, Learn” culture. <talk about BML> Let me give you a recent example…
- #56 Time: 2 minutes Flow: We have had an end to end ALM solution for many years with Team Foundation Server on-premises which is built on a core set of capabilities to help development teams: Source control Agile planning tools Team rooms Test case management Feedback management and Automated builds Layered on top of that core ALM functionality, Team Foundation Server has the ability to Manage, deploy to, and test on environments with Lab Management Configure and manage releases with Release Manager Perform on premises load testing Integrate completely with SharePoint and customized SharePoint sites Integrate seamlessly with Project Server and System Center We are now introducing Visual Studio Online which builds on top of that core ALM functionality as well and provides additional capabilities and unique services like: Cloud Build Service Cloud Load Testing Service Application Insights Continuous deployment to Windows Azure Lightweight Code Editing Experience Microsoft’s ALM toolset is flexible and allows you to adopt a mixture of on-premises or cloud-based services. You can even use TFS on-premises and use services on Visual Studio Online that make sense to your team like the Load Testing Service and Application Insights. Transition to next slide: We’ll explore these services and more in depth throughout the rest of this presentation.