ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx

•Transferir como PPTX, PDF•

0 gostou•8 visualizações

Slides used by Alexander Perathoner in his presentation of the paper "On the Impact of Hitting System Resource Limits on Test Flakiness". Presentation took place on 14th April 2024 at the International Flaky Test Workshop, co-located at the International Conference on Software Engineering (ICSE 24) in Lisbon. The paper was written by F. Leinen, A. Perathoner and Prof. Dr. A. Pretschner.

Apresentações e oratória

On the Impact of Hitting System
Resource Limits on Test Flakiness
FTW 2024
April 14, 2024
Fabian Leinen Alexander Perathoner Alexander
Pretschner Technical University Munich

Motivation
Intuition: “PC under load provokes lags and flakiness”
Study Gap: resources are not stressed, just measured
Hypothesis: “Reaching CPU limits promotes timing-related
flakiness”
2

Approach: UI Tests & macOS
Weaker Resources
More expensive
3

Does Hitting Natural Resource Limits Impact Flakiness?
We measure ratio of:
- time at resource limit
- total runtime
in executions of flaky / non-flaky tests
5

Does Hitting Natural Resource Limits Impact Flakiness?
We measure ratio of:
- time at resource limit
- total runtime
in passing / failing executions of flaky tests
6

Gathering Data
select:repo
(type:file content:XCUIApplication lang:Swift)
(repo:has.tag(macos) OR
repo:has.tag(osx) OR
repo:has.topic(macOS))
236 repositories
7

236 repositories
Manual filtering
20 projects
Gathering Data
6.5k
Sparkle
8.1k
Maccy
8

tum-i4/workflow-telemetry-action
measuring:
CPU, RAM, network, disk
GitHub
Workflow
Gathering Data in the GitHub CI
9

Does Hitting Natural Resource Limits Impact Flakiness?
13

Does Hitting Natural Resource Limits Impact Flakiness?
14

Result Discussion
Alternative explanations for RQ1:
- Flakiness linked to complexity of tests (size) rather
than resource limits?
No quantitative explanations for RQ2
15

What to Take Away From This
We analyzed the impact of resource limits:
1. on tests being flaky → they do spend more time at
limits
2. on flaky failures → they do not
16
Upgrade the server to boost stability
But don't expect it to magically ace tests

Call for Action
● More projects!
● More resources!
● Less overhead!
Remember the GitHub Action!
17

Mais conteúdo relacionado

Semelhante a ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx

Advances in genomics and data analytics create new opportunities for cancer research and personalized medical treatment via large-scale federation of genomic, clinical, imaging and other data from many thousands of patients across institutions around the world. Despite these opportunities and promising early results, cancer research is often stymied by information technology barriers. One major barrier is a lack of tools for the reliable, secure, rapid, and easy transfer, sharing, and management of large collections of human data. In the absence of such tools, security and performance concerns often prevent sharing altogether or force researchers to resort to slow and error prone shipping of physical media. If data are received, timely analysis is further impeded by the difficulties inherent in verifying data integrity and managing who can access data and for what purpose. I will discuss how the mature Globus data management platform addresses these obstacles to discovery and explain how its intuitive, web-based interfaces enable use by researchers without specialized IT knowledge. I also describe how Globus technologies can be extended to meet the security requirements of human data so as to enable use in data-intensive cancer research.

Streamlined data sharing and analysis to accelerate cancer research

Ian Foster

The proliferation of mobile devices in the form of smartphones and tablet devices has put into our hands computational power and capability previously limited to desktop until recently. Couple this with the connectivity of these devices to the internet and the trend of increased capability and accessibility in smaller devices continues. This presentation will provide an overview of our efforts to provide access on mobile devices to a large chemistry database, ChemSpider, containing over 25 million unique chemical compounds and associated data including patents, publications, properties and analytical data. We will also discuss how, by providing programming interfaces and Open Data, it has been possible to produce a Spectral Game (www.spectralgame.com) for scientists and students to practice their spectral interpretation skills. We will discuss some of the technology hurdles associated with delivering such capabilities to the various mobile platforms and how modern technologies can significantly enhance the user experience.

Chemistry in the hand: The delivery of structure databases and spectroscopy g...

US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure

Among Viruses, Trojans, and Backdoors:Fighting Malware in 2022

Marcus Botacin

On the Malware Detection Problem: Challenges & Novel Approaches

Marcus Botacin

"Data Provenance: Principles and Why it matters for BioMedical Applications"

Pinar Alper

Break -> inspect -> fix is the Ops workflow for infrastructure stacks of the past. Distributed infrastructure and applications claim to be the new generation, but why is it so much more painful to maintain and troubleshoot them? Much of the pain comes from outdated operational models relying on reactive or, worse yet, manual monitoring and Ops. This talk lays out a proactive Ops model for container infrastructure. By focusing on event monitoring, infrastructure state monitoring, trend analysis, and distributed log collection, a proactive Ops model delivers observability for distributed apps that was not possible before. Using real-world examples from Swarm and Kubernetes, we'll demonstrate the tools used and how we relieve Ops pain in container orchestration.

Proactive ops for container orchestration environments

Docker, Inc.

Belak_ICME_June02015

Jim Belak

OpenPOWER Academia and Research team's webinar - Presentations from Oak Ridg...

Ganesan Narayanasamy

[A talk presented at Oak Ridge National Laboratory on October 15, 2015] We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In big-science projects in high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to develop suites of science services to which researchers can dispatch mundane but time-consuming tasks, and thus to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers. I use examples from Globus and other projects to demonstrate what can be achieved.

Accelerating Discovery via Science Services

Ian Foster

Domains such as drug discovery, data science, and policy studies increasing rely on the combination of complex analysis pipelines with integrated data sources to come to conclusions. A key question then arises is what are these conclusions based upon? Thus, there is a tension between integrating data for analysis and understanding where that data comes from (its provenance). In this talk, I describe recent work that is attempting to facilitate transparency by combining provenance tracked within databases with the data integration and analytics pipelines that feed them. I discuss this with respect to use cases from public policy as well as drug discovery. Given at: http://ccct.uva.nl/content/ccct-seminar-21-february-2014

Transparency in the Data Supply Chain

Paul Groth

Accelerating Data-driven Discovery in Energy Science

Ian Foster

Performance Analysis of Idle Programs

greenwop

PhD public defense: A Measurement Framework for Analyzing Technical Lag in ...

Ahmed Zerouali

Windows Live Forensics 101

Arpan Raval

Containers in Science: neuroimaging use cases

Krzysztof Gorgolewski

Current Topics paper A4 submission 4.30.2015 Master Copy

Tommie Walls

Practical Chaos Engineering

SIGHUP

Virtual Gov Day - IT Operations Breakout - Jennifer Green, R&D Scientist, Los...

Splunk

Stream based data / event / message processing becomes preferred way of achieving interoperability and real-time communication in distributed SOA / microservice / database architectures. Beside lambdas, Java 8 introduced two new APIs explicitly dealing with stream data processing: - Stream - which is PULL-based and easily parallelizable; - CompletableFuture / CompletionStage - which allow composition of PUSH-based, non-blocking, asynchronous data processing pipelines. Java 9 will provide further support for stream-based data-processing by extending the CompletableFuture with additional functionality – support for delays and timeouts, better support for subclassing, and new utility methods. More, Java 9 provides new java.util.concurrent.Flow API implementing Reactive Streams specification that enables reactive programming and interoperability with libraries like Reactor, RxJava, RabbitMQ, Vert.x, Ratpack, and Akka. The presentation will discuss the novelties in Java 8 and Java 9 supporting stream data processing, describing the APIs, models and practical details of asynchronous pipeline implementation, error handling, multithreaded execution, asyncronous REST service implementation, interoperability with existing libraries. There are provided demo examples (code on GitHub) using Completable Future and Flow with: - JAX-RS 2.1 AsyncResponse, and more importantly unit-testing the async REST service method implementations; - CDI 2.0 asynchronous observers (fireAsync / @ObservesAsync);

Stream Processing with CompletableFuture and Flow in Java 9

Trayan Iliev

Cloud Resilience with Open Stack

Jorge Cardoso

Semelhante a ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx (20)

Streamlined data sharing and analysis to accelerate cancer research

Chemistry in the hand: The delivery of structure databases and spectroscopy g...

Among Viruses, Trojans, and Backdoors:Fighting Malware in 2022

On the Malware Detection Problem: Challenges & Novel Approaches

"Data Provenance: Principles and Why it matters for BioMedical Applications"

Proactive ops for container orchestration environments

Belak_ICME_June02015

OpenPOWER Academia and Research team's webinar - Presentations from Oak Ridg...

Accelerating Discovery via Science Services

Transparency in the Data Supply Chain

Accelerating Data-driven Discovery in Energy Science

Performance Analysis of Idle Programs

PhD public defense: A Measurement Framework for Analyzing Technical Lag in ...

Windows Live Forensics 101

Containers in Science: neuroimaging use cases

Current Topics paper A4 submission 4.30.2015 Master Copy

Practical Chaos Engineering

Virtual Gov Day - IT Operations Breakout - Jennifer Green, R&D Scientist, Los...

Stream Processing with CompletableFuture and Flow in Java 9

Cloud Resilience with Open Stack

Último

Getting started with Amazon Bedrock Studio and Control Tower

Vladimir Samoylov

Pollinator Ambassador Earth Steward Day Presentation 2024-05-22

LHelferty

Hi-Tech Industry 2024-25 Prospective.pptx

ShivamM16

Sharpen existing tools or get a new toolbox? Contemporary cluster initiatives...

Orkestra

• For a full set of 420+ questions. Go to https://skillcertpro.com/product/oracle-database-administration-i-1z0-082-exam-questions/ • SkillCertPro offers detailed explanations to each question which helps to understand the concepts better. • It is recommended to score above 85% in SkillCertPro exams before attempting a real exam. • SkillCertPro updates exam questions every 2 weeks. • You will get life time access and life time free updates • SkillCertPro assures 100% pass guarantee in first attempt.

Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf

SkillCertProExams

A proposal aimed at transforming a key site in Canoga Park into a vibrant, mixed-use development. The project includes comprehensive location analysis, neighborhood context evaluation, and zoning and planning assessments. By leveraging the site’s strengths and addressing potential challenges, we plan to create a space that integrates residential, commercial, and public areas. The development will enhance connectivity, support community objectives, and provide economic benefits, all while aligning with local zoning regulations.

The Canoga Gardens Development Project. PDF

Rahsaan L. Browne

123445566544333222333444dxcvbcvcvharsh.pptx

gargh1099

05232024 Joint Meeting - Community Networking

Michael Orias

Breathing in New Life_ Part 3 05 22 2024.pptx

FamilyWorshipCenterD

527598851-ppc-due-to-various-govt-policies.pdf

rajpreetkaur75080

Acorn Recovery: Restore IT infra within minutes

IP ServerOne

Have you ever wondered how search works while visiting an e-commerce site, internal website, or searching through other types of online resources? Look no further than this informative session on the ways that taxonomies help end-users navigate the internet! Hear from taxonomists and other information professionals who have first-hand experience creating and working with taxonomies that aid in navigation, search, and discovery across a range of disciplines.

Eureka, I found it! - Special Libraries Association 2021 Presentation

Access Innovations, Inc.

Bridging the Divide: Enhancing Public Engagement in Urban Development This paper delves into the critical role of public engagement in urban development, emphasizing the need for community involvement to create sustainable and inclusive urban spaces. It discusses the motivations behind public participation, the challenges faced in engaging diverse communities, and the strategies for overcoming these barriers. The paper also highlights successful case studies and explores the use of technology and effective communication to facilitate broader and more meaningful engagement. By examining these elements, the paper underscores the importance of bridging the gap between planners and residents to ensure urban development reflects the collective vision and needs of the community.

Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...

Rahsaan L. Browne

What is biology? In simple terms, biology is the study of living organisms and their interactions with one another and their environments. This is a very broad definition because the scope of biology is vast. Biologists may study anything from the microscopic or submicroscopic view of a cell to ecosystems and the whole living planet . Listening to the daily news, you will quickly realize how many aspects of biology are discussed every day. For example, recent news topics include Escherichia coli outbreaks in spinach and Salmonella contamination in peanut butter. Other subjects include efforts toward finding a cure for AIDS, Alzheimer’s disease, and cancer. On a global scale, many researchers are committed to finding ways to protect the planet, solve environmental issues, and reduce the effects of climate change. All of these diverse endeavors are related to different facets of the discipline of biology. Biology is a science, but what exactly is science? What does the study of biology share with other scientific disciplines? Science (from the Latin scientia, meaning “knowledge”) can be defined as knowledge that covers general truths or the operation of general laws, especially when acquired and tested by the scientific method. It becomes clear from this definition that the application of the scientific method plays a major role in science. The scientific method is a method of research with defined steps that include experiments and careful observation. The steps of the scientific method will be examined in detail later, but one of the most important aspects of this method is the testing of hypotheses by means of repeatable experiments. A hypothesis is a suggested explanation for an event, which can be tested. Although using the scientific method is inherent to science, it is inadequate in determining what science is. This is because it is relatively easy to apply the scientific method to disciplines such as physics and chemistry, but when it comes to disciplines like archaeology, psychology, and geology, the scientific method becomes less applicable as it becomes more difficult to repeat experiments.

Introduction of Biology in living organisms

soumyapottola

ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx

1. On the Impact of Hitting System Resource Limits on Test Flakiness FTW 2024 April 14, 2024 Fabian Leinen Alexander Perathoner Alexander Pretschner Technical University Munich

2. Motivation Intuition: “PC under load provokes lags and flakiness” Study Gap: resources are not stressed, just measured Hypothesis: “Reaching CPU limits promotes timing-related flakiness” 2

3. Approach: UI Tests & macOS Weaker Resources More expensive 3

4. Identifying Potentially Flaky Tests 4

5. Does Hitting Natural Resource Limits Impact Flakiness? We measure ratio of: - time at resource limit - total runtime in executions of flaky / non-flaky tests 5

6. Does Hitting Natural Resource Limits Impact Flakiness? We measure ratio of: - time at resource limit - total runtime in passing / failing executions of flaky tests 6

7. Gathering Data select:repo (type:file content:XCUIApplication lang:Swift) (repo:has.tag(macos) OR repo:has.tag(osx) OR repo:has.topic(macOS)) 236 repositories 7

8. 236 repositories Manual filtering 20 projects Gathering Data 6.5k Sparkle 8.1k Maccy 8

9. tum-i4/workflow-telemetry-action measuring: CPU, RAM, network, disk GitHub Workflow Gathering Data in the GitHub CI 9

10. 10 Gathering Data in the GitHub CI

11. 11 Gathering Data in the GitHub CI

12. How Does the Dataset Look? 12

13. Does Hitting Natural Resource Limits Impact Flakiness? 13

14. Does Hitting Natural Resource Limits Impact Flakiness? 14

15. Result Discussion Alternative explanations for RQ1: - Flakiness linked to complexity of tests (size) rather than resource limits? No quantitative explanations for RQ2 15

16. What to Take Away From This We analyzed the impact of resource limits: 1. on tests being flaky → they do spend more time at limits 2. on flaky failures → they do not 16 Upgrade the server to boost stability But don't expect it to magically ace tests

17. Call for Action ● More projects! ● More resources! ● Less overhead! Remember the GitHub Action! 17

18. Root Causes of Flaky Tests 18

ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx

Semelhante a ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx (20)

Último

Último (14)

ICSE 24, FTW: On the Impact of Hitting System Resource Limits on Test Flakiness.pptx