One of the main activities in software development is monitoring, which plays a vital role in verifying the proper implementation of processes, identifying errors, and discovering opportunities for improvement. Continuous Integration (CI) covers a set of widely adopted practices that enhance software development. However, there are indications that developers may not adequately monitor all CI practices. In this thesis, we delve deep into the ocean of monitoring CI practices. Our goal is to discover how this monitoring is conducted, demonstrate the advantages of monitoring CI practices, and highlight the challenges that need to be overcome. We conducted a series of studies to analyze the impact of specific CI practices on the volume of Pull Requests and bug-related Issues, as well as the importance attributed to these practices by developers and the support for monitoring provided by the most popular CI tools. Finally, we evaluate monitoring in real-world scenarios, conducting a case study on three projects from three different organizations, where we investigate more deeply the developers' interest in monitoring CI practices, their benefits, challenges, and the evolution of CI practices over a two-month period.
In our first study, we analyze the impact of specific CI practices on the volume of Pull Requests and bug-related Issues. Our findings revealed a positive correlation between CI practices and the increase in the number of merged pull requests. We also identified a significant correlation between the number of bug-related issues and the number of bugs. Additionally, our findings suggest that higher values of CI practices can indicate better quality in the development process.
In our second study, we investigate the importance attributed to these practices by developers and the support for monitoring provided by the most popular CI services. We find that developers usually monitor only code coverage and basic build metadata (e.g. build duration and build status). Developers expressed interest in monitoring CI practices if provided the opportunity. Furthermore, we have identified that several leading CI services still have initial support in monitoring CI practices.
Finally, we evaluate monitoring in real-world scenarios, conducting a case study on three projects from three different organizations, where we investigate more deeply the developers' interest in monitoring CI practices, their benefits, challenges, and the evolution of CI practices over a two-month period. The case study revealed that monitoring CI practices provides several benefits to the project and is cost-effective. Participants demonstrated a strong desire to integrate CI monitoring dashboards with popular CI services.
Presentation discussing the development of an innovative evaluation and certification methodology for security systems, as part of the European project. By Dr Thordis Sveinsdottir.
Presentation discussing the development of an innovative evaluation and certification methodology for security systems, as part of the European project. By Dr Thordis Sveinsdottir.
Agile techniques that utilize iterative development are broadly used in various industry projects as a lightweight development technique which can satisfy the continuous changes of requirements. Short repetitions are used that are required for efficient product delivery. Traditional and old software development methods are not much efficient and effective to control the rapid change in requirements. Despite the benefits of Agile, criticism on agile methodology states that it couldn’t succeed to pay attention to architectural and design issues and therefore is bound to produce small design-decisions. The past decade has observed numerous changes in systems development with many organizations accepting agile techniques as a viable methodology for developing systems. An increase in the number of research studies reveals the growing demand and acceptance of agile methodologies. While most research has focused on acceptance rate and adaptation of agile practices, there is very limited knowledge of their post-adoption usage and incorporation within organizations. Several factors explain the effective usage of agile methodologies. A combination of previous research in Agile Methodologies, Diffusion of Innovations, Information Systems implementation, and Systems Development has been carried out to develop a research model that identifies the main factors relevant to the propagation and effective usage of agile methodologies in organizations.
Project Collaboration on Infrastructure Megaprojects - Webinar, June 3, 2015Aconex
Parsons, one of the world’s largest construction and engineering companies, discuss best practices on how they structure decisions and processes on complex infrastructure projects.
Boston Consulting Group provide insight on how they assist some of the world’s largest JVs and PPPs.
See a demonstration of process management, including submittals, RFIs, commissioning and handover using Aconex.
RT332: Measuring Progress and Productivity in Model-based EngineeringAVEVA Group plc
CII is an organization consisting of owners, constructors, designers, and academia dedicated to improving the construction industry as a whole, providing meaningful research to yield useful tools to improve all aspects of the construction industry. Learn how AVEVA is collaborating with CII's Research Team 332 on researching and developing adoptable and adaptable solutions for the task of progressing and measuring productivity within 3D modeling environments.
Presented by:
Dr. Mani Golparvar-Fard—University of Illinois at Urbana-Champaign
Gustavo Garciat—University of Illinois at Urbana-Champaign
Derwin Cartmel—Day & Zimmermann
Discover how AVEVA can transform your business today
www.aveva.com
What are the differences between popular innovation project management methodologies? Why does project management often fail? Learn how risk assessment should define your methodology in order to become a real innovation factory. The waterfall methodology has been promoted for years as the best practice for IR&D management. These days agile and scrum are increasingly popular as alternative. Hater or believer? Good or bad? Get guided through our body of knowledge as published in the Inspire Toolbox.
Theory of Change or Change of Theory - lessons from Humidtropics CRPPiet van Asten
The integrated systems and ‘open agenda’ approach responds to the needs of key stakeholders (farmers, policy makers), but created relatively few engagement incentives for CGIAR scientists. The Humidtropics ToC is stil valid, but the CRP-II programs may need to invest in crossing the divide between research and enduser incentives if we wish to accelerate scaling and achieve ‘site integration’
Towards a Metamodel for a Requirements Engineering Process of Embedded SystemsTarcísio Couto
In the embedded systems (ES) area, more than 50% of problems occur at system delivery and are related to
misconconceptions in capturing requirements. According to our systematic literature review (SLR), no evidence explicitly depicts how an embedded system must be elicited and specified. However, understanding the embedded systems and their environment is a strenuous activity. Even though current approaches present some contributions, the definition of a systematic requirements engineering process remains a challenging issue. Based on this shortcoming, we developed a metamodel that defines concepts and relationships that must be taken into account the development of an ES. From the metamodel, we define a precise requirements engineering process. This research presents the main results of a SLR, a resource model, and a sketch of a process to guide the requirements development of embedded systems.
Agile techniques that utilize iterative development are broadly used in various industry projects as a lightweight development technique which can satisfy the continuous changes of requirements. Short repetitions are used that are required for efficient product delivery. Traditional and old software development methods are not much efficient and effective to control the rapid change in requirements. Despite the benefits of Agile, criticism on agile methodology states that it couldn’t succeed to pay attention to architectural and design issues and therefore is bound to produce small design-decisions. The past decade has observed numerous changes in systems development with many organizations accepting agile techniques as a viable methodology for developing systems. An increase in the number of research studies reveals the growing demand and acceptance of agile methodologies. While most research has focused on acceptance rate and adaptation of agile practices, there is very limited knowledge of their post-adoption usage and incorporation within organizations. Several factors explain the effective usage of agile methodologies. A combination of previous research in Agile Methodologies, Diffusion of Innovations, Information Systems implementation, and Systems Development has been carried out to develop a research model that identifies the main factors relevant to the propagation and effective usage of agile methodologies in organizations.
Project Collaboration on Infrastructure Megaprojects - Webinar, June 3, 2015Aconex
Parsons, one of the world’s largest construction and engineering companies, discuss best practices on how they structure decisions and processes on complex infrastructure projects.
Boston Consulting Group provide insight on how they assist some of the world’s largest JVs and PPPs.
See a demonstration of process management, including submittals, RFIs, commissioning and handover using Aconex.
RT332: Measuring Progress and Productivity in Model-based EngineeringAVEVA Group plc
CII is an organization consisting of owners, constructors, designers, and academia dedicated to improving the construction industry as a whole, providing meaningful research to yield useful tools to improve all aspects of the construction industry. Learn how AVEVA is collaborating with CII's Research Team 332 on researching and developing adoptable and adaptable solutions for the task of progressing and measuring productivity within 3D modeling environments.
Presented by:
Dr. Mani Golparvar-Fard—University of Illinois at Urbana-Champaign
Gustavo Garciat—University of Illinois at Urbana-Champaign
Derwin Cartmel—Day & Zimmermann
Discover how AVEVA can transform your business today
www.aveva.com
What are the differences between popular innovation project management methodologies? Why does project management often fail? Learn how risk assessment should define your methodology in order to become a real innovation factory. The waterfall methodology has been promoted for years as the best practice for IR&D management. These days agile and scrum are increasingly popular as alternative. Hater or believer? Good or bad? Get guided through our body of knowledge as published in the Inspire Toolbox.
Theory of Change or Change of Theory - lessons from Humidtropics CRPPiet van Asten
The integrated systems and ‘open agenda’ approach responds to the needs of key stakeholders (farmers, policy makers), but created relatively few engagement incentives for CGIAR scientists. The Humidtropics ToC is stil valid, but the CRP-II programs may need to invest in crossing the divide between research and enduser incentives if we wish to accelerate scaling and achieve ‘site integration’
Towards a Metamodel for a Requirements Engineering Process of Embedded SystemsTarcísio Couto
In the embedded systems (ES) area, more than 50% of problems occur at system delivery and are related to
misconconceptions in capturing requirements. According to our systematic literature review (SLR), no evidence explicitly depicts how an embedded system must be elicited and specified. However, understanding the embedded systems and their environment is a strenuous activity. Even though current approaches present some contributions, the definition of a systematic requirements engineering process remains a challenging issue. Based on this shortcoming, we developed a metamodel that defines concepts and relationships that must be taken into account the development of an ES. From the metamodel, we define a precise requirements engineering process. This research presents the main results of a SLR, a resource model, and a sketch of a process to guide the requirements development of embedded systems.
Introduction to angular with a simple but complete projectJadson Santos
A simple front end project with angular. Its show how to create your first components, include bootstrap templates, create routes and build the project to production.
Conheça o Junit, Mockito e PowerMock, Frameworks que permitem você criar testes unitários para sistemas de informação. Até menos sistemas não foram pensados para testes.
Usando JMeter para testar sua aplicação JSFJadson Santos
Testes de performance são uns dos testes fundamentais que toda equipe de desenvolvimento com o mínimo de profissionalismo deveria executar.
Essa apresentação mostra um exemplo básico de com usar JMeter para realizar testes de performance em aplicações web com JSF.
ISI 2024: Application Form (Extended), Exam Date (Out), EligibilitySciAstra
The Indian Statistical Institute (ISI) has extended its application deadline for 2024 admissions to April 2. Known for its excellence in statistics and related fields, ISI offers a range of programs from Bachelor's to Junior Research Fellowships. The admission test is scheduled for May 12, 2024. Eligibility varies by program, generally requiring a background in Mathematics and English for undergraduate courses and specific degrees for postgraduate and research positions. Application fees are ₹1500 for male general category applicants and ₹1000 for females. Applications are open to Indian and OCI candidates.
Professional air quality monitoring systems provide immediate, on-site data for analysis, compliance, and decision-making.
Monitor common gases, weather parameters, particulates.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
What is greenhouse gasses and how many gasses are there to affect the Earth.moosaasad1975
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...University of Maribor
Slides from talk:
Aleš Zamuda: Remote Sensing and Computational, Evolutionary, Supercomputing, and Intelligent Systems.
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Inter-Society Networking Panel GRSS/MTT-S/CIS Panel Session: Promoting Connection and Cooperation
https://www.etran.rs/2024/en/home-english/
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
Salas, V. (2024) "John of St. Thomas (Poinsot) on the Science of Sacred Theol...Studia Poinsotiana
I Introduction
II Subalternation and Theology
III Theology and Dogmatic Declarations
IV The Mixed Principles of Theology
V Virtual Revelation: The Unity of Theology
VI Theology as a Natural Science
VII Theology’s Certitude
VIII Conclusion
Notes
Bibliography
All the contents are fully attributable to the author, Doctor Victor Salas. Should you wish to get this text republished, get in touch with the author or the editorial committee of the Studia Poinsotiana. Insofar as possible, we will be happy to broker your contact.
Seminar of U.V. Spectroscopy by SAMIR PANDASAMIR PANDA
Spectroscopy is a branch of science dealing the study of interaction of electromagnetic radiation with matter.
Ultraviolet-visible spectroscopy refers to absorption spectroscopy or reflect spectroscopy in the UV-VIS spectral region.
Ultraviolet-visible spectroscopy is an analytical method that can measure the amount of light received by the analyte.
hematic appreciation test is a psychological assessment tool used to measure an individual's appreciation and understanding of specific themes or topics. This test helps to evaluate an individual's ability to connect different ideas and concepts within a given theme, as well as their overall comprehension and interpretation skills. The results of the test can provide valuable insights into an individual's cognitive abilities, creativity, and critical thinking skills
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Sérgio Sacani
Since volcanic activity was first discovered on Io from Voyager images in 1979, changes
on Io’s surface have been monitored from both spacecraft and ground-based telescopes.
Here, we present the highest spatial resolution images of Io ever obtained from a groundbased telescope. These images, acquired by the SHARK-VIS instrument on the Large
Binocular Telescope, show evidence of a major resurfacing event on Io’s trailing hemisphere. When compared to the most recent spacecraft images, the SHARK-VIS images
show that a plume deposit from a powerful eruption at Pillan Patera has covered part
of the long-lived Pele plume deposit. Although this type of resurfacing event may be common on Io, few have been detected due to the rarity of spacecraft visits and the previously low spatial resolution available from Earth-based telescopes. The SHARK-VIS instrument ushers in a new era of high resolution imaging of Io’s surface using adaptive
optics at visible wavelengths.
Earliest Galaxies in the JADES Origins Field: Luminosity Function and Cosmic ...Sérgio Sacani
We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest
imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters
spanning 0.4−0.9µm) and novel JWST images with 14 filters spanning 0.8−5µm, including 7 mediumband filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data
at > 2.3µm to construct an ultradeep image, reaching as deep as ≈ 31.4 AB mag in the stack and
30.3-31.0 AB mag (5σ, r = 0.1” circular aperture) in individual filters. We measure photometric
redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts
z = 11.5 − 15. These objects show compact half-light radii of R1/2 ∼ 50 − 200pc, stellar masses of
M⋆ ∼ 107−108M⊙, and star-formation rates of SFR ∼ 0.1−1 M⊙ yr−1
. Our search finds no candidates
at 15 < z < 20, placing upper limits at these redshifts. We develop a forward modeling approach to
infer the properties of the evolving luminosity function without binning in redshift or luminosity that
marginalizes over the photometric redshift uncertainty of our candidate galaxies and incorporates the
impact of non-detections. We find a z = 12 luminosity function in good agreement with prior results,
and that the luminosity function normalization and UV luminosity density decline by a factor of ∼ 2.5
from z = 12 to z = 14. We discuss the possible implications of our results in the context of theoretical
models for evolution of the dark matter halo mass function.
A Deep Dive into Continuous Integration Monitoring Practices
1. A Deep Dive into Continuous
Integration Monitoring Practices
UNIVERSIDADE FEDERAL DO RIO GRANDE DO NORTE
CENTRO DE CIÊNCIAS EXATAS E DA TERRA
DEPARTAMENTO DE INFORMÁTICA E MATEMÁTICAAPLICADA
PROGRAMA DE PÓS-GRADUAÇÃO EM SISTEMAS E COMPUTAÇÃO
Jadson José dos Santos
PhD Candidate
Doctoral Defense
Advisor: Prof. Dr. Uirá Kulesza
Co-Advisor: Prof. Dr. Daniel Alencar da Costa
Natal-RN
4:30PM Thursday, Apr. 25, 2024
Background Wallpaper Credits: Jon Hutchinson. https://www.teachwire.net/news/6-ideas-for-taking-a-curriculum-deep-dive-with-resources/
2. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 2
INTRODUCTION
“Continuous Integration (CI) is a
software development practice
where members of a team integrate
their work frequently, usually each
person integrates at least daily.”
Martin Fowler
3. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 3
• There is extensive research into the benefits and
costs of Continuous Integration
INTRODUCTION
Hilton et. al. 2016.
Usage, costs, and benefits of continuous integration in open-source projects.
Costs
• CI systems have too high
maintenance costs
• CI takes too long to set up
• Our project doesn’t have automated
tests
• The developers on my project are
not familiar enough with CI
• …
Benefits
• CI makes us less worried about
breaking our builds
• CI helps us catch bugs earlier
• CI helps us deploy more often
• CI makes integration easier
• CI lets us spend less time
debugging
• …
4. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 4
LITERATURE
REVIEW
Zhao et al. 2017
After the transition to
using Travis CI: (i) the
increasing number of
merge commits, (ii)
test suite sizes seem
to increase, etc...
Vasilescu et al. 2015
After CI: (i) Teams are significantly
more effective at merging pull
requests. (ii) Core developers are able
to discover significantly more bugs
than in teams not using CI
Bernardo et al. 2018
After CI: a large increase in the
number of submitted, merged and
delivered PRs per release.
5. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 5
• However
• Most of studies focused in miner repositories and
have examined the overall impact of CI
• However … there is a lack of studies specifically
addressing the monitoring CI practices
LITERATURE
REVIEW
6. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 6
LITERATURE
REVIEW
42.5%
Of the primary studies did not provide explicit criteria
for identifying CI usage in projects.
56.25%
Among studies that applied only one criterion, CI
service configuration was the most common
Soares et al. 2021
7. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 7
LITERATURE
REVIEW
Felidré et al 2019
(i) The average code coverage of Java projects it
was 63%;
(ii) 60% of the projects have infrequent commits;
(iii) 85% of the projects have at least one broken
build that take a long time to be fixed
(iv) most of projects have a build that executes with
more than 10 minutes.
A fact known as CI THEATER.
8. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 8
AGENDA
OBJECTIVES
METHODOLOGY
STUDY 1
STUDY 2
STUDY 3
CONCLUSION
9. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 9
OBJECTIVES
This thesis aims to answer the following general research question:
• How important is monitoring CI practices?
• We hypothesize that development teams typically do not adequately monitor
project CI practices
• Consistently monitoring these practices over time can lead to more efficient
and optimized CI processes
10. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 10
METHODOLOGY
11. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 11
Study 1: Investigating the Impact of Continuous
Integration Practices on the Volume of Merged
Pull Requests and Closed Bug-Related Issues
of Open-Source Projects
Background Wallpaper Credits: Lombardy.https://www.en.regione.lombardia.it/wps/portal/site/en-regione-lombardia/discover-lombardy/statistic-and-observatories
12. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 12
OBJECTIVES
• Verify the potential influence of 5 CI practice* in the Volume of
merged Pull Requests (related to project productivity) and in the
Volume of bug-related Issue (related to project quality)
Study 1
2 years
First commit
travis.yml
Last Build
* 1) build duration, 2) build activity, 3) build health, 4) time to fix a broken build, and 5) commit activity
13. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 13
METHODOLOGY*
*
Similarly
to
Zhao
et
al.’s
work
Study 1
Most starred GitHub projects Relevant projects**
27,883 3,143 2,029 776
90
Have used Travis-CI 2 years using Travis-CI
Filter active projects **** Collect Info ***
*** PRs, Issues, commits, and builds
** more than 100 stars, 100 PRs, 100 Issues,10MB **** at least 1 build in all 24 month of study
Build LR Models and D.A.
14. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 14
METHODOLOGY
• RQ1: Which CI practices contribute to higher volume of merged
pull requests?
• We fit linear regression models (Ordinary Least Squares, OLS) to find
associations between the number of merged PRs and the 5 CI practices.
Study 1
15. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 15
RESULTS
• RQ1: Which CI practices contribute to higher volume of merged
pull requests?
Regression model
R2 0.353
Study 1
16. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 16
METHODOLOGY
• RQ2: Which CI practice contribute to higher volume of closed bug-
related issues?
• The number of bug-related IR present an excess number of zeros. We applied
zero-inflated negative binomial regression models (ZINB)
Study 1
24 months
0 0 0 0 2 0 0 0 0 1 0 0 0 1 0 0 0
Number of closed
bug-related issues
17. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 17
RESULTS
• RQ2: Which CI practice contribute to higher volume of closed bug-related issues?
*
Study 1
18. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 18
METHODOLOGY
• RQ3: Can consistently applied CI practices be an indicator of process quality?
Study 1
Projects where
practices are
above the
overall median
Projects where
practices are
below the
overall median
* a confidence level of 95% and a confidence interval of 10%
Randomly selected
91 and 88 PR*
PR Comments
19. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 19
RESULTS
• RQ3: Can consistently applied CI practices be an indicator of process quality?
Projects with worst CI practices Projects with best CI practices
Study 1
20. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 20
Background Wallpaper Credits: Lisa Melonçon. https://tek-ritr.com/survey-research/
21. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 21
OBJECTIVES
To better understand the importance and prevalence of
monitoring 7 CI practices, we:
• Document Analysis 1: Investigate whether developers express
the need for monitoring CI practices in their project repositories
• Survey: Where we ask questions regarding the monitoring of CI
practices
• Document Analysis 2: Perform a document analysis of the
official documentation of current state-of-the art CI tools related
to monitoring capabilities
Study 2
22. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 22
METHODOLOGY*
Study 2
Most starred GitHub projects * Using GitHub Actions
11,014 7,658 2,488
121
2 years using CI **
Filter active projects **** Relevant projects***
** with 6 months recent builds
* with more than 100 stars, 100 PRs, 100 Issues,10MB
*** more then 100 builds, issues, PRs, commits and coverage inputs inside study period
Identify Survey Participants
776
**** build information for all 24 months
Identify CI workflows
500
(28 answers)
23. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 23
METHODOLOGY
RQ1: Do developers express the need to monitor CI practices in
PRs?
Study 2
PR Comments
(4,892 comments)
Randomly selected
382 PRs*
73,004 PRs
from 121 projects
* a confidence level of 95% and a confidence interval of 5%
24. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 24
RESULTS
• RQ1:Do developers express the need to monitor CI practices in
PRs?
Study 2
25. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 25
• RQ2: What is the perceived importance of monitoring CI practices?
• Most projects (12/19): automatic execution of the CI build and the automatic execution
of tests define the project CI level
• Test related practices (31/85) are the most monitored CI practices
Study 2
26. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 26
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
• Evaluation of CI practices
• Most most developers (20/28),
71%, have given positive
comment about the 7 CI
practices
• Almost half of developers
(11/24) would like to improve
CI practices values
Study 2
27. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 27
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
Developer 10
Yes, these metrics will provide valuable information in the
status of our Project over time.
Developer 15
depends on practice, for example “commit frequency” is not
important at all, but fixing a broken build is”
Developer 18
Absolutely! It saves time in the long run and increases
confidence when refactoring is applied”
Study 2
28. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 28
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
• Evaluation of CI
practices
• 68% of the
developers (19/28)
would like to
receive regular
updates of our
survey because
they think the
monitor the 7 CI
practices is useful
Study 2
29. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 29
METHODOLOGY RQ3: Which monitoring features are currently supported by existing
CI tools?
• We conduct a Grey Literature Review (GLR) to compile the existing monitoring
capabilities of existing CI services and third-party tools
• We conduct a search on the official documentation of most popular CI services*
• We examined the sections within the documentation that could be directly
correlated with the concept of monitoring
• We collect all references to third-party tools cited in the official CI services
documentation
• We conduct a search on the official documentation of third-party tools
* TRAVISCI, GHACTIONS, CIRCLECI, APPVEYOR, AZURE, GITLAB and JENKINS.
(Golzadeh, M., Decan, A., Mens, T.: On the rise and fall of ci services in github.
In: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER))
Document Analysis
+
Thematic Analysis
Study 2
30. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 30
RESULTS • RQ3: Which monitoring features are currently supported by
existing CI tools?
• CI services currently offer limited built-in support for monitoring CI
• Build metadata tracking is the most commonly supported monitoring practice (“Build
Duration” and “Build Status”)
• There are various complementary tools that can enhance this monitoring with a
predominant focus on fundamental information about the performance of the
CI pipeline
• However, they often come with additional costs and complexity
Study 2
31. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 31
RESULTS
Study 2
32. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 32
Study 3: Do you monitor CI
practices? I don’t know. You tell me:
A case study
Background Wallpaper Credits: Gratis vector handgetekende case study-sjabloon voor bestemmingspagina’s. Freepik. Free License
33. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 33
OBJECTIVES • We conducted a multiple case study with mixed methods.
• Collection of metrics in three public Brazilian organizations (5 participants).
• 5 Research Questions
• Over a period of 8 weeks, we conducted a series of interviews with team
members to gain insights into the benefits and challenges of monitoring CI
practices
RQ1, RQ2,
RQ3, RQ4,
RQ5
RQ1 RQ5 RQ4
Study 3
34. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 34
MONITORING
TOOL • We developed an extensive* suite of metrics tool that automatically
collect the evolution of 7 CI practices over time
* Supports: 17 metrics (7 CI + 4 DORA + 6 Basics) and integration 4 tools with GHActions, GITLAB, Sonarqube and CodeCov
Study 3
35. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 35
RESULTS
• Thematic Analysis
Study 3
36. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 36
RESULTS
• RQ1: Were the developers satisfied with the project’s CI maturity
levels?
• Most of the projects (
!
"
) were satisfied with the CI maturity level prior to the case
study.
• They did not utilize any metric-driven assessment to evaluate CI maturity
Participant P1
The tool, like the study, also emphasizes
the importance of providing an accurate
idea of how the CI is performing,
enabling individuals to understand the
metrics and reference values.
Participant P2
It’s something we’re still maturing,
we started from scratch.... Then, one
thing ended up helping the other,
seeing these metrics and what
needed improvement in our projects
Study 3
37. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 37
RESULTS
• RQ2: What benefits does CI monitoring bring?
• Give a overview of CI stage of a project
• Highlight problems related to CI
• Awaken a desire for improvements
• Improve the quality of CI adoption
• Increase communication within the team.
• CI monitoring also proved to be not costly to apply
Participant P1
It brings precisely this visibility of those aspects that can be
considered a problem. As I was talking about this coverage, we already
have this information on the Sonarqube. But the other ones are new
aspects that bring a more qualitative view of the CI practices we did
not have in other tools ...
Study 3
38. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 38
RESULTS
• RQ2: What benefits does CI monitoring bring?
Participant P2
It helped to highlight... The coverage is
evident; it was interesting because it showed
that there was coverage data, and suddenly, it
disappeared. I think it’s helping.
Participant P3
I think it ... pointing out that we are having problems with the code coverage
metric. Regarding the build, we already felt that the build took a long time and that
the VMs were slow, but not that much, more than 8 hours of build time. We have
tests, tests that involve the instantiation of other modules in VMs, ...., but almost
10h is a very high value.
Participant P5
It brings more confidence related
to some CI needs, for sure. It provided a
better view of this process.
Study 3
39. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 39
RESULTS
• RQ3: What are the challenges/disadvantages for using it on a daily
basis?
• The time dedicated to other development tasks as the main difficulty in
monitoring CI practice
• The lack of integration with other CI tools
• Legacy projects and the internal culture
• A complex development environment, reflected in the instability of the CI
environment
Participant P1
I get a little worried about not having time to
present what has been completed if I’m going to
do all the tests, then we end up having to fit the
tests into the free time we have in the
sprint
Study 3
40. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 40
RESULTS
• RQ4: Are developers interested in using the tool?
Participant P3
I received the email and
the problem of code
coverage appeared, as I
already knew about code
coverage, I did not pay
much attention to the tool
because I thought that
perhaps it would be more
of a priority to look at the
issue of the pipeline
Study 3
41. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 41
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization A
Participant P1
… We already knew
about the coverage
problem from the
beginning, but it
became very evident.
With this information,
we began to act to
resolve
this problem
Study 3
42. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 42
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization B
Participant P2
… Last week we had
some problems with
infrastructure, we
ended up having to
do some configurations
and such. Because of it,
the build health metric
started to have a
lower value
Study 3
43. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 43
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization C
Participant P5
… There was also a
period at the end of last
year of the build being
generated in a broken
way … We had already
observed this in Gitlab,
but the monitoring tool
also contributed, in a
way, complemented this
need to fix it
Study 3
44. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 44
Background Wallpaper Credits: https://stock.adobe.com/br/search?k=overview
45. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 45
Thesis
Main
Contributions Better Development Process
Quality: Our findings suggest that
higher values of CI practices may
indicate better quality in the
development process
Overlooked of CI monitoring: Most of
developers consider just run automatic builds
and coverage (tests execution) to evaluate
maturity of CI process
Monitoring CI Bring Benefits: We discovered strong evidences
indicating that developers place significant importance on monitoring
Continuous Integration (CI) practices and, this monitoring brings
several benefits for the project
46. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 46
Thesis
Main
Contributions
Basic Tools Support: Leading CI services and
complementary tools still offer basic support for monitoring
CI practices. Focus on monitoring fundamental
information related to the performance of the CI pipeline
47. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 47
Thesis
Main
Contributions
Monitoring Suite Tool: We developed an extensive suite
of metric tool that automatically calculate the values of seven
CI practices.
https://github.com/jadsonjs/holter-devops
48. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 48
Future
Work
?
49. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 49
What’s
Next? Extensions for new Practices:
Extending this work to assess additional
CI/CD practices in areas like DevOps
Extensions for new
Contexts: Our evaluations
can be replicated in more
extensive projects, testing the
robustness of our results
Tooling: 1) Improvements to our monitoring suite tool can be made. 2) We
strongly recommend that CI services (such as Gitlab, Github, CircleCi,
Jenkins, among others) strive to support CI practices monitoring natively
Measure CI maturity:
Deeper exploration of the
optimal practices values for
different contexts,
establishing ideal values to
determine the CI maturity
level
Extensions for CI practices benefits:
Exploring other ways of evaluating the
benefits of CI practices in software
development projects
50. 27/04/2024 50
A Deep Dive into Continuous Integration Monitoring Practices
Thank you http://caseufrn.github.io/
Background Wallpaper Credits: https://www.localdivethailand.com/phuket-diving-frequently-asked-questions/