[DSC Europe 22] Machine learning algorithms as tools for student success pred...DataScienceConferenc1
The goal of higher education institutions is to provide quality education to students. Predicting academic success and early intervention to help at-risk students is an important task for this purpose. This talk explores the possibilities of applying machine learning in developing predictive models of academic performance. What factors lead to success at university? Are there differences between students of different generations? Answers are given by applying machine learning algorithms to a data set of 400 students of three generations of IT studies. The results show differences between students with regard to student responsibility and regularity of class attendance and great potential of applying machine learning in developing predictive models.
[DSC Europe 22] Machine learning algorithms as tools for student success pred...DataScienceConferenc1
The goal of higher education institutions is to provide quality education to students. Predicting academic success and early intervention to help at-risk students is an important task for this purpose. This talk explores the possibilities of applying machine learning in developing predictive models of academic performance. What factors lead to success at university? Are there differences between students of different generations? Answers are given by applying machine learning algorithms to a data set of 400 students of three generations of IT studies. The results show differences between students with regard to student responsibility and regularity of class attendance and great potential of applying machine learning in developing predictive models.
Neeraj Trivedi - Training of district officials in BiharPOSHAN
Presentation by Neeraj Trivedi on "Training of district officials in Bihar" at Developing a nutrition training roadmap to support India’s nutrition progress (17-18 Dec 2019)
Social Recommendation A Review ACM Transactions on Intelligent Systems and Technology 2013 Jiliang Tang · Xia Hu · Huan Liu Associate professor at Michigan.
Optimizing Data Synthesis and Visualization in Real-Time Decision-MakingCSSI_Inc
CSSI’s Kim Bender was a speaker at 2014's AMS Summer Community Meeting: Improving Forecasts and the Communication of Forecasts. Kim was a member of the panel on “Synthesizing Forecasting Information” which discussed the plethora of information forecasters have to guide their decisions.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/10/bias-in-computer-vision-its-bigger-than-facial-recognition-a-presentation-from-santa-clara-university/
Susan Kennedy, Assistant Professor of Philosophy at Santa Clara University, presents the “Bias in Computer Vision—It’s Bigger Than Facial Recognition!” tutorial at the May 2023 Embedded Vision Summit.
As AI is increasingly integrated into various industries, concerns about its potential to reproduce or exacerbate bias have become widespread. While the use of AI holds the promise of reducing bias, it can also have unintended consequences, particularly in high-stakes computer vision applications such as facial recognition. However, even seemingly low-stakes computer vision applications such as identifying potholes and damaged roads can also present ethical challenges related to bias.
This talk explores how bias in computer vision often poses an ethical challenge, regardless of the stakes involved. Kennedy discusses the limitations of technical solutions aimed at mitigating bias, and why “bias-free” AI may not be achievable. Instead, she focuses on the importance of adopting a “bias-aware” approach to responsible AI design and explores strategies that can be employed to achieve this.
Chapter 9 – Proofreading Exercise
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
A critical process in technology management is measuring performance and
managing information. As noted by numerous authors and researchers, focusing on the
wrong areas can lead to actions that don’t help the organization
relize there strategic goals. This is one of the ways that a balanced
scorecard approach can prove beneficial, since it helps the firm identify the
proper measures by allaying them with the firms vision
and strategy. Specifically, performance measures should align with
the principal factors that determinate the success for that
particular firm.
It’s impotent to note that a firm could be meeting these
specific requirements and still not be in a position to acheive it’s
long-term goals. This problem can be addressed at least in part by
ensuring that the performance measures are used at the appropriate level
within the organization. If the organization ensures that there are goals for
each level of the organization, and appropriate measures that
correspond too these goals, than it stands a much better chance
of aligning its performance measures with it’s strategic intent. This means
that their should be goals and corresponding measures that are used at
the corprate level, the factory level (in the case of a firm in the
manufacturing industry) and the production line level.
As we can see, identifying the proper performance measures can be
difficult. That having ben said, so long as the firm uses the
proper criterion to select data and uses that data correctly it will
have a much better chance of finding and sustaining an competitive
advantage.
ITS 832
Chapter 15
Visual Decision Support for Policy Making: Advancing
Policy Analysis with Visualization
Information Technology in a Global Economy
Professor Michael Solomon
Introduction
• Background
• Approach
• Case Studies
• Optimization
• Social Simulation
• Urban Planning
• Conclusion
Background
• Assessing policy options for societal problems is difficult
• Decision making methods
• Data driven
• Model driven
• Visual decision supports helps in evaluating model output
• Information visualization and visual analytics
• Makes complex results accessible to many
• Policy analysis
• Part of process aimed at solving societal problems
Data Visualization
Policy cycle
Approach
• Characterization of stakeholders
• Policy makers
• Policy analysts
• Modeling experts
• Domain experts
• Public stakeholders
• Bridging knowledge gaps
• With information visualization (IV)
• Cohesive view of model representation
Visual Support for Policy Analysis
Approach, cont’d.
• Synergy effects of applying IV to policy analysis
• Communication - facilitated
• Complexity - reduced
• Subjectivity - reduced
• Validation - improve ...
This presentations covers Definition of Operations Research , Models, Scope,Phases ,advantages,limitations, tools and techniques in OR and Characteristics of Operations research
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Neeraj Trivedi - Training of district officials in BiharPOSHAN
Presentation by Neeraj Trivedi on "Training of district officials in Bihar" at Developing a nutrition training roadmap to support India’s nutrition progress (17-18 Dec 2019)
Social Recommendation A Review ACM Transactions on Intelligent Systems and Technology 2013 Jiliang Tang · Xia Hu · Huan Liu Associate professor at Michigan.
Optimizing Data Synthesis and Visualization in Real-Time Decision-MakingCSSI_Inc
CSSI’s Kim Bender was a speaker at 2014's AMS Summer Community Meeting: Improving Forecasts and the Communication of Forecasts. Kim was a member of the panel on “Synthesizing Forecasting Information” which discussed the plethora of information forecasters have to guide their decisions.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/10/bias-in-computer-vision-its-bigger-than-facial-recognition-a-presentation-from-santa-clara-university/
Susan Kennedy, Assistant Professor of Philosophy at Santa Clara University, presents the “Bias in Computer Vision—It’s Bigger Than Facial Recognition!” tutorial at the May 2023 Embedded Vision Summit.
As AI is increasingly integrated into various industries, concerns about its potential to reproduce or exacerbate bias have become widespread. While the use of AI holds the promise of reducing bias, it can also have unintended consequences, particularly in high-stakes computer vision applications such as facial recognition. However, even seemingly low-stakes computer vision applications such as identifying potholes and damaged roads can also present ethical challenges related to bias.
This talk explores how bias in computer vision often poses an ethical challenge, regardless of the stakes involved. Kennedy discusses the limitations of technical solutions aimed at mitigating bias, and why “bias-free” AI may not be achievable. Instead, she focuses on the importance of adopting a “bias-aware” approach to responsible AI design and explores strategies that can be employed to achieve this.
Chapter 9 – Proofreading Exercise
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
A critical process in technology management is measuring performance and
managing information. As noted by numerous authors and researchers, focusing on the
wrong areas can lead to actions that don’t help the organization
relize there strategic goals. This is one of the ways that a balanced
scorecard approach can prove beneficial, since it helps the firm identify the
proper measures by allaying them with the firms vision
and strategy. Specifically, performance measures should align with
the principal factors that determinate the success for that
particular firm.
It’s impotent to note that a firm could be meeting these
specific requirements and still not be in a position to acheive it’s
long-term goals. This problem can be addressed at least in part by
ensuring that the performance measures are used at the appropriate level
within the organization. If the organization ensures that there are goals for
each level of the organization, and appropriate measures that
correspond too these goals, than it stands a much better chance
of aligning its performance measures with it’s strategic intent. This means
that their should be goals and corresponding measures that are used at
the corprate level, the factory level (in the case of a firm in the
manufacturing industry) and the production line level.
As we can see, identifying the proper performance measures can be
difficult. That having ben said, so long as the firm uses the
proper criterion to select data and uses that data correctly it will
have a much better chance of finding and sustaining an competitive
advantage.
ITS 832
Chapter 15
Visual Decision Support for Policy Making: Advancing
Policy Analysis with Visualization
Information Technology in a Global Economy
Professor Michael Solomon
Introduction
• Background
• Approach
• Case Studies
• Optimization
• Social Simulation
• Urban Planning
• Conclusion
Background
• Assessing policy options for societal problems is difficult
• Decision making methods
• Data driven
• Model driven
• Visual decision supports helps in evaluating model output
• Information visualization and visual analytics
• Makes complex results accessible to many
• Policy analysis
• Part of process aimed at solving societal problems
Data Visualization
Policy cycle
Approach
• Characterization of stakeholders
• Policy makers
• Policy analysts
• Modeling experts
• Domain experts
• Public stakeholders
• Bridging knowledge gaps
• With information visualization (IV)
• Cohesive view of model representation
Visual Support for Policy Analysis
Approach, cont’d.
• Synergy effects of applying IV to policy analysis
• Communication - facilitated
• Complexity - reduced
• Subjectivity - reduced
• Validation - improve ...
This presentations covers Definition of Operations Research , Models, Scope,Phases ,advantages,limitations, tools and techniques in OR and Characteristics of Operations research
Semelhante a DREAMS 2021 presentation- Evaluation metrics in human-in-the-loop for autonomous systems (20)
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
DREAMS 2021 presentation- Evaluation metrics in human-in-the-loop for autonomous systems
1. REVIEW OF EVALUATION METRICS USED IN LITERATURE + PROPOSED IDEA
DREAMS, EDCC 2021
13 September 2021
Evaluation of Human-in-the-Loop Learning
based Autonomous Systems
Prajit Thazhurazhikath Rajendran, Huascar Espinoza, Chokri Mraidha (CEA, DILS-
LSEA),
Agnes Delaborde (LNE)
2. | 2
DREAMS 2021 | Prajit T Rajendran
Safety challenges of DL/AI components
• The use of DL/AI components in autonomous
systems comes with various challenges:
• Vulnerable to out of distribution data
• Adversarial inputs
• Anomalies
• Lack of transparency
• Stochastic nature
• Unknown unknowns
• Uncertainty
• Safety is an emergent property- it is not as a
property of any particular component individually
• Regulation/qualification/certification of such DL/AI
components is an ongoing work by the community
• Traditional approaches do not facilitate safe learning
• Humans can guide the system to safe behavior with
their knowledge, experience and adaptability
normal
anomaly
Out-of-Distribution Samples
3. | 3
DREAMS 2021 | Prajit T Rajendran
Categories of human-in-the-loop learning methods
Active learning
• Semi-supervised ML where only a subset of the training data is labelled
• Human queried interactively to label data points of interest from the unlabelled set
• PROS: Reduces data labelling requirement
• CONS: Selecting the right points to query is important
4. | 4
DREAMS 2021 | Prajit T Rajendran
Categories of human-in-the-loop learning methods
Demonstration
• Human is in full control and provides demonstrations to train the agent
• Agent can mimic human data to use as a safe starting point
• PROS: Leads to safer policies
• CONS: More human effort needed, may be subjective, train-test distribution shift
5. | 5
DREAMS 2021 | Prajit T Rajendran
Categories of human-in-the-loop learning methods
Intervention
• Human, agent share control and human intervenes when necessary
• Human takes over control to avoid catastrophic states and agent learns from these
• PROS: Leads to safer policies
• CONS: Need to keep human in the loop for long, slow response time
6. | 6
DREAMS 2021 | Prajit T Rajendran
Categories of human-in-the-loop learning methods
Evaluation
• Agent in full control and human provides feedback for tasks
• Human gives feedback based on known objective or preference, which the agent
learns
• PROS: Leads to safer policies
• CONS: Need to keep human in the loop for long, credit attribution problem,
subjective feedback
7. | 7
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
Rate of task completion
Safety
Performance
Data
requirement
User trust
Time
User
satisfaction
Rate of catastrophies
Response time
Training time
Subjective measures
Likert scale
Binary feedback
Type of interactions
Number of queries
Average reward
Deviation from thresholds
Number of interventions
8. | 8
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
Safety
• Learning from intervention used
• Human intervenes to avoid undesirable events or catastrophies
• Policy constrained to safer regions
• Evaluated based on number of occurences of catastrophies
Trial without error- Towards safe RL via human intervention, William Saunders et.al
Trial without error- Towards safe RL via human intervention, William Saunders et.al
9. | 9
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
Performance
One shot imitation learning, Yan Duan et.al
One shot imitation learning, Yan Duan et.al
• Learning from demonstration + meta-learning used
• Train networks that are not specific to one task and can adapt to
new tasks
• Evaluated based on average rate of success/task completion
10. | 10
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
Time
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• DAgger approaches: Learning from demonstration + intervention
• Start with imitation of expert policy, collect data
• Train the next policy under the aggregate of all collected datasets
• Hand over control to expert if necessary based on rulesets
• Evaluated based on number of training iterations needed to reach
a significant level of performance
Dropout Dagger- A Bayesian approach to safe imitation learning- Kunal Menda et.al
11. | 11
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
Data
requirement
Overcoming blindspots in the real world: Leveraging complementary
abilities for joint execution, Ramya Ramakrishnan et.al
Overcoming blindspots in the real world: Leveraging complementary
abilities for joint execution, Ramya Ramakrishnan et.al
• Learning from demonstration + intervention used
• Agent and human both are considered to have blindspots
• Choose actor (human vs agent) based on blindspot activation level
• Evaluated based on number of human queries needed vs average
reward
12. | 12
DREAMS 2021 | Prajit T Rajendran
Common metrics in HITL learning methods
User
satisfaction
• Trust ~ Extent that the human agrees with the AI
• Questionnaire about use of system, biological
data, number of interventions, “humanness” etc
• Users could be quick to distrust AI with easily
identifiable incorrect result
• Interpretability improves trust
User trust
• Satisfaction w.r.t interaction, performance,
design
• Could be subjective
• Questionnaires, evaluative feedbacks
• Necessary for successful adoption and
widespread use in society
13. | 13
DREAMS 2021 | Prajit T Rajendran
Limitations of prior approaches
• Assumptions made about humans (even experts) being always correct
• Interactions between human and AI may not always be flawless
• Uncertainty of DL components not considered
• Presence of errors in data
• No existing measure for data quality
• Data quality may be defined in terms of completeness, accuracy and efficiency
Cognitive overload Slow response Incorrect response Lack of attention
Errors in perception Errors in planning Errors in execution
14. | 14
DREAMS 2021 | Prajit T Rajendran
Proposed approach
• Hypothesis: Bad demonstration samples affect safety; Full self-exploration by system is
also infeasible
• Premise: Infeasible to start training afresh due to large training time, unsafe exploration
Data store
Unsupervised
anomaly detector
Feature
extractor
Environment
dynamics
Anomaly
predictor Policy
learning
module
Correct samples
Erroneous
samples
Non-exploratory
training phase
Candidate
samples
Human-in-the-loop
Environment
Exploratory
training phase
Historical data, demonstrations etc.
15. | 15
DREAMS 2021 | Prajit T Rajendran
Proposed approach
• Non-exploratory training phase:
• Data from the data store is used to train the anomaly predictor and policy learning modules
• Can use human-in-the-loop to classify outliers as correct or erroneous
• Correct samples can directly be used for policy training
• Erroneous samples can be used to predict future anomalies/faults by combining with model of
environment dynamics
Data store
Unsupervised
anomaly detector
Feature
extractor
Environment
dynamics
Anomaly
predictor Policy
learning
module
Correct samples
Erroneous
samples
Non-exploratory
training phase
Candidate
samples
Human-in-the-loop
Environment
Exploratory
training phase
Historical data, demonstrations etc.
16. | 16
DREAMS 2021 | Prajit T Rajendran
Proposed approach
• Exploratory training phase:
• System interacts with the environment but chooses actions based on predicted anomaly score
• Facilitates safe exploration by taking previous human feedback into consideration
Data store
Unsupervised
anomaly detector
Feature
extractor
Environment
dynamics
Anomaly
predictor Policy
learning
module
Correct samples
Erroneous
samples
Non-exploratory
training phase
Candidate
samples
Human-in-the-loop
Environment
Exploratory
training phase
Historical data, demonstrations etc.
17. | 17
DREAMS 2021 | Prajit T Rajendran
Future work
• Evaluation of suitable datasets used in autonomous systems policy/control development
• Development of experimental procedure for design and test of proposed model
• Implementation of human-in-the-loop sample classifier, and anomaly predictor
• Evaluation of system on pre-decided metrics on target domain
18. | 18
DREAMS 2021 | Prajit T Rajendran
Conclusions
• Identified necessity of human-in-the-loop learning, discussed its categories
• Explored the various evaluation metrics of human-in-the-loop approaches presented in
literature
• Defined the requirements for ”quality data“ with characteristics such as accuracy,
completeness or efficiency
• Proposed a method to measure and improve data quality in human-in-the-loop
approaches
19. Commissariat à l’énergie atomique et aux énergies alternatives
Institut List | CEA SACLAY NANO-INNOV | BAT. 861 – PC142
91191 Gif-sur-Yvette Cedex - FRANCE
www-list.cea.fr
Établissement public à caractère industriel et commercial | RCS Paris B 775 685 019
Prajit Thazhurazhikath Rajendran
prajit.thazhurazhikath@cea.fr Thank you