Fostering Friendships - Enhancing Social Bonds in the Classroom
What do students want? Towards an instrument for students’ evaluation of quality of learning analytics services
1. What do students want? Towards an instrument for
students’ evaluation of quality of learning analytics
services
Alexander Whitelock-Wainwright, Dragan Gašević, & Ricardo Tejeiro
A.Wainwright@Liverpool.ac.uk
2. Aims
• Highlight the importance of service quality in learning analytics.
• Develop an instrument to explore student expectations towards
learning analytics.
• Steps towards a model of learning analytics use.
3. What is service quality?
• Subjective assessment of the degree to which a service user’s needs
or expectations were met (Parasuraman, Zeithaml, & Malhotra ,
2005).
Expectations
Service
Usage/Exposure Perceptions Attitude
4. What is service quality? (Contd.)
• Encourages users to use own service over competitors (Parasuraman,
Zeithaml, & Berry, 1988).
• Service quality in higher education (Spooren, Brockx, & Mortelmans,
2013).
• Ideological gap (Ng & Forbes, 2009).
5. Service Quality in Learning Analytics
• Learning analytics services designed to support learning.
• Various stakeholder groups within learning analytics (e.g., students,
teachers, managers) (Clow, 2012).
• Quality indicators of learning analytics tools (Scheffel, Drachsler,
Stoyanov, & Specht, 2014).
6. Questionnaire Development
• Measuring student expectations of learning analytics.
• Development of institutional policies (SHEILA).
• Theories of human behaviour.
7. Questionnaire Development (Contd.)
• Identifying themes within past literature (Ifenthaler & Schumacher,
2016; Sclater, 2016; West, Heath, & Huijser, 2016):
• Ethics and Privacy
• Meaningfulness
• Agency
• Interventions
10. Pilot Study Results
• Instrument reduced to 19 items.
• Two factor solution for both scales:
• Service expectations:
o Desires scale – 0.88 alpha.
o Predictive scale – 0.88 alpha.
• Ethical expectations:
o Desires scale – 0.82 alpha.
o Predictive scale – 0.86 alpha.
16. Future Directions
• Develop the corresponding perceptions scale.
• Model intentions towards using learning analytics.
Attitudes
Social
Norms
Intentions to Use
Learning Analytics
Perceived
Behavioural
Control
The aims of our conceptual paper were to…
Firstly… highlight the importance of service quality in learning analytics… in particular we wanted to convey how future learning analytics services should be subject to quality assurance measures…
In other words… we cannot assume that any service in place won’t be without its problems…
For example… students may perceive the tools or service offered to not align with what they initially expected… such as the information not being relatable to their needs…
So if we are not able to identify these problems early on… there is a risk that learning analytics services will not be used by majority of students… due to their negatives attitudes towards using it…
To address this issue… we aimed to develop an instrument that measures the expectations of students towards learning analytics services…
And… I will be presenting some results from our pilot study of this questionnaire…
I will also present some ideas for future directions… particularly on the need for an instrument to assess perceptions of learning analytics services… and how all this can be placed within a theoretical framework of human behaviour…
We can consider service quality as a user’s subjective assessment of whether their expectations about a service were met…
To elaborate upon this… here is a simplistic overview of service quality…
Firstly… before we engage with any service we hold pre-trial beliefs… also known as expectations
For example… if a university offered a learning analytics service designed to provide regular updates about my performance… I would hold various beliefs about the timeliness and accuracy of the information provided…
These beliefs may have formed through receiving various information from the university… peers… or instructors
As I then start to utilise the learning analytics service… these cognitive beliefs will be disconfirmed… which can be either positive or negative… dependent on whether the service meets or fails to align with these expectations…
So the system may provide me timely updates on my learning… and this information may be both accurate and relevant to my needs…
What I am then perceiving is an alignment between the service offered and my expectations… as a result… I am more likely to hold an attitude towards using the service that will be positive…
On the other hand… a service that I perceive as not meeting these expectations will lead me to hold a negative attitude towards using the service…
If we consider this from the perspective of the theory of planned behaviour… or the theory of reasoned action… these attitudes that arise… following the disconfirmation of beliefs… can affect intentions to perform future behaviours… such as to reuse the learning analytics service in the future
So… providing a good quality of service can be an important factor in encouraging users… to continue using your service…
Service quality is not constrained to only businesses… it is also important for higher education… through the use of teacher evaluations and student surveys… not only can the results aid prospective students in deciding where to study… it can help the university identify aspects of a course that require improvement…
Without taking measurements of service quality… there is a risk of higher education institutes perpetuating an ideological gap… this means that there is a clear gap between what students expect from a university service… and what a higher education institute believes the service it is providing should be…
Persistence of this gap can then be assumed as a cause of student dissatisfaction…
We can view learning analytics as a service offered by higher education institutes… as it is designed to support students during their learning… such as providing visualisations of how an individual’s performance compares to their peers…
If we view learning analytics as a service… then it should be subject to quality assurance measures… as without a continual evaluation of such services then it is likely that problems could endure without resolution…
Persistence of these problems are then likely to lead to students holding negative attitudes towards using the learning analytics services…
What is problematic… is that there are different stakeholder groups within learning analytics… all of whom will hold different expectations of what a learning analytics service should achieve… or what features it should have…
For example… an instructor may require real-time updates about their students learning within a course… whereas… a manager may want to know how a handful of modules are running…
These variations in expectations could be detrimental to the overall service… as it may result in a service that satisfies the expectations of one group over and above the rest…
The DELICATE checklist does provide a method to standardise the evaluation of learning analytics tools… so as a community we are not ignoring the pivotal role of evaluation… but we should now move towards exploring how students perceive these services to be… as a way of improving attitudes of the main stakeholders
With this in mind… our aim was to develop a questionnaire to explore student expectations of learning analytics…
We have chosen to only concentrate on expectations… as it stands… due to majority of universities not having a general learning analytics services in place… so perception items cannot be developed as of yet…
The results from exploring expectations will still be beneficial… as with the development of policies about learning analytics… this questionnaire can provide a way of collecting students views about learning analytics… and these can be accounted for in any policy developments…
What’s more… this questionnaire can be used in conjunction with theories of human behaviour to explain students intentions towards using learning analytics…
In order to create the survey items… I conducted a literature review of learning analytics papers concerning ethics and privacy issues…
Within these papers… a number of themes regularly came up… which can be grouped into the following types of expectations
Ethics and privacy... This can relate to the idea of whether students expect their data to kept securely… and remain confidential…
Meaningfulness… this is mainly concerned with the feedback from learning analytics being provided in a format that is accessible… and understandable…
Agency… this relates to learning analytics being student-centred… so will students have control over whether they can make sense out of their own data… or will the institution do this for them
And… interventions… which is concerned with what students expect results of learning analytics to be used for… will an intervention be aimed at improving academic skills like writing… or will there be more focus on emotional support
A further issue relates to the term expectation being ambiguous… it cannot be thought simply as what an individual expects…
Instead… expectations can be thought of as either… predictive… which an individual’s pre-trial belief of how what a service will achieve…
Or… desired… which is the pre-trial belief of what level of service performance would be necessary to please the individual
By separating expectations out like this… a more detailed understanding of satisfaction can emerge…
So… if the perceptions of a service meet… or exceeded my desired expectations… then I will be satisfied with the service
Meeting the predictive expectations… on the other hand… will lead to a feeling of indifference…
Failure to meet either expectation… will be a cause of dissatisfaction
Following the literature review… I created 79 items… which were then subject to peer review…
The peer review allowed me to reduce the number of items… as the feedback highlighted items that were quite similar…
We were then left with 37 items for a pilot study… each item contains two subscales… one for predictive expectations… and one for desired expectations…
210 respondents took part in the pilot study… where they completed the questionnaire and provided qualitative comments about the clarity of each question…
An exploratory factor analysis was ran on the quantitative data collected from the questionnaire… which allowed for the identification of underlying latent variables…
The qualitative comments were also used to determine whether items needed changing or removing…
The factor analysis showed a two factor solution to be sufficient for both scales… with the same items loading on these factors across both scales
These factors can be viewed as service and ethical expectations…
For the service expectations… desires and predictive subscales had 0.88 reliability
For ethical expectations… the desires scale had a 0.82 reliability and the predictive scale had a 0.86 reliability…
The next five slides will present a brief overview of the pilot data…
Firstly… the item with the lowest average response related to whether teachers were obligated to act if a student was found to be underperforming.
This plot shows the highest average response for the predictive expectations scale… this was for the item relating to the university ensuring that all collected data will be kept securely…
The highest average response item for the desires expectation scale was for the item concerning students being asked for consent before data was sent to third party companies…
The smallest difference between both subscales was for the item relating to using identifiable data… so from this it can be inferred that students have strong opinions towards the university asking for any consent for identifiable data usage…
Whereas… the greatest difference between subscales was for the item concerning the feedback being easy to understand…
The next steps for this research project… are to first… re-distribute the questionnaire to a larger sample of students… whose responses will be subject to a further exploratory factor analysis…
Then a final distribution of the questionnaire will be undertaken… and a confirmatory factor analysis ran to assess the validity of the scale.
Once the expectation scale has been developed… there is a need for the corresponding perception scale… once this has been achieved we can then explore whether service expectations are being met…
The complete instrument for measuring expectations and perceptions can be an important step in introducing theory into learning analytics… as it can placed within the theory of planned behaviour framework as to model the reasons why students may or may not use the learning analytics systems in place…
This is important to consider… as the provision of a tool designed to support learning may be beneficial… from a managerial point of view… but if the student holds a negative attitude towards using it… then they may have no intention to use it at all…
So if students have no intention of using a learning analytics service or tool… then how will it be beneficial to learning?
