Assessing digital literacy levels: the design of an instrument - AILA 2014

1. Assessing digital literacy levels:
the design of an instrument
AILA 2014
Brisbane – Australia -
August
Rosinda de Castro Guerra
Ramos
F. Katherine Asega

2. This research aims to investigate how to
measure the level of Digital Literacy (DL) of
in-service English teachers in Brazil in order to
promote their digital inclusion.
RESEARCH OBJECTIVE

3. MOTIVATION
SOCIAL:
Changes in society.
Transition between "old
and new, real and virtual,
the printed and digital"
(Buzato, 2006) requires a
set of competences
related to Technologies.
Educating efficient digital citizens.

4. MOTIVATION
Three reasons to include ICT
in education:
1. Potential benefits of ICT for
teaching and learning.
2. The need to acquire Digital
Competence to be functional
in our knowledge society
(Eshet-Alkalai, 2004).
3. Digital inclusion depends more
on knowledge and skills than
on access and use (Erstad,
2010b).
EDUCATIONAL:
Integration of ICT in practice.
Educating efficient digital citizens.

5. MOTIVATION
PROFESSIONAL:
The need for teacher development.
Educating efficient digital citizens.
• To become digitally literate.
• To help students to become
collaborative, problem
solving, creative learners
through using ICT.
BUT...
What level of digital literacy
should teachers have to
make a more appropriate
use of ICTs in practice?

6. Transition between "old and new, real and virtual,
the printed and digital" (Buzato, 2006) requires a set of
competences related to Technologies.
Three reasons to include ICT in education:
1. Potential benefits of ICT for teaching and learning.
2. The need to acquire Digital Competence to be functional
in our knowledge society (Eshet-Alkalai, 2004).
3. Digital inclusion depends more on knowledge and skills
than on access and use (Erstad, 2010b).
SOCIAL REASONS

7. Our objective is to present
the process of designing
an assessment instrument
to evaluate
the level of DL
of Brazilian English Teachers in-service.
PRESENTATION OBJECTIVE

8. Digital Literacy and its concepts.
Ala-Mutka, K. (2011); Bawden, D. (2001); Bawden, D. (2008);
Erstad, O. (2010b); Eshet-Alkalai, Y. (2004); Eshet-Alkalai, Y., &
Chajut, E. (2010); European Commission (2013); European
Commission (2012); Martin, A. (2006); Martin, A., & Grudziecki,
J. (2006); UNESCO (2011).
THEORETICAL FOUNDATION
Digital Competence.
Ala-Mutka, K. (2011); Bawden, D. (2001); (Eshet-
Alkalai, 2004). European Commission (2013);
European Commission (2012).
Assessment.
Bachman & Palmer (1996).

9. “Digital Literacy is the awareness, attitude and ability
of individuals to appropriately use digital tools and
facilities to identify, access, manage, integrate,
evaluate, analyze and synthesize digital resources,
construct new knowledge, create media expressions,
and communicate with others, in the context of specific
life situations, in order to enable constructive social
action; and to reflect upon this process.”
(Martin & Grudziecki, 2006).
THEORETICAL FOUNDATION
Digital Literacy Concept

10. "Digital Competence involves the confident and critical
use of Information Society Technology (IST) for work,
leisure and communication. It is underpinned by basic
skills in ICT: the use of computers to retrieve, assess,
store, produce, present and exchange information, and
to communicate and participate in collaborative
networks via the Internet."
(European Parliament and the Council, 2006).
THEORETICAL FOUNDATION
Digital Competence Concept

11. • The DIGCOMP study was launched by European
Commission's Joint Research Centre (JRC) - Prospective
Technological Studies (IPTS).
• Carried out between January 2011 and December 2012.
• AIM: to identify descriptors and develop conceptual
framework of Digital Competence.
• RESULTS: two different interrelated outputs - a self-
assessment grid and a framework.
THEORETICAL FOUNDATION
Digital Competence Project (DIGCOMP)

12. Self-Assessment Grid
• Proposes the areas of Digital
Competence and descriptors
for three proficiency levels.
• It can be used as a tool to:
– identify the level of digital
competence of each
citizen;
– to understand how to
improve digital
competence.
THEORETICAL FOUNDATION
Framework
• Identifies (for each area) the
related competences.
• Provides descriptors on three
levels and examples
knowledge, attitudes and
skills.
• It can be used by curricula and
developers to develop the
digital competence of a
specific target group.
Digital Competence Project (DIGCOMP)

13. THEORETICAL FOUNDATION
Digital Competence – Areas and Competence

14. THEORETICAL FOUNDATION
Digital Competence Descriptors

15. DIGITAL LITERACY TEST DESIGN
Test
Design
Putting into
Practice - Stage 3
Conceptual
Mapping
Browsing
Assessment
Conceptual
Mapping
Browsing
Digital Literacy
Available
Tests
Checking and
comparing
Conceptual
Mapping
Looking for
Descriptors
Methodology

16. Online Proficiency Test that will
measure the Digital Literacy (DL)
considering three different levels of
knowledge, attitudes and skills: basic,
intermediate, advanced.
Digital Literacy Test description
Aims to identify the level of DL of in-service English teachers
in Brazil and inform the participants how to improve digital
competence according to the stated needs.
DIGITAL LITERACY TEST DESIGN

17. PHASE 1
Computer Literacy
TOOL FOCUS TEST
• Aims to evaluate:
- the skills to operate a variety of
computer applications
packages (word processing,
databases, spreadsheets);
- general IT skills, such as
copying disks and generating
hard-copy printout.
first step towards advanced
knowledge, skills and attitudes.
PHASE 2
Digital Competence
KNOWLEDGE, ATTITUDE AND SKILL
FOCUS TEST
• Aims to evaluate digital
competence into three levels:
basic, intermediate, advanced.
Taking into account 5 different
areas: information,
communication, content-creation,
safety and problem-solving.
DIGITAL LITERACY TEST DESIGN
General Description

18. Question Description
PHASE 1 – Computer Literacy
• Total of questions: 15
- 5 questions of Computer Basics
(computer devices and its
functions).
- 10 questions of Computer
Concept (software and
programs usage).
- Total amount of time: 7 min.
Close ended questions, choose
from a list of choice to identify the
computer skills.
Question Sample
1. Which computer devices(components)
do you know how to use?
Choose the skills that best describe your experience:
( ) Monitor
( ) Touch Screen
( ) Keyboard
( ) Mouse
( ) Touchpad
( ) CD-Rom drive
( ) USB drive
( ) External drive
( ) Printer
( ) Scanner
DIGITAL LITERACY TEST DESIGN

19. Question Description
PHASE 2 – Digital Competence
• Total of questions: 22
- 3 questions of Information: browsing, evaluating and storing.
- 6 questions of Communication: interaction, sharing, engaging,
collaborating, online norms and digital identity.
- 5 questions of Content-creation: developing content, integrating
resources, copyright, program modification.
- 4 questions of Safety: protecting devices, personal data, health,
environment.
- 4 questions of Problem solving: solving technical problems,
Identifying needs and technological responses, innovating and
creatively using technology.
• Total amount of time: 22 min.
Close ended questions, multiple
choice with questions that
represent the competence level.
DIGITAL LITERACY TEST DESIGN

20. Question Sample
PHASE 2 – Digital Competence
Safety area
DIGITAL LITERACY TEST DESIGN
Basic level
Intermediate level
Advanced level Close
Need to start from
Computer Literacy
Each and every area
will have its own
result, thought they
are related.

21. Preliminary Results - Stage 3: pilot phase
PHASE 1 – Computer Literacy
• Submitted the questions to analysis and interviewed the
participants:
- 2 different Information Technology specialists.
- 2 different English teachers (basic level of experience).
• The results:
- one specialist considered the questions obsolete due to used
terminology use and technology advances. The other suggested
some terms adjustment.
- both English Teachers know how to use the computer but they are
not familiar with the used terms.
• Conclusion: we are rewriting the questions based on the observation,
adjusting the terms and focusing more on the use of software and
programs based on daily task and use experience.
DIGITAL LITERACY TEST DESIGN

22. PHASE 2 – Digital Competence
• Will Submit the questions to:
- 2 different Information Technology specialists.
- 10 different English teachers (from basic to advanced level).
• The expected results:
- to map the level proficiency.
- to adjust questions and answers.
- to map exam time consuming.
- to experience a different online interface based on programming
and automatic results and reports.
DIGITAL LITERACY TEST DESIGN
Preliminary Results - Stage 3: pilot phase

23. FRAMEWORK
• The framework is based on the recognition that both cognitive and
technical skills are necessary for people to be functional in a digital
society.
• Target group: adults.
• It supports a vision of digital
• literacy as the ability to create, communicate and collaborate to
organise and produce
• information; to understand and apply knowledge of the functions of
ICT; to use ICT for
• thinking and learning; and to develop a critical appreciation of the
role of ICT in society.
• Target group: primary and lower secondary schools.

24.  Ala-Mutka, K. (2011). Mapping Digital Competence: Towards a Conceptual
Understanding. Seville: JRC-IPTS. Retrieved from
http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=4699
 Bachman, L., & Palmer, A. (1996). Language testing in practice: Designing and
developing useful language tests. Oxford: Oxford University Press.
 Bawden, D. (2001). Information and digital literacies: a review of concepts. Journal
of Documentation, 57(2), 218-259.
 Bawden, D. (2008). Origins and Concepts Of Digital Literacy. In C. Lankshear & M.
Knobel (Eds.), Digital Literacies: Concepts, Policies & Practices (pp. 17-32).
 Erstad, O. (2010a). Conceptions of Technology Literacy and Fluency. In P. Penelope,
B. Eva & M. Barry (Eds.), International Encyclopedia of Education (pp. 34-41).
Oxford: Elsevier.
 Eshet-Alkalai, Y. (2004). Digital Literacy. A Conceptual Framework for Survival Skills
in the Digital Era. Journal of Educational Multimedia & Hypermedia, 13(1), 93-106.
 Eshet-Alkalai, Y., & Chajut, E. (2010). You can teach old dogs new tricks: The factors
that affect changes over time in digital literacy. Journal of Information Technology
Education, 9, 173-181.
REFERENCES

25.  European Commission (2011). Mapping Digital Competence: Towards a Conceptual
Understanding. Luxembourg: Publications Office of the European Union.
 European Commission (2011). Mapping Digital Competence: Towards a Conceptual
Understanding. Luxembourg: Publications Office of the European Union.
 European Commission (2012). Digital Competence in Practice: An Analysis of
Frameworks. Luxembourg: Publications Office of the European Union.
 European Commission (2013). DIGCOMP: A Framework for Developing and
Understanding Digital Competence in Europe.
 Martin, A. (2006). Literacies for the Digital Age. In A. Martin & D. Madigan (Eds.),
Digital Literacies for Learning (pp. 3-25). London: Facet.
 Martin, A., & Grudziecki, J. (2006). DigEuLit: Concepts and Tools for Digital Literacy
Development. ITALICS: Innovations in Teaching & Learning in Information &
Computer Sciences, 5(4), 246-264.
 UNESCO (2008) . ICT Competency Standards for Teachers. United Nations
Educational, Scientific and Cultural Organization.
REFERENCES