This lecture will analyze the increasingly important topic of assessment and evaluation in e-government. Different models, methodologies and approaches will be presented.
Dimitris Sarantis, Researcher, United Nations University, PT
3. A common case
• Assess different
maturity level of e-Services
4. Some things we want to improve
• Efficiency
• Fast execution of core process/improved service delivery
• Simplification of procedures
• Reduced paper work
• Reduced communication cost
• Transparency
• Reliable information delivery
• Comprehensive information delivery
• Easy access to information
• Fairness
• Interactivity
• Improved interaction (with internal actors, actors belonging to other related organizations,
beneficiaries, citizens, enterprises)
• Decision support
• Improved planning and decision-making
• Better Monitoring and control
5. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
6. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
7. Assessment Source
• Policy Document
• Strategy Documents
• Government Portal/Websites
• Reports
• Publications
• Raw Statistical Data
• Agencies
• Stakeholders
• …
13. eGovernment Assessment Perspective
DemandSupply
Environment
Leadership and political commitment
Οrganisational and governance
frameworks
Legal environment
Government policy and vision
Data-driven culture
Social and cultural environment
Clear business cases
Business environment
International co-operation
Infrastructure
Service Maturity
Connectivity and technology infrastructure
Cross-border Mobility
Key Enablers
Telecommunications Infrastructure Index (TII)
Interoperability
User Centric
Citizen Service Experience
Citizen Satisfaction Survey
Citizen Service Delivery
Experience
Consumer and business
adoption
User Engagement
Transparent
Government
Transparency, openness
and inclusiveness
Engagement and
participation
Coherent use of digital
technologies
14. eGovernment Assessment Cubes Assessment Sector
National Level
Local Level
Organisational Level
Assessment Scope
Supply
Demand
Assessment Perspective
15. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
16. Begin with a measure
• There are various measures you could choose to use – a number, a
percentage etc.
• For example:
Incorrect: Citizens in need of financial assistance
Correct: Number of citizens in need of financial assistance or % of citizens
in need of financial assistance
17. Indicators are neutral in terms of what should
be achieved
• Do not specify a direction in which you would like to see the change
occur, for example increase or decrease.
• Do not specify quantities - these are specified in your target
• For example:
Incorrect: Increased number of citizens voted in local elections or
80% of citizens voted in local elections
Correct: Number of citizens voted in local elections
18. Specify just one result per indicator
• Do not combine two results in one indicator. For example:
Incorrect: Number of public employees trained and
career guidance resources distributed
Correct: Number of public employees trained
Number of career guidance resources distributed
19. Indicators are specific and unambiguous
• Do not use “suitcase words”: unpack them.
• For example:
Incorrect: Number of demoralized public employees
Correct: Number of public employees reporting burnout
20. Necessary features for indicators
• Reliable: the indicator should be accurate enough for its intended use and
respond to changes in the level of performance.
• Well-defined: the indicator needs to have a clear, unambiguous definition so that
data will be collected consistently, and be easy to understand and use.
• Verifiable: it must be possible to validate the processes and systems that produce
the indicator.
• Cost-effective: the usefulness of the indicator should justify the cost of collecting
the data.
• Appropriate: the indicator should avoid unintended consequences and
encourage service delivery improvements, and not give managers incentives to
carry out activities simply to meet a particular target.
• Relevant: the indicator should relate logically and directly to an aspect of the
assessment object, and the realisation of strategic goals and objectives of the
process
21. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
22. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
23. e-Government Development Index (EGDI)
• United Nations Department of Economic and Social Affairs (UNDESA)
• Benchmark e-Government development achieved by all Member
States of the UN
• Performance rating of national governments relative to one another
• A relative decline in rankings does not necessarily imply that
countries have done less but rather those holding leadership
positions have advanced or performed better than others.
24. EGDI considerations
• New, better and faster public services.
• e-Government to support sustainable development in the three
dimensions – economic growth, social inclusion and environmental
protection.
• Reduce digital divide.
• Comparing one country's public processes and performance metrics
with other countries. In this way, they learn how well the targets
perform and, more importantly, the practices that explain why these
countries are successful.
25. i-Voting & OOP, Estonia
• Internet voting (I-voting or online voting) is one of the options for
elections in addition to other voting methods in Estonia.
• I-voting in this context means voting through the Internet, not voting
by using a special voting device.
• Internet voting was first introduced in the local elections of 2005, 2%
of all registered voters.
• 2019, 44%
• i-Voting saves over 11,000 working days per election
• Application of OOP
26. Standards - UK
• United Kingdom deployed new web technologies such as HTML5, as
part of the aim to make its national portal GOV.UK accessible to the
widest possible audience.
• Resulted in savings of £1.7bn in 2014 through its digital and
technology transformation.
• Eighty-five per cent of tax filing is now done through online channels
and over 98 per cent of driving tests are now booked online
27. Digital mental health gateway - Australia
• Head to Health focuses on digital services and resources, providing a
one-stop shop for services and resources delivered by some of
Australia’s most trusted mental health service providers.
• Resources include
• free or low-cost apps
• online support communities
• online courses and phone services
• chatbot for pointing people in the right direction.
28. Crowdsourcing - France
• In France, use of crowdsourcing in policy-making to enact its new
digital law. Over 20,000 people and organisations to go online to vote
and comment on the text of the digital law.
• DemocracyOS - Debates were sorted around 6 main categories:
• citizen initiatives that can inspire the Parliament
• early stage consultations about current bills
• interactions with the legislative procedure
• participation to the evaluation of law implementation
• combination of online and offline participation
• other ideas
29. EGDI components
• Online Service Index (OSI)
• Telecommunication Infrastructure Index (TII)
• Human Capital Index (HCI)
• EGDI =1/3 (OSI +TII + HCI)
30. Telecommunication Infrastructure Index (TII)
1. estimated internet users per 100 inhabitants
2. number of main fixed telephone lines per 100 inhabitants
3. number of mobile subscribers per 100 inhabitants
4. number of wireless broadband subscriptions per 100 inhabitants;
5. number of fixed broadband subscriptions per 100 inhabitants.
Data source: The International Telecommunication Union
31. Human Capital Index (HCI)
The Human Capital Index (HCI) consists of four components, namely:
1. adult literacy rate
2. the combined primary, secondary and tertiary gross enrolment ratio
3. expected years of schooling
4. average years of schooling
Data source: United Nations Educational, Scientific and Cultural
Organization (UNESCO)
32. Online Service Index (OSI)
• Assessors: Researchers, including UN experts and online United
Nations Volunteers (UNVs)
• In the native language
• What: national portal, e-services portal, e-participation portal,
sectorial portals
• Each country is assessed by at least two researchers.
• Relevant features could be found and accessed easily
33. OSI Measurements
• Each question calls for a binary response.
• The total number of points scored by each country is normalised to
the range of 0 to 1.
34. OSI Indicators
• National government portal
• List of all services in the national portal
• Guidance or tutorials to citizens in understanding and using online services
• Features available online allowing communication between citizens and government to improve online
services
• Open Government datasets available on education
• Sources of archived information (policies, budget, legal documents etc)
• Submit online for Income taxes
• Register for: Job seeking services
• Register online for: Vehicle (car, truck, motorcycle, and others)
• Register online for: A new company or business entity registration
• Apply online for: Birth certificates
• Apply online for: Drivers licenses
• Pay online for: Any fees for government services or fines (such as motor vehicle violations)
• Online government service available to the following vulnerable groups:( poor disabled elderly
immigrants women youth)
• Find results of any government procurement/bidding process online
• Information about any upcoming e-Participation activities
• Datasets available in machine-readable non-proprietary formats
35. EGDI Ranking
2016 2018
Country EGDI Country EGDI
United Kingdom 0.9193 Denmark 0.9150
Australia 0.9143 Australia 0.9053
Republic of Korea 0.8915 Republic of Korea 0.9010
Singapore 0.8828 United Kingdom 0.8999
Finland 0.8817 Sweden 0.8882
Sweden 0.8704 Finland 0.8815
Netherlands 0.8659 Singapore 0.8812
New Zealand 0.8653 New Zealand 0.8806
Denmark 0.8510 France 0.8790
France 0.8456 Japan 0.8783
36. e-Participation Index (EPI)
• e-Information: Enabling participation by providing citizens with
public information and access to information without or upon
demand
• e-Consultation: Engaging citizens in contributions to and deliberation
on public policies and services
• e-Decision-making: Empowering citizens through co-design of policy
options and coproduction of service components and delivery
modalities.
37. EPI Ranking
2016 2018
Country EGDI Country EGDI
United Kingdom 1.0000 Republic of Korea 1.0000
Japan 0.9831 Denmark 1.0000
Australia 0.9831 Finland 1.0000
Republic of Korea 0.9661 Netherlands 0.9888
Netherlands 0.9492 Japan 0.9831
New Zealand 0.9492 New Zealand 0.9831
Spain 0.9322 Australia 0.9831
Singapore 0.9153 Spain 0.9831
Canada 0.9153 United Kingdom 0.9831
Italy 0.9153 United States of America 0.9831
39. Criteria Groups
• Technology criterion, basic features of the website are assessed;
• Content Provision criterion, existence of essential information is
examined;
• Services Provision, delivery of fundamental electronic services is
assessed;
• Participation and Engagement the existence of relevant participation
and engagement mechanisms and initiatives.
40. 60 indicators
• Each of the 60 indicators is ascribed a “value 1” if it is found in a
municipality website, “value 0” if it is absent and nothing if it is not
applicable.
• The LOSI value of a municipality is the sum of the values of all the 60
indicators for that municipality.
41. Technology criterion
• Is MGP accessible through mobile devices?
• Is a sitemap (or index) available on the MGP?
• Is a search feature available on MGP?
• Is MGP compliant with the markup validity standards set by the World
Wide Web Consortium (W3C)?
• Is MGP compliant with Web Content Accessibility Guidelines
(WCAG1.0)?
42. Content Provision criterion
• Is there a "contact us" feature on MGP (e.g. email, phone, address)?
• Is information available on the MGP about the organizational
structure or chart?
• Does MGP provides sources of information (budget or policy) related
to its municipality budget?
• Does the website provide information about health issues?
• Does the website provide information about environmental issues?
• Is there a privacy policy or statement available on the MGP?
• Is there a Open Government data policy available online?
43. Services
• Is there any kind of authentication required to access online services
and restricted access areas on the MGP (e.g. digital ID; login and
password; mobile key)?
• Can a person modify its own data online?
• Does the municipality respond to a question sent by email?
• Does the government provide an e-procurement platform for bidding
processes / submission of tenders?
• Can users make a declaration to the municipality police online?
• Can users notify the government of moving/changing an address
online?
44. Participation
• Are there any online tools available on the MGP to collect public
opinion from citizens to inform policy deliberation processes?
• Does the portal have any social networking feature(s)?
• Does the portal provide participation possibility to the user regarding
participatory budgeting?
• Does the government indicate any outcome of e-consultation resulted
in a new policy decision/regulation/service?
45. Implementation of Technology indicators in
municipalities’ websites
100%
95%
90%
88%
83%
65%
60%
58%
55%
55%
43%
40%
Browser compatibility
Mobile device accessibility
Ease of portal finding
Internal search mechanism
Navigability
Foreign Language support
Internal advanced search mechanism
Alignment with accessibility standards
Alignment with markup validation standards
Alignment with display standards
Portal loading speed
Customization of display features
46. Implementation of Content Provision
indicators in municipalities’ websites
100%
98%
95%
90%
90%
88%
85%
85%
83%
83%
83%
80%
80%
73%
70%
68%
68%
68%
68%
65%
63%
60%
60%
58%
55%
45%
Names and contacts of heads of departments
Contact details
Municipality information
Information about provided services
Health information
Education information
Environmental information
Sport and culture information
Organization structure
Social welfare information
Links for government agencies
Budget related information
Procurement annoucements information
Open data provision
Evidence of portal content update
Facilitation of free internet access
Privacy policy
Evidences of smart cities initiatives
Statistical data and studies
Online user support
Procurement results information
Open data policy
Information on online services use
Open data metadata
Third parties partnerships information
Evidences of emerging technologies use
47. Implementation of Services Provision
indicators in municipalities’ websites
68%
60%
55%
55%
50%
48%
45%
38%
38%
35%
35%
25%
25%
Portal authentication
e-Procurement service
Online vacancies
e-Payment of service fees or fines
Online building permit
Municipality responsiveness to emails
Personal data accessibility
Police online declaration
Quality of email response
Personal data updating
Address change notification
Online residentship
Delay of email response
48. Implementation of Participation indicators in
municipalities’ websites
85%
85%
55%
53%
48%
43%
40%
35%
23%
Feedback/complaint submission
Social networking features
Online deliberation processes
Announcement of upcoming e-participation activities
Reporting of occurrences in public spaces
Real time communication
Feedback about consultation processes
Participatory territorial master plan
Participatory budgeting
51. Member States Questionnaire
(MSQ)
The objective of this questionnaire is to gather information from the Member
States in preparation of the United Nations E-Government Survey 2020.
52. Institutional Framework
• URLs for specific portals (e-Services, e-Participation, Open
government data, Public procurement etc.)
• Name(s) and URL(s) of the government agency/department/ministry
at the national level in charge of e-Government.
• Chief Information Officer (CIO) to manage national cross-agency e-
government programs/strategies
• Names and URLs of the government
agencies/ministries/departments at the national level in charge of
the following (education, health, environment etc.)
53. Strategy and Implementation
• National development strategy incorporating the Sustainable Development Goals (SDGs)
• National e-Government strategy/digital readiness strategy
• National e-government strategy or equivalent:
• has an implementation plan.
• is aligned with the national development strategy
• is aligned with the Sustainable Development Goals (SDGs).
• is aligned with sub-national/local digital development strategy.
• has an emphasis on digital-first principle
• has an emphasis on digital by default; digital by design; mobile-first principle
• has an emphasis on once-only (data) principle
• has an emphasis to ‘leave no one offline’ or to ‘leave no one behind’; or other specific measures to
ensure e-government is accessible by the most vulnerable groups
• makes specific reference to e-participation, digital inclusion and/or engagement.
• makes specific reference to the use of social media in the government.
• makes specific reference to the use of new technologies such as artificial intelligence, blockchain,
big data
54. Legal Framework
• Is there any legal framework on
• access to information such as Freedom of Information Act
• personal data protection including digital security
• open government data
• digital identity
• digital certification/signature
• e-procurement
• digitally publishing government expenditure
• data interoperability
• digital government as a right
55. Usage of online services
• Do you collect usage statistics of e-Government services?
• If yes, do you publish results online and share those with the public
institutions concerned?
User satisfaction
• Do you measure satisfaction of citizens on e-government services?
• If yes, do you publish results online and share those with the public
institutions concerned?
56. Social Media
• How does your government use social media at the national level to
interact with and engage people in e-Government activities? Please
also explain if there were any guidelines for government officials on
the use of social media.
57. New Technologies
• Does your government have a specific national strategy on one or more of
following new technologies?
• Artificial Intelligence
• Blockchain
• Big data
• Smart cities
• Robotics
• Internet of Things
• Quantum computing
• Virtual reality
• Augmented reality
• Does your government have any government body at the national level
working specifically related to the new technologies?
58. Indicators use
• What is the percentage of the population satisfied with their last
experience of online public services?
• What percentage of your GDP is allocated for ICT investment in the
public sector?
• What is the proportion of persons employed in central government
organizations routinely using ICTs?
• What is the proportion of persons employed in central government
organizations routinely using the Internet?
• If any, what kind of indicators do you collect/use to track digital
literacy at the national level?
59. International and Regional Cooperation
• Is your government part of any sub-regional, regional or international
cooperation on e-Government?
• Is your government offering (or planning to offer) support to other
countries in the area of e-government?
• Are there any ongoing public-private partnerships and
multistakeholder partnerships, focusing on e-Government?
60. Contents
• e-Government Assessment Dimensions
• Indicators Development
• e-Government Assessment Methods
• United Nations
• e-Government Development Index (EGDI)
• Local Online Service Index (LOSI)
• Member States Questionnaire (MSQ)
• European Commission
• The Digital Economy and Society Index (DESI)
• National Interoperability Framework Observatory (NIFO)
• Sectorial Assessment Instruments
• Hospital Website Assessment Instrument (HWAI)
61. The Digital Economy and Society Index (DESI)
A composite index
• Summarises relevant indicators on Europe’s digital performance and
• Tracks the evolution of EU member states in digital competitiveness.
• Data are mostly collected by the European Commission services (DG CNECT,
Eurostat) and by ad-hoc studies launched by the Commission services.
63. 1. Connectivity
Deployment of broadband infrastructure and its quality. Access to fast
and ultrafast broadband-enabled services.
• Fixed Broadband
The extent to which they connect to the Internet via a fixed broadband connection
• Mobile Broadband
How widely 4G services are available and used (4G coverage)
• Fast broadband
Availability and use of fast Internet connections (at least 30 Mbps download speed)
• Ultrafast broadband
Availability and use of ultrafast Internet connections (at least 100 Mbps download
speed)
• Broadband price index
How affordable it is to have a broadband Internet connection
64. Households with a fast broadband (at least 30
Mbps) subscription (% of households), 2018
65. 2. Human Capital/Digital skills
Measures the skills needed to take advantage of the possibilities
offered by digital technologies.
• Basic Skills and Usage
Captures the digital skills level of the general population.
• Advanced skills and Development
Concerns the workforce and its potential to maintain and grow the digital
economy.
67. 3. Use of Internet Services by citizens
Accounts for a variety of online activities, such as:
• Content
Measures the extent to which a country’s Internet users get online content
(videos, music, games, etc.) via their broadband connections.
• Communication
Measures the extent to which a country’s Internet users communicate and
interact online (e.g. video calls) using their broadband connections.
• Transactions
Captures the propensity of Internet users to perform transactions online
(online shopping and banking).
69. 4. Integration of Digital Technology by
businesses
Measures the digitisation of businesses and e-Commerce. By adopting
digital technologies, businesses can enhance efficiency, reduce costs
and better engage customers and business partners. Furthermore, the
Internet as a sales outlet offers access to wider markets and potential
for growth.
• Business digitization
Takes stock of the level of adoption of digital technologies by businesses
• e-Commerce
Focuses on the exploitation of the online sales channel by small and medium
enterprises
71. 5. Digital Public Services
Measures the digitisation of public services, focusing on eGovernment
and eHealth. Modernisation and digitisation of public services can
lead to efficiency gains for the public administration, citizens and
businesses alike.
• eGovernment
Captures the level of development of a country’s eGovernment services
• eHealth
Captures the percentage of people who used health and care services provided
online without having to go to the hospital or doctors surgery (for example, by
getting a prescription or a consultation online)
73. Weights attributed to the DESI dimensions
Dimension Weight
1 Connectivity 25%
2 Human Capital 25%
3 Use of Internet Services 15%
4 Integration of Digital Technology 20%
5 Digital Public Services 15%
74. NIFO provides policy makers, researchers, and business stakeholders with the latest
developments on digital government and interoperability across Europe.
75. NIFO Factsheets
Information on interoperability.
Tool for aligning NIFs with the EIF
Country factsheets
Data Source: Eurostat, e-Government Benchmark, Questionnaire to national
representatives
• NIFO factsheets
The NIFO factsheets describe the latest interoperability policies and
initiatives in European countries.
• eGovernment factsheets
The eGovernment factsheets describe the latest eGovernment policies and
initiatives in European countries and at the EU level.
77. EIF principles
• Subsidiarity and proportionality
• Openness
• Transparency
• Reusability
• Technological neutrality and data portability
• User-centricity
• Inclusion and accessibility
• Security and privacy
• Multilingualism
• Administrative simplification
• Preservation of information
• Assessment of Effectiveness and Efficiency
80. Interoperability Agreements
• Formal arrangements for cooperation through interoperability
agreements.
• Standards, specifications, legislation at EU or national level or via
bilateral and multilateral agreements.
• Agreements to address operational matters. For example,
memoranda of understanding (MoUs), service level agreements
(SLAs), support/escalation procedures and contact details, referring, if
necessary, to underlying agreements at semantic and technical levels.
81. Interoperability Governance
Identifying and selecting standards and specifications
• Identifying candidate standards and specifications based upon specific
needs and requirements;
• Assessing candidate standards and specifications using standardised,
transparent, fair and non-discriminatory methods;
• Implementing the standards and specifications according to plans and
practical guidelines;
• Monitoring compliance with the standards and specifications;
• Managing change with appropriate procedures;
• Documenting standards and specifications, in open catalogues, using a
standardised description.
83. eGov infographics
• Facilitate the communication of their content,
• Key indicators:
• Usage of Internet to interact with Public authorities, obtain info, download or
send forms;
• eGov state of play indicators: User centricity, Transparency, Citizen & business
cross border mobility and eGov key enabler;
• eGov Services to business and citizens
• Strategy & Legal Framework
• eGov Actors
• Infrastructure
87. Assessment Levels
Criterion 1
Indicator 1.1
Sub Indicator
1.1.1
Sub Indicator
1.1.2
Indicator 1.2
Sub Indicator
1.2.1
enable measurement of progress
towards the achievement of the key
objectives for each criterion
cover the whole spectrum of
the identified assessment
elements of our literature
review
Each indicator is assessed based on the
presence of certain sub-indicators
characterizing them
88. Hospital Web Assessment Instrument (HWAI)
Content Technology Services Community Interaction
Hospital Information
Quality Metrics
Organisational Structure
and Medical Information
Patient Information
Research and Teaching
Navigability
Accessibility
Usability/Readability
Credibility
Privacy/Security
Administration
Procedures
Appointments
Patient Care
Participation
Media
Advertising/Marketing
Criteria
Indicators
90. Hospital Information
• Hospital history
• Hospital name and logo on page header
• Welcome message
• Statement of purpose (Mission, Vision, Values)
• Hospital annual report for the previous year
• Financial data availability
• Quality Management Certification (e.g. ISO, EFQM)
• Hospital contact information (postal address, telephone, fax, e-mail)
• Hospital VAT number
• Map of the hospital area
• Ways of reaching the hospital location (e.g. car, public transport)
• Area covered by the hospital (population served)
• Complementary services (e.g. press, cafeteria, television, telephone, parking,
religious service)
• Phone directory
• Emergency information
91. Quality Metrics
• Number of hospital beds disclosed
• Waiting list disclosed
• Waiting time consultation
• Waiting time surgery
• Date of last monitoring of the waiting list disclosed
• Hospital report of the number of admissions in the previous year
• Nosocomial infection rate disclosed
• Inpatient mortality rate disclosed
• Surgical mortality rate disclosed
• Information on births per year
• The site provides patients' data (anonymized)
• Results of surveys regarding patient satisfaction are provided
92. Organisational Structure and Medical
Information
• Organization chart (medical management, nursing management, hospital management)
• Services charter
• Listing of services available at the hospital
• Detailed list of outpatient hospital services available (consultation, diagnostic services)
• List of departments or units providing user services
• Location of departments or units providing user services
• Telephone and/or fax number and/or e-mail address of departments or units providing user
services
• Working hours of departments or units providing user services
• List of employed doctors (alphabetical order, specialization)
• Doctors' curricula/information disclosed
• Photos of the medical team (physicians, nurses) available
• Possibility to read online or to download health-care booklets
• Medical glossary available
• Conditions and treatments
93. Patient Information
• Information regarding patient privacy
• Patient's rights and obligations
• Different types of admissions are disclosed
• Information and rules to be followed on admission, during the hospital stay, on
patient discharge, by visitors
• Information and procedure for obtaining a copy of the medical documentation
• Location of patient care service or unit
• Business hours of patient care service or unit
• Telephone and/or fax of patient care service or unit
• Details of how to pay prescription charges or fees
• Information about private consultations/services and fees
• List of consultations/services with fees available
• Information for foreigner patients is provided
94. Research and Teaching
• Scientific studies that the hospital promotes or is involved in
• Publications of the hospital itself
• Undergraduate or postgraduate courses that are held at the hospital
• Schedule of activities that take place at the hospital (e.g. courses,
congresses, conferences)
• Library details (address, business hours, publications catalogue,
services available (e.g. reading, loans, copies))
96. Navigability
• Website name appears on browser title bar
• Active part of the site appears on browser title bar
• Best browser version for the website is indicated
• Interwebsite links are distinguished from intrawebsite links
• Interwebsite links show a full description of the linked website
• Functioning intrawebsite and interwebsite links
97. Accessibility
• Compliance with level WCAG 1.0 W3C
• Website is validated through W3C CSS Validation Service
• Website is validated through W3C Markup Validation Service
• Website listed on the first page of results after performing a Google search with
its name
• For individual sub-pages, there is a specific and meaningful description provided
via the META/description tag
• Website is compatible with the most popular browsers (Internet Explorer, Mozilla
Firefox, Google Chrome, Safari)
• Access from mobile devices
• The website provides useful/useable information even when CSS, JavaScript, and
images are disabled or missing
98. Usability/Readability
• Website map available
• Website search engine
• Access to the website in foreign languages
• Website load time (<5 sec)
• Illustrations/pictures/photos accompany text to assist description
• Graphics open conveniently (images/graphics are quick to load)
• Website pages can be printed
• Individual sub-pages have specific and meaningful titles
• The layout of the website is responsive (i.e. does it adapt to varying screen sizes), or there is a
separate version for mobile devices
• The website offers means to adjust (increase) the text size without compromising the
functionality of the website
• The website offers means to adjust (increase) the contrast of textual information for visitors with
visual impairments
• The website provides a (consistently accessible) menu structure for navigating the department’s
sub-pages
• Website does not include pop-up advertising
99. Credibility
• The text is grammatically correct
• The text does not have spelling errors
• Interest conflict declaration are shown
• Date of last website update
• Pages have dates associated with them (there are indications of
updates to materials)
• Website has HON (Health on the Net) foundation code certification
• Webmaster characteristics
• Sources and references are clearly listed
102. Administration Procedures
• Provision of online forms
• Possibility of forms downloading
• Possibility of filled forms uploading
• Provision of digitally signed documents
• Acceptance of digitally signed documents
103. Appointments
• Medical examination appointment via e-mail / web form
• Schedule admission via e-mail / web form
• Schedule visits to outpatient consulting rooms via e-mail / web form
• Electronic payment
104. Patient Care
• Asynchronous communication with the doctor via message exchange
system / e-mail
• Electronic directory with patient's exams
• Interactive communication tool (chat with the doctor), synchronous
communication
• Provision of telemedicine (video-conference system) services
• Patient telemonitoring (e.g. specific vital signs, blood glucose, peak
flow rate, blood/urine chemistry)
• Availability of membership or signing in area (Customer Area)
106. Participation
• Suggestions/complaints via web form / e-mail
• Information request via web form / e-mail
• Information about job opportunities at the hospital
• Discussion forum is present
• If the website has a forum it is possible to identify the administrator
• If the website has a forum the access is identified
• FAQ
• Associations that work at the hospital (e.g. voluntary associations,
patient associations, associations for the defense of patients' rights)
107. Media
• The hospital in the media (press review, news that appeared in
press/radio/TV)
• Website provides an up-to-date news/events schedule/newsletter
• Virtual visit to the hospital
• Links to other websites of interest (e.g. hospitals, scientific societies,
institutions)
• Work hours of public relations office
• Location of public relations office
• Telephone and/or fax number and/or e-mail address of public
relations office
• Hospital news (hospital’s innovations, new techniques used by the
centre specialists, infrastructure improvement)
108. Marketing/Advertising
• Website sponsors and investors are disclosed
• Advertising and contents are differentiated
• Advertising is not contradictory with respect of the website contents
• Information on how to donate to the hospital
• Facebook use
• Twitter use
• LinkedIn use
• Affiliated insurance companies
109. The ‘Big Picture’ in Portugal
Indicator Average percentage of Sub Indicators coverage
Accessibility (T) 74%
Hospital Information (C) 72%
Navigability (T) 64%
Usability/Readability (T) 64%
Patient Information (C) 55%
Organisational Structure and Medical Information (C)
53%
Media (P) 45%
Community Interaction (P) 42%
Privacy/Security (T) 38%
Credibility (T) 33%
Quality Metrics (C) 25%
Research and Teaching (C) 17%
Administration Procedures (S) 14%
Advertising/Marketing (P) 14%
Communication with Others (S) 5%
Appointments (S) 4%
Patient Care (S) 3%
Inter-Hospital Communication (S) 0%
110. HWAI Perspectives
• The instrument has been developed based on relative research literature, tested and revised in real
assessment cases.
• HWAI:
• offers to hospital management aggregated and at the same time scientifically sound information to
improve their provided services
• offers to citizens the possibility to compare and select health service provider
• assists health sector policy-makers to monitor and further develop eHealth policy
• Be of interest to technology practitioners and researchers, as the findings shed light on the further
development of performance measurements for hospital websites.
• Future work:
• Determine relative weights for each indicator and sub-indicator.
• Apply it in other countries hospitals’ website assessment and discuss the results
• HSWAI could be applied in country level indicating different trends and policies. It could be used to identify
areas where there is still room for health sector improvement.
catalogue, where governments arrange information in a catalog
Vertical integration. This stage, involves integration with higher level systems within similar functionalities or jurisdictions.
This stage initiates the transformation of government services rather than automating its existing processes. It focuses on integrating government functions at different levels, such as those of local governments and state governments.
Connected: governments transform themselves into a connected entity that responds to the needs of its citizens by developing an integrated back office infrastructure. gue accessible by citizens and businesses
Efficiency The IT-enabled government processes are expected to simplify procedures, execute faster, minimize use of papers and save costs while communicating with government. The enhanced efficiency is captured through this variable.
Transparency The variable encompasses transparency aspect of a service. An e-governance service is expected to bring transparency in government-controlled operations. A government service has to be trustworthy, thorough, unbiased and accessible without any difficulty to end users.
Interactivity An e-governance service targeting citizens is expected to facilitate interactions at various levels, i.e. within constituting units of a government department, with other departments associated with the service and with recipient of the service. The variable is conceptualized to capture such interactions.
Decision support Digitization of services and online transactions contribute to better decision-making, monitoring and control at the level of officials as well as beneficiaries, which is captured through this variable. For example, a farmer who has online access to commodity prices/arrivals information and storage facilities can monitor prevailing prices, store his produce or select a market for selling his produce when conditions are favourable. This micro-variable reflects better decision support in terms of improved planning and decision-making
The scope of e-government has been noted as follows (Heeks, 2001):
•e-administration – improving government processes by reducing costs, managing performance, making strategic connections within government, and creating empowerment.
•e-citizens and e-services – connecting citizens to government by communicating with citizens, supporting accountability by listening to citizens, supporting democracy, and improving public services.
•e-society – building interactions beyond the boundaries of government by working better with business, developing communities, building government partnerships, and building society.
possible sources for gathering related data are identified.
Stakeholders - to understand and engage the stakeholders in e-Government planning for successful implementation.
The stakeholders are required to participate in e-Government Readiness Assessment to supply critical information for strategy development including specific business requirements and drivers for IT, and expectations regarding e-services.
The scope of e-Government Assessment defines the levels of government covered by the assessment, helping to estimate the human resources and the time-frame required for the survey exercise and to ensure proper survey and instruments design.
instruments to gather data from different sources
Environment sub dimension assesses the adequacy of the public administration environment in order to successfully implement and support the digital transformation in a country. It includes indicators such as government policy and vision, leadership and political commitment, social and cultural environment, human capital, institutional capacities, legal environment etc.
Infrastructure sub dimension assesses the technical capability of public sector to support the required operations and to provide the expected services. This sub-dimension includes indicators like service maturity, connectivity and technology infrastructure, cross-border mobility capacity, key enablers, telecommunications infrastructure etc.
User Centric sub dimension evaluates citizens' and businesses' satisfaction regarding the accepted eGovernment services. This sub-dimension includes indicators like citizen service experience, citizen satisfaction survey, consumer and business adoption level, online service index etc.
User Engagement sub dimension assesses the level of citizen and business participation in digital transformation. This sub-dimension includes indicators like transparent government, openness and inclusiveness, engagement and participation, coherent use of digital technologies etc.
Most experts look at the “what” questions—what are the goals?
what are the indicators?—and not the “why” questions: Why do we
want to measure something? Why is there a need in a particular
country to think about these issues? Why do we want to embark on
building sustainable results-based M&E systems?
When trying to find an indicator or sign of change, start with the question: “How will I know or what will I see if this specific result has happened?” There are a number of guidelines that can assist you in
developing useful indicators. Four useful guidelines are that indicators should:
1. Begin with a measure;
• There are various measures you could choose to use – a number, a percentage or a ration.
For example:
Incorrect: Learners in need of financial assistance
Correct: Number of learners in need of financial assistance or
% of learners in need of financial assistance
Indicators are neutral in terms of what should be achieved. In other words:
• Do not specify a direction in which you would like to see the change occur, for example
increase or decrease.
• Do not specify quantities - these are specified in your target
For example:
Incorrect: Increased number of learners enrolled in FET institutions or
80% of learners enrolled in FET institutions
Correct: Number of learners enrolled in FET institutions
3. Specify just one result per indicator:
• Do not combine two results in one indicator
For example:
Incorrect: Number of teachers trained and career guidance resources distributed
Number of learners who have received career guidance and been awarded a bursary.
Correct: Number of teachers trained
Number of career guidance resources distributed
4. Indicators are specific and unambiguous:
• Do not use “suitcase words”: unpack them.
For example:
Incorrect: Number of demoralized teachers
Correct: Number of teachers reporting burnout
In addition, the Government Wide M&E System specifies a number of criteria for indicators.
An indicator should be:
Reliable: the indicator should be accurate enough for its intended use and respond to changes in the level of performance.
Well-defined: the indicator needs to have a clear, unambiguous definition so that data will be collected consistently, and be easy to understand and use.
Verifiable: it must be possible to validate the processes and systems that produce the indicator.
Cost-effective: the usefulness of the indicator should justify the cost of collecting the data.
Appropriate: the indicator should avoid unintended consequences and encourage service delivery improvements, and not give managers incentives to carry out activities simply to meet a particular target.
Relevant: the indicator should relate logically and directly to an aspect of the institution's mandate, and the realisation of strategic goals and objectives20.
national government websites were analysed for the content and services available that the average citizen would most likely use. The
presence, or absence of specific features contributed to determining a country's level of progress.
https://blogs.microsoft.com/eupolicy/2019/05/10/electronic-voting-estonia/
The government began the legislative process in 2001 and introduced the new voting system in 2005. By 2002, Estonia had also introduced an ID card system and by 2005 almost 80% of the electorate had this ID card. At the time, Estonians were saying they did everything with their computer – their banking, taxes, signing documents – and asked: “why not voting?”
The United Kingdom deployed new web technologies such as HTML5, as part of the aim to make its national portal GOV.UK “accessible to the widest possible audience. This achievement was also won through efficiency gains which resulted in savings of £1.7bn in 2014 through its digital and technology transformation. Eighty-five per cent of tax filing is now done through online channels and over 98 per cent of driving tests are now booked online
In management terms, an indicator is a variable that is used to assess the achievement of results in relation to the stated goals/objectives
The e-participation index (EPI) is derived as a supplementary index to the UN E-Government Survey. It extends the dimension of the Survey by focusing on the use of online services to
facilitate provision of information by governments to citizens (“e-information sharing”),
interaction with stakeholders (“e-consultation”),
and engagement in decision-making processes (“e-decision making”).
Deployment of broadband infrastructure and its quality. Access to fast and ultrafast broadband-enabled services is a necessary condition for competitiveness.
Fixed Broadband
Whether citizens have the possibility to connect to the Internet via a fixed broadband connection, and on the extent to which they do in fact connect to the Internet that way
Mobile Broadband
How widely 4G services are available (4G coverage) and whether citizens use the broadband capabilities of their mobile devices
Fast broadband
The availability and use of fast Internet connections (defined as those offering at least 30 Mbps download speed).
Ultrafast broadband
The availability and use of ultrafast Internet connections (defined as those offering at least 100 Mbps download speed).
Broadband price index
How affordable it is to have a broadband Internet connection
The interoperability principles are fundamental behavioural aspects to drive interoperability actions. This chapter sets out general interoperability principles which are relevant to the process of establishing interoperable European public services. They describe the context in which European public services are designed and implemented.
Ensure that national interoperability frameworks and interoperability strategies are aligned with the EIF and, if needed, tailor and extend them to address the national context and needs.
Publish the data you own as open data unless certain restrictions apply.
Ensure internal visibility and provide for European public services.
Reuse and share solutions, and cooperate in the external interfaces development of joint solutions when implementing European public services
Do not impose any technological solutions on citizens, businesses and other administrations that are technology-specific or disproportionate to their real needs.
Use multiple channels to provide the European public service, to ensure that users can select the channel that best suits their needs.
Ensure that all European public services are accessible to all citizens, including persons with disabilities, the elderly and other disadvantaged groups.
Define a common security and privacy framework and establish processes for public services to ensure secure and trustworthy data exchange between public administrations and in interactions with citizens and businesses.
Use information systems and technical architectures that cater for multilingualism when establishing a European public service.
Simplify processes and use digital channels whenever appropriate for the delivery of European public services, to respond promptly and with high quality to users’ requests and reduce the administrative burden on public administrations, businesses and citizens
Formulate a long-term preservation policy for information related to European public services and especially for information that is exchanged across borders.
Evaluate the effectiveness and efficiency of different interoperability solutions and technological options considering user needs, proportionality and balance between costs and benefits.
Public administrations need to identify, negotiate and agree on a common approach to interconnecting service components.
Decide on a common scheme for interconnecting loosely coupled service components and put in place and maintain the necessary infrastructure for establishing and maintaining European public services.
The conceptual model promotes the idea of interoperability by design. It means that for European public services to be interoperable, they should be designed in accordance with the proposed model and with certain interoperability and reusability requirements31 in mind.
Internal information sources and services
Develop a shared infrastructure of reusable services and information sources that can be used by all public administrations.
Building blocks (information sources and services) should make their data or functionality accessible using service-oriented approaches.
Catalogues
Put in place catalogues of public services, public data, and interoperability solutions and use common models for describing them.
Security and privacy
Consider the specific security and privacy requirements and identify measures for the provision of each public service according to risk management plans.
Coordination function
The coordination function ensures that needs are identified and appropriate services are invoked and orchestrated to provide a European public service. This function should select the appropriate sources and services and integrate them. Coordination can be automated or manual. The following process phases are part of ‘integrated public service provision’ and executed by the coordination function.
Ιntegrated Public Services: To achieve public services integration at EU level, compatible models, standards and common infrastructure need to be agreed.
Public administrations need to exploit services delivered outside their organisational boundaries by third parties, such as payment services provided by financial institutions or connectivity services provided by telecommunications providers. They need also to exploit external information sources such as open data and data from international organisations, chambers of commerce, etc. Moreover, useful data can be collected through the Internet of Things (e.g. sensors) and social web applications.
Legal interoperability is about ensuring that organisations operating under different legal frameworks, policies and strategies are able to work together. This might require that legislation does not block the establishment of European public services within and between Member States and that there are clear agreements about how to deal with differences in legislation across borders, including the option of putting in place new legislation.
Ensure that legislation is screened by means of ‘interoperability checks’, to identify any barriers to interoperability. When drafting legislation to establish a European public service, seek to make it consistent with relevant legislation, perform a ‘digital check’ and consider data protection requirements.
Organisational interoperability
This refers to the way in which public administrations align their business processes, responsibilities and expectations to achieve commonly agreed and mutually beneficial goals. In practice, organisational interoperability means documenting and integrating or aligning business processes and relevant information exchanged. Organisational interoperability also aims to meet the requirements of the user community by making services available, easily identifiable, accessible and user-focused.
Document your business processes using commonly accepted modelling techniques and agree on how these processes should be aligned to deliver a European public service.
Semantic interoperability ensures that the precise format and meaning of exchanged data and information is preserved and understood throughout exchanges between parties, in other words ‘what is sent is what is understood
Technical interoperability
This covers the applications and infrastructures linking systems and services. Aspects of technical interoperability include interface specifications, interconnection services, data integration services, data presentation and exchange, and secure communication protocols
Use open specifications, where available, to ensure technical interoperability when establishing European public services.
Organisations involved in European public service provision should make formal arrangements for cooperation through interoperability agreements. Setting up and managing these agreements is part of public service governance.
Agreements should be detailed enough to achieve their aim, i.e. to provide European public services, while leaving each organisation the maximum feasible internal and national autonomy.
Country profile: national description of each country presenting its basic data, political structure and key relevant indicators
Highlights: to put focus on the most significant change and progress achieved in 2017
Strategy: the main strategic objectives and principles at national level
Legal Framework: the main legal texts impacting on the development of eGovernment
eGovernment Actors: to list and explain the main roles and responsibilities
eGovernment Infrastructure: description of the technical infrastructure components supporting eGov implementation in the country
eGovernment Services for Citizens and Businesses: availability and sophistication of eServices for both citizens and businesses, grouped by thematic
Electronic Identification (eID): electronic identification solution to determine if the user is who he claims to be. Using eID enables online transactions, saves time and reduces costs for all actors involved.
Electronic Documents (eDocuments): an electronic document reduces offline paper processes by allowing citizens and businesses to send authenticated documents online.
Authentic Sources: base registries used by governments to automatically validate or fetch data relating to citizens or businesses. It facilitates prefilling of online forms and the implementation of the ‘once-only principle’, which implies that governments re-use data to deliver services automatically.
Digital Post: public administrations should allow citizens to receive mail in a digital format and help reduce paper mailing. Governments should provide the possibility to communicate electronically-only with citizens or entrepreneurs through personal mailboxes or other digital post solutions.
