This publication is a Science for Policy report by the Joint Research Centre (JRC), the European Commission’s science and knowledge service. It aims to provide evidence-based scientific support to the European policy-making process. The scientific output expressed does not imply a policy position of the European Commission. The inclusion of commercial products in the report is not an endorsement of these products in any way. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of this publication.

Contact information

Name: Yves Punie

Address: Edificio Expo, Calle Inca Garcilaso 3, E-41092 Seville, Spain

E-mail: Yves.PUNIE@ec.europa.eu

Tel.: +34 9544-88229

JRC Science Hub

https://ec.europa.eu/jrc

JRC Mission

As the science and knowledge service of the European Commission, the Joint Research Centre’s mission is to support EU policies with independent evidence throughout the whole policy cycle.

Europe Direct is a service to help you find answers to your questions about the European Union Free phone number (*): 00 800 6 7 8 9 10 11
(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed.

A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server http://europa.eu

How to obtain EU publications

Our publications are available from EU Bookshop (http://bookshop.europa.eu),
where you can place an order with the sales agent of your choice.

The Publications Office has a worldwide network of sales agents.
You can obtain their contact details by sending a fax to (352) 29 29-42758.

JRC104031

EUR 28294 EN

PDF | ISBN 978-92-79-64441-2 | ISSN 1831-9424 | doi:10.2791/955210
EPUB | ISBN 978-92-79-74184-5 | ISSN 1831-9424 | doi:10.2791/326911

Luxembourg: Publications Office of the European Union, 2016

© European Union, 2016

Reproduction is authorised provided the source is acknowledged.

How to cite: Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., Vuorikari, R. (2016). Research Evidence on the Use of Learning Analytics - Implications for Education Policy. R. Vuorikari, J. Castaño Muñoz (Eds.). Joint Research Centre Science for Policy Report; EUR 28294 EN; doi:10.2791/955210.

All images © European Union 2016

Research Evidence on the Use of Learning Analytics and Their Implications for Education Policy – Case Studies, inventory and Literature Review report

Learning analytics is an emergent field of research that is growing fast. It takes advantage of the last decade of e-learning implementations n education and training as well as of research and development work in areas such as educational data mining, web analytics and statistics. In recent years, increasing numbers of digital tools for the education and training sectors have included learning analytics to some extent, and these tools are now in the early stages of adoption. This report reviews early uptake in the field, presenting five case studies and an inventory of tools, policies and practices. It also provides an Action List for policymakers, practitioners, researchers and industry members to guide work in Europe.

LF-NA-28-294-EN-E

Research Evidence on the Use of Learning Analytics

Implications for Education Policy

Foreword

JRC research on Learning and Skills for the Digital Era started in 2005 with the aim to provide evidence-based policy support to the European Commission on harnessing the potential of digital technologies to innovate education and training practices; improve access to lifelong learning; and to deal with the rise of new (digital) skills and competences needed for employment, personal development and social inclusion. More than 20 major studies have been undertaken on these issues with more than 100 different publications.

Recent work on capacity building for the digital transformation of education and learning, and for changing requirements on skills and competences has focussed on the development of digital competence frameworks for citizens (DigComp), educators (DigCompEdu), educational organisations (DigCompOrg) and consumers (DigCompConsumers). A framework for opening-up Higher Education Institutions (OpenEdu) was also published in 2016, as well as a competence framework for entrepreneurship (EntreComp). Some of these frameworks are accompanied by (self-) assessment instruments. Additional research has been undertaken on computational thinking (CompuThink), Learning Analytics and MOOCs (MOOCKnowledge, MOOCs4inclusion).

This report aims to understand the state of the art in the implementation of learning analytics for education and training in both formal and informal settings. It also aims to understand the potential for European policy to be used to guide and support the take-up and adaptation of learning analytics to enhance education in Europe. This study, called the Implications and Opportunities of Learning Analytics for European Educational Policy (henceforward the Study), therefore has an international scope, although the policy perspectives are discussed from the point of view of the EU. The research was conducted between September 2015 and June 2016. The key findings seek to inform, guide and inspire practitioners, researchers and policy makers at all levels (institutional, local, regional, national, international) in implementing learning analytics in European education and training.

More information from all our studies can be found on the JRC Science hub: https://ec.europa.eu/jrc/en/research-topic/learning-and-skills.

Yves Punie

Project Leader
DG JRC Unit Human Capital and Employment
European Commission

Acknowledgements

The study described in this report was designed by the JRC (Unit B.4) to help European policymakers set out an agenda for stimulating high-quality, innovative ways of learning and teaching through the use of learning analytics. The study was carried out between September 2015 and June 2016 by The Open University (OU), based in the UK, and forms part of the University’s on-going commitment to the field of learning analytics.

As a distance-learning institution, the OU has been carrying out work on learning analytics since its first students enrolled in the 1970s. It has taken an active lead in many areas of this field, including the development of its ethics, and the linking of learning design with learning analytics (Ferguson, 2012). The university is currently engaged in a far-reaching strategic investment project in learning analytics, led by senior management.

The editors would like to thank not only those who carried out the study, but also those who participated in the expert workshops held in Amsterdam in March 2016, in which the preliminary results of the study were presented and further developments were discussed. A list of all participants can be found in Annex 5.

Disclaimer

The inclusion of commercial products and project names in this report is not an endorsement of these products or projects in any way.

Executive summary

Policy context

The Europe 2020 Strategy acknowledges that Education and Training (E&T) have a strategic role to play in helping Europe to remain competitive, overcome the current economic crisis and grasp new opportunities. Digital transformation of E&T systems is included in several Europe 2020 flagship initiatives and boosting digital skills and digital learning is among President Juncker’s priorities.

From 2013, the European Commission’s action plan Opening up Education has focused on challenges in the field of education, particularly on those that have been brought about by digitalisation of every aspects of our lives - including education and training.

“Technology makes it possible to develop new solutions for better personalised learning, by allowing teachers to have a more accurate and up-to-date follow up of each learner. Through learning analytics, new and more learner-centred teaching methods can emerge since the evolution of learners who use ICT regularly can be closely monitored.” (p.5)

One of the key transformative actions in this area has been to promote research and innovation on adaptive learning technologies, learning analytics and digital games for learning (European Commission, 2013).

The study described in this report aimed to find and document evidence on the implementation of learning analytics for education and training in order to better understand their implications and opportunities for European educational policy. A key outcome of the study is the Action List for Learning Analytics which offers educators, researchers, developers and policymakers a step-by-step list of actions to ensure that learning analytics will fully embrace open and innovative education and training.

Main findings

Learning analytics is a field of research that has developed over the last decade and continues to grow quickly. Though practical applications are beginning to emerge, the technology is still not widely used in educational settings. Learning analytics involve

the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs¹.

Learning analytics has its roots in many fields of educational and technical research, including assessment, personal learning and social learning, and also in business intelligence and data mining. It draws on theory and methodologies from disciplines such as statistics, artificial intelligence and computer science (Dawson et al., 2014).

What do we know about implementing Learning Analytics in Europe?

Between September 2015 and June 2016, the JRC-led study on “The Implications and Opportunities of Learning Analytics for European Educational Policy” gathered evidence of implementation of learning analytics in educational contexts. The focus was on the use and the processes of implementing learning analytics in any tier of education.

The study gathered evidence from two sources:

• An inventory of examples of tools, practices and policies from all tiers of the educational system, including informal and non-formal learning;

• Five case studies that provide insights into current and recent practices in the implementation of learning analytics focusing on understanding the enablers and obstacles for implementation.

Although the Inventory is not exhaustive, it illustrates the work currently being done and the kind of practical applications of learning analytics that are already possible today. Together, the examples of tools, practices and case studies show that work across Europe in the area of learning analytics is promising, but currently fragmented.

Regarding available tools and their usage to improve - and innovate - education, there is a wide gap between the potential roles for learning analytics that have been identified in research literature as a whole and the dominant themes in learning analytics as put into practice by ICT/learning technology vendors, developers and researchers. Firstly, much of the current work on learning analytics concentrates on the supply side – the development of tools, data, models and prototypes. There is considerably less work on the demand side – i.e. on how analytics connect with education and the changes that school administrators, teachers and students want these tools to make in order to support their everyday learning, teaching and assessment work. More attention needs to be paid to the demand side - like, for example, the work carried out by Kennisnet in the Netherlands. This sought to help schools articulate what they want from ICT vendors, mediating requirements and exploring possible solutions, thus ensuring that learning analytics products have useful features for their end users.

Secondly, tools seem to be focusing currently on visualising engagement and activity developing systems that provide early alerts and eventually target interventions. What can be seen, though, is that these data visualisations are not necessarily ‘actionable’ in the way that learning analytics should be. In other words, they do not reveal what actions should be taken to improve learning and teaching. Also, efforts focus mainly on identifying students who may drop out and less on innovative pedagogical processes and practices, or on helping educational organisations to fully embrace the digital era.

Another issue with current tools is finding evidence for their formal validation (e.g. whether the tools fulfil their intended purpose, such as having a positive impact on learning; encouraging more efficient learning; or more effective learning). The issue is partly related to the timeframe; very little hard evidence is currently available that is based on anything other than short-term studies. Some positive work is cited in the LACE Evidence Hub but, at this stage, there is no overwhelming evidence that learning analytics have fostered more effective and efficient learning processes and organisations. However, there is convincing evidence in the Inventory and case studies that companies and organisations believe they can do this in the future, and are prepared to invest time and resources in order to achieve this.

Some European countries, notably Denmark, the Netherlands and Norway, are beginning to develop national approaches and are creating infrastructure to support and enable endeavours in learning analytics. A few European universities, such as Nottingham Trent, Dublin City and The Open University, have developed implementations, some large-scale, others smaller-scale. We also find that organisations such as Kennisnet (NL), Jisc (UK), Apereo (international) and the LACE project (a European research network that reached the end of its project funding in June 2016) are helping many educational institutions and also companies in Europe to develop their capabilities in learning analytics.

However, these implementations do not seem to be widely known, and there seem to be only limited opportunities to share experience and good practice in the area of use and implementation of learning analytics in an educational context. In order for other educational institutions to follow the lead of these early adopters, and to encourage them to build on what they have already achieved, more work is needed on areas related to adoption and implementation.

From this study, we also learn that most policies that have an influence on learning analytics were developed in other contexts of educational technologies. Even though policies related to technical standards for interoperability already exist, many need to be amended or even replaced to take learning analytics into account. As regards data protection and privacy, Europe’s General Data Protection Regulation (GDPR) entered into force in May 2016 and it can be foreseen that it will affect learning analytics in many ways. As Europe has taken the position that individual privacy is important, some changes to current practices in general analytics are evident. Institutions will need to understand their responsibilities and obligations with regard to data privacy and data protection and will have to put procedures in place to ensure that they are compliant with the legislation. There will also be an increased need to help parents and students understand how data are used. This study has identified some pioneering work in this area.

Much of the work that is underway in Europe seems to address some of the strategic objectives or priorities at an institutional or regional level. However, at a higher level, there is a little coherence and convergence towards common topics and goals: for example, those of the new priorities for European cooperation in education and training (European Union, 2015). As a result, companies and researchers focus heavily on only some areas, e.g. reduction of drop-out rates and identification of at-risk students, while others, for example new and more learner-centred teaching methods, remain relatively untouched. In order to reap the potential benefits from modernising education systems and improving learning outcomes, work is needed to make links between learning analytics, European priority areas for education and training, and the beliefs and values that underpin these areas.

Key conclusions

The evidence shows that the use of learning analytics to improve and to innovate learning and teaching in Europe is still in its infancy. The high expectations, for example those outlined in the policy context above (‘through learning analytics, new and more learner-centred teaching methods can emerge’), have not yet been realised. Though early adopters are already taking a lead in research and development, the evidence on practice and successful implementation is still scarce. Furthermore, though the work across Europe on learning analytics is promising, it is currently fragmented.

This underlines the need for a careful build-up of research and experimentation, with both practice and policies that have a unified European vision. Therefore, the study suggests that work is needed to make links between learning analytics, the beliefs and values that underpin this field, and European priority areas for education and training 2020 (European Union, 2015). As a way of guiding the discussion about further development in this area, the Action List for Learning Analytics is proposed.

The Action List for Learning Analytics focuses on seven areas of activity. It outlines a set of actions for educators, researchers, developers and policymakers in which learning analytics are used to drive work in Europe’s priority areas for education and training. Strategic work should take place to ensure that each area is covered, that there is no duplication of effort, that teams are working on all actions and that their work proceeds in parallel.

Policy leadership and governance practices

• Develop common visions of learning analytics that address strategic objectives and priorities

• Develop a roadmap for learning analytics within Europe

• Align learning analytics work with different sectors of education

• Develop frameworks that enable the development of analytics

• Assign responsibility for the development of learning analytics within Europe

• Continuously work on reaching common understanding and developing new priorities

Institutional leadership and governance practices

• Create organisational structures to support the use of learning analytics and help educational leaders to implement these changes

• Develop practices that are appropriate to different contexts

• Develop and employ ethical standards, including data protection

Collaboration and networking

• Identify and build on work in related areas and other countries

• Engage stakeholders throughout the process to create learning analytics that have useful features

• Support collaboration with commercial organisations

Teaching and learning practices

• Develop learning analytics that makes good use of pedagogy

• Align analytics with assessment practices

Quality assessment and assurance practices

• Develop a robust quality assurance process to ensure the validity and reliability of tools

• Develop evaluation checklists for learning analytics tools

Capacity building

• Identify the skills required in different areas

• Train and support researchers and developers to work in this field

• Train and support educators to use analytics to support achievement

Infrastructure

• Develop technologies that enable development of analytics

• Adapt and employ interoperability standards

1 Introduction to the Report

Learning analytics is an emergent research field that is growing quickly. It involves:

the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs².

In other words, the field takes the data that are generated as people engage in learning, and uses these data to help improve learning and teaching.

The definition draws on the field of web analytics, in which user data are collected ‘for the purposes of understanding and optimizing web usage’ (Web Analytics Association, 2008). Just as its definition is drawn from another field of study, learning analytics is also rooted in many fields of educational and technical research dating back some 30 years into topics such as personal and social learning, and assessment, and in disciplines such as business intelligence and data mining. It also draws on theories and methodologies from statistics, artificial intelligence and computer science (Dawson et al., 2014).

The emergence of learning analytics as a field has been attributed to three principal drivers (Ferguson, 2012):

• Big data: the introduction of institutional databases and virtual learning environments (also known as learning management systems) means that educational institutions deal with increasingly large amounts of data, and are looking for ways of using these to improve learning and teaching.

• Online learning: The rise of Big Data in education is accompanied by an increase in take-up of online and blended teaching and learning, and by growth in the number of learners worldwide learning informally using open educational resources (OERs) and massive open online courses (MOOCs). There is therefore a worldwide interest in ways of optimising learning in these settings.

• National concerns: Countries and international groupings are increasingly interested in measuring, demonstrating and improving performance in education and are looking for ways to optimise learning and educational results in order to benefit society and the individuals within it.

The research community that has formed around learning analytics was stimulated by the first international conference on Learning Analytics and Knowledge in 2011 (LAK11). Following the publication of the LAK11 call for papers, the term ‘learning analytics’ (Figure 1, see blue upward curve) became increasingly more popular than in the initially more used term ‘educational data mining’ (Figure 1, see red line).

Figure 1: 2010-2016 Google trend analysis for ‘educational data mining’ (red) and ‘Learning Analytics’ (blue) shows interest in Learning Analytics increasing from 2011

A clear understanding of the global uptake of learning analytics is needed for the development of policies that can foster their potential to support more effective and efficient learning processes and organisations within the EU. The study ‘Implications and Opportunities of Learning Analytics for European Educational Policy’ (henceforward simply ‘the study’), therefore has an international scope, although the policy perspectives are discussed from the point of view of the EU. The research was conducted between September 2015 and June 2016.

The study addressed three research questions:

RQ1: What is the current international state of the art in the implementation of learning analytics for education and training in both formal and informal settings? (Answered in Section 2)

RQ2: What are the prospects for the implementation of learning analytics for education and training over the next 10–15 years? (Answered in Section 4)

RQ3: What is the potential for European policy to be used to guide and support the take-up and adaptation of learning analytics to enhance education in Europe? (Answered in Section 4)

The results of the study are documented in this report. Evidence was gathered for this study from two sources:

• An evidence-based inventory of the implementation of learning analytics in all tiers of the educational system, including informal and non-formal learning (Annex 1), and

• Five illustrative case studies that provide insights into current and recent practices in the implementation of learning analytics (Annex 2).

The study also included a review of literature related to the issue of implementation of learning analytics (not part of this report) and a brief overview of the issues that prompted the emergence of learning analytics from existing research in fields such as data mining, personal learning, assessment and social learning (Annex 3).

The vocabulary used in this emerging field is explained in the Glossary (Annex 4). It is split into sections and includes terms in general use, terms used by developers, and terms used by researchers.

2 The State of the Art in the Implementation of Learning Analytics for Education and Training

Evidence on the implementation of learning analytics in education and training, capturing the state of the art in this area, was gathered for this study. The scope is international and across educational sectors. In this section, the main results from the two sources of evidence, namely the Inventory and the case studies, are summarised. An expert workshop was organised to validate the research results.

What do we know about the current implementation of learning analytics?

The Inventory offers a short overview of 60 tools, practices and policies in the field of learning analytics. The aim was to collect evidence of practical implementations of learning analytics in the context of education and to document the state-of-play in early 2016, when the study was carried out. The aim was not to be exhaustive, but to showcase the diversity that is currently emerging worldwide.

What types of learning analytics tools are available and for whom?

The Inventory covered 28 tools, most of which were developed in Europe or North America either by e-Learning vendors (18), universities (2) or as a collaborative project involving various stakeholders (e.g. vendors, universities, non-profits). These tools are divided into different categories: tools for school level, higher education, workplace learning and those that can be used in multiple contexts.

These tools serve various purposes within education – for example, they can alert learners and educators to problems with performance and identify learners in need of support. Some also make predictions about the future behaviour of learners and their success whereas others recommend suitable resources or activities, or adapt course materials and activities to suit the knowledge level of individual students (the latter is also known as ‘adaptive learning’). Other tools serve the purpose of more general analytics tools geared to assessment, or to the design and planning of educational interventions.

The underlying use of analytics by tools also differs: some use analytics for summarising and describing the available data, whereas others use data for statistical inferences, for example to form judgements about a population of learners or to judge the reliability of certain statistical relationships. The majority of learning analytics tools in the Inventory present data about learners in a usable form, either through visualisations or by summarizing and describing the data. This can provide useful opportunities for reflecting on work that has been carried out and for making comparisons between individual learners, specific cohorts or institutions.

Lastly, the tools in the Inventory are very varied in terms of supply-model. Most of them seem to be self, or privately, hosted server software; some are desktop tools; some are shared service models which can be integrated into existing Learning Management Systems/Virtual Learning Environments/Managed Information Systems. The latter take advantage of existing data in these systems, whereas others are stand-alone tools which generate their own data for the purposes of modelling, alerting, prediction, etc. Some tools also take advantage of third party data such as social media and statistical services.

To illustrate the variety of tools, we take a look at the following three examples:

• Cognitive Tutor software: this focuses only on one area - mathematics. It provides personalised learning activities and feedback to the learner using specific models of domain knowledge and cognitive models based on learners’ responses. Teacher and learner get information on progress and mastery of each achievable skill, including pre-test and post-test information. See the Inventory No: 6.

• Civitas Learning: these individually-tailored services fulfil the needs of a specific institution, making use of already available student data, e.g. from VLE, social media, “card swipes” (e.g. students using their card to go to library). They then make available to institutional leaders and student service providers historic and predictive data on learners’ performance and success across modules and they predict programme completion. See the Inventory No: 5.

Conexus Vokal: an overall learning environment tool with an extra module that makes learning analytics available to support evaluation and improvement of pedagogical practices. It provides analysis and reporting at individual and group level on the basis of data it gathers from different sources. The provider of this tool works with several school book publishers, whose content it can use to generate data for analytics, and it also gathers data from student surveys and national statistical data sources, e.g. national tests. See the Inventory No: 7.

Most of the tools (13) in the Inventory were developed for use with students in secondary or post-compulsory education, and a further 8 were used in Higher Education. Six of the tools were designed for use in a wide range of settings and some of these can also be used in informal learning settings in which learners select their own goals and means of achieving them. Notably, none of the tools in the Inventory was designed specifically for informal learning.

Some current tools take advantage of novel, innovative pedagogy and theoretical approaches to teaching and learning. Some examples are listed below, particularly in the areas that have been highlighted as priorities for Europe (as in ‘New priorities for European cooperation in education and training’ by European Union, 2015)

Focus on innovative education and training:

• Improving students’ learning habits: CLARA (see the case study of University of Technology, Sydney 123). This tool aims to make students aware of their learning dispositions (the habits of minds they bring to their learning). The survey tool platform generates a ‘learning power’ profile visualisation for each student, and also interventions based on these learning profiles. In addition, students receive coaching and mentoring from trained peers and staff

• Helping students to reflect: Open Essayist (see the Inventory no: 17). This tool provides automated feedback to learners on draft essays in order to support learner reflection and development. It presents a computer-based analysis of the most important sections and key words in a draft so that learners can compare those to what they intended to convey, and adjust their writing in the light of that comparison.

Focus on skills and competences:

• Providing analytics for informal learners: Khan Academy analytics (see the Inventory no: 25). The Khan Academy provides for free online video-centric learning resources on a wide range of subjects, principally focusing on declarative and procedural knowledge. The platform provides a dashboard for learners that shows progress against skills and activity patterns over time, and against different skills. This is an example of the use of analytics to support informal learning.

• Supporting collaborative or group learning: SNAPP (see the Inventory no: 12) The Social Networks Adapting Pedagogical Practice (SNAPP) tool performs real-time social network analysis and data visualisation of forum discussion activity on commercial and open source learning management systems. The tool can be used to identify isolated students, facilitator-centric network patterns, group malfunction and users who bridge smaller networks.

• Analytics for 21st-century skills: Connected Intelligence Centre (see the case study of University of Technology, Sydney). This Australian centre is developing learning analytics associated with the 21st-century qualities that are important for all university staff and students.

• Supporting skills development: Skillaware (see the Inventory no: 22). This learning environment software is designed to support skills development in the context of workplace learning and training. The programme is used together with existing company software or procedures to determine worker effectiveness and to identify areas where training may be useful.

Focus on Higher Education attainment, student retention and inequalities:

• Helping students to make the right choices: Degree Compass (see the Inventory no: 14). On average, students in the US take 20% (on average) more classes than are needed to graduate. Providing help with course selection therefore can cut tuition costs and help increase retention and graduation rates at college. Degree Compass is designed to increase student success by providing students with academic advice from the time they start school, monitoring progress, offering on-going personalised course and degree path recommendations, and reducing time-to-degree with better course selection.

• Narrowing the attainment gap: Georgia State University (see the Inventory no: 35). At the university, predictive analytics have been used to tackle the achievement gap for low income and first-generation students. GSU’s graduation rate rose from 32% in 2003 to 54% in 2014. In the process, the university claims to have removed the achievement gap between students from minority backgrounds or lower socioeconomic status, and their peers.

• Aligning analytics with student support: Student Success Plan (see the Inventory no: 19). This software is designed to improve retention, academic performance, persistence, graduation rates, and time to completion. Through counselling, web-based support systems, and proactive intervention techniques, students are identified, supported and their progress is monitored.

Focus on quality and efficiency of compulsory education systems:

• Analysing test result data from student to district level: the LUVS dashboard (see the Inventory no: 4). Teachers and administrators can view and analyse the results of tests at the level of individual student, classroom, school or district. The tool is produced by Cito, a Dutch company, which has been commissioned by the Dutch government to produce testing and examination services for primary and secondary education.

• Providing data analysis and reporting tool for schools: FFT Aspire (see the Inventory no: 8). This is a data analysis and reporting tool for schools that draws on the national data available in the UK. It provides several dashboards showing facets of school performance, such as progression, attendance and future performance estimates. Its collaboration dashboard enables comparison of the performance of schools, taking into account factors such as social deprivation. It thus highlights areas of inequality where action needs to be taken.

Since 2011, when learning analytics emerged as a distinct field, validation of the tools used has been an issue. The Inventory provides evidence of the maturity and utility of each tool. Some tools are validated “by use” – i.e. their success is indicated by the number of organisations and users who continue to engage with them. For example, Schoolzilla (see the Inventory no: 11) is used by 58 schools across the USA and Bingel (see the Inventory no: 3) claims to be used by a large percentage of Dutch-speaking schools in Belgium. Some learning management systems and digital technologies can visualise data in a way that may be labelled ‘learning analytics’, and wide-scale use of these has been reported. For example, Conexus Vokal (see the Inventory no: 7) is used in 75% of Norwegian primary schools (however, not necessarily the analytics model), and the itslearning platform (see the Inventory no: 9) claims to have over 7 million active users internationally. This issue of data visualisation, however, can be used to highlight the difference between learning analytics research and the level of current implementation and deployment. These data visualisations are not necessarily ‘actionable’ in the way that learning analytics should be – in other words, they do not reveal what actions should be taken in order to improve learning or teaching.

As regards formal validation of tools (e.g. whether the tools fulfils its intended purpose such as impact on learning; more efficient learning; more effective learning), there is little to report. One reason might be that not enough time has passed. Following the emergence of learning analytics in 2011, important first steps involved overcoming bureaucratic and technical constraints in order to bring data together and present them in usable form. By 2013, relatively few early adopters were in a position to start developing algorithms and then test them using real student data. Those who went on to do this using the data they gathered from their next two student cohorts were ready to trial their algorithms on students at the start of the 2015 academic year and to begin reporting their initial findings in 2016. This might be one reason why validation is so scarce.

There are also several tools, particularly those that were developed before the emergence of learning analytics as a field, which have undergone more robust study. Statistics from 2005 – 2011 show that students using the Student Success Plan were five times more likely to graduate than others (see the Inventory no: 19). Studies at Tennessee schools have shown that at-risk students who used the Degree Compass tool (see the Inventory no: 14) earned higher grades than others. The CourseSmart Engagement Index (see the Inventory no: 13) has been shown to be a significant predictor of course grades across disciplines and educators. Some tools reported in the Inventory are still under development, and some developers have not openly shared evidence about success rates.

In general, one could conclude from the broad but shallow list of examples in the Inventory that currently tools seem to focus mainly on success in school and university courses. They offer a new type of “digital era” support for teachers, school leaders and other educational staff based on data.

1. What kinds of institutional and policy practices exist in Europe and elsewhere?

The ‘Practices’ section of the Inventory illustrates the work that is currently being done and the kind of practical learning analytics applications that are already possible today. The descriptions of the practices are divided into different categories: institutional pilots, at scale implementations, and also a number of initiatives linked to learning analytics at national level. We also report on practices related to the ethical use of learning analytics. This part of the Inventory also describes international and local networks and organisations concerned with research, development and practices around learning analytics.

Formal education in Europe

Practice examples in Europe include the following illustrative examples:

• An example of a practice on an institutional scale in England is the student dashboard deployed by Nottingham Trent, which uses engagement data (e.g. library use, attendance, use of the online learning platform) of all undergraduate students (see the Inventory no: 36).

• An institutional pilot is being run in Dunchurch Infant School which is trialling the use of learning analytics to support teachers in recording the activity of very young learners. Teachers can use data visualisations to see reports on their pupils’ strengths and weaknesses (see the Inventory no: 32). Another pilot is being run by Dublin City University, which supports students on some Moodle courses by providing targeted predictions and resources (see the Inventory no: 31). The Open University deploys software that predicts which students are at risk and has conducted several scientific pilots (see the Inventory no: 18).

• From the tools’ Inventory, we have reports on the use of learning analytics services which use national statistics, e.g. in the Netherlands (Cito LUVS, see the Inventory no: 4), in the UK (FFT Aspire, see the Inventory no: 8) and in Norway (Vokal, see the Inventory no: 7).

• Additionally, according to the descriptions of tools, various types of software are being used in primary and secondary schools around Europe that deploy features of learning analytics, e.g. Bingel (the Inventory no: 3); Conexus (the Inventory no: 7); itslearning (see the Inventory no: 9).

Some European countries are developing national approaches and are beginning to create the infrastructure to support learning analytics, e.g.:

• In Norway, several developments are on-going (see the Inventory no: 40):

Actions related to technical infrastructure and interoperability are being carried out. UNINETT, which develops and operates the Norwegian national research and education network, is rolling out a service platform, Dataporten (Norwegian for "data gate")³, that connects data sources and end-user applications. This will eventually allow better sharing of data in general and also for the purposes of learning analytics.

In Standards Norway, the national standards body of Norway, discussions have centred around three projects: Data sharing, vocabularies for activity descriptions, and Privacy and best practice guidelines. All three have the potential to enable applications such as learning analytics.

The research centre ‘SLATE’, which is partly funded by the Ministry of Education, is set to conduct a National Overview study to better understand ‘Possibilities and Challenges for Learning Analytics in Norway’ (this will include a Norwegian inventory of tools, ethics and privacy and some guidelines).

• In the Netherlands, two Dutch government-funded organisations are working on learning analytics, which is seen as one of the key issues in ICT for education.

Kennisnet advises sector councils and schools within compulsory education (more details in Case Study of Kennisnet)

SURF Foundation is a public collaborative organisation for ICT in higher education and research⁴.

Apart from support and advocacy, both the above play a key role in developing standards through EduStandaard, the Dutch educational standards body.

• In Denmark, the Ministry of Education is adopting technology infrastructure that can support the large-scale adoption of learning analytics across the country (User Portal Initiative, see the Inventory no: 39).

A great deal of work on ethics and privacy when implementing learning analytics has taken place in Europe. An institution-specific example is the Ethical use of student data policy that has been put into practice at The Open University in the UK (see more details in the case study on The Open University, UK). A more general set of guidelines has been developed by the UK organisation ‘Jisc’, with the intention that these should form the basis for discussion and policy development in different contexts. The Jisc code of practice for learning analytics focuses on issues of responsibility, transparency and consent, privacy, validity, access, enabling positive interventions, minimising adverse impacts, and data stewardship (see the Inventory no: 42).

More generally, the Analytical Review produced by the British government’s Department for Education (DfE) focused in 2013 on the roles of research, analysis, and the use of data within the department and its schools and children’s services. The report suggested that the government should lead culture change by setting the expectation that evidence is an integral part of education policy and delivery and that research skills are the key to professional improvement and freedom. The government should also make the sharing of real-time data easier, more efficient and more attractive. Finally, it should encourage a flourishing secondary market to improve data access and analysis by parents, schools and other interested parties (see the Inventory no: 52).

Formal education in the USA

The USA has taken the lead in the field of learning analytics both in research and in practice. Several documents have also been produced which inform policy and policy makers. Many big e-learning technology vendors are US-based companies, which is also clearly mirrored in the number of examples in the Inventory. For example, 10 (out of 28 tools) are US companies. These include Civitas Learning (see the Inventory no: 5) and Knewton (see the Inventory no: 15), which are both leading-edge vendors in the field, and also companies such as Blue Canary which has been acquired by Blackboard (see the Blue Canary case study).

The Practice section of the Inventory also includes many institutional practices at scale that are US-based. What emerges from these examples is that there seems to be a fair amount of interest in topics such as student retention and students who could be identified as “at-risk”. Arizona State University has been using Knewton’s analytics tools since 2011, creating personalized learning paths for thousands of students in remedial math (see the Inventory no: 29). Georgia State University claims that its use of learning analytics has removed the achievement gap between students from minority backgrounds or who have lower socio-economic status, and their peers who previously had higher graduation rates (see the Inventory no: 35). Course Signals from Purdue University (the Inventory no: 33) and the pilot from Rio Salado College (see the Inventory no: 30) have the same focus. Many of the tools in the inventory explicitly focus on identifying students at-risk – for example, Degree Compass by Desire2Learn (see the Inventory no: 14); X-Ray Analytics (now acquired by Blackboard - see the Inventory no: 21); Knewton (see the Inventory no: 15) and Schoolzilla (see the Inventory no: 11).

Interesting documents have been produced to inform policy and policy makers. Enhancing teaching and learning through learning analytics and educational data mining is a policy brief produced by the US Department of Education in 2012 (see the Inventory no: 55). It advises educators and administrators to be intelligent consumers of data and to generate demand for products that have useful features. Institutions are advised that the adoption of analytics initiatives, and the technical requirements of analytics requirements, may exceed their current technical capacity. Policymakers are advised to align the technical requirements of their policies with online learning and to consider privacy, policy and legal issues when storing and analysing personally-identifiable information.

In 2012, Educause, an American educational organisation that has been active in the promotion of learning analytics, produced a report on Understanding and managing the risks of analytics in higher education (see the Inventory no: 61). This focuses on the challenges associated with learning analytics and deals with areas of concern around data governance, including legal data protection requirements, data collection and storage methods, and access to student data. The report also considers data quality and the issues associated with missing, incorrect or misleading data with legal and institutional compliance, the use of third-party systems and issues around ethics and privacy.

In 2014, the Alliance for Excellent Education published Capacity enablers and barriers for learning analytics (see the Inventory no: 53), which considers the implications of these subjects for policy and practice. The Alliance is an advocacy organisation dedicated to ensuring that all students, particularly those traditionally under-served, graduate from high school ready for success in college, work and citizenship. Moreover, the report of the Alliance makes a series of recommendations that will support the take-up of learning analytics. According to the report, it is important to develop a clear understanding of the potential and rationale for learning analytics. It is necessary to build capacity for the implementation of learning analytics, including development of a culture of informed decision-making, infrastructure and human capital. To make this possible, funding models must be explored and developed. To support these processes, research on adoption and emergence of effective practice is needed. Alongside this work, policies must be identified and developed to support and enable learning analytics, including aspects of technology procurement, teacher development and privacy.

Formal education in Australia

Extensive work on learning analytics is being carried out in Australia. Interesting large-scale practices are reported in a report called Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement (see more at no: Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement). One of the case studies focuses on the University of Technology in Sydney which has created a data-intensive strategy based on learning analytics (see case study of University of Technology, Sydney).

In 2012, a networking event in Sydney brought people interested in learner-centred, data-driven practices from across the continent together for the first time. In 2013, the Australian Government Office for Learning and Teaching funded a report on improving the quality and productivity of the higher education sector (Siemens et al., 2013). The aim of the report was principally to advise the Australian government on interventions it could make to enable its higher education establishments to exploit learning analytics in order to achieve increased levels of educational success, and thus build a competitive advantage for Australia. It identified five factors that could enable the development of a national agenda.

1. Higher education leaders coordinate a high-level learning analytics task force.

2. Leverage existing national data and analytics strategies and frameworks.

3. Establish guidelines for privacy and ethics.

4. Promote a coordinated leadership programme to build institutional leadership capacity.

5. Develop an open and shared analytics curriculum (to develop systemic capacity for learning analytics by training skilled professionals and researchers).

A subsequent report by the Australian Government Office for Learning and Teaching on “Improving the quality and productivity of the higher education sector” (see the Inventory no: 56) concludes that most Australian universities are in the early stages of adopting successful learning analytics practices. It stresses that learning analytics is a complex system, which requires the development of six key areas: academic content, conceptualisation of the purpose for learning analytics, leadership, university strategy, stakeholder feedback, technology and an understanding of the specific university context. The report identifies areas that will need further consideration and support if learning analytics are to provide meaningful impact.

The report notes that people form a critical ingredient in the early stages of learning analytics implementations, and it calls for broader stakeholder involvement and discussion at all levels about learning analytics and their potential. These discussions need to include national conversations that identify ethical issues in this area and ways of dealing with these.

Capacity building is also an issue and it is discussed here partly in terms of skills that require programmes of professional development, academic courses and secondment opportunities. Capacity building is also discussed in terms of educational leadership. The study found that implementations of learning analytics in Australian higher education fell into two broad groups. The first of these treated analytics as a way to enhance existing practices, and therefore focused on performance measurement and retention interventions. The second group looked more deeply into learning as a pursuit of understanding and viewed retention as an important proxy for student engagement.

Focus on work

The Inventory identified only two tools that were designed for use in vocational or training settings (see the Inventory ‘Tools: workplace learning’), though this might be in part because workplace training is business specific and sometimes business sensitive, so these tools may not be shared externally. However, there is a positive side which shows that some tools are already being aligned with the need to focus on learning outcomes for employability and innovation, although there is obvious potential for this work to be developed further (see also EU projects such as Edu-works⁵ which focuses on labour market matching processes).

The Learning analytics at the workplace manifesto (see the Inventory no: 57) makes a start in the area of workplace training by providing advice for industry leaders, employers, workers, universities, teachers, social partners, teacher unions and trade unions. It calls for the EU to bring together relevant stakeholders with a view to identifying 21st-century skills that are needed and then improving the training of Europe’s workforce in order to meet the needs of industry and society.

Organisations and networks

The Practice part of the Inventory also includes organisations that are concerned with the development of the field. In the Netherlands and the UK, practitioners aiming to implement learning analytics can call upon support organisations such as Kennisnet (see more at Case Study of Kennisnet) and Jisc (the Inventory no: 41) to develop their learning analytics capability through advice and guidance, the establishment of a technical platform with free and charged services, and integration with institutional systems, and the support of a series of pilots using the platform. The more internationally-available initiative is Apereo (see The Apereo Foundation Learning Analytics Initiative case study). More research and academically-oriented networks include the international Society for Learning Analytics Research (SoLAR) - see the Inventory no: 49), Europe’s Learning Analytics Community Exchange (LACE) (see the Inventory no: 48), and the Spanish Network for Learning Analytics Research (SNOLA) (see the Inventory no: 50).

To conclude the results of the Inventory of learning analytics implementations, we can say that in general, examples from Australia, Europe and North America show that there are growing opportunities to share lessons learned and examples of good practice, even though learning analytics are only being used at scale in a small number of institutions. Regarding the issues that emerge from the current review of policies and practices, we see that most policies that are related to education, data protection, privacy and technical standards all influence learning analytics, but were not originally designed with learning analytics in mind. However, there is growing awareness of the need for policy in this area, and the Inventory contains examples of policies and policy briefings (see individual reports in the Inventory under ‘Policy documents’). Some of these briefings include recommendations that can be implemented at national or international level and should therefore be taken into account when developing learning analytics policy at European level.

What are the insights from the case studies?

The five in-depth case studies carried out represent good coverage of different continents and education levels. The first focuses on national work in this area and showcases recent work in the Netherlands. It is followed by two examples of educational institutions that have rolled out learning analytics at scale: the Open University, UK, and University of Technology Sydney, Australia. The last two cases deal with learning analytics development and implementation and focus on Apereo, an international initiative designed to accelerate the development of learning analytics tools, and Blue Canary, a predictive learning analytics software company. The case studies can be found in Annex 2 of this document.

In Australia, the University of Technology, Sydney, committed itself to becoming a data-intensive university in 2011. This work began with a series of internally-funded projects that tested the potential of data mining in relation to student retention. The importance of data as a business, learning and research priority within the University became increasingly clear. A university strategy was developed and a new centre was opened in 2014, which focuses on research into next-generation learning analytics tools. These tools are now being developed and piloted, and preliminary results are currently emerging.

The example of the University of Technology in Sydney shows that the introduction of learning analytics is a long-term process that entails changes to strategy, policy and structure, as well as shifts in pedagogy and technology. The path from initial pilot studies to validated analytics takes years, even when a university is fully committed to the area.

In the Netherlands, work on learning analytics by public organisation Kennisnet has also taken time to mature. Kennisnet built up its activity in the area of learning analytics after a horizon-scanning exercise in 2011. The organisation helps schools articulate what they want from technology vendors, and has brought them together to increase their influence with vendors, so they will be better able to deliver effective solutions for learning analytics, among other ICT issues. In other words, Kennisnet helps schools to generate demand for products that have useful features, especially seen from the end users’ point of view and not only from that of the vendors’. Standardisation and interoperability are seen as key issues by both vendors and schools, and Kennisnet expects to continue its work in this area for several years.

The other case studies show the same pattern of extended development. Blue Canary’s commercial work on learning analytics built on several years of research and pilot studies (see the Blue Canary case study). The Open University’s ethics policy, which deals with data use and analytics, was developed through a multi-year process of research, consultation and pilot studies (see The Open University, UK case study). The Apereo Learning Analytics Initiative supports the development of learning analytics software through a structured innovation process. The organisation is already looking ahead 10-15 years, during which time it hopes to become the baseline framework for open learning analytics initiatives (see the Blue Canary case study).

Each case study provides an example of an organisation building its experience on learning analytics over time. Each of them has not only high hopes for learning analytics, but also the conviction that they will be successful. Their extensive knowledge of the field means that these hopes are grounded in an understanding of what is possible now, and what could be possible in the future. Each is aware that learning analytics require a robust infrastructure, and system of quality assurance and validation that gives confidence to all stakeholders.

Insights from Expert Workshop

In order to discuss the issues arising from the Inventory and Case Studies in more depth, and to prioritise them, experts from across Europe took part in a workshop (see list of participants in Annex 5). A 2-day workshop was held in Amsterdam, NL, in March 2016. The expert participants identified four immediate issues for learning analytics in Europe:

1. A European roadmap for learning analytics development is needed in order to build and develop a set of interoperable learning analytics tools that are tailored to the needs of Europe and have been shown to work in practice.

2. Stakeholder engagement needs to be increased by reaching out to groups including teachers, students, staff, employers and parents.

3. As legislation changes and individuals become more aware of data use, institutions need help to understand their responsibilities and obligations with regard to data privacy and data protection.

4. More empirical evidence is needed about the effects of learning analytics, in order to support a process of quality assurance.

Workshop participants also identified the following policy priorities:

• Innovative pedagogy

The top priority in the short term is to develop innovative pedagogy that drives innovation and the use of data to solve practical problems.

• Teacher education

The top priority in the longer term is for media competencies and learning analytics knowledge to be built into training for both new and existing teachers.

• Ambassadors

Learning analytics need more outreach, with ministries and politicians spreading the word and encouraging local communities and schools to engage.

• Evidence hub

It is also important to gather scientific evidence on the impact of learning analytics. Currently the LACE Evidence Hub does it (see details at: Inventory No. 51). Securing sustainable funding for such a site is crucial.

• Identify success cases and methodologies

A coordinated approach to quality assurance should be developed and successful work on which to build should be identified.

• 21st-century skills

Work should be funded that develops learning analytics for important skills and competencies that are difficult to measure, particularly 21st-century skills.

• Orchestration of grants

European funding for work on learning analytics should be orchestrated around an agreed reference model that makes it clear what work is needed and where the gaps are.

• Crowd-sourced funding support

A system to crowd-source funding for tools teachers need could be developed, with European top-up funding available for successful candidates.

• Open access standards

Standards for European analytics should be established and an open access forum should be set up to enable the creation of standards from practice.

• Data privacy

A clear statement is needed from privacy commissioners about controls to protect learners, teachers and society.

• Decide which problems we want to solve

A series of collective discussions should be set up to identify priorities for learning analytics in the future.

• Facilitate data amalgamation

Work on ways of combining data sources should be supported to provide multi-faceted insights into the problems we seek to solve.

The inputs from the expert workshop, together with the results of the study, were used to create the Action List for Learning Analytics which is included in Section 4 of this report.

3 Summary of the Results and Further Steps

Since 2011, when an international conference raised awareness of learning analytics research, the use of data to understand and optimise learning and the contexts in which it occurs has offered a popular vision worldwide – particularly in Europe, Australia and North America.

Our first research question was: what is the state of the art internationally in the implementation of learning analytics for education and training in both formal and informal settings? The short answer is that early adopters are already taking the lead in research and development. However, the evidence on practice and successful implementation to improve - and innovate - learning and teaching is still scarce, as we have seen in the previous section.

Together, the examples of tools, practices and case studies show that work across Europe in learning analytics is potentially promising. Currently, however, it is fragmented. Some European countries, notably Denmark, the Netherlands and Norway, are developing national approaches and are beginning to create the infrastructure to support and enable endeavours in learning analytics. A few European universities, such as Nottingham Trent, Dublin City and the Open University, have developed implementations, some large-scale, others smaller-scale, which focus on some of the key areas of implementation such as tools or privacy. Most policies that influence learning analytics seem to have been developed in other contexts of educational technologies. From this study, we also learn that policies related to data protection, education, privacy and technical standards already exist, but many may need to be replaced or amended to take learning analytics into account.

Regarding available tools and their usage to improve - and innovate - education, there is a wide gap between the potential roles for learning analytics that have been identified by the research literature as a whole, and the dominant themes in learning analytics as they are put into practice at scale. The emphasis currently seems to be on visualising engagement and activity, making use of intelligent tutoring systems and adaptive content platforms, and developing systems that provide early alerts and target interventions. Most effort is focused on identifying students who may drop out; but less effort is made as regards innovative pedagogical processes, practices and developing educational organisations that fully embrace the digital era.

Another issue with current tools is finding evidence relating to formal validation of tools (e.g. whether the tools fulfil their intended purpose such as having an impact on learning; or making learning more efficient or more effective). Indeed, there is little to report (in the Inventory template as ‘Maturity and Evidence of Utility’). The issue is partly related to the timeframe; the topic of learning analytics first emerged in 2011 and at that point early adopters focused on how to bring the data together. A few years later a very few educational institutions were in a position to start validating tools and their impact. This means that very little hard evidence based on anything other than short-term studies is currently available. Some positive work is cited in the LACE Evidence Hub but, at this stage, there is no overwhelming evidence that learning analytics have fostered more effective and efficient learning processes and organisations. However, there is convincing evidence in the Inventory and Case Studies that companies and organisations believe learning analytics will do this in the future, and are prepared to invest time and resources in order in them.

Much of the work that is underway in Europe seems to address some of the strategic objectives or priorities at an institutional or regional level. However, at a higher level, there is a little coherence and convergence towards common topics and goals, for example those of the new priorities for European cooperation in education and training (European Union, 2015). As a result, companies and researchers are heavily focused on only some areas, e.g. reduction of drop-out rates and identification of at-risk students, while other areas, for example new and more learner-centred teaching methods, remain relatively untouched. Moreover, it appears that many people who implement learning analytics are likely to consider philosophy and pedagogy to be abstruse, academic or difficult. Additionally, few technology-enhanced learning implementation projects or policy documents from government level downwards are likely to deal with culture or values in their documentation. This, arguably, represents a risk. In particular, work is needed to link learning analytics with European priority areas for education and training, and the beliefs and values that underpin those areas.

Organisations such as Kennisnet in the Netherlands (see the Kennisnet case study), Apereo Foundation (see the The Apereo Foundation Learning Analytics Initiative case study) and Jisc in the UK (see the Inventory no: 41) are providing support to multiple educational institutions and also to companies within Europe to develop their learning analytics capability. Nevertheless, the results of this research do not seem to be widely known, and there seem to be only limited opportunities to share experience and good practice, especially in the area of use and implementation of learning analytics in an educational context. At a European level, the EU-funded project Learning Analytics Community Exchange (LACE) has integrated communities working on learning analytics so that they can share effective solutions to real problems. Since this project ended in summer 2016, no single network/organisation is bringing together people and evidence across Europe. On an international level, the Society for Learning Analytics Research (SoLAR) coordinates efforts on research initiatives related to conferences, summer schools, a journal and training initiatives. In order for others to follow the lead of these early adopters, and to encourage them to build on what they have already achieved, more work is needed on areas related to adoption and implementation.

The study “The Implications and Opportunities of Learning Analytics for European Educational Policy” conducted between September 2015 and June 2016, leads us to conclude that work across Europe in the area of learning analytics implementation in an educational context can be seen as promising. However, it is currently unevenly distributed and fragmented. In particular, the evidence that implementation of learning analytics can improve - and innovate - learning and teaching is still scarce, and high expectations have not yet been realised. This underlines the need for a careful build-up of research and experimentation with both practice and policies.

Further steps

In order to answer the two remaining research question set for this study, namely ‘What are the prospects for the implementation of learning analytics for education and training in the next 10–15 years?’ and ‘What is the potential for European policy to be used to guide and support the take-up and adaptation of learning analytics to enhance education in Europe?’, we take a closer look at the new European Union’s priority areas for education and training (European Union, 2015). One of them is open and innovative education and training, which fully embraces the digital era. Help achieving this goal could involve the use of learning analytics, which could help to improve learning and teaching by making use of the data generated as people engage in learning. This will require, however, a focus on another priority area, the provision of strong support for teachers, trainers, school leaders and other educational staff.

Matching the vision outlined above with reality is a process that will require time and experience, as the case studies make clear. A unified European approach could fill these gaps and could also build on previous learning analytics work that has taken place around the world. It could ensure that learning analytics are used effectively in all areas of education and training across the continent. In addition, a unified European approach could move this work forward by taking a strong lead in this area.

Much of the work by early adopters has been focused on the use of predictive analytics to identify students who are likely to drop out or to fail, so that they can be targeted for support. This is a worthwhile goal, aligned with one of Europe’s 2020 strategy targets for education (‘reducing school drop-out rates to less than 10%), but it is also a limited one. Learning analytics could play a larger role in ‘improving the quality and efficiency of education and training’ and in the provision of ‘open and innovative education and training’. In future, they could also be used to support other important European priority areas such as employability, innovation, active citizenship and wellbeing. Future work could align learning analytics much more closely with these priority areas for education and training in order to find common goals and convergence towards a shared vision so that there is no duplication of effort and that work proceeds in parallel.

The Action List for Learning Analytics is proposed as a way of guiding the discussion of further development in this area. It sets out a set of actions that will align the activities of educators, researchers, developers and policymakers to ensure that learning analytics are used to better drive work in Europe. These groups can use the Action List to ensure that open and innovative education and training, which fully embraces the digital era, becomes a reality in Europe, as it proposed in ‘the New priorities for European cooperation in education and training’.

The Action List for Learning Analytics focuses on seven areas of activity and identifies actions that need to be taken in each of these areas. It is important that initial strategic work takes place to ensure that each area is covered, that there is no duplication of effort, that teams are working on all actions and that work proceeds in parallel. The Action List for Learning Analytics is presented in the following section of this report.

4 The Action List for Learning Analytics

Introduction

Based on the examples of learning analytics and their implementations presented in this Study, practitioners, researchers and companies seem to be heavily focusing on areas such as identifying students at risk of drop-out and predicting students’ success. Results of current implementation and practices, however, do not seem to be widely known, and especially for policymakers at local, national and European levels, there are only limited opportunities to share experience and good practice.

The Action List for Learning Analytics offers the potential to resolve this problem by aligning work across Europe so that there is no duplication of effort and that work proceeds in parallel. The Action List for Learning Analytics is proposed to guide the discussion of the further development so that it could be more strongly aligned with the European Union’s new priority areas for education and training, namely, to ensure that open and innovative education and training, which fully embraces the digital era, becomes a reality (European Union, 2015).

Policymakers can use the Action List for Learning Analytics as a strategic planning tool in order to develop comprehensive policies for the effective uptake of learning analytics at local, national and European levels. Researchers and developers, including commercial companies, can use it to guide their work. Educational institutions (including primary and secondary schools, suppliers of vocational education and training, and higher education institutions) can use it to identify the resources and training that they require.

The Action List for Learning Analytics is comprised of 21 items. They are divided into 7 areas which are the following; Policy Leadership and Governance practices; Institutional Leadership and Governance practices; Collaboration and Networking; Teaching and Learning practices; Quality assessment and assurance practices; Capacity building; and Infrastructure.

These 7 areas are similar to those of the European Framework for Digitally-Competent Educational Organisations which aims to help educational organisations to fully integrate digital-age learning (Kampylis et al., 2015).

Policy Leadership and Governance Practices

i. Develop common visions of learning analytics that address strategic objectives and priorities

At a broad level this action refers to the strategic objectives and priorities of Europe, including its priority areas for education and training. At different levels, it also refers to the strategic objectives and priorities of member states, regions and individual organisations. All of these will take into account local context to an extent that is not possible at an international level.

What we measure shows what we value. Much of the current discussion about learning analytics focuses on performance metrics and how these affect teachers, learners and policy makers. Defining what to measure, and not measuring easily available data, includes an important debate about the vision of what learning analytics could and should do. This discussion, however, does not always take into account the delight and motivation that are inherent within learning. A danger with such learning analytics is that they prompt educational institutions to generate data that can be processed easily, prompting a focus on formal assessment that shows how much information has been retained.

Analytics should empower learners and teachers to make the right decisions for their needs. There is a need to do more work on that empowerment, with a focus on building rich datasets that will enable us to support the human side of learning. If we want to encourage teachers and learners to make use of analytics, then those analytics should provide delight.

It is also important that learning analytics do not become stuck in a rut, aligned only on performance metrics, for example. The possibilities for learning analytics keep developing as new pedagogies and technologies are introduced. Already, data from learning management systems can be supplemented with data from sensors embedded in the physical environment or from personal tracking data relating to movement or to vital signs.

At the same time, Europe is facing new learning challenges. For example, highly talented people from across the world are coming to Europe. Learning analytics could be used to shorten the time to recognising their competences and existing qualifications by putting individuals into realistic scenarios, and comparing their data with European benchmarks. This work could be linked to existing vocational training quality frameworks.

Planning for the development of analytics that address strategic objectives and priorities is therefore not a one-time activity. As Europe changes, and new possibilities emerge, plans for analytics will need to be developed alongside strategic objectives and priorities.

Action point: Develop a common vision in Europe by working with a multi-stakeholder group to consider priority areas for education and training and identify what learning analytics should do, how they should look and which beliefs and values should underpin them.

ii. Develop a roadmap for learning analytics within Europe

A European roadmap for learning analytics development could be used to drive the construction and development of a set of interoperable learning analytics tools that are tailored for the needs of Europe and that have been shown to work in practice. This would give a firm basis to build on in the future and would boost user confidence.

A roadmap aligned with Europe’s vision for learning analytics (i.e. as defined above) would identify current gaps in the European learning analytics toolkit. A learning analytics roadmap would support the development of sustainable tools and practices that outlive individual projects and can be deployed outside the settings in which they are developed. In addition, the European grants system could work more efficiently and effectively to support the development of learning analytics if grants were orchestrated around a roadmap. This would avoid replication of work, would fill obvious gaps and would demand evidence of the successful application of learning analytics in practice. It would also take into account the need for some work to be carried out on a more extended timescale than is currently funded, to include the time necessary for development, implementation and evaluation.

The roadmap would be tailored to the needs of the European community. It would also take into account the need for research and experimentation that can help to make our national education systems stronger.

The Open Learning Analytics framework (Siemens et al., 2011) proposed by SoLAR provides an example of a roadmap that brings different elements together, including the design, implementation and evaluation of an open platform.

Action point: At European level, work with learning analytics experts, educators, vendors and policymakers to develop a roadmap for learning analytics within Europe that is aligned with Europe’s priority areas, fills gaps in the European toolkit and supports the development of sustainable tools and practices.

iii. Align learning analytics work with different sectors of education

European priorities cover all areas of education and training, with a focus on making lifelong learning and mobility a reality and increasing opportunities for open education and training. However, as the LACE Evidence Hub (see the Inventory no: 51) indicates, much of the work in this area takes place within formal education, usually in the context of learning at secondary level and above.

Learning analytics should be tailored for different settings, including different levels of schooling, education, informal learning such as MOOCs, and workplace training. They should be responsive to individual needs, but should also support social learning.

In the case of businesses that use learning analytics to support training, their analytics processes and results are likely to be commercially sensitive, so there are few opportunities to share experience. The Learning analytics at the workplace (LAW) manifesto (see the Inventory no: 57) is one of the few documents to place learning analytics in the context of development in manufacturing such as 3D printing, the Internet of Things, digital disruptions and Industry 4.0.

In the case of informal learning providers such as MOOC platforms, their business models may not include resources to develop and deploy learning analytics.

Europe must therefore act to ensure that learning analytics can be adopted in all areas of education. This will involve extending to different sectors of education – including informal education and MOOCs – the work currently being carried out in the higher education sector to identify the different elements that need to be taken into account when deploying learning analytics. It will also require qualitative studies to understand how learning analytics can be aligned with the perceived purpose of education in different contexts, and which aspects of different educational contexts will support or constrain the use of learning analytics.

Action point: At European and national levels, explore the possibility of funding and supporting learning analytics work that extends into the workplace, work that focuses on implementation in informal learning settings, and qualitative work that considers the factors influencing success or failure when learning analytics are applied in different contexts.

iv. Develop frameworks that enable development of learning analytics

Analytics make use of quantitative and qualitative data. However, European priorities cover areas including employability, innovation, active citizenship and well-being and Inclusive education, equality, equity, non-discrimination and the promotion of civic competences. These are all areas that are difficult to quantify.

In order to use analytics to promote the development of these areas, there is a need for agreed frameworks that set out what these skills and competencies entail and how progression can be identified and measured.

A model for this work is the Digital Competence Framework for Citizens (DigComp)⁶, which deals with digital competence. The framework has five dimensions, which cover: competence areas that have been identified; competences that are pertinent to each area; proficiency levels that are foreseen for each competence; examples of the knowledge, skills and attitudes applicable to each competence; and examples of the applicability of the competence to different purposes. Similar framework is that of European Framework for Entrepreneurship Competence⁷.

As these frameworks are developed at a European level, and widely disseminated, they form a unified basis for future work so that alignment between analytics projects in these areas will be possible.

Action point: At European level, fund work on the development and deployment of frameworks that support learning analytics work related to skills and competencies.

v. Assign responsibility for development of learning analytics within Europe

Learning analytics work within Europe requires a strong lead. This will enable Europe to follow its roadmap for learning analytics, rather than including analytics as an element in a variety of different strategic frameworks. Having a strong lead would also mean that different national and European funding bodies would be aware of learning analytics work that has been completed or is currently in progress and would not put out calls for work that has already been funded in a different context.

Although learning analytics are a comparatively recent addition to the digital learning toolbox, there are already many European-funded projects in progress. These receive support from different programmes, including Erasmus+, Marie Skłodowska-Curie Actions, FP7 and H2020.

The following list gives a snapshot of projects already working in the field with a clear focus on analytics:

FP7-funded projects:

LACE (Learning analytics community exchange)⁸ A coordination and support action, LACE is focused on pressing learning analytics issues including interoperability and ethics.

LEA’s Box (Learning analytics toolbox)⁹ A specific targeted research project, LEA’s Box provides a central hub where teachers can find the best analytics solutions for their students.

PELARS (Practice-based experiential learning analytics research and support)¹⁰ PELARS uses multi-modal data to enable students to learn to make better decisions in small groups, and to help them reflect on the process.

Erasmus+funded projects:

PBL3.0 (Integrating learning analytics and semantics in problem-based learning)¹¹ This project will make recommendations about best practices and policies in the context of problem-based learning.

SHEILA (Supporting higher education to incorporate learning analytics)¹² SHEILA is a project that is intended to have an impact on policy development.

STELA (Successful transition from secondary to higher education by means of learning analytics)¹³ STELA is another project, this time supporting a successful transition from secondary to higher education.

Other:

LAEP (Implications and opportunities of learning analytics for European educational policy)¹⁴ Funded by the JRC, LAEP is the project responsible for this report.

Projects that incorporate learning analytics, but do not have them as a main focus:

BEACONING (Breaking educational barriers with contextualised pervasive and gameful learning)¹⁵ BEACONING is a new Horizon 2020 project, which makes use of games and gamification in different domains and settings.

RAGE (Realising an applied gaming ecosystem)¹⁶ RAGE is a Horizon 2020-funded project with a focus on serious games. It is building a full infrastructure that will streamline the process of applying learning analytics to games.

WATCHME (Workplace-based e-assessment technology for competency-based higher multi-professional education)¹⁷ An FP7-funded project on workplace-based learning that uses an e-portfolio system to collect information about activities and the learning context.

The challenge for projects from various funding schemes is to contact and learn from the experience of others, or for consortium members to gain an overall sense of a European analytics strategy and where responsibility for this lies. The LACE project has been working to link the different projects within a European research network, but when it reached the end of its project funding in June 2016, individual projects are likely to find themselves isolated from each other.

Action point: At European level, identify some responsible entity for leading and coordinating work on learning analytics and implementing the learning analytics roadmap in order to facilitate peer learning and not to duplicate work. Network also organisations and individuals who will be key national contacts in different European countries.

vi. Continuously work on reaching common understanding and developing new priorities

The process of learning analytics is often presented as a cycle of learning design, learning activity, learning analytics, and reflection on learning analytics. The same is the case at European scale. Analytics have the potential to produce significant change at every level of education and training. As they are implemented, they will therefore change the learning landscape so that priorities for education and training after 2020 will look significantly different from those that are key in 2016. In order to develop the field of learning analytics, stakeholders need to engage in collective discussions about future directions and priorities.

Learning analytics is a relatively new field, which opens up different possibilities, not all of them positive. In order to open up thinking about these possibilities, a study was carried out to investigate with different stakeholders how learning analytics may develop internationally in the next decade¹⁸. In order to do this, eight plausible futures were drawn up intended to act as provocations and to elicit strong reactions. Each vision contrasted the situation in 2015 with a potential situation in 2025. The full visions were each around 100 words long. In summary, they were:

In 2025, classrooms monitor the physical environment to support learning and teaching,

In 2025, personal data tracking supports learning,

In 2025, analytics are rarely used in education,

In 2025, individuals control their own data,

In 2025, open systems for learning analytics are widely adopted,

In 2025, learning analytics systems are essential tools of educational management,

In 2025, most teaching is delegated to computers,

In 2025, analytics support self-directed autonomous learning.

Study participants considered these visions in workshop sessions or via an online survey and considered whether they were feasible or desirable, and what actions would be required in order for them to become a reality. The report on this work¹⁹ highlighted some of the reasons that stakeholders should be involved throughout the learning analytics process. For instance, it revealed disagreements between educational sectors and showed that practitioners do not necessarily welcome the systems and methods produced by developers

Action point: At European and national levels, organise regular events involving a range of stakeholders in order to discuss future directions, priorities but also possible dangers, in the field of learning analytics.

Institutional Leadership and Governance Practices

vii. Create organisational structures to support use of learning analytics and help educational leaders to implement these changes

A core goal for most learning analytic projects is to move from small-scale practice, innovation and research towards broader implementation, but this introduces a new set of challenges because educational institutions are stable systems, resistant to change. To avoid failure and maximize success, implementation of learning analytics at scale requires explicit and careful consideration of the entire TEL (Technology-Enhanced Learning) complex: the different groups of people involved, the educational beliefs and practices of those groups, the technologies they use, and the specific environments within which they operate. It is crucial not only to provide analytics and their associated tools, but also to begin with a clear strategic vision, assess institutional culture critically, identify potential barriers to adoption, develop approaches that can overcome these, and put in place appropriate forms of support, training, and community building.

Piecemeal, simplistic, and non-systemic approaches to learning analytics implementation will struggle to gain traction. Analytics implementation requires a change to a wide range of practices across an institution. Educators need to be involved in designing the tools and able to evaluate any implementation of analytics tools in order to use them effectively. Learners need to be convinced that analytics are reliable and will improve their learning without unduly intruding into their privacy. Support staff need to be trained to maintain the infrastructure and to add data to the system. Library staff need to be able to use the analytics to shape their practice and resources. University administrators need to be convinced that the implemented analytics provide a sound return on investment and demonstrably improve teaching and learning quality. IT staff need to put workflows into place so that raw data are collated, prepared for use, and made readily available to end users. In order to convince all these stakeholders to put in the sustained effort necessary to make use of learning analytics, a clear vision of the gains to be made is required at the outset and should be maintained throughout. The University of Technology Sydney Case Study (see the University of Technology, Sydney case study) shows how learning analytics can be aligned with strategic objectives and priorities.

The European SHEILA project²⁰ offers a seven-step approach to the institutional implementation of learning analytics: define a clear set of overarching policy objectives; map the context; identify the key stakeholders; identify learning analytics purposes; develop a strategy; analyse capacity and develop human resources; and develop a monitoring and learning system (evaluation). This is an iterative process, and these steps can be repeated many times.

In order to implement analytics effectively, leaders are likely to require skills in change management. The European SHEILA project is currently identifying the different elements that need to be taken into account when deploying learning analytics, with a view to helping higher education carry out this process.

Action point: At European level, identify ways in which the funding system can be explored and adapted to support learning analytics implementation that works systemically.

Action point: At European level, fund projects that extend the work in this area in order to support the deployment of learning analytics in the schools and workplace sectors as well as within informal learning provision.

viii. Develop practices that are appropriate to different contexts

The culture, values, and existing practices that apply in the education or training setting in which learning analytics is implemented influence multiple aspects of what is done and how it is done. Research suggests that even when learning analytics are acknowledged by institutional leaders to provide new insights, they may still fail to influence institutional planning and strategic decision-making (Macfadyen et al, 2014). This may be the result of lack of attention to institutional culture, lack of understanding of the degree to which individuals and cultures resist innovation and change, and lack of understanding of approaches to motivating social and cultural change.

The beliefs of potential users of a learning analytics system – for example about its ease of use, utility, changes to workload or potential threats – are critical factors in acceptance and adoption, and may outweigh any assumptions about objective benefits. Educational organisations can make use of the existing tools, for example of DigCompOrg Framework²¹, to guide a process of self-reflection on their progress towards comprehensive integration and effective deployment of learning analytics and other digital learning technologies.

More work is needed to explore these areas and to develop appropriate practices. This could explore questions such as: How do people behave when learning analytics initiatives are undertaken? What is the current state of awareness, acceptance, and beliefs about applying analytics to teaching and learning? How are analytics perceived in terms of usefulness and relevance? How significant are differences in regional or sector culture, values, and professional practice, in relation to implementing learning analytics? Which norms of professional practice, power and influence do learning analytics challenge?

Action point: At European and national levels, fund and support work that explores the influence of culture, values and existing practices on the implementation of learning analytics.

Action point: At local level, make use of the existing tools, for example the DigCompOrg Framework, to support progress towards effective deployment of learning analytics.

ix. Develop and employ ethical standards, including data protection

Europe’s General Data Protection Regulation (GDPR)²² entered into force in May 2016 and will affect the learning analytics field in many ways. Europe has taken the position that individual privacy is important and that changes to current practices in general analytics are needed. Moving forward, the definition of personal data will be larger and more complex, and these legal changes will mean universities become data containers rather than data processors, with new responsibilities for control of data.

Institutions will need to understand their responsibilities and obligations with regard to data privacy and data protection and will have to put procedures in place to ensure that they are compliant with the legislation. There will also be an increased need to help parents and students understand how data are used.

A concern is that organisations, schools and companies that are privacy sensitive will be cautious and slow about adoption of learning analytics, while those that are not will be the ones first on the market. Companies in the United States are not as constrained by data protection regulation as those in Europe, which could give them a competitive advantage.

Students should feel that analytics are there to support them, not as a form of surveillance. They should not be frightened, shocked or scared by the use of their data. Rather, they should feel empowered to add their own data in order to provide a broader picture of their learning activities and capabilities. There is a need to distinguish learning analytics from the negative portrayals of big data in the media. Analytics should not be seen as a way of manipulating emotions, exploiting personal data or putting unaccountable algorithms in charge of individuals’ learning. Transparency is important – analytics processes should be open to scrutiny and subject to correction.

There have been several significant European initiatives in this area. Following a consultation period, The Open University in the UK has developed and implemented an ethics policy (see OU Ethics policy at the Inventory no: 44). The LACE project has been responsible for a series of workshops on ethics and privacy in learning analytics (EP4LA²³), which have been responsible for driving and transforming activity in these areas.

In the UK, Jisc has built upon this work to produce a code of practice that is intended to help universities and colleges to develop effective approaches to a variety of issues relating to the practice of learning analytics. Rather than providing a prescriptive code of practice, the approach taken is to clarify a set of principles that can be put into practice according to the policies and practices already in place in universities and colleges.

The Jisc code of practice (see the Inventory no: 42) deals with issues related to responsibility, transparency and consent, privacy, validity, access, enabling positive interventions, minimising adverse impacts and stewardship of data. Although these are fairly general areas, the code of practice has been developed with higher education and British laws in mind and so there is still work to be done on developing codes that take local legislation into account and are suitable for use in schools, in workplace training and in informal settings

Cormack (2016) has proposed a Data protection framework for learning analytics that reduces the significance of the boundary between protected personal data and unprotected, non-personal data ensuring that all processing includes appropriate safeguards. The proposed framework appears in a special issue of the Journal of Learning Analytics that deals with issues of ethics, privacy and data protection (Ferguson, 2016).

Action point: At national level, develop and share model policies on data privacy and data protection, and support institutions to understand their responsibilities and obligations in these areas and to put procedures in place to ensure that they are compliant with the legislation.

Action point: At local level, adopt data privacy and data protection policies, and work with staff and students to ensure they are aware of their rights and responsibilities.

Collaboration and Networking

x. Identify and build on work in related areas and other countries

As the results of the Study demonstrate, some Member States have already devoted considerable resources to the development and implementation of a strategy for learning analytics, and especially focusing on standards and infrastructure to enable them. In Denmark, the Ministry for Children, Education and Gender Equality is working with both central-level data and local data. One big initiative is a data warehouse, designed for school leaders, which links data to the country’s educational goals. Currently, the latest Danish initiative is the development of new dashboards that are targeted towards parental choice about schools but such work can also offer new possibilities for learning analytics. Denmark is also formulating standards for data exchange, and the Ministry is currently developing platforms on which central data can be combined with local data. We have reported also work on data standards in Norway and the Netherlands.

As well as work by governments and standardisation bodies, significant development work is being carried out by companies in the private sector. This ranges from the work of large companies such as Desire2Learn’s (See the Inventory no: 14) work on predictive analytics to the work of smaller companies, such as the learning tracker tool develop by start-up company Claned (See the Inventory no: 24).

On the other hand, the LACE Evidence Hub (see the Inventory no: 51) brings together some of the research evidence on the impact of learning analytics that is available internationally relating to learning analytics. The Hub puts forward four propositions, that learning analytics: improve learning outcomes, improve learning support and teaching, are taken up and used widely, and are used in an ethical way. Research is gathered in the Evidence Hub if it supports or challenges these propositions.

This Inventory of learning analytics tools, policies and practices provides a good starting point for investigating the current state of the art in different areas. In order to avoid good work in learning analytics being trapped in institutional, project or even national silos, there is a need for ways to share experience and practice at national and European level. The Case Study of Kennisnet (see the case study of Kennisnet) shows ways of organising knowledge transfer and the sharing of good practice at a national level. One model for good practice comes from the Netherlands, where SURF arranges workshops that spark dialogue between diverse groups including data scientists, teachers and education leaders. In some areas, it is schools or companies that are taking the initiative, or individuals within organisations (see the case study of Kennisnet). On the other hand, work from Australia provides a good example of a strategy that is being based on sound research (Siemens et al., 2013; Colvin, 2015).

In order to avoid duplication of work, Europe should keep itself up to date with significant developments and policy reports in this area from around the world, but also invest in bringing together a range of different types of stakeholder to build on work in related areas and other countries.

Action point: at national level, support and develop active national networks of learning analytics stakeholders.

Action point: At European level, commission reports from countries that are active in the field of learning analytics and update these on a regular basis.

Action point: At European level, support the development of an accessible repository for learning analytics evidence, building on the model of the LACE Evidence Hub.

xi. Engage stakeholders throughout the process to create learning analytics that have useful features

There are many different stakeholders involved with learning analytics. At a macro level, governments and regional authorities are beginning to see how they could be used to help achieve national and international objectives. Employers and educational institutions are looking for ways to increase the success of their organisations by providing effective support for learning. Within institutions, managers, learners, educators and developers are approaching analytics from different angles. On the other hand, trade unions and student unions are identifying ways in which analytics could benefit their members, and looking for ways to avoid potential pitfalls.

Despite the multitude of stakeholders, much of current work on learning analytics concentrates on the supply side – the development of tools, data, models and prototypes. There is clearly less work on the demand side – how analytics connect with education and the changes that teachers want these tools to have in order to support their everyday teaching and assessment work. More attention needs to be paid to the demand side; learning analytics systems should work for the teacher, not the other way around. Moreover, to make good use of learning analytics, students should be aware of how to act on the output of these analytics. They should also have some idea of how results are derived, so that they can be aware of their limitations.

Dialogues are needed to align the views and aims of different stakeholders. Initiatives that do not take into account these different views and experiences are unlikely to succeed. There is a need to bring people and stakeholders on board by reaching out to groups including teachers, students, staff, employers and parents.

Action point: At national level, involve a wide range of stakeholders, including employers and organisations such as unions and student unions, in discussions to identify ways in which analytics could benefit them, their members and their employees, and to find ways of avoiding potential pitfalls.

Action point: At local level, involve learners and teachers in decision making and co-designing of tools so that they include features that they find useful for their own use. Offer training and support so that they can effectively use of these tools in their learning and teaching.

Action point: At local level, policies of ethics and data protection should support students to make informed decisions about the use of their data. Students should be made aware of these policies and of learning analytics practices within their institution.

xii. Support collaboration with commercial organisations

Across the world, companies are developing and marketing learning analytics tools, a sample of which is presented more fully in the Inventory. I can give an idea of the range of work that is being carried out within Europe and beyond.

At present, there is a distinction between the research and development work that is carried out in the commercial sector and the work that is carried out in the academic sector. This gap needs to be narrowed.

Some initiatives are already in place. More work needs to be done to bring the two groups together because, as the Case Study on Blue Canary (see the Blue Canary case study) notes, collaboration between educational institutions and companies is critical to moving the field forward. Equally, the work carried out by Kennisnet in the Netherlands is important in order to ensure that learning analytics products have useful features for their end users, e.g. school administrators, teachers and students.

Action point: At European, national and local levels, promote work on learning analytics that brings together academic and commercial partners together with end users.

Teaching and Learning Practices

xiii. Develop learning analytics that makes good use of pedagogy

Successful analytics do not begin with a set of data; they begin with an understanding of how people learn. There is a need for novel, innovative pedagogy (theorised approaches to teaching and learning) that drives innovation and makes use of data to solve practical problems, particularly those highlighted as priority areas for Europe. Some current tools and practices point the way in this area, some of which are found in this Inventory.

Improving students’ learning habits: CLARA This tool, based on 15 years of research, makes students aware of their learning dispositions (the habits of minds they bring to their learning). The survey tool platform generates ‘learning power’ profile visualisations for each student, as well as interventions that are based on the learning profiles. In addition, students receive coaching and mentoring from trained peers as well as from staff (see case study of University of Technology, Sydney).

Helping students to reflect: Open Essayist This tool provides automated feedback to learners on draft essays in order to support learner reflection and development. It presents a computer-based analysis of the most important sections and key words in a draft so that learners can compare those to what they intended to convey, and adjust their writing in the light of that comparison (the Inventory no: 17).

Supporting collaborative or group learning: SNAPP The Social Networks Adapting Pedagogical Practice (SNAPP) tool performs real-time social network analysis and data visualisation of forum discussion activity on commercial and open source learning management systems. The tool can be used to identify isolated students, facilitator-centric network patterns, group malfunction and users who bridge smaller networks (the Inventory no: 12).

Action point: At national level, once plans for development and deployment of analytics aligned with European priority areas are in place, identify areas of relevant expertise, and analytics work that can be developed and aligned with European priority areas.

Action point: At local level, identify how current work and expertise can be aligned with European priority areas and with other work in these areas.

xiv. Align analytics with assessment practices

As assessment drives the behaviour of both teachers and students, old assessment strategies can limit the potential for learning analytics and, more broadly, for learning technologies. Learning analytics could potentially help to shift education to more authentic types of learning that equip students with and assess them on the 21st-century competencies that will be crucial in their future lives. If national assessment policies remain focused on the high-stakes end of the year exams, then analytics will be tied to these areas.

A shift towards student reflection, formative assessment, and the development of skills and competencies will move analytics away from a focus on current measurable outcomes and towards support for the holistic process of learning. This will need to be done in the context of both formal and informal learning.

Action point: At national level, fund studies to make recommendations about changes to assessment processes at all levels of education.

Action point: At local level, trial new methods of assessment, particularly in areas such as non-formal learning (e.g. MOOCs) where the assessment regimes are still under development.

Quality assessment and assurance practices

xv. Develop a robust quality assurance process for the validity and reliability of tools

More empirical evidence is needed about the effects of learning analytics. This will form part of a process of quality assurance, which will be essential to the development of user trust in learning analytics. Currently, some companies and institutions are making grand claims for analytics based on limited or dubious evidence while, at the other end of the spectrum, some teachers and students are not acting on good recommendations because they have not been convinced that they are valid or reliable.

Another problem is the ‘black-box’ nature of many learning analytics. Data are entered and results are generated, but it is not clear to the end user how those results have been generated. In cases of machine learning, even the people who developed the system may not be sure of the criteria that are used to generate final results. This can work against equality and equity. If, for example, students of a particular ethnic background or gender have tended not to be successful on a course in the past, algorithms are likely to take those demographic details as indicators that future students are likely to be unsuccessful. This could produce a form of automated discrimination that blames learners for failure, rather than prompting consideration of the ways in which the learning design or teaching are failing certain groups of students

European educational institutions and qualifications are subject to rigorous quality assurance. This should also be the case with learning analytics, and it should be clear who is responsible for this process. Quality assurance will involve checking the quality of data used, the validity and reliability of tools, and whether they are employed effectively in specific contexts. Some of this work must be carried out at an institutional level, but there is also a role for national or international quality assurance.

Action point: At European level, develop a coordinated approach to quality assurance, and a coordinated way of identifying and sharing successful cases, tools and methodologies.

xvi. Develop evaluation checklists for learning analytics tools

There are many learning analytics tools available, so it is very difficult for teachers to choose between them and to select the ones that will provide solutions to their problems. A simple list of tools is both uninformative and uninspiring. There should be more resources that help schools and teachers to decide which practices will work for them in a realistic real-world setting. Most teachers currently do not have the knowledge, nor time, to separate one system from another. Europe needs effective evaluation checklists or frameworks that can help them to make these decisions.

These frameworks would help teachers to ask the right questions in order to identify a tool that is evidence based, that has been shown to support learning, that is appropriate for their context, covered by their budget and is likely to help them to achieve their educational goals. The checklist would take into account both open and commercial learning analytics tools.

The checklist could also be associated with an evidence base in the form of testimonials and user stories. These would help to bring these tools to life and would provide information about practices that work and tools that help to improve teaching. It would be important that this evidence was quality assured. One way of doing that would be through a European learning analytics network with members sharing experiences, offering alternatives, building knowledge together, and providing feedback on frameworks and the evidence.

In many cases, decisions about learning analytics are made at institutional level, rather than by teachers. An evaluation checklist should prompt decision makers to consult with teachers and agree a solution rather than imposing it.

Action point: At national level, develop evaluation checklists for learning analytics, making use of the models provided by the Framework of characteristics for analytics (Cooper, 2012) and Quality indicators for learning analytics (Scheffel, 2014).

Capacity building

xvii. Identify the skills required in different areas

Research in Australia has found that systemic capacity for using learning analytics is hampered by lack of access to skilled professionals and researchers (Siemens et al., 2013).

For adoption of learning analytics it is important that both the developers but also the users of the analytics have the right set of skills. Currently, we don’t know exactly which skills are needed, and how many people already possess them.

For example for those implementing or procuring learning analytics systems, it is important that they are sufficiently knowledgeable to critically evaluate the system qualities that influence validity and appropriate use.

An obvious example of skills under consideration is those which a ‘data scientist’ or ‘data wrangler’ might possess. For example, quality of data is important when developing analytics. Datasets may be incomplete for a variety of reasons. They may also be out of date. Teachers and students may be able to see obvious errors, but not have permission levels that allow these to be corrected. Data may be entered wrongly, or people may supply inaccurate data (for example, many social media users supply fictitious details about their age, birthdate and employment). Analytics based on low quality data will be flawed and misleading, so institutions need policies in place to ensure that data collection is carried out consistently and that the process is quality checked.

Additionally, regarding the outputs of learning analytics, it is important that those who make decisions on the basis of visualisations, statistics and predictions really understand what they all really mean.

Action point: At European and national level, the higher education sector should partner with learning analytics experts and researchers to research data literacies in this area and to develop an open and shared learning analytics curriculum, for example to support the role of teaching support staff as analytics interpreters, and to develop more intuitive and easily interpretable analytics outputs.

xviii. Train and support researchers and developers to work in this field

Some of the data literacies and competencies required for learning analytics are more generic, and will be increasingly required in a Europe where big data and analytics are commonly deployed in many areas of life.

A further set of competencies that may be important in implementing learning analytics, reflecting the fact that teaching and learning is a complex space, are those required for evaluation and research. The implementation of learning analytics requires a reflexive process, so there is not only a need for evaluation and research skills to be available but also for learning analytics to be implemented in an exploratory fashion.

There is also a need for researchers and developers to be skilled in both technical and pedagogical areas. Early work in learning analytics often claimed that the tools that had been developed would be suitable for any pedagogical approach. This was sometimes the case, but such claims often disguised a lack of awareness of different approaches and assumed that teaching and learning would always take the form of direct instruction. Equally, some pedagogically strong work had the side effect of bringing the learning management system in which it was implemented to a standstill, because the processing power required for implementation had not been taken into account.

Researchers and developers also need to take into account that end-users are unlikely to share their knowledge of data processing and interpretation. Skills related to visualisation methods and to effective ways of presenting information to users are important if analytics are to be effective.

Action point: At national and local level, training for researchers in the field of learning analytics should include both technical and pedagogic elements.

Action point: At local level, researchers should provide guidelines on how to interpret learning analytics indicators, and should clearly state the limitations of indicators in order to prevent misinterpretation.

xix. Train and support educators to use analytics to support achievement

Teachers are the engine of innovation in education and any development that does not take their experience, constraints and requirements into account is unlikely to succeed. It is therefore important that the field of learning analytics does not simply focus its attention on developers and learners – it needs to involve teachers in order to succeed.

Teachers have established ways of working, and may not be confident in working with quantitative data and analytics. If they are to make effective use of learning analytics, many will need to increase their skills and confidence in this area. They will also need to be convinced that these new tools offer real value for their students.

Digital competence, a good understanding of data literacies and learning analytics knowledge need to be built into training for both new and existing teachers. This should include the ideas behind learning analytics and data mining, and the associated challenges and dangers. Such training could be carried out formally in face-to-face settings, or through informal courses such as MOOCs. In both cases, it should enable teachers to use the solutions that have already been developed to benefit their students, and prepare them to use the solutions that are currently under development.

Action point: At national level, incorporate digital competence and learning analytics knowledge within teacher and lecturer training, as well as within provision for continuing professional development.

Infrastructure

xx. Develop technologies that enable development of analytics

Work in Australia suggests a need to develop national data inventories, identifying gaps in data collection that need to be addressed through additional data collection activity and instruments. It also suggests the development of centralised databases that are accessible to educational institutions, decision makers and researchers (See ‘Formal education in Australia’, p. 14).

Learning analytics require rich data, but learning management systems simply provide activity data, such as how many times people have clicked on a web page, and when they have done that. Relying on one set of data can be dangerous. For example, high-achieving students may not participate in an activity because they have already developed that knowledge, whereas low-achieving students might not participate because it is too difficult. Activity data imply that these different sets of students form one group. The limitations of different datasets should therefore be identified, and this information should be shared with end users.

Data from other sources are needed to complement this activity data. These could include data from formative assessment (assessment for learning, rather than of learning) and data about student dispositions. Pilot studies that are being carried out in the Netherlands, coordinated by SURFnet²⁴, provide a model for this work. These make use of very rich datasets, including survey data, formative assessment, activity data and data about learning dispositions. More work is needed on ways of combining different datasets to increase the value of learning analytics for learners and teachers.

There is a need for systems that facilitate the collection and amalgamation of these different datasets at national or international scale. This is already being done in some countries, for example the Conexus Vokal tool draws on a range of anonymised data from Statistics Norway (see the Inventory no: 7). These systems should also be easy to interrogate, so more work is needed on ways in which these data can be presented and visualised in ways that are comprehensible to end users.

Lastly, higher education institutions that are pursuing learning analytics adoption often view data warehouses as a key enabling technology. These systems provide a way of integrating, organizing, and summarizing large datasets. Again, work in this area is fragmented, and there are not yet any studies of how different data warehouses work in practice, their advantages and their limitations. Denmark is taking a lead here at the national level. The country is developing a data warehouse to strengthen evaluation and follow-up initiatives across its entire education sector. The aim is to facilitate access to a range of performance data for schools and municipalities.

At both institutional and national levels, there is a need to explore whether the most appropriate infrastructure for learning analytics matches existing systems. In many cases, a substantially different approach to the management and storage of data will be required if analytics are to be implemented effectively.

Action point: At European and national levels, fund work that studies how different data warehouses work in practice, their advantages and their limitations.

Action point: At national level, develop model policies that can be used to ensure that data collection is carried out consistently and that the process is quality checked.

Action point: At local level, adapt model policies to local needs and apply them.

Action point: At European and national levels, compile data inventories in order to identify and address gaps in data collection.

Action point: At national level, where appropriate, develop centralised databases to facilitate the collection and amalgamation of datasets that can be used to support learning analytics work, that are accessible to stakeholders, and that are acceptable to the learners and teachers whose data they store.

xxi. Adapt and employ interoperability standards

If learning analytics systems are to build on each other and to interact with each other, then they need to be interoperable. A report on Learning analytics interoperability: requirements, specifications and adoption²⁵ provides a detailed survey of current interoperability initiatives that is designed to inform roadmaps and the choices of educational policy makers and managers.

Two competing specifications for gathering data about learning activities are emerging, both developed in the USA. These are Caliper from IMS Global and xAPI (also called TinCan) from ADL. The two specifications are attracting and generating ecosystems of other specifications, architectures and applications.

IMS Global is a closed membership organisation, mainly made up of large vendors, but also including some universities and national agencies. IMS specifications are developed in private, drawing on use cases from members, and then published openly. The organisation offers a set of interoperability specifications, of which Caliper is the most recent. The ambition is to provide complete coverage of the needs of education.

Open development has several advantages. It engages a wide range of stakeholders worldwide and can incorporate contributions from the wider academic community and research projects. It is also able to build momentum by coordinating around an open specification and architecture.

Overall, an open approach to learning analytics, making use of open-source software, offers certain advantages, including the possibility of reducing cost, no need to be tied to a single vendor, and options to draw on the resources of an international developer community. The Open Academic Analytics Initiative (OAAI) undertook a multi-year project to research the issues associated with scaling up learning analytics, particularly focussing on the use of open source software. This demonstrated the deployment of infrastructure and analytical methods across different kinds of higher education institution. The Apereo Foundation (see the case study on The Apereo Foundation Learning Analytics Initiative) is taking this work forward both within Europe and more broadly.

From a European perspective, the choice is not simply between an open and a closed architecture. Issues of data privacy have a higher profile in Europe than the USA, with legal controls on data collection and storage. Caliper and xAPI both have data stores that could include privacy controls, and xAPI developers seem to have been more active in addressing this issue. An additional consideration is that business models based on ownership, transfer and analysis of data may not be compatible with European approaches to data protection.

In general, any individual organisation can make a coherent decision to stick to a closed model, but European agencies have reasons to promote plurality, choice and localisation.

European Committee for Standardisation (CEN) instruments for consensus in this area are currently inactive, and there is currently no pan-European instrument for the harmonisation of learning analytics. National and European beneficiaries of learning analytics therefore need to provide support and leadership in the development of interoperability standards. This can be done in collaboration with stakeholder groups such as the Apereo Foundation or SoLAR (see the Inventory).

At an institutional level, the increasing diversity of software and physical devices used to access that software represents a growing challenge for those who would like to integrate data for learning analytics. Significant problems are that most institutional data systems are not interoperable and are controlled by different sections of an institution, so aggregating administrative data, library data, assessment data, classroom data and online data is likely to pose challenges.

The issue of interoperability is not purely concerned with data access. Analysis and interpretation require that the meaning of the data, including differences between contexts, needs to be taken into account. For example, the term ‘learner’ may be used by the student records system to refer to everyone who has registered, in the classroom to refer to everyone who has registered and has gone on to take part in classes, and in the exams office to refer to everyone who has taken an exam. As a result of this lack of standardisation, different parts of the institution will produce different learner counts, preventing meaningful integration of the data. At a wider level, ‘learner’ may refer to a young child in one context and to a postgraduate student in another. If these data are stripped of their context, this may lead to mistaken attempts to amalgamate findings about sets of learners.

Action point: At European and national levels, work with stakeholder groups such as the Apereo Foundation and SoLAR to provide support and leadership in the development of interoperability standards.

Action point: At national level, work to share interoperability standards widely and to adapt them to local language and context, where appropriate.

5 Concluding remarks

Learning analytics offer the opportunity to take data that are generated as people engage in learning, and use these data to help improve learning and teaching. This is a vision that has proved popular around the world and, as a result, learning analytics has become a fast-developing field. Many of the early adopters are based in Europe, and countries such as the Netherlands, Norway and Denmark are already taking a lead in this area.

While learning analytics have developed very quickly in the past five years, educational policy in Europe has developed at a slower pace. Most policy that influences learning analytics was developed in other contexts. As a result, current policy may need to be reassessed in order for it to work as an enabler for implementation of learning analytics, for example, in areas such as data protection. In addition, policy is not yet supporting strategic development within this field.

Today, many learning management systems and digital technologies can produce visualisations of data in a way that may be labelled ‘learning analytics’. These data visualisations are not necessarily ‘actionable’ in the way that learning analytics should be – they do not reveal what actions need to be taken in order to improve learning or teaching. In many cases, there is little or no research evidence to show that these tools genuinely improve learning and teaching.

Much of the current work on learning analytics is concerned with predicting which students are likely to drop out, with a view to providing those students with additional support. This is a worthwhile aim, but learning analytics offer many other possibilities. Learning analytics could be used to tackle big problems and European priority areas for education and training such as open and innovative education and training; learning outcomes that focus on employability, innovation, active citizenship and well-being; and recognition of skills and qualifications to facilitate learning and labour mobility.

The Action List for Learning Analytics set out in this report offers a way of resolving these problems by aligning work across Europe. The Action List focuses on seven areas of activity. It proposes a set of actions that will align the work of educators, researchers, developers and policymakers so that learning analytics are used to drive work in Europe’s priority areas for education and training. These groups can use the Action List to ensure that open and innovative education and training, which fully embraces the digital era, becomes a reality.

The Action List’s points set out a programme of work at European, national and local levels. This work should begin with strategic actions at European level by creating a common European vision outlining strategic objectives. This should be followed by the development of a roadmap for learning analytics within Europe, according to which responsibilities would be aligned for development of learning analytics within Europe. The Action List for Analytics points set out how this work should begin.

2. Develop a common vision in Europe: Work in a multi-stakeholder group to consider priority areas for education and training and identify what learning analytics should do and how they should look within that area.

3. Develop a roadmap: Work with learning analytics experts, educators, vendors and policymakers to develop a roadmap for learning analytics within Europe that is aligned with Europe’s priority areas, fills gaps in the European toolkit and supports the development of sustainable tools and practices.

4. Assign responsibility: Identify responsible organisations and people for leading and coordinating work on learning analytics and implementing the learning analytics roadmap, as well as the individuals and organisations who will be key national contacts in different European countries.

These three actions will provide a firm basis for further action to develop and implement learning analytics within Europe.

Annex 1: Inventory of Tools, Practices and Policies

This section provides a three-part Inventory that brings together evidence of practical implementations of learning analytics and documents the state of the art. It covers:

• Tools

• Policy documents

• Practices.

The Inventory was developed using existing academic literature, policy documents, practitioner-generated reports (grey literature) and contributions from the learning analytics community. It provides a ‘broad-but-shallow’ collection of reference points.

The Inventory is also openly available online on the Cloudworks site at http://cloudworks.ac.uk/cloudscape/view/2959, where it can be extended or amended by researchers, practitioners and anyone with a knowledge of the field.

Each entry begins with a brief synopsis of the tool, practice or policy. Entries end with details of the items maturity and any evidence that it has proved useful in practice, as well as links to key resources and references that can be used to access more detailed information.

All entries include

• Inventory type – what the item is used or intended for

• Keywords – specialist terms are explained in the Glossary below

Entries relating to tools include

• Role of analytics – the different uses of analytics

• Data sources – where the data originate

• Learning – educational sector in which the tool is used

• Supply model – how the tool is accessed

• Origin – where the tool originated

• Ethics and privacy – details of these where available

• Language – the language used by the tool

Entries relating to policies include

• Document source – where the policy originated

• Geographical – region where the policy applies

• Relationships – areas covered by the policy

Entries related to practices include:

• Learning – educational sector to which the practice applies

• Geographical – where the practice is applied

• Pedagogic – theory of teaching and learning that underpins the practice

• Tools used – any relevant tools

• Design and implementation – how the practice developed and is applied

Tools: school level

1. ASSISTments

Synopsis

ASSISTments is an intelligent tutoring system developed by Neal Heffernan and colleagues that is researched at Worcester Polytechnic Institute (WPI) in collaboration with a variety of universities and organisations in the United States. The core system was designed to give progressive hints to students who answer a question incorrectly, in order to simulate the type of instantaneous directed feedback a tutor would provide. From this platform there have been a variety of studies of the system focusing on how to use the student log data generated from the system effectively. For example, studies have been carried out to see how these data can influence parent engagement or predict performance on high stakes tests.

Classification

Inventory type:

learner support tool

analytics for assessment

Role of analytics:

alerting

visualisation

prediction

recommendation

Data sources:

uses own data

Keywords:

intelligent tutoring system

Tool in Context

Learning:

secondary education

Supply model:

Privately hosted software

Origin:

Worcester Polytechnic Institute, United States of America

Ethics and privacy:

Languages:

English

Maturity and Evidence of Utility

ASSISTments has been used as a research platform from WPI in association with a variety of universities. It has been expanding in terms of adoption. Two hundred and sixteen counties in the United States used the system between 28 February and 28 April 2012.

Further Information

Tool provider’s website: http://bit.ly/1SXSbbh

Highlighted research: http://bit.ly/1X5TCne, http://bit.ly/1P97xWv

Map of US districts that used ASSISTments in spring 2012: http://bit.ly/1X5TlRd

2. Bettermarks

Synopsis

The Bettermarks program supports mathematics teaching through the use of adapted content, connected with over 100 textbooks. Teachers can either assign online lessons to students or let the system assign them based on students’ skill levels. As students complete lessons, Bettermarks analyses their performance and behaviours to detect gaps in knowledge, suggest lessons for improvement or provide additional challenges.

The program also incorporates a teacher centre, where student performance data can be accessed. Teachers can access ‘at-a-glance’ reports on completion and pass rates across the module. Additionally, they may look at individual student results and progression.

This system uses any web browser and does not require downloaded software.

Classification

Inventory type:

smart system

Role of analytics:

summarisation & description

adaptation

recommendation

Data sources:

uses own data

Keywords:

adaptive modelling

Tool in Context

Learning:

school

Supply model:

self-hosted server software

Origin:

Bettermarks, Germany

Ethics and privacy:

Little information is available about the company’s ethics and privacy policies. On their website, they state that ‘login data and exercise data’ are saved. They further explain that students’ email addresses or real names are not required, and that no data are shared with third parties. However, no information is available about data storage methods.

Languages:

English, German, Dutch, Spanish

Maturity and Evidence of Utility

Currently no information is available about examples of the program’s use or effectiveness. Preliminary studies have suggested that students who use the system receive better marks than those who do not. However, this information was internally sourced (see link below) and has not been peer reviewed.

Further Information

Tool provider’s website: http://bettermarks.com/

Interview with CEO: http://bit.ly/1P0Ia78

Result: http://bit.ly/1PWplBH

Example of use:

• ‘Success Stories’ brochure: http://bit.ly/1QIkXuX

3. Bingel

Synopsis

Bingel is a Belgian-based online exercise platform for primary education. It is currently used by more than 70% of Dutch-speaking students, and has recently been introduced in Finland and Sweden. The platform includes over 3,500 course-related exercises in eight subjects, and is available for grade levels 1-6.

Bingel is an adaptive platform that incorporates online exercises, and provides automatic corrections and real-time feedback to students. Teachers can use the platform in the classroom or assign students tasks to carry out at home, and the system can be used on PCs or tablets. Individual and personalised tasks can be assigned to each student, and the tool itself can generate personalised learning paths through the materials. The tasks adopt a gamified approach to learning.

Classification

Inventory type:

learning environment tool

smart system

Role of analytics:

modeling

adaptation

Data sources:

uses own data

Keywords:

adaptive learning

gamified learning

Tool in Context

Learning:

school

Supply model:

self-hosted server software

Origin:

Sanoma Group, Finland

Ethics and privacy:

Bingel has a privacy policy that explicitly outlines the use of student data. Data are stored on the platform only for the current school year and can be accessed by teachers. During the summer holidays, student data are permanently deleted.

Languages:

Dutch, Finnish, Swedish

Maturity and Evidence of Utility

Bingel has been offered to schools for over five years, and is now used by a large percentage of Dutch-speaking schools in Belgium. However, there is no information available on its website in regards to evidence of learning gains. Thus, research-backed findings are needed to further demonstrate maturity and evidence of utility.

Further Information

Tool provider’s website: http://www.bingelsite.be/ (In Dutch)

Vendor website: https://sanoma.com/

4. Cito LUVS

Synopsis

LUVS is a tool for planning and tracking school-aged students’ online educational activities. It is produced by Cito, a Dutch company which produces testing and examination services for primary and secondary education and is commissioned by the Dutch government. Examinations are available for all mandatory school subjects.

The LUVS tool connects with currently existing school administration systems to aggregate student assessment results across subjects and grade levels. Within the LUVS dashboard, teachers and administrators can view and analyse test results on the individual student, classroom, school or district level.

The tool is an additional add-on for schools already incorporating Cito testing services.

Classification

Inventory type:

Analytics for assessment

Role of analytics:

summarisation

description

Data sources:

Uses own data

Keywords:

assessment

academic performance

performance

Tool in Context

Learning:

school

Supply model:

self-hosted server software

Origin:

Cito, Netherlands

Ethics and privacy:

No information about ethics or privacy is available at this time

Languages:

Dutch

Maturity and Evidence of Utility

No information is currently available on the Cito website in regards to evidence-based results of using their product. However, the product’s main function is description and consolidation of data for teachers and administrators. In this context, the product is well used and appears stable.

Further Information

Tool provider’s website: http://bit.ly/1wWo09J (in Dutch)

Description of Cito’s role in the Netherlands: http://bit.ly/1XfovHJ

5. Civitas Learning

Synopsis

Civitas Learning is a US-based company that works directly with higher education institutions to build bespoke data science and learning analytics tools that make use of currently available student data. Stated potential data sources include virtual learning environments, social media, card swipes, libraries and housing.

Civitas Learning’s Student Insights Platform aggregates student data and uses a variety of tools for analysis and visualisation. The Illume tool demonstrates historic and predictive student data for institutional leaders and student service providers. The Inspire for Faculty tool provides real-time analysis of student engagement and behaviours in specific modules, as well as data visualisation tools and predictive modelling. Similarly, the Inspire for Advisor tool visualises student performance and success across modules and predicts programme completion. Degree Map helps students and advisors make individual degree plans. Additionally, the Hoot.me tool helps teachers build module-specific Facebook Q&A sections. Finally, Civitas Learning provides a course-scheduling platform for module enrolment.

Each of this wide variety of tools is individually developed with partnering institutions to fit their analytics needs, so platform uses and data sources vary widely. Civitas currently work with over 70 partnering institutions in the USA.

Classification

Inventory type:

smart system

Role of analytics:

alerting

summary and description

statistical inference

prediction

data visualisation

Data sources:

sources data from other systems: management information systems, virtual learning environment, social media, card swipe (varies by institution)

Keywords:

analytics, prediction, retention, student performance, visualisation

Tool in Context

Learning:

post-compulsory

Supply model:

varies by institution

Origin:

Civitas Learning, United States of America

Ethics and privacy:

Civitas Learning builds platforms in partnership with subscribing universities. No specific ethics and privacy statement is listed on the Civitas Learning website, and policies may vary by institution.

Languages:

English

Maturity and Evidence of Utility

Despite its wide use, relatively little empirical research has been conducted to test the effectiveness of Civitas platforms at partnering institutions. The research that does exist shows limited results. For instance, in a randomised control trial at University of Maryland University College, users of Civitas’ Illume application outperformed non-users by just 3%. Thus, more rigorous, empirical evidence of the platform’s maturity is suggested for the future.

Further Information

Tool provider’s website: https://www.civitaslearning.com/

Civitas platform options: http://bit.ly/1ZN7xhP

List of partnering institutions: http://bit.ly/1ZNbh2D

Example(s) of use:

• Case study at University of Maryland University College: http://bit.ly/1lKdKxo

• Case study in three contexts: http://bit.ly/1PNqCes

6. Cognitive Tutor software

Synopsis

Cognitive Tutor is an intelligent tutoring software provided by the US company Carnegie Learning. This web-based software is mainly used to teach mathematics to 9-12 grade students. The software provides personalised learning activities and customised feedback for several prepared mathematics courses based on a domain, tutoring, and student skill models.

Two learning analytics relevant components of this software are the 'Skillometer' and the teacher reports. The 'Skillometer' is a visual indicator of students' progress in mastering skills. It gives the student an indicator of skill mastery for each achievable skill of a learning unit. The level of mastery shown by the tool expresses a prediction about the ability to demonstrate this skill in future again. The data for this visualisation stem from the tracking of the interaction of the student with the software.

Teachers are supported with several reports that are generated by the software. The class progress report shows the amount of active students on each unit. The class skill alert report shows for each skill the skill mastery level for each student. The student detailed report shows for each student the amount of mastered skills, time spent, amount of completed units, sections, and problems. The detail by section report shows information for each student on a unit-by-unit level. Another report shows aggregated data for each unit. The student skill alert report shows units of underperformance. The class assessment reports allow comparison of pre-test with post-test results by topic, or by problem on class level. The student assessment reports show pre-test and post-tests results by topic, or by problem on student level. These reports are intended to support teachers with their instructional decision-making.

Classification

Inventory type:

learning environment tool

smart system

learner support tool

analytics for assessment

Role of analytics:

alerting

summary and description

visualisation

prediction

modelling

adaptation

Data sources:

uses own data

Keywords:

adaptive, cognitive tutor, knowledge tracing

Tool in Context

Learning:

school

Supply model:

desktop tool (Java Webstart application or browser based)

Origin:

Carnegie Learning, United States of America

Ethics and privacy:

The company provides a privacy policy.

Languages:

English

Maturity and Evidence of Utility

Research on Cognitive Tutor dates back to the 1980s. The software was extensively trialled, for example, the Cognitive Tutor algebra 1 course was used by 2000 US schools in 2004. Furthermore, several scientific reviews have been published.

Further Information

Tool provider’s website: https://www.carnegielearning.com/learning-solutions/software/cognitive-tutor

History of Cognitive Tutor: http://ctat.pact.cs.cmu.edu/index.php?id=timeline

7. Conexus – Vokal

Synopsis

Conexus is a Norwegian company with a number of products and services focused on the use of data for school-level education, professional development and management.

The product known as Vokal compiles background, activity and assessment data from various sources. It provides analysis and reporting at individual and group level, as well as tools to support the evaluation and improvement of pedagogic practice. Data are gathered from a range of external sources – Conexus has worked with several publishers – and is combined with anonymised data from Statistics Norway, the student survey and national tests.

Vokal also includes support for adaptivity; Knewton is used for progression analysis in individual subjects. Conexus emphasises, however, that its tools are intended to support pedagogic practice, and that Vokal is not an automated teaching system.

Classification

Inventory type:

learning environment tool

smart system

design and planning tool

analytics for assessment

Role of analytics:

summarisation & description

statistical inference

visualisation

modelling

adaptation

Data sources:

uses data from statistical services, sources data from other system(s): management information systems, virtual learning environment, publisher online content, assessment systems

Keywords:

Tool in Context

Learning:

school

Supply model:

desktop tool/self-hosted server software/privately-hosted software/shared service model

Origin:

Conexus, Norway

Ethics and privacy:

Languages:

Maturity and Evidence of Utility

Conexus state that Vokal is used in 75% of Norwegian primary schools.

Further Information

Tool provider’s website: http://www.conexus.no/vokal/ (Norwegian language site)

Tool provider’s website: http://en.conexus.no (English language site, with less detail)

Presentation by Yngve Lindvig (Conexus): http://bit.ly/Conexus-Lindvig

Case study on data sharing by LACE Project (section 2.7): http://bit.ly/Conexus-LACE

Gartner Vendors Report: http://bit.ly/Conexus-Gartner (subscription required to access report)

8. FFT Aspire

Synopsis

FFT, the Fischer Family Trust, is a UK non-profit organisation that provides services for UK-based education, such as the National Pupil Database for the Department for Education, and school analyses.

The software FFT Aspire is a data analysis and reporting tool for schools. It provides several dashboards showing facets of school performance, such as past attainment, progression, attendance and future performance estimates. It targets several users groups, such as teachers, subject leaders, department heads, senior school leaders, advisors, local authorities and governors.

The range of dashboards includes an overall school dashboard, a subject dashboard for department heads, subject leaders, and teachers, a governor dashboard (helping schools to share information with their governing bodies), a student explorer dashboard, a collaboration dashboard (to compare school performance with other schools), and a target-setting dashboard (school performance targets). Furthermore, the tool supports the creation of custom analyses and dashboards such as a three-year dashboard, a dashboard relating to children with special educational needs, and a dashboard of high attainers.

Classification

Inventory type:

design and planning tool

Role of analytics:

alerting

summary and description

visualisation

prediction

Data sources:

uses data from statistical services, sources data from other system(s): management information systems

Keywords:

data analysis, reporting, future planning

Tool in Context

Learning:

school/

Supply model:

shared service model

Origin:

Fischer Family Trust, UK

Ethics and privacy:

Languages:

English

Maturity and Evidence of Utility

FFT Education Ltd was established in 2001. FFT Aspire is the successor of FFT Live. Virtually all local authorities in England and Wales have a FFT Live subscription. Similar coverage is assumed for FFT Aspire.

Example of use in England: http://bit.ly/1Ziwdzw

Further Information

Tool provider’s website: https://fftaspire.org/

Example(s) of use:

Case studies: https://fftaspire.org/help/casestudies

About FFT: http://www.fft.org.uk/about-us/Fischer-Family-Trust.aspx

Documentation:

https://fftaspire.org/help/support

https://www.youtube.com/channel/UC0HdON1oVddKt9ZEojI5VC

9. itslearning

Synopsis

Developed for K-12 classrooms, itslearning is a learning management system with functionality for course management and delivery, curriculum management, reporting and analytics. The reporting and analytics features incorporate functionality for standards mastery reporting (enabling teachers to see the percentage of students who have mastered each course standard), and a content recommendation engine that ‘provides remediation and enrichment activities based on student performance against learning objectives’. This enables the identification of students who are struggling to meet learning objectives and assigns them activities for reinforcement. The itslearning recommendation engine can automate ‘most’ of the process of ‘identification of students who are struggling to meet learning objectives and assign them activities for reinforcement’.

The reporting features enable students, teachers, administrators, mentors and parents to view student aspects of students’ progress via their personalised dashboard. Teachers and administrators can filter views of how classes have performed with respect to specific learning objectives by date, or by status (for example, to show only the students who have exceeded a particular learning objective). A parent dashboard enables parents to see their child’s progress on tasks, grades and towards learning objectives, as well as their individual learning plans, behaviour and attendance.

Classification

Inventory type:

learning environment tool

smart system

learner support tool

design and planning tool,

analytics for assessment

Role of analytics:

summary and description

recommendation

Data sources:

uses own data, uses data from statistical services, sources data from other system(s): management information systems, virtual learning environment, audio/video playback, assessment system, forums

Keywords:

reporting, recommendation system

Tool in Context

Learning:

school (K-12)

Supply model:

self-hosted server software

Origin:

Itslearning, Norwa (started as a computer engineering project at Bergen University College in 1998)

Ethics and privacy:

Privacy matters have been considered in the software design and service provision: there is a privacy section in which administrators can edit settings.

Languages:

Maturity and Evidence of Utility

The itslearning platform was established in 1999, developing from a computer-engineering project at Bergen University College. It now has over 7 million active users.

Tolgfors, B., & Öhman, M. (2015). The implications of assessment for learning in physical education and health. European Physical Education Review. doi: 10.1177/1356336x15595006‘

Further Information

Aggregated learning objectives report: https://vimeo.com/118518649

Tool provider’s website: http://www.itslearning.net/reporting-analytics

Brief description of recommendation engine http://www.itslearning.co.uk/mobile-and-byod

itslearning company background http://www.itslearning.net/our-story

10. Metacog

Synopsis

Metacog uses a content pool of interactive ‘learning objects’ to personalise content and pace for individual learners. Students are asked to complete a real-world task using the platform, and data are collected that relate to their usage behaviours, including click data, time stamps and correct/incorrect responses. The platform’s API analyses student interactions in order to assess their understanding of the content. It is possible to use Metacog in collaboration with pre-existing resources.

Students using Metacog have access to information about whether they have performed a task correctly. A leader board is also created so that students can compare their performance with peers. For teachers, the platform colour-codes performance as green, yellow or red to indicate understanding of the material on individual tasks or over time. The platform also helps teachers to group students based on their current understanding, in order to provide individualised assignments or additional resources. Teachers can additionally review which part of a task is proving to be a stumbling block for individual students or for the class as a whole. On an administrative or publisher level, the platform can be used on a macro scale to help determine where to invest additional resources by highlighting gaps in understanding across classrooms.

Classification

Inventory type:

smart system

Role of analytics:

summary and description

visualisation

adaptation

Data sources:

uses own data: student behaviours within the platform

Keywords:

adaptation, visualisation

Tool in Context

Learning:

school

Supply model:

self-hosted server software

Origin:

Metacog; United States of America

Ethics and privacy:

The platform only collects data that is specified by the organisation using it. Individual organisations may choose to exclude information such as student identification. The company has a Student Privacy Pledge, which highlights that student data will be kept private and secure, and will not be shared with third parties.

Languages:

English

Maturity and Evidence of Utility

Metacog’s website does not currently share examples of the platform’s use and no empirical studies of its utility have been found. Examples of practice and results are necessary to assess its maturity and evidence of utility.

Further Information

Tool provider’s website: http://metacog.com/

Introductory White Paper: http://bit.ly/1R93rAm

11. Schoolzilla

Synopsis

Schoolzilla provides a data warehouse and associated data dashboard targeted at the K-12 US market. It provides ‘connectors’ that allow data to be integrated into the system through nightly updates from multiple sources such as assessment, behaviour, enrolment, grade, observation, and student information databases.

Schoolzilla provides multiple views of these integrated data through a dashboard library. Representations for teachers, school leaders, school district leaders and system administrators are provided in the library, and system administrators may customise these using Tableau’s data visualisation products.

Teachers can use dashboards such as the ‘Early warning signs’ report to identify at-risk students. For example, this dashboard brings together data on attendance, behaviour and grades, and allows users to view data for schools as a whole, to compare schools (for district leaders) and to drill down to view data about individuals. System administrators can monitor the quality of the data within the system using dashboards that present the results of data audits including automatic checks for missing or malformed data.

Classification

Inventory type:

design and planning tool

analytics for assessment

Role of analytics:

alerting

summary and description

visualisation

Data sources:

sources data from other system(s): management information systems, virtual learning environment, assessment system

Keywords:

data warehouse

Tool in Context

Learning:

school (targeting the US K-12 market)

Supply model:

privately hosted software

Origin:

Aspire Public Schools: institutional project, United States of America

Ethics and privacy:

The Schoolzilla terms of service include paragraphs about intellectual property rights, confidentiality and privacy. These terms of service include a ‘plain English’ version of each section.

Languages:

English

Maturity and Evidence of Utility

The basis of Schoolzilla was developed by staff at Aspire Public Schools, and used within the Aspire Schools group for three years before being spun off as a separate entity in 2013. As of January 2016, it was in use by 580 schools across the US: https://schoolzilla.com/infographic-2015-year-in-review/

Further Information

Tool provider’s website: https://schoolzilla.com/

Example(s) of use:

There are some reviews of Schoolzilla available, however some of these provide demonstrations of the system as opposed to views on use in practice: https://www.edsurge.com/product-reviews/schoolzilla/educator-reviews

12. SNAPP

Synopsis

The Social Networks Adapting Pedagogical Practice (SNAPP) tool performs real-time social network analysis and data visualisation of forum discussion activity on commercial and open source learning management systems. Reasons for using such a tool include the identification of isolated students, facilitator-centric network patterns, group malfunction, and users who bridge smaller networks.

Some basic descriptive data are available about the users, including total number of posts, number of posts per user, post and reply frequencies by user, and who is interacting with whom.

Research conducted with the tool includes: monitoring student networks, participant interaction over time, and assessing broad-based admissions.

Classification

Inventory type:

learner support tool

design and planning tool

analytics for assessment

Role of analytics:

statistical inference

visualisation

summary and description

modelling

Data sources:

uses data from LMS discussion boards

Keywords:

social network analysis, visualisation

Tool in Context

Learning:

post-secondary education

Supply model:

privately hosted software

Origin:

University of Wollongong, Australia

Ethics and privacy:

Languages:

English

Maturity and Evidence of Utility

The project includes both national and international partners. There have been many research studies conducted with the tool.

Further Information

Tool provider’s website: http://www.snappvis.org/

Highlighted research – ten research publications related to SNAPP: http://bit.ly/1R9kXnS

13. VitalSource CourseSmart

Synopsis

CourseSmart Analytics are available to teachers whose institutions participate in an integration between the institution's LMS and CourseSmart's eTextbook. The integration is effected using IMS Global’s Learning Tools Interoperability standard (LTI). CourseSmart’s analytics dashboard presents a measurement of students’ engagement with digital course materials. A centrepiece of this dashboard is the CourseSmart Engagement Index Technology™, a proprietary algorithm that evaluates standard usage data – such as number of pages read, number of times a student opened/interacted with the digital textbook, number of days the student used the textbook, time spent reading, number of highlights, number of bookmarks, and number of notes – and assimilates these data to provide an overall assessment of students' engagement with the material.

The analytics are intended to give teachers insights into their students’ engagement with and patterns of usage of e-books, with a view to enabling teachers make interventions based on this data.

VitalSource acquired CourseSmart in early 2014, and press releases issued in October 2015 announced ‘the upcoming re-launch of our analytics product’. However, there have been no further announcements.

Classification

Inventory type:

general analytics tool

learning environment tool

Role of analytics:

Alerting

summary and description

visualisation

Data sources:

uses own data, sources data from other system(s): virtual learning environment,

Keywords:

e-book

Tool in Context

Learning:

school, post-compulsory

Supply model:

self-hosted server software

Origin:

CourseSmart, United States of America

Ethics and privacy:

VitalSource has a Privacy & Cookies Policy

Languages:

English

Maturity and Evidence of Utility

CourseSmart was founded in 2007 by a conglomeration of publishers. Beta testing of CourseSmart Analytics started in late 2012, and the first version was released in summer 2013.

Junco & Clem (2015) carried out a study of 236 students using CourseSmart in the Spring 2013 semester. They found that CourseSmart Engagement Index ‘was a significant predictor of course grades across disciplines, instructors, and course sections’. However, ‘the number of days students spent reading was a more powerful predictor of course outcomes. This suggests that the calculated Engagement Index does not yet capture the important factors related to engagement with the textbook’. Juno & Clem conclude that the ‘CourseSmart Engagement Index needs to be refined’ and this may be happening.

Further Information

Tool provider’s website: https://www.vitalsource.com

Description by JISC consultant: http://analytics.jiscinvolve.org/wp/2014/10/06/more-sophisticated-learner-engagement-metrics-and-doing-something-with-them/

Descriptions from help material: Navigating the Analytics Dashboard, About Analytics

Junco, R., & Clem, C. (2015). Predicting course outcomes with digital textbook usage data. The Internet and Higher Education, 27, 54-63. doi: 10.1016/j.iheduc.2015.06.001

Tools: Higher Education

14. Degree Compass (Desire2Learn)

Synopsis

Course selection can prove challenging for students. Desire2Learn cites research by Complete College America, which found that students take 20% (on average) more classes than are needed to graduate. Providing help with course selection can therefore cut tuition costs. At-risk students who are not as likely to make it to graduation are potentially the population that is in the most need of support in decision making, in order to help increase retention and graduation rates at college.

Using information about other students’ enrolments, this system provides recommendations as to which courses the students should take in order to complete their degree as well as which courses they are most likely to complete.

The Degree Compass application aims to increase student success by:

• Providing students with academic advice from the time they start school;

• Monitoring progress and offering on-going personalised course and degree path recommendations;

• Reducing time-to-degree with better course selection.

Classification

Inventory type:

learning environment tool

smart system

learner support tool

Role of analytics:

statistical inference

prediction

modelling

recommendation

Data sources:

uses own data

Keywords:

at-risk students, course selection

Tool in Context

Learning:

post-secondary education

Supply model:

privately hosted software

Origin:

Desire2Learn, United States of America

Ethics and privacy:

Languages:

English

Maturity and Evidence of Utility

Studies at Tennessee schools have shown that at-risk students who use this tool have earned higher grades. More than 90% of students who took a 4-star course as recommended by this system earned an A or B in the course.

Further Information

Press release giving details of Degree Compass: http://bit.ly/1PEQKMx

Detailed Educause overview of Degree Compass: http://bit.ly/1VUIMPz

Related research:

Denley, Tristan (2012), ‘Austin Peay State University: Degree Compass’ in Oblinger, Diana, (ed.) Game Changers: Education and Information Technologies. Educause, 2012.

15. Knewton

Synopsis

Knewton is an adaptive learning software company that provides platforms for personalised education. The company was founded in 2008 and formed a partnership with Pearson Education in 2011. Over ten million students have used their adaptive learning platforms at the primary, secondary and university levels. Many programs are available at different educational levels, and Knewton often works with schools or universities to create custom platforms that fit institutional needs. From a student perspective, the program uses algorithms based on student performance and behaviours to suggest lessons via differentiated instruction, as well as to provide students with information about their progress. It incorporates immediate feedback, community collaborative forums and gamification to encourage participation. From an educator perspective, the program supports data summarisation and visualisation at the classroom or individual student level. Using a ‘stop light’ system, students are categorised for interventions as ‘ahead of track,’ ‘on track,’ ‘off track’ or ‘very behind.’

Classification

Inventory type:

smart system

Role of analytics:

adaptation

visualisation

summary and description

Data sources:

uses own data

Keywords:

adaptive learning

classification

Tool in Context

Learning:

school, post-compulsory

Supply model:

self-hosted server software

Origin:

Knewton, United States of America

Ethics and privacy:

Little information is available about privacy and ethics. This is likely to vary by institution.

Languages:

English

Maturity and Evidence of Utility

Knewton is perhaps the most established adaptive learning software, and partners with big names in the education world, such as Pearson Education and Houghton Mifflin Harcourt, and in the tech world, such as HP and Microsoft. Considering the vast number of students using their platforms, only a limited amount of evidence is promoted on the Knewton website at both the school and university level. For example, it is argued that an increase in retention was seen at Arizona University from 64% to 75%, however the student cohorts examined were of varying size and cohorts studied the courses in different academic years.

Further Information

Tool provider’s website: http://www.knewton.com

Platform summary white paper: http://knewt.ly/1rCMS61

Technical white paper: http://bit.ly/1XfovHJ

16. Loop

Synopsis

Loop is an open source analytics tool funded by the Australian Office for Learning and Teaching. The tool can be connected with Moodle or Blackboard to provide a tool for teachers to visualise student behaviours in their learning management system. The dashboard component displays student log data through the learning management site, such as class materials accessed, discussion forum activity, and assessment performance. These data can be viewed at the classroom or individual student level. At the same time, the tool incorporates information about the course structure and schedule within its visualisations. In this sense, the project aims to incorporate a ‘pedagogical helper tool’ to aid teachers in data interpretation that make sense in their specific context. In 2015, the tool was piloted with four courses run by three Australian universities, with hopes of a wide-scale release following soon.

Classification

Inventory type:

learning environment tool

Role of analytics:

visualising

Data sources:

Uses data from other systems: Moodle or Blackboard

Keywords:

visualisation

learning management system

Tool in Context

Learning:

post-compulsory

Supply model:

self-hosted server software

Origin:

collaborative project, Government funded, Australia

Ethics and privacy:

No information about ethics or privacy is available at this time

Languages:

English

Maturity and Evidence of Utility

The Australian government funds this project, which is the product of collaboration between three universities and nine leading researchers. However, Loop is currently in an initial pilot stage, with four courses across three universities adopting the tool for one academic year in 2015. At the end of the year, a qualitative study with course instructors is planned, but no findings have yet been released. A full analysis of this initial pilot will be necessarily to confirm the tool’s maturity and evidence of utility.

Further Information

Tool provider’s website: http://bit.ly/24DaLfi

Related papers:

http://bit.ly/1O87cpZ

http://bit.ly/1Nnnc7i

Presentation of the tool: http://bit.ly/24EObQa

17. Open Essayist

Synopsis

Open Essayist, developed by The Open University, UK, is designed to provide automated reflective feedback to learners on draft essays. The underlying idea is to present a computer-based analysis of the most important parts and key words in the writing, so that learners can compare those to what they intended to convey, and adjust their writing in the light of that comparison.

Learners upload their draft essay, and the system then generates a series of different views based on analysis of the text, including: the most prominent words and a graphical view of their distribution through the text; the key sentences in the text, with hints to aid reflection; and a graphical view of the structure of the essay.

The tool is intended as a formative, developmental tool rather than for summative assessment.

Classification

Inventory type:

learner support tool

analytics for assessment

Role of analytics:

summary and description

visualisation

Data sources:

uses own data (learner uploads)

Keywords:

assessment, natural language processing, visualisation

Tool in Context

Learning:

higher education

Supply model:

privately hosted software

Origin:

Open Essayist/SAFeSEA projects: collaborative project, OU, United Kingdom

Ethics and privacy:

Feedback is given direct to the individual learner, not shared or distributed to others.

Languages:

English

Maturity and Evidence of Utility

The tool has been trialled successfully with Masters-level students, and the project team is currently looking for wider take-up.

Further Information

Project website: http://www.open.ac.uk/researchprojects/safesea/

Trials with Masters students: http://oro.open.ac.uk/42041/1/lak15_submission_46.pdf

18. OU Analyse

Synopsis

The Knowledge Media Institute (KMi) of The Open University, UK developed OU Analyse – software that predicts students at risk. OU Analyse builds upon two previous projects (Retain and the OU-Microsoft Research Cambridge project). OU Analyse uses machine-learning techniques to develop predictive models based on demographics and VLE usage data.

The software provides a dashboard reporting the aggregated prediction value of several models for all students of a module. Furthermore, the tool discloses the reasoning that underlies its prediction. Currently, the institute is developing a tool that can recommend activities to students to improve their performance. Module chairs, module teams, and student support teams use the predictions of OU Analyse to contact and support students.

Classification

Inventory type:

learner support tool

analytics for assessment

Role of analytics:

alerting

summary and description

visualisation

prediction

modelling

recommendation

Data sources:

sources data from other system(s): management information systems, virtual learning environment, assessment system

Keywords:

prediction

Tool in Context

Learning:

higher education

Supply model:

privately hosted software

Origin:

collaborative or institutional project, OU, United Kingdom

Ethics and privacy:

The Open University has set out ethical guidelines on the use of data for learning analytics.

Languages:

English

Maturity and Evidence of Utility

OU Analyse's development was accompanied by several scientific pilot studies. The software is used across the university and received substantial coverage in the press.

Further Information

Tool provider’s website: https://analyse.kmi.open.ac.uk

Example(s) of use:

http://www.laceproject.eu/publications/analysing-at-risk-students-at-open-university.pdf

http://www.bbc.co.uk/news/technology-3367547

https://www.timeshighereducation.com/news/week-higher-education-%E2%80%93-30-july-2015

http://www.ft.com/cms/s/2/634624c6-312b-11e5-91ac-a5e17d9b4cff.html

Rienties, Bart; Cross, Simon and Zdrahal, Zdenek (2016). Implementing a learning analytics intervention and evaluation framework: what works? In: Kei Daniel, Ben and Butson, Russell eds. Big Data and Learning Analytics in Higher Education: Current Theory and Practice. Heidelberg: Springer.

Kuzilek, Jakub; Hlosta, Martin; Herrmannova, Drahomira; Zdrahal, Zdenek and Wolff, Annika (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, LAK15, pp. 1–16.

See also LAEP Inventory records:

• Ethical use of student data policy – The Open University

• Tribal's Student Insights

19. Student Success Plan

Synopsis

Student Success Plan (SSP) is software to support case management of student support: counselling, coaching and pastoral care. It has lightweight data analytics, principally focused on the management and enhancement of student support services. It is being adopted to support action in relation to predictive analytics.

SSP is designed to improve retention, academic performance, persistence, graduation rates and time to completion. Through counselling, web-based support systems and proactive intervention techniques, students are identified, supported and monitored. The software provides case management tools for handling staff, student, and student-services communications, action planning, planning academic choices, alerting, student self-assessment and progress monitoring.

SSP is not a single ‘out of the box’ solution, but a set of configurable components adopting an open architecture so that they can be integrated into a variety of system landscapes. An Open Source Software edition is available, overseen by the Apereo Foundation.

Classification

Inventory type:

learner support tool

design and planning tool

Role of analytics:

summary and description

Data sources:

uses own data, sources data from other system(s): management information systems

Keywords:

case management, open source

Tool in Context

Learning:

post-compulsory

Supply model:

self-hosted server software

privately hosted software

Origin:

Unicon: technology-enhanced learning vendor (open source)

Previously Sinclair Community College: institutional project

Ethics and privacy:

Ethics and privacy matters were considered from an early stage; the software was developed in an educational setting around existing norms of professional practice in student support.

Languages:

English

Maturity and Evidence of Utility

Student Success Plan was developed by Sinclair Community College (SCC), supported by grant funding, and has been in use for ten years. It has received 11 awards in the USA and is now adopted by Unicon, an Open Source Software development, hosting, and support services provider.

According to Sinclair statistics from 2005 – 2011, students using SSP were five times more likely to graduate. For quarter-to-quarter retention rates (Fall ’10 to Winter ’11), transitioned SSP students (students who had completed the SSP process) had a 37% higher rate of retention than students who qualified for the programme but did not participate and a 26% higher rate of retention than students not designated ‘at risk’ [figures from Unicon web site].

Further Information

Tool provider’s website: http://www.studentsuccessplan.org/

Unicon distribution and services for SSP: http://bit.ly/UniconSSP

Apereo Foundation, the Open Source custodian of SSP: http://bit.ly/SSPApereo

Example(s) of use:

• Educause review article describing rationale and development in SCC: http://bit.ly/SSP-2011

• Gateway to College Network research and evaluation report: http://bit.ly/SSP-GTC

See also LAEP Inventory record:

• Effective learning analytics pilots – Jisc

20. Tribal’s Student Insights

Synopsis

Tribal, based in the UK, is a global provider of software solutions and specialises in products supporting the management of education.

Tribal's Student Insights is a piece of software that is currently being developed to predict student performance and 'at-risk' students from data available in student information systems, including academic performance at entrance, demographics, and assessment results, as well as activity data, such as student interaction, VLE usage, and library usage.

The software generates predictive models about a student's likelihood of passing a module. The software provides dashboards that present this information at student and module level. University educators and managers can use this information, for example, to provide individual student support, or to monitor modules with regard to their predicted performance.

Classification

Inventory type:

learner support tool

analytics for assessment

Role of analytics:

alerting

summary and description

visualisation

prediction

modelling

Data sources:

management information systems, virtual learning environment, assessment system,

Keywords:

prediction

Tool in Context

Learning:

higher education

Supply model:

shared service model

Origin:

Tribal, United Kingdom

Ethics and privacy:

Languages:

English

Maturity and Evidence of Utility

The software is under development. Tribal is working in collaboration with the University of Wolverhampton.

Further Information

Tool provider’s website: http://www.tribalgroup.com/technology/sitsvision/Documents/Tribal%20Student%20Insight.pdf

Example(s) of use:

http://blog.tribalgroup.com/2014/11/24/understanding-your-students-and-strengthening-their-success

http://blog.tribalgroup.com/2016/01/14/business-intelligence-reducing-costs-and-improving-productivity-by-effectively-analysing-data

http://analytics.jiscinvolve.org/wp/2014/10/27/tribal-student-insight-an-interview-with-chris-ballard

21. X-Ray Analytics

Synopsis

X-Ray Analytics is a predictive modeling tool, linked with Moodle and Moodlerooms, which was acquired by Blackboard in 2015. The dashboard provides teachers with visualisations of past behaviours in their learning management system at multiple levels: course, multiple course and intuitional. Its algorithms then make predictions about future performance and behaviours in order to identify ‘at-risk’ students who may be in need of an intervention. The tool also considers student engagement by analysing contributions to online collaborative tools, such as discussion forums, using social network analysis. Students can be identified as at risk depending on the time they have spent in the course, their grades and their discussion forum engagements. X-Ray Analytics uses a cloud-based model and analyses pre-existing data in the learning management system. The tool is expected to be available for all Blackboard courses in the near future.

Classification

Inventory type:

learning environment tool

Role of analytics:

visualisation

summary and description

alerting

Data sources:

sources data from other system(s): virtual learning environment

Keywords:

prediction

predictive modeling

social network analysis

visualisaton

Tool in Context

Learning:

post-compulsory

Supply model:

self-hosted server software

Origin:

Blackboard, United States of America

Ethics and privacy:

Data are stored via a cloud-based model. At present, no information is available that specifically addresses ethics or privacy.

Languages:

English

Maturity and Evidence of Utility

X-Ray Analytics has been acquired by Blackboard, with plans to make the tool available to all users in the near future. There is little information available related to evidence of utility or results of use.

Further Information

Tool provider’s website: http://bit.ly/1Npiu9d

Blackboard press release: http://bit.ly/1L4fSst

Informal account: http://bit.ly/1WXTRSF

Tools: workplace learning

22. Skillaware

Synopsis

Skillaware is a company based in Italy that designs learning environment software for workplace learning and training. The program is used with pre-existing company software or procedures to determine worker effectiveness and areas where training may be useful. Using a variety of tools, Skillaware captures user activities and behaviours within existing software.

The SkillEditor function captures user behaviours and automatically suggests trainings to make workers’ use of various forms of software more productive. The SkillAgent function provides suggestions for next steps in a task when a user appears to need assistance.

In addition, the SkillAnalyzer tool allows company analysts to watch real-time user activity and provide data visualisation for management staff.

Classification

Inventory type:

learner support tool

design and planning tool

Role of analytics:

alerting

recommendation

Data sources:

uses own data, sources data from other systems (varies by user)

Keywords:

data visualisation, user modelling

Tool in Context

Learning:

workplace

Supply model:

self-hosted server software

Origin:

Skillaware: analytics vendor

Ethics and privacy:

No explicit ethics or privacy policies can be found. However, the company works to provide programs for individual use within existing company practices, and ethics practices may vary between customers.

Languages:

English, Italian, German

Maturity and Evidence of Utility

Preliminary analysis in the form of a conference paper supports the software’s validity. However, there are relatively few case studies or examples of use of the software. More empirical evidence will be needed in the future to validate the tool's maturity and evidence of utility.

Further Information

Tool provider’s website: http://skillaware.com/

Preliminary analysis: http://bit.ly/1OPqqxm

White papers: http://skillaware.com/en/documents/

23. WATCHME Project

Synopsis

WATCHME is a European-funded project that uses learning analytics to improve workplace-based feedback and professional development. The acronym stands for Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education. The project has built an electronic portfolio system, which can be used to provide trainees with visualisations and feedback on their development. Their dashboard incorporates data from multiple sources, including self-reporting, online activity data, and qualitative narratives.

A particular type of data model is used to aggregate data and provide ‘Just-in-Time’ feedback to support continued learning. Members of the team of researchers on this project come from multi-disciplinary backgrounds, including areas such as human medicine, veterinary medicine, teacher training and information technology. A prototype of the tool has been developed and the project is currently testing usability.

Classification

Inventory type:

general analytics tool

Role of analytics:

summary and description

visualisation

Data sources:

uses own data, sources data from other system(s): workplace training environments

Keywords:

bayesian network

workplace learning

Tool in Context

Learning:

workplace

Supply model:

This information is not provided on the project’s website

Origin:

collaborative project: EU funded

Ethics and privacy:

No information about ethics and privacy is included on the project’s website

Languages:

English

Maturity and Evidence of Utility

The project is still in its testing phase and relatively little empirical evidence has been released on its usability and impact on workplace learning. The project is a large-scale collaboration with leading researchers in multiple disciplines, which gives weight to its academic rigour.

Analysis of its use can be expected before the project ends in 2017.

Further Information

Tool provider’s website: http://www.project-watchme.eu/

Project collaborators: http://bit.ly/1UBV1mM

Project presentations and publications summaries of use: http://bit.ly/26WXkFM

Tools: all levels, non-and informal learning, other

24. Claned

Synopsis

Claned provides a learning environment that can be used for e-learning in subjects as diverse as medical education and dance education. Claned aims to provide tools that make the learning process visible for both students and teachers, thus implementing components of learning analytics. In the Claned environment, one can embed e-learning materials or upload videos, documents, and slideshows. The system provides automatic keywords and topics, and tracks everything that a learner does. It also provides analytics on the interactions between different learners, focused on collaboration. Claned provides data to teachers by looking for groups of students who act in similar ways, or have similar motivational patterns. The aim is to make the learning process visible to the teacher, so it is clear where supporting materials might be useful, or more support is needed on topics experienced as challenging. Claned also gives the data back to the learner, using a learning tracker tool. The next phase will be to use the data to provide suggestions for individualised learning paths, tailored to help individuals achieve their learning goals.

Classification

Inventory type:

learning environment tool

smart system

learner support tool

analytics for assessment

recommendation

Role of analytics:

Adaptation, description, visualisation

Data sources:

uses own data

Keywords:

Personalisation,

Tool in Context

Learning:

School, training, informal

Supply model:

Privately hosted software

Origin:

technology-enhanced learning vendor, analytics vendor A start-up company with bases in Helsinki, Dubai, Singapore and London.

Ethics and privacy:

The website says “We respect individuals and the privacy of their information. We do not gather data on individuals nor is our technology designed to gather any.”

Languages:

Maturity and Evidence of Utility

Further Information

White paper “- Sampling experiences to enhance learning process” http://www.claned.com/wpcontent/uploads/SAMPLING-EXPERIENCES-TO-ENHANCE-LEARNING-PROCESS_170715.pdf

http://www.claned.com/,

25. Khan Academy analytics

Synopsis

Khan Academy is a set of freely accessible online video-centric learning resources, principally focusing on declarative and procedural knowledge, covering a wide range of subjects at levels suitable for school-aged and adult learners. Learning analytics figure in three ways: as the engine for services offered by the Khan Academy through the web pages; as access to data for analytics processes undertaken by third parties; and as a means of continuous design enhancement.

Khan Academy provides information to teachers/coaches on individual and class-level performance. This provides summary estimates of effort, engagement, and difficulty with the material. The learning materials are mapped to a set of skills, with various mastery levels for each; the teacher/coach can drill down to this level and use the information on progress or difficulty to recommend materials for follow-on or under-pinning skills, or to instigate an alternative learning activity (perhaps outside Khan Academy).

Khan Academy provides a dashboard for learners and this shows progress against skills (as for the teacher/coach) and activity pattern in time and against different skills.

Data access by third parties is via a web-standards-based API and gives differentiated access according to the data type. Video, playlist, topic/skill maps, and exercise data are open access. User-level activity and progress logs are secured, requiring login and authorisation.

Classification

Inventory type:

general analytics tool (access to the API), learner support tool, smart system

Role of analytics:

adaptation, description, modelling, recommendation, summarisation, visualisation

Data sources:

uses own data

Keywords:

adaptation, personalisation, student model

Tool in Context

Learning:

Informal, post-compulsory, school, vocational education and training

Supply model:

privately hosted software: free access

Origin:

Khan Academy: non-profit, United States of America

Ethics and privacy:

Khan Academy is a Student Privacy Pledge signatory and has a public statement of privacy principles, including how data are collected, how it is used, retention, sharing, and user control. They make explicit reference to child users. https://studentprivacypledge.org/

Languages:

There are separate versions of the Khan Academy site in English, French, Norwegian, Portuguese, Spanish and Turkish. Content is available in over 30 languages

Maturity and Evidence of Utility

The data-centred services offered by Khan Academy have continued to evolve with analytics on service usage being a significant source of evidence in the development. There are numerous examples of use worldwide.

Further Information

Tool provider’s website: https://www.khanacademy.org/

Privacy policy: https://www.khanacademy.org/about/privacy-policy

Case studies from the Khan Academy website: http://bit.ly/KhanCaseStudies (general)

Implementations in schools: http://schools.khanacademy.org/

26. Digital Assess – adaptive comparative judgement

Synopsis

Digital Assess provides support for workflow around assessment of coursework or other evidence-based assessment scenarios. The system can be used for conventional assessor marking or for peer assessment. Learning analytics are used to drive a process known as adaptive comparative judgement, which increases the reliability of the assessment.

Adaptive comparative judgement is a development of the assessment approach in which pairs of work by students are compared, using some defined dimensions of quality. Learning analytics drives the adaptive element by automatically determining which pairs to present to which individuals undertaking the assessment, in order to maximise the increase in the reliability of the grading in each round of comparison. Over several rounds of comparative judgement, reliability statistics are computed, as well as statistics that identifies student work that is problematic. The process can also support year-on-year standardisation. The method is particularly applicable to cases where a detailed marking scheme is ill-suited to the object of assessment – for example for creative subjects or ‘soft skills’ – or would be excessively time-consuming, or where peer assessment has a pedagogic role.

Research undertaken by academics and high-stakes awarding bodies has demonstrated that adaptive comparative judgement is a reliable method, exceeding the inter-rater reliability typical of conventional essay marking.

Classification

Inventory type:

analytics for assessment

Role of analytics:

statistical inference

adaptation

Data sources:

uses own data

Keywords:

adaptive comparative judgement, peer assessment

Tool in Context

Learning:

school, vocational education and training, post-compulsory, informal

Supply model:

shared service model

Origin:

Digital Assess: technology-enhanced learning vendor

Ethics and privacy:

The Digital Assess system is designed to support secure high-stakes assessment. Peer assessment is undertaken anonymously, but any free-form assessment has some risk of re-identification.

Languages:

English

Maturity and Evidence of Utility

The tool has been rigorously evaluated by an awarding body (responsible for high-stakes public assessment), and has been piloted at scale at the University of Edinburgh. In 2015, Digital Assess reported raising $3million in new investment.

Further Information

Tool provider’s website: http://digitalassess.com

Outline of adaptive comparative judgement component: http://bit.ly/DA-ACJ

Story about new investment: http://bit.ly/DA-3m

Example(s) of use:

• Case study, adaptive comparative judgement at University of Edinburgh: http://bit.ly/DA-UoE

• Report of an exploratory study undertaken by the Centre for Education Research and Policy of the Assessment and Qualifications Alliance (a UK awarding body): http://bit.ly/DA-AQA

Research: Alastair Pollitt (2012): The Method of Adaptive Comparative Judgement, Assessment in Education: Principles, Policy & Practice, DOI:10.1080/0969594X.2012.665354

27. Learning Analytics Processor

Synopsis

The Learning Analytics Processor (LAP) is software to manage a learning analytics workflow. Typically, this type of workflow is referred to as a pipeline and consists of three distinct phases: input, model execution, and output. The pipeline is built using an open architecture that exposes output from the pipeline via a collection of web service APIs. The LAP is a general-purpose tool designed to meet the need for scaling up learning analytics from manually driven processes to automation of routine technical tasks. The essential purpose of the LAP is to streamline data pre-processing, predictive model use, and results post-processing to make this a more efficient and reliable process. It is configurable, not tied to particular data sources, and agnostic as to the way in which the results of the predictive model are used.

Currently, LAP supports the Marist College Open Academic Analytics Initiative Early Alert and Risk Assessment model but development of additional models as well as feature and scalability enhancements are underway.

Classification

Inventory type:

general analytics tool

Role of analytics:

prediction

modelling

Data sources:

LAP can use data from different sources

Keywords:

workflow, pipeline, predictive analytics, open source

Tool in Context

Learning:

School, vocational education and training, post-compulsory, informal

Supply model:

desktop tool, self-hosted server software, privately-hosted software, shared service model

Origin:

OAAI Project (led by Marist College): collaborative project

Unicon: technology-enhanced learning vendor (open source)

Ethics and privacy:

The original OAAI project was undertaken with ethical research oversight. Since the LAP is a system to automate an analytics pipeline, rather than being a user-facing application, the main concern is system security.

Languages:

Not applicable

Maturity and Evidence of Utility

The LAP arose out of the Open Academic Analytics Initiative (OAAI), led by Marist College (USA), and was developed to automate the processing pipeline that OAAI demonstrated.

It is currently work in progress, being one of the Apereo Foundation’s incubation projects, and is under development by Unicon and Marist, having been selected in a competitive tendering process as a component for the Jisc Effective Learning Analytics pilots.

Further Information

Tool provider’s website, Apereo Foundation, open source custodian: http://bit.ly/LAP-Apereo

LAP and Open Learning Analytics, outline: http://bit.ly/LAP-OLA

LAP features and technical architecture: http://bit.ly/LAP-Details

Jayaprakash, S. M., Moody, E. W., Lauria, E. J. M., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: an open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47. [describes the Open Academic Analytics Initiative project]

See also LAEP Inventory record:

• Effective learning analytics pilots – Jisc

28. Realising an Applied Gaming Eco-system (RAGE)

Synopsis

RAGE is a European-funded project coordinated by the Open University Netherlands, in collaboration with gaming industry professionals and universities in ten European countries. The project focuses on supporting development of ‘applied’ or ‘serious’ games through the use of pilot testing and analytics in real-world educational scenarios. The overall aim is to develop serious games more easily, more quickly and more cost-efficiently. Partnering members belong to an ‘Ecosystem,’ which is a designated social space for collaboration between partners at all levels: commercial, educational, policy, research, and others. The project provides centralised access to software, resources and data, as well as training for developers and educators. Unique to the project is its pilot testing phase, during which developed games can be used in real-world educational scenarios, then analysed for effectiveness using learning analytics and trace data.

Classification

Inventory type:

design and planning tool

Role of analytics:

summary and description

Data sources:

Uses data from other systems: various developed games

Keywords:

games

games-based learning

Tool in Context

Learning:

all levels

Supply model:

unknown

Origin:

Various technology-enhanced learning vendors

Ethics and privacy:

No information about ethics or privacy is available at this time

Languages:

Multiple

Maturity and Evidence of Utility

RAGE is currently running pilot studies on 11 different games in various European countries. As the project is on-going, little evidence has yet been released on project outcomes. However, the large number of collaborations with researchers and industry professionals lends to the project’s maturity and potential for success. The project has also established a business plan for continued work after the European funding has ended.

Further Information

Tool provider’s website: http://rageproject.eu

List of collaborators: http://rageproject.eu/project/partners/

List of pilot projects: http://rageproject.eu/project/pilots/

Dissemination materials and publications: http://rageproject.eu/downloads/

Practices: institutional pilots

29. Arizona State University

Synopsis

Arizona State University (ASU) partnered with private company Knewton Enterprises in 2011 to make use of the Knewton Math Readiness program for its online and blended mathematics modules. The program created personalised learning paths for over 5,000 students registered on remedial mathematics modules. Knewton’s website highlights that the system, ‘continually assesses their mathematical proficiency and adapts accordingly.’ After adopting the system, Knewton states that ASU retention in the remedial mathematics programme increased from 64% to 75%.

In 2015, ASU announced a partnership with Cengage Learning and Knewton Enterprises to create ‘Active Adaptive’ modules. These modules will use analytics similar to the Knewton Math Readiness programme, which adapts students’ learning paths through the module according to their demonstrated proficiency. In combination, Cengage Learning will provide study tools to enhance resources such as note taking and collaboration with classmates.

Classification

Inventory type:

pilot

Keywords:

adaptive

Context of Practice

Learning:

post-compulsory

Geographical:

national: USA

Pedagogic:

This institutional practice relies on adaptive content in remedial and entry-level modules, based on students’ demonstrated proficiency.

Practical Matters

Tools used:

Knewton Enterprises – adaptive learning paths software, Cengage Learning – online study tools

Design and implementation:

Relatively little information about the programme is provided on the Arizona State University website. However, informal press accounts highlight that the system was put into place in 2011 for remedial mathematics courses. Further partnerships with Knewton and Cengage Learning were announced in 2015 to develop more adaptive modules university wide. Informal accounts highlight some push back by university staff, due to the lack of pilot testing or consultation with staff prior to partnerships.

Maturity and Evidence of Utility

Knewton-powered classrooms were rolled out to students without pilot testing. The Knewton website claims an increase in retention from 64% to 75%, however the student cohorts examined were of varying size – 2,419 students without Knewton program and 1,565 with the program – and cohorts were studied at ASU at different time points. Thus, a more robust randomised control trial would be useful to clarify results. An informal account on Inside Higher Ed highlights wide variation in retention rates between individual module sections. Thus, more quantitative and qualitative research are suggested.

Further Information

Overview of Knewton tool: http://knewt.ly/1WK4FCq

Informal account: http://bit.ly/1ZYMI8s

Case study: http://knewt.ly/1nJWsCl

Cengage press release: http://bit.ly/1Nv8TXl

30. Progress and Course Engagement (RioPACE) – Rio Salado College

Synopsis

Rio Salado College is a community college located in Arizona in the USA, which has an online enrolment of over 40,000 students. The college introduced its Progress and Course Engagement (RioPACE) system across the university in 2010. The system uses data modelling and predictive analytics to target interventions aimed at low-performing students.

The system analyses virtual learning environment (VLE) behaviours and compares students to previously successful students. Weekly warning labels are provided individually on a colour-coded traffic light system similar to that employed by Purdue’s Course Signals. Teachers receive weekly reports on student progress and predicted completion, enabling them to target students for interventions if necessary.

Students can also view their warning labels by accessing the RioPACE system within the VLE. Students with a yellow or red indicator are prompted to contact their module teacher for help getting back on track.

Classification

Inventory type:

pilot

Keywords:

prediction, predictive modelling, data mining, classification

Context of Practice

Learning:

post-compulsory

Geographical:

national: USA

Pedagogic:

Rio Salado College is not explicit in its support of one pedagogic framework over another. This institutional practice emphasises the importance of teacher interventions.

Practical Matters

Tools used:

RioPACE is a custom-built system that functions within the institution’s VLE, RioLearn

Design and implementation:

RioPACE has been implemented institution-wide across all modules. The system was created by Rio Salado College. However, the college did collaborate with Purdue University and modelled its system on Purdue’s Course Signals. The college also participates in the Gates-funded WCET project as part of the Predictive Analytics Reporting (PAR) Framework.

Maturity and Evidence of Utility

Preliminary research appears to support the accuracy and validity of RioPACE’s predictive modelling. However, little empirical research has been published or shared with regard to increases in retention as a result of the programme’s adoption.

Further Information

Programme website: http://bit.ly/1nKnTvQ

Interview with associate dean: http://bit.ly/1ZNxs8R

Academic study:

Smith, V., Lange, A., & Huston, D. (2012). Predictive modelling to forecast student outcomes and drive effective interventions in online community college courses. Journal of Asynchronous Learning Networks, 16(3), 51-61.

See also LAEP Inventory records:

• PAR Framework

• Course Signals – Purdue University

31. PredictED – Dublin City University

Synopsis

Dublin City University (DCU) initiated a new learning analytics programme called PredictED in 2014 for ten modules.

PredictED analyses student behaviours in the Moodle virtual learning environment (VLE), and compares them with previously successful students on the same module.

Once a week, participating students receive an email with an updated prediction of whether they are likely to pass or fail the module. Those who appear to be struggling receive study suggestions and resources to support their study. The emails also contain information about how their VLE activity compared with that of their classmates during the previous week.

Classification

Inventory type:

pilot

Keywords:

predictive analytics, self-regulation

Context of Practice

Learning:

post-compulsory

Geographical:

national: Ireland

Pedagogic:

The approach taken by PredictED has not been explicit in respect of pedagogy. The system focuses on student support through the use of predictive analytics. Use of the system is by students for self-regulation.

Practical Matters

Tools used:

PredictED was developed by DCU’s Insight Centre for Data Analytics. It functions within the university’s VLE system, Moodle.

Design and implementation:

The programme is currently only available for a small number of modules. Students must opt in to participate. The system is designed for first-year students in their first term at the university. During the initial trial, around 75% of eligible students opted to participate.

Maturity and Evidence of Utility

Informal accounts highlight that students who opted to participate in the PredictED trial had 3% higher scores than those who did not participate. However, this perceived improvement does not take into account self-selection bias or consider demographics of those who opted in versus those who did not. Thus, more rigorous testing of the system is needed to further determine the system’s maturity and evidence of utility.

Further Information

Academic poster on use of data to predict which students are at risk: http://bit.ly/1SLPZ6A

Informal accounts: http://bit.ly/1PSyk74, http://bit.ly/1UsIGhC

32. Dunchurch Infant School

Synopsis

Dunchurch Infant School is an Early Years institution in the UK that teaches children from pre-school age through their first year of primary school. At the school, observations of students’ play and interactions within the classroom are made and recorded, using the Development Matters system.

Development Matters is non-statutory guidance, produced with support from the Department for Education, to support those working in early childhood education settings to implement the requirements of the Statutory Framework for the Early Foundation Stage. It includes guidelines for seven aspects of learning, which are further divided into seventeen subsections.

Nearly 8,700 observations are recorded in the school in a given year, which has prompted the school to use learning analytics to help manage and interpret the large volumes of data on individual pupils.

The school has a dedicated data analyst who collects observations and creates data visualisation charts for classroom teachers. Teachers can then use these reports as a snapshot of their pupils’ strengths and weaknesses.

The school claims that the percentage of students reaching ‘a good level of development’ has risen from 55% to 77% since adopting analytics.

Classification

Inventory type:

pilot

Keywords:

data visualisation, observation

Context of Practice

Learning:

school

Geographical:

national: UK

Pedagogic:

Dunchurch Infant School uses the Development Matters framework, produced by The British Association for Early Childhood Education.

Practical Matters

Tools used:

The school previously used 2 Build a Profile, an app designed for recording observations. However, a dedicated staff member now develops visualisations in house.

Design and implementation:

This data visualisation and analytics system has been introduced school-wide in all classrooms. Over 75 pre-school children and nearly 60 first-year pupils are involved. The school has a dedicated staff member who collects data and creates visualisations for classroom teachers.

Maturity and Evidence of Utility

The school claims that the percentage of students reaching ‘a good level of development’ has risen from 55% to 77% since adopting analytics. Their school ratings have also improved since the adoption. No empirical evidence is currently available, although a detailed evaluation is planned.

Further Information

Dunchurch Infant School: http://dunchurchinfantschoolandnursery.co.uk/

Informal account of analytics at the school: http://bit.ly/1SiVU1o

Development Matters: http://bit.ly/1nOID5s

School inspection reports: http://bit.ly/1WNyQII

Practices: institutional at scale

33. Course Signals – Purdue University

Synopsis

Course Signals is a predictive learning analytics system originally produced at Purdue University in the USA. The system uses student data to predict those who are at risk of not successfully completing a course. By using predictive modelling of student data and activity in the learning management system (LMS), each student is assigned to a ‘risk group,’ the colours of which are those of a traffic signal – red, yellow, or green.

To use the system, a lecturer or tutor must manually run the model to receive students’ ‘signals’, which they can then use to provide targeted feedback or additional resources to those at risk of low performance. Course Signals incorporates the use of intervention emails, which can be written by the teacher and sent to those in each risk group. Notifications can also be given in a student’s LMS course page.

Course Signals enables educators to give real-time feedback as early as the second week of class, and it can be used at multiple points during the term. In research published at LAK12, it was suggested that there was a 21% retention rate improvement at Purdue between students who took at least one course that used Course Signals, compared with those who did not. However, this has since been disputed.

Classification

Inventory type:

example at scale

Keywords:

predictive analytics, predictive modelling

Context of Practice

Learning:

post-compulsory

Geographical:

national: United States of America

Pedagogic:

Purdue Course Signals is not explicitly aligned with a pedagogic framework.

Practical Matters

Tools used:

Data used by Course Signals include student grades, demographic information, academic history, and use of the learning management system.

Design and implementation:

This system was produced at Purdue University in the USA. It uses student data from Blackboard, although no explicit connection between developers of Course Signals and Blackboard is described.

As of 2012, over 2,300 students in more than 100 courses had used the system. At that time, it was suggested a further 20,000 students would gain access within the next 18 months. However, more current data has not been made available.

At present, courses at Purdue are not required to use Course Signals, thus it has not been mobilised yet on an institution-wide scale. Lecturers may choose to adopt Course Signals within their own courses, but the project website suggests it is most effective for classes with over 50 students.

Maturity and Evidence of Utility

Course Signal’s effectiveness was highlighted in a paper presented at LAK12, claiming a 21% improvement in the retention rate of students who took at least one course that used the programme. However, criticisms have been made about the methods underlying these claims. As no follow-up studies have yet been published, it will be necessary to address these issues to demonstrate maturity and utility of the system.

Further Information

Informal accounts: http://bit.ly/22S5InZ, http://bit.ly/1Saf8aK

Comparison of Course Signals and Blackboard Retention Center: http://bit.ly/1Rf95l0

Criticisms of claims http://bit.ly/22S2K2Q, http://bit.ly/1OGwsSd

Academic study:

Arnold, Kimberley E, & Pistilli, Matthew. (2012). Course Signals at Purdue: using learning analytics to increase student success. Paper presented at LAK12, Vancouver, Canada.

See also LAEP Inventory record: Progress and Course Engagement (RioPACE) – Rio Salado College

34. E²Coach

Synopsis

High enrolment introductory courses in science, technology, engineering and mathematics (STEM) at the University of Michigan (UoM) applied learning analytics to provide personalised messages to students. In predicting student performance they found grade point average (GPA) in other courses to be the strongest predictor of success in a course. The university also asked students about their goals for the course and reason for taking the course as additional information to help tailor communications.

In order to generate content, the project team interviewed faculty members about the advice they would give to students who had a variety of backgrounds, goals and circumstances. The team also surveyed students who had completed the course in order to gather information about a spectrum of learners and advice they had received about the courses. They interviewed students who performed better than expected and worse than expected in order to create student testimonials related to the courses. Using all of this information from students and faculty, they created a content bank designed to provide personalised advice for students with a variety of backgrounds, goals and circumstances.

Users of E²Coach out-performed non-users. Occasional users outperformed non-users by 0.15 letter grades, while frequent users out-performed non-users by 0.32 letter grades.

At UoM the Third Century Initiative is investing 1.4 million US dollars to expand programmes including E²Coach at the university.

Classification

Inventory type:

example at scale

Keywords:

adaptive

Context of Practice

Learning:

post-compulsory

Geographical:

national: United States of America

Pedagogic:

personalisation

Practical Matters

Tools used:

MTS – Michigan Tailoring System, Student Information System

Design and implementation:

A coaching team and a student information system feed information to the MTS in order to provide personalised and tailored advice to students on introductory STEM courses. The coaching team includes previous students, behaviour change experts and instructors.

Maturity and Evidence of Utility

The MTS System is a mature open source platform

Further Information

Educause report on E²Coach: http://bit.ly/1VGzsPo

Details of project grant from Next Generation Learning Challenges: http://bit.ly/1QXbrV0

Campus Technology blog post on topic: http://bit.ly/1KnulD0

University of Michigan Third Century Initiative: http://bit.ly/1NOCN9c

Michigan Tailoring System: http://bit.ly/20DXb64

35. Georgia State University

Synopsis

At Georgia State University (GSU), predictive analytics have been used to tackle the achievement gap for low income and first-generation students. The university found that students were dropped from courses due to non-payment even when they had high grade point averages (GPAs) and were close to graduation. GSU graduation rate went from 32% in 2003 to 54% in 2014. In the process, the university claimed it removed the achievement gap between students from minority backgrounds or lower socio-economic status, and their peers who had higher graduation rates. GSU states that it achieved these results by systematically accumulating smaller victories. The university took a series of measures to assist students with costs that were preventing them from staying enrolled in the university. The university used as tutors existing students who were obliged to work for the university as part of their financial aid package. The university also helped students select courses based on predictions of likelihood that they would pass the course.

Classification

Inventory type:

example at scale

Keywords:

predictive analytics

Context of Practice

Learning:

post-compulsory

Geographical:

national: United States of America

Pedagogic:

This institutional practice relies on information about course grades from historic students, students who are on work studies, and information about course fee payments.

Practical Matters

Tools used:

GSU’s Office of Institutional Research compiled data from multiple systems and created a comprehensive data warehouse.

Design and implementation:

By creating Panther Retention Grants, 200 students were given hundreds of dollars to remain enrolled in courses. When students were dropped from a course due to failure to pay course fees, the university examined their GPA and proximity to graduation, and funded those who were most likely to graduate. These grants resulted in many of the recipients going on to graduation.

The university also tackled gate-keeper courses, introductory courses that were good indicators of success for a given major. If a student was performing poorly in a gate-keeper course in their major, the university would hire a student who had a work study agreement, and who had previously taken the course, to tutor the struggling student.

The university also created an advice system using a database of 2.5 million grades from the past 10 years to advise current students about the courses they were likely to succeed in based on their current grades. The same system advises students on what their major could be and saw first-year undeclared majors drop by 40% over two years.

Maturity and Evidence of Utility

The implementation has prompted congressional testimony in the USA. Gate-keeper courses have been researched at a variety of grade levels across primary, secondary, and post-compulsory education.

Further Information

Video of congressional hearing on this programme: http://bit.ly/1QE6y10

Report – Building a Pathway to Student Success at Georgia State University: http://bit.ly/20lQn0c

University Innovation Alliance blog post on programme (2015) - http://bit.ly/1m9YGcO

Bill & Melinda Gates Foundation Case Study of the programme: gates.ly/1P9nGey

36. Nottingham Trent University Student Dashboard

Synopsis

Nottingham Trent University (NTU) in the UK has developed, trialled and deployed a Student Dashboard for all undergraduate students.

The system draws engagement data from a range of sources: library use, attendance, use of the online learning environment, ID card swipes in to university buildings, and academic grades. It uses these to generate a composite engagement score and displays this graphically, together with the average for everyone on the course, and gives a rating of high, good, average or low. Automatic alerts are sent to a student’s tutor for triggers such as ‘no engagement for a fortnight’ or ‘academic failure’.

The primary users of a student’s score are the student themselves and their tutors; the scores are also available to other tutors on the course, course administrators, and student support staff, but not to other students.

Classification

Inventory type:

example at scale

Keywords:

predictive analytics, visualisation

Context of Practice

Learning:

higher education

Geographical:

National: United Kingdom

Pedagogic:

The student dashboard does not explicitly embed a particular pedagogical approach, but implicitly relies on measures of engagement being useful indicators of learning.

Practical Matters

Tools used:

The system was developed with DTP SolutionPath’s Predictive Analytics service.

Design and implementation:

A Student Engagement Manager led the dashboard development, drawing on input from other stakeholders, with an initial trial with a smaller group of students before being rolled out more widely.

Maturity and Evidence of Utility

After pilot work in 2013/14 with 400 students, the system was made available to all students in September 2014, and enhanced further in 2015. The system won the Times Higher Education award in 2014 for Outstanding Support for Students.

Further Information

Current guide for students: http://bit.ly/1T2Ahnx

Upgrade announcement to students: http://bit.ly/1T2AoQ3

Video lecture on the use of learning analytics to increase student engagement: https://vimeo.com/114081815

Practices: national level

37. Ceibal

Synopsis

Uruguay has adopted a 1:1 approach to its education system. After delivering laptops (or tablets) to its students and providing software such as an adaptive mathematics tutor, the country examined the impact that upgrading the internet connection had on completion rates of learning activities using the software.

During the five-year project, the country delivered 450,000 XO laptops to students. As well as putting devices into the hands of students and teachers the project implemented an intelligent tutor. The goal was to remove the digital gap between students who had access to technology and those who did not.

Some critics raised the point that this was a large investment to access technology and questioned whether more emphasis should have been placed on the pedagogy of effective use of technology. However, the project did take a systematic approach to deployment, taking into account distribution, Internet access, training, repair and disposal. Access to the Internet is considered to be a human right.

Estimated cost of the project was put at £159 per student with an estimated on-going annual maintenance cost of £13 per student. During the five-year project the cost was under 5% of the national budget for education.

Classification

Inventory type:

example at scale

Keywords:

adaptive

Context of Practice

Learning:

compulsory

Geographical:

national: Uruguay

Pedagogic:

formative evaluation, project based learning, personalization,

Practical Matters

Tools used:

Plan Ceibal Information System, LMS Crea, PAM (adaptive math tutor), ZABBIX (infrastructure monitor), Data warehouse

Design and implementation:

A study was conducted to examine the impact of upgrading the internet connection on completion rates of learning activities on the software. The analysis used a random and stratified sample across two populations: Interior Urban (IU) and Montevideo Metropolitan Area (MMV). Upgrading the Internet connection for schools was associated with an order of magnitude of improvement in the use of the math tutor software in IU schools. The report described the IU as having an initial condition of a less favourable learning environment.

Maturity and Evidence of Utility

The technology-based project has been running across the country for more than a decade and Ceibal is now making moves to integrate learning analytics within the system and to take a lead on the introduction of learning analytics across South America. http://bit.ly/1Ull7Wa

Further Information

Project overview: http://bit.ly/1nCa2XT

Ceibal website (in Spanish): http://www.ceibal.edu.uy/

BBC account from 2009: http://bbc.in/1KTCIRy

World Bank blog post on the project’s next steps (2013) http://bit.ly/1PcRFAc

Paper, ‘How can Plan Ceibal land into the age of Big Data?’: http://bit.ly/1nC1ujU

38. Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement

Synopsis

The Australian government commissioned this in-depth look at the state of learning analytics practices in the country in 2015. Study 1 identified two categories of implementation:

1.) Universities focused on performance measurement and retention interventions

2.) Universities focused more deeply on learning as a pursuit of understanding, who viewed retention as an important proxy for student engagement

This highlighted opposing views about the purpose of using learning analytics to support retention: as a tool for supporting university needs or as a tool for supporting the student academic and social experience. In this study, more universities belonged to Cluster 1 than Cluster 2. The report highlights that institutional learning analytics policies require more than technical readiness, as universities’ views on the benefits of learning analytics are also important drivers.

Study 2 highlighted important factors for success.

The report concludes that most Australian universities are in the early stages of adopting successful learning analytics practices. It stresses that learning analytics form a complex system, which requires the development of six key areas: academic content, conceptualisation of the purpose for learning analytics, leadership, university strategy, stakeholder feedback, technology and an understanding of the specific university context.

Classification

Inventory type:

candidate for mainstreaming

Keywords:

implementation, performance measurement, retention

Context of Practice

Learning:

post-compulsory

Geographical:

national: Australia

Pedagogic:

Some universities considered student retention by analysing student data to determine interventions that support success in retention. In these cases, retention was viewed as a final goal and a marker of success. Other universities viewed retention as one factor that influences success. In these cases, retention was important as a support to the final goal of student learning. The report highlighted that university leaders’ conceptualisations of learning and the role of learning analytics helped shape the use of analytics.

Practical Matters

Tools used:

This report did not examine specific tools used by universities and instead focused on the ways in which tools are adopted.

Design and implementation:

The report gives the following suggestions and considerations for designing and implementing meaningful learning analytics programmes:

1.) Senior institutional leaders’ commitment and strategic plan

2.) Compatibility with existing university systems

3.) A platform that can easily and ethically share data

4.) Transparency of learning analytics operations and data

5.) User-friendly tools to provide feedback

6.) Coordination with educators to design appropriate learning analytics tools

7.) Empowerment of students to develop agency in their learning

Maturity and Evidence of Utility

This report uses robust mixed methods to consider the state of the art of learning analytics in Australia. It also incorporates viewpoints of international experts. The report includes an in-depth description and full explanation of its methods. Although the report focuses on the Australian context, it is of interest to an international audience. This report looks more broadly at trends in the adoption of learning analytics across universities, which provides useful insights and tips for moving the field forward. However, a more in-depth analysis of specific institutional practices will be useful in the future.

Further Information

Student retention and learning analytics report: http://he-analytics.com/

39. Denmark: User Portal Initiative

Synopsis

The Danish Ministry of Education has recently released a national User Portal Initiative, which aims to develop a common learning management system and standardisation framework for exchanging data for all school-aged students in the country. The initiative aims to go live during the 2016-2017 academic year. These initiatives are in collaboration with several technology-enhanced learning vendors, with the common goal of allowing an integration of data nationwide that can be used to develop and inform local or district-wide initiatives.

By 2016, it is expected that all schools in Denmark will adopt technology infrastructure to begin the large-scale adoption of learning analytics. The Ministry is involved in developing and supporting a wide range of resources and programs for schools. Several of these online portals consolidate and summarise resources and evidence of their utility, including EMU, SkoDa, and Materialeplatformen. The creation of common educational objectives, well-being objectives and national testing by the Ministry is also associated with a broader adoption of learning analytics tools and data sharing. Enrolment in secondary education takes place through a digital process called Accession, allowing for easy collection of student demographic data. Finally, a data warehouse that allows for comparisons of student data between institutions, districts or regions is available to the public.

Classification

Inventory type:

pilot

Keywords:

data sharing

standardisation

Context of Practice

Learning:

school

Geographical:

National: Denmark

Pedagogic:

The Ministry of Education will require in 2016 that all schools incorporate IT infrastructure to support these initiatives. Little information has been released about how this may change or disrupt current teaching practices.

Practical Matters

Tools used:

The Ministry plans to collect data from local IT infrastructures at individual schools. National online testing will form a common practice across all schools.

Design and implementation:

The Ministry of Education in Denmark has initiated these practices, in collaboration with local schools through a pilot study conducted by Ramboll Management Consulting. A current challenge is the need to encourage institutions to adopt a ‘data culture’ and to prepare teachers and administrators through the development of digital competencies.

Maturity and Evidence of Utility

As the project is in its initial pilot phases, it is too early to draw conclusions about maturity or evidence of use. However, in its development phase in 2014, an assessment of the potential of learning analytics use in schools was conducted by Ramboll Management Consulting and incorporated into the practice design.

Further Information

Press release (in Danish): http://bit.ly/1OaRrPa

LACE summary: http://bit.ly/1YhwceJ

Data warehouse (in Danish): https://www.uddannelsesstatistik.dk/

40. Norway: various initiatives at the national level

Synopsis

In Norway, a number of software tools deploying features of learning analytics are available through the commercial sector. For instance, Conexus (see description in 7), a Norwegian educational software company that was set up in 2000, provides learning analytics tools for data aggregation and visualisation. Conexus software also provides tools for assessment, adaptive learning and targeting interventions. Another example is itslearning²⁶ (see description in 9), a learning platform with analytics features, which was originally developed at Bergen University College in 1998. In early 2014, Norwegian largest textbook company Gyldendal²⁷ announced a partnership with the adaptive learning software company Knewton to design an adaptive learning textbook program for primary schools, called Multi Smart Øving²⁸, which also incorporates learning analytics tools. An important driver for such vendor and tool development is organised by IKT-Norge²⁹, an interest group for the Norwegian ICT industry.

In order to support and guide the up-take with the issues around learning analytics, the Centre for ICT in Education (Senter for IKT i utdanningen), with a mandate to promote ICT in Norwegian schools, has organised workshops and drafted policy-oriented advice for schools. The Centre’s report on learning analytics (Laeringsanalyse) by Morten Dahl provides an introduction to the subject, written in Norwegian. This gives examples of use within Norway and in a global context. The report identifies potential problems with learning analytics. These include lack of teacher training in the skills necessary to use analytics effectively; threats to privacy and information security; the complex learning analytics market in which there are currently no guidelines, national framework or infrastructure, and a lack of understanding of which data are relevant for promoting quality in learning. The report also deals with the privacy challenges associated with learning analytics and asks how far schools can proceed with recording, compiling and analysing data about students without coming into conflict with their right to privacy. In Norway, schools may only make use of personal data for learning analytics if they can identify a legally valid reason for that use. If personal data are used, schools will be responsible for assuring the quality of those data, for ensuring that they are used to support learning, and for ensuring that students, teachers and parents or guardians are able to access, correct and delete their data on demand.

In 2015, the Ministry of Education and Research committed 25 million Norwegian kroner (approximately 2.7 million euros) to the establishment of a research centre on learning analytics. To determine the location of this new centre, the Ministry invited bid submissions. After a review process, the University of Bergen was selected as the host institution, and the centre was named the Centre for The Science of Learning and Technology (SLATE). The Ministry will contribute five million Norwegian Kroner (approximately 540,000 Euro) per year to the centre, and the University of Bergen will contribute additional research funding. Although the current contract for the centre is for five years, there is a possibility that it will be extended for an additional five. SLATE will have a broad scope, encompassing life-long learning and applying a multitude of research viewpoints and approaches. Learning Analytics are one element in SLATE’s activities.

In 2016, several developments are on-going, especially focusing on enabling the underlying infrastructure:

• Actions related to technical infrastructure and interoperability are being carried out in Norway. UNINETT, who develops and operates the Norwegian national research and education network, is rolling out a service platform, Dataporten (Norwegian for "data gate")³⁰, that connects data sources and end-user applications. This will eventually allow for better sharing of data also for the purpose of learning analytics.

• Within Standards Norway³¹, the national standards body of Norway, discussions have cantered around three projects: Datasharing, vocabularies for activity descriptions, and Privacy and best practice guidelines, all potential underlying enablers for applications such as learning analytics.

Classification

Inventory type:

analysis of policy-related issues

Keywords:

Infrastructure, data protection, ethics, privacy

Policy Context

Learning:

kindergarten, primary, secondary, teacher training

Geographical:

National: Norway

Relationships:

Maturity and Evidence of Utility

Further Information

ICT practice blog (in Norwegian): http://bit.ly/1PStxsi

Location of policy document (in Norwegian): https://iktsenteret.no/ressurser/laeringsanalyse

Centre website in English: http://bit.ly/1VHc2cH

41. Effective learning analytics pilots – JISC

Synopsis

Jisc, a UK not-for-profit organisation with a mission to develop the exploitation of digital technologies for education and research in universities and colleges, is in the early stages of a national initiative to accelerate those institutions towards effective use of learning analytics through: the development of advice and guidance, the establishment of a technical platform with free and charged services and integration with institutional systems, and the support of a series of pilots using the platform.

Envisaged use includes:

• access by students to measures of their own levels of participation and indicators of disengagement or falling-behind, as an aid to self-regulation

• use by staff to trigger interventions as part of a student support process.

The first set of pilots entered their Discovery Phase in autumn 2015. During this phase, institutions assess their readiness as a baseline for implementation planning.

Classification

Inventory type:

pilot

Keywords:

predictive analytics, self-regulation, student support

Context of Practice

Learning:

post-compulsory

Geographical:

national: UK

Pedagogic:

The emphasis of the technical system and tools is on student support rather than on teaching and learning. In this respect, the implicit approach is mainstream in that the emphasis is on monitoring engagement and performance and using predictive analytics to prompt appropriate staff to the possible need for an intervention. Use by students for self-regulation is in scope but is of secondary importance.

Practical Matters

Tools used:

Tribal Student Insight – learning analytics processor and staff dashboard

Unicon Learning Analytics processor and open dashboard

Learning Locker (HT2) – storage of activity records using xAPI

Student Success Plan – for managing alerts and interventions

A bespoke student app and a student consent service are also being developed.

Design and implementation:

A workshop in September 2014 formed part of the co-design process. This workshop identified and prioritised three actions: the development of a solution with a particular funding model, a code of practice covering ethical, privacy, and legal matters; and support for a peer network.

During a two-year pilot phase, Jisc anticipates that between 20 and 40 institutions will complete the Discovery Phase.

Maturity and Evidence of Utility

Findings from the first tranche of pilots are not yet available. The overall approach taken by Jisc is illustrative of a co-design approach involving stakeholders from across UK universities and colleges. This employs technical architecture which offers choice to institutions, and a multi-stranded approach to accelerating institutions towards adoption of learning analytics

Further Information

Informal account of Jisc Effective Learning Analytics challenge: http://analytics.jiscinvolve.org/

Discovery Phase of the pilots: http://bit.ly/JiscDiscovery

Overview of tools available: http://bit.ly/JiscTools

Background on the technical system architecture, ‘Developing an open architecture for learning analytics’ paper http://bit.ly/JiscEUNIS-Paper, slides http://bit.ly/JiscEUNIS-Slides (EUNIS Congress 2015)

See also LAEP Inventory records:

• Code of practice for learning analytics

• Student success plan

42. Code of practice for learning analytics – Jisc

Synopsis

This code of practice was developed to help universities and colleges to develop effective approaches to a variety of issues relating to the practice of learning analytics. It is a concise document that would be suitable for development of local strategies and policies. Rather than providing a prescriptive code of practice, the approach taken is to clarify a set of principles that can be operationalised according to the policies and practices already in place in universities and colleges.

The topics covered are, as described by the authors:

1. Responsibility – allocating responsibility for the data and processes of learning analytics within an institution

2. Transparency and Consent – being open about all aspects of the use of learning analytics, and ensuring students provide meaningful consent

3. Privacy – ensuring individual rights are protected and compliance with data protection legislation

4. Validity – making sure that algorithms, metrics and processes are valid

5. Access – giving students access to their data and analytics

6. Enabling positive interventions – handling interventions based on analytics appropriately

7. Minimising adverse impacts – avoiding the various pitfalls that can arise

8. Stewardship of data – handling data appropriately

The Code was developed for use in the United Kingdom, and refers to some national law, but most aspects are generally applicable, drawing particularly on thinking from North America, Europe, and Australia. It is published under a Creative Commons Licence.

Classification

Inventory type:

good practice advice

Document source:

Jisc, a UK Charity (non-profit)

Keywords:

responsibility, transparency, consent, privacy, validity, ethics

Policy Context

Learning:

post-compulsory

Geographical:

national: UK

Relationships:

The Code of Practice is not formally linked to other policy initiatives but forms part of a systematic programme of initiatives being undertaken by Jisc to assist universities and colleges in the UK in the implementation of learning analytics.

The Code relates to existing policies on privacy and recent work by The Open University to develop its policy on the Ethical use of Student Data for Learning Analytics.

Maturity and Evidence of Utility

Representatives, with diverse roles, from the UK higher and further education sectors were consulted, and they identified the need for a code of practice as a prerequisite for effective implementations of learning analytics.

Following a series of open publications and expert workshops, including a literature review of recent work on ethics and legal matters and a workshop meeting, a draft code of practice was developed and made openly available for comment. A steering group with members drawn from the National Union of Students and UK universities and colleges had oversight of the development process.

Further Information

Location of policy document: http://bit.ly/JiscCoP

Ethics and legal issues literature review: http://bit.ly/JiscLitReview

Taxonomy of ethical, legal and logistical issues (draft issues and workshop report): http://bit.ly/JiscTaxonomy

See also LAEP Inventory records:

Ethical use of student data policy – The Open University

Effective learning analytics pilots – Jisc

43. PAR Framework

Synopsis

The Predictive Analytics Reporting (PAR) Framework is a non-profit provider of analytics-as-a-service to a range of types of higher education institution in the USA (two- and four-year courses of study, public and private, traditional and non-traditional institutions).

It undertakes benchmarking, prediction and work to understand the signs of risk versus progress to completion. In addition to prediction, an aim of PAR is to support the identification of good practice in student retention through data analysis, shared models and benchmarking across institutions.

The PAR Framework motivations are two-fold: a) that there is a cost saving in having a central analytics service with highly skilled staff, covering multiple aspects of expertise from data science to policy and higher education practice; b) cross-institutional benchmark studies provide valuable information on effective strategies to promote achievement, engagement and progress, which a single-institution analytics activity would be unable to reveal.

Classification

Inventory type:

candidate for mainstreaming

Keywords:

predictive analytics, analytics as a service

Context of Practice

Learning:

post-compulsory

Geographical:

national: USA

Pedagogic:

The PAR Framework is not explicit in supporting any pedagogic practice.

Practical Matters

Tools used:

Exchange of data and results between the member institutions and PAR uses files produced according to a set of Common Data Definitions, which are published under a Creative Commons licence.

Design and implementation:

PAR is a membership organisation in which each member institution contributes its data to a central database and receives the results of student-level analysis on its own data. PAR maintains a team including data scientists and researchers. Benchmark data are available to all member institutions. Governance is member-led.

Each member institution is required to follow its normal institutional approval process for human subject research (ethics committee or institutional review board) and the PAR team all have certification in human subject research.

Maturity and Evidence of Utility

PAR is essentially already mainstream in that numerous higher education institutions in the United States are member institutions, but it is classified here as a candidate for mainstreaming as the model has yet to be replicated in other geographical regions.

PAR is now an independent non-profit organisation but it has evolved over a number of years, having been a service managed by the WICHE Cooperative for Educational Technologies (WCET) non-profit organisation until late 2014, with funding from the Bill and Melinda Gates Foundation, at which point 16 institutions were part of the collaborative venture. Previously, PAR had been a smaller-scale pilot project. By autumn 2015, 33 campuses were participating in the collaboration.

A 2012 academic paper deals with the PAR Framework proof of concept study and its initial findings. files.eric.ed.gov/fulltext/EJ982674.pdf

Further Information

Overview of the PAR Framework: http://www.parframework.org/about-par/overview/

Practices and policies on the ethical use of LA

44. Ethical use of student data policy – The Open University

Synopsis

The Open University (UK) policy documents relating to the ethical use of student data include both a formal policy and guidance documents. The aim of the policy documents is to set out how the University intends that student data should be used to inform the delivery of student support in ways which conform to the University’s charter principle to ‘treat each other with dignity and respect’.

The policy, which covers use of data for both student-level interventions and institutional-level strategies and processes, but not use for academic research, is based on eight principles, which are:

1. Learning analytics is an ethical practice that should align with core organisational principles, such as open entry to undergraduate level study.

2. The OU has a responsibility to all stakeholders to use and extract meaning from student data for the benefit of students where feasible.

3. Students should not be wholly defined by their visible data or our interpretation of that data.

4. The purpose and the boundaries regarding the use of learning analytics should be well defined and visible.

5. The University is transparent regarding data collection, and will provide students with the opportunity to update their own data and consent agreements at regular intervals.

6. Students should be engaged as active agents in the implementation of learning analytics (e.g. informed consent, personalised learning paths, interventions).

7. Modelling and interventions based on analysis of data should be sound and free from bias.

8. Adoption of learning analytics within the OU requires broad acceptance of the values and benefits (organisational culture) and the development of appropriate skills across the organisation.

Guidance documents expand upon the policy, to summarise the principles for staff, and to provide answers to the ‘frequently asked questions’ of students about how data about them is used in practice.

Classification

Inventory type:

adoption implementation advice

analysis of policy-related issues

formal policies

good practice advice

strategy-level white paper

Document source:

The Open University: UK higher education establishment

Keywords:

ethics, data protection, privacy, student support

Policy Context

Learning:

post-compulsory

Geographical:

national: UK

Relationships:

The policy is explicitly linked to The Open University Student Charter and to policy and legal requirements for data protection.

Maturity and Evidence of Utility

The policy is a relatively new creation, having been adopted in September 2014. It is, however, underpinned by: a series of peer-reviewed scholarly works combining original formulations of the problem space and review of existing related practice in higher education; and consultation with key institutional stakeholders. It is not yet fully integrated into daily practice such as registration.

Further Information

Location of policy document: http://bit.ly/OU-LAPolicy (comprises the formal policy and informative guidance)

Slade, Sharon and Prinsloo, Paul (2014). Student perspectives on the use of their data: between intrusion, surveillance and care. In: Challenges for Research into Open & Distance Learning: Doing Things Better – Doing Better Things, pp. 291–300 http://oro.open.ac.uk/41229/

45. Learning analytics: a guide for students’ unions – NUS

Synopsis

The UK’s National Union of Students (NUS) has compiled a brief good practice guide for student unions within the UK. The start of the guide defines learning analytics as ‘using the increasing potential of data insight to improve students’ learning.’

The guide goes on to highlight the types of data that universities may use, such as virtual learning environment behaviours, use of books or assessment marks. Next, potential benefits of adopting learning analytics are discussed, such as avoiding drop-outs and reducing demotivation.

The remainder of the document focuses on risks associated with learning analytics, and considerations for student unions in schools that use student data. The risks highlighted include: privacy, data sharing with third parties, consent, and formative versus summative data.

The document also links to JISC’s Code of Practice and contact information for help from NUS.

Classification

Inventory type:

good practice advice

adoption/implementation advice

Document source:

Educational establishment: NUS

Keywords:

ethics

student unions

Policy Context

Learning:

post-compulsory

Geographical:

National: UK

Relationships:

This document was created by the NUS specifically to inform student unions of their rights and areas of concern. It also explicitly links with JISC’s Code of Practice, which was created in consultation with NUS.

Maturity and Evidence of Utility

This good practice document is relatively short, but is an excellent summary for those new to learning analytics. It is also one of the few policy documents available that are explicitly written from a student perspective and address students as agents in the process of adopting learning analytics. Its connection with the more formal JISC Code of Practice document also lends to its maturity and evidence of utility.

Further Information

Location of policy document: http://bit.ly/1X2j13W

JISC Code of Practice: http://bit.ly/1T8zbrC

See also LAEP Inventory records:

• JISC Code of Practice

46. Evaluation of policy frameworks for addressing ethical considerations in learning analytics

Synopsis

This LAK13 conference paper analyses policy frameworks from two large distance education universities, The Open University in the UK and the University of South Africa.

The paper reports that although a great deal of data was collected from and about students by the institutions, learning analytics were not explicitly addressed within the policies of either institution at the time of writing. Both institutions’ policy frameworks were focused on national and international legislative issues around intellectual property, data privacy and data protection.

The review highlights the irregularity of learning analytics where the institution is the only stakeholder with decision-making power, determining the scope, definition and use of educational data without the input of other stakeholders – specifically students.

It is clear from the existing policy frameworks of both institutions that the definition and scope, harvesting and analysis of data are imbalanced and non-transparent affairs.

This research indicates that some higher education institutions’ policy frameworks may no longer be sufficient to address the ethical issues in realising the potential of learning analytics.

Classification

Inventory type:

analysis of policy-related issues

Document source:

LAK13 Proceedings of the Third International Conference on Learning Analytics and Knowledge, educational establishment

Keywords:

data protection, ethics, privacy

Policy Context

Learning:

post-compulsory

Geographical:

International: UK, South Africa

Relationships:

This analysis relates to policy documents of The Open University in the UK and the University of South Africa, dated in or before 2013.

Maturity and Evidence of Utility

The analysis discusses issues that are pertinent for any university that is using or wishes to use learning analytics, but that has not considered the potential policy implications.

It considers issues arising from two different educational contexts so findings should be applicable to institutions operating in either of the contexts and potentially beyond these.

Further Information

Location of policy document: http://oro.open.ac.uk/36934/

Slade, Sharon and Prinsloo, Paul (2013). Learning analytics: ethical issues and dilemmas. American Behavioral Scientist, 57(10) pp. 1509–1528 http://oro.open.ac.uk/36594/

See also LAEP Inventory records:

Ethical use of student data policy – The Open University

Practices: interest groups and networks

47. Further education learning technology action group: FELTAG

Synopsis

FELTAG, the Further Education Learning Technology Action Group, includes members from across the further education system, including learning providers, accreditation and funding bodies, and industry. The group has as its goal ‘Create the conditions for the agile evolution of the FE system, support employers and drive economic growth’ and believes that ‘Government cannot and should not provide all the answers. Ownership by the FE sector’. The group emphasises putting people ahead of the technology and investing in teachers and administrators. One suggestion offered by the group is to build an innovation network to enable staff to drive digital innovations.

Classification

Inventory type:

example at scale

Keywords:

further education

Context of Practice

Learning:

post-compulsory

Geographical:

national: UK

Pedagogic:

There is a focus on empowering learners and engaging them actively.

Practical Matters

Design and implementation:

FELTAG recommendations:

•   Learners should be empowered as digital leaders, increase their influence in providers’ learning strategy, become more aware of assistive technology, and be prepared to demonstrate online skills.

•   Employers should participate in further education curricula development and scale up best practices, offer leading-edge apprenticeships, encourage providers to use collaborative MOOCs, and consider how small and medium enterprises can build the digital capability of staff.

•   Skill providers should assess organisations’ use of technology, accredit learning technology, have regional support centres play a role in further education, and teach the teachers how to design their own learning materials.

In terms of investment, regulation, and funding, the group outlines efforts that need to be made in order to keep up with the pace of technology. It identifies that infrastructure concerns such as broadband need to be taken into account, and suggested that publicly funded programmes in 2015/16 should have mandated a 10% wholly-online component, increasing to 50% in the following year. Funding should encourage ‘learning presence’ not ‘physical attendance’.

Maturity and Evidence of Utility

This work involved a variety of key stakeholders and examined the challenge from a comprehensive perspective illustrating the roles of different types of organisations in producing improved learning.

Further Information

FELTAG coalition: http://feltag.org.uk/feltag-coalition/

FELTAG report: http://bit.ly/1SVsbgt

48. Learning Analytics Community Exchange (LACE)

Synopsis

The Learning Analytics Community Exchange is a European-funded project in the 7th Framework Programme, which involves nine partners from across Europe. LACE partners are passionate about the opportunities afforded by current and future views of learning analytics (LA) and educational data mining (EDM) but are also concerned about missed opportunities and failing to realise value. The 30-month project aims to integrate communities working on LA and EDM from schools, workplace and universities by sharing effective solutions to real problems.

The LACE project brings together existing key European players in the field of learning analytics and EDM who are committed to building communities of practice and sharing emerging best practice in order to make progress towards four objectives.

Objective 1 – Promote knowledge creation and exchange

Objective 2 – Increase the evidence base

Objective 3 – Contribute to the definition of future directions

Objective 4 – Build consensus on interoperability and data sharing

Classification

Inventory type:

network

Keywords:

research network, research collaboration

Context of Practice

Learning:

school, post-compulsory, workplace

Geographical:

international

Pedagogic:

The focus of LACE is on analytics to deal with questions of interest to an educator, trainer or reflective learner. These include questions directed towards improving effectiveness or efficiency with regard to teaching and learning, developing assessment with greater relevance and other forms of pedagogically driven decision making.

Practical Matters

Tools used:

The LACE project has developed tools for use by the learning analytics community, including a framework of quality indicators for learning analytics, the DELICATE checklist for a trusted implementation of learning analytics, and the LACE Evidence Hub, which provides access to research evidence.

Design and implementation:

LACE has engaged with learners, educators, organisations and policymakers across Europe. It has organised many events, including a successful series of workshops on ethics and privacy in learning analytics (EP4LA)

Maturity and Evidence of Utility

LACE was a 30-month project, which ran from January 2014 until June 2016. Its tools and resources remain available online.

Further Information

LACE project website: http://www.laceproject.eu/

DELICATE checklist: http://bit.ly/1XT2iNO

LACE YouTube channel, containing video interviews with international experts: https://www.youtube.com/user/LaceprojectEu

LACE Review papers: http://www.laceproject.eu/learning-analytics-review/

49. Society for Learning Analytics Research (SoLAR)

Synopsis

The Society for Learning Analytics Research (SoLAR) is an inter-disciplinary network of leading international researchers who are exploring the role and impact of analytics on teaching, learning, training and development. SoLAR has been active in organising the International Conference on Learning Analytics & Knowledge (LAK) and the Learning Analytics Summer Institute (LASI), launching multiple initiatives to support collaborative and open research around learning analytics, promoting the publication and dissemination of learning analytics research, and advising and consulting with state, provincial and national governments.

SoLAR priorities to advance the field of learning analytics globally are:

• Foster the highest standards of academic research into learning analytics

• Promote the development of open educational resources in learning analytics

• Raise awareness of learning analytics amongst policy and decision-makers in educational institutions and governments

• Create opportunities for the diverse stakeholders in learning analytics to communicate, collaborate and debate. These stakeholders include academic researchers, product developers, educators, students, institutional administrators and government policy analysts.

Classification

Inventory type:

example at scale

Keywords:

network, organisation, society

Context of Practice

Learning:

All levels

Geographical:

international

Pedagogic:

various

Practical Matters

Tools used:

SoLAR makes use of a range of technologies to support international communication. These include Google Groups, the use of EasyChair to manage conference submissions, and Zoom for executive meetings.

Design and implementation:

The Info Hub on the SoLAR website brings together learning analytics resources and reports from the field. SoLAR also provides a dataset of research literature, which can be used to test computational methods of analysis

Maturity and Evidence of Utility

SoLAR was founded in 2011. In 2016, its annual conference attracted 460 participants.

Further Information

SoLAR website: https://solaresearch.org

SoLAR Info Hub: https://solaresearch.org/core/

LAK Dataset: https://solaresearch.org/initiatives/dataset/

50. Spanish Network of Learning Analytics (SNOLA)

Synopsis

SNOLA (Spanish Network of Learning Analytics) is a collaborative community that is building practice for learning analytics researchers in Spain. The primary aim of SNOLA is to share resources and findings among members through online depositories, webinars and events. The project also encourages collaboration between members on learning-analytics-related projects. One prominent example is ATHENA-I (translation from Spanish: Application of analysis techniques and adaptation of the educational process in the Cloud for the provision of Interoperable Learning Spaces), which analyses the effects of new technologies, such as learning management systems and MOOCs, within schools. Other collaborations include a learning analytics extension for Khan Academy and edX, and MakeWorld, a digital program for science, technology, engineering and mathematics. SNOLA currently has around 75 members based in a variety of universities and businesses across Spain.

Classification

Inventory type:

example at scale

Keywords:

research network

research collaboration

Context of Practice

Learning:

All levels

Geographical:

National: Spain

Pedagogic:

SNOLA members come from a wide variety of research backgrounds, so it is not possible to highlight one specific pedagogic framework that applies to the entire network

Practical Matters

Tools used:

Different members make use of different tools.

Design and implementation:

The Spanish Ministry of Economy and Competitiveness (MINECO) called for ‘networks of excellence’ of scientific research and SNOLA was formed in response to this call. In 2015, SNOLA was designated as an accepted ‘network of excellence’ by the government. At present, any Spanish researcher can join SNOLA by completing an online form.

Maturity and Evidence of Utility

Although a relatively new collaborative research group, SNOLA has already provided an important boost to learning analytics research in Spain. In 2015, a Learning Analytics Summer Institute (LASI) was hosted in at the University of Deusto, in collaboration with SNOLA members, and another LASI is planned for 2016. In 2016, a well-received webinar was held by SNOLA members, entitled ‘Applying Quantitative Techniques for Analysis of Educational Data.’ Several collaborative projects between partners are already underway.

Further Information

Group website: http://snola.deusto.es/

Zotero group: https://www.zotero.org/groups/snola

LASI Bilbao 2016: http://lasi16.snola.es/

Evidence-base

51. LACE Evidence Hub on Learning Analytics

Synopsis

The LACE Evidence Hub is designed as a tool to help people to make evidence–based decisions about learning analytics, whether they are teachers, managers, researchers or policymakers.

The Evidence Hub gathers research evidence from around the world on learning analytics. The results are is organised around four key propositions.

• Learning analytics improve learning outcomes: including cognitive gains, improved assessment marks, better scores on tests and attainment results. In June 2016, there were 28 pieces of research evidence out of which 26 support the positive/neutral proposition.

• Learning analytics improve learning support and teaching, including retention, completion and progression, but are not direct learning gains by the learner. In June 2016, there were 15 pieces of research evidence out of which 12 support the positive/neutral proposition.

• Learning analytics are taken up and used widely, including deployment at scale. In June 2016, there were 16 pieces of research evidence out of which 14 support the positive/neutral proposition

• Learning analytics are used in an ethical way. There are 7 pieces of research evidence out of which 4 support the negative proposition

The Hub provides summaries of, and links to, the research evidence related to learning analytics. This evidence can be searched and interrogated in various ways, including by country and by sector (schools, higher education, workplace and informal learning)

Classification

Inventory type:

research tool

Role of analytics:

summary and description

visualisation

Data sources:

published research data, project data

Keywords:

evidence, learning analytics, research

Tool in Context

Learning:

school, higher education, vocational education, informal learning

Supply model:

openly available tool

Origin:

Learning Analytics Community Exchange (LACE) project

Ethics and privacy:

One of the Evidence Hub propositions relates to the ethical use of learning analytics

Languages:

English

Maturity and Evidence of Utility

The Evidence Hub has been developed since 2014 and is now integrated with the submission and acceptance system for the Learning Analytics and Knowledge conferences (LAK).

Further Information

Evidence Hub: http://evidence.laceproject.eu

Evidence of the Month: Highlighted additions to the Evidence Hub: http://www.laceproject.eu/blog/category/evidence/

Policy documents

52. Analytical review – UK Department for Education

Synopsis

The Analytical Review, which was published in April 2013, was concerned with the role of research, analysis, and the use of data within the UK Government Department for Education (DfE) and its schools and children’s services (in the UK, education and social services for children are related through legislation). The review has two parts: ‘Building Evidence into Education’ by Ben Goldacre and ‘Data Systems’ by Roger Plant.

In addition to covering matters of education research and the potential for more evidence-based policy and practice, and modernisation of the statistical work undertaken in the DfE, the report considered matters that relate more directly to the conduct of teaching and learning. It can therefore be considered as dealing with learning analytics, although neither of the two parts of the review explored classroom practice. A key conclusion which over-arches much of the report on data and analysis is that the system should move away from periodic centralised data collection, which is often seen as being a burden at school level and provides low reward at that level, to more real-time data exchanges with greater utility at school level.

The report asserts that more fluid and timely data exchanges would: ‘Support teaching and learning directly. The system will be able to cater for broadening data demands particularly in relation to performance and pedagogical data held in systems such as learning platforms.’

The report recommended that the DfE should, among other things:

• Lead culture change: setting an expectation that evidence is an integral part of education policy and delivery and that research skills are the key to professional improvement and freedom.

• Make sharing real-time data easier, more efficient and more attractive.

• Encourage a flourishing secondary market to improve data access and analysis by parents, schools and others.

It also identified the importance of interoperable IT systems in delivering real-time data exchanges.

Classification

Inventory type:

analysis of policy-related issues

Document source:

national government: UK (commissioned report)

Keywords:

real-time data, interoperability, research skills (teachers)

Policy Context

Learning:

school

Geographical:

national: UK

Relationships:

The Analytical Review was primarily concerned not with learning analytics but with performance management of education and evidence-based policy and practice. As such the Review envisions a future state in which more ambitious learning analytics would become feasible.

Maturity and Evidence of Utility

This is a major external policy review that stimulated a large capital project, the School Performance Data Programme, although this was subsequently cancelled due to delivery problems, and replaced with some small-scale pilots.

Further Information

Location of policy document: http://bit.ly/DfE-AR

Report on cancellation of the Schools Performance Data Programme: http://bit.ly/DfE-SDP-CW

53. Capacity enablers and barriers for learning analytics – Alliance for Excellent Education

Synopsis

This report, subtitled ‘Implications for Policy and Practice’, was published in 2014 by the Alliance for Excellent Education, a US-based policy and advocacy organisation dedicated to ensuring that all students, particularly those traditionally under-served, graduate from high school ready for success in college, work and citizenship.

It explores trends and policy enablers and barriers to adoption of effective learning analytics at Federal, State, and School District level. It goes on to describe opportunities and make recommendations aimed at policy-makers and education leaders. These recommendations are, in outline:

• Develop a clear understanding of the potential and rationale for learning analytics.

• Build capacity for the implementation of learning analytics, including development of a culture of informed decision-making, infrastructure, and human capital.

• Identify and develop policies to support and enable learning analytics, including aspects of privacy, technology procurement and teacher development.

• Develop funding models to support learning analytics.

• Conduct research to support the capacity building and policies critical for learning analytics, to study adoption and emergence of effective practice.

Classification

Inventory type:

analysis of policy-related issues

Document source:

Alliance for Excellent Education: non-governmental policy and advocacy organisation supported by several philanthropic foundations

Keywords:

policy, practice

Policy Context

Learning:

school

Geographical:

national: USA

Relationships:

The Alliance for Excellent Education is concerned with a broad range of policy issues that it believes underpin the achievement of its mission.

The report is explicit in identifying existing policies at Federal and State level that are relevant to adoption of learning analytics and calls for these to be reviewed and implemented in ways that enable rather than inhibit the adoption of effective learning analytics.

Maturity and Evidence of Utility

The report is a credible assessment of the US school policy space, having been produced by a combination of Alliance senior staff and associated consultants, with a track record in education and educational policy innovation and reform, who drew evidence from 13 interviews of public policy experts and school district senior staff.

Further Information

Location of policy document: http://bit.ly/ALL4ED

Executive summary: http://bit.ly/ALL4ED-X

Summary of policy-related issues: http://all4ed.org/issues/

54. Education governance: the role of data – Organisation for Economic Co-operation and Development

Synopsis

This document is a summary of a conference hosted by the Organisation for Economic Co-operation and Development (OECD) in Tallinn, Estonia from 12-13 February 2015. The conference theme was ‘Education Governance: The Role of Data’. The document outlines conference themes and provides a summary of keynote speakers, workshops and panels.

The conference included three keynote speakers. The first, Marc Tucker from the National Centre on Education and the Economy in the USA, discussed factors necessary for good education governance.

The second keynote, by Kim Schildkamp from University of Twente in the Netherlands, highlighted the kinds of data that exist in education, challenges of using such data and potential solutions.

The final keynote, by Birgit Lao-Peetersoo (Foundation Innove) and Aune Valk (Estonian Ministry of Education and Research), looked at data in education in an Estonian context.

A panel is also described, which discussed the tension between data that are available versus data that should ultimately be used in education. Additionally, four workshops are described, which covered developing data systems, data and trust, learning analytics, and the Estonian data system.

The learning analytics workshop highlighted scepticism on behalf of workshop participants that learning analytics would be able to deliver measurable changes in education. Also described was a general fear of exploitation of student data.

Classification

Inventory type:

good practice advice

Document source:

International Conference for the OECD/CERI Governing Complex Education Systems project (GCES)

Keywords:

governance

Policy Context

Learning:

all levels

Geographical:

international

Relationships:

This paper is a conference summary and is not formally linked to explicit policies. The conference was part of a larger OECD project, Governing Complex Education Systems (GCES).

Maturity and Evidence of Utility

This document describes the keynote talks and workshop events during an OECD conference. Although it provides an interesting insight into speaker and participant views of learning analytics and the use of student data, the piece does not offer much empirical evidence or concrete advice for developing or implementing learning analytics systems. However, the video recordings of conference talks may be of use. These are available in full on YouTube.

Further Information

Location of policy document: http://bit.ly/1mKTpJI

Full conference recordings available at: http://bit.ly/1IVVE71

55. Enhancing teaching and learning through learning analytics and educational data mining – US Department of Education

Synopsis

This policy brief, written by Marie Bienkowski, Mingyu Feng, and Barbara Means, was published in an issue brief by the US Department of Education. The goal of this brief is to educate both policymakers and administrators about how analytics and data mining have been applied as well as how they could be applied for educational improvement.

The report defines both learning analytics and educational data mining. There is a diagram of the components of an adaptive learning system. Adaptive learning is described from the student perspective (using Khan Academy) and the teacher perspective (using ASSISTments).

Adoption and Implementation advice

1. Advice for educators and administrators

a. be intelligent consumers of data

b. generate demand for products that have useful features.

2. Institutional guidance

a. the cost to adopt analytics initiatives can exceed the technical capacity of the institution.

Policy-related advice

1. Advice for educators and administrators

a. Align technical requirements of local government policies with online learning.

b. Consider privacy, policy, and legal issues when storing and analysing personally identifiable information from students.

Classification

Inventory type:

adoption/implementation advice

analysis of policy-related issues

Document source:

SRI funded by US Department of Education

Keywords:

impact, ethics

Policy Context

Learning:

school and post-compulsory

Geographical:

United States

Relationships:

The policy brief is linked to privacy, ethics and institutional capacity.

Maturity and Evidence of Utility

This is a document about learning analytics written in 2012 by a well-respected research organisation, SRI international. It provides a good starting point for concepts such as adaptive systems as well as some advice about how a variety of stakeholders can influence the development of the field.

Further Information

Location of policy document: http://1.usa.gov/1SVU6gl

56. Improving the quality and productivity of the higher education sector – Australian Government Office for Learning and Teaching

Synopsis

The Australian Government Office for Learning and Teaching (OLT) commissioned this report, subtitled ‘Policy and Strategy for Systems-Level Deployment of Learning Analytics’. It was produced in late 2013 by three leading figures in the Society for Learning Analytics Research (SoLAR).

It considered ten case studies from universities in Australia, the USA, and the UK and explored the strategic issues pertinent to effective use of learning analytics. The work was also informed by webinar contributions from several individuals offering their experiences and analysis of the problem space of systemic adoption of learning analytics.

The aim of the report was principally to guide the Australian Government in the ways in which it should intervene to enable its higher education establishments to exploit learning analytics to achieve increased levels of educational success, and to build a competitive advantage for Australia.

The key enabling factors identified in the report for a national agenda are:

1. Australian higher education leaders coordinate a high-level learning analytics task force.

2. Leverage existing national data and analytics strategies and frameworks.

3. Establish guidelines for privacy and ethics.

4. Promote a coordinated leadership program to build institutional leadership capacity.

5. Develop an open and shared analytics curriculum.

The report is published under a Creative Commons licence.

Classification

Inventory type:

strategy-level white papers

analysis of policy-related issues

Document source:

SoLAR, funded by national government

Keywords:

SoLAR

Policy Context

Learning:

post-compulsory

Geographical:

national: Australia

Relationships:

The report refers to national regulatory and statistical data collection and to a perception that school-level data collection, analysis and sharing had progressed beyond that in Australian higher education.

Maturity and Evidence of Utility

The authors of the report have considerable collective experience of learning analytics research and emerging practice, on the basis of case studies and expert contribution from practitioners.

The enabling factors identified in the report do not appear to have been met by specific initiatives but subsequent projects funded by OLT are working on several of the implied tasks, for example the development of a roadmap, maturity model, or similar to guide the uptake of learning analytics tools and practices.

Further Information

Location of policy document: http://bit.ly/OLT-2013

Rogers, T., Colvin, C., West, D., Dawson, S., & Dawson, S. (2015). Learning Analytics in Oz: What’s happening now, what’s planned, and where could it (and should it) go? In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge – LAK15 (pp. 432–433).

57. Learning analytics at the workplace manifesto – LACE

Synopsis

The Learning Analytics at the Workplace (LAW) manifesto is a document created for the Learning Analytics Community Exchange (LACE). Its creation followed a 2015 ‘workplace learning’ session that formed part of a learning analytics workshop in Brussels, supported by the European Parliament.

The manifesto first highlights the current state of the art of European manufacturing, as well as potential industry changes in the future. In particular, 3D printing, Internet of Things, digital disruptions and Industry 4.0 are discussed. Next, the document highlights the 21st-century skills needed to address and embrace these changes, and proposes adoption of learning analytics to support increased workplace learning of these skills.

The stakeholders for adopting learning analytics for workplace learning are described in detail in three primary areas: industry, education and society. Suggestions are offered at multiple levels, including advice for industry leaders, employers, workers, universities, teachers, social partners, teacher unions and trade unions. Finally, the future of learning analytics for workplace learning is addressed.

Classification

Inventory type:

good practice advice

analysis of policy-related issues

Document source:

non-governmental public body: LACE

Keywords:

Workplace learning

Policy Context

Learning:

VET

Geographical:

International: Europe

Relationships:

This manifesto is not explicitly linked to any formal policies, but rather gives policy suggestions for those wishing to adopt learning analytics measures in the workplace. Members of the LAW work group include representatives from SkillAware and the EU-funded WatchMe project.

Maturity and Evidence of Utility

Experts from across Europe with diverse roles within both the higher education and industry sectors compiled this policy document. The sources the document draws upon are also diverse and include empirical studies. However, other than the LAW working group, there are no stated collaborations with either practitioners or researchers, which may be a consideration for future policy statements. This document is also written from a European perspective, although those from other countries may find it useful.

Further Information

Location of policy document: http://bit.ly/1PAExbF

LACE Learning Analytics at the Workplace group: http://bit.ly/1StC4SZ

58. Opening up education: innovative teaching and learning for all through new technologies and open educational resources – European Union

Synopsis

This Communication from the European Union set out a European agenda for stimulating high-quality, innovative ways of learning and teaching through new technologies and digital content. ‘Opening up education' proposes actions towards more open learning environments to deliver education of higher quality and efficacy, thus contributing to the Europe 2020 goals of boosting EU competitiveness and growth through better skilled workforce and more employment.

The Communication specifically mentioned learning analytics, noting that: ‘Technology makes it possible to develop new solutions for better personalised learning, by allowing teachers to have a more accurate and up-to-date follow up of each learner. Through learning analytics, new and more learner-centred teaching methods can emerge since the evolution of learners who use ICT regularly can be closely monitored: teachers may know the exact learning outcomes of each individual and identify needs for additional support.’ The communication also noted that, through Erasmus+ and Horizon2020, the commission would promote research and innovation on learning analytics.

Classification

Inventory type:

formal policies

Document source:

Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

Keywords:

innovation, new technology, digital literacy, open university, teaching quality, digital technology, teacher, education, vocational training, youth

Policy Context

Learning:

general education

Geographical:

International

Relationships:

This document was explicitly linked to Europe’s 2020 goals. Related documents are linked to at http://bit.ly/1UappQf

Maturity and Evidence of Utility

This document is dated 25 September 2013. Many of the Erasmus+ and H2020-funded learning analytics projects that have been launched since that date owe their existence, at least in part, to this communication.

Further Information

Location of policy document: http://bit.ly/22r9V0n

OpenEdu: a study of strategies for Opening Up Education: http://bit.ly/1UapKCg

59. Policy brief on learning analytics – UNESCO

Synopsis

This policy brief, written by Simon Buckingham Shum, was published in 2012 by the UNESCO Institute for Information Technologies in Education.

The aim of the report is to describe and define learning analytics and to provide real-world examples of their use. In doing so, it divides learning analytics into three levels – micro (individual student), meso (institution), and macro (region/state/national/international) – and highlights potential benefits of learning analytics for each. Examples of learning analytics forms are also given, including LMS/VLE dashboards, predictive analytics, adaptive learning analytics, social network analytics and discourse analytics.

The report also highlights debates in the learning analytics field. Topics include the perceived ‘neutrality’ of data, conceptualising the definition of student ‘success’, and various ethical implications of using and sharing student data.

Finally, the policy brief provides recommendations for higher education institutions in the light of the state of the art at the time of publication:

1.) Using analytics as a tool to debate visions of teaching and education in the 21st century

2.) Training staff and researchers to use and develop analytics tools

3.) Developing an analytics infrastructure for research at an institutional level

4.) Collaborating with other institutions to develop trusted partnerships and robust learning analytics methods (for example, through an open analytics platform)

This policy is published under a Creative Commons licence.

Classification

Inventory type:

strategy-level white papers

Document source:

UNESCO Institute for Information Technologies in Education: non-governmental (195 member countries, 10 associate member territories)

Keywords:

impact, ethics

Policy Context

Learning:

post-compulsory

Geographical:

international: global

Relationships:

The policy brief is not formally linked to other policy initiatives or policies. Rather, it is a general document aimed at describing and defining the state of learning analytics at the time of publication.

Maturity and Evidence of Utility

The report is a credible assessment of learning analytics, written by a leading researcher in the field and with an extensive list of source materials. However, no information is available about the editing or peer review process.

Further Information

Location of policy document: http://bit.ly/1NmyqDh

60. What matters most for education management information systems framework paper – EMIS

Synopsis

The World Bank Education Management Information Systems (EMIS) ‘What Matters Most for Education Management Information Systems Framework Paper’ was published in 2014 as part of its Systems Approach for Better Education Results (SABER) working paper series. SABER is an initiative to produce comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems.

The working paper focuses on data and their analysis as a tool for management, school-system oversight, and policy. The term ‘learning analytics’ is never used. Nevertheless, the vision described in the paper includes feedback of student learning and other outcomes back to school level for action by teachers, students, so enablement of learning analytics practices is implicit.

The paper states that ‘an effective EMIS is one that has a fully functioning information cycle. This cycle demonstrates that an EMIS is more than a simple annual school census, that the coverage of statistics goes beyond administrative census data. An EMIS is a dynamic system that has a defined architecture, the capacity to perform analytics, and the ability to serve its users. The functioning of this cyclical process results in more effective data sharing and coordination.’

The paper notes that the complexity of education data means that an institutionalised system is needed that can look at an entire education system in a comprehensive, structured and systematic matter. It asserts that ‘a system to collect, maintain, and disseminate timely and relevant information about the education system is critical.’

The paper concludes with a rubric for assessing progress toward the key policy goals identified, which gives indicative statements against four levels of maturity in relation to a large number of indicators organised under policy headings including: legal framework, human resources, infrastructural capacity, data-driven culture, methodological soundness, openness, timeliness and data coverage.

Classification

Inventory type:

strategy-level white papers

analysis of policy-related issues

Document source:

World Bank: non-governmental (188 member countries)

Keywords:

Policy, goal, self-assessment, management information systems

Policy Context

Learning:

school

Geographical:

international: global

Relationships:

The framework paper is rooted in the World Bank’s mission to work for a world free of poverty and its aim to improve learning as a vehicle to this end, by helping countries improve data collection, data and system management, and data use in decision making.

Maturity and Evidence of Utility

The report was written with the benefit of a review of global evidence, with input from multiple sources, and peer-reviewed by World Bank staff.

Further Information

Location of policy document: http://bit.ly/SABER-EMIS

61. Understanding and managing the risks of analytics in higher education: a guide – Educause

This document, written in 2012 by Randy Stiles for Educause, provides practical information about the risks associated with adopting (or not adopting) learning analytics in higher education institutions. In its introduction, the document states that it ‘provides frameworks, suggestions, and resources that may prove helpful in considering risk and performing analytics at both ends of a possible spectrum – not doing enough or doing too much, too soon’.

The document highlights the risks for institutional leaders that are associated with adoption of analytics, including their premature or inappropriate use and imposing an inappropriate data-oriented culture on the institution. On the other side of the argument, the risks of ignoring analytics altogether are explored.

Data governance is considered next. The document highlights several areas of concern, including legal data protection requirements, data collection and storage methods, and access to student data.

The section that follows looks at data quality, and issues of missing, incorrect or misleading data.

Finally, smaller sections consider issues of legal or institutional compliance (from a primarily American perspective), ethics and privacy, and using third party systems.

Classification

Inventory type:

good practice advice

adoption/implementation advice

Document source:

Educause: Non-profit based in the United States

Keywords:

risks, ethics, compliance

Policy Context

Learning:

post-compulsory

Geographical:

National: United States

Relationships:

The policy brief is not formally linked to other policy initiatives or policies. Rather, it is a general document aimed at describing and defining the risks associated with adopting (or not adopting) learning analytics at higher education institutions.

Maturity and Evidence of Utility

This policy document’s author has considerable practical experience in higher education, which is apparent throughout. However, no stated collaborations, either with other practitioners or with researchers, contributed to the writing of this document, which diminishes its maturity and utility.

The sources that this document draws upon are primarily other policy documents, think pieces, or opinion pieces, with relatively little empirical evidence examined.

This document is written from a US perspective, although readers from other countries may still find it useful. The document provides good, general suggestions for implementing sound analytics policies (which are often taken from other sources), but readers should look elsewhere for more specific advice on implementation.

Further Information

Location of policy document: http://bit.ly/1Z8NRtE

Annex 2: Case Studies of Implementation of Learning Analytics

The Case Studies that are presented in this section will contribute to the understanding of existing practices in the use and implementation of learning analytics for educational purposes. These Case Studies were selected from the wider Inventory of tools, policies and practices by using the selection criteria described below. Each of these Case Studies focuses on the role and impact of learning analytics in relation to the development of more effective learning processes and organisations.

The approach taken to positioning the Case Studies was to start with an important aspect of learning analytics adoption and to use an example of this in practice to explore the area, rather than starting with an example and focusing on its associated issues. Therefore, although a particular case could potentially be used to explore a wide variety of associated issues, each Case Study focuses particularly on one of these so that, overall, the studies provide detailed coverage of areas of interest.

These Case Studies support critical reflection on the impact, potential and limits of learning analytics. They also provide indicators of emerging issues related to implementation of learning analytics that could help to shape future policy, and identify obstacles and enablers that can guide and support the take-up, adaptation and further development of this technology to enhance education in Europe. This reflection, in turn, will provide input for work to support the take-up and adaptation of learning analytics at a European level.

This structure of the Case Studies is presented below, including the leading questions for each of the sub-sections:

Introduction

• Could you describe the subject of this Case Study in a few sentences?

Context of the case studied

• What motivated the activity considered in this Case Study? Were there any explicit strategic drivers?; Could you describe for me the educational and organisational setting? How does this relate to practice in the sector as a whole? How does it relate to practice at a local, national or international level?; Who are the key stakeholders, and have they changed over time?

Design and implementation process

• How did your aims translate to particular objectives?; How was the implementation conceived, which stakeholders were involved, and how did they contribute?; What needed to change? (Cover IT, resources, processes and practices, organisation and policy.); How was change managed? What were the steps taken to move from ideas to reality?; How did you plan for sustainability?

Experience

• What steps have you taken to monitor, evaluate or reflect?; What changed and how was this evidenced? Did you identify impact and benefits?; What issues, limitations and obstacles did you encounter?; What will be your next steps and prospects?

Policies

• In what was, if any, has educational policy supported or limited your work?; Based on your experience of learning analytics, how would you like to see educational policy change in the future?

Kennisnet

Developing school sector awareness, knowledge and skills around learning analytics in the Netherlands

Kennisnet: Introduction

Kennisnet³² is a public organisation in the Netherlands that is fully funded by the Dutch government. Kennisnet started as a network infrastructure project in the 1990s, providing Internet access and national ICT infrastructure for schools, notably fast Internet access. As that project matured, Kennisnet’s role transitioned to that of an expertise organisation, although it still maintains some infrastructure. Its aim is to share and develop knowledge, expertise and best practices around the use of ICT in education. It also advises sector councils in the areas of primary, secondary and vocational education. Kennisnet’s annual ‘Education Days’ (Dé Onderwijsdagen)³³, organised in partnership with SURF³⁴ (public collaborative organisation for ICT in higher education and research in the Netherlands), are a key annual event for the Dutch ICT-in-education sector.

Kennisnet has built up its activity in the area of learning analytics after identifying it as an area through horizon scanning in 2011. In 2014 the organisation set up a project to overcome the obstacles Dutch schools face in ICT in education. Learning analytics were identified as an issue for a small number of pioneering schools, and as a likely issue for the sector as a whole in the future.

Kennisnet also commissions research and provides information and articles about learning analytics, dashboards and personal learning. Additionally, it is playing a key role in developing standards in this area for The Netherlands, through EduStandaard,³⁵ the Dutch educational standards body. This has included a recent standard on exchange of assessment data.³⁶

Kennisnet: Context

In the Netherlands, the government sets goals for schools and provides direct funding, but schools are free to decide for themselves how to achieve these goals, choosing their own principles of teaching and organising their teaching themselves. In the Netherlands, this freedom is seen as a key feature of the educational system.³⁷ Some schools operate entirely individually, and some work together as groups. Thus, Dutch schools have a degree of autonomy and low-level budget holding that is unusual in Europe, although it is similar to that of Academy schools in the English system. As a result, most schools rely on ICT vendors to support their curriculum and technology choices, and make heavy use of materials from educational publishers. There has been a big focus on personalised learning.

The impetus for activities around learning analytics came from several sources. Originally, in 2011, Kennisnet’s innovation department explored the potential for learning analytics as part of its horizon-scanning activity. Some time later, around 2014, the topic emerged as an issue for schools with which Kennisnet works. The main impetus came from Kennisnet’s Doorbraakproject³⁸ or ‘Breakthrough’ project, which was set up in 2014 to overcome obstacles Dutch schools have with the use of ICT. In the initial phase of this work, Kennisnet approached schools to ask them what issues they had in using ICT. Issues around learning analytics made up one of the themes that emerged, as a consequence of which Kennisnet started to scale up its work in this area.

Through the ‘Breakthrough’ project, Kennisnet identified ‘forerunner’ schools that were ahead of others in their use of information and communication technology (ICT). Some of these schools encountered difficulties in using learning analytics technologies. The most common motivation for schools was to have a dashboard, a visual display of progress information that could give insight into each student’s activity with regard to different skills and subjects.

This is not simple to achieve, particularly if a school wants to use multiple vendors. If a school is using a single vendor’s products, and the vendor provides a dashboard, there is usually no technical problem in making this work. However, if a school uses products from multiple sources, there is a significant problem with a lack of interoperability: different systems do not work together easily. Standardisation, so that these products will work together, would help greatly.

At the moment, the work of Kennisnet is mainly focused on the forerunner schools that are working in this area, but the aim is for the work to develop in order to benefit the sector as a whole. The stakeholders in this work include Government, Kennisnet, schools, vendors, educational publishers and standards bodies.

Kennisnet: Design and implementation process

Most of Kennisnet’s work in the area of learning analytics has been conceived in close partnership with the forerunner schools who propose issues they would like to solve in relation to ICT use. These proposals had to come from the head of a school or from the governing body for a group of schools. The proposals were reviewed to determine which were individual issues and which were issues for a wider group of schools or the entire sector, with most effort to be aimed at those issues with widest applicability.

Since 2014, Kennisnet has worked with PO-Raad,³⁹ the primary education sector organisation for the Netherlands, to provide a Versnellingsvragen, or Acceleration Questions service.⁴⁰ School boards submit to the website the problems they encounter in the development or implementation of ICT in education. They can see questions submitted by other schools and endorse them if they have that issue as well. PO-Raad and Kennisnet help answer these questions, updating the website with information and progress reports, and also use these questions to drive their work.

Schools can request help from Kennisnet using email, a phone hotline, a ‘virtual critical friend’ who can review an ICT plan.⁴¹ However, it does not provide in-school help. It is able to provide information, and to explain how other schools are using technology, but ultimately schools are responsible for implementation. Kennisnet always requires senior approval from each school, but also works with individual teachers and ICT staff.

Kennisnet has sought to help schools articulate what they want from ICT vendors, so it can act as a broker to the vendors, mediating requirements and exploring possible solutions. Kennisnet can suggest what might be useful for vendors to produce but the drive has to come from the schools, as they are the purchasers. Kennisnet groups requests together to increase their influence with vendors, with the intention that the organisation will be better able to deliver effective solutions related not only to in ICT issues in general, but also to learning analytics in particular.

The main change sought is to improve the technical solutions available from vendors. However, it is not just vendors who need to change. Schools need to understand and articulate their needs better, and to ground this work in educational considerations. They also need to develop teachers’ skills so they are willing and able to work with the new technology. There is also a role for expectation management in the case of enthusiastic schools that want new solutions immediately.

Kennisnet also showcases good practice around the use of ICT in education using a variety of methods, this work includes presentations, workshops, responding to one-to-one queries, as well as ongoing research and standardisation efforts. Face-to-face events include the annual ‘Education Days’ event for schools⁴², a research conference and regional meetings. The organisation also provides a wide range of publications, including horizon-scanning reports, magazines and brochures.⁴³

Where appropriate, Kennisnet also commissions research that involves formal evaluation of ICT use in education. It recently commissioned the University of Twente and Radboud University to carry out two studies of Snappet⁴⁴, an adaptive educational platform with some learning analytics’ features used by many primary schools. These studies explored whether children learned better, how teachers used the platform, and whether they were able to implement interventions based on the data effectively.⁴⁵ A preliminary paper has been published recently on its effect on students' arithmetic skills⁴⁶. The results indicate that students in the Snappet condition make significantly more progress on arithmetic skills in grade 4. However, much of the work in this project is still at least a year away from being in a state where final evaluation would be appropriate.

Kennisnet has sought to help schools articulate what they want from ICT vendors, mediating requirements and exploring possible solutions.

In the longer term, the project on learning analytics will end, although it is likely to continue for several years. Kennisnet expects that its learning analytics work will end up in the areas of technology infrastructure or standardisation, which will need to be evaluated thoroughly and then developed further. In the case of standardisation, Kennisnet works together with SURF to staff EduStandaard,⁴⁷ which is responsible for the management and implementation support of standards and reference architectures for education and research in the Netherlands. This work is progressed through workshops and a formal Standardisation Council and Architecture Council.

SURF has also been interested in learning analytics since they rose to prominence in 2011, and has run a series of projects in the area.⁴⁸ The main distinction between the two organisations is that Kennisnet covers schools, while SURF covers higher education. SURF is running an Innovation Programme from 2015 to 2018, working with Dutch higher education institutions to get them working with learning analytics. It is currently working on learning analytics readiness – developing instruments and infrastructure, and holding workshops that involve information technology and education departments and that, at national level, bring different sectors together to solve problems. A report on pedagogical models is currently being developed, as well as a report on privacy. In spring 2016, SURF published a whitepaper on How data can improve the quality of higher education⁴⁹.

Kennisnet has worked closely with European projects in the area of learning analytics. In particular, the standards work done in the LACE project⁵⁰, funded by the European Commission through the Seventh Framework Programme, has been particularly useful for Kennisnet’s work with schools. Work with European projects has enabled Kennisnet to contribute to wider discussions about what is happening in education in other countries, and to understand a wider range of approaches.

Work with European projects has enabled Kennisnet to contribute to wider discussions about what is happening in education in other countries, and to understand a wider range of approaches.

Kennisnet: Experience

In order to ensure that Kennisnet provides what the schools want, on-going work is carried out in close collaboration with schools, involving detailed dialogue with them. This is on the agenda for every school: almost all include personalised learning in their vision statement. Wietse van Bruggen, a project manager at Kennisnet with a longstanding interest in learning analytics, sees that vendors are delivering more products that are more useful in realising this vision. However, he believes profound educational change – using analytics on a deep level, not simply enhancing current practice – has not yet taken place. He does see a positive change in vendors’ involvement. Initially, they were apprehensive. However, now the issues are clearer, they can see more easily where they might fit into the picture, and are exploring their position in this new world of digital education.

Standardisation and interoperability are seen as key issues by vendors, schools and Kennisnet, particularly when it comes to exchanging information between systems that involve more than simple test results. Kennisnet has worked through EduStandaard to develop Uitwisseling Leerlinggegevens en Resultaten (UWLR), or Student Data and Results Exchange, a Dutch standard for exchanging test information.⁵¹ van Bruggen comments that it proved very hard to reach agreement on these data, and that the work ahead, to extend this to formative assessment and other data, will be tough.

Some of this work can draw on existing international standards, such as Experience API⁵², which specifies how data about learning experiences can be exchanged between learning management systems, learning record stores, and IMS Caliper⁵³, which IMS claims is ‘the world’s first interoperability standard for educational click stream data’.⁵⁴ However, van Bruggen firmly believes that whatever standard is used, there needs to be a conversation between the stakeholders about how to interpret it. He sees a role for Kennisnet in the facilitation of these discussions, through EduStandaard and its direct work with schools.

A question that remains to be answered in this area concerns the potential challenge to vendors’ business models that is implied by the need to ensure sufficient diversity in the market. It is most straightforward for a school to deploy a single integrated system from one vendor. This means there is a risk that in future there will be a very small number of suppliers with a very high market share. van Bruggen suggests that one way to create diversity would be to separate the learning record store and visualisation packages, with them working together in an interoperable way. However, he can see that this approach would present issues for vendors’ business models.

The next step for Kennisnet will be to make sure that standards work around the exchange of test information is implemented effectively, by running evaluations and trials and by continuing engagement with schools. After that, the organisation will move on to a roadmap for standardisation that will enable the exchange of more and broader learning information, working first with forerunner schools to get a clearer picture of what kinds of information they want to exchange, and how that can be standardised so it can be transferred between systems. van Bruggen believes that experimentation with schools will help to clarify what is needed, and will help Kennisnet and the vendors develop what schools want.

Kennisnet: Policies

Educational policy requires Dutch schools to provide every child with a certain number of hours of education in the classroom each year. To a certain extent, this limits the possibilities for fundamental changes to how schools work, in particular in relation to approaches that could achieve learning outcomes faster (each child must receive the set number of hours of education) or by different means (each child must spend those hours in the classroom, not elsewhere).

Sometimes schools are hesitant to change, to introduce innovations, because they are concerned that this may lead to negative evaluations.

All schools in the Netherlands are evaluated on their performance by the Inspectie van het Onderwijs,⁵⁵ which is a government-funded organisation. van Bruggen argues that there may be a barrier associated with perceptions of educational policy in general. Sometimes schools are hesitant to change, to introduce innovations, because they are concerned that this may lead to negative evaluations. But in reality, van Brugge explains, the evaluation organisation is very open to schools that want to try different things, and is keen to make sure that its performance and evaluation framework can work with the school rather than against them.

There is extensive discussion in the Netherlands at the moment about what should be in the curriculum, and what should be changed, with a vision up to 2032.⁵⁶ However, this is mainly concerned with what children should be taught, not how they should be taught. Personalised learning is up to individual schools to take up as they see fit rather than being something required by the government. In such cases, there is no requirement to move away from traditional teaching approaches; with the freedom in the Dutch system, the incentives have to come from the schools themselves.

van Bruggen does not see a significant need for policy change in the Netherlands. The current system provides a lot of freedom for schools to innovate and do new things. In his experience, he sees some schools are unhappy with the pace of vendors implementing new solutions and therefore suggest that the Government should step in to enable implementation to take place more quickly. However, he believes it is currently unrealistic for this to happen, because it is not yet clear what should be done, so developing legislation and policy is not yet possible. The current structure sets out the goals clearly, and leaves it up to the schools to decide how to achieve them in a very flexible way. This independence is part of the Dutch education system.

Goals for schools are very broadly defined, so van Bruggen is concerned that schools can be overly reliant on publishers to supply a structure for what they have to teach. He worries that, instead of working towards their own educational goals in line with the Dutch freedom of schools, they rely on what is in existing textbooks. Some forerunner schools do set their own goals and timelines for what children should learn and by when, but most do not.

Portion of Learning Analytics Infographic CC:BY Kennisnet, 2014.
For the full picture, see http://www.laceproject.eu/blog/infographic-learning-analytics/

The Open University, UK

The process of developing an institutional ethics policy

OU: Introduction

This Case Study focuses on the process of developing an institutional policy for ethical use of student data. The Open University (OU) in the UK has collected and analysed student data for many years, and has used these data in a variety of ways, including as a way of targeting efforts towards student support and retention. As learning analytics emerged as a field, the university began to take a strategic interest in it. The need for a policy for ethical use of student data grew out of a growing awareness within the university of the range and volume of data collected, and how these data could be used to provide effective and timely guidance to students. The policy⁵⁷ was made available in July 2014.

OU: Context

In early 2013, the OU set up a strategic project to explore learning analytics. This project included a number of practical and technical sub-projects that focused on the development of learning analytics solutions for the benefit of OU staff and students. These included progress reporting and data visualisation. The development of an institutional policy for ethical use of student data was the focus of a specific sub-project that ran alongside the other learning analytics sub-projects. There were no external drivers such as national legislation that prompted the development of the policy.

The initial team was a group of five people chaired by an academic from the university’s business school who had a developing interest in ethical issues related to learning analytics. The team included an expert in data protection issues, the head of the OU’s Information Office, an academic expert on ethics and research from the University’s Institute of Educational Technology (IET), and a project manager from its Learning and Teaching Centre.

When the ethical policy sub-project started, the team began by examining what was going on outside the OU. They found that, at the time, no other universities had policies that dealt with the ethics of learning analytics. Many had data protection policies, but none was exploring issues relevant to analytics, such as issues around classification and ownership of data, and consent issues.

The development of the policy was felt to be particularly important in the OU for two reasons. First, the university operates an open entry policy, so the backgrounds and experiences of students vary greatly. Second, it is a distance teaching university, so face-to-face meetings between staff and students occur infrequently in comparison with conventional universities, if at all. These two factors mean that the OU has to rely on information that can be gathered to make decisions for the benefit of students. In some cases, the information gathered through analytics is the main or only source of knowledge about aspects of student study.

When work on the policy started, the sub-project team members were aware of no other institutions working on similar ethical policy issues. A published review of institutional policies within the UK and South Africa, co-authored by a team member, had found no reference to the ethics of learning analytics⁵⁸.

OU students were and are the main stakeholders with respect to the policy and its development. Stakeholders from within the university’s structure included faculty staff, student-facing support staff, the university’s IT unit, the university’s academic policy and governance unit, and the university’s Pro-Vice Chancellors. The other stakeholders were the project team, which expanded when a senior academic joined following his appointment to the OU in 2014.

OU: Design and implementation process

The objective of the team was to produce a policy, however the university identified no precise requirements. The team determined that the policy should be transparent – it should be clear about what the university does with student data, without causing distress or creating misunderstanding amongst students.

The process of generating the policy began with a review of related policies and drew on existing research carried out by the team’s chair. This led to the development of a set of general principles. These were tested, reviewed and refined over time through a series of consultations with stakeholder groups⁵⁹, leading to the final versions of the principles presented in section 4 of the ‘OU ethical use of student data for learning analytics policy’ document⁶⁰:

• Principle 1: Learning analytics is an ethical practice that should align with core organisational principles, such as open entry to undergraduate level study.

• Principle 2: The OU has a responsibility to all stakeholders to use and extract meaning from student data for the benefit of students where feasible.

• Principle 3: Students should not be wholly defined by their visible data or our interpretation of that data.

• Principle 4: The purpose and the boundaries regarding the use of learning analytics should be well defined and visible.

• Principle 5: The University is transparent regarding data collection, and will provide students with the opportunity to update their own data and consent agreements at regular intervals.

• Principle 6: Students should be engaged as active agents in the implementation of learning analytics (e.g. informed consent, personalised learning paths, interventions).

• Principle 7: Modelling and interventions based on analysis of data should be sound and free from bias.

• Principle 8: Adoption of learning analytics within the OU requires broad acceptance of the values and benefits (organisational culture) and the development of appropriate skills across the organisation.

From the student side, the main contributions to the review and refinement of the policy were from two dedicated online student consultation forums. These forums involved an established group of volunteer students that is representative of OU students as a whole. This group of around 90 students had been recruited to participate in consultations with the University on a range of issues.

General principles that would underpin the ethics policy were tested, reviewed and refined over time through a series of consultations with stakeholder groups.

The first forum discussed the initial draft of the principles that form part of the policy. These principles were posted to the forum, along with a number of questions intended to explore participants’ understanding of the principles. Issues that were discussed contributed to the drafting of initial versions of the policy.

Once an initial version of the policy had been drafted, two representatives of the Open University Students Association (OUSA) participated in discussions with the project team to refine the policy further. These discussions focused on the issue of consent, considering if or when students would be asked to consent to their data being used for learning analytics. OUSA’s aim was to ensure that the student voice was heard, and that the university recognised any concerns that students have in relation to data collection and analytics.

The team carried out many other consultations with various university committees, and had to seek and gain approval from these committees. As the work on the policy progressed, the team become aware of stakeholders who might have conflicting views, such as the unit responsible for registering students with the university.

One of the team’s recommendations was to pursue informed consent, so every student would have to give consent before their data could be used for learning analytics purposes. However, this was flagged as a potential barrier to registration as it had the potential to deter some students from registering. A compromise resulted, in that the profile of the policy was raised and communicated to students in a variety of ways.

The team worked to create versions of the policy that are meaningful and understandable to students. They also liaised with staff members who work on student-facing websites in order to highlight the policy and to engage more proactively with students, encouraging them to update their own data.

Team members realised early on that simply creating a policy changes very little. The team worked to create versions of the policy that are meaningful and understandable to students. They also liaised with staff members who work on student-facing websites in order to highlight the policy and to engage more proactively with students, encouraging them to update their own data. This relates to one of the policy’s principles, which is concerned with the mutual responsibility of students and University to enable students to make sure that the information stored about them is up to date. Case studies and practical guidance have been developed for student-facing staff to see what the policy means in practice.

The creation of the policy for ethical use of student data required some small changes to other institutional policies. For example, wording had to be changed or added within the OU’s data protection policy, and its terms and conditions of student registration, in order to promote and link to the ethical policy.

For the policy team, the remit was to develop the policy, which went live in 2014. This policy did not include a position on consent, and discussion with stakeholders about this issue continued into 2016. There were two main stakeholder groups with different views on the consent issue: students and university staff. There was no common ground, so the team aligned with the staff perspective, recommending a position of informed consent. This was given formal approval by the University’s Student Experience Committee in February 2016.

Once it had been finalised and approved, the policy was handed to the academic policy and governance unit for maintenance and development (if required). This unit is responsible for providing the University with services for academic and student policy, standards and processes, and for institutional governance and regulatory compliance. One of the project team is from this unit, which has eased the transition of ownership from the team to the unit.

OU: Experience

Members of the team have written papers about the development of the policy, and have engaged in related work outside the OU. For example, the team’s chair consulted on the Jisc project that led to the publication of the Jisc Code of Practice for Learning Analytics⁶¹ in June 2015. Involvement in the development of this code of practice prompted reflection on the stance that the OU’s policy had taken on student consent to use of their data for academic purposes. The Jisc code of practice recommends that students should give consent to use of their data for the purpose of learning analytics. The difference between the two positions may be due to the fact that the OU has to deal with the practicalities of implementing its own policy, whereas the responsibility for implementing the code of practice lies with institutions and not with Jisc.

The strategic project to explore learning analytics is scheduled to finish in June 2016 and the ethical policy sub-project will finish at this time.

There remain ethically interesting and controversial aspects to be investigated. For example, it is possible that educational institutions will not have the resources necessary to provide proactive support for all the students identified as in need of extra support through the use of analytics. Decisions will need to be taken about how to target available resources. At the moment there are no principles or guidance to inform this kind of decision.

It is possible that educational institutions will not have the resources necessary to provide proactive support for all the students identified as in need of extra support through the use of analytics. Decisions will need to be taken about how to target available resources

OU: Policies

Overall, the work of the ethical policy sub-project team was neither supported, nor limited by existing policies, although it took into account legislation such as that relating to data protection regulations about use of sensitive data. The OU had existing policies that made reference to the ways in which it should use student data and about the ways that it should support students, and one of the things the team had to do was to add detail to these.

It would be valuable to see other higher education institutions develop policies on ethical use of student data based on this experience. Overall, there is a need to realise that students are the key stakeholders, but that other issues such as retention and completion rates are likely to affect the nature and implementation of policies in this area.

University of Technology, Sydney

Learning analytics in the context of a data-intensive strategy

UTS: Introduction

In 2011, the Australian University of Technology Sydney (UTS) committed itself to a vision of becoming a data-intensive university. This strategy, led by Deputy Vice-Chancellor and Vice-President Professor Shirley Alexander, now makes use of university data⁶² to support the decision-making process of all university stakeholders⁶³.

As part of this strategy, a dedicated institute, the Connected Intelligence Centre (CIC)⁶⁴, was formed in August 2014. The CIC was created as a response to the growing importance of data in UTS learning and research. It is spearheading the UTS learning analytics initiative and is key to the learning.futures programme⁶⁵ that is currently shaping the future of UTS student learning.

UTS: Context

UTS is an Australian university that was founded in 1988 and by 2015 had 40,636 students enrolled⁶⁶. Since 2008, the University has invested AU$1 billion (about 675 million euros) in campus redesign. Alongside the renewal of its campus learning spaces, UTS has also renewed its learning practices, guided by its learning.futures strategy and by that strategy’s predecessor, Learning2014⁶⁷.

The initiator of the programme to become a data-intensive university was Deputy Vice-Chancellor and Vice-President (Teaching and Learning) Shirley Alexander⁶⁸. The aim was to create a university where staff and students understand data and, regardless of the volume and diversity of these data, can use and reuse them, store and curate them, apply and develop the analytical tools to interpret them. The programme reflected a recognition that there has been an explosion of data in society, and that this trend has implications for the whole University, how it works as an organisation, what and how it teaches to prepare its students for the future, as well as how researchers will operate in the future.

The project began in 2010-13 with a series of internally funded projects in which computer science researchers tested the potential of data-mining techniques in relation to issues of student retention. In 2011, learning analytics emerged as a human-centred field seeking to integrate data science with education. Within UTS, the growing importance of data as a university business, learning and research priority became increasingly clear.

These factors led to the data-intensive university (DIU) strategy and in 2014 the opening of Connected Intelligence Centre (CIC) with the mandate to advance learning analytics within UTS. The institute’s name reflects a UTS staff decision to describe this as a connected intelligence project rather than a data-intensive university project. They made this decision partly on the grounds that the phrase ‘data-intensive university’ might alienate some. At the same time, ‘connected intelligence’ better reflects the strategic aim of the project, which is to understand the consequences of the data revolution on education. CIC defines its purpose broadly⁶⁹:

The CIC operates outside University faculties as a hybrid research lab conducting applied research as well as offering selected courses. Its primary audience consists of UTS students and staff. The centre sits directly within the portfolio of the Deputy Vice-Chancellor (Education and Students). Besides connecting with the faculties, CIC collaborates with the Advanced Analytics Institute⁷⁰ and the business unit, as well as with the business intelligence unit that manages the UTS data warehouse. The CIC also collaborates with other universities, for example on the projects that surveyed the state of the art of learning analytics in Australia⁷¹, which were funded by the Australian Office for Learning and Teaching.

Outside the academic sphere, the CIC has strong relations with industry and government. These relations are not only visible in its research collaborations with corporate partners, but also in the interest of corporate partners in the Master of Data Science and Innovation programme⁷², which is filling a skill gap that is currently opening up around data scientists. External partners – including big consulting companies, governmental departments, start-ups and non-governmental organisations (NGOs) – contribute to teaching as well as enrolling their staff on the programme.

Connected Intelligence Centre is developing learning analytics associated with graduate attributes, the 21st-century qualities that are important for all students

UTS: Design and implementation process

CIC currently focuses on research into next-generation learning analytics tools as well as the provision of courses preparing students for a data-intensive world.

Early learning analytics work at UTS focused on student attrition, the so-called ‘killer subjects’ that deterred students, student study and engagement patterns. Today, CIC is working on social media learning analytics and collaborative teamwork. Currently, its learning analytics focus is on UTS graduate attributes⁷³ 74,, the 21st-century qualities that are important for all staff and students. This aim is directly derived from UTS learning.futures, an innovation strategy that is transforming learning spaces and learning practices at the University in order to have a positive impact on student satisfaction and engagement⁷⁵.

One of the key areas for development is the design of analytics that deliver learning.futures experiences to students. For example, academic writings analytics are developed to support students’ analytical and reflective writing skills. Growth of students’ agency and resilience is fostered with learner profile analytics. In this area, CIC is employing participatory design methods in order to involve all stakeholders in the analytics design.

One major CIC research project focuses on the potential of automated analysis of writing, using various technologies, to give formative feedback to students about their writing drafts⁷⁶. These may be drafts of traditional academic scholarly writing, but could also be drafts of more personal reflective writing, which has great importance for reflective practitioners as well as for how students think about how they are developing as learners.

The CIC involved several stakeholders in the development of this innovation. The development process involved pedagogical experts who specialise in reflective writing and provided the underlying instructional design for teaching reflective writing. The process also involved academics from different faculties, who provided subject-specific expertise. The academic writing tool uses a language platform that is provided with the support of a corporate research lab, with additional language technologies now being added. The challenge here was to align the generic language technology, parts of which are based on an externally hosted corporate partner language platform, with the requirements of the academics. CIC chose a co-design process for the product, which involved the corporate partner and UTS academics as well as students.

Another major CIC project uses the self-assessment survey tool CLARA⁷⁷, a tool which was developed to make students aware of their learning dispositions (the habits of minds they bring to their learning). The survey tool platform generates ‘learning power’ profile visualisations for each student, as well as suggesting interventions that are based on the learning profiles.

A key development was the implementation of a scalable process that provides mentoring and coaching to the hundreds of students who use the tool.

This learning power self-assessment tool is based on educational research and has been in development for 15 years. From a technical point of view it is a simple survey platform.

A key development was the implementation of a scalable process that provides mentoring and coaching to the hundreds of students who use the tool. This was a challenge because mentoring is inherently difficult to scale up. For example, 900 Science first-year students carried out self-analysis using the survey tool. It was not possible to provide 900 undergraduates with a 1:1 coaching conversation. CIC therefore involved the Science Faculty in the design of a coaching programme. This programme trained third- and fourth-year students in the methodology associated with the tool and introduced them to fictional students with fictional learning power profiles, in order to provoke reflection. Those personas were based on descriptions by academics of the types of student who study their courses.

The innovation process as a whole involved a wide range of stakeholders. Academics were involved in defining student personas. Senior students were trained in coaching. The UTS Peer Mentoring programme manages the entire peer-training programme.

On the teaching side, CIC currently offers the Master of Data Science and Innovation⁷⁸, a doctoral programme⁷⁹, and the elective course Argument, Evidence and Intuition⁸⁰. This course forms part of the learning.futures strategy that is raising the level of data literacy within the University. It teaches staff and students basic concepts of statistics and improves their data literacy and ability to argue about and criticise the kinds of data prevalent in everyday life.

UTS: Experience

Although CIC has only existed since August 2014, some observations can be drawn from its collaborative work within the University.

CIC engages UTS staff in discussions, both to receive feedback and to inform staff about the relatively new concept of learning analytics and what they can mean for learning and teaching. Many educators are very excited about the CIC’s work on learning analytics. For example, they see great potential in the CIC’s academic writing analytics tool⁸¹, as it provides rapid feedback at any time of the day or night to students on drafts (not something academics can deliver). Feedback from students is also broadly positive. Some educators were initially concerned about data and data analytics, which they sometimes associated with reductionist forms of education. When they engaged with CIC staff, they were reassured to find that they care deeply about education. This dispersed concerns that analytics implied a certain type of learning and led to a change in perceptions of learning analytics.

General challenges have arisen during the transition from the old system of learning and teaching to the new system, but these challenges are not specific to learning analytics. Learning and teaching are embedded within organisational processes and information systems that were established and created for one type of pedagogy. The learning.futures strategy leads the transition to new pedagogies through corresponding change in organisational processes and information systems.

UTS: Policies

The current policies of the University are seen as great enablers for learning analytics and the work of the CIC. The UTS learning.future⁸² policy views learning analytics as essential. It states the need for fast formative feedback, for more authentic assessment, for more data and data analytics, in order to facilitate the learning of qualities such as the higher order graduate attributes that will prepare students for our data-saturated society.

Although senior executives initiated the programme, it is not a top-down strategy but instead introduces innovation bottom-up by working with academics and early adopters to show the University what successful learning analytics look like⁸³. Such success stories help the buy-in of more and more people at the University to learning analytics.

Beyond UTS, at a national level, policy change will be necessary to shift current views of assessment. As assessment drives teachers’ and students’ behaviour, old assessment strategies can limit the potential for learning analytics and more broadly for learning technologies. Learning analytics can help to shift education to more authentic types of learning that equip students with and assess them on the 21st-century competencies⁸⁴ that will be crucial in their future lives.

Although senior executives initiated the programme, it is not a top-down strategy but instead introduces innovation bottom-up by working with academics and early adopters to show the University what successful learning analytics look like.

The Apereo Foundation Learning Analytics Initiative

Open-source software and architecture as an option

Apereo: Introduction

In 2014, Learning Analytics Initiative (LAI)⁸⁵, which is designed to accelerate the development of learning analytics software, support pilot studies at member organisations and avoid duplication of software and institutional developments, was announced by Apereo. Apereo Foundation⁸⁶ is an umbrella organisation to foster the development and maintenance of open-source software ‘for the academic enterprise’ and the communities that surround it.

LAI’s formation stemmed from discussion within the international Society of Learning Analytics Research (SoLAR) community about an open learning analytics framework. One problem seen in the field by members at the time was that using fragmented systems for learning analytics was not a viable long-term solution. An integrated platform was needed to aggregate all data in one place with tools for data mining. Another key driver was the foundation’s collaborative culture, as Marist College (USA), the University of Amsterdam (NL), the University of Hull (UK) and Unicon (USA) felt their individual work in the field could be combined to build a cohesive platform to support learning analytics activities.

The Learning Analytics Initiative began by identifying five major components of a successful learning analytics platform which are Collection; Storage; Analysis; Communication, and Action.

The narrative for such platform is the following. First, the data must be collected and aggregated from different learning systems into a centralised storage component. Next, an analysis component is needed to make meaning from the data. Results of that analysis should then be pushed to a dashboard component for communication to educators, administrators or students. Finally, components are needed to initiate actions, such as advice and interventions. Under the Apereo umbrella, projects are currently underway to address each of these five areas.

Apereo: Context

The Apereo Foundation exists as an umbrella organisation to foster the development and maintenance of open-source software ‘for the academic enterprise’ and the communities that surround it. It also incorporates an incubation process during which emerging open-source software can be supported in its transition to a sustainable product. The Learning Analytics Initiative, which forms just one part of Apereo’s initiatives, has promoted two relevant products: the Student Success Plan (see the Inventory no: 19) for student support case management; and the Learning Analytics Processor (see the Inventory no: 27), which controls an analytics workflow and focuses on predictive modelling of student data.

Discussions that led to the formation of the Apereo Foundation began in 2010. The initial motivation was a merger between Sakai (an open-source learning management system) and Jasig (a non-profit organisation in the US). The two organisations had worked closely together since 2006 on the development of open source educational solutions. The official formation and naming of Apereo followed in 2012. Today, Apereo functions as an umbrella organisation for a global network of over 180 partnering institutions on six continents⁸⁷, with each contributing to a wide range of education-related projects and communities. There is currently strong representation in the organisation from the United States and Europe. In addition, Apereo projects have connections with more than ten commercial affiliates.

The underlying purpose of Apereo is to foster collaboration between these stakeholders in an open-source manner. This means that development of open-source software (with open licences and open coding source) and community building are both essential elements. Within the foundation, learning analytics make up just one aspect or community among Apereo’s wider initiatives in the education field.

Learning Analytics Initiative is designed to accelerate the development of learning analytics software, support pilot studies and avoid duplication of developments

Although the majority of those working on Apereo projects are employed through member institutions or commercial affiliates, the foundation has some paid members of staff, including an executive director and a community coordinator. The foundation also has a 15-member board of directors, made up of volunteers from partnering entities. Beyond this, the foundation is made up of a series of overlapping and interlocking software, regional and thematic communities: ‘Software Communities’ (stakeholders in the development of particular software programs) and ‘Communities of Interest’ (those formed around common interest areas). The Apereo Foundation, at its core, is a bottom-up initiative, with member institutions setting the standards for the foundation’s visions, rather than being managed top down by its directorial body.

In terms of funding structures, resources often come from the individual projects. However, projects are typically collaborative in nature, with individual institutions contributing and developing elements of a program or platform. Funding also occasionally comes from grants or contracts from outside entities, such as the European Union and Jisc (UK not-for-profit organisation), and Apereo members frequently collaborate to submit funding bids. Additionally, the foundation relies on volunteers and the sharing of skills between community members to contribute to one another’s work.

Perhaps the largest learning analytics projects developed by Apereo are Student Success Plan (SSP) and Learning Analytics Progress (LAP). These programs are two independent projects underneath the Apereo and LAI umbrella, but can also be integrated with each other. Using this model, institutions have the freedom to adopt the full Apereo software stack or to integrate one or the other with existing programs. In keeping with the Apereo mission, both programs are open source and are configurable to the specific needs of individual institutions.

SSP is an endorsed project that has already been adopted in approximately 50 institutions, mostly in the United States. The program is case-management software that includes a suite of tools aimed at promoting student success. These tools support areas such as academic advising, student resources, coaching or counselling, disability accommodations, and data aggregation. Thus, SSP operates in the collection and aggregation domains of the LAI framework, as well as providing tools for action, such enrolling students in coursework.

On the other hand, Learning Analytics Progress focuses on predictive modelling of student performance and completion, and fits within the analysis and communication components of the framework. The program is designed to help consolidate big data at educational institutions for early alerts and data visualisation, with the final aim of providing resources for determining interventions. Learning Analytics Progress is currently in Apereo’s incubation process (see below), and has been piloted at a handful of universities.

Apereo: Design and implementation process

Projects developed within the Apereo community undergo an incubation process⁸⁸. To progress to this stage, the program must be largely developed and a potential candidate for large-scale adoption (e.g. The Learning Analytics Processor, see the Inventory no: 27). To graduate from incubation, the project must meet an established list of exit criteria⁸⁹, upon which it will receive an Apereo endorsement. This endorsement is an indicator of the program’s maturity and sustainability. The incubation process also leads to a common infrastructure around testing and development of programs created under the Apereo umbrella, as well as providing a robust method for global collaboration.

Several other components play a role in Apereo’s operations. First is the use of standards and community building that allows universities to share work and collaborate. Second is the transferability between universities of their predictive models for student success. For example, an empirical study at Marist College in the USA analysed how well Apereo models perform when developed at one institution and then deployed in a different institutional setting⁹⁰. Initial findings were positive. Finally, the Apereo framework makes it possible to build many types of dashboards to fit individual organisational needs. This means that using Apereo’s open-source software is more economical than building a new dashboard from scratch, although resources on site (such as a software engineer) may still be needed.

Using Apereo’s open-source software is more economical than building a new dashboard from scratch.

Much of Apereo’s current work on learning analytics is concerned with scaling up existing programs and preparing for massive implementation. For instance, the foundation has worked closely with Jisc in the UK to develop a national initiative⁹¹ for learning analytics using Apereo software. In the near future, every university in the UK will have access to programs such as Learning Analytics Progress, potentially providing support for millions of students. The UK is the first national initiative for Apereo, but other countries are already considering following suit. Apereo is therefore working to scale systems up to a cloud-based service in anticipation of wide-scale use. In the UK, an early-release pilot version of Learning Analytics Progress is expected during the summer 2016, with a large-scale release planned for some time in the following academic year.

In addition to software development, the foundation has also taken steps to develop and build its learning analytics community. As the community is global, one important aspect is its online presence. To encourage online collaboration, the foundation uses a wide variety of tools, including email lists, wikis, GitHub repositories and messaging on platforms such as Slack.

Additional steps are taken to translate this online community to face-to-face contexts. Apereo hosts an annual international conference, as well as several regional and community-specific conferences. To facilitate discussion and collaboration, these often include networking and icebreaker activities. The foundation also hosts webinars (online seminars), hackathons (collaborative computer coding events), workshops, seminars and showcases on a regular basis, and individual communities maintain contact with each other. The incubation process also helps to bring people together through collaborative work on projects both face-to-face and online.

Apereo: Experience

The impact of Apereo’s learning analytics activities has been strong, especially considering that it is a relatively young foundation. In terms of software, Student Success Plan is a fully functioning baseline system that is highly modifiable to meet the needs and cultures of individual institutions. This has led to its adoption by around 50 institutions, mainly in the USA, and the foundation hopes an additional 350 institutions will adopt it in the future.

The foundation’s work on projects such as Learning Analytics Progress has also led to a national initiative in the UK. Another significant impact has been the increased collaboration between institutions and stakeholders, as Apereo is currently the only open-source global learning analytics initiative. Foundation members frequently co-create, co-author and peer review in ways not possible before Apereo’s creation and that would be unlikely to take place in a commercial setting.

The Apereo model provides evidence that a foundation that is a global community-based initiative built by volunteers⁹² (working as employees of other institutions) has several strengths. For example, the foundation argues that the people involved in the foundation demonstrate a deeper passion for their work and are motivated in a way that might not be possible in a for-profit industry. Additionally, Apereo members feel that the amount of innovation in the foundation can be credited to its strong community and flat (non-hierarchical) organisation. However, there has been little empirical exploration of this notion beyond these personal reflections.

This structure also poses several challenges. For example, disagreements between community members sometimes occur. Other issues include the formation of cliques and changing or rotating membership. The focus on volunteer efforts means that there are no dedicated staff members in areas such as marketing and web presence. The international nature of collaborations means national interests and initiatives sometimes distract or hinder progress. Despite these issues, Apereo members feel strongly that to control the foundation’s organisation system and community efforts would decrease its potential for innovation.

Apereo hopes to become the baseline framework for open learning analytics initiatives, delivering an infrastructure for longitudinal data throughout the lifecycle of the learner.

In the near future, the foundation’s learning analytics efforts will be focused on the UK national initiative, and on preparing programs such as Learning Analytics Progress for wide-scale adoption. Increasing awareness of the foundation’s work and diversifying those who use its programs are also priorities. Looking ahead to the next 10-15 years, Apereo hopes to become the baseline framework for open learning analytics initiatives, delivering an infrastructure for longitudinal data throughout the lifecycle of the learner.

Apereo: Policies

Several policy-driven initiatives are viewed as important to the future success of Apereo’s work in the learning analytics field. One key issue is national policies related to data aggregation and data privacy. In many countries, there is little or no access to real-time education statistics. In the UK, for example, higher education statistics are reported to the Higher Education Statistics Agency (HESA), but there is a significant lag in their publication and the 2013 university enrolment statistics were not available until 2016.

In other contexts, policies and laws are out-dated and may pre-date the Internet. In the USA, for example, privacy requirements for student records are mandated by the Family Educational Rights and Privacy Act (FERPA), which was developed in 1974 and last amended in 2001. This means that many schools have difficulties when it comes to accessing student data for learning analytics. National and international policies on student data collection and access need to be revisited in order for learning analytic programs, such as those developed by Apereo, to reach their full potential.

To achieve their goals, Apereo foundation members recognise that a common data dictionary and data sharing policies will be necessary for predictive models to be run on an international scale. In a European context, for example, one current policy roadblock is the lack of European-wide data aggregation and an associated inability to share data between institutions. A more integrated and international policy would allow each university in Europe to have a baseline set of analytics. One step towards achieving this goal would be a large-scale project with an international scope, such as a European-wide project focusing on the use of the Apereo software stack for learning analytics.

One current policy roadblock is the lack of European-wide data aggregation and an associated inability to share data between institutions. A more integrated and international policy would allow each university in Europe to have a baseline set of analytics.

In addition to international and national policies, individual institutional policies are important to the success of Apereo’s work. Many universities currently do not have an established information strategy. This results in a lack of control of their data and little understanding of how to aggregate and analyse them. Institutional ethics policies related to data sharing, such as Jisc’s recently published Code of Practice (see the Inventory no: 42), will be necessary for learning analytics. Top-down institutional strategies are therefore key drivers in the successful adoption of programmes like Apereo. For Apereo to integrate with the university structure, a university culture that recognises and prioritises the fact that information has value is vital.

Blue Canary

Commercial providers of learning analytics can move the whole field forward

Blue Canary: Introduction

Blue Canary was, until the end of 2015, a commercial provider of customised solutions for predictive analytics, primarily focused on predicting which students were at risk in terms of course completion. It was then acquired by Blackboard, a developer of a virtual learning environment and course management system. The case of Blue Canary illustrates how the efforts of a community – including funders, universities, researchers, states and entrepreneurs – can create a path to success not only for a start-up learning analytics company, but also for the field of learning analytics as a whole.

Blue Canary: Context

In 2011, a million-dollar grant from the Bill & Melinda Gates Foundation was awarded to develop the Predictive Analytics Reporting (PAR) Framework (see the Inventory no: 43). The goal was to identify variables that influence student retention and progression, and to guide decision-making that would improve post-secondary US student completion rates. The Predictive Analytics Reporting framework brought together data representing more than 400,000 student records from across six higher education institutions belonging to Western Interstate Commission for Higher Education⁹³. Each of the six participating institutions had been exploring or implementing analytics projects on their own student data. The PAR Framework enabled them to expand on this work by exploring the patterns that could be derived when the six institutional datasets were considered as a single, unified sample.

Given the experience of the project, in 2013, one of the project participants went on to start a company called Blue Canary. The founder, Mike Sharkey, saw that a one-size-fits-all model of predictive analytics is unlikely to work in every context. Some early insights into the experience had led him to believe a customised solution at the institution level could be a viable strategy for a predictive analytics company. One insight he presented was that in the above-mentioned data set, which brought together records from two community colleges, two for-profit universities, and two four-year public universities, the data had a lot of commonalities⁹⁴. These commonalities, together with university-specific elements, needed to be taken into consideration in order to develop predictive analytics for student retention.

Blue Canary: Design and implementation process

In 2012, at the international conference on Learning Analytics and Knowledge (LAK12)⁹⁵, the founder of Blue Canary provided a description of the predictive model at the University of Phoenix and how this model differed from the Predictive Analytics Reporting (PAR) framework that had been developed. In his talk, he illustrated how some of the PAR indicators, which were used to predict risks to student retention and progress toward degree completion, had low value in relation to the standard practices at the University of Phoenix.

In 2013, Blue Canary was started taking a bootstrap approach (an approach that requires low levels of initial capital) to building a company that focused on predicting which students were at risk in terms of course completion. Mike Sharkey, the founder, worked with a partner that focused on markets such as health care (through a company called Clairvoyant), and together they produced customised predictive analytics solutions for their clients.

Blue Canary worked on the basis that there are two parts to the student retention problem. The first problem is to identify students at risk and the second problem is to develop intervention strategies that retain those students.

There are two parts to the student retention problem. The first problem is to identify students at risk and the second problem is to develop intervention strategies that retain those students.

The company focused on solving the first of these problems for institutions. Therefore, the business model was based on collaboration between Blue Canary and its customer institutions. The company predicted which students were at risk each week, and it was then up to each client university to take action on those predictions and to make interventions that would retain those students.

Each week it provided customers with a list of students who predictive analytics had identified as likely to drop out in the coming week. It was estimated that a small, targeted list (including around 12 at-risk students) was more helpful to his customers than a list of 300 students at various degrees of risk.

The aim was not to replace humans with analytics but rather to augment human decision-making with data based on predictive models. In doing this, the company worked to ensure the data used in its predictions were as transparent as possible. This transparency of the predictive model provided critical information that could be used by institutions to make decisions about intervention. As the solution was customized to institutions on a case-by-case basis there was no generic model developed in which the same variables were always considered critical.

By 2014, Blue Canary had started to gain momentum and the key challenges to growth were getting brand recognition and using a salesforce to penetrate the market. Part of the sales strategy was to identify an ideal customer. In the case of Blue Canary, the team determined that an ideal customer would have five attributes. These were not focused on learning and teaching but on institutional strategy and leadership.

1. An institutional goal of improving student retention

2. Key influencers aware of the power of analytics

3. Existing data footprint

4. Stakeholder commitment to action and intervention

5. Defined purchase decision process.

Given that Blue Canary saw itself as only half of the solution (predicting which students were at risk), these criteria helped to ensure that staff spent their time working with customers who would capable of providing the other half of the solution. The company looked for customers who would successfully purchase their product, provide the necessary data for the product to work, would be interested in and capable of taking action on the predictions that Blue Canary could provide, and would understand the value that predictive analytics could provide. The vendor alone could not solve the problem of student retention; it was only by working in partnership with an institution that the predictive analytics could be used to support students.

Some customers ‘got it’ and brought on board not only people who could collaborate on the technical work with Blue Canary but also people who could take action based on the information provided by Blue Canary. However, at other institutions, it was not clear who was held accountable for student retention and this lack of accountability made it more difficult for Blue Canary to provide an effective service. While the company could provide predictions, institutions needed to have infrastructure and staff in place to focus on solving the problem of student retention. The institution required not only data but also strategic leadership in order to work with a learning analytics provider.

The Learning Analytics and Knowledge (LAK14) conference in 2014, like those in previous and subsequent years, included an annual data challenge that made a data set public so that members of the learning analytics community could tackle a specific challenge using the same data set. That year Blue Canary won the data challenge, helping this relatively new company to establish international brand recognition⁹⁶. The LAK Data Challenge provides an example of how opportunities can be created to showcase the work of those who are fully engaged in practice to a research community. The challenge also provides an opportunity for practitioners to inform researchers about technical possibilities.

While the company could provide predictions, institutions needed to have infrastructure and staff in place to focus on solving the problem of student retention.

Awards can play an important part in establishing the brand of a start-up. In 2015, Blue Canary’s partner company Clairvoyant won the Governor’s Celebration of Innovation (GCOI) Award in the Start-Up of the Year category recognising the innovative analytics work done by Blue Canary. The award is also an example of how regional support, in this case provided by the Arizona Commerce Authority⁹⁷, can help a new learning analytics company to establish itself.

Blue Canary: Experience

At the end of 2015 Blackboard acquired Blue Canary where Sharkey is now Vice President of Analytics, in charge of a suite of products including Blue Canary. He believes that this gives him the opportunity to work on products that not only have brand recognition but also have a sufficient sales force to support a broader impact.

In 2016, Sharkey was one of the Chairs of the Practitioner Track at the annual Learning Analytics and Knowledge conference, LAK16. He considers that his role from the early years of the conference has been to represent the practitioner, and he has been happy to help to cultivate a role for the practitioner at the conference and to develop the role of the practitioner in moving the field forward. He sees that collaboration between researchers and practitioners, and between the education sector and the for-profit sector, is critical to moving the field forward. He has seen at first hand the value of working with data across institutions and also the implications for ethics, data protection and privacy. Blue Canary explicitly avoided analysis of data across clients. The company carefully kept client data separate, and this was an important element of its data privacy agreement with its customers.

Collaboration between researchers and practitioners and between the education sector and the for-profit sector is critical to moving the field forward.

Blue Canary: Policies

Data privacy offers both challenges and opportunities for potential collaboration between for-profit companies and universities. Blue Canary found that data privacy policies not only restricted opportunities for analysis but also prevented clients from developing insights across institutions. Such policies may limit opportunities for vendors to improve products for their clients and ultimately reduce their benefits for students.

Blue Canary has worked to reduce the division between researchers and practitioners, and the division between educational institutions and vendors. While for-profit organizations do not function in the same way as educational institutions, there needs to be a viable way for those organisations to collaborate. It is important that data privacy policies take into account how collaboration between universities and for-profits can be supported, rather than prevented.

Annex 3: Background to Learning Analytics

Research topics that have contributed to current thinking on learning analytics

Learning analytics research is a fast-developing field that has been taken up worldwide – particularly in Europe, Australia and North America. By 2011, scholars were already beginning to formalise what learning analytics would mean in relation to stakeholders, processes and values. This led, in 2012, to the publication of several different approaches based on a survey of the literature. One of these, a reference model, followed the practice of earlier surveys of the educational data mining literature by focusing on analytical methods. It mapped out the ‘what, who, why, and how’ of learning analytics (Chatti et al., 2012). A complementary map of the field’s drivers, developments and challenges (Ferguson, 2012), published in the same journal, was unusual in giving attention to technical, practical and political drivers of the field.

At the same time, although not a literature survey, Greller and Drachsler (2012) developed a generic framework, which is essentially a model of the learning analytics domain encompassing internal limitations, external constraints, stakeholders, instruments, objectives, and data. The influence of these three papers is demonstrated by Google Scholar citation counts of 146, 217 and 180, respectively, by May 2016.

As studies from many different research areas have shaped today’s thinking about learning analytics, the following sub-sections examine the research topics that emerged prior to and in parallel with learning analytics.

Educational data mining (EDM)

Writing at the time when educational data mining was establishing its own identity, Romero and Ventura charted its emergence from 1995 to 2005 by defining those aspects that set educational data mining apart from commercial applications (Romero & Ventura, 2007). The authors’ emphasis is frequently on tools and analytical methods – such as data pre-processing, clustering, association rules, classification and visualisation – as is the case with the later review of the field by Baker and Yacef (2009), which appeared in the inaugural volume of the Journal of Educational Data Mining. Other literature surveys have provided more recent analysis of the methods, algorithms, processes, and data-sources used. Some of these are technically detailed (Peña-Ayala, 2014) and others more conceptual (Steiner et al.,2014).

Adaptive and ‘intelligent’ systems

The development of intelligent tutoring systems is a particular theme in the educational data mining literature, building on earlier work on adaptive hypermedia through ‘attempt[s] to be more adaptive by building a model of the goals, preferences and knowledge of each individual student and using this model throughout the interaction with the student in order to adapt to the needs of that student’ (Romero & Ventura, 2007).

The late 2000s saw an improvement in the student models that drive intelligent tutoring systems as a key area of application of educational data mining, that is to say representations of a ‘student’s characteristics or state, such as the student’s current knowledge, motivation, meta-cognition, and attitudes’ (Baker, 2009). As well as improved performance in the knowledge domain, advances were made in the detection of elements as varied as gaming the system, self-efficacy and motivational/affective state.

This trend of increasing research effort focussing on the treatment of learning strategies, affect and metacognitive state has continued (Papamitsiou & Economides, 2014). A detailed literature review provides an overview of technical approaches to student modelling and charts the changing emphasis on performance v behaviour modelling between 2010 and 2013 (Peña-Ayala, 2014).

Personal learning and self-regulation

Some practitioners and scholars and innovators envisioned the relationship of individual learners with their studies and technology as one of self-organisation served by a personal learning environment. They and reacted against the automated view of personalisation embodied by intelligent tutoring systems and other adaptive systems. Many of the same technical approaches to learner modelling apply, but the emphasis is on using analytics methods to support learner agency. It is worth noting, however, that research on intelligent tutoring systems has tended to focus on school-age learners while interest in personal learning environments developed primarily in higher education and reflects differing assumptions about which aspects of learning learners should control.

Although personalised learning has been seen as a desirable aim of learning analytics, it was not reflected in learning analytics research published before 2012 (Chatti, 2012), and remained as an opportunity for learning analytics research in a literature survey of 2014 (Papamitsiou, 2014).

Insight into student performance and progress

The limitations of the traditional teacher-learner relationship, in terms of what is practical for teachers and other actors in the learning process to monitor unaided, is a clear theme in the literature. One of the lines of thought which coalesced with others as learning analytics emerged was ‘academic analytics’. These analytics emphasise the business-oriented concerns of higher education administrators and managers. Academic analytics have a history that stretches back into the late 1990s, with the detection of at-risk students becoming a common theme by the end of the century (Chatti, 2012).

Earlier work tended to rely on conventional educational data such as attendance records, assessment data, course, and curriculum goals. Activity records from learning management systems were noted as a potential source but were not systematically exploited (Romero & Ventura, 2007). Later work began to include more activity data from learning management systems. The Signals tool/process developed at Purdue University that made use of this type of data was, in many ways, a flagship for interest in learning analytics which straddled both research and business interests in higher education (Ferguson, 2012). Signals is also an interesting case from the perspective of a discussion of learning analytics adoption because, in addition to the method and the claims for efficacy, the people involved at Purdue had taken care to address some of the cultural aspects necessary for adoption, although this aspect of the work is not well captured in the academic literature.

Recent research activity has incorporated more detailed treatment of signs of engagement and mood and matched them to task-level performance (Papamitsiou & Economides, 2014). Overall, predicting performance is an area that has seen sustained activity (Sin, 2015).

Assessment and feedback

The role of analytics in assessment and feedback is here separated from the use of assessment data for analytics. Earlier work tended to focus on the latter and, although an early review (Chatti, 2012) notes that 13% of the papers surveyed had related to assessment and feedback, the assessment and feedback process aspect formed only part of a broader picture.

A substantial part of the work on assessment has been on the inferences that can be drawn from learner responses to objective questions (Peña-Ayala, 2014). This work includes activity aimed at inferring what learners know, as well as research on the test instruments.

Work is continuing on the less objectively definable aspects of providing feedback that is more compelling and actionable from a learner point of view. Another focus is on assessment in more authentic settings than formal testing, including assessment of process rather than product (Steiner et al., 2014; Papamitsiou & Economides, 2014b). This aspect of assessment and feedback appears to be attracting increasing attention, while research activity on the more objective aspects of assessment appears to be declining (Peña-Ayala, 2014).

Insight into engagement and social learning

Use of activity data from general-purpose educational software applications initially tended to focus on monitoring and non-predictive analysis. These were frequent topics in the literature available in 2011, particularly in papers from the first international conference on Learning Analytics and Knowledge, alongside research on intelligent tutoring systems and adaptive systems, which flourished at International Educational Data Mining conferences (Chatti, 2012).

A particular topic of interest, driven by widespread engagement with the social-constructivist learning paradigm in the European and North American technology-enhanced learning community is social network analysis. This was an existing field of research in the social sciences that aims to understand how actors (such as learners and teachers) relate through their actions or opinions (Ferguson, 2012b). These social networks are presented graphically as networks in which individual actors appear as points and their interactions are represented as lines (edges) that connect those points.

The application of social network analysis provides an early example of learning analytics research being explicit about its underpinning pedagogic theory. This was in contrast to previous work in learning analytics that had not dealt with theories of how learning and teaching take place, and also contrasted with visualisations of learner and teacher activity that were supposedly pedagogically neutral (Vuorikari & Scimeca, 2013).

Resource recommendation systems

The use of clustering and association rule algorithms to recommend educational content was an important field of activity when educational data mining emerged at the beginning of the century, according to citation figures (Baker, 2009). Similarly, both content-based recommendations and collaborative filtering (which use the textual content and data about user preferences, respectively) figured significantly in technology-enhanced learning research in the late 2000s (Chatti, 2012; Manouselis et al., 2011). However, little new activity on this topic was evident by 2014 (Papamitsiou & Economides, 2014).

Game-based learning and serious games

Game-based learning has an established history and this area saw a flourishing of research interest at roughly the same time as learning analytics. However, research activity initially failed to focus on the insights that learning analytics methods could bring to a game-based learning scenario (Papamitsiou & Economides, 2014). There is now evidence that the conjunction of learning analytics and serious games is exciting attention and it is clear that earlier work can be re-conceptualised in a game-based learning setting (Steiner et al., 2014). Game-based learning was identified as one of the top three topics at the 2014 International Conference on Educational Data Mining (Sin, 2015).

Insight into effective curriculum design and pedagogic strategies

Curriculum design, learning design and the selection of pedagogic strategies form a set of related topics which are present in the learning analytics literature but never really prominent. Early work on educational data mining included research on ‘relating a student’s later success to the amount of each type of pedagogical support the student received up to that point’ (Baker, 2009). More recent studies note research work on a variety of factors, from the way teachers use online tools to the estimation of prerequisite structures in subject material (Peña-Ayala, 2014). This research appears to be somewhat fragmented and lacking a unifying idea.

One issue that has been touched on in many strands of learning analytics research is the modelling and discovery of behavioural patterns. This is an aspect of the student models created for intelligent tutoring systems, as well of research into MOOC data (Papamitsiou & Economides, 2014).

Distance & online education, and MOOCs

Early educational data mining literature tended to contrast distance and online education with a traditional education setting. On one hand, the issues caused by reduced levels of personal contact in distance education could partially be overcome by extracting more from the data. On the other hand, online education generated activity data that was potentially useful in many ways (Romero & Ventura, 2007).

The sheer volume of data produced by massive open online courses (MOOCs), in a relatively consistent and therefore more easily analysed form, was a gift to learning analytics, particularly to those using data mining methods. Researchers were quick to explore this data in relation to many existing research topics, including behaviour discovery/modelling, promoting engagement, and the identification of early signals of drop out (Papamitsiou & Economides, 2014)..

Refinement and validation of educational theories

There has been relatively little emphasis on the refinement and validation of educational theories, which we take to include informal conventional wisdom such as the relationship between self-discipline and likely learning gains. This is surprising, as this work was identified as a key area of application in the inaugural edition of the Journal of Educational Data Mining (Baker, 2009). Low levels of reference to specific educational theoretical frameworks in learning analytics literature remains a shortcoming in the research (Nistor, 2015).

Annex 4: Glossary

The Glossary contains key terms in the field and is intended for use when reading the wider learning analytics literature. Not all terms within it are used in this report.

The Glossary is divided into three sections, each of which is arranged alphabetically:

• Terms commonly used in relation to learning analytics

• Technical terms relating to learning analytics

• Academic terms relating to learning analytics

Terms commonly used in relation to learning analytics

Term

Definition

academic analytics

The process of evaluating and analysing organisational data from the systems of educational institutions for reporting and decision-making reasons. If a distinction is drawn with learning analytics, academic analytics are typically focused at the level of the institution or above, whereas learning analytics are typically focused at the level of the individual.

adaptive

Adaptive learning

Of some learning activity or environment, means that the system adapts to characteristics or behaviours of the individual learner.

affect

Emotions or moods.

algorithm

A process or set of rules to be followed in problem-solving operations, especially by a computer.

analytics

Processing of data to produce meaningful patterns and inferences, or individual metrics that convey information about a large dataset.

API

Application programming interface, the means by which software components exchange data or direct processing.

at-risk students

Predictive analytics are used to identify students who are at risk of dropping out or failing a course

big data

A loose term for situations where the amount of data to be processed is so large that traditional approaches do not work, or for using data processing approaches that were originally developed to deal with very large datasets.

clickstream

A clickstream records the parts of the screen a computer user clicks on. It forms a record of pages a user has visited and shows the route taken through different websites.

cognitive tutor

A type of intelligent tutoring system in which feedback is provided to the learner based on cognitive models of the learner (typically inferred from their responses to the system) and of the knowledge domain to be learned. As a trademark, systems of this type produced commercially by Carnegie Learning.

dashboard

A dashboard is a visualization that presents a set of data in a single display. In educational settings a dashboard may include summary information about learners’ attendance and attainment. It may also show an aggregated summary of information about a group of learners such as a class and provide a facility for its user to explore the individual scores that make up this aggregated summary.

data mining

Algorithms and techniques for discovering patterns and regularities in large datasets.

data protection

Laws and rules concerning the processing of personal data, and the associated processes and procedures for ensuring that processing complies with these. Within the EU, there is clear and relatively strict legislation aimed at ensuring privacy and fairness in the processing of personal data. Similar legislation exists in other OECD countries, with the exception of the USA, where the law is substantially more permissive, except for personal data about children.

data warehouse

A central repository of integrated data, usually from several sources, that is designed for queries and analysis.

educational data mining or EDM

An emerging discipline, concerned with developing methods for exploring the unique types of data that come from the educational setting, and using those methods to better understand students, and the settings which they learn in. In contrast with learning analytics, it is typically concerned with finer-grained detail about individual learner behaviours, and is closer to computer science as a discipline.

engagement

A broad term with a range of meanings. I can mean a substantial affective investment of a learner in the process of learning (as in a deep orientation to learning). It can also mean use of learner activity data to infer how long learners spent on particular activities.

intelligent tutor & intelligent tutoring system

Software that gives immediate, adaptive and individual responses to learners, such as instruction and feedback, generally without requiring input from a human tutor.

interoperability

Ability of different technologies to communicate, exchange data and to use the data that has been exchanged.

learning analytics

The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs. In the context of this report, the term is used more broadly to cover both academic analytics and educational data mining.

learning management system (LMS)

A learning management system is used to administer, document, track, report and deliver online learning resources and courses. Examples include Blackboard and Moodle. Also referred to as a virtual learning environment.

massive open online course or MOOC

An online course open for anyone to study without pre-requisites or charge, intended for a larger number of learners than a traditional course.

open-source software

The source code for open-source software is made available so that it can be studied, changed and distributed to anyone and for any purpose. Open-source software is often developed collaboratively and in public. For example, the Moodle virtual learning environment is provided freely as open-source software that can be adapted, extended or modified by anyone.

prediction

predictive modelling

Predictive modelling is used to create a statistical model of future behaviour and thus to make predictions about future events, such as whether a student will pass or fail a course.

privacy

Keeping personal data so that it is not observed by others, or by unauthorised people.

real-time data

Data that is delivered as soon as it is collected. These could include a learner’s actions while an activity is in process.

recommendation system

recommender system

A system that uses patterns of behaviour to predict the rating a user would give to an item. These systems can be used to recommend course materials or activities.

reliability

Whether a particular method gives the same result given input that is essentially the same (see also, validity).

retention

In universities, keeping students who have enrolled on a course until they complete that course (reducing drop-out). The retention rate is the fraction of students who started a course who complete it, as distinct from the pass rate, which is the fraction of students who passed the course’s assessment. Can apply to an individual module, semester or course, or to an entire degree programme.

social learning analytics

Analytics that focus on how learners build knowledge together in their cultural and social settings.

Society for Learning Analytics Research or SoLAR

An inter-disciplinary network of leading international researchers exploring the role and impact of analytics on teaching, learning, training and development.

validity

Whether a particular method does what it is supposed to do, or measures accurately what it is intended to measure (see also, reliability).

visualisation

A graphical or visual display of information, intended to help the viewer to understand a set of data.

Technical terms relating to learning analytics

affective computing

Computing that takes into account the emotional state or mood of the user.

API

An application program interface (API) specifies how software applications should interact. It enables them to communicate and to share data with each other.

association rule

In data mining, a strong association discovered between items using methods that look for patterns where items co-occur (are associated), as distinct from sequence rules, which are the result of sequence mining: looking for patterns where one item happens after another (in sequence)

Bayesian knowledge tracing

A particular way of inferring the cognitive model of learners based on whether their answers are correct or incorrect. Typically used in cognitive tutors.

Bayesian network

A probabilistic model of the relationships between variables, typically ‘learned’ from a large dataset.

causal discovery

In data mining/machine learning, algorithms and techniques that seek to discover causal relationships between variables, as opposed to mere associations (for example wet pavements and open umbrellas are associated, but one does not cause the other – they share a common cause, rain).

classification

In machine learning, algorithms and techniques for determining which category an observation belongs in, based on categories developed from a training set of data. An example would be whether a student’s learning activity is ‘on track’ or ‘in trouble’ based on a comparison with data from students from a previous instance of the same course.

cluster analysis clustering

In data mining/machine learning, algorithms and techniques for grouping data so that each group (cluster) contains items that are more similar to each other than they are to items in the other clusters.

dynamic Bayesian networks

A Bayesian network concerned with how variables change over time.

A probabilistic model of the relationships between various variables at one point in time and another.

feature engineering feature selection

In machine learning, the often-challenging process of identifying or developing features (data that could be useful for prediction or classification) for algorithms to work on.

hierarchical clustering

A particular sort of cluster analysis that aims to group (cluster) data into groups (clusters) that form some sort of hierarchy.

knowledge tracing

Algorithms and techniques for inferring the cognitive model of the learner, typically used in cognitive tutors.

log files

Computer files that contain lists of past events. For instance, in a learning environment, a log file might contain an entry for each time the learner clicked on an item, showing which item was clicked and when. Analysis of log files can be useful for tracking learner behaviour and for improving learning environments.

logistic regression

In machine learning, a particular algorithm used for classification when the data are to be classified into discrete categories, such as ‘pass’ or ‘fail’.

machine learning

The use of computer algorithms to detect patterns in data, such as cluster analysis or predictive modelling.

matrix decomposition
matrix factorisation

Algorithms that take a matrix and determine two factors (i.e. two new matrices) that, when multiplied together, give the original matrix. Often used to develop systems that can recommend particular resources to a learner based on other learners’ behaviours or outcomes.

natural language processing NLP

Within computational linguistics, algorithms and techniques for relating human languages (natural language) to computer language. Used to enable computer systems to communicate using human language.

predictive modelling

Finding patterns in data and using those patterns to make predictions about other data, such as whether a student will pass or fail a course.

process mining

Looking for data patterns that relate to learning processes, and using the models developed for purposes such as uncovering those learning processes.

regression

A broad set of statistical tools and algorithms for modelling and analysing the relationships between variables.

sequence mining

Looking for patterns where items happen in sequence (one after another), as distinct from patterns where they co-occur (are associated, as in association rules).

social network analysis

Algorithms and techniques for analysing the relationships between individuals (social relationships) based on network and graph theory. The underlying model is one of ‘nodes’ (individuals or things) and ‘edges’ (relationships or interactions between them).

text mining

Algorithms and techniques for finding useful patterns in text, often using natural language processing.

visual analytics

Processing of data to produce meaningful visual patterns, or individual visualisations that convey information about a large dataset.

xAPI

An open source application program interface (API) that enables different applications to share data about human performance.

Academic terms relating to learning analytics

affect detection

Of a computer system, the ability to detect the emotions or moods of learners.

cognitive modelling

The process of developing models of the cognitive processes in learners, typically for the purposes of a cognitive tutor.

computational linguistics

An established interdisciplinary field concerned with using computers to analyse human languages (natural language).

design research

Research into the processes of design or, more recently, research that forms part of a process of design.

digital literacy

The skills needed to find, evaluate, make use of, share and create content using digital technologies.

discourse analytics

Collective term for a wide variety of approaches to the analysis of series of communicative events, typically those that involve speech or written communication.

evidence-centred design

A method for the design and evaluation of educational systems that focuses on higher-level knowledge.

formative assessment

Any type of assessment that contributes to learning by providing actionable feedback to the learner.

eye tracking

Determining where someone’s eyes are focused and, typically, using this information to inform design or research.

game-based learning

A type of game play that is associated with working towards the achievement of defined learning outcomes.

item response theory

The study of how learners’ responses to individual questions (items) in tests relate to their underlying abilities, typically using probabilistic approaches.

learning curves

Graph showing amount of learning over time (often using test scores) or repeated attempts at a task.

peer assessment

Students mark the work of their fellow learners, based upon benchmarks provided by an educator.

psychometrics

Field of study concerned with the measurement of psychological variables. In this context, typically used for the construction and validation of questionnaires and tests.

self regulation

Self-regulated learning is guided by thinking about your own thinking, acting strategically and being motivated to learn.

student model
learner model
user model

Student models represent information about a student’s characteristics or state, such as their current knowledge, motivation, meta-cognition and attitudes.

summative assessment

Any form of assessment that demonstrates the extent to which a learner has met the assessment criteria.

Annex 5: Experts

The Expert Workshop held in Amsterdam in March 2016 as part of this study included attendees from 13 countries and ten European projects.

Name

Country

Affiliation

Invited experts

1

Adam Cooper

UK

Tribal Group

2

Alan Berg

The Netherlands

Apereo Foundation

3

Alex Rayón Jerez

Spain

Universidad de Duesto

4

Andrew Cormack

UK

Jisc

5

Anne Boyer

France

Université de Lorraine

6

Barbara Wasson

Norway

University of Bergen

7

Charlotta Grönqvist

Norway

Sanoma

8

Daniel Spikol

Sweden

Malmö University

9

Dirk Tempelaar

The Netherlands

Maastricht University

10

Ed Foster

UK

Nottingham Trent Uni

12

Ian Dewes

UK

Dunchurch Infant School

13

Jocelyn Manderveld

The Netherlands

SURFNet

14

Kristel Rillo

Estonia

MoE

15

Kristian Ørnsholt

Denmark

Ministeriet for BOL

16

María Jésus García San Martin

Spain

Ministry of Education

17

Mark Brown

Ireland

Dublin City University

18

Susan Flocken

Belgium

ETUCE

19

Tim Vogelsang

Germany

iversity

20

Topi Litmanen

Finland

Claned Group

European project representatives and associated individuals

21

Anouschka van Leeuwen

The Netherlands

22

Baltasar Fernández Manjón

Spain

RAGE
BEACONING

23

Bert Bredeweg

The Netherlands

24

Bert Slof

The Netherlands

25

Gábor Kismihók

The Netherlands

www.eduworks-network.eu

26

Indra Posthumus

The Netherlands

27

Jan-Paul van Staalduinen

The Netherlands

STELA

28

Jeroen Donkers

The Netherlands

WatchMe

29

Katerina Riviou

The Netherlands

PBL3.0

30

Liina Malva

Estonia

31

Marieke van der Schaaf

The Netherlands

WatchMe

32

Marius van Zandwijk

The Netherlands

33

Michael Kickmeier-Rust

Austria

LEA’s Box

34

Noelia Cantero

Brussels

SHEILA

35

Stefan Mol

The Netherlands

36

Tom Broos

Belgium

STELA

37

Wietse van Bruggen

The Netherlands

LACE

Organising team and European Commission representatives

37

Geir Ottestad

Belgium

European Commission

38

Konstantin Scheller

Belgium

European Commission

39

Jonatan Castaño Muñoz

Spain

Joint Research Centre

40

Riina Vuorikari

Spain

Joint Research Centre

41

Yves Punie

Spain

Joint Research Centre

42

Doug Clow

UK

LACE

43

Hendrik Drachsler

The Netherlands

LACE

44

Maren Scheffel

Germany

LACE

45

Andrew Brasher

UK

LAEP

46

Bart Rienties

UK

LAEP

47

Garron Hillaire

UK

LAEP

48

Jenna Mittelmeier

UK

LAEP

49

Rebecca Ferguson

UK

LAEP

References

Arnold, K.E., et al. Building institutional capacities and competencies for systemic learning analytics initiatives. in Fourth International Conference on Learning Analytics And Knowledge – LAK14 (24-28 March). 2014. Indianapolis, USA: ACM.

Baker, R.S.J.d. and K. Yacef, The state of educational data mining in 2009: a review and future visions. Journal of Educational Data Mining, 2009. 1(1): pp. 3-17.

Chatti, M.A., et al., A reference model for learning analytics. International Journal of Technology Enhanced Learning, 2012. 4(5/6): pp. 318-331.

Colvin, C., et al., Student Retention and Learning Analytics: A Snapshot of Australian Practices and a Framework for Advancement, 2015, Australian Government: Office for Learning and Teaching: Sydney, Australia.

Cormack, A., A data protection framework for learning analytics. Journal of Learning Analytics, 2016. 3(1).

Daniel, B., Big Data and analytics in higher education: opportunities and challenges. British Journal of Educational Technology, 2015. 46(5): pp. 904-920.

European Commission. (2013). Opening up Education: Innovative teaching and learning for all through new Technologies and Open Educational Resources (Communication No. /* COM/2013/0654 final */. Retrieved from http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52013DC0654

European Union, New priorities for European cooperation in education and training. 2015 Joint Report of the Council and the Commission on the implementation of the strategic framework for European cooperation in education and training (ET 2020). Official Journal of the European Union, C 417, 15 December 2015

Ferguson, R., et al., Setting learning analytics in context: overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 2015. 1(3): pp. 120-144. http://oro.open.ac.uk/42115/

Ferguson, R., et al., Special section on ethics and privacy in learning analytics. Journal of Learning Analytics, 2016. 3(1).

Ferguson, R., Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning (IJTEL), 2012. 4(5/6): p. 304-317. http://oro.open.ac.uk/36374/

Ferguson, R., The State of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, 2012, Knowledge Media Institute, The Open University: Milton Keynes, UK.

Greller, W. and H. Drachsler, Translating learning into numbers: a generic framework for learning analytics. Educational Technology and Society, 2012. 15(3): pp. 42-57.

Macfadyen, L., et al., Embracing Big Data in complex educational systems: the learning analytics imperative and the policy challenge. Research & Practice in Assessment, 2014. 9 (Winter): pp. 17-28.

Manouselis, N., et al., Recommender systems in technology enhanced learning., in Recommender Systems Handbook, P.B. Kantor, et al., Editors. 2011, Springer: USA. pp. 387-415.

Nistor, N., M. Derntl, and R. Klamma, Learning analytics: trends and issues of the empirical research of the years 2011–2014, in Design for Teaching and Learning in a Networked World, G. Conole, et al., Editors. 2015, Springer International Publishing. p. 453-459.

Papamitsiou, Z. and A.A. Economides, Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Educational Technology & Society, 2014. 17(4): pp. 49-64.

Papamitsiou, Z. and A.A. Economides, Temporal learning analytics for adaptive assessment. Journal of Learning Analytics, 2014. 1(3): pp. 165-168.

Peña-Ayala, A., Educational data mining: a survey and a data mining-based analysis of recent works. Expert Systems with Applications, 2014. 41(4 PART 1): pp. 1432–1462.

Romero, C. and S. Ventura, Educational data mining: a survey from 1995 to 2005. Expert Systems with Applications, 2007. 33(1): pp. 135-146.

Siemens, G., S. Dawson, and G. Lynch, Improving the Quality of Productivity of the Higher Education Sector: Policy and Strategy for Systems-Level Deployment of Learning Analytics, 2013, SoLAR. http://bit.ly/1KNZ9HH

Siemens, George, et al. Open Learning Analytics: an integrated & modularized platform. Diss. Open University Press, 2011.

Sin, K. and L. Muthu, Application of Big Data in educational data mining and learning analytics – a literature review. ICTACT Journal on Soft Computing, 2015. 695 (July): pp. 1035–1049.

Steiner, C., M. Kickmeier-Rust, and M.A. Türker, Review article about LA and EDM approaches (D3.1), 2014: Graz. http://bit.ly/25BLmQv

Vuorikari, R. and S. Scimeca, Social learning analytics to study teachers’ large-scale professional networks, in Open and Social Technologies for Networked Learning, T. Ley, et al., Editors. 2013, Springer: Berlin. pp. 25-34.

Web Analytics Association, Web Analytics Definitions – Draft for Public Comment, 2008. Retrieved from:
http://www.digitalanalyticsassociation.org/Files/PDF_standards/WebAnalyticsDefinitions.pdf

Notes

¹ http://bit.ly/25k2NEQ

² http://bit.ly/25k2NEQ

³ https://www.uninett.no/en/service-platform-dataporten

⁴ https://www.surf.nl/en/innovationprojects/customised-education/learning-analytics.html

⁵ www.eduworks-network.eu

⁶ https://ec.europa.eu/jrc/digcomp

⁷ https://ec.europa.eu/jrc/entrecomp

⁸ http://www.laceproject.eu/

⁹ http://www.leas-box.eu/

¹⁰ http://www.pelars.eu/

¹¹ https://www.ou.nl/web/welten-research/pbl3.0

¹² http://bit.ly/1T46Zoq

¹³ http://bit.ly/1Mz5jMW

¹⁴ http://bit.ly/1Xu9v6E

¹⁵ http://beaconing.eu/

¹⁶ http://rageproject.eu/

¹⁷ http://www.project-watchme.eu/

¹⁸ http://www.laceproject.eu/the-lace-visions-of-the-future-of-learning-analytics/

¹⁹ http://bit.ly/26gNoXw

²⁰ http://bit.ly/1T46Zoq

²¹ http://bit.ly/1T2Xfwf

²² http://bit.ly/1IjvPgK

²³ http://www.laceproject.eu/ethics-privacy-learning-analytics/

²⁴ https://www.surf.nl/en

²⁵ http://bit.ly/1VDkjkW

²⁶ http://www.itslearning.net/

²⁷ http://www.gyldendal.no/ (in Norwegian)

²⁸ http://www.smartoving.no/ (in Norwegian)

²⁹ https://www.ikt-norge.no/english/

³⁰ https://www.uninett.no/en/service-platform-dataporten

³¹ https://www.standard.no/en/

³² https://www.kennisnet.nl/about-us/

³³ https://www.deonderwijsdagen.nl/ (in Dutch)

³⁴ https://www.surf.nl/en

³⁵ https://www.edustandaard.nl/ (in Dutch)

³⁶ http://bit.ly/1Uj0fyP (in Dutch)

³⁷ http://bit.ly/24kke5R

³⁸ http://bit.ly/1sOyVSM (in Dutch)

³⁹ https://www.poraad.nl (in Dutch)

⁴⁰ http://bit.ly/1P8t1kj (in Dutch)

⁴¹ http://bit.ly/1XSzJSD (in Dutch)

⁴² https://www.deonderwijsdagen.nl/ (in Dutch)

⁴³ https://www.kennisnet.nl/publicaties/ (in Dutch)

⁴⁴ https://nl.snappet.org/

⁴⁵ http://bit.ly/1KSE6UL (in Dutch)

⁴⁶ http://dl.acm.org/citation.cfm?id=2883892

⁴⁷ https://www.edustandaard.nl (in Dutch)

⁴⁸ http://bit.ly/22Aa9m8

⁴⁹ http://bit.ly/295mHN4

⁵⁰ http://www.laceproject.eu/

⁵¹ http://bit.ly/1Uj0fyP (in Dutch)

⁵² http://1.usa.gov/1UwH1bV

⁵³ https://www.imsglobal.org/activity/caliperram

⁵⁴ http://bit.ly/1TY78bV

⁵⁵ http://www.onderwijsinspectie.nl/english

⁵⁶ http://onsonderwijs2032.nl/ (in Dutch)

⁵⁷ http:/bit.ly/10zbH54

⁵⁸ http://oro.open.ac.uk/36934/

⁵⁹ http://oro.open.ac.uk/41229/

⁶⁰ http:/bit.ly/10zbH54

⁶¹ http://bit.ly/1Pg3hDK

⁶² http://bit.ly/1sqRuww

⁶³ http://bit.ly/1pqkJhm

⁶⁴ http://utscic.edu.au/

⁶⁵ http://bit.ly/1J3vr2l

⁶⁶ http://bit.ly/OgHVfR

⁶⁷ http://bit.ly/1sPhX6U

⁶⁸ http://bit.ly/1X0Q74a

⁶⁹ http://bit.ly/1t44RmY

⁷⁰ http://bit.ly/1QxWnJm

⁷¹ http://he-analytics.com/

⁷² http://bit.ly/24G3okW

⁷³ http://bit.ly/1LYuFDw

⁷⁴ http://bit.ly/1XT6Ci1

⁷⁵ http://bit.ly/1J3vr2l

⁷⁶ http://utscic.edu.au/tools/awa/

⁷⁷ http://utscic.edu.au/tools/clara/

⁷⁸ http://utscic.edu.au/learn/mdsi/

⁷⁹ http://utscic.edu.au/research/phd/

⁸⁰ http://utscic.edu.au/learn/aei/

⁸¹ http://utscic.edu.au/tools/awa/

⁸² http://bit.ly/1J3vr2l

⁸³ http://oro.open.ac.uk/42115/

⁸⁴ http://bit.ly/25zvhum

⁸⁵ http://bit.ly/1oAFrKW

⁸⁶ https://www.apereo.org/

⁸⁷ http://bit.ly/1PZr3VS

⁸⁸ http://bit.ly/1TMkOID

⁸⁹ http://bit.ly/1W4ejTb

⁹⁰ http://bit.ly/1LLOjCC

⁹¹ http://bit.ly/JiscTools

⁹² http://bit.ly/1RmRcOB

⁹³ http://wcet.wiche.edu/

⁹⁴ http://bit.ly/1UElS0W

⁹⁵ https://www.youtube.com/watch?v=sZNIVRA2ru8

⁹⁶ http://bit.ly/1LYaBBi

⁹⁷ http://bit.ly/1TXRQVc