Vendors are rapidly developing products for analysing learners and their activities. There’s a battle going on between the companies whose primary product is the VLE, those which sell student information systems (SISs) and those who have developed business intelligence systems for industry in general but are now targetting the education sector as a key market.
In previous posts I’ve given an overview of the different types of learning analytics systems available and described some of the simple reporting tools for the VLE/LMS. I’ve also looked at how data can be brought in from other sources to provide more sophisticated indicators of engagement, and how systems are beginning to build in workflow around managing the subsequent automated and human interventions. This post is about the emerging analytics systems built around the VLE, the ones with an online learning-centric view of the World.
Given the vast amount of data accumulated by students as they use their VLE, and the fact that for many learners this is the primary way in which they’re interacting with their institution online, it’s not surprising that VLE vendors have spotted an opportunity to develop products which use this data to help institutions improve student success. But VLE log files are only part of the picture. Much of the rest of the data needed for learning analytics is held in the SIS. Vendors such as Blackboard and Desire2Learn have therefore been building in functionality to combine data from some of the most commonly used SISs with data from their own systems, and developing ever more sophisticated analytics products to sit alongside their VLEs.
Blackboard Analytics for Learn
This product is designed to provide more in-depth analysis of student activity and performance than is available in the Blackboard Retention Centre, enabling the correlation of student behaviours to educational outcomes. Whereas Retention Centre works at level of the course and uses data from Learn only, Analytics for Learn integrates data from the VLE with the student information system and allows analysis of individuals, groups and instructors across multiple courses by staff such as deans, senior management and educational researchers.
The software facilitates the analysis of student activity and performance patterns, the identification of at-risk student behaviour, and the measurement of learning outcomes against course grades. It also allows the tracking of KPIs through institutional dashboards. Blackboard gives some examples of the questions which Analytics for Learn can help answer:
- How does student performance differ between courses where the VLE is used versus where it is not used?
- Is there a difference in student performance where the instructor went through a training class and those that did not?
- How are our efforts at improving course quality making a difference?
The second question highlights another key difference with Retention Centre. Analytics for Learn allows the analysis of teachers’ performance as well, something that will prove controversial in many institutions but will be increasingly of interest to senior managers who wish to monitor the quality of teaching in ways that were never possible in traditional classroom settings.
A whole host of reports are provided on areas such as activity and gradebook exceptions, student performance against learning outcomes, organisational unit performance against learning outcomes, “student-at-a-glance”, “course-at-a-glance”, most active instructors, and aggregated activity by organisational unit. Learning-related metrics include:
- gradebook scores
- submissions to interactive tools such as discussions
- session and course logins; time, tools and content accessed; time on task
- organisational unit and term information for aggregration and comparison
- “academic standing”
Unlike Course Signals, the software is for carrying out analysis alone, and Blackboard has chosen not to facilitate interventions or any workflow features at this stage through Analytics for Learn.
Analytics for Learn is part of the Blackboard Analytics Suite which consists of a data warehouse with data models, a data transformation process and “out of the box” integration with Blackboard Learn and three student information systems: Oracle PeopleSoft, Datatel, Inc. and SunGard Higher Education. Integration with other student information systems is, according to Blackboard documentation, “easy”. Good luck to you though if you have a bespoke SIS.
The software is of course only for users of Blackboard Learn. Visualisations can be done with the Pyramid business intelligence tool which integrates with the data warehouse. Other BI tools can be used as well. Blackboard provides consultancy for implementing Analytics for Learn. They perform the initial installation which takes “several days”. Institutions must have skills in the core technologies such as SQL Server and Windows Server.
Desire2Learn Insights
Desire2Learn have recently rebranded their VLE package as Brightspace. Built on top of this is a system called Insights which enables reporting on engagement, assessment and outcomes. It includes predictive modelling aimed at helping identify at-risk learners. A large number of data visualisation and analytics dashboards are available. Insights includes the Student Success System which is specifically aimed at enhancing retention. Target users are instructors, educational researchers and senior staff such as deans.
Desire2Learn has done some rapid development and added an impressive array of analytics capability to Insights in a short space of time. It provides a number of pre-configured reports, including some on learning outcomes and how well students have met them. “Competency” reports are available for individuals and at other levels including across courses.
Reports can also be obtained on the use of course resources and on “engagement” by users, measured by their logins to the VLE. Other reports are available on enrolments and withdrawals, course grades, quiz scores and statistics on the items themselves which may help to identify questions which discriminate well or poorly between the best or weakest students.
There are further reports available on students at risk, showing differences between courses and highlighting individuals. There is also a series of “data mining reports”, enabling insight into student behaviour, grade achievements and usage patterns – as well as the effectiveness of learning content. Reports can be obtained on tool access by users. Predictive analytics are deployed to assess risk levels of students using historical course data. Weekly predictions are then prepared, generating a “success index” for each student in the course.
An “assessments predictive chart” shows a student’s performance across all course assessments, and compares their performance with their classmates. Arrows inside the trend signs indicate whether predictions are negative or positive compared to the previous week. The dashboard can be filtered based on success levels.
Unlike Analytics for Learn, the system does facilitiate some interventions and workflow management. Emails can be directed to all students in a particular risk category, for example. A pin can be added as a visual reminder beside those students the instructor wishes to follow up with.
Insights is tightly integrated with the Brightspace virtual learning environment (formerly Desire2Learn), and built using the IBM Cognos business intelligence system. Data comes from “multiple source systems” – in particular the Brightspace VLE. Austin Peay State University reports that they are integrating data from their student information system.
A large number of metrics are provided relating to course access, content access, social learning, assessments and preparedness. It really is becoming a quite impressive offering. What a pity that hardly anyone this side of the Atlantic is using Brightspace…
Further thoughts
Most higher and further education institutions in the UK use Moodle or Blackboard Learn. If you’re using Moodle and you want to carry out in-depth learning analytics there are no sophisticated Moodle-specific analytics systems currently available, and you may need to use a business intelligence tool.
Moodle hosting organisations are beginning to provide their own reporting functionality to their clients. Two of these are the University of London Computer Centre (ULCC) and MoodleRooms, headquartered in Baltimore, Maryland. MoodleRooms’ services include the reporting of activity, grades and engagement proxies, flagging lack of activity and low grades – both within and across courses. MoodleRooms also claims to provide “a 360 degree view of your students by integrating with industry-wide SIS systems”, so they are basing their analytics not just on data from Moodle.
Both ULCC and MoodleRooms no doubt see learning analytics as key potential differentiators for their services and will be looking to further develop their reporting and analytics tools and deploy them across their growing client bases. Already 33% of UK higher education institutions are using hosted services for their institutional VLE (See UCISA Technology Enhanced Learning Survey 2014). Learning analytics provided alongside such hosted services will no doubt prove attractive for many institutions, particularly smaller ones without data scientists or sufficient technical expertise to install and maintain their own analytics systems. On the other hand one institution I have spoken to wants to get usage data from its VLE hosting service into its own data warehouse alongside other institutional data to carry out its own analytics.
Blackboard Learn users can deploy Analytics for Learn if they want more than the simple tools offered in Retention Centre. But not many institutions have yet used Analytics for Learn in anger – certainly not in the UK. One Blackboard presentation mentions three organisations where it’s being piloted: Montgomery County Community College, University of Maryland Baltimore County and Grand Rapids Community College. Given the necessity to enhance retention in many unviersities and colleges and the high penetration of Blackboard Learn there are likely to be many more institutions investigating purchasing the software.
Meanwhile only two of the 96 UK higher education institutions who completed the UCISA technology enhanced learning survey in 2014 reported using Brightspace as their primary virtual learning environment. It’s not clear if either of these is yet using Insights.
So is the battle for supremacy in learning analytics systems being lost by the VLE vendors? It would seem so at the moment – at least in the UK. Certainly many educational institutions are already using business intelligence software for their broader requirements and are beginning to explore the potential of these for analysing learners and learning. The business intelligence software is necessary anyway: Analytics for Learn is built on top of the Microsoft business intelligence stack and Insights requires IBM Cognos.
If only the VLE-centric analytics products could be made more interoperable then Moodle users for example could plug in Analytics for Learn or Insights and benefit from the innovations of companies based in the world of online learning – as well as benefitting from the underlying business intelligence software and visualisation tools.
Meanwhile the SIS vendors are moving quickly into the market and trying to promote an SIS-centric view of learning analytics, some of them even co-developing their products with UK universities.
