In discussions around the ethics and legal issues of learning analytics I’ve found the same issues cropping up again and again. Almost always they’ve already been covered somewhere in the growing collection of publications on learning analytics. Sometimes they’re expressed in different ways but boil down to the same underlying problem.
The literature review of these issues aims to provide the background for the development of a code of practice for learning analytics. But it’s a large and unwieldy document to refer to so I’ve attempted to distil and group the many issues that I’ve come across so far.
I’ve given each of the resulting 86 issues a name and have provided a question which attempts to capture the issue. Many of these cannot be answered simply; almost all could be responded to with “It depends…” Most have both an ethical and a legal dimension. Some are related more to logistics than ethics or law. And some are already dealt with by existing institutional policies.
I’ll be taking this taxonomy to a workshop next week in Paris with the Lace Project and Apereo where I hope the issues and questions can be clarified and further refined. Then it’s over to the Code of Practice Advisory Group in the UK for their advice on how to translate this into a useful Code of Practice.
| Area | Issue | Question |
|---|---|---|
| Validity | Minimisation of inaccurate data | How does an institution minimise inaccuracies in the data? |
| Minimisation of incomplete data | How does an institution minimise incompleteness of the dataset? | |
| Optimum range of data sources | How many and which data sources are necessary for increasing accuracy in the analytics? | |
| Verification of algorithms and metrics | How should an institution verify its algorithms and metrics to ensure accuracy? | |
| Spurious correlations | How can institutions avoid drawing misleading conclusions from spurious correlations? | |
| Evolving nature of students | To what extent can analytics be accurate when students’ identities and actions evolve as they progress through their studies? | |
| Authentication of public data sources | How can institutions ensure that student data taken from public sites is authenticated to their students? | |
| Ownership and control | Control of data for analytics | Who in the institution decides what data is collected and used for analytics? |
| Breaking silos | How can silos of data ownership in institutions be broken into in order to obtain data for analytics? | |
| Control of analytics processes | Who in the institution decides how analytics are to be created and used? | |
| Overall responsibility | Who is responsible in the institution for the appropriate and effective use of learning analytics? | |
| Ownership of data | What data does the institution own and what is owned by the student? | |
| Awareness | Student awareness of data collection | What should students be told about the data that is being collected about them? |
| Student awareness of data use | What should students be told about the uses to which their data is being put? | |
| Student awareness of algorithms and metrics | To what extent should students be given details of the algorithms used for learning analytics and the metrics and labels that are created? | |
| Proprietary algorithms and metrics | What should institutions do if vendors decline to make details of their algorithms and metrics public? | |
| Student awareness of potential consequences of opting out | What should students be told about the potential consequences of opting out of data collection and analysis of their learning? | |
| Staff awareness of data collection and use | What should staff be told about the data that is being collected about their students and what is being done with it? | |
| Consent and opting out | When to seek consent | In what situations should students be asked for consent to collection and use of their data for learning analytics? |
| Consent for anonymous use | Should students be asked for consent for collection of data which will only be used in anonymised formats? | |
| Consent for outsourcing | Do students need to give specific consent if the collection and analysis of data is to be outsourced to third parties? | |
| Clear and meaningful consent processes | How can institutions avoid opaque privacy policies and ensure that students genuinely understand the consent they are asked to give? | |
| Right to opt out | Does a student have the right to opt out of data collection and analysis of their learning activities? | |
| Partial consent | Can students consent to some data collection and analysis but opt out elsewhere? | |
| Right to withdraw | Does a student have the right to withdraw from data collection and analysis after previously having given their consent? | |
| Right to anonymity | Should students be allowed to provide pseudonyms to disguise their identity in certain circumstances? | |
| Adverse impact of opting out on individual | If a student is allowed to opt out of data collection and analysis of their activities could this have a negative effect on their studies? | |
| Adverse impact of opting out on group | If individual students opt out will the dataset be incomplete, thus potentially reducing the accuracy and effectiveness of learning analytics for the group? | |
| Lack of real choice to opt out | Do students really have a choice if pressure is put on them by the institution or there’s a chance of adverse impact on their academic success by opting out? | |
| Student input to analytics process | Should students have a say in what data is collected and how it is used for analytics? | |
| Change of purpose | Should institutions request consent again if the data is to be used for purposes for which consent was not originally given? | |
| Legitimate interest | To what extent can the institution’s “legitimate interests” override privacy controls for individuals? | |
| Unknown future uses of data | How can consent be requested when potential uses of the (big) data are not yet known? | |
| Consent in open courses | Are open courses (MOOCs etc) different when it comes to obtaining consent? | |
| Use of publicly available data | Can institutions use publicly available data (e.g. tweets) without obtaining consent? | |
| Student access | Student access to their data | To what extent should students be able to access the data held about them? |
| Student access to their analytics | To what extent should students be able to access the analytics performed on their data? | |
| Data formats | In what formats should students be able to access their data? | |
| Metrics and labels | Should students see the metrics and labels attached to them? | |
| Right to correct inaccurate data | What data should individuals be allowed to correct about themselves? | |
| Data portability | What data about themselves can the learner take with them? | |
| Privacy | Out of scope data | Is there any data that should not be used for learning analytics? |
| Access to employers | Under what circumstances would it be appropriate to give employers access to analytics on students? | |
| Tracking location | Under what circumstances is it appropriate to track the location of students at campuses? | |
| Staff permissions | To what extent should access to individuals’ data be restricted within an institution? | |
| Unintentional creation of senstitive data | How do institutions avoid creating “sensitive” data (e.g. ethnicity, religion) from other data sources? | |
| Use of metadata to identify individuals | Can individuals be identified from metadada even if personal data has been deleted? | |
| Requests from external agencies | What should institutions do when requests for student data are made by external agencies e.g. educational authorities or security agencies? | |
| Sharing data with other institutions | Is it appropriate for institutions to share student data with other institutions in order to increase the dataset and enhance the analytics? | |
| Enhancing trust by retaining data internally | If students are told that their data will be kept within the institution will they develop greater trust in and acceptance of learning analytics? | |
| Action | Institutional obligation to act | What obligation does the institution have to intervene when there is evidence that a student could benefit from additional support? |
| Student obligation to act | What is the student’s obligation to act on learning analytics designed to help them? | |
| Conflict with study goals | What should a student do if the suggested advice is in conflict with their study goals? | |
| Obligation to prevent continuation | Is there an obligation on the institution to prevent students from continuing on a pathway if analytics show that it is not in the student’s or institution’s interests for them to continue? | |
| Type of intervention | How are the appropriate interventions decided on? | |
| Distribution of interventions | How should interventions resulting from analytics be distributed among different stakeholders in the institution? | |
| Conflicting purposes of interventions | How does the institution ensure that it is not carrying out multiple interventions whose purposes conflict? | |
| Staff incentives for intervention | What incentives are in place for staff to intervene? | |
| Failure to act | What happens if an institution fails to intervene? | |
| Need for human intermediation | Are some analytics better presented to students via e.g. a tutor than via a system? | |
| Triage | How does an institution allocate resources for learning analytics appropriately for learners with different requirements? | |
| Triage transparency | How transparent should an institution be in how it allocates resources to different groups? | |
| Opportunity cost | How is spending on learning analytics justified in relation to other funding requirements? | |
| Favouring one group over another | Could the intervention strategies favour one group of students over another? | |
| Consequences of false information | What should institutions do if it is determined that a student has given false information to e.g. obtain additional support? | |
| Audit trails | Should institutions record audit trails of all predictions and interventions? | |
| Unexpected findings | What infrastructure is in place to deal with something unexpected arising in the data? | |
| Adverse impact | Labelling bias | Does a student profile or labelling bias institutional perceptions and behaviours towards them? |
| Oversimplification | How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances? | |
| Undermining of autonomy | Is student autonomy in decision making about their learning undermined by predictive analytics? | |
| Gaming the system | If students know that data is being collected about them will they alter their behaviour to present themselves more positively, thus skewing the analytics and distracting them from their learning? | |
| Abusing the system | If students understand the algorithms behind learning analytics will they abuse the system to obtain additional support? | |
| Adverse behavioural impact | If students are presented with data about their performance, likelihood of failure etc. could this have a negative impact on their behaviour, leading to increased likelihood of failure and dropout? | |
| Reinforcement of discriminatory attitudes and actions | Could analytics reinforce discriminatory attitudes and actions by profiling students based on e.g. their race or gender? | |
| Reinforcement of social power differentials | Could analytics reinforce social power differentials and learners’ status relating to each other? | |
| Infantilisation | Could analytics “infantalise” students by spoon-feeding them with automated suggestions, making the learning process less demanding? | |
| Echo chambers | Could analytics create “echo chambers” where intelligent software reinforces our own attitudes or beliefs? | |
| Non-participation | Will knowledge that they are being monitored lead to non-participation by students? | |
| Stewardship | Data minimisation | Is all the data being held on an individual necessary in order to carry out the analytics? |
| Data processing location | Is the data being processed in a country permitted by the local data protection laws? | |
| Right to be forgotten | Can all data regarding an individual except that necessary for statutory purposes be deleted? | |
| Unnecessary data retention | How long should data be retained for? | |
| Unhelpful data deletion | If data is deleted does this restrict the institution’s ability to refine its models, track performance over multiple cohorts etc? | |
| Incomplete knowledge of data sources | Can an institution be sure that it knows where all personal data is held? | |
| Inappropriate data sharing | How can we prevent data being shared within or outside the institution with parties who have no legitimate interest in seeing it or may use it inappropriately? | |
| Risk of re-identification | Does anonymisation of data becomes more difficult as multiple data sources are aggregated, potentially leading to re-identification of an individual later? |
