Methodology
Methodologically, the paper draws on the case study approach favoured in Science and Technology
Studies (STS) where particular attention is paid to epistemic ‘controversies’. These controversies take
the form of disagreements and debates that illustrate how social aspects influence the otherwise see-
mingly ‘objective’ process of knowledge production in science and engineering domains (Law 2016).
In this paper, we treat the academic debate around the DKT case as a minor controversy that can
illuminate the sociotechnical factors involved in the production of knowledge about data science and
AI techniques in education. In particular, our paper presents an interpretative analysis of the
disciplinary debate itself. This approach therefore follows the work of Kelty and Landecker (2009)
by focusing on a corpus of formal knowledge analysed as an ethnographic informant: ‘something to
be observed and engaged as something alive with concepts and practices not necessarily visible
through the lens of single actors’ (177). In terms of conventional educational research, this consti-
tutes a relatively experimental and unconventional method. However, we argue that complex
phenomena like ‘algorithmic education’ can only be studied by focusing on their digital and episte-
mic manifestations, and (it follows) through a pragmatic yet careful use of multiple methods that
stretch well beyond a traditional reliance on interviews and other qualitative self-report methods.
In the remainder of the paper, we therefore take the eight articles that comprise the DKT contro-
versy to develop an ethnographic understanding of the following three elements of a ‘relational
framework’:
(1)
The educational data-set and broader digital ‘learning’ platform. The first focus is on one of the
specific ‘educational’ datasets involved in the DKT controversy. As mentioned earlier, the DKT
debate considered six datasets in total. In this paper, we focus on the most prevalent dataset in
the controversy that featured in six of the eight articles. This dataset is from a US-based Intel-
ligent Tutoring system called ASSISTments, designed to teach (mostly) the topic of algebra. For
this element of our analysis we examine the technical documentation relating to ASSISTments
and treat the actual dataset as a digital-ethnographic artefact.
(2)
The AI method. Second, we focus on the specific machine learning method implicit in the DKT
debate: recurrent neural networks (RNNs). Here we attempt an interpretative reading of the
papers, looking beyond their face value as ‘objective’ empirical reports. The interpretation
focuses on tension between the ‘new knowledge’ that RNNs tried to discover in the data, and
the ‘existing knowledge’ codified in the data environment as a result of an epistemic consensus
amongst the educationalists that created it (i.e., pedagogic consensus about how the topic of
algebra is learnt, and a pedagogic consensus about learning progression).
(3)
The cultural, discursive and economic aspects of data science in education. Third, we examine the
DKT debate as a specific instance of epistemic discourse. In particular, this involves analysing
the patterning of a specific learning-related keyword (‘performance’) across the papers as indica-
tive of problematic cultural assumptions. We then place this discursive contestation in the con-
text of competitive relations that the DKT studies mediated between universities, corporate
entities and the six digital educational datasets.
Weak AI: hype, backlash and the complexities of ‘learning from data’
Before examining each of these three elements, it is important to develop a good working under-
standing of the AI method that is under scrutiny here. In particular, we outline the specific machine
learning method implicit in the DKT debate: recurrent neural networks (RNNs), which is one of
many methods that can be used in AI. As shall be clear when we go on to consider the three
elements, understanding the logic of this method and the way it differs from other machine learning
approaches is an important pre-requisite to making sense of the DKT controversy.
click here to read -- paper
click here to watch making of B-AIM: