I will be the first to state that I am not an expert in the field of conducting psychometric models, Bayesian networks, Bayesian analyses, but I have been struggling to find any blog posts about conducting a bayes net with latent variables that uses Stan. The purpose of this post is to walk through Stan and some bayes net terminology to get a basic understanding of some psychometric models conducted using Bayesian inference.
To get started, make sure you follow the detailed instructions on installing RStan or PyStan. I know if using Mac, make sure to also download Xcode so that Stan will work correctly. For posts having to do with Bayes Net, I will be doing my programming in either R or Python, while calling on Stan to conduct the Markov Chain Monte Carlo (MCMC) sampling.
For these models, I’ll be using the ECPE dataset from the edmdata package in R. This dataset has 28 items for 2,922 students with 3 latent attributes/variables. A 1 on for each item indicates whether a student got the item correct and a 0 indicates they got the item incorrect.
While I will be discussing bayes net through an educational measurement lens, bayes net can be used outside of education to show that individuals have skills that are not directly measured. Instead of items on an assessment, tasks that capture each skill can be assessed. Before walking through some bayes net terminology, it is important to note that this model is simply for educational purposes. Components of the psychometric models I will be writing about require expert opinion and domain knowledge. For example, bayes net models require expert opinions on the assignment of items to skills. Additionally, bayes net models require expert opinion on the priors for the lambda (\(\lambda\)) parameters.
Since there are different opinions on using different terms, I am going to stick to the following terms.
Node = Observed variable
Edge = Path/arrow between two nodes
Parent = The node that has an arrow/edge directed at another node
Child = The node that the arrow/edge is directed at from another node
DAG = Directed Acyclic Graph (visual representation of nodes [similar to path analysis in visualization])
Q-matrix = A matrix showing the relationship between the assessment items and each attribute/skill, with each row being an item and each column being a skill.
For this introductory post into bayes net, I thought it would be best to include some visuals for the models I’ll be testing. For both R and Python, I will be using the cmdstan version; whether it be cmdstanpy or cmdstanr. Finally, I will state that while this is introductory to a bayes net model, this post assumes that you have a basic understanding of Bayesian inference.
Below shows the first 10 rows for all the items in the dataset. Each row is a student and each cell is whether or not that student got the correct answer for each item.
Okay, now on to the Q-matrix. For the ECPE dataset, the q-matrix has already been created from expert knowledge. Another option could be to try and learn the q-matrix from a structural learning algorithm, but that would have to be for another post.
Each row in the Q-matrix corresponds to an item with each column corresponding to a latent attribute. So for the first row, the 1 values for the first and second attributes indicates that item 1 measures attribute 1 and attribute 2, but not attribute 3.
The shape of both the data (y) and the Q-matrix is important for how we will load data into Stan. The dataset has 2922 rows (number of students) and 28 columns (number of items). The Q-matrix will also be included in the list for Stan, where we’ll take the number of attributes (3) and the values for each attribute in the Q-matrix.
Attribute Profile Matrix
In addition to the student data and the Q-matrix, I also am going to create an attribute profile matrix where each row is a latent class and the columns are the attributes. These classes correspond to whether the class has mastered each attribute. For instance, class number 1 will not master any attributes. I like to think of these as profiles of mastery. For instance, the class with no attributes mastered is the class profile 000.
I am going to create this matrix by creating every possible combination of skills, which will create every potential latent class. Then I will just add each row as a numbered class. Below is the final matrix created for 3 skills.
So now we have everything to build our bayes net model. Before we get to that, I do want to visually show the models I will be creating in this series.
Models
The first model I will go over is a Log-Linear Cognitive Diagnostic Model (LCDM). This model is useful as a general diagnostic model that allows for different probabilities for each latent class. I’ll then make small changes to the model to include edges between the latent attributes in different ways.
Below are DINA models through a Bayesian Network framework. Essentially, the model assumes that each student has mastered all skills in order to correctly respond to an assessment item. See here for an excellent post about the DINA model.