I will be the first to state that I am not an expert in the field of conducting psychometric models, Bayesian networks, Bayesian analyses, but I have been struggling to find any blog posts about conducting a bayes net with latent variables that uses Stan. The purpose of this post is to walk through Stan and some bayes net terminology to get a basic understanding of some psychometric models conducted using Bayesian inference.
To get started, make sure you follow the detailed instructions on installing RStan. I know if using Mac, make sure to also download Xcode so that Stan will work correctly. For this post, I will be doing all my programming in R, while calling on Stan to conduct the Markov Chain Monte Carlo (MCMC) sampling. Maybe a future post will follow this tutorial using PyStan, Cmdstanpy, or PyMC, but there are just more readily available tools using R so I will be using R instead. I’m also creating some data to be used in the following posts on latent bayes nets. For these posts, I’ll be creating binary data that will represent items for an education assessment where a 1 indicates that a student has answered the item correctly and a 0 indicates they did not answer the item correctly. The model will also include three latent attributes/skills/variables where a 1 would indicate that the student has mastered the skill and a 0 would indicate that they do not have mastery of the skill.
While I will be discussing bayes net through an educational measurement lens, bayes net can be used outside of education to show that individuals have skills that are not directly measured. Instead of items on an assessment, tasks that capture each skill can be assessed. Before walking through some bayes net terminology, it is important to note that this model is simply for educational purposes. Components of the psychometric models I will be writing about require expert opinion and domain knowledge. For example, bayes net models require expert opinions on the assignment of items to skills. Additionally, bayes net models require expert opinion on the priors for the lambda (\(\lambda\)) parameters.
Since there is different opinions on using different terms, I am going to stick to the following terms.
Node = Observed variable
Edge = Path/arrow between two nodes
Parent = The node that has an arrow/edge directed at another node
Child = The node that the arrow/edge is directed at from another node
Attribute/Skill = Latent (unobserved/hidden) node
DAG = Directed Acyclic Graph (visual representation of nodes [similar to path analysis in visualization])
Q-matrix = A matrix showing the relationship between the assessment items and each attribute/skill, with each row being an item and each column being a skill.
For this introductory post into bayes net, I thought it would be best to create some artificial data and show visually the models I will be planning on creating using R and Stan. I will be using cmdstanr instead of rstan for my Stan computations. The main difference between the two packages is that rstan avoids using R6 classes, while cmdstanr uses R6 classes. If you’d like more information on trade-offs of different object-oriented programming classes, you can read more here. Finally, I will state that while this is introductory to a bayes net model, this post assumes that you have a basic understanding of Bayesian inference.
The first thing I am going to do is create a function that would create a bernoulii distribution. I decided on some random numbers for the probabilities of correct responses to the 15 different items and decided to create some fake student IDs for each row. Below is a table to look into the data if you want.
Okay, now on to the Q-matrix. As previously stated, I am creating this q-matrix to be as simple as possible. This means that in a realistic scenario, you would either want to use a structural learning algorithm to see what nodes have edges to our three latent nodes, or you should probably have experts on your latent attributes to declare what items measure what latent attribute.
Above, I created a q-matrix that follows a pattern where each attribute has 5 items that correspond to that attribute. The table above allows you to search which items correspond to each attribute by typing 1 into the filter bar above each column.
Attribute Profile Matrix
If we only wanted to examine how the posterior distributions compare to each student and their responses, then I would only need to have my student data and the Q-matrix. However, I also want to put students into latent classes. Because of this, I also have to create an attribute profile matrix. I am going to create this matrix by creating every possible combination of skills, which will create every potential latent class. Then I will just add each row as a numbered class. Below is the final matrix created for 3 skills.
Note: Latent classes are different from our latent nodes/attributes/skills. The matrix created above (alpha) is a matrix where each row is a different latent class and each column corresponds to each of the skills.
So now we have everything to build our bayes net model. Before we get to that, I do want to visually show the models I will be creating in this series.
The first model I will go over is a naive bayes model; however, naive bayes models do not correct for what I have labeled as true positive and false positive probabilities. This model also mimic a deterministic inputs, noisy “and” gate (DINA) model. Essentially, the model assumes that each student has mastered all skills in order to correctly respond to an assessment item. See here for an excellent post about the DINA model.
The second model is a bayes net model that looks very similar to the first model. Now there are edges between the three latent nodes, where depending on whether a student has the previous skill, the probability differs for having the following skill. In the next post I will be estimating the first bayes net model and doing some posterior checks to see how the model works.
