mmpf : Monte-Carlo Methods for Prediction Functions

Machine learning methods can often learn high-dimensional functions which generalize well but are not human interpretable. The mmpf package marginalizes prediction functions using Monte-Carlo methods, allowing users to investigate the behavior of these learned functions, as on a lower dimensional subset of input features: partial dependence and variations thereof. This makes machine learning methods more useful in situations where accurate prediction is not the only goal, such as in the social sciences where linear models are commonly used because of their interpretability. Many methods for estimating prediction functions produce estimated functions which are not directly human-interpretable because of their complexity: for example, they may include highdimensional interactions and/or complex nonlinearities. While a learning method’s capacity to automatically learn interactions and nonlinearities is attractive when the goal is prediction, there are many cases where users want good predictions and the ability to understand how predictions depend on the features. mmpf implements general methods for interpreting prediction functions using Monte-Carlo methods. These methods allow any function which generates predictions to be be interpreted. mmpf is currently used in other packages for machine learning like edarf and mlr (Jones and Linder, 2016; Bischl et al., 2016). Marginalizing Prediction Functions The core function of mmpf, marginalPrediction, allows marginalization of a prediction function so that it depends on a subset of the features. Say the matrix of features X is partitioned into two subsets, Xu and X−u, where the former is of primary interest. A prediction function f (which in the regression case maps X→ y) where y is a real-valued vector might not be additive or linear in the columns of Xu, making f difficult to interpret directly. To obtain the marginal relationship between Xu and f we could marginalize the joint distribution so that we obtain a function fu which only depends on the relevant subset of the features: fu(Xu) = ∫ f (Xu, X−u)P(Xu|X−u)P(X−u)dX−u. This however, can distort the relationship between Xu and f because of the inclusion of dependence between Xu and X−u (specifically Xu|X−u), which is unrelated to f . An alternative is to instead integrate against the marginal distribution of X−u as in 2.1, as suggested by (Friedman, 2001): f̃u(Xu) = ∫ f (Xu, X−u)P(X−u)dX−u. To illustrate this point, suppose data are generated from an additive model, f (·) = x1 + x2 and (x1, x2) ∼ MVN(0, Σ) where the diagonal entries of Σ are 1 and the off-diagonals are 0.5. That is, (x1, x2) are correlated by construction. Now if we want to understand how f depends on x1 we could integrate against the true joint distribution, as in 2.1. However, this distorts the relationship between x1 and f , because the conditional probability of x1 given x2 is higher at values of (x1, x2) which are more extreme (due to their correlation). Since x2 is related to f , this has the effect of distorting the relationship between x1 and f , and, in this case, makes the relationship appear more extreme than it is, as can be seen in the left panel of Figure 1. This distortion of the relationship between x1 and f can be made more misleading if x2 interacts with x1 to produce f , or if x2 has a nonlinear relationship with f , as can be seen in the right panel of Figure 1. Integrating against the marginal distribution of x1 recovers the true additive effect (left panel) and the average marginal effect (x1 + 0.5x1x̄2, in the right panel) respectively.


Marginalizing Prediction Functions
The core function of mmpf, marginalPrediction, allows marginalization of a prediction function so that it depends on a subset of the features.Say the matrix of features X is partitioned into two subsets, X u and X −u , where the former is of primary interest.A prediction function f (which in the regression case maps X → y) where y is a real-valued vector might not be additive or linear in the columns of X u , making f difficult to interpret directly.To obtain the marginal relationship between X u and f we could marginalize the joint distribution so that we obtain a function f u which only depends on the relevant subset of the features: This however, can distort the relationship between X u and f because of the inclusion of dependence between X u and X −u (specifically X u |X −u ), which is unrelated to f .An alternative is to instead integrate against the marginal distribution of X −u as in 2.1, as suggested by (Friedman, 2001): To illustrate this point, suppose data are generated from an additive model, f (•) = x 1 + x 2 and (x 1 , x 2 ) ∼ MVN(0, Σ) where the diagonal entries of Σ are 1 and the off-diagonals are 0.5.That is, (x 1 , x 2 ) are correlated by construction.Now if we want to understand how f depends on x 1 we could integrate against the true joint distribution, as in 2.1.However, this distorts the relationship between x 1 and f , because the conditional probability of x 1 given x 2 is higher at values of (x 1 , x 2 ) which are more extreme (due to their correlation).Since x 2 is related to f , this has the effect of distorting the relationship between x 1 and f , and, in this case, makes the relationship appear more extreme than it is, as can be seen in the left panel of Figure 1.This distortion of the relationship between x 1 and f can be made more misleading if x 2 interacts with x 1 to produce f , or if x 2 has a nonlinear relationship with f , as can be seen in the right panel of Figure 1.
Integrating against the marginal distribution of x 1 recovers the true additive effect (left panel) and the average marginal effect (x 1 + 0.5x 1 x2 , in the right panel) respectively.

Using mmpf
In practical settings we do not know P(X) or f .We can use f , estimated from (X, y) as a plug-in estimator for f and can estimate P(X −u ) from the training data, allowing us to estimate the partial dependence of X u on f Friedman (2001).
The R Journal Vol.10/1, July 2018 ISSN 2073-4859 This the behavior of the prediction function at a vector or matrix of values for X u , averaged over the empirical marginal distribution of X −u .
The function marginalPrediction allows users to compute easily partial dependence and many variations thereof.The key arguments of marginalPrediction are the prediction function (predict.fun),the training data (data), the names of the columns of the training data which are of interest (vars), the number of points to use in the grid for X u , and the number of points to sample from X −u (n, an integer vector of length 2).Additional arguments control how the grid is constructed (e.g., uniform sampling, user chosen values, non-uniform sampling), indicate the use of weights, and instruct how aggregation is done (e.g., deviations from partial dependence).Below is an example using the Iris data (Anderson, 1936): library(mmpf) library(randomForest) data(iris) iris.features= iris[, -ncol(iris)] # exclude the species column fit = randomForest(iris.features,iris$Species) mp = marginalPrediction(data = iris.features,vars = "Petal.Width", n = c(10, nrow(iris)), model = fit, uniform = TRUE, predict.fun= function(object, newdata) predict(object, newdata, type = "prob")) print(mp) ## Petal.Width setosa versicolor virginica ## 1: 0.1000000 0.6374133 0.2337733 0.1288133 ## 2: 0.3666667 0.6374133 0.2337733 0.1288133 ## 3: 0.6333333 0.6356267 0.2350533 0.1293200 ## 4: 0.9000000 0.1707200 0.5997333 0.2295467 ## 5: 1.1666667 0.1688267 0.6016267 0.2295467 ## 6: 1.4333333 0.1688133 0.5880800 0.2431067 ## 7: 1.7000000 0.1640400 0.4242800 0.4116800 ## 8: 1.9666667 0.1619867 0.2066667 0.6313467 ## 9: 2.2333333 0.1619867 0.2047867 0.6332267 ## 10: 2.5000000 0.1619867 0.2047867 0.6332267 In this case, f returns a probability of membership in each class for each value of the variable The R Journal Vol.10/1, July 2018 ISSN 2073-4859 In fact, any function of the marginalized function fu can be computed, including vector-valued functions.For example the expectation and variance of fu can be simultaneously computed, the results of which are shown in Figures 3 and 4. Computing the variance of fu can be used for detecting interactions between X u and X −u (Goldstein et al., 2015).If the variance of fu (X u ) is constant then this indicates that X −u does not interact with X u , since, if it did, this would make f more variable in regions of the joint distribution wherein there is interaction between X u and X −u .

Figure 1 :
Figure1: The marginal relationship between x 1 and f as estimated by integrating against the marginal distribution of x 2 (the blue line) or the joint distribution of (x 1 , x 2 ) (the red line).The true relationship is shown by the dashed line.In the left panel f is an additive function of x 1 and x 2 and in the right panel x 1 and x 2 interact via multiplication to produce f .

Figure 2 :
Figure 2: The expected value of f estimated by a random forest and then marginalized by Monte-Carlo integration to depend only on "Petal.Width."

Figure 3 :Figure 4 :
Figure 3: The expected value of f estimated by a random forest and then marginalized by Monte-Carlo integration to depend only on "Petal.Width" and "Petal.Length."