Two topics that I like to blog about are lens models and decision trees. Today I want to offer up suggestions for how lens models might be constructed from decision trees.

Recall that a lens model looks something like this (taken from this blog):

Recall also that a fully specified decision tree looks something like this (taken from this blog):

Notice that the decision tree includes two factors: how much nitrogen to apply (100k, 160k or 240k per acre) and quality of the growing season (poor, average, good). In the context of a lens model, these might be viewed as indicators of what the yield might be at the end of growing season. In other words, if the "intangible state" we are trying judge is the amount of corn we will get at the end of a growing season, then two critical indicators are how much nitrogen is applied and what the quality of the growing season will be like (which in turn might be indicated by the amount of rain). We have control over one of those indicators (how much nitrogen to apply) but not the other (what the weather will be like). The main point I want to make here is that it is relatively easy to convert a decision tree to a lens model by making each factor in your decision tree an indicator in your lens model.

So, not only can we use multiple linear regression to specify the indicators in our lens model, but we can also use decision tree learning algorithms (PDF link) to specify the indicators in our lens models.

I don't want to get into the technical details of how decisions tree algorithms work but in general they work by recording various "features" that are associated with a target outcome you are interested in. For example, if you want to make a decision about whether a c-section will be required to deliver a baby, you can look at all the c-section births and all the non c-section births and record standardized information about all those cases. Then you start looking for the best feature that discriminates between c-section and non c-section births. That feature will likely not be a perfect discriminator so you take all the remaining cases where you used the best feature to sort cases and then use the next best feature to discriminate between cases that require c-section births and non c-section births. If you do this you come up with the decision tree shown below which can be captured more simply in an if-then rule which is also shown below:

We can construct a lens model from this tree, or from the in-then rule, where each of the three factors is an indicator in our lens model. If we use the thickness of the line connecting the judge to the indicator to represent the strength of the relationship, the first indicator would have a thicker line than the second indicator which would be thicker than the third indicator. The first indicator captures the most variance followed by the second followed by the third. This is how algorithms that generate decision trees work so when we construct lens models based on them, we should expect them to have a certain form.

The point of this blog is to show that there are several formal techniques we might use to generate a lens model. Multiple linear regression is one previously discussed technique. Today I discussed the use of decision tree algorithms as another technique. A decision tree algorithm also suggests a plausible psychological strategy for coming up with indicators; namely, pick an indicator that accounts for most of the target cases. If there are some cases it doesn't handle, pick another indicator that might filter out the more of the cases it doesn't handle, and so on. You might not have to use many indicators before you arrive at a set of indicators that captures enough of the data to satisfy you.

Multiple linear regression and decision tree algorithms are two formal techniques you can use to make the indicators used in judgement explicit and which offer up concrete approaches to thinking about how common sense, which we often find difficult to explain, might work and be improved upon. Doctors making decisions about c-sections might have relied upon common sense which included consideration of the factors studied but the formal techniques helped to identify the relevant indicators and the overall strength of the relationship between the indicators and the need for a c-section. Where multiple regression is a more wholistic/parallel method of finding indicators, decision tree learning algorithms strike me as a more analytic/sequential method of finding judgement indicators.

Below is a lecture by machine learning guru Tom Mitchell on decision tree learning that is set to start with him discussing the c-section example.