Open aka Incomplete Definition Rules - simple version
Let’s compare a Rule about the influence of some Variables on a downstream Variable …
Downstream Variable, independently measurable Upstream Variable(s), independently measurable
… with a Definition Rule:
!Rule Downstream Variable, not independently measurable Upstream Variable(s), independently measurable
We could say that in the first case, we have a Rule telling us what to expect the Level of the downstream Variable to be, but we can also measure (or otherwise assess or judge) it, so we can compare the two.
In the second case, the Variable is defined by the Definition Rule, so all we need is the Rule, there is nothing to check or compare:
!Rule Sum of the two Variables ((integer)) Variable A ((integer)) Variable B ((integer))
Or is there?
What do we do with something like this?
!Rule Tolerance increases in the whole country (some rule) Tolerance increases in North Region
The above, rather crass, example seems pretty obviously flawed. On the one hand there is obviously some kind of definitional link between the two Variables, on the other hand, the link is not complete.
In the above example, there is certainly no causal relationship but a kind of incomplete Definition. One of the most tricky challenges in evaluation is when we are faced with partially or incompletely defined Variables27.
In real life this kind of tricky underlying logic (or lack of logic) is more likely to be linguistically disguised and harder to spot, as in this example
Communities in Nepal embrace tolerance (Defined Variable??) Inter-religious tolerance improved in seven districts
We liberals like to think that open = good, exciting and closed = bad, boring. But there is a lot to be said for a complete Definition and a lot of disadvantages with incomplete ones.
From the confusing cases discussed above, there is one special case when the upstream Variable(s) can be considered as part of the downstream Variable, for example by representing part of a broader concept and/or some part of a larger number of units. So we can treat the downstream Variable as incompletely defined by the upstream Variable(s).
Presumably more defining Variables would be needed to completely define the downstream Variable, and these are not in the picture.
!Rule Communities in Nepal embrace peace and understanding partial (?) Inter-religious tolerance improved in three Northern districts Inter-religious tolerance improved in two Southern districts
Conceptual parts: “falls under” or “is an example of”
Very often we see this kind of relationship at the downstream end of a Logframe, at what are usually called, suggestively “the highest levels of the Logframe”. These kinds of structures are often more about positioning actual project outcomes (like “Inter-religious tolerance improved in seven districts”) in relation to a donor’s strategic aims (like “Communities in Nepal embrace tolerance”).
This problem can be compounded when the language used is vague and doesn’t really correspond between the upstream and downstream Variables, as in the next example.
!Rule (?) Communities in Nepal embrace peace and understanding Inter-religious tolerance improved in seven districts
Is inter-religious tolerance in seven districts part of what we mean by “peace and understanding” in the whole country? In a way yes, because if there was appallingly low tolerance in those Districts, we wouldn’t assent to “Communities in Nepal embrace peace and understanding”.
Let’s look at this example. An NGO, proTolerance, has long been conducting workshops for mainstream children on tolerance towards Roma children. They are always looking for funding. They see a call for tenders on an Embassy’s webpage under the heading “Increasing tolerance amongst children” and they submit a logframe which looks a bit like this:
!Rule (?) Children are more tolerant ((lo-hi)) Children are more tolerant towards Roma people ((lo-hi)) Children participate in our tolerance workshops direction=TB
Now as we well know, children becoming more tolerant towards Roma people doesn’t necessarily cause them to be more tolerant in general. By all means, there might be some causal Mechanism such that when children have learned to be tolerant towards one group, this generalises and they become tolerant to all. But the opposite is possible too. No, we most often get this kind of logframe when an organisation is trying to show that its project goal (children are more tolerant to Roma) falls under a more general heading - and therefore, perhaps, is eligible for funding under that heading.
Both of these are perfectly good Variables. The Embassy has been dealing with its own particular goal for a decade, and proTolerance likewise. But it is enough just to understand the meaning of the words to see there is some definitional relationship between these two goals.
Professional duty of interpretation
Looking at a previous example again:
!Rule (add?) Child's overall well-being Child's social well-being Child's emotional well-being
So child’s overall well-being here just is defined as a combination of these two areas. So well-being is improving if and only if the project is improving these two things.
But suppose an evaluator comes and looks at the project and the Logframe and says, well I can’t sign off on this way of understanding the Logframe; too many of the children are unfit and overweight or obese. They might seem quite well-adjusted in many ways but it just doesn’t make sense to say a child who is struggling with fitness can have high well-being. In other words, the evaluator is saying that even though the concept might be literally defined as merely the combination of two other concepts, the use of ordinary-language words to name (overall well-being) the overall concept is certainly going to attract the attention of donors, lay people and other stakeholders and they are going to understand these words in a certain way.
Of course, others might not agree. The evaluator is then more in the position of a judge assembling arguments than a scientist assembling data.
Evaluators have a moral and/or professional duty to ensure that, the Labels of defined Variables are reasonable and do not, for example, promise more than they can really deliver.
The evaluator might for example prefer this:
!Rule:complete Child's well-being Child's social well-being Child's emotional well-being Child's physical well-being
Alternatively, the evaluator might annotate the relevant part of the original Theory like this:
!Rule:incomplete Child's well-being Child's social well-being Child's emotional well-being
In either case, whether or not the project logical framework or Theory of Change is actually amended, there is good reason for a final or mid-term evaluation to address in particular parts of the meaning of the overall concept which are perhaps not covered by the two Variables suggested by existing project documentation.
This is related to the problem, familiar to those constructing and using logical frameworks, of “choosing indicators”, see xx. But the problem of incompletely defined Variables described here is much more general than that particular case. Here we are not primarily concerned with measurement but with concepts.
Convention: assume definitions are total
If not otherwise specified, assume a defined Variable is totally defined by its defining Variables.
!Rule Child's well-being Child's social wellbeing Child's emotional wellbeing Child's physical wellbeing
… is equivalent to this:
!Rule:complete Child's well-being Child's social wellbeing Child's emotional wellbeing Child's physical wellbeing
Notice this is different from the convention for the influence of Influence Variables on Consequence Variables, which we assume to be incomplete unless otherwise stated.
This is precisely parallel to the chapter above in which we discussed Consequences which were incompletely determined by their Influences↩