We apply the method appropriate to the question. Evaluations of humanitarian action mainly use qualitative methods,[1] interviews with ‘key informants’ and personal observation.[2] This is not because qualitative methods are the most appropriate for measuring change. Qualitative methods are suited to exploring the reasons for change, explaining the array of motivations for change, or even whether people feel a change has occurred. Qualitative methods are not appropriate to confirm if a change may have occurred.
At their best, qualitative investigations are in-depth and exploratory within a focused area, so questions should be informed by quantitative findings. They cannot be wholly representative simply because numbers are always too small and the methods for involving participants are nearly always arbitrarily selective, often even self-selective (though in well applied qualitative methods this is not the case).
The tendency to rely solely upon qualitative methods in measuring impact is mainly because they can be less time-consuming and logistically less demanding. They are therefore the most convenient methods regardless of appropriateness. The position of Measured Change is that the method (qualitative or quantitative) should be chosen based on the question being asked, and should not be determined by financial or time constraints or a familiarity with the methods.
When evaluation or operational research objectives include terms such as effectiveness, efficiency or impact they must include quantitative methods. Without measurements of difference those objectives cannot be achieved. Our approach is to use qualitative and quantitative appropriately in order to answer the questions being asked, and to apply the same rigorous approach to understanding why something is happening as to whether it is happening. Mixed methods, not mixed-up methods.
________________________________
1 Buchanan-Smith M, Cosgrave J. 2013. Evaluation of Humanitarian Action. ALNAP.
2 This is often not presented as a weakness, but rather quantitative methods such as field surveys are actively discouraged for a range of reasons which include excessive cost, lack of time, lack of a sampling frame and the absence of baseline measures (a circular argument as these are derived from field surveys). These excuses are largely the consequence of poor planning, oversampling and poor knowledge of sampling techniques. The form of inquiry, it is argued, is determined by extrinsic factors, not the nature of the question being asked.
