Many of the big conservation organizations use assessments to score protected area management effectiveness. Few of these evaluation approaches look directly at outcomes and how changes in these could be attributed to the existence of a protected area. Tools such as the Tracking Tool or RAPPAM are more assessing the management capacity of the PAs, such as adequacy of staff and budgets.
Nolte & Agrawal (2013 Cons Biol) test how well these types of measures link to the capacity of PAs to deliver real conservation outcomes. They use data from the Management Effectiveness Tracking Tool (METT) and look at how it relates to the effectiveness of PAs in reducing fire occurrence in the Amazon rainforest. They do it by applying matching methods, nowadays widely used to estimate PA effectiveness (Andam et al 2008 PNAS).
They find that protected areas were a METT assessment has been carried out are more effective in stopping forest fires. But a bit surprisingly, this does not relate to whether the protected area fell within the group of high or low METT scores. During our journal club we discussed why this might be the case. We were a bit concerned about the sample and how the authors divided it into the two groups of high or low METT score (eg. 1.33 for complete sample). This is a very low score and perhaps defining a threshold based on some objective criteria might have made more sense. The METT scores ranges from 0 to 3 and looking at the original questionnaire it seems that what I would classify as an effective area clearly would be above 2, closer to 3.
For example:
0 There are no mechanisms for controlling inappropriate land use and activities in the PA
1 Mechanisms for controlling inappropriate land use and activities in the PA exist but there are major problems in implementing them effectively
2 Mechanisms for controlling inappropriate land use and activities in the PA exist but there are some problems in effectively implementing them
3 Mechanisms for controlling inappropriate land use and activities in the PA exist and are being effectively implemented
As this grouping underlies much of the results it would have been nice to have some more justifications for setting this threshold. Looking at figure 3 it seems the METT scores were rather low (all below 2) and did not show much variation. It was interesting that the study was also done for individual PAs, and had the sample size been bigger, this is likely to have provided much more interesting results. However, these two approaches, and a rather complex methodology, made it sometimes difficult to follow the text, and some of us suggested that a diagram of the analyses would have helped the unfamiliar reader.
As the authors also do in the discussion, we also discussed how well fire worked as a proxy for habitat destruction/deforestation. Specifically, how well does it link with the indicators in the METT assessments? This could perhaps in part explain why some of the indicators traditionally believed to be crucial for effectiveness weren’t found in this study. Personally I found the results linked to institutional variables, and the inconsistency they showed, to be very interesting.
In the discussion Nolte and Agrawal raise some very interesting points about the purpose and faith of these types of PAME assessments. If, as they show in the Amazon, these do not correlate with objective measures of PA effectiveness, what are their value and how should conservation organizations then treat them? This might be very region-specific, and similar types of studies should be carried out elsewhere to test for the validity of the results in different settings. All in all, the paper was interesting and the topic is very current and generated a great discussion.
Ref.
Nolte and Agrawal 2013. Linking management effectiveness indicators to observed effects of protected areas on fire occurrence in the Amazon rainforest. Conservation Biology 27: 155-165.
DOI: 10.1111/j.1523-1739.2012.01930.x