Performance measurement using trained observer ratings: The effect of environmental variables on comparative measures in local government

Comparative performance measurement is an anathema to most government officials. While internal comparison with national benchmarks or other similar agencies is an accepted management analysis tool, public comparisons are deemed too politically risky for all but a few governments, agencies, and non-profits. To avoid the unappealing prospect of being unfavorably ranked versus comparable agencies, governments cite two primary objections. First, performance measurement is considered an inexact science, subject to manipulation of input data and outcome scores that affects comparisons. Second, comparative measurement is considered unfair by governments because they maintain each operates in a unique environment of demographic and physical characteristics beyond their control which have significant effects on their ability to deliver quality services. This research addresses both of these objections to comparative performance measurement. It uses trained observer ratings, an accepted objective measurement system, and utilizes a scoring scheme that results in a ratio variable outcome score appropriate for comparisons and statistical analysis. The model was employed for external random testing of street appearance across 30 governments in Broward County, Florida, eliminating the possibility of manipulation by those being measured. In addition, the research evaluates the effects of twelve independent variables, which measure characteristics of each community's unique environment. The research question is whether the effects of the external environmental factors on the composite street rating scores are so significant as to make comparisons meaningless, as government officials contend. The findings of this research do indicate that white, educated, affluent residents tend to live in cities with better streets and that environment is an important element in street appearance. Further, the standard deviation scores indicate that the more diversity in the environment of a city, the more diversity in the quality of its streets. Nonetheless, the conclusion of the research is that although environment is significant, the methods and findings of the model can be employed to control for environmental effects in future comparative performance efforts to assuage the concerns of governments being compared.