Monash Ratepayers Association has received the following material from a concerned Monash Ratepayer. It throws into question the validity of much of the Council’s Customer Satisfaction survey.
The ratepayer (Name and address provided but withheld) writes
In October each year for the past nine years Monash council has organised a questionnaire to determine how important their services are, and how satisfied randomly selected rate-payers are, with the provision of those services.
One question asked how satisfied I was with the way in which Council encouraged Industry, The survey consists of a series of questions where the respondent is asked to give ratings of “Very Low”, “Moderate”, “High” and “Very High”. There is no “I don’t know” option. I did not have any experience in many of the areas of the survey and could therefore not judge if the Council was doing a good job or not.
Commerce and Employment. I have not seen any figures of how many businesses have moved into Monash because the Council encouraged them. I don’t know what awards are given to local industry by Council, etc. My answer to this question therefore was “I don’t know” but there was no provision for this answer.
Childrens Services was another area that I could not answer because my children are grown up.
Approximately 20% of the questions were ones to which I could not give an informed answer.
The Public Amenity Section lumps together Fire Prevention, Animal Control, and Car Parking and then asks for an overall rating. These topics have nothing to do with each other and yet the respondent is asked to give an overall rating.
Letter to each Councillor sent 19th July 2004: (Name and address provided but withheld)
In October 2003 a Customer Satisfaction Survey was sent out to a number of City of Monash ratepayers. I was one of those ratepayers.
I was most concerned about the construction of the survey. The form of the construction tells the Council what it wants to know. In other words the results of the survey are known before Council receives the data back from those being surveyed. The fact that there is no “don’t know” or “not applicable” column in the survey forces the respondent to make an uninformed judgement in order to fill out the form. Most people when faced with this problem do not want to show their ignorance and will tick 3 – that is the middle option. The survey will then give a predetermined result of about 70% - that you will notice, is the approximate result of each section of your survey.
I rang the researchers (Name withheld) to ask why there was no “don’t know” column. (They) said it was not necessary to complete a section if the answer was “I don’t know” this option was not given as a possibility in the covering letter.
I rang Monash Council about my concerns relating to the survey and was told that the survey was only to look as “rate payer perception.” It was not for “service design” and not to “ measure customer needs.”
The survey does not allow for any indication where ratepayers think where money should be spent to maximize benefits to ratepayers.
The survey does not allow for ratepayers to indicate priorities.
So what does the survey measure?
I would like the following questions answered:
What is the purpose of the survey if it is not to measure the needs of the ratepayer, and how well is Council meeting these needs?
On what basis did you choose the survey pollsters.
Who was responsible for approving the format of the survey in Monash Council?
Is there a statutory requirement for such a survey?
What was the cost of running the survey?
How long has this version of the survey been running?
Are all surveys audited by State Parliament?
The ratepayer received the following response from Council:
The City of Monash is committed to undertaking an annual community satisfaction and reporting the results to Council by December each year. The survey is a valuable planning tool and provides a measure of community satisfaction with Council performance in key service areas. It also contributes to policy development and is an input into the development of the annual Council Financial and Business plans. It is also appropriate that the Council continue to monitor performance and report the outcome to the community in the interests of effective and transparent governance.
The October 2003 survey was the ninth annual survey conducted for the City of Monash by (Name withheld). The survey involved a mail out to 2400 randomly selected households in the municipality, with a total of 1000 responses received.
The information gathered from the survey informs Council of the effectiveness of its activities, communication with the community and informs both service planning and design. The survey results are considered along with other methodologies used by the Council, such as the Victorian Government’s survey of local government performance and the Council’s ‘improving Value” Program to identify continuous improvement opportunities for service delivery. Services and activities reporting lower satisfaction levels in the survey are investigated further to enable Council to respond appropriately.
(Name and address provided but withheld)
The response given by Council at the Council meeting gives no real information. A follow up letter was sent to (the ratepayer’s) home indicating that Council uses (Name withheld) because other Councils use this Company also, and comparisons can be made between Monash and other Councils.
It appears that the methodology that Monash Council uses may not be the best that could have been chosen. Just because a survey has been running for nine years and is used by other Councils does not make it right.
(We) understand from information provided by other sources, that some councils base the performance of their Chief Executive on the results of this survey. This of course can decide whether or not the CEO is entitled to additional income for good performance.
The survey is a Perception Survey and not a real Satisfaction Survey. Can we afford to run sections of our Council Research based on “perception” rather than “reality”? If we want to know what is happening in our community and we want to operate in a truly constructive manner there would appear to be better ways of doing the research.
Additional Letter From Mayor Cr Banerji, Dated 28 July 2004
(Other Councillors have not responded)
Dear (Name withheld),
Thank you for your letter received 23 July 2004, regarding Council’s Community Satisfaction Measurement Survey (CSMS). I have the following comments to make in response to the questions and issues raised in your letter.
The CSMS is a survey that seeks to obtain the community’s perception about Council services. This is achieved by asking people firstly to rate how important they think a particular service is, regardless of whether or not they use that service and secondly, to rate how well they think Council performs in that service. The survey is not designed to measure knowledge or experience with Council services but rather to measure the broader community perception of the services delivered by the Council.
The research obtained in the CSMS is utilised as only one element of Council’s planning and evaluation methodology for service delivery. Council uses a variety of survey types to measure opinions based on specific knowledge by users of particular services.
The construction of the CSMS is one that is well tested and widely used across a number of Councils by (name withheld). (It) is a member of the Association of Market Research Organisations and specialises in strategic and customised research services.
(It) has conducted the CSMS for the City of Monash for the past nine (9) years and has been selected on the basis of their expertise in conducting market research covering a wide range of satisfaction research, telephone surveys, focus groups and workshops and strategic planning. The information obtained during this time has provided the City of Monash with valuable trend data regarding the community’s perception about Council services.
Staff and Councillors at the City of Monash are consulted annually on the content and appropriateness of the survey prior to it being distributed to the community. The cost of conducting the 2003 Customer Satisfaction Survey was $15,000.
There is no statutory requirement for this survey or requirement for the CSMS to be audited by State Parliament. However, in addition to the CSMS, the City of Monash also participates in the Annual Community Satisfaction coordinated by the Department for Victorian Communities (DVC) during February and March each year. This survey is undertaken by the independent research group (name withheld) on behalf of DVC to measure how Victorian residents’ rate the performance of their local governments. A number of results from this survey, including the overall Council performance, advocacy and community engagement, are published in Councils’ Annual reports and form part of the Victorian Local Government Indicators. Those Councils who do not participate in this survey are required to conduct their own survey and report the results in their Annual Report.
I hope this information clarifies the concerns that you have raised and I thank you for your interest,
Yours sincerely,
(Original signed by Cr Joy Banerji, Mayor.)
The Australian Journal of Market Research, Vol 7, No2 July 1999 states on Page 30:
The consequence of selective perception for any applied social research is that we cannot ask a subject to make an effective response to a question about a topic to which he / she has not attended. The subject has to give the non-committal “Don’t know / Not Applicable” or he / she has to make an uninformed judgement in order to keep the researcher happy. The social researcher will then have an incomplete data set, plus data which has been collected outside the intended behavioural domain – the experience of the customer of the product or service.
Even a perception survey requires that the respondent has at least some basic knowledge of the subject matter. Otherwise their inputs are guesses or “stabs in the dark” and, as the ratepayer points out in the letter, there is a tendency for the respondent to go for the middle ground, giving erroneous data.
The MRA has been advised that the Monash survey appears to use a “Likert Scale”. Further, whilst a Likert Scale may use an “I don’t know” type of option, more often than not it does not do so. However, when it does not include an “I don’t know” option the survey questionnaire should include in its instructions a statement that it is not necessary to complete a section if the respondent has no knowledge of the subject matter. The Monash Community Satisfaction Measurement Survey does not do this. As a result the findings of the survey are likely to be flawed.
Having sections of a survey with high levels of no response is in itself useful data: indicating that many ratepayers have no knowledge or perception of the subject and that, perhaps the Council needs to be more communicative with the ratepayers and residents on some subjects. Also the “no response” data can be included or excluded from the findings analysis. The findings outcome can be quite different depending on whether it is or not.
The initial response from Council offers little clarification and does not answer the questions put by the ratepayer. What it seemingly does do is place much more importance on the survey than would be the case for a perception survey.
The Mayor’s letter in response does to some extent answer most of the questions put by the ratepayer. What it also does is put a different emphasis on the use of the survey data when compared to council’s initial response.
However, above all, whether deliberately or by oversight, the Council has failed to recognise or address the most fundamental flaw of the survey – the failure to provide the option of an “I don’t know”, or an instruction that a particular question does not have to be answered if the respondent does not have any knowledge or perception of the subject.
In her response letter, Cr Banerji refers to the survey being used to identify continuous improvement opportunities. The immediate continuous improvement opportunity, as the MRA sees it, is to improve the survey questionnaire itself. – to eliminate the flaws rightly identified by this ratepayer.
The ratepayer’s letter and Council’s responses were penned in July of this year. The 2004 questionnaire should have been circulated in October 2004 – ample time for the survey to have been rectified.
If the survey is to be used for continuous improvement of Council and its services, it needs to give factual data and not data that provides Council with want it wants to hear and feels comfortable with.
The Monash Ratepayers Association will provide the following questions to Council:
· Has Council rectified the survey questionnaire for 2004?
· If not, is it going to rectify it for future surveys?
· Will Council make publicly available:
1. The data received from the last 9 years of surveys and this year’s survey?
2. The findings and recommendations arising from the data for each of the last 9 years and this year?
If you have recently received a survey questionnaire and would like to make any comments to the Monash Ratepayers Association about it, please contact us on the e\mail or postal address shown on this website’s home page.