Monday, September 11, 2017

Response From ABS to Marriage Law Postal Survey Questions

On 16 August I sent the ABS a list of fourteen questions regarding the conduct of the Marriage Law Postal Survey, in particular regarding count quality assurance issues.  Some of the questions were answered in subsequent public debate.  The response below was received today, September 11, from Michael Wilson of ABS and is reproduced in full.  My questions as sent are in italics.  My thanks to the ABS for their detailed responses at this busy time.

I have added some comments of my own below the responses, and may add more later.



ABS Responses To Questions
--------------------------------------------------------------------------------------------------------------------------

1. Will, as appears to be the case, survey responses be being processed and counted while the time window for submitting responses is still open?  
The survey responses will be data captured during the collection period, which is a common process for the ABS. The survey responses will not be finalised until after the 7th of November.

2. If yes to 1, what will happen if a respondent reports that their survey has been stolen (or that they did not receive a survey) after their vote has already been received and processed.  
If an eligible Australian reports that a form has been stolen, the ABS will mark the original barcode as invalid. The ABS will count the replacement form when it is returned, rather than the original form. The processing of valid and invalid forms will be conducted at the end of the collection period so there is no risk the survey return is already processed.

3. Will automatic character recognition be used to count survey responses?  
No. The ABS will use optical mark recognition not character recognition to identify Yes and No responses.  

4. If yes to 3, will there be manual verification (eg by checking images on a computer screen as done in the 2016 Senate election) that every survey response has been scanned correctly?
There will be processes for quality assuring and reviewing the automated mark recognition process to ensure confidence in the results.

5. To what extent will the observers appointed by Members of Parliament be allowed to observe the counting process in respect of each specific survey form?
These external observers will be nominated by two committees, representing each side of the debate. The nominations committees will comprise Members of the House of Representatives and Senators. The observers will be nominated by mid-September and the observation process will begin 3 October 2017, and continue through to 10 November 2017 (or earlier, should processing finish earlier).  Compared to the role of an election scrutineer two key differences are that observers will not have:
· the ability to view every survey form
· the ability to seek an administrative review of the survey form result validity.

6. Will every Member of Parliament, or failing that at least every party represented in the Parliament and every independent MP, be entitled to nominate their own observers?
An invitation to participate on the nomination committees was provided to each parliamentary party leader, as well as to every independent MP and Senator. Nomination of observers through two committees provides for a more coordinated and efficient system of arranging observers, than if every parliamentarian were to appoint people directly.

7. Will there be a process for observers to challenge any apparent errors they may observe, as scrutineers may do in an election, and to what level will it be possible to escalate such challenges?  
Yes. The role of an observer will be to witness randomly selected sample batches of survey responses of; yes responses, no responses and invalid responses (including those where the response is unclear). Observers will be able to specify if they agree or disagree with the survey result that the ABS has recorded. Where any doubt is raised these will be escalated to a further ABS officer.

8. Will clear and detailed validity guidelines on what will be interpreted as a valid survey response be posted online prior to the mailing of any survey forms?  
The website will have clear instructions on how to provide a valid survey response. Observers and ABS staff will have clear guidelines on the recognition of valid marks.

9. Will the survey form require the respondent to write a word or to tick a box?  
Respondents will only need to clearly mark one box.

10. If observers have the ability to observe counts being made before the window for survey responses has closed, will there be any prevention on them publishing, or causing to be published, comments about evidence they have gained in the process about the likely outcomes of the survey (without disclosing sensitive information in the process) and if not how will such behaviour (including via “leaks” to anonymous third parties) be prevented?  
Processes will be put in place to keep all information associated with the survey secret. Observers will examine a sample of the same number of 'yes' and 'no' survey responses to assure the correct recognition of marks, not the computer tabulation of totals. Observers will be required to sign deeds of confidentiality. The Australian Statistician will publish the statistical information on 15 November 2017 through the ABS website.

11. What will be the design of the envelope system used to return survey forms?  It has been mentioned that a scannable code will appear on the survey forms.  Aside from this what components (including outer envelopes) will contain identifying information, if any, whether coded or not?  
The return envelope will not have any identifying information or barcode on it. The survey form will have a scannable code.

12. Will there be a requirement for a respondent to sign any part of the returned material (such as a detachable flap on an internal envelope as used in Tasmanian local government elections) that can be used to determine whether a response was submitted by the voter or not?  
No.

13. Will there be any pre-defined provision for a request for a recount in the event that the outcome is especially close?  
No. The ABS has in place robust systems and controls for the processing, interpreting and publishing of statistical data and will be employing these systems and controls in relation to the Australian Marriage Law Postal Survey. The survey assurance process will include external observers representing the ‘yes’ and ‘no’ views. And the ABS will also engage an independent auditor to monitor a number of processes such as the dispatch of survey forms, the receipt and destruction of forms. The Australian Statistician will publish a statement on the quality and integrity of the survey on 15 November 2017.

14. It is stated that the Australian Statistician will publish a statement on the quality and integrity of the survey on 15 November 2017.  However this comes only eight days after the last date for receipt of responses.  It is possible – as is frequently the case with results by booth in federal elections - that concerns about the validity of some results might only be apparent once figures are released.  Will there be provision for investigating and officially responding to any quality concerns raised after 15 Nov?  In the event that errors in the final figures are identified, will there be provision for a revised result to be released?  
The ABS is confident it can assure the results of this statistical collection by 15 November.

--------------------------------------------------------------------------------------------------------------------------

My Comments

The response to question 2 is particularly pleasing as the previous information had suggested a very serious concern: that an unscrupulous person could illegally steal a survey form, vote and then have that stolen vote counted, disenfranchising the form's original owner if (as is likely) they did not quickly realise what had happened.  The ABS response indicates that the processes of data capture and data processing will be distinct in a way that prevents this from being a problem.  

However it is clear from the responses to questions 5, 7 and 10 that the opportunities for the observers to examine the process will be limited and will not even remotely approach the standards of normal election scrutineering.  This means that in the event of a close result, either overall or by electorate, it will be very difficult for observers to authoritatively answer any concerns that might be raised about the figures.  

Concerning scrutineering I also draw readers' attention to the concerns raised by Ian Brightwell in submission 2 to the Senate Standing Committee enquiry.  

My own concern regarding scrutiny of the MLPS is that it fundamentally differs from normal statistical processes such as the Census in view of the accuracy demands.  In a Census process, it is not important whether 18.10% or 18.11% of the residents of a particular suburb identify as Anglicans (for example).  But in the Marriage Law Postal Survey, the government's imposition of a 50% cutoff for same-sex marriage legislation to be brought on means that the normally insignificant difference between 49.999% and 50.001% could, in theory, in the end determine the issue.  

Even if the overall vote is not that close, there is also the likelihood that specific MPs may impose such a cutoff on an electorate or state basis in deciding their vote on a bill to legalise same-sex marriage.  In all likelihood the result won't be so close for the difference between a statistically accurate and an election-level accurate (and scrutinised as such) count to matter, but if it does, there will be trouble.

I may add other comments later.  

9 comments:

  1. Since they'll be using mark recognition machines rather than character recognition, it seems we could fill in the whole Yes* box, or put a horizontal line across it as often advised on multi-choice test forms. Still, a tick might be more satisfying. (*Or No for the change-o-phobic)

    ReplyDelete
  2. Or a nice big equals sign [=] would do it!

    ReplyDelete
  3. I've got a few questions.

    1) The ABS has said that the results will be published as Yes, No, and Response Not Clear. This opens the possibility of Yes winning with a plurality but not a majority of the vote. Would a bill then proceed on this basis or would this utterly fail to resolve the issue?

    2) Why won't the ABS exclude unclear responses to provide a cleaner Yes:No split much like the AEC does in elections?

    3) If the responses are going to be progressively counted, why won't the ABS provide indicative updates? Again, much like the AEC does for elections before the formal declaration of the poll. A lot of measures seem to be put in place to ensure secrecy so we can have some kind of grand reveal on Nov 15 - seems needless.

    Your thoughts?

    ReplyDelete
  4. I've got a few questions.

    1) The ABS has said that the results will be published as Yes, No, and Response Not Clear. This opens the possibility of Yes winning with a plurality but not a majority of the vote. Would a bill then proceed on this basis or would this utterly fail to resolve the issue?

    2) Why won't the ABS exclude unclear responses to provide a cleaner Yes:No split much like the AEC does in elections?

    3) If the responses are going to be progressively counted, why won't the ABS provide indicative updates? Again, much like the AEC does for elections before the formal declaration of the poll. A lot of measures seem to be put in place to ensure secrecy so we can have some kind of grand reveal on Nov 15 - seems needless.

    Your thoughts?

    ReplyDelete
    Replies
    1. Deepest apologies for the slow response. Re (3) first, what the AEC says in answer to Qs 1 and 2 combined makes me think that the counting (as opposed to capturing) process is only going to happen sometime between Nov 7 and 15. I also think that if the counting is only going to happen after the close of submissions then there might be some ability to release interim totals, which would certainly be a lot more fun. However not enough detail has been provided about the counting method to say if this would be the case.

      Re (1) and (2) it is unclear to me based on what the ABS has said (https://marriagesurvey.abs.gov.au/survey-results) whether the ABS will be releasing these results in percentage form, or just as counts (raw numbers) and then leaving everybody else to calculate their own percentages.

      The commentary from PM Turnbull has at no stage canvassed any result other than Yes winning or No winning, so any result with more Yes than No will be widely interpreted as a win for Yes while any result with more No than Yes will be widely interpreted as a win for No, irrespective of issues with unclear responses.

      It would then be a very brave government to try to immediately go against whatever result had been expressed and, as Dr Tad Tietze said on Twitter today, we haven't had a brave government for a long time.

      Still, in the event of a very close and (somehow) disputed Yes result, I wouldn't take anything for granted.

      Delete
    2. PS: An interesting aspect of the original Directive here is that it doesn't direct the inclusion of a "Response Not Clear" category so that must have been decided subsequently. (Also the original directive does refer to "proportion" so I suspect we will see percentages.) The general answer to why the ABS doesn't do things like an election is because they have a culture of doing "surveys".

      Delete
  5. No problem and thanks for your clear responses.

    But I have a follow-up question.

    The ABS has repeatedly said they will post the results of their 'survey' on their website at 11:30am, 15 Nov. Given the DoS attack on the Census website last year, and even the High Court's website crashing last week around the time of the HC's postal survey judgement, isn't there a real risk that the ABS website will crash - or come under attack - just before 11:30am, 15 Nov? Would it not be better for the Statistician to announce the result at a press conference?

    ReplyDelete
  6. I am not an expert on websites crashing but it would make sense for them to at least have some form of back-up release plan. Also in the lead-up to the release, the whole question of any potential hacking of the count and what manual processes could exist to detect and rectify a hack were one to somehow occur is something I would like to see more discussion of.

    ReplyDelete
  7. Going back to Dylan's previous questions - given the form of the form, and the fact that returning it is optional, you'd surely expect than less than 1% will be "response not clear". The scenario of a "plurality" but not a majority voting yes is a phantasm. But don't let me stop you from scaring yourself, if that's what you enjoy.

    ReplyDelete