Build the data collection instruments

This sub-process describes the activities to build the collection instruments to be used during the phase 4 (Collect). The collection instrument is generated or built based on the design specifications created during phase 2 (Design). A collection may use one or more modes to receive the data, e.g. personal or telephone interviews; paper, electronic or web questionnaires; SDMX hubs. Collection instruments may also be data extraction routines used to gather data from existing statistical or administrative data sets. This sub-process also includes preparing and testing the contents and functioning of that instrument (e.g. testing the questions in a questionnaire). It is recommended to consider the direct connection of collection instruments to the statistical metadata system, so that metadata can be more easily captured in the collection phase. Connection of metadata and data at the point of capture can save work in later phases. Capturing the metrics of data collection (paradata) is also an important consideration in this sub-process.

-------------------

The content of this page was adapted from Household Sample Surveys in Developing and Transition Countries, United Nations Statistics Division, 2005 (see in particular Chapter III, An overview of questionnaire design for household surveys in developing countries, by Paul Glewwe)

A five-step process will lead to the production of a questionnaire formalizing the best compromise between the constraints facing data producers and the objectives of the surveys.

Step 1: Formulating clear objectives

Data that do not serve clearly defined objectives should not be collected. These objectives must be ranked by importance. Technical and financial constraints limit the scope and coverage of the survey. With a large number of objectives, it may not be possible to collect all the information needed. Low priority objectives or those that can be met using existing data from other sources should then be dropped.

Step 2: Establish a questionnaire design team

A team of experts and stakeholders must be formed at the very beginning of the process to ensure that all necessary expertise is available and that no aspect of the survey is neglected. The team should have representatives from several key groups: policy makers, researchers and data analysts, and data collectors (interviewers, supervisors, data entry staff and computer technicians).

Step 3: Assessing constraints

The financial resources available will limit both how many households can be surveyed and how many questions can be asked of a given household. Large sample sizes or highly detailed household questionnaires may exceed the capacity of the implementing organization to undertake the survey at the desired quality level. This constraint is often underestimated. As a result, many surveys have produced large data sets of questionable quality, and thus of limited usefulness.

The willingness and ability of the respondents to provide the desired information is another limitation. Extremely long survey questionnaires and highly sensitive questions will likely result in high rates of refusal and/or data that are incomplete or inaccurate. Even when respondents are cooperative, they may not be able to answer complex questions or questions requiring them to recall events that occurred months or years ago.

Step 4: Drafting the questionnaire using a module approach

A household survey questionnaire is usually composed of modules. A module consists of a series of questions used to collect information on a particular topic such as housing, employment or health. The design of the questionnaire can then be broken down into two steps: (i) deciding which modules are necessary and the order the modules should follow, and (ii) Design each module, question by question.

The order of the modules should match the order in which the interview will be conducted, so that the interviewer can complete the questionnaire by starting with the first page and then continuing on, page by page, until the end of the questionnaire. Exceptions may be necessary in some cases, but overall, it is "natural" for the modules to be ordered in this way.

The first modules in the questionnaire should consist of questions that are relatively easy to answer and that pertain to topics that are not sensitive. By doing so, the interviewer will be able to put the household members at ease and develop a rapport with them.

A third principle is to group modules that are likely to be answered by the same household member. This allows the person to answer all the questions of these modules and then end his or her participation, letting other household members answer the remaining modules.

Once the modules have been selected, and their order determined, the detailed task of choosing the specific questions and writing them out, word for word, must be performed.

When carrying out this work in a given country, it is useful to begin by reviewing past household surveys on the same topic that have been conducted in that country, perhaps in other countries (see the IHSN collection of questionnaires). It is useful also to review guidelines available from international survey programs, if available for the topic. Over-time (and in some time cross-country) comparability should be maintained to the extent possible. For key socio-economic indicators, the formulation of the question, the modalities, and the instructions to interviewers should be identical across national surveys so that various surveys can produce comparable indicators. National statistical agencies should maintain a set of standard questions, classifications, and concepts to be used in all national surveys (what could be qualified as a "national question bank"). See our section on harmonization of survey methods for a more detailed justification of the importance of consistency across data sources, and of the importance of metadata management for achieving this objective.

The questions should be written out on the questionnaire so that the interviewer can conduct the interview by reading each question from the questionnaire. This ensures that the same questions are asked of all households. The advantage of writing out all questions was clearly demonstrated in an experimental study by Scott and others (1988): questions that had not been written out in detail produced 7 to 20 times more errors than did questions that had been written out in detail.

Reference: Scott, Christopher, and others (1988). Verbatim questionnaires versus field translations or schedules: an experimental study. International Statistical Review, vol. 56, No. 3, pp 259-78.

Questions must be kept as short and simple as possible, using common, everyday terms. In addition, all questions should be checked carefully to ensure that they are not leading or otherwise likely to induce the respondent to give biased responses. If the question is complicated, it should be broken down into two or more separate questions.

The questionnaire should be designed so that the answers to almost all questions are pre-coded. The coding scheme for answers should be consistent across questions.

The questionnaire should include "skip codes" which indicate which questions are not to be asked of the household, based on the answers to previous questions.

Step 5. Field-testing and finalizing the questionnaire

No household survey questionnaire should be finalized without being tried out on a small number of households to check for problems in the questionnaire design. Field-testing the draft questionnaire can be divided into two stages. The first stage, called pre-testing, involves trying out selected sections (modules) of the questionnaire on a small number of households (for example, 10-15), to obtain an approximate idea of how well the draft questionnaire pages work. This can be done more than once, starting in the early stages of the questionnaire design process. The second stage is a comprehensive field test of a draft questionnaire. It is often referred to as the pilot test. This is a larger operation, typically involving 100-200 households.

Another key point is that in countries where more than one language is spoken, the questionnaire should be translated into all major languages and the pilot test should be carried out in those languages. A useful reference for questionnaire translation is Harkness, Van de Vijver and Mohler (2003).

Reference: Harkness, Janet A., Fons J.R. Van de Vijver and Peter Mohler (2003). Cross-Cultural Survey Methods. New York: Wiley

The pilot test should test not only the draft questionnaire but also the entire fieldwork plan, including supervision methods, data entry, and written materials such as interviewer manuals. Only by testing the entire process can the team be assured that the survey is ready for implementation. A useful last step is to undertake a quick analysis of the data collected in the pilot test to check for problems that may otherwise be overlooked.

For a more detailed treatment see:

Grosh, Margaret, and Paul Glewwe, eds. (2000). Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. New York: Oxford University Press (for World Bank).

Converse, Jean M., and Stanley Presser (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, California: Sage Publications.