Part IV: User Survey

Part IV: User Survey

Introduction

Any evaluation of network technologies and services will benefit from an evaluation from the users' perspective. Some academic institutions, such as Indiana University, have successfully administered user surveys assessing the networked environment on a regular basis and have a powerful data base of longitudinal information that assists in network planning and development. When used in addition to the data collection techniques and the measures outlined previously in the manual, a user survey can broaden the scope and quality of data collection.

In this manual, users are defined as all members of the authorized service community. This group includes students, faculty, staff, and others. Users of the manual should know who the users are, what their needs and expectations are, and whether those needs and expectations are being met. Users' responses to questions about how they use network services and their degree of satisfaction with those services can be analyzed as part of the evaluation process. The user survey included in this section can be administered to both known users and authorized users who do not use the campus network.

The development of a survey that fits the needs of all institutions of higher education is an impossible task. Thus, the guidelines and model survey included in this section offer an approach that users of the manual can customize and refine to fit the needs of their particular institution. The model survey offered in this section is a beginning basis from which a short and effective survey can be developed.

Designers of network user surveys may benefit by reviewing surveys that have been developed by other organizations and institutions. Increasingly, one can locate such surveys and results on various academic Web pages. Appendix D describes three national survey efforts that provide a good introduction to the types of survey questions and techniques that might be modified for use in local settings.

Purpose of Model User Survey

The primary purpose of the user survey is to obtain a profile of the people who use the network, to learn how they use it, to assess their satisfaction with it, and to find out how campus networks are affecting users. The following questions can be answered by this user survey:

  • Who are the users?

  • When do they use the network?

  • How do they access the network?

  • How do they use the network, i.e., what types of activities do they engage in while using it?

  • How satisfied are they with the network?

  • How useful do they believe the network is?

  • What do they think is successful about the network and what needs to be improved?

  • What impact is the network having on various aspects of the users' activities?

    Answers to these questions, especially when combined with other techniques described in Part II and III can provide very useful information for decision makers as they plan and develop the network.

    Issues Related to Method

    Errors, Reliability, and Validity. There is always some amount of error in data collection. Random errors are unpredictable but can be reduced by selecting a larger and more representative sample (Fink, 1995). Errors can occur if the survey does not accurately measure what it proposes to measure. The objective is to produce results which are as reliable (reproducible) and valid (accurate) as possible. In some instances specific, reliable measures can reduce the richness in meaning (validity) of general concepts. Because of this, Babbie (1992) recommends using several different approaches that will highlight different aspects of the concept under investigation.

    Bias. Survey data accuracy can be diminished by social desirability bias, i.e., the tendency of respondents to give answers to questions "in a way that conforms to dominant belief patterns among groups to which the respondent feels some identification or allegiance" (Dillman, 1978, p. 62). The surveyed users may give answers that make them look good but are not representative of what they really think or feel. One way to avoid bias is by constructing questions that are neutral and avoid asking people to think that there is a "right way" and a "wrong way" to answer (Babbie, 1992).

    Who Can Administer the Survey. This survey can be administered by a central network services office, by distributed computing departments, or by libraries, depending upon the organizational structure of networking services, the objectives for doing the survey, the budget, and the procedures and format used to administer it. Changes and modifications in specific questions included in the model survey may depend on the unit identified to be most appropriate to administer the survey.

    Choice of Survey Administration Method

    In general, there are two types of survey methods: self-administered and interviews. Each has advantages and disadvantages, but, assuming that most institutions have limited resources for conducting an evaluation, a self-administered mail survey may be the most feasible choice.

    Self-administered surveys. This type of survey uses questionnaires which the respondents complete themselves. The most commonly used medium of the questionnaire is paper, and responses are hand-written. Computerized forms of questionnaires are likely to become more prevalent and will offer several important advantages: data do not have to be re-keyed for subsequent analysis, responses may be more complete, and the response rate may be higher. Completed surveys should be anonymous unless respondents are informed in advance to the contrary.

    Self-administered surveys can be implemented in a number of ways ranging from supervised to unsupervised (Fink, 1995):

  • One-to-one supervision: An interviewer and a respondent are alone together. The administrator is available to answer questions about the survey, but confidentiality is injured. This is a very labor-intensive method and would not be cost effective for large sample groups.

  • Group supervision: Questionnaires are given to people who are together in large groups in a classroom or auditorium. The administrator is available to answer questions and monitor the completion of the survey. A disadvantage of this method is that it is difficult to organize and schedule large groups who are willing to complete a survey.

  • Semi-supervised: Questionnaires are handed to people along with verbal and written instructions, but they are not closely supervised when they complete the survey. Semisupervised surveys could be passed out to people standing in lines or when they are entering or exiting buildings. Although this kind of survey is fairly inexpensive, it is difficult to achieve a representative sample and to ensure that the surveys will be completed and returned.

  • Unsupervised: Questionnaires, along with a cover letter and instructions, are mailed to people who assume the responsibility for completing and returning the survey. This is the least expensive method of administering surveys, and can be sent to a representative sample of the user community, making the likelihood of receiving a representative response possible. Unfortunately, the administrator has no control over who responds.

    Each of the above approaches have their own strengths and weaknesses. Time and financial constraints will also affect the decision as to which approach might be better for a given situation.

    Interviews. Survey interviews are conversations; an interviewer asks a respondent a series of prepared questions. Interviewing is more expensive to conduct than a self-administered survey, but the quality of the response can be enhanced by the interviewer who can interact effectively with respondents to explain questions, probe for further information, and obtain complete responses. This method allows for the most control of responses, and does not have to rely on a person's motivation to complete and return the survey. There are two major types of interviews (Fink, 1995):

  • In-person interviews are similar to one-on-one self-administered surveys, because they are conducted with individuals and the interviewer asks questions and records the responses. Although this method has high costs for hiring, training, and deploying interviewers, it can produce better response rates.

  • Telephone interviews are conducted by interviewers who call people who are either on a predetermined list or randomly selected. Costs can still be significant as personnel and telephone facilities must be used.

    It may also be beneficial to combine these approaches and conduct some interviews in person while other interviews are conducted via the telephone. By deleting and adding questions, and creating an interview script, this mail survey could be modified for use as an interview instrument.

    Electronic Questionnaires

    The electronic questionnaire can be thought of as a logical extension of the more commonly used face-to-face and mail-in varieties. Since the survey involves the use of, and perceptions of, the electronic network, the on-line questionnaire seems a natural application. An electronic survey will be most effective when the purpose is to focus more clearly on a specific aspect of the academic network involving knowledgeable network users. There are, however, some issues to consider in using an electronic survey:

  • Respondents to an on-line questionnaire will likely be those with more than minimal networking skills and their attitudes may be biased in favor of networking. This technique should therefore be used to clarify certain "user" issues and not to address the fundamental questions of whether individuals use the network, how much they use it, or whether they like it or not.

  • The audience must be properly selected to elicit responses directed at clearly defined issues. They should be current network users, as determined through pre-screening, and should comprise a pool that will offer a range of responses.

  • The questions should be thoughtfully selected to focus on specific issues. The aim could be to focus, for example, could be on assessment of applications, data base quality, and/or training requirements.

  • The results must be carefully analyzed to detect bias or a lack of variability. Results that do not show a range of response may indicate an improperly designed study. Changes in the participant selection and questions may be required to improve the validity of the results.

    Nonetheless, use of an electronic survey for targeted users may be a more efficient means to administer the survey.

    The electronic survey can be used to elicit information about the effectiveness, efficiency, and impacts of academic networks if carefully and skillfully administered. One important application is to focus on specific issues with a carefully chosen target population. An example would be an electronic questionnaire administered to law school students to assess the perceived quality of resources on the network, as compared to in-house resources, and the ease and accessibility of those networked resources. It may be especially useful to compare results from the electronic survey to data collection using other techniques.

    Getting Ready

    Obtain approval. In order to administer a survey, approval should be obtained from the parent organization. Many college administrations require that proposals for research that will use people as subjects be reviewed in advance. The purpose of the review is to protect the welfare and confidentiality of the participants, i.e., the user survey respondents. Before proceeding with the survey, contact the appropriate office on campus. Obtaining approval may take several weeks or months, so plan accordingly.

    Determine a budget. There are costs associated with implementing a survey. Personnel time, equipment, facilities rental, printing, postage, and other costs should be anticipated and budgeted in advance of administering the survey. The costs incurred will depend upon the method of survey chosen. Self-administered surveys are the least expensive compared to in-person and telephone interviews. Questionnaires administered by mail cost about 50% less than those administered by telephone and 75% less than those administered by personal interview (Fink, 1995).

    Select a survey administration method. This is dependent on variables previously discussed, such as funding and the objectives of the surveys. Nevertheless, the approach must be determined and specific tasks to implement it should be identified.

    Inform people. It will be easier to have people cooperate with the evaluation if they are told about the survey and why it is being done. The more they know about and understand the purpose of the survey and its procedures, the more smoothly the process will unfold. The entire networked computing services staff should be informed about the survey, preferably by the person in charge of the evaluation. One benefit of disclosure is that the staff will know how to answer questions about the survey or will be ready to refer questions to the appropriate person. Second, personnel may feel threatened by the survey, fearing that, as individuals, they will be targets for evaluation. If they understand that the purpose of the survey is not to scrutinize individuals, their fears can be alleviated. Third, the staff may have suggestions about how to improve the survey process (Van House, et al., 1990). Results of the survey should be made available to respondents in a timely fashion.

    Set a timeline. All activities related to preparation and administration of the survey as well as analysis of the data collected should be plotted on a timeline. Specific deadlines for completion of each activity should be established. Key steps to consider include:

  • Step 1: clarify the survey objectives.

  • Step 2: identify the target group and select a representative sample.

  • Step 3: modify/revise the survey, pretest, and prepare final version.

  • Step 4: print and copy the survey.

  • Step 5: distribute the survey.

  • Step 3: send reminders, collect the data.

  • Step 4: analyze the data and prepare reports.

    Modify the user survey. The user survey provided in this manual will likely need some modification to fit an organization's particular needs. The survey should be used as a template and it should be customized as necessary. For example, if the institution does not have a graduate program, then question #1 can be modified to exclude "graduate student" as a choice. Questions that relate more closely to the characteristics of your user groups, networked computing's structure and services, and the goals and objectives of the evaluation may be added.

    Pretest the survey. Once the survey is modified, it is important that it is pretested with a small group of individuals similar in composition to those who will actually receive the survey. The pretest group should read the survey and, in addition to completing it, provide feedback about reading comprehension problems, typographical errors, design layout and flow difficulties. Pretesting will provide an opportunity to gauge how long the survey will take to complete and can identify other problems that may need to be considered and corrected before the final version is printed and distributed.

    Administering the Survey

    Distribute questionnaires. Ideally, all members of the authorized service community should be surveyed. It is advisable to select a random sample that is of sufficient size to minimize the standard error (i.e., the difference between sample statistics and their corresponding population parameters). In this way evaluators can be more confident that the sample is representative of the population of interest. Different groups of users, students, faculty, staff, and others, can be surveyed. Existing guidelines should be consulted for determining sample size and selecting random samples (Babbie, 1992).

    Once a target sample group is identified, and the final set of questions are prepared, it is time to administer the survey. A cover letter should be included with a self-administered mail survey that describes the importance of the survey and the need for the user to respond. Be sure to include a deadline and return address for the completed surveys.

    Collect the responses. A good to very good response rate ranges from 50-70%. Thus, it will almost always be necessary to send reminders to the targeted users. Dillman (1978) recommends sending up to three reminders over a period of seven weeks.

    Data Analysis

    Data should be analyzed in a systematic manner. First, the completed forms must be reviewed for errors and missing values. Second, the data must be encoded so that they can be converted into a computerized format. Third, the encoded data must be processed for analysis. Statistical, database, and spreadsheet software packages exist that will perform this function (Van House, 1990).

    The Importance of Longitudinal Data

    Although the survey can provide valuable baseline data after the initial distribution, the real value of the instrument comes with repeatedly surveying the user community. Repeated surveying and longitudinal analysis will allow administrators to track their progress in improving services and correcting problems. It will also identify changes in the expectations and behaviors of the user community.

    Customizing the Survey

    The user survey offered on the following pages is a menu of possible questions and topics for data collection. Users of this manual will want to select topics and questions from the survey to obtain information of particular interest, or obtain that information from specific target groups. In addition, it may be necessary to customize the wording of specific questions to match terminology or phrases in use at a particular campus. Do not administer this survey without reducing its overall length and customizing it for your particular campus.