Doing Survey Research: A Guide to
Quantitative Methods
2006, Allyn & Bacon Publishers
Suggested Answers to Exercises
© Peter M. Nardi
Note: Many of these exercises
have multiple correct answers and these are just some of the possible ones.
CHAPTER 1:
Interpret: What Do These Real Examples Tell Us?
1a. Faulty reasoning and cause-effect: It assumes that the teacher training is the source of the problems in public schools today. While it might be a factor, alternative causal explanations (such as per pupil expenditures, quality of the school building, social class background of the children, availability of classroom resources like computers and textbooks, and salaries) might be stronger explanations of the “bad education public school kids” get. Also, incorrect generalization: not all public school education is bad and perhaps the media stories are focused more on the problem schools than the good ones.
1b. Another untested cause and effect (If…then) relationship: It assumes that requiring national standardized tests will result in teachers emphasizing the test content and preparing students to pass tests. This logic may be accurate, as some studies are beginning to suggest, but what is the evidence for this? How might it be verified.
2a. Since the goal is to understand “why” variations in drug use occur, the study is seeking explanations and can be considered an “explanatory” study.
2b. The excerpt illustrates “descriptive” research since it focuses on providing the demographic characteristics of those who are caregivers. It does not tell us why they are caregivers.
2c. This finding is primarily an example of explanatory research since it tell us the reasons why some people are less likely to fear crime and perceive themselves more vulnerable to assaults. The “cause” is how well integrated they are in the community. Although the findings describe those who are less fearful as being happy with neighbors and more satisfied with their city, it does not tell us the demographic characteristics of them, that is, male or female, race/ethnicity, social class, or age, to name a few.
Consult: What could be done?
1. This is faulty cause and effect reasoning. It implies that moving into the dorm will cause you to get higher grades.
2. It does not take into account alternative explanations or other plausible causes. Research would tell us what other factors are involved, such as describing who actually lives there. What if more women are in the dorm and on this campus, women tend to have higher GPAs and better study skills? What if there is a racial/ethnic factor to explain the grades or how well the students bond together in a sense of community?
3. Perhaps the cause-effect direction is the other way: the dorm attracts the best students. We need to do some research on this and not just base it on rumor.
Decide: What Do You Do Next?
1. Perhaps you heard that those who watch more TV tend to believe in stereotypes more. We know from research that heavy TV viewers for example believe they will more likely be a victim of stranger violence even though more people are victims of violence from people they know. Everyday thinking problems of overgeneralizing, faulty samples, and incorrect cause and effect relationships could result in racist conclusions.
2. (a) Explore: carry on some focus groups simply asking people what images they get from TV of various ethnic/racial groups or conduct some content analysis of the highest rated prime time TV shows. (b) Design a questionnaire to find out which kinds of people are more likely to believe the stereotypes and media depictions: men or women, higher educated or less educated, people of color or whites, etc., and find out what their opinions are. (c) Interview heavy and light TV viewers and compare the results of their opinions toward different groups and see if there is a relationship between the number of hours watching TV and their attitudes.
3. You would need to rule out other possible reasons that might explain their opinions, for example their own race/ethnicity and educational levels, and show that their attitudes were there before they watch the shows.
CHAPTER 2
Interpret: What Do These Real Examples Tell Us?
1a. The authors use such ideas as strain theory, cultural deviance theory, and social disorganization theory to generate ideas for investigating the links between immigratin and crime. Note that the theories are not about immigration but focus on the origins of crime.
1b. Deductive since the theories are being used to generate ideas for a study on immigration and crime.
1c. For example, using Sutherland’s cultural deviance theory, if the immigrant group has distinctive cultural traditions that promote or accept criminal behavior, then we can expect a higher crime rate.
1d. For example, using strain theory, we could ask what are the blocked socioeconomic opportunities faced by immigrants, such as access to education and good schools, discrimination in the job market, or getting lower wages at work.
1e. What is the impact on the immigrants of asking questions about their attitudes toward crime? What if they are illegal immigrants, how can confidentiality be assured? How confidential can we be when asking people about criminal behavior? What privacy issues are at stake if seeking public records or police files?
2a. Anonymity is assured, use of opaque envelope demonstrates that the answers from individuals cannot be seen or read by others and thereby linked to a particular person, people not connected to the particular school are proctoring the survey so confidentiality is further emphasized since they don’t know the students personally, permission from parents was sought for their children’s participation, names were not recorded again assuring anonymity, respondents were told they could skip questions and thus not forced to participate either in the study or parts of it.
2b. Other ethical concerns: how will the findings be used, given the sexual nature of the survey? How will it be handled for those students who were denied permission by parents to participate, especially if most do complete the survey?
2c. The absence of a signed form could possibly mean that the parents never even saw the form, and never had the chance to deny permission. But asking to deny permission probably results in more students participating. On the other hand, having parents sign to give permission could result in forged permission slips, or forms never returned on time and thus result in fewer participants.
Consult: What could be done?
1. This is a
sensitive topic that could be raising issues in the respondents yet to be
worked out. Confidentiality needs to be assured and
2. Is this really voluntary? How safe are they from physical or mental harm?
3. Is this really voluntary when extra credit is offered by a teacher?
4. It would be unethical if respondents did not know what the topic of the survey was and did not have the option of opting out or skipping questions. Otherwise, these are legitimate topics for research.
5. Has permission
been granted by parents or guardians for these minors? Given the topic of drug
use, is there
6. Is there physical
harm in giving people high fat foods, especially if they have not been screened
for previous health concerns? Are they participating with
7. Nothing unethical
unless control group subjects were not debriefed about the placebo. Some people
feel that any experiment involving a group of subjects not receiving the treatment
is deceptive and should sto
Decide: What Do You Do Next?
1. Communications and media journals, sociology and psychology databases, ethnicity/race sources
2. Television and race and attitudes; media images and race; stereotypes and television
3. Students should search for media web pages and studies on racial images,
4. Different
racial/ethnic categories, types of TV shows monitored (news, sitcoms, dramas,
TV movies), portrayal of characters (criminals, leaders, power roles, targets
of violence or jokes, lead characters or secondary roles), attitudes towards
minorities (favor equal treatment, re
CHAPTER 3:
Interpret: What Do These Real Examples Tell Us?
1. Age: interval/ratio
Ethnicity: dichotomous nominal measure, could be used as dummy ordinal or interval/ratio measure
Education: if years of schooling expressed in grades (such as tenth grade, or eight grade), then interval/ratio. If expressed as “some high school,” “high school graduate,” “some college,” “college graduate,” and so on, then it’s an ordinal measure.
Income: ordinal categories
Religious: dichotomous nominal measure, could be used as dummy ordinal or interval/ratio measure
2a. one-directional; independent variable is age of organization, dependent variable is likelihood of adopting technology.
2b. one-directional; independent is number of personnel with technical backgrounds, dependent is when (perhaps number of years ago) they adopted technology.
2c. two-directional, independent is size and complexity of organizations, dependent is usage of technology.
3. Criterion validity of the concurrent type was established by using an external measure to verify the accuracy of the results from the LPT, namely the Barnell-Loft Series of reading. It is valid for the at-risk groups listed. Reliability was demonstrated presumably by giving 46 students the LPT at two different times. The results were compared and the result was close to the perfect relationship of 1.0.
Consult: What could be done?
1. First they need to develop some research questions or hypotheses and construct a research design. They need to make a list of variables to measure the concepts: plans to go to college, grades, and self-esteem. How will they operationalize the variables?
2. The school
administrators need to consider the reliability and validity of the measures
they will develop, and find out what they are for various standardized
self-esteem scales. Use the Internet or a resource book in the library to find
scales and compare the reliability and validity of the available self-esteem
scales, such as the
3. You need to advise
the administrators about the ethics of studying minors who need permission of
parents to participate in the research. It is also important to consider what
the impact would be on them if asked questions about self-esteem and what you
will say to allow them
CHAPTER 4
Interpret: What Do These Real Examples Tell Us?
1a. Good that it specifies a time frame (“during the past week”), but question implies that one was read (“which newspaper was that?”). Should prompt the interviewer to skip the second part if respondent says “none.” A self-administered survey would have to indicate some branching either visually or in words.
1b. Like the previous one, some prompts should be given to the interviewer if respondent says “no” to seeing or hearing a show. “Political or social issues” might need to be defined with an example or two. A self-administered survey would also have to indicate some branching.
1c. Might be necessary to specify that you are asking about a political party, or who they voted for in the last election, or if they are actually registered with a party. But
it does say “generally speaking” so it should be clear that this is about an identity. On a self-administered survey, it should be made specific since there is no one to ask to clarify.
1d. Some might see this as a double-barreled item in that they could consider government and public affairs as two separate things. A self-administered questionnaire would use a Likert scale format for the choices (“most of the time” = 1, etc.).
2a. If living with both their own mother and father, the interviewer must skip to question 8 about what kind of work the father did, otherwise ask item A for the remaining answers to find out what happened to parents.
2b. closed-ended items are also provided in Item A, and for 8 E. Open-ended items are asked about occupation and place of work (Items for 8).
2c. Could be confusing, especially the box (IQ-3 Interviewer) with further instructions to check back on Q. 7.
2d. It is important that an interviewer rehearse this section given all the contingency items and branching. It also saves time during the interview and prevents the respondent from feeling that the interviewer doesn’t know what he or she is doing, and thus call into question the reliability and validity of the study.
Consult: What could be done?
1. Such things as: bad format and biased instructions (assumes quality of service and food needs improving), items not numbered, more than one per line, vague questions without time frames (ate how often in the past week? Month? When?), items often have more than one “right” answer, options not mutually exclusive (cereal is a vegetarian meal and fruit can be a dessert; what if you are 5’ 5” tall?) or exhaustive (other items available in cafeteria), double-barreled items (ham and eggs), wordy biased items (“Some people feel…”), invoking an authority biases item (“a nutritionist from the Department of Health”), no branching (last item “Why do you thing she is not wrong” not only is poorly worded but assumes previous answer is “No”), no date when survey is due to be returned, invites comments but doesn’t say about what, scale from 1 to 10 doesn’t specify if 1 or 10 is the best quality food, and so on. Numerous mistakes to find.
Here is an example of a much better survey:
Cafeteria
Food Survey
We are interested in hearing your candid opinions about the food and
service in the cafeteria. Your responses are anonymous.
A. Here are some questions
about the cafeteria. Please circle the responses that best capture your
experiences.
1. Within the past week, did you go to the cafeteria at least once for a
meal?
What was your main reason
for not going this past week? Skip to Question 3
(1) Yes (2)
No
2. For each of these meals, please indicate how frequently you went to
the cafeteria for a meal this past week:
None
1 to 3 times 4 to 6 times Every Day
(a) Breakfast 0 1 2 3
(b) Lunch 0
1 2 3
(c) Dinner 0 1 2 3
3. On a scale of 1 to 7, where
7=Terrific, please evaluate the following based on your last visit to the
cafeteria:
(a) quality of the food (terrible) 1 2 3 4 5 6 7 (terrific)
(b) selection of foods 1 2 3 4 5 6 7
(c) friendliness of staff
1 2 3 4 5 6 7
(d) layout of the room 1 2 3 4 5 6 7
4. Now we’d like to get your opinions about the cafeteria.
Agree
Agree Disagree Disagree
Somewhat Somewhat
(a) I look forward to eating here.
1 2 3 4
(b) The food needs much
improvement 1 2 3 4
(c) The people who work there
are friendly 1 2 3 4
(d) The place is too dirty 1 2 3 4
(e) There are not enough choices
in the
food served 1 2 3 4
(f) It is a comfortable place to eat 1 2 3 4
B. Finally, we’d like to know
a little about you.
5. Gender: Male……… 1
Female……2
6. Your age: Under 21………..…1
21-25……………...2
26-30……………...3
31-35……………...4
36-40……………...5
41-45……………...6
46-50……………...7
51 or
more………..8
7. Please add any additional comments you have about your experiences in
the cafeteria.
Please return your survey to
the Human Resources Office by November
10.
CHAPTER 5:
Interpret: What Do These Real Examples Tell Us?
1. A probability sample using some (unspecified) interval: given the special listing of subscribers, there could already be a bias in the original list, unless the study is focusing on subscribers. Strength is that it is a random sample.
2. Random digit dialing is a good way of generating a random probability sample, although it excludes those who cannot afford telephones. An adult member willing to participate, however, may create a volunteer bias and limit the generalizability.
3. Two stages or clusters used in this probability technique, resulting in a potentially good random sample, given the high response rate.
4. Starting out as a random sample is a strength, but a longitudinal panel design resulted in a high attrition rate, calling into question the generalizability of findings from the more recent sample.
5. Probability random sampling within strata of sex and job category seems appropriate for a study on salaries, especially since pay differs by sex and type of job. This technique guarantees finding people randomly within different jobs and gender.
6. Small convenience sample of volunteers at a university limit how much the findings can be generalized to other students or non-students.
7. First convenience sampling is used by finding respondents from organizations that are not selected randomly. Then, snowball sampling is a non-probability technique used to generate respondents in difficult to find groups. Although this resulted in a sample of people needed for a study on Asian issues, the results cannot be generalized beyond the 64 people who participated.
Consult: What could be done?
1. A population study was done and if everyone (or at least a very large majority of them) completed the survey, then it would be accurate to make conclusions about the entire campus. However, with only 25 percent responding, it is possible that only the drug users and alcohol drinkers took an interest in the topic and answered the questionnaire. This could result in an over-sampling of users.
2. The conclusions should be made only about the 25 percent who responded, not about the entire campus. Demographic data should be reviewed to see if there is a
pattern as to who completed the
questionnaires and compare that
3. The university should be advised to conduct a random sample of students, or to send follow-up notes and surveys to increase the initial response rate, or to employ convenience sampling methods and distribute surveys in large classes, again with the intention of increasing the response rate.
CHAPTER 6
Interpret: What Do These Real Examples Tell Us?
1a. The variables are ordinal levels of measurement with equal-appearing intervals typical of Likert-type scales. These are suitable for means and standard deviations.
1b. For Blacks, racial-ethnic identity is more important when at work then when at home (2.89 versus 2.42), more important when at home for multiracials, and about the same at home and at work for whites. Compared to whites and multiracials, racial-identity is more important for Blacks at work (2.89 versus 1.82 versus 1.95) and at home. The standard deviations tell us that Blacks also are more dispersed on these measures than multiracials and whites, that is, their scores on these measures vary more than it does for others.
2a. The number of partners is not normally distributed and more than likely positively skewed, namely very few people at the higher end of number of partners (that is with 1,016 partners!).
2b. Half of the 3126 respondents (that is, 1563) had under 3 partners and the other half had more than 3. The most common response is 1 partner.
2c.
The range is 1016 – 0 = 1016. Other useful
3a. Because the mean, median, and mode are virtually the same, it is very close to a normal curve.
3b. i. 68 percent, because 63.3 is 4 inches below the mean, and 71.3 is 4 inches above the mean, that is, one standard deviation unit above and below the mean. 68 percent of all scores on a normal distributions are within one standard deviation unit plus and minus.
3b. ii. any height shorter than 63.3 inches, because 16 percent of a normal distribution are more than 1 standard deviation units away.
3b. iii. 50 percent since 67 inches is the median height.
3b. iv. People either shorter than 59.3 inches or taller than 75.3 inches would be statistically different from the mean because they are more than two standard deviation units (8 inches) above or below the mean.
3b. v. Between 59.3 and 75.3 inches, since around 95 percent of all the scores fall between those heights.
Consult: What could be done?
1. First step is to review frequency distributions (tables, graphs) for each variable to make sure that they are all variables and not constants. Any variable that does not have an adequate distribution cannot be used for further data analysis.
2a. mode, bar graph or pie chart
2b. mean, median, standard deviation, histogram or frequency graph
2c. median, pie chart or bar graph [ordinal measure]
2d. median (mean if normally distributed), frequency graph or histogram
2e. mode, bar graph or pie chart
CHAPTER 7:
Interpret: What Do These Real Examples Tell Us?
1a. -.255, the minus sign does not make it weaker than .098.
1b. An inverse relationship, so those respondents with more education tend to watch less television, or conversely, those with less education watch more TV.
1c. A correlation of .098 is virtually zero, therefore almost no correlation exists. Low correlation coefficients can be significant when sample sizes are large.
2a. There is no relationship between companies having political action committees and lobbying the government. There is no relationship between companies having political action committees and donating to charities.
2b. 67% of companies without PACs do not lobby the government, compared with 75% of companies with PACs that do lobby the government. In words: having a PAC is related to lobbying. Not having a PAC is related to not lobbying and not giving to charity.
2c.
The number of the chi-square itself does not give any
2d. Reject the null hypotheses and conclude that there is indeed a relationship between companies having PACs and lobbying and giving to charity. Those who have PACs tend to lobby more and are slightly more likely to give to charity than those without PACs.
2e. Because these are dichotomous measures, gamma could be used and even a Pearson r correlation.
Consult: What could be done?
1. Construct a crosstable with gender in the columns (independent variable) and the names of the sections in the rows (dependent variable), such as sports, news, fashion, entertainment, comics, and so on. Then calculate a chi-square and lambda to see if there is a significant association between these nominal measures.
2. frequency of reading the paper could be measured on an ordinal scale from “daily” to “infrequently” or with an interval/ratio measure of number of hours or times per week. If income is measured as ordinal categories, then you are limited to chi-square and gamma. If both are interval/ratio, then Pearson r. Assuming education, age, and family size are also interval/ratio measures, then Pearson r could be used. If any of these are in ordinal form, then chi-square and gamma would be appropriate.
3. In addition to favorite sections and frequency of reading, research might measure race/ethnicity, geographic location, occupation and assess their relationship with opinions of the quality of the paper, frequency of reading, favorite sections, type of newspaper, and so on.
CHAPTER 8:
Interpret: What Do These Real Examples Tell Us?
1a. It is a comparison of means between two categories (male and female).
1b. There is no difference in age, years of education, and days trained between men and women. For age and days trained, the probability of the t-test values occurring by chance is less than 1 in one thousand, and for years of education it is less than 5 in one hundred. In short, they are all statistically significant and the null hypotheses would be rejected.
1c. There is a difference in age, years of education, and days of training between men and women: Men are older, have more education, and required more days of on-the-job training.
2a. It is comparing autonomy and conformity scores among five different social class groups of Chinese workers.
2b. There is no difference in mean autonomy scores and mean conformity scores among the different social classes (cadre, professional, etc.). Both F values are statistically significant at the .001 level, and the null hypotheses can be rejected.
2c. There are significant differences in autonomy and conformity among the social classes: Nonmanual, cadre, and professionals are higher in autonomy, while supervisors and manual workers value conformity more.
Consult: What could be done?
1. Collect data from the campus bookstore and calculate the mean cost of books in these five disciplines. Since we are comparing more than two means, use ANOVA to test if the differences in cost are statistically significant.
2. Ask for confidential data of employees’ years of education and salaries. No names need be attached. Alternatively, you could ask employees to self-report their years of education and salaries on an anonymous survey. If data remain in interval/ratio form, then a Pearson r correlation could be calculated, but to answer the specific question of higher educated versus lower educated, the employees would be divided into two groups (higher and lower educated, perhaps determined by cutting at the median, or deciding to take those with less than college education and compare to those with some graduate education). Then a mean salary (assuming it is in interval/ratio form and not
ordinal categories) would be calculated for the two groups and a t-test would be used to statistically test the difference in salaries.
CHAPTER 9:
Interpret: What Do These Real Examples Tell Us?
1a. There is no relationship between Asian American’s GPA and how academically prepared they are, how confident they are in finishing their degree, and their employment status. Regression is used since there is more than one independent variable and all are measured as either equal-interval ordinal or ratio/interval measures.
1b. Academically prepared and confidence are statistically significant, while employment status is not. Academically prepared is very strong, almost twice that of confidence.
1c. It says that 26% of the reasons why Asian Americans vary in their GPA can be explained by these three independent variables. R is a PRE when squared.
1d. Those Asian Americans with the highest GPA tend to be better prepared academically and have higher confidence in finishing their degree. It does not seem to matter what their employment status is.
2a. Women are more likely to graduate than men, or men are less likely to graduate than women.
2b. This relationship is statistically significant. Getting a chi-square value of 6.79 for a table with 1 degree of freedom could occur by chance nine times in 1000.
2c. The first table (those with a B grade or lower) shows that there is very little difference between graduation rates of men and women. 59.8% of women graduate compared with 54.6% of men. For those with higher grades, 71.8% of women graduate compared with 60% of the men, a much larger difference.
2d. The chi-squares confirm that the relationship between sex and graduation rates is statistically significant only for those with grades of B+ or higher.
2e. Specification: The control variable of grades specifies the original relationship. It disappears for those with grades of B or under, and is replicated for those with grades of B+ or higher.
Consult: What could be done?
1. Before you can declare a cause and effect relationship, it is necessary to first establish that a correlation exists between income and political party and that alternative variables do no alter the original relationship. One can calculate ANOVA to compare the average incomes among respondents from different political parties, or if income is in ordinal categories, to construct a crosstable and test with lambda and chi-square. Then introduce a control variable, such as educational level or age, and see if the original relationship remains.
2. The dummy variable of Democrat and Republican can be used as a dependent variable in a regression analysis, and income, age, educational level could be entered as independent variables. Beta weights will tell us which are good predictors of voting Republican or Democrat.
CHAPTER 10:
Interpret: What Do These Real Examples Tell Us?
1. Clearly specifies the time and location of gathering a census of homeless people, both in shelters and on the street. The use of the word “census” implies that this is an attempt to document the population from which a sample could later be drawn. As described in the next paragraph, a stratified random probability sample was used (here called a random quota). Interviews were conducted with a structured and open-ended questionnaire. It specifies the sampling strategy and the types of questions being asked. In general, a decent summary of the methods, although it might specify who (race, gender, etc.) is conducting the interviews, how they were trained, and more about the survey questions and their validity and reliability.
2. It begins by
describing the location of the study, then discusses
the exploratory early stage of the project which used focus groups and
interviews with key
Consult: What could be done?
1. They should ask about the sampling used, who conducted the survey, what kinds of official data were made available by the school, how reliable and valid were the questions, and was it a self-report survey or interviews. They should also inquire about definitions used (what is considered “truancy” or “substance abuse” for example?).
2. Given the
potentially damaging
3. Executive
summaries of the report (the most likely way of distributing