So I think that I need to share what I learnt from my peer review. I know, I am a little late with this, but I am sharing none the less. I did my peer review on President or Dictator? A Comparison of Cuban American Media
Coverage of Cuban News. I think that my biggest problem with this article is that they did not tell the reader how they arrive at their percentages. They took for granted that the reader will just accept what they presented us with. As a result they severely compromised their validity and reliability - weakening their argument.
O. R. Holsti and R. P Weber were the two sources that I consulted vigorously while looking into Content Analysis. They were both very helpful resources and I totally recommend them (even if they are a little dated), especially since I have taken my notes from them and won't need to borrow them from the Inforum again.
I had originally considered doing my Peer Review on the "President or Dictator" article but ended up choosing a different article - one, with a research methodology that was more familiar to me. However, I do remember coming across some glaring problems in this article. I wish that I had noticed your post sooner so that my observations could be of some use to you. Anyway, here they are:
ReplyDelete1. Sample Limitations: One of the hallmarks of a poor research article, or a weak argument in general, is a lack of discussion of one's biases and limitations. In the case of this study, Callahan et al. (2012) draw observations from four Cuban-American newspapers: One Spanish-language newspaper from Miami, and one from New York; as well as one English-language newspaper from Miami, and one from New York. The authors explain their reasoning for targeting the Miami newspapers, but offer no explanation for choosing the New York ones.
Moreover, no efforts are made by the authors to explain why these samples, in particular, are representative of Cuban-Americans newspapers at large. It is claimed that the greatest concentration of Cuban-Americans is in Miami, but the researchers themselves admit that "Focusing exclusively on the Miami-Dade area to understand how all Cuban-Americans think about politics would be akin to only studying Native Americans on reservations... While these represent easily accessible populations, they may not, in fact, represent the majority views of the group." (Callahan et al., 2012, p.4). This entails that the results of their study can be generalized to Cuban-American magazines in Miami and in New York, but not to Cuban-American magazines at large within the United States.
These inherent sample biases and limitations are alluded to in the literature review section of the paper, but their relevance for interpreting the results of this study are never discussed.
2. Ambiguously Methodology: The only reason I was able to swiftly identify the statistical test used in Callahan et al. (2012) is because I had taken some in-depth classes on statistics throughout my undergrad. The X2 symbol, combined with the format of the presented tables, indicate that the authors used a Chi-squared test. It is not enough for the authors to merely state that they used "tests of differences" (Callahan et al., 2012, p.9), for there are many tests of difference within the field of statistics (differences between independent means, differences between paired observations, etc.). The authors should have explicitly stated their adopted statistical test.
Moreover, not enough information is given regarding the data collection process. It is stated by the authors that certain words and phrases were searched for in the selected magazines over a six month period, but there is no indication of how the frequency of these words was contextualized. Was the frequency of words determined by how often they appeared per word, per sentence, per paragraph, per page, per article, etc.? Furthermore, only a few examples are given for the words and phrases that were targeted. A full list of targeted terms is not provided by the researchers.
By not explicitly communicating the methodology of their study, Callahan et al. (2012), make it difficult, if not impossible, for their peers in the academic community to conduct replication studies in order to verify or challenge their results. The authors additionally make it difficult, if not impossible, for their peers to evaluate the study's methodology based on research that has employed similar or identical techniques to study other topics.