Journal Reviewing Advice?

Herp literature forum.

Moderator: Scott Waters

Post Reply
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Journal Reviewing Advice?

Post by CCarille »

Hey all,
I'm looking for some advice on something I've been asked to review for a journal.

I know many of you probably get a ton of requests to review stuff and have done it a lot, but I've only started teaching at a higher education level within the last year.

Being a first-time reviewer, I'd appreciate any advice anyone (who has had experience) can give on how to provide feedback and what should ultimately be the limit on mistakes before denying the manuscript. I've done some research and have general guidelines for myself, but hearing from people with experience is always helpful.

Thanks!
Chris
WW**
Posts: 240
Joined: June 7th, 2010, 6:32 am

Re: Journal Reviewing Advice?

Post by WW** »

If you are reviewing a research paper, the fundamental question is whether the question is phrased appropriately and whether the data presented are adequate to answer the question. If the answer to those questions is basically positive, then your job consists of making constructive suggestions to improve the manuscript in terms of analysis, presentation etc. If the data don't support the conclusions, then you already have your answer.

A lot will also depend on whether you are dealing with a first submission or a revision. I would be more forgiving of mistakes/problems in a first submission than a revision.

If you think it's a brilliant manuscript, then it is worth writing a paragraph or two to highlight the strengths of the ms. A review that simply says "it's brilliant" is surprisingly useless to an editor, who needs to balance what are often divergent viewpoints among reviewers, and can use good arguments from the reviewers to help reach a conclusion.

A more difficult aspect is the question how significant the research is and is it important enough for the journal... never an easy one.

Finally, bear in mind that reviewing is never an exact science. Some manuscripts are awkward, and teasing them apart can be a difficult and thankless task. I remember as editor coming across manuscripts that were opaque to start with and it was only after two rounds of review that it became obvious what the authors had really done and that the entire ms was fatally flawed - my name came off a few Christmas card lists that way....
User avatar
Hans Breuer (twoton)
Posts: 3230
Joined: June 8th, 2010, 3:19 am
Location: Kuching, Sarawak (Borneo)
Contact:

Re: Journal Reviewing Advice?

Post by Hans Breuer (twoton) »

WW** wrote:my name came off a few Christmas card lists that way....
LOL
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

Thanks WW! I appreciate the insight. I had some of my own ideas of how to go about it all and you've helped to narrow some of that down. I would of went to some people in my dept, but I figured I'd see what "herp" focused colleagues would suggest since there is no one in either department I'm a part of that deals with herps at all and the paper is herp related.
Hans Breuer (twoton) wrote:
WW** wrote:my name came off a few Christmas card lists that way....
LOL
I agree... haha
User avatar
gbin
Posts: 2292
Joined: June 10th, 2010, 4:28 pm

Re: Journal Reviewing Advice?

Post by gbin »

Congratulations and condolences, Chris, on becoming a reviewer! And kudos for taking the responsibility as seriously as it deserves, too! :thumb:

I'm not particularly herp-focused as a scientist, but I thought you might benefit from my general comments, some of which overlap with what Wolfgang already offered you.

The first question I always ask myself when I receive a journal submission to review is "Am I an appropriate person to review this particular paper?" The answer isn't always yes, though especially early on in one's scientific career there's an understandable compulsion to accept everything sent one's way. (I was once even asked to ghost review a paper by a boss who liked to pretend that she knew things she didn't but who realized that in this particular case she wouldn't be able to fake her way through it. :shock: ) Sure, ideally every paper should be written such that pretty much any good scientist could reasonably evaluate it in basic terms, but in reality that's sometimes not the case; this is especially true of papers that involve considerable or unusual statistical analyses which lie outside the realm of the asked reviewer. Too, at least some familiarlity (and the more expertise the better) with the methods, literature, etc. of the field in question is part of what the editor is seeking. It might escape an editor's attention, but it's painfully obvious to an author when his/her paper has been reviewed by someone who didn't understand it or have sufficient knowledge of its field but wrote a bunch of bogus criticisms rather than admitting that someone else should have been entrusted with the job. Don't put yourself in a position where you might be such a reviewer! Just respond to the editor with a reply such as "I'm afraid I'm not quite the right person to review this particular paper, but I believe so-and-so person at such-and-such place might be." The editor will greatly appreciate both your helping to preserve the integrity of the peer review process and your suggestion on where s/he should go next (when you are able to provide such a suggestion), and will have no qualms about sending you additional papers to review.

Then it's on to the basics. Does the paper lay out a clear hypothesis, and a test that is both relevant and meaningful for examining it? Is the methodology adequately detailed for you to fully understand what was done, and if so, is it appropriate to the test it's meant for? Don't be surprised if a fair number of papers fail in one or more of these, but write your review keeping in mind that the fault(s) might lie in the work itself, in the writing or both. That is, some problems might be correctable in a subsequent draft of the manuscript and some might not, and which is the case won't always be clear to you.

Beyond the basics (and within the framework for reviews requested by the journal), there are two problems that are so common in scientific papers that I always keep a watchful eye out for them:

First and foremost is the unsupported statement. In my experience, these are by far the most common errors one encounters, and are easily made by a majority of writers in a majority of papers (and yes, I've occasionally been caught making them, too, as careful as I try to be to avoid them). Every factual statement in a paper - excepting only the most basic, extremely well established bits of information that the authors might be reminding readers of as background - needs to be backed up by data, either data found in that same paper or in another which is cited where the statement is made. Unsupported statements can pop up nearly anywhere in a paper, but conclusion sections in particular often include statements that are at most only partly supported (as people think more of their data than they seem to have cause for). Sometimes the statement needs to be removed, and sometimes it just needs to be rewritten so that it's clear that the authors are only expressing their belief/interpretation or a possibility rather than an assertion actually backed by evidence. Lots of unsupported statements get by both reviewers and editors, I know, but each in its own way damages our knowledge base (it's generally much harder to dislodge false knowledge that's become established in the literature than it is to fill in areas of ignorance remaining in a field), and every one you root up helps.

Second, and often related to the problem above, is the misused citation. These too are extremely common, and emphasize the need for some expertise on the part of the reviewer. Do you suspect that a paper is using a citation that doesn't actually say something or present data it's purported to? Check the cited paper and see! I have very often seen an author cite someone else, sometimes even for crucial aspects of his/her current work, when in fact the cited paper simply cites someone else in turn for the same matter. Sometimes long chains of such empty citations are thus created. This phenomenon could just be dismissed as laziness, I suppose, except that even under the best of circumstances each iteration offers the opportunity for the information conveyed to change, as in the childhood game of telephone. And the circumstances are often far less than their best, too (as I'll touch upon in a moment). If the truly original citation - the one with the actual data in it - for a statement isn't something out of downright antiquity or unavailable for some other reason, it should be cited. A less (but still fairly) common problem with citations is when people don't make unsupported statements themselves but instead cite the unsupported statements of others as if they were evidence-based assertions. As I said, false knowledge becomes established in the literature this way all too easily. Still less common (but still common enough to be disturbing) is when a citation doesn't actually say anything or present any data that even remotely resembles what it's being cited for. I don't know whether this happens due to deliberate deception or some kind of serious brain fart, but it's bad news for the scientific literature. Much less of an issue but looming seemingly large in the minds of some reviewers is literature that's not cited but should be; just remember that if you recommend a whole slew of papers by a single author be cited, the folks who wrote the manuscript will almost certainly figure out that you are the seminal author in question. :lol: And, of course, there's always the mundane problem of citations appearing in the text but then not in complete form in the literature cited section, or vice versa. These don't generally raise any serious red flags, but are just something to be noted so that they're corrected. Yeah, tracking - not to mention tracking down - citations while reviewing a paper can take some time, especially if the authors went overboard in citing the literature (which is another valid criticism to make on papers so affected - and a criticism that editors don't take lightly given that more citations means more print and paper). But it's important.

Speaking of importance, as Wolfgang mentioned, it can be difficult to assess the originality and significance of the research in question. Impossible, really, if one doesn't have sufficient expertise in the field. Just do your best with papers you truly feel you can handle, and pass on the rest as I said.

Wolfgang raised a worthwhile point, too, in mentioning that it's good to identify particular strengths as well as weaknesses of a paper. You're meant to provide the editor with an objective review first and foremost, and fortunately as few papers are all bad as are all good. Providing the authors with constructive criticism for improving a subsequent draft is secondary but also important. And on that last note...

Reviewing papers written by folks for whom English is a second language can be especially difficult, and sometimes things are just so unclear no matter how hard you try to read past the writing issues that the best you can do is say something like "This paper requires additional editing for English before it can be properly reviewed." It's not your job to rewrite a paper for anyone in any event, but if you can see where the authors were trying to go and you're willing to put in the additional effort to make some clarifying suggestions, I can assure you that assistance will be greatly appreciated. I think I would even have been put on a couple folks' Christmas card lists ;) for giving them such help when I could see they really had something worth getting out, had they known who I was; certainly a few have thanked "an anonymous reviewer" quite warmly in the acknowledgements of the final draft.

Enjoy yourself! And try to hold onto the fun of reviewing your first paper as long as you can! ;)

Gerry
WW**
Posts: 240
Joined: June 7th, 2010, 6:32 am

Re: Journal Reviewing Advice?

Post by WW** »

gbin wrote:Reviewing papers written by folks for whom English is a second language can be especially difficult, and sometimes things are just so unclear no matter how hard you try to read past the writing issues that the best you can do is say something like "This paper requires additional editing for English before it can be properly reviewed." It's not your job to rewrite a paper for anyone in any event, but if you can see where the authors were trying to go and you're willing to put in the additional effort to make some clarifying suggestions, I can assure you that assistance will be greatly appreciated. I think I would even have been put on a couple folks' Christmas card lists ;) for giving them such help when I could see they really had something worth getting out, had they known who I was; certainly a few have thanked "an anonymous reviewer" quite warmly in the acknowledgements of the final draft.
For manuscripts by non-English-speaking authors with poor English, you can always recommend the SSAR's pre-submission manuscript checking service: http://www.ssarherps.org/pages/presub.php

One has to salute the brave volunteers who take this on - editing a poorly written ms into something publishable is often more work than writing it from scratch!
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

Thanks for the thorough advice Gerry!
gbin wrote:Congratulations and condolences, Chris, on becoming a reviewer!
Funny and something I've read a lot about. I can imagine how some can be overwhelmed by reviewing when one has their own research to focus on.

I'll be sure to fact check and follow-up on questionably citations. I've gotten pretty used to doing this for my undergrad students since I assign a fair amount of papers throughout each semester. Luckily, I've had a fair amount of writing experience between a book and articles as well.... and my g/f is always on my ass because she is a writer. haha
WW** wrote: For manuscripts by non-English-speaking authors with poor English, you can always recommend the SSAR's pre-submission manuscript checking service: http://www.ssarherps.org/pages/presub.php
Thanks for the link WW! I'll keep it tucked away for when the occasion arises. Back when I was a HS teacher I had to call upon many different resources for grading papers and finals.... students speaking Mandarin, students speaking Spanish (although my Spanish is pretty decent, but not the best for science), and I even had a student that was partially deaf, only spoke a little English and a little Spanish... he mainly spoke Zapoteca (sp?) a dialect found in the mountains of Mexico!

I always couldn't imagine being a HS English teacher. Now, grading undergrad papers I can't imagine being an English/Writing Professor. It's amazing how awful people are with their spelling and grammar.
User avatar
gbin
Posts: 2292
Joined: June 10th, 2010, 4:28 pm

Re: Journal Reviewing Advice?

Post by gbin »

CCarille wrote:... I've gotten pretty used to doing this for my undergrad students since I assign a fair amount of papers throughout each semester...
Definitely a good form of preparation for reviewing journal submissions, as students manage to display every conceivable problem at one time or another in the papers they write. :thumb:

Gerry
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

gbin wrote: Definitely a good form of preparation for reviewing journal submissions, as students manage to display every conceivable problem at one time or another in the papers they write. :thumb:
Gerry
That could not be more true! I sometimes think I'm grading papers written by 5th graders or non-English speaking students. Once again why I would never want to be an English professor/teacher.
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

snakemanaustralia wrote:“If the data don't support the conclusions, then you already have your answer.”

Well said Wolfgang.
However it is a pity you and your mate Robert Hansen don’t practice what you preach here!
Lets take a look at your latest rant in herp review as an example!
Then take a look at how it breaches the SSAR’s own ethics guidelines as published on their website...
Want some examples?
It's fine if you have issues with other people, everyone does. Please leave the unnecessary comments out of posts I've started. I was simply looking for help from colleagues, not to hear people air arguments and issues with each other. Have some decency and respect please... at least enough not to muddy up posts of people you do not know in any context.
User avatar
Hans Breuer (twoton)
Posts: 3230
Joined: June 8th, 2010, 3:19 am
Location: Kuching, Sarawak (Borneo)
Contact:

Re: Journal Reviewing Advice?

Post by Hans Breuer (twoton) »

snakemanaustralia wrote:CCarille, I just thought it may be appropriate that you be aware of some of the unethical practices of Mr Wuster.
It's not appropriate. I think Mr Carille has made that quite clear when he wrote
It's fine if you have issues with other people, everyone does. Please leave the unnecessary comments out of posts I've started. I was simply looking for help from colleagues, not to hear people air arguments and issues with each other. Have some decency and respect please... at least enough not to muddy up posts of people you do not know in any context.
Which part of the above is unclear to you?

Please refrain from further hijacking this thread. I, for one, am interested in the original subject and would like to see more people's opinions on the matter...which will not happen if you scare them off.

Thank you in advance for your kind consideration.
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

Well the process turned out to be a little disappointing....

I sat down to a first read and rejected the paper immediately for plagiarizing the first couple paragraphs strait from wikipedia! Now I don't know if the request to review the paper was a joke. Sad if it isn't [a joke], since the paper was supposedly written by a PhD and even my high school students knew not to plagiarize.
User avatar
chris_mcmartin
Posts: 2447
Joined: June 9th, 2010, 12:13 am
Location: Greater Houston TX Area
Contact:

Re: Journal Reviewing Advice?

Post by chris_mcmartin »

CCarille wrote:I sat down to a first read and rejected the paper immediately for plagiarizing the first couple paragraphs strait from wikipedia!
How did you know; are you familiar with that subject's wikipedia page and recognized the similarity, or did you run the paper through an analysis program? I've never used those, but fortunately for me, in most cases with my students it's (painfully) obvious they wrote their papers unassisted. :lol:
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

chris_mcmartin wrote: How did you know; are you familiar with that subject's wikipedia page and recognized the similarity, or did you run the paper through an analysis program? I've never used those, but fortunately for me, in most cases with my students it's (painfully) obvious they wrote their papers unassisted. :lol:
The paper involved ball pythons [to some capacity] and the information about them seemed a little off and rudimentary for a PhD to be writing. I just had a suspicion that it was copied and I had an idea it was from wikipedia. Surprising how accurate I was, as I just googled "ball python" and checked wikipedia.... copied word for word (even with a mistake or two! haha - and not enough to be written in the author's own words). Sad if the paper was really written by a PhD, even if they aren't familiar with herpetology/zoology.

At the high school level I didn't find much plagiarism, simply because we didn't have a lot of papers... not part of the NYS Living Environment curriculum. I have found it at the undergrad level though and it has been "painfully obvious". that they did plagiarize. I have found that most writing at the hs and undergrad levels is ridiculously awful... so much so that I dread reading through some papers. I have a stack right now I'm procrastinating on... I actually graded the final instead. It's amazing how bad grammar has become and how often students write run-on sentence, don't spell words completely (ok vs okay), and write sentences that don't make any sense. Hell, even reading the paper that I was reviewing, there were grammatical errors, spelling errors, and scientific names weren't written correctly. The US education system is quickly falling behind I think... and that's not a shot at you or any teacher in particular (I was a hs teacher). Standards are being lowered, kids are being coddled more, and the basic skills attained in elementary school are either not being taught or not being retained (most schools don't even teach script). Sorry for the rant.
User avatar
gbin
Posts: 2292
Joined: June 10th, 2010, 4:28 pm

Re: Journal Reviewing Advice?

Post by gbin »

Ouch! :shock: Plagiarism is a problem I haven't run into in Ph.D.s' work before (though I found it rampant in college student reports). And from wikipedia, at that! Could it indeed have been a prank?

When I was working on my doctorate, one of my fellow graduate students punked a professor on her committee so well that it almost got her booted out of the department's graduate program. In preparation for a meeting with her advisors, she wrote two very different dissertation proposals. One was a good, solid presentation of what she intended to do for her scientific hypothesis-driven research, and the other was a fuzzy-soft, meandering piece of tripe in which she proposed a study that was as far from real science as she could think to make it. She was inspired to do this because the aforementioned professor was a relatively young guy who the worst kind of hotshot, who on the basis of a small body of good work that got him quite a bit of attention viewed himself as some kind of extraordinary gift to the field of zoology as a whole and to our department specifically, and who was always ranting about how the rest of the department wasn't up to his standards and how impatient he was to get everyone there; he was often downright abusive in the process, as you might expect. (On one particularly memorable occasion, in the setting of a faculty meeting at which I and numerous other graduate students were also attending - we were encouraged to sit in such meetings, and even to offer our thoughts to the assembly if they seemed relevant - this guy cut off a brief ramble by one of the department's most senior professors with an almost spittle-laden shout of "Get to the f-ing point!" that silenced the whole room for a long moment. He was also inordinately fond of ambushing finishing graduate students with what he hoped were tricky and devastating statistical issues during their exit seminars.) Her idea was to pass the fake proposals around some weeks before the meeting, let everybody have a good laugh (and hopefully thereby lighten this particular guy's mood) and then give them the real proposal when they contacted her or her advisor about the fake. It apparently worked just as expected, too, except for with this guy. She waited and waited without hearing anything from him, finally decided that he must have just gotten a copy of the real proposal from her major advisor without saying anything to her about it, and went to her meeting...

... Where the guy promptly ripped her to shreds for her joke. Apparently the deal was that he hadn't bothered to so much as look at the proposal he'd been given until late on the night before the meeting, mistook it for what she really intended, and was so apoplectic that he called her advisor about it in the middle of the night to complain about her severe inadequacies as one of the department's graduate students. Her advisor didn't really help matters, as he simply informed the guy that it was a joke - and he would have promptly gotten the real proposal to review had he not waited until the absolute last minute to do his job. So the guy went to the meeting thinking that the joke was aimed at him and him alone, and determined to make her pay for it.

She was still pretty shaken up by that meeting when she told me about it sometime later, but there was also a certain glint in her eye that told me she'd have done the exact same thing again had she to do it over again. I accordingly and sincerely told her that she was now one of my heroes, and in my mind she remains such to this day. :D

Anyway, I hope my tale lightens your mood at least a bit. Don't worry, you'll get plenty more papers to review in the future - and some of them will be real gems that you'll be one of the first people privileged to get a peek at. ;)

Gerry
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

Thanks Gerry! That's a pretty wild story... I've definitely met people like that new professor before. Unfortunately, the world is filled with many people that think they're above everyone else.

I'm sure the paper isn't a prank on me particularly, as I checked the journal and it isn't a fake, but wonder if the submission itself was "real". The topic was a little strange to begin with, but within reason for research.
User avatar
chris_mcmartin
Posts: 2447
Joined: June 9th, 2010, 12:13 am
Location: Greater Houston TX Area
Contact:

Re: Journal Reviewing Advice?

Post by chris_mcmartin »

CCarille wrote:how often students write run-on sentence, don't spell words completely (ok vs okay),
1. "run-on sentences." 8-)
2. I don't know if I'd allow either "ok" or "okay" for the style of writing we use at my institution...sounds too conversational vs. professional.

The US education system is quickly falling behind I think... and that's not a shot at you or any teacher in particular
I agree, and I don't take offense because I'm not the guy primarily responsible for imparting grammatical knowledge; I just correct it when I find fault (and as the SWCHR Bulletin editor, I'm immensely and personally embarrassed when a missed typo or two slips past the proverbial goalie). I'm currently instructing at the graduate level, so the level of mediocrity hasn't quite become overwhelming just yet, but trends are clearly visible. I don't know how much blame the education system itself should shoulder; lack of parental involvement during crucial developmental years is the primary culprit, at least in my opinion.
User avatar
CCarille
Posts: 380
Joined: January 8th, 2011, 5:51 am
Location: NY

Re: Journal Reviewing Advice?

Post by CCarille »

chris_mcmartin wrote: 1. "run-on sentences." 8-)
Awful on my part there! haha

"okay" comes up in papers because I look for opinions on several topics throughout the year. I realize it's equivalent to using "really" and "very" among other non-descriptive terms, but it works for my purposes with non-majors I teach.
chris_mcmartin wrote: I agree, and I don't take offense because I'm not the guy primarily responsible for imparting grammatical knowledge; I just correct it when I find fault (and as the SWCHR Bulletin editor, I'm immensely and personally embarrassed when a missed typo or two slips past the proverbial goalie). I'm currently instructing at the graduate level, so the level of mediocrity hasn't quite become overwhelming just yet, but trends are clearly visible. I don't know how much blame the education system itself should shoulder; lack of parental involvement during crucial developmental years is the primary culprit, at least in my opinion.
I agree with you in part on the parental involvement. It's just hard for me to say so without being a parent. I believe a lot of the blame should rest on parents. Simply, placing all the blame on teachers doesn't seem appropriate when students get much of their work ethic from their home lives.
User avatar
gbin
Posts: 2292
Joined: June 10th, 2010, 4:28 pm

Re: Journal Reviewing Advice?

Post by gbin »

I guess I habitually use "ok" in informal writing :oops: ; I use neither "ok" nor "okay" in formal writing.

The internet itself contributes heavily to people (not just students) having a more and more tenuous grasp of proper English. The problem is that a person see so many mistakes (and so many of the same mistakes) online that over time s/he unconsciously begins to incorporate them. I myself used to be a superb speller, for example, but nowadays I find myself checking a dictionary for spelling more and more often just because I've seen various misspellings so often that doubt has been raised in my mind about what's correct.

I'll work with students in the context of their reports/colleagues in the context of their papers, but no one could pay me enough to take on the job of English teacher.

Gerry
User avatar
Hans Breuer (twoton)
Posts: 3230
Joined: June 8th, 2010, 3:19 am
Location: Kuching, Sarawak (Borneo)
Contact:

Re: Journal Reviewing Advice?

Post by Hans Breuer (twoton) »

gbin wrote:I guess I habitually use "ok" in informal writing :oops: ; I use neither "ok" nor "okay" in formal writing."
It's "OK", anyway - capitals.
User avatar
gbin
Posts: 2292
Joined: June 10th, 2010, 4:28 pm

Re: Journal Reviewing Advice?

Post by gbin »

Well then I'm wrong every which way, ain't I? :P

Gerry
User avatar
Hans Breuer (twoton)
Posts: 3230
Joined: June 8th, 2010, 3:19 am
Location: Kuching, Sarawak (Borneo)
Contact:

Re: Journal Reviewing Advice?

Post by Hans Breuer (twoton) »

gbin wrote:Well then I'm wrong every which way, ain't I? :P
:-)
Post Reply