"Why didn't the author's use the CMU code?" This part of a scathing review made me want to crawl through the Internet and strangle the reviewer. I had an entire section that detailed why I didn't use the CMU code. From the comments, it was clear he/she didn't read the paper.
In a peer-reviewed conference or journal, each paper is given to 3-5 people that rate the paper. The best rated papers make it in. This system is not perfect, but it is a good one.
Can you make it better? Yes. If you are invited to peer-review a paper here are some suggestions:
- As you read through the paper, keep notes. I like reading the paper on one monitor, or on paper, and keeping notes in a simple text editor. Do this the first read of the paper.
- For each note, refer to specific parts by section, paragraph, line, whatever.
- Number each point
- Point out things with the paper that you liked.
- No matter how bad the paper is, find something nice.
- Be specific. If there are common problems throughout the paper, find a specific example.
- Make suggestions on what would work
- If you don't have time to read the entire paper, don't do a review on it.
The goal of peer-reviewed publications should be to further humanity. So if reviewer feedback doesn't help educate the writer of the article, it is pointless. Great publications may come from those who originally submitted poor work. If constructive feedback is not given, the author may never become great.
