Scholarship Skills

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Scholarship Skills

Tim Sheard, PSU

Todd Leen, OGI-OHSU

Lecture 19: Reviewing

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Reviewing

Basically an altruistic activity

Why do it?

–Altruism is a good thing

–Intelligent, thoughtful reviews enhance the quality of thejournal or conference

–Enhance your reputation (with editor or program chair)

–Step to becoming an editor or program committee member

–Shows professional participation

You may be called upon to review a journal orconference paper as a grad student. (Either directly orby a professor.)

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Basic Job

You must determine

–Are the results worthwhile?

–Are they probably right? (How muchtime/effort is expected?)

–Are they presented well?

You might

– suggest ways to improve investigation

– suggest ways to improve exposition

– point to work authors may have overlooked

– break anonymity if it’s helpful

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Read the Paper

Start by reading the paper carefully

–take notes as you go

–mark typos on the MS

–if it’s dreadful writing, can say that theexpositional deficiencies prevented you fromgiving it a careful reading

–In extreme cases can abandon paper -- butTELL THE EDITOR WHY!

–checking every example and proof?

•not always, but SAY SO

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Evaluation

Evaluate results, defense, presentation

•Is it technically sound, validmethodology?

Is there a solid defense?

•Is it significant?

If not, need to say why it is obvious, simple or oflow utility

Archival quality -- interesting for 5+ years?

•Is length appropriate to contribution

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Evaluation(cont’d)

•Is it novel?

–either results, or methods, or conceptualsynthesis

•Is it interesting?

Motivation, context, interpretation

Helps understand nature of computation

Can be used to solve problems

Will spawn other research

Timely to the field

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Evaluation(cont’d)

•Is it deep?

Might you get it as a homework problem in agraduate class?

•Is it concise?

Irrelevant parts, needless detail

But not too concise: shouldn’t be hard to follow

•Can it be understood by people in thefield?

Mostly self-contained

Something for non-experts in its particular area

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Evaluation(cont’d)

•Is the paper complete?

Methodology adequately described

No gaps in the development

•Is the research complete?

•Style and polish

Has it been prepared with thought and care?

Easy to read?

•Does it give proper credit to others?

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Decide

•Conferences sometimes have anumeric scale, journals may ask fornumerical judgment in several areas(novelty, completeness, importance,clarity).

–Reject -- no hope (rare)

–Revise – for exposition, may also askfor additional experiments ortheoretical elements

–Accept as is (unlikely)

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Professional Responsibility

•Be objective and fair

Put aside personalities and favored approaches

Alert editor or program chair to conflicts of interest

•Speed

Try for deadline in cover letter – editorial staff willsend reminders

•Make an acceptable paper as good aspossible -- that is, tell the author how toimprove the paper

•Confidential

•Courteous and honest report

How certain and qualified are you in yourjudgements?

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Typical Reports

•Out of scope

Wrong journal or conference

•Already done

Give citation

•Too simple

You could solve it if posed as an exercise

•Not very significant

Maybe okay if shortened

•Major errors

Essential parts are wrong

•Not obligated to determine if fixable, but give anopinion if worth trying to fix, or fixable.

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Typical Reports(cont’d)

•Boring

Grind through the details, tedious experiments

•High overhead for results

Lots of work for little output

•Poorly presented

Not a good job of writing the paper

Is there a basis for a good paper? (Tellingpeople how to salvage their paper is timeconsuming. These tend to be the longest andmost tedious reviews to write.)

•Acceptable with minor (or no)revisions

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Typical Reports(cont’d)

There is sometimes a review formprovided (e.g. IEEE).

Don’t feel constrained by it, especially for ajournal (TkL almost never uses them.)

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Typical Reports(cont’d)

For editor (PC chair) and authors

Shorter for a conference paper

•Statement of significance

•If you are recommending rejection,say why.

•General comments, suggestions forimprovement

•Detailed notes

Usually by page or section

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Kinds of Comments

Typos, grammar, confusing statements

References to add

Requests to compare to other work

Additional points to address in paper

Organization

Sketchy sections, parts to omit, pieces to move

Presentation

Better formulation of definitions, need for examples

Technical errors

If you think some major point is wrong, try to include acounterexample.

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Cover LetterPrivate Remarks to Editor

Not seen by the authors

•Identify paper

–title

–author

–for which journal or conference

–number

•Your expertise and effort (time spent),caveats (“this isn’t an area of my greatestexpertise ...”)

•Summary of paper and your opinion

One or two paragraphs

Scholarship Skills

Tim Sheard & Todd Leen

Lecture 19

Cover LetterPrivate Remarks to Editor

•Your recommendations andjustification

•Parts of report you want to emphasize-- specific comments

•Anything else not for the authors