Skip to content

How we score science.

Each paper receives an adjusted score 0-100. That score combines two independent judgements: the quality of the work, and the evidential weight that the study design carries for the paper's claims.

The ratings

Quality, 0 to 100

Review-based assessment of the paper's rigour, clarity, transparency, and validity of its conclusions.

Design weight, 0 to 100

Evidential weight the study design gives the paper’s claims, ranked on its field’s evidence hierarchy.

Adjusted score, 0 to 100

Q adjusted by D via an interaction term. Quality dominates; design shifts it by up to 7.5 points, most when the two align.

Quality: the Q score

Q rates a paper’s execution, independent of study type, from two parallel streams: reviewer-raised issues, each scored by criticality, and reviewer-identified merits, each scored by calibre.

The criticality anchors
  • 0Negligible.A trivial concern. We record it for completeness, but it has almost no effect on the soundness of the work. Typographical issues, minor stylistic preferences, small omissions that a careful reader can barely notice.
  • 5Moderate.A genuine problem that a reasonable reviewer will expect the authors to address. The issue does not invalidate the work, but it weakens a claim, a conclusion, or part of the methodology.
  • 10Fatal.A defect so severe that the reader cannot trust the work in its current form. Causes include a fundamental design flaw, a breakdown in the chain of evidence, an analytical error that invalidates the central conclusions, or an ethical breach. Polish cannot rescue the paper; it needs substantive rework.
The calibre anchors
  • 0Negligible.A strength so minor that it almost does not register. The work does the task competently, but at a level that does not distinguish it from other adequately executed papers in the field.
  • 5Moderate.A solid, real merit. The work does something clearly above baseline competence.
  • 10Exceptional.A standout strength. A new methodological standard, evidence of rare quality, an insight that changes the approach to a question, or execution at the top of the field.
Reading the numbers

The numbers are a navigation aid. One high-criticality issue can matter more than a dozen minor ones. One exceptional-calibre merit can justify a paper that has noticeable issues elsewhere. The written reasoning that accompanies each score holds the real evaluative work.

0Negligible55ModerateModerate1010FatalExceptionalCriticalityIssues · 0-10CalibreMerits · 0-10
Fig. 1Criticality and calibre share one scale: a zero in the middle, severity to the left, strength to the right.
Criticality
How seriously an issue undermines the work. Not the difficulty of a repair, and not its prominence in the paper.
Calibre
How impressive or valuable a strength is, compared with the norms of the field.

The review files each issue and each merit into one of eight categories

01
Research Design
Foundational decisions: methodology, study structure, selection of participants or materials, and the overall approach’s fit to the question.
02
Data and Evidence
Raw material the work rests on: soundness of data collection, assembly of evidence, measurement methods, and quality of underlying sources.
03
Analytical Approach
What the author does with the collected data: statistical models, qualitative coding, computational pipelines, formal reasoning, or argumentative structure, in both choice and execution.
04
Scholarly Grounding
How fully the work situates itself in its field: engagement with relevant literature, strength of theoretical underpinnings, proper credit to prior work, and clear positioning of its contribution.
05
Reporting Quality
Transparency and completeness: enough detail (methods, data, code, justification) for a reader to understand, evaluate, and reproduce the work.
06
Interpretive Rigour
Leap from results to conclusions: do findings support the claims, and does the author acknowledge limitations, hedge claims, and examine plausible alternative explanations?
07
Ethical Conduct
Ethical treatment of subjects, informed consent, declared conflicts of interest, data integrity, and adherence to broader norms of responsible conduct.
08
Contribution
What the work adds. A paper can be technically impeccable yet contribute little, or rough in execution yet advance understanding in important ways.

Design weight: the D score

D is the evidential weight a study design gives the paper’s claims. Strong evidence is field-dependent: resistance to confounding in medicine, logical validity in mathematics, binding authority in law, community legitimacy in Indigenous studies.

What the hierarchy measures

Epistemic warrant: how fully the study design shields the claim from error, for the claim type of that field.

What it does not measure

The novelty, importance, or contribution of the work. A lower position on the hierarchy is a different type of warrant, often the strongest available for that question, not worse research. A score of 3 in ecology means the paper uses a natural experiment or a quasi-experimental design. That design is appropriate, and often the only ethical option, for many ecological questions.

The core question is different in each field

Field

The question it asks

Hierarchy applied

Clinical / Health
“Does X cause Y in humans?”
Hierarchy of causal strength
Mathematical Sciences
“Is X necessarily true?”
Hierarchy of proof rigour
Physical / Chemical Sciences
“Can X be measured reliably?”
Hierarchy of experimental reproducibility
Social / Economic
“Does X generalise?”
Hierarchy of internal + external validity
Humanities / Interpretive
“Is the reading of X warranted?”
Hierarchy of argumentative quality
Law
“Is X binding?”
Hierarchy of legal authority
Engineering
“Does X work in practice?”
Hierarchy of validation rigour
Indigenous Studies
“Is X legitimate to this community?”
Hierarchy of relational / cultural validity
Fig. 2Eight fields, the question each one asks, and the hierarchy that answers it.

Research domains

23 domains

Agricultural, Veterinary & Food Sciences

OCEBM Levels of Evidence 2011 (adapted for experimental and observational biology: clinical, production and breeding research)

Animal production · Crop and pasture production · Horticultural production · Veterinary sciences

Level

Study / evidence type

Scale: 1 (strongest) → 5 (weakest)

1
Systematic review /​ meta-analysis of randomised, replicated controlled experiments or trials - OR a model /​ genomic-prediction validated by accurate performance on independent populations or environments
2
A single high-quality randomised, replicated controlled trial or experiment (field, pen or clinical), or a controlled experiment showing a large, consistent effect across units
3
Controlled experiment lacking randomisation or replication, OR a strong observational /​ quasi-experimental study (cohort, case-control, cross-sectional, diagnostic-accuracy, or systematically sampled pathology/​parasitology survey)
4
Uncontrolled or descriptive work - case series, single-site/​single-season trial, single-specimen anatomical/​physiological/​microbiological characterisation, or exploratory measurement without a comparator
5
Mechanistic/​physiological reasoning, an unvalidated model or in silico prediction, or expert husbandry/​clinical opinion without empirical confirmation
Fig. 2.1The hierarchy applied to Agricultural, Veterinary & Food Sciences. Bar length is the position of the level on its scale.
  • Formal
  • Adapted
  • Informal

The adjusted score:
Q and D combined

S combines Q and D into a 0–100 score via an interaction term rather than a sum, rewarding well-executed work in demanding designs, penalising poorly-executed work in forgiving ones, and adjusting off-diagonal cases (high Q/low D, low Q/high D) less.

Adjusted score

83.38

+3.38

Gold

80

75

The adjustment peaks at ±7.5 where quality and design align, and at ±3.75 where they do not.

−7−6−5−4−3−2−1+1+2+3+4+5+6+7+7.5 · high quality with a high-weight design−7.5 · low quality with a low-weight design+3.75 · off-diagonal−3.75 · off-diagonalD = 50 · S = Q050100050100Q, qualityD, design weight

Q 80 D 75

S = 80 + 3.38 = 83.38Gold

Fig. 3The adjustment to Q over the (Q, D) square. Contours at each whole point; the dotted line is the neutral design weight; the marked points are the worked example. Drag to read any point.

Worked example: medicine

Fifteen papers: five study designs from the medical hierarchy, each at three levels of quality. Select one to place it on the instrument.

Study design

Strong

Q = 80

Average

Q = 50

Weak

Q = 20

Systematic review / meta-analysis of RCTs

D = 100

Individual RCT or observational study with large effect

D = 75

Non-randomised controlled cohort

D = 50

Case-series / case-control

D = 25

Mechanism-based reasoning / expert opinion

D = 0

Across a row, the top rung lifts strong work the most and the bottom rung costs weak work the most.

Individual RCT or observational study with large effect

Strong work Q 80 D 75

80 + 3.38 = 83.38

The design adds 3.38 points to Q.

Above-average work in a demanding design. Quality and design align, so the adjustment moves toward its maximum, 7.5.

From Q to S, step by step

Five steps take the quality score to the adjusted score. The arithmetic follows the point on the instrument above, so move it and read the steps again.

Q = 80 D = 75

The formula

S = clip(Q + Δ · r, 0, 100)

r = s · m + b · (1 − m)

  1. 1

    Start with the quality score.

    Q is the review's verdict on the work. The next four steps shift it by at most 7.5 points either way.

    Q

    80

  2. 2

    How demanding was the study design?

    50 is the midpoint of the design weight. Above it the design counts for the paper, below it against. The further from 50, the larger the shift.

    Δ = D − 50

    +25 = 75 − 50

  3. 3

    Do quality and design tell the same story?

    They align when a demanding design is done well, or an undemanding one done poorly. m is near 1 when they align and near 0 when they do not.

    m = Q / 100

    0.80 = 80 / 100

    D is 50 or more

  4. 4

    How strongly should the design count?

    Alignment earns the strong rate, s. Misalignment earns the gentle rate, b, half as strong. m blends the two into one rate.

    r = s · m + b · (1 − m)

    0.1350 = 0.15 × 0.80 + 0.075 × 0.20

  5. 5

    Apply the shift.

    The distance from the midpoint, at that rate, is the shift. Add it to Q and keep the result between 0 and 100.

    S = Q + Δ · r

    83.38 = 80 + 3.38

    +25 × 0.1350 = +3.38

The tiers

  • 85–100PlatinumExceptional work whose claims withstand scrutiny.
  • 70–84GoldStrong work whose claims hold up, bar a few minor points to address.
  • 60–69SilverBroadly sound work, though with substantive points still to resolve.
  • 50–59BronzeLargely sound work, but with issues that temper its claims.
  • 30–49CobaltNotable weaknesses that weigh on the strength of its claims.
  • 10–29TinDeeper problems that leave several of the central claims undersupported.
  • 0–9FlawedFundamental issues that undercut the paper's central claims.
Fig. 4The seven tiers as a reading of the 0 to 100 adjusted score.

See it on your own paper.