How a paper is scored.

Every paper receives a single adjusted score between 0 and 100. That score combines two independent judgements: the quality of the work itself, and the evidential weight that the study design carries for the paper's claims. This page explains how each is built, how they combine, and why the combination is structured the way it is.

Quality, 0 to 100

A review-based assessment of how well the paper is executed: rigour, clarity, transparency, and the validity of its conclusions. Built from issues (scored by criticality) and merits (scored by calibre) across eight categories.

Design weight, 0 to 100

The evidential weight that the study design carries for the paper's claims: how well those claims are shielded from alternative explanations. What counts as strong evidence depends on the field.

Adjusted score, 0 to 100

The two ratings combined through an interaction-aware adjustment. Quality remains the dominant signal; the design weight nudges it up or down by at most a few points, depending on how well the two agree.

1Quality: the Q score

Q reflects how well the paper itself is executed, independent of what kind of study it is. It is built up from two parallel streams: the issues a reviewer raises, each scored by criticality, and the merits a reviewer identifies, each scored by calibre.

The numbers are a navigation aid. A single high-criticality issue may matter more than a dozen minor ones, and a single exceptional-calibre merit may justify a paper that has noticeable issues elsewhere. The written reasoning that accompanies each score is where the real evaluative work lives.

Criticality

Issues · 0-10

How seriously an issue undermines the work. Not how hard it is to fix, and not how prominent it is in the paper, but how much it bears on the validity and integrity of the contribution.

Negligible.A trivial concern. Worth flagging for completeness, but essentially no bearing on whether the work stands up. Typographical issues, minor stylistic preferences, small omissions a careful reader would barely notice.

Moderate.A genuine problem that a reasonable reviewer would expect to see addressed. The work is not invalidated, but the issue meaningfully weakens some claim, conclusion, or aspect of methodology, and a thoughtful response or revision is warranted.

Fatal.A defect severe enough that the work, as it stands, cannot be relied upon: a fundamental flaw in design, a breakdown in the chain of evidence, an analytical error invalidating central conclusions, or an ethical breach. No amount of polish can rescue the paper without substantive rework.

Calibre

Merits · 0-10

How impressive or valuable a particular strength is beyond the norms of the field. Calibre is the positive mirror of criticality: it captures what the paper does well, and how exceptionally.

Negligible.A strength so minor it barely registers. The work does the thing competently, but at a level that wouldn't distinguish it from any other adequately executed paper in the field.

Moderate.A solid, real merit. The work does something genuinely well, clearly above baseline competence, in a way a reviewer would want to call out and a reader would benefit from noticing.

Exceptional.A standout strength. A methodological move that sets a new bar, evidence of rare quality, an insight that reshapes how a question is approached, or execution at the very top of what the field produces.

Each issue and each merit is filed into one of eight categories, so the shape of a paper's strengths and weaknesses is legible at a glance.

01
Research Design
The foundational architectural decisions: choice of methodology, study structure, selection of participants or materials, and whether the overall approach is fit for the question being asked.
02
Data and Evidence
The raw material the work rests on: the soundness of data collection, how evidence was assembled, how things were measured, and the quality of underlying sources.
03
Analytical Approach
What the author does with the data once it is in hand: statistical models, qualitative coding, computational pipelines, formal reasoning, or argumentative structure, both choice and execution.
04
Scholarly Grounding
How well the work situates itself in its field: engagement with relevant literature, strength of theoretical underpinnings, proper crediting of prior work, and a clear sense of where the contribution sits.
05
Reporting Quality
Transparency and completeness: is there enough detail (methods, data, code, justification) for a reader to understand, evaluate, and (where applicable) reproduce the work?
06
Interpretive Rigor
The leap from results to conclusions: are claims supported by what was found, limitations honestly acknowledged, claims appropriately hedged, and have plausible alternative explanations been seriously considered?
07
Ethical Conduct
Responsible-research dimensions: ethical treatment of subjects, informed consent, declared conflicts of interest, data integrity, and adherence to the broader norms of responsible conduct.
08
Contribution
What the work actually adds. A paper can be technically impeccable and contribute little; another can be rough around the edges but advance understanding in important ways.

2Design weight: the D score

D reflects the evidential weight a study design carries for the paper's claims. What counts as strong evidence depends on the field: resistance to confounding in medicine, logical validity in mathematics, binding authority in law, community legitimacy in Indigenous studies. The sections that follow show the hierarchy applied to several research domains, and why that hierarchy is the right one for that kind of knowledge.

What the hierarchy measures: how well a study design shields its claim from being wrong. The hierarchy ranks one thing: epistemic warrant for the claim type of that field, given the study design.

What it does not measure: the novelty, importance, or contribution of the work. A landmark qualitative study can matter more than a mediocre RCT. D speaks to how well a claim is shielded, never to how much the claim is worth.

D is not a value judgement. Lower on the hierarchy does not mean worse research; it means a different kind of warrant, often the strongest available for that question. A score of 3 in ecology is not “mediocre”; it means the paper uses a natural experiment or quasi-experimental design, which is appropriate and often the only ethical option for many ecological questions.

The core question differs by field

Before examining individual schemas, understand that different fields ask fundamentally different questions, and that determines which hierarchy is appropriate.

Field	The question it asks	Hierarchy applied
Clinical / Health	“Does X cause Y in humans?”	Hierarchy of causal strength
Mathematical Sciences	“Is X necessarily true?”	Hierarchy of proof rigour
Physical / Chemical Sciences	“Can X be measured reliably?”	Hierarchy of experimental reproducibility
Social / Economic	“Does X generalise?”	Hierarchy of internal + external validity
Humanities / Interpretive	“Is the reading of X warranted?”	Hierarchy of argumentative quality
Law	“Is X binding?”	Hierarchy of legal authority
Engineering	“Does X work in practice?”	Hierarchy of validation rigour
Indigenous Studies	“Is X legitimate to this community?”	Hierarchy of relational / cultural validity

A note on schema labels

In the list of research domains below, every discipline is assigned exactly one of three labels indicating where its hierarchy comes from.

Formal

An existing, named, published schema applied exactly as defined. The schema name and primary source are cited verbatim.

Adapted

An existing named schema has been adapted for this field. All level-name mappings from the original to the adapted version are shown explicitly.

Informal

No published schema exists for this field. A hierarchy has been constructed from discipline norms to reflect the epistemic standards of the field.

Expand any domain to see its disciplines and the applicable hierarchy. Where disciplines within a domain use different schemas they appear as separate blocks.

23 domains

3The adjusted score: combining Q and D

The quality score Q and the design-weight score D are combined into a single adjusted score S, also in the range 0 to 100, through an interaction-aware adjustment. The construction treats Q and D as two independent signals that interact rather than simply summing: well-executed work in a demanding design is rewarded, and poorly-executed work in a forgiving design is penalised, while off-diagonal cases (high Q with low D, or low Q with high D) receive only modest adjustments.

Definitions

\Delta = D - 50

m(Q, D) = \begin{cases} \dfrac{Q}{100} & \text{if } D \geq 50 \\[6pt] \dfrac{100 - Q}{100} & \text{if } D < 50 \end{cases}

Adjusted score

S = \mathrm{clip}\!\left(\, Q + \Delta \cdot \bigl[\, s \cdot m(Q, D) + b \cdot (1 - m(Q, D)) \,\bigr],\; 0,\; 100 \,\right)

Qreview-based quality score, 0 to 100Dstudy-design weight, 0 to 100Deltacentred design weight, -50 to +50malignment magnitude, 0 to 1s = 0.15interaction strengthb = 0.075baseline strength

What the adjustment does, in plain terms

The adjustment to Q depends on how well Q and D align: they align when both are high or both are low, and pull apart when one is high and the other low. In the formula, the interaction term (s · m) grows as the two align, while the baseline term (b · (1 - m)) carries the off-diagonal cases. The four cases below show which one dominates when.

High quality with a high-weight design (e.g. a well-executed meta-analysis of RCTs): alignment is high, the interaction term dominates, and the paper receives the largest positive adjustment. The system rewards executing a demanding study design well.

Low quality with a low-weight design (e.g. a poorly-executed expert opinion): alignment is again high but in the opposite direction; the interaction term dominates and the paper receives the largest negative adjustment. Doing easier work poorly compounds against the score, since this kind of work should be straightforward to do well.

Off-diagonal cases (high Q with low D, or low Q with high D): alignment is low, the baseline term dominates, and the adjustment is small. A well-executed expert opinion still earns most of its quality credit; a struggling meta-analysis is treated more leniently because the format is genuinely harder.

A neutral design weight. If D = 50, then the centred weight is 0 and the quality score is left unchanged: S = Q.

Worked example: medicine

Using the medical hierarchy as the reference, the table below shows the adjusted score S (and the size of the adjustment in parentheses) for three quality levels: strong (Q = 80), average (Q = 50), and weak (Q = 20), across all five tiers of the design hierarchy.

Study design	D	Strong (Q = 80)	Average (Q = 50)	Weak (Q = 20)
Systematic review / meta-analysis of RCTs	100	86.75 (+6.75)	55.62 (+5.62)	24.50 (+4.50)
Individual RCT or observational with large effect	75	83.38 (+3.38)	52.81 (+2.81)	22.25 (+2.25)
Non-randomised controlled cohort	50	80.00 (0.00)	50.00 (0.00)	20.00 (0.00)
Case-series / case-control	25	77.75 (-2.25)	47.19 (-2.81)	16.62 (-3.38)
Mechanism-based reasoning / expert opinion	0	75.50 (-4.50)	44.38 (-5.62)	13.25 (-6.75)

Bounds. The maximum possible adjustment is +/-7.5 points, occurring at the perfectly aligned corners (Q = 100 with D = 100, or Q = 0 with D = 0). At the perfectly misaligned corners (Q = 100 with D = 0, or Q = 0 with D = 100), the adjustment is exactly half: +/-3.75 points.

Classification: ANZSRC 2020 Fields of Research, Australian Bureau of Statistics & Stats NZ, released 30 June 2020

OCEBM schema: Oxford Centre for Evidence-Based Medicine, Levels of Evidence Working Group (2011). "The Oxford Levels of Evidence 2." cebm.ox.ac.uk

ESSA / WWC schema: US Dept of Education, What Works Clearinghouse Procedures and Standards Handbook v5.0 (2022)

Melnyk & Fineout-Overholt schema: Melnyk, B.M. & Fineout-Overholt, E. (2023). EBP in Nursing & Healthcare, 5th ed. Wolters Kluwer

IPCC schema: IPCC Guidance Note for Lead Authors on the Use of Expert Judgment and Treatment of Uncertainty (2010)

Campbell Collaboration: Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR), current version

APA Div 12: Chambless, D.L. et al. (1998). Update on empirically validated therapies, II. The Clinical Psychologist, 51(1), 3-16

EBSE: Kitchenham, B. et al. (2004). Evidence-based software engineering. Proc. ICSE 2004

AIATSIS: AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research (2020)

Mathematical typesetting: KaTeX

The weight assigned to any study reflects its position in the applicable hierarchy for that field, not the novelty, importance, or contribution of the work.

See it on your own paper. Free.

Upload a paper, free→How it works

1 free review · no card required