Review risk-of-bias decisions before generating structured assessments
This guide explains the complete ROB Master workflow, from preparing study files and reviewing AI assessment recommendations to processing studies, downloading outputs, and using the audit timeline.
RoB 2 overall proposals are Low when all five domains are Low, High when any domain is High, and otherwise Some concerns for complete assessments. Two or more Some concerns domains trigger a combined-impact review, not automatic High. Before confirmation, justify retaining Some concerns or changing to High because the concerns together substantially lower confidence. Incomplete assessments remain pending. Older saved ratings keep their original rule; use Reassess saved answers in Review to create a new revision without another AI call.
What ROB Master does
Upload your papers and review the checklist recommendations. Each main report has one card with its supporting files and readiness. Click Process Files. For each new RoB 2 study, paste your review protocol once in the box at the top. With individual result review off (the default), each question receives the protocol, full report and supplements. There is no separate result-identification step and one assessment is created per paper.
Protocol mode is a study-level adaptation of RoB 2. The model considers protocol-relevant outcomes and explains material differences. Conflicting responses remain unresolved for review instead of being combined silently. Turn on individual result review to use the result-specific workflow and choose a reported result before processing.
Each new study contributes at most one automatic assessment. The estimated credit total includes one assessment per study awaiting identification and all existing saved selections. Supporting files add no assessment charge. Existing assessments and prior selections are preserved; inspect or remove an unprocessed saved selection in Assessment details.
Saved identification attempts are reused. Stale or uncertain attempts require an explicit Retry primary-result identification action, which may make another AI request. No paid retry is automatic. The two digital NICE checklists remain study-level assessments and require no outcome selection.
Detailed manual scope forms and Add another result are hidden in the standard workflow. The pasted protocol or selected result remains attached to each saved assessment and output. Results & Downloads remains accessible while other selections or jobs are pending.
Check files shows the main report, supporting documents, extracted text length and available page counts. Routine notes, such as embedded figures or an unavailable independent page count, do not require confirmation. Actionable issues, such as very short text or unresolved tables, appear in one batch review. Accept the listed issues only after reviewing them; changing sources requires a new review. Ready studies can process while other studies remain on hold. Missing text, failed extraction or known missing pages cannot be overridden. Properly attached figure supplements remain allowed.
Answer choices are restricted by question in the provider request and checked again by the server. Invalid model output is still possible and remains a processing error. Unresolved means no rating was supplied; it is not another answer option. NR means genuinely not reported after checking the appropriate evidence, and continues the assessment. NA requires actual inapplicability. Missing or incorrect source material must be corrected, not labelled NR across an entire paper. Summaries are never filled automatically to finish.
NICE Quantitative intervention uses a study-level digital assessment with 27 questions. Select the main study and its supplements, then Process Files. Its separate results tab uses ++, +, −, NR (not reported) and NA (not applicable) for items. Internal and external validity summaries allow only ++, + and −; no combined score is calculated. Failed or unfinished assessments remain unresolved. Editing an item clears unchanged summary proposals for reassessment. NICE Word, Excel and JSON downloads use the saved revision and do not include a combined judgement graphic. This pilot remains experimental.
In the RoB 2 tab of Results & Downloads, an Excel-style table shows one row per assessed result and questions across the columns. The same study can have multiple rows for different outcomes. Supplements support their main study; they do not need a separately charged assessment. Click an answer to read the complete question, reasoning and original AI evidence. Use Download ZIP beside Review in each saved digital assessment row to download that exact revision. Use the solid checklist ZIP buttons in the lower Download Results card for the displayed saved assessments in each checklist, regardless of the selected table tab. It includes one combined Excel workbook with a row per assessment and one combined Word document with a section per assessment. Individual row downloads remain per assessment. Incomplete outputs are labelled as drafts. Other / historical results in that card contains historical or other-checklist ZIPs; Input ZIP is separate. Use Edit answers only when a correction is needed, enter a reason and save. The Word, Excel and judgement graphic downloads use those same saved answers. Editing or downloading does not call AI again. In Edit answers, fields that need input are marked for confirmation. You can correct one question and save an incomplete draft. Reason for changes is required to record why you changed it; the message beside Save changes explains any blocker. Switching questions keeps your edits and reason. Save or Cancel edits before changing the saved revision. Revisions include assessment setup, processing started and the AI assessment, so revision 3 can be the first AI output rather than a third manual edit. History keeps the original numbers and shows each saved purpose. Needs attention means incomplete or failed processing, not a high risk-of-bias judgement. Editing creates another revision and clears prior review confirmation. Confirming your review does not validate the method: legacy question/scoring discrepancies remain under review, while the former three-domain automatic High rule has been removed for new assessments. Only parallel-group trials and assignment effects are supported. Existing ZIPs remain historical file-only outputs and are not imported as reviewed answers.
An uncertain AI response requires attention and is not silently submitted again. Completed proposals settle the reservation; failed or uncertain runs release it. Queued assessments start automatically as capacity becomes available. Safely interrupted jobs continue from saved progress without repeating recorded questions or reserving another credit unit. No manual resume is needed. Overall progress shows processed assessments and counts for completed, processing, queued and needs-review results. Uncertain responses still require review. Recorded counts can include a saved error response; read its question-specific explanation. Results remain accessible while another assessment is pending. Scroll horizontally inside the answer table. No-information answers, unanswered questions, skipped questions and processing errors remain distinct.
One assessment unit is reserved for each confirmed result when Process Files is clicked; the price and total for new digital results are shown beforehand. Preparing AI result suggestions does not reserve another assessment unit. Other checklists still produce their usual document outputs, separately from the digital RoB 2 table. RoB 2 is not sent through both processing paths. Do not rerun an old project solely to make its historical ZIP appear as editable answers.
If the digital checklist cannot load, ask your administrator to check the service connection. If a document says Extract text first, finish its text extraction before requesting suggestions. If the AI identifies an unsupported design, do not relabel it as a parallel trial to proceed. A Needs attention result requires source review; do not treat missing answers as low risk.
ROB Master helps teams prepare risk-of-bias and quality-assessment outputs for systematic literature reviews. You upload or select study documents, let the system recommend an assessment family and scale, review those recommendations, then generate structured assessment results for download.
The tool is designed for researcher control. AI can classify documents and prepare assessment outputs, but the user-reviewed ROB type and scale are the decisions that drive processing.
| What you provide | What ROB Master prepares | What you receive |
|---|---|---|
| Project title and study documents | File readiness and extracted text status | Input ZIP for traceability |
| Main publications and optional supplements | AI ROB type and scale recommendations | Reviewable recommendation table |
| User-confirmed assessment choices | Grouped processing and final output generation | Result ZIP with structured outputs |
Prepare sources and choose what to assess
Upload the full main article and attach its supplements to that report. A figure-only PDF is not automatically a full article. Supporting documents contribute evidence to the same assessment; they do not create another assessment charge.
Use Check files to see each filename, its main or supporting role and extracted text. A red little-text warning asks you to inspect the files; it does not prove the study is invalid. Correct a mistaken main/supplement association in Upload Documents. Missing or incomplete extraction can block processing. Routine extraction notes do not need acknowledgement. Specific source issues can be reviewed together for the batch; ready studies can proceed while others remain on hold.
Digital question-by-question review is available for RoB 2, NICE Quantitative intervention, and NICE Quantitative studies reporting correlation and association. RoB 2 currently supports parallel-group randomized trials and the effect of assignment. NICE uses one study-level assessment without an outcome picker. Other available checklists retain the Word/Excel document workflow. GRADE and ROB-ME are not available as digital implementations and are not planned in this release.
For correlations and associations, choose NICE → Quantitative studies reporting correlation and association, check the main report and supplements, then Process Files. There is no outcome picker. Its 19 questions use native item ratings and separate internal/external validity summaries. Displayed question 4.4 corresponds to published Appendix G question 4.6 on precision of association; no questions have been added to fill the numbering gap. The UK setting question is retained. Apply NA only with a reason why that item does not apply; NR describes genuine nonreporting, not a processing failure.
- In Recommendations, review the checklist assigned to each main report.
- For RoB 2, paste your protocol and click Process Files. One protocol-guided assessment is created per paper, without result-selection approval.
- If individual result review is enabled, select the reported result before processing. In protocol mode, review outcome conflicts after processing. Source warnings remain available in one batch review.
- Check the selected assessment count and quoted total before Process Files. For example, three assessments at two credits each total six credits. Selecting outcomes alone does not start assessment generation or reserve its credits.
Trace an answer back to its saved source
Click a table answer to open that question directly, or use Review and the Question selector to reach any question. Where available, Original AI answer and evidence shows the original proposal, source filename and saved supporting excerpt. Main articles and supplements remain separately attributed.
Verified evidence means the excerpt was associated with the supplied source text. It does not guarantee that the interpretation, study design or scientific judgement is correct. Compare the quotation with the full paper and relevant checklist guidance. Page or section references should only be relied upon when actually recorded; the export does not invent them.
A valid NICE NR response can mean the information was not reported and may legitimately have no quotation. Missing input, an unresolved judgement, an invalid response and a processing failure are different states. Do not replace them with a positive answer just to finish the checklist. Original AI evidence stays read-only; a revised answer and its rationale are recorded separately in history.
Edit a draft and understand its progress
Saved results remain reachable while other jobs are pending. Resume appears only for jobs the system can safely continue. It is not offered for an active request or an uncertain completed attempt; avoid repeated processing to work around an error. Needs attention describes an incomplete or failed workflow, not a high risk-of-bias judgement.
- Open Review, select the question and read its native answer choices, rationale and evidence. NICE summaries accept only ++, + or −; item questions also allow NR and NA. Do not substitute a universal numeric score.
- Select Edit answers and correct the necessary answers or rationale. Fields needing input are marked; evidence remains read-only.
- Enter Reason for changes. This required audit explanation is separate from the rationale for an individual answer. The message beside Save changes explains any blocker.
- Save a valid incomplete draft even if other questions still need review. Switching questions retains your pending edits and change reason. Cancel edits restores the saved values; save or cancel before changing revisions or exporting that assessment.
- Confirm my review becomes available only when the saved assessment is ready. For NICE, changing items can require reassessing the internal and external validity summaries. Confirmation does not validate the methodology.
Revision numbers count saved stages, not just manual edits. For example, revision 1 can be setup, revision 2 processing started and revision 3 the first AI output. History identifies the purpose and available date, actor and change reason while retaining earlier saved results.
Choose an assessment ZIP or a checklist ZIP
Download ZIP beside Review exports one saved assessment. The lower Download Results card has solid buttons for each available checklist, independent of the selected tab. A checklist ZIP contains one combined Excel workbook, one combined Word document, the saved assessment JSON and a contents record. The template header is retained and each assessment fills the next data row. The displayed count covers the loaded eligible latest saved rows, up to 500 per download, and may not represent every record in a large project.
Incomplete saved outputs are labelled drafts. Queued and processing placeholders are not complete downloads. A checklist export fails visibly if one member cannot be exported; it does not silently omit that assessment. Save or cancel pending edits first. Editing and downloading saved results do not run AI again.
Word retains the existing answer and full saved rationale in the main assessment tables, followed by concise references such as E1 and E2. A Supporting evidence table after the assessment tables lists each reference, question, friendly source filename and full verified quotation. Long saved rationale is preserved, not automatically summarized. Excel retains its separate Verified evidence column.
Clean exports use friendly source names and the main report's leading numeric file ID: 2868_Ren_2021.pdf displays 2868. Leading zeros are retained; files without a reliable numeric prefix have no invented ID. Different outcomes may share that file ID while remaining separate assessments. Internal development banners are omitted from clean reports, while the original audit information is retained separately.
Input ZIP remains in its own card. Other / historical results retains existing file-based outputs. Some older versions can only offer an explicitly labelled original-format cached ZIP, if one exists; it may retain older labels and banners. Do not rerun AI solely to change export formatting.
If something looks wrong
- Very little text: open Check files, inspect the full report and correct source associations or extraction before acknowledging.
- No selectable RoB 2 outcome: check the design, missing reported details and evidence. Refresh recommendations explicitly only when appropriate; not every suggested result is eligible.
- Save disabled: read the visible blocker, especially the required change reason. A missing confirmation answer does not by itself prevent saving a valid draft.
- Needs attention: inspect the recorded question and explanation. Preserve the saved output and review missing information; do not assume Resume or retry is always safe.
- Export fails: read the error in Download Results. An unsupported historical version may require its original cached export. If unavailable, retain the assessment and report the failure rather than regenerating answers.
Example walkthrough: upload a trial and its supplement, check their text, choose RoB 2, request outcomes and select the reported 12-month comparison. Review the count and credits, process, then open an answer to inspect its source quotation. Correct one answer with a change reason, save the draft, complete the remaining review and download either that assessment or the RoB 2 checklist ZIP from Download Results.
Digital assessments remain experimental and require scientific review. Evidence association and reviewer confirmation do not establish clinical or statistical correctness. The latest interface and Word-layout changes have source-level checks and regression fixtures; runtime and visual acceptance checks remain pending. Other checklist adapters and broader RoB 2 designs are not implied by the existing document catalogue.
The complete workflow
ROB Master separates file preparation, recommendation review, and final processing so each high-impact decision can be checked before credits are spent on assessment generation.
- 1
Create or open the project
Start a standalone ROB project or open a ROB stage linked to a master project.
- 2
Upload or select study files
Add direct upload files, or select documents from Workflow PDF Manager when using a master project.
- 3
Confirm text readiness
Wait until text extraction is completed or skipped for every required file.
- 4
Get AI recommendations
Ask ROB Master to suggest the assessment family and scale for each main study file.
- 5
Review user selections
Accept or override AI choices so every processable file has the correct user ROB type and scale.
- 6
Process and download
Generate structured ROB outputs, monitor grouped progress, and download the result package.
Prepare study files before requesting recommendations
ROB Master accepts PDF, TXT, DOC, and DOCX files. Direct uploads support up to 100 files, with each file limited to 20 MB. In a master project, files are selected from Workflow PDF Manager under the ROB tab rather than uploaded directly inside the ROB page.
Use predictable file names
Main study files should follow a study ID, author, and year pattern such as 7158_Viswanath_2022.pdf. The ID helps the system and reviewers keep publications, supplements, recommendations, and outputs aligned.
| File type | Accepted pattern | Purpose |
|---|---|---|
| Main publication | ID_Author_Year.pdf | Creates the main row for AI recommendation and processing. |
| Supplement | ID_Author_Year_SUPP1.pdf, SUPPL, APPENDIX1, or DATASET1 | Provides additional evidence for the matching main publication. |
| Master project file | Selected in Workflow PDF Manager | Uses the master project's linked document set for ROB. |
Wait for text extraction to finish
After files are uploaded or selected, ROB Master extracts text and tracks a status for each file. AI recommendations can start only when required files are extracted or intentionally skipped.
| Status | Meaning | User action |
|---|---|---|
| Pending | Text extraction has not started for the file. | Wait for automatic extraction or start it when available. |
| Queued | The file is waiting for extraction capacity. | No edit is needed unless the queue remains blocked. |
| Processing | The system is extracting text from the document. | Wait for the file chip to update. |
| Retrying | A temporary extraction issue occurred and retry is active. | Wait, or use retry if the file later fails. |
| Extracted | Text is ready for recommendation and processing. | Continue once every required file is ready. |
| Failed | Text extraction could not complete. | Use retry where available or replace the source document. |
Review AI ROB type and scale recommendations
The recommendation step classifies each main study file into an assessment family and, where required, a specific scale. These are suggestions. The user ROB type and user ROB scale are the values used for final processing.
| Assessment family | Available scale choices | Common use |
|---|---|---|
| NICE | Quantitative intervention; quantitative correlation and association; qualitative; economic | Digital study-level review for intervention and correlation/association; qualitative and economic retain document outputs. |
| Newcastle-Ottawa | Cohort-Studies; Case-Control | Observational cohort and case-control study assessment. |
| JBI | Analytical cross sectional, case control, case reports, cohort studies, economic evaluations, prevalence studies, qualitative research, RCT, text and opinion | JBI critical appraisal across multiple study designs. |
| Drummond's scale | No separate scale required | Economic evaluation quality assessment. |
| Downs and Black scale | No separate scale required | Health-care intervention study quality assessment. |
| Cochrane Risk of Bias | ROB v2; ROBINS-I | Randomized trials or non-randomized intervention studies. |
| AMSTAR 2.0 | No separate scale required | Methodological quality assessment for systematic reviews. |
Edit one file or apply choices in bulk
Recommendation cards show the file name, AI type, AI scale, user type, user scale, and AI reason. Use the reason to decide whether the classification is acceptable, then edit individual rows or select multiple rows and apply a bulk type or scale.
| Control | Use it when | Effect |
|---|---|---|
| Select All / Unselect All | You want to apply the same assessment choice to many studies. | Changes which rows receive bulk edits. |
| User ROB Type | The study needs a different assessment family than AI suggested. | Resets the scale choices to those valid for that type. |
| User ROB Scale | The assessment family has multiple supported scales. | Stores the scale used for final processing. |
| Retry | A file recommendation failed. | Requests a new recommendation for the failed file. |
Files skipped from recommendation
Files that exceed the ROB recommendation page limit can be marked as skipped. Skipped files do not require a user type or scale and are not included in the processable recommendation count.
Process files after selections are valid
Once every processable file has a valid user selection and no recommendation is still pending, queued, or processing, ROB Master can generate the assessment outputs. The project may enter queued or processing states depending on worker capacity.
| Project phase | Meaning | What to watch |
|---|---|---|
| Recommendations-ready | AI suggestions are available for review. | Missing user type or scale warnings before processing. |
| Queued | The project is waiting for processing capacity. | Queued status and group progress once available. |
| Processing | Assessment generation is running. | Completed groups, failed groups, and estimated remaining time. |
| Finalizing | Completed groups are being merged into final outputs. | Merge, graph generation, and download preparation status. |
| Completed | Final outputs are ready. | Review decisions, input ZIP, and result ZIP. |
| Failed | Recommendation, processing, or finalization encountered an error. | Error messages and retry actions. |
Download inputs, results, and review decisions
Completed ROB projects provide a review-decision summary, an input ZIP, and a result ZIP. The summary helps confirm which AI recommendations were accepted and which user choices were applied.
| Output | What it provides |
|---|---|
| Review Decisions | Table of AI ROB type, AI ROB scale, user ROB type, and user ROB scale for each file. |
| Input ZIP | A downloadable package of the input files associated with the project. |
| Result ZIP | The generated ROB assessment outputs prepared by the processing workflow. |
| Timeline | Project history for uploads, AI recommendations, manual overrides, processing, and output events. |
Use the timeline to understand project history
The ROB timeline records project lifecycle events, file activity, recommendation events, manual overrides, processing groups, output preparation, actors, and errors. It is useful when a team needs to understand what happened after a refresh, retry, or handoff.
| Filter | Use it to find |
|---|---|
| Phase | Project, upload, recommendation, processing, or output events. |
| Category | Lifecycle, setup, file, or process events. |
| Search | Titles, summaries, actors, file names, result paths, and action names. |
Review decisions before processing
| Do | Avoid |
|---|---|
| check_circle Confirm the study design before accepting an AI recommendation. | cancel Process every AI suggestion without reviewing the reason. |
| check_circle Keep supplementary files tied to the matching main publication. | cancel Upload supplements as unrelated main study files. |
| check_circle Use bulk edits only for studies that truly share the same assessment choice. | cancel Apply one ROB type to mixed study designs. |
| check_circle Resolve missing user type or scale warnings before processing. | cancel Treat AI type and scale as enough when user fields are empty. |
| check_circle Use the timeline to investigate retries, overrides, and output issues. | cancel Rely only on the current stepper state after a long-running job. |
| check_circle Download both input and result packages when archiving a project. | cancel Keep only result files without the source-file context. |
Before starting processing
- check The project title clearly identifies the review or assessment set.
- check Every main file follows the expected ID, author, and year naming pattern.
- check Supplementary files share the same study identity as their main publication.
- check Every required file has completed or skipped text extraction.
- check AI recommendation reasons have been reviewed for questionable files.
- check Every processable file has a valid user ROB type.
- check Every non-optional assessment type has a valid user ROB scale.
- check Skipped files and failed files are understood before final processing starts.