About this toolbox

App version 1.0.0-pilot.6 · item bank v2.0 (798 items, under expert review)

What this is

The GenAI Healthcare Evaluation Toolbox turns the question "is our generative-AI application good enough for healthcare?" into a structured, evidence-based, exportable evaluation plan. In four steps it captures a plain-language description of an application, screens it with three gateway questions (need, benefit, risk), tailors a roadmap of evaluation items selected from a literature-derived item bank spanning ten dimensions — from technical performance to ethics, law, economics, environmental sustainability and patient involvement — and produces a numbered checklist you can complete, export and audit.

The item selection is not a black box: a transparent term-matching (TF-IDF) model, running entirely in your browser, ranks the item bank against your description, and you accept or reject every item. No generative AI writes your plan, and nothing you type leaves your device unless you opt into the research.

Who runs it

The toolbox is developed and maintained by Associate Professor Laura-Maria Peltonen, University of Eastern Finland, who is the principal investigator and data controller for the associated research (see the privacy notice).

It is developed within the ITEA 4 project 22021 PROFIT, with funding from Business Finland (grant 3964/31/2024).

Scientific basis

The evaluation items were systematically distilled from published evaluation frameworks, reporting standards, regulation, and domain literatures — including FUTURE-AI, the WHO guidance on ethics and governance of AI for health, the EU AI Act, NASSS, TEHAI, QUADAS-AI, TRIPOD-LLM and the sources listed below. Each item is traceable to its source, tagged to one of ten evaluation dimensions and one of three levels (individual, organisational, system).

Item bank v2.0 contains 798 candidate items and is currently undergoing formal content-validity assessment by an expert panel (item-level and scale-level content validity indices, inter-rater reliability). A reduced, validated item set will ship as a new versioned release; every generated plan and every research record carries the exact app and item-bank versions used, so results remain interpretable across versions. The toolbox itself is being evaluated in a phased research programme (usability, field pilot, live observational study, psychometric validation).

How to cite

Peltonen L-M. GenAI Healthcare Evaluation Toolbox [software], version 1.0.0-pilot. University of Eastern Finland; 2026. Available from: https://genaieval.org

A methods and validation paper is in preparation; until it appears, please cite the toolbox directly as above.

Sources behind the item bank

Items in the bank carry a short source tag (visible via ⓘ on any item). The tags map to the following published works:

ISO/IEC 42001:2023 AIMS

Reporting standards cluster (MI-CLAIM, CONSORT-AI, SPIRIT-AI, TRIPOD+AI, DECIDE-AI, STARD-AI)

EU AI Act (Reg 2024/1689)

TRIPOD-LLM (Gallifant/Bitterman 2025)

FUTURE-AI (Lekadir 2025)

NICE ESF + HTA Core Model/EUnetHTA + Drummond/Wolff economics

NASSS + NASSS-CAT + CFIR

CHAI Responsible Health AI Framework + Assurance Reporting Checklists

Domain 7 Economic Viability synthesis (Drummond et al. methods; Wolff 2020 & Voets 2022 systematic reviews of AI/ML economic evaluations; Bélisle-Pipon HTA-for-AI; NICE Evidence Standards Framework economic tier)

Domain 4 deep extraction — Data protection, liability & IP for GenAI in healthcare (GDPR, EU AI Act, MDR, professional-liability literature, IP/training-data provenance)

Environmental sustainability tools (Green Algorithms, ML CO2 Impact, CodeCarbon, Strubell 2019, Luccioni 2023, Patterson 2021, NHS Net Zero)

Domain 10 — Patient Engagement and Involvement (INVOLVE/NIHR PPI, WHO Ethics & Governance of AI for Health, FUTURE-AI, TRIPOD-LLM)

Model Cards (Mitchell 2019) + Datasheets (Gebru 2021) + Model Facts (Sendak 2020)

TEHAI (Reddy 2021)

WHO Ethics & governance of AI for health (2021)

QUEST framework (Tam et al., npj Digital Medicine 2024)

CREOLA clinical-text hallucination/omission framework (npj Digital Medicine 2025)

Domain 1 — Stakeholder Co-Design and Involvement (Madaio 2020 co-design checklists; CBPR; INVOLVE; NASSS adopter-system)

Domain 5 Societal Impact synthesis (Topol 2019; Obermeyer 2019; NASSS Greenhalgh 2017; WHO 2021 ethics & equity; HEIA; workforce-impact assessment)

Domain 9 Cultural and Contextual Adaptation (FUTURE-AI Universality; NASSS; cross-cultural adaptation methods; KU cultural-appropriateness)

QUADAS-AI + QUADAS-2 (Sounderajah 2021; Whiting 2011)

Glossary

Item / item bank
A single evaluation criterion phrased as a checkable statement; the bank is the full curated collection (798 in v2.0).
Dimension
One of ten evaluation domains: stakeholder co-design, technical performance, ethics, legal & regulatory, societal impact, environmental sustainability, economic viability, long-term sustainability, cultural adaptation, patient engagement.
Levels (Individual / Organisational / System)
Whether an item concerns individual users and patients, the deploying organisation, or the wider health system.
Gateway screening
The three-question triage (need, benefit, risk) producing STOP, PROCEED WITH CAUTION, or PROCEED before any detailed evaluation effort is spent.
Relevance model / TF-IDF
A classical, deterministic text-matching technique that weights terms by how distinctive they are; used here to rank items against your description — inspectable, not generative.
Content validity / CVI
The degree to which an instrument's items relevantly cover the construct, quantified by expert-panel Content Validity Indices at item (I-CVI) and scale (S-CVI) level.
Model drift
Degradation of an AI system's real-world performance over time as data, practice, or populations change.
FUTURE-AI
International consensus guideline for trustworthy and deployable AI in healthcare (fairness, universality, traceability, usability, robustness, explainability).
NASSS / NASSS-CAT
Framework (and its assessment tools) for why health technologies are not adopted, are abandoned, or fail to scale, spread and be sustained.
CFIR
Consolidated Framework for Implementation Research — determinants of successful implementation in health services.
TEHAI
Translational Evaluation of Healthcare AI framework (capability, utility, adoption).
QUADAS-2 / QUADAS-AI
Quality-assessment tools for diagnostic accuracy studies, with an AI-specific extension.
TRIPOD+AI / TRIPOD-LLM
Reporting guidelines for prediction-model studies, extended for machine learning and for large language models.
CONSORT-AI / SPIRIT-AI
Reporting extensions for clinical trials (and their protocols) involving AI interventions.
DECIDE-AI
Reporting guideline for early, small-scale clinical evaluation of AI-based decision support.
STARD-AI
Reporting standard for AI-centred diagnostic test accuracy studies.
MI-CLAIM
Minimum information standard for clinical AI model reporting.
QUEST
Framework for human evaluation of large language models in healthcare (npj Digital Medicine, 2024).
CREOLA
Framework for classifying hallucination and omission errors in AI-generated clinical text (npj Digital Medicine, 2025).
CHAI
Coalition for Health AI — responsible health-AI framework and assurance reporting checklists.
ISO/IEC 42001
International standard for AI management systems (AIMS) in organisations.
EU AI Act
Regulation (EU) 2024/1689 establishing risk-based rules for AI systems in the European Union.
MDR
EU Medical Device Regulation 2017/745 — governs software that qualifies as a medical device.
GDPR / DPIA
EU General Data Protection Regulation, and the Data Protection Impact Assessment it requires for high-risk processing.
NICE ESF
NICE Evidence Standards Framework for digital health technologies, including its economic-evidence tiers.
HTA / EUnetHTA
Health Technology Assessment and the European HTA Core Model for structured technology appraisal.
PPI / INVOLVE
Patient and Public Involvement in research, per the UK NIHR INVOLVE tradition.
Model cards / datasheets / model facts
Structured transparency documentation for models, datasets, and clinical model deployments.
HEIA
Health Equity Impact Assessment — structured appraisal of how an intervention affects health equity.
CBPR
Community-Based Participatory Research — co-design methodology sharing power with affected communities.

Support & contact

Questions, feedback, or problems: laura-maria.peltonen@uef.fi. During the pilot we aim to respond within five working days.

← Back to the toolbox · Privacy notice