R Package Qualification with riskmetric: Assess and Document Package Risk

Assess and document the risk of the R packages a regulated analysis depends on — score maintenance, documentation, testing, and adoption with the R Validation Hub’s riskmetric, and read which metrics evaluate in a sealed validated environment

A practical, runnable guide to qualifying the R packages you use in a regulated (GxP) analysis with riskmetric, the R Validation Hub’s risk-based assessment package. Learn the regulatory question behind it — is R “validated” for clinical use, and how a risk-based approach answers it — then run the pkg_ref() |> pkg_assess() |> pkg_score() |> summarize_scores() workflow on a real installed package (admiral), interpret the per-metric score table and the overall score in plain language, and read honestly which metrics evaluate offline in a sealed validated environment versus which need live CRAN or GitHub. Every assessment block runs here against the installed package, offline.

TipKey takeaways

• You qualify an R package by assessing its risk, not by “certifying” the language. R itself is a tool, not a validated product; the accepted approach is to document that each package you depend on is well-maintained, documented, and tested enough for its intended use.
• riskmetric automates that assessment. It reads a package’s metadata and activity and turns it into a set of 0–1 metric scores — maintenance, documentation, testing, adoption, code size, and dependencies.
• The workflow is one pipe: pkg_ref("admiral") |> pkg_assess() |> pkg_score() produces one score per metric, and summarize_scores() collapses them into a single overall number.
• In a sealed validated environment, the metadata metrics are what you audit. The metrics that need live CRAN or GitHub (downloads, reverse dependencies, remote checks, bug status) return NA offline — which is the honest picture of a qualified GxP environment with no open internet.
• riskmetric is evidence, not a verdict. It gives a documented, reproducible starting point for a package-qualification decision; a human still sets the acceptance criteria and signs off.

Introduction

Your ADaM (Analysis Data Model) pipeline depends on admiral to build the datasets, dplyr to reshape them, and rtables to lay out the tables. At an inspection, a regulator asks the fair question: how do you know those packages are fit for use? You cannot answer “they’re on CRAN” — CRAN hosts packages of every maturity. And you cannot point to an official stamp: R and its packages are open-source tools, not a “validated product” a vendor certifies for you.

The industry’s answer is risk-based qualification. Instead of proving a package is perfect, you assess and document its risk — is it actively maintained, is it documented, is it tested, is it widely used — and decide, on the record, whether that risk is acceptable for how you use it. This is the position of the R Validation Hub, a cross-industry initiative whose white paper on validating R packages lays out a risk-based framework for using R in a regulated setting. GxP here is the umbrella for the “good practice” quality regulations (Good Clinical, Laboratory, and Manufacturing Practice) that govern work submitted to health authorities.

This lesson uses the R Validation Hub’s own package, riskmetric, to turn that assessment into runnable, reproducible code. We assess a real installed package (admiral), read its scores, and — importantly — are honest about which metrics evaluate in a sealed, offline validated environment and which need the open internet.

The eight metadata checks at the top pass cleanly; the three low scores at the bottom are size penalties, not quality failures (more on that below); and six metrics are missing because they need the network. The rest of this lesson is how to produce this, read it correctly, and fit it into a qualification SOP — a standard operating procedure, the written, controlled process an organisation follows.

What riskmetric measures

riskmetric groups its checks into families. Each check reads something about the package and returns a 0–1 score, where higher is lower risk. Crucially for a validated environment, the families split by where the evidence lives: package metadata (shipped inside the installed package, readable offline) versus live signals (CRAN download counts, the reverse-dependency graph, remote check results, the issue tracker) that only exist on the network.

Family	What it asks	Example metrics	Needs network?
Maintenance	Is someone responsible, and is the package actively kept up?	has_maintainer, has_source_control, has_bug_reports_url, has_news, news_current	No — from DESCRIPTION / NEWS
Documentation	Can a user learn and verify it?	has_website, has_vignettes, has_examples, export_help (fraction of exports documented)	No — from the installed help/vignettes
Size & complexity	How much surface area is there to review?	size_codebase, exported_namespace, dependencies	No — from the installed code
Testing	Is it tested and does it pass checks?	covr_coverage, r_cmd_check, remote_checks	Yes — needs source build / remote check results
Adoption & activity	Do others rely on it, and are bugs handled?	downloads_1yr, reverse_dependencies, bugs_status	Yes — CRAN logs, reverse-dep DB, issue tracker

The split is the whole story for a GxP setup: a qualified environment is typically sealed (no open internet), so the metadata families are exactly what it can audit, and the network families are what you either accept as NA or gather separately, once, from a controlled source. See the riskmetric metric reference for the full list and each metric’s definition.

Assess a package

The workflow is a three-step pipe. pkg_ref() names the package to assess; pkg_assess() runs every metric assessment; pkg_score() turns each raw assessment into a 0–1 score. We point pkg_ref() at an installed package (admiral), so riskmetric reads the local library copy — no download, no network.

library(riskmetric)

scores <- pkg_ref("admiral") |>   # an installed-package reference (reads the local library copy)
  pkg_assess() |>                 # run every metric assessment
  pkg_score()                     # convert each assessment to a 0-1 score

class(scores)                     # a named list: one scored metric per element

[1] "list"

length(scores)

[1] 19

pkg_score() returns a named list — one element per metric, each a single 0–1 score carrying a human-readable label. That is convenient to compute on but not to read, so turn it into a tidy table: one row per metric, its score rounded, sorted so the weakest scores surface first.

score_df <- data.frame(
  metric = names(scores),
  score  = round(vapply(scores, function(x) as.numeric(x), numeric(1)), 3),
  row.names = NULL
)

score_df[order(score_df$score, na.last = TRUE), ]

                 metric score
18         dependencies 0.018
12        size_codebase 0.035
6    exported_namespace 0.142
17         has_examples 0.989
2              has_news 1.000
4          news_current 1.000
7         has_vignettes 1.000
8           export_help 1.000
9           has_website 1.000
10       has_maintainer 1.000
13   has_source_control 1.000
14  has_bug_reports_url 1.000
1         covr_coverage    NA
3         remote_checks    NA
5           r_cmd_check    NA
11          bugs_status    NA
15        downloads_1yr    NA
16 reverse_dependencies    NA
19              license    NA

That table is the assessment. Every non-NA row is a documented, reproducible piece of evidence about admiral — and every NA row is a metric that could not evaluate offline, which we read honestly in a moment.

Read the scores

Take the metadata metrics that evaluated (the non-NA rows) and read them in plain language:

• The maintenance and documentation checks pass at 1.0. admiral has a maintainer, a source repository, a bug-reports URL, a NEWS file with an entry for the current version, a website, and vignettes; export_help is 1.0 (every exported function is documented) and has_examples is 0.99 (nearly all help files carry examples). For qualification, that is the good news you want on the record: the package is maintained and documented.
• The three low scores are size penalties, not quality failures. dependencies (0.02), size_codebase (0.04), and exported_namespace (0.14) are logistic ratings where a bigger package scores lower — more dependencies, more lines of code, and more exported functions each mean more surface area to review, which riskmetric treats as higher risk. admiral is a large, many-function package, so it scores low here by construction. This is not “admiral is bad code”; it is “admiral is big, so budget more review effort.” Misreading these three as defects is the most common first-time error.

So the per-metric scores are not a pass/fail list — they are a risk profile. A small, well-documented helper package would show high size scores and possibly weaker documentation; admiral shows the opposite. Qualification is about reading that profile against how you use the package, not chasing every score to 1.0.

The overall score

summarize_scores() collapses the per-metric scores into a single number — a convenient headline for a qualification record.

summarize_scores(scores)

[1] 0.5166239

About 0.52. Read it carefully, because the default weighting is doing something you must understand: summarize_scores() averages across metrics and, by default, the network metrics that returned NA count as missing evidence and pull the number down. So 0.52 is “admiral’s score given that six checks could not run in this offline environment,” not “admiral is a medium-risk package.”

If you instead average only the metrics that actually evaluated — the honest offline evidence — the picture is very different:

offline <- score_df[!is.na(score_df$score), ]

nrow(offline)                        # 12 metrics evaluated offline

[1] 12

round(mean(offline$score), 3)        # mean of the available evidence

[1] 0.765

Averaging the 12 available metrics gives about 0.77 — the same package, scored on the evidence that exists in a sealed environment. Neither number is “the” answer; the point is that an overall score is only as meaningful as the weighting behind it, and in a validated environment that weighting has to account for the metrics that legitimately cannot run. Document which number you used and why.

The metrics that need the network

Six metrics returned NA, and one more (license) is NA for a different reason. Being explicit about this is not a limitation to hide — it is the correct picture of a qualified, sealed environment, and a regulator expects you to know exactly which evidence a metric needs.

Six metrics need live CRAN or GitHub, so they are NA offline:

Metric	What it needs	Why it is NA here
covr_coverage	A source build + running the test suite	No source/tests in a sealed environment
r_cmd_check	Downloading the package and running R CMD check	No download in a sealed environment
remote_checks	CRAN/GitHub hosted check results	No web access
bugs_status	The last-30-bugs closed fraction (issue-tracker API)	No GitHub API access
downloads_1yr	CRAN download logs (cranlogs)	No web access
reverse_dependencies	The CRAN reverse-dependency database	No web access

In an offline validated environment these degrade to NA without error — riskmetric simply records that the evidence was unavailable. In practice you gather these signals once, from a controlled source, during environment setup (for example, a dated CRAN snapshot or an internal package mirror that records download and check data), and file them alongside the offline assessment.

One more NA is not a network problem at all: license. In riskmetric 0.2.7 the license assessment captures the package’s license string offline, but the scoring function is a deliberate stub that returns NA — the metric is captured but not yet scored in this version. So a missing license score is expected behaviour of the package version, not evidence that your environment failed to reach the network. (You still record the license separately; an acceptable open-source license is a standard qualification checklist item.)

How it fits a package-qualification SOP

riskmetric produces evidence; a qualification SOP is the controlled process that turns that evidence into a documented decision. A typical flow:

1. Inventory the packages the analysis depends on (and pin them — see the reproducible-environment lesson, which freezes the exact versions in a committed lockfile).
2. Assess each package with pkg_ref() |> pkg_assess() |> pkg_score(), and archive the score table as an artefact of the qualification record.
3. Interpret against acceptance criteria your organisation sets in advance — for example, “documented exports, a maintainer, and a current NEWS entry are required; a low size score triggers extra review rather than rejection.” riskmetric does not set these thresholds; your SOP does.
4. Supplement the offline gaps with the network evidence gathered from a controlled source, and record the license.
5. Decide and sign off — a human accepts (or rejects) each package for its intended use, and the assessment, the criteria, and the decision are stored together.

The value riskmetric adds is that steps 2 and 4 become reproducible code instead of a manual spreadsheet: re-run the pipe and you regenerate the same evidence, which is exactly the property a validated environment is built on.

🟢 With an AI agent

Ask Prova “how do I assess the packages my ADaM pipeline depends on with riskmetric — run pkg_assess on admiral and rtables, read the metric scores, and explain which ones return NA in a sealed validated environment?” — it answers grounded in this pillar’s lessons, with runnable riskmetric code you can try on installed packages. The runtime is the judge. Ask Prova →

Common issues

Reading the NA network metrics as failures. Six metrics (covr_coverage, r_cmd_check, remote_checks, bugs_status, downloads_1yr, reverse_dependencies) need live CRAN or GitHub and return NA in a sealed, offline environment. That is expected, not a broken assessment — riskmetric ran correctly and recorded that the evidence was unavailable. Gather those signals separately from a controlled source; do not treat an offline NA as a low score.

Pointing pkg_ref() at a remote reference in an offline environment. riskmetric can reference a package from CRAN or a source URL, but those need the network and will fail (or hang) in a sealed environment. To assess what you actually run, reference the installed package — pkg_ref("admiral") reads the local library copy — so the assessment reflects the exact version in your qualified library and needs no download.

Expecting a license score and getting NA. In riskmetric 0.2.7 the license string is captured offline but the license score is a deliberate stub returning NA. A missing license score is the package version’s behaviour, not a network failure and not a problem with your environment. Record the license from the assessment separately; do not chase a score that the version does not produce.

Frequently asked questions

NoteIs R validated for clinical use?

R is a programming language and tool, not a product a vendor “validates” for you — so the question is not “is R validated?” but “have you qualified the R packages you use for their intended purpose?” The accepted approach, set out by the R Validation Hub in its white paper, is risk-based: assess and document each package’s maintenance, documentation, testing, and adoption, and decide whether that risk is acceptable for how you use it. riskmetric automates that assessment. There is no FDA endorsement of a specific package; the sponsor owns the qualification decision.

NoteWhat is riskmetric?

riskmetric is an R package from the R Validation Hub that assesses the risk of using an R package. It reads a package’s metadata and activity and returns a set of 0–1 scores across families — maintenance, documentation, testing, adoption, code size, and dependencies — via the pipe pkg_ref("pkg") |> pkg_assess() |> pkg_score(), with summarize_scores() for an overall number. It gives you reproducible evidence for a package-qualification decision; it does not, by itself, approve a package.

NoteWhat is the R Validation Hub?

The R Validation Hub is a cross-industry initiative (a working group of the R Consortium) that develops the framework and tools for using R in a regulated, GxP setting. Its white paper, “A Risk-based Approach for Assessing R Package Accuracy within a Validated Infrastructure”, argues that package qualification should be risk-based rather than an all-or-nothing certification, and the group maintains the riskmetric package that operationalises it.

NoteHow do I qualify an R package for a regulated analysis?

Inventory and pin the packages your analysis depends on, then assess each one: pkg_ref("admiral") |> pkg_assess() |> pkg_score(). Archive the resulting score table, interpret it against acceptance criteria your SOP defines in advance (for example, a maintainer and documented exports are required; a low size score triggers extra review, not rejection), supplement the offline gaps with network evidence from a controlled source, and have a human accept or reject the package for its intended use. riskmetric supplies the reproducible evidence; your SOP and a qualified reviewer make the decision.

Noteriskmetric vs manual validation — what’s the difference?

Manual validation documents a package’s suitability by hand — a reviewer fills in a checklist about maintenance, documentation, and testing. riskmetric produces that same evidence as reproducible code: the metric scores regenerate identically each time you re-run the pipe, so the assessment is consistent across packages and repeatable in an audit. riskmetric does not replace the human decision — you still set the acceptance criteria and sign off — but it removes the manual, error-prone data-gathering step and makes the evidence reproducible.

Test your understanding

ImportantYour turn: assess and compare two packages

Assess rtables (the table-layout package, installed in this environment) the same way we assessed admiral: run pkg_ref("rtables") |> pkg_assess() |> pkg_score(), build the tidy score table, and answer two questions. (1) Which metrics returned NA, and are any of those a network failure you should worry about in a sealed environment? (2) Does rtables score higher or lower than admiral on the size_codebase metric, and what would that difference actually mean?

TipHint

Reuse the score-table code, swapping "admiral" for "rtables". The NA rows will be the same six network metrics plus license (a by-design stub, not a network failure). For the size comparison, remember that size_codebase is a logistic rating where a smaller codebase scores higher — so a higher score means less code to review, i.e. lower size risk.

TipSolution

library(riskmetric)

scores <- pkg_ref("rtables") |> pkg_assess() |> pkg_score()
score_df <- data.frame(
  metric = names(scores),
  score  = round(vapply(scores, function(x) as.numeric(x), numeric(1)), 3),
  row.names = NULL
)
score_df[order(score_df$score, na.last = TRUE), ]

(1) The NA rows are the six network metrics (covr_coverage, r_cmd_check, remote_checks, bugs_status, downloads_1yr, reverse_dependencies) plus license. None is a problem: the six need live CRAN/GitHub and are expected to be NA in a sealed environment, and license is a by-design stub in riskmetric 0.2.7 — you record it separately. (2) rtables is a smaller codebase than admiral, so it scores higher on size_codebase. That does not make rtables “better” — it means there is less code to review, i.e. lower risk on the size axis. Qualification reads each metric for what it measures, not as a leaderboard.

NoteQuick check: admiral scores 0.02 on the dependencies metric. What does that tell you?

A. admiral’s code is low quality and should be rejected. B. admiral has many dependencies, which riskmetric treats as higher risk on that axis — budget more review, not automatic rejection. C. riskmetric could not evaluate the dependencies metric offline.

TipShow answer

B. dependencies is a logistic rating where more dependencies score lower, because each dependency is more surface area to review — so 0.02 says “admiral depends on a lot,” a size/complexity signal, not a quality verdict. A misreads a size penalty as a defect (the very trap this lesson warns about); C is wrong because dependencies reads the installed DESCRIPTION and evaluates offline — it returned a real score, not NA.

Conclusion

Qualifying an R package is not about certifying the language — it is about assessing and documenting risk, the way the R Validation Hub’s risk-based framework prescribes. riskmetric makes that assessment reproducible: pkg_ref() |> pkg_assess() |> pkg_score() turns a package’s metadata and activity into a 0–1 score per metric, summarize_scores() gives a headline number, and a tidy score table becomes an artefact of the qualification record. The discipline is in reading it honestly — the low size scores are penalties for surface area, not defects; the overall number depends on how the missing metrics are weighted; and in a sealed validated environment the metadata metrics are the evidence you audit while the network metrics legitimately return NA. riskmetric supplies the evidence, reproducibly; your SOP and a qualified reviewer still make the call. That combination — automated, repeatable evidence plus a documented human decision — is what lets you answer an inspector’s “how do you know these packages are fit for use?” with something better than “trust me.”

Note

This lesson is reproducible: every assessment on this page was produced by the code shown, run offline against the installed package — copy any block and run it to reproduce them. The runtime is the judge.

References

• riskmetric (CRAN) · riskmetric documentation — the R Validation Hub’s package-risk assessment tool, and its metric reference.
• R Validation Hub — the cross-industry initiative for using R in a regulated setting.
• A Risk-based Approach for Assessing R Package Accuracy within a Validated Infrastructure — the R Validation Hub white paper that frames package qualification as a risk-based activity.
• R Consortium Submissions Working Group — Pilot 1 — an FDA pilot submission built with open-source R packages, an example of the risk-based approach in practice.

Related lessons

• Reproducible environment in R with renv — pin the exact package versions you qualify here, so the qualified library rebuilds identically. · Double programming in R with diffdf — validating an analysis’s outputs; package qualification validates the tools those outputs are built with. · Defensive programming in R with checkmate and assertthat — guarding a script’s inputs, the complementary QC discipline. · Create ADSL in R with admiral — the pipeline whose dependency on admiral this lesson qualifies.
• Where this fits: building ADaM datasets with admiral → validating their outputs (double programming with diffdf) → defending their inputs (assertions with checkmate) → freezing the environment (reproducibility with renv) → qualifying the packages that environment contains (package risk assessment with riskmetric — you are here). Qualification is how the tools themselves earn a place in a validated pipeline.

Was this page helpful?

Prove you can do it. Master the whole Clinical Programming in R (pharmaverse) series — track your path, build projects, and earn a certificate.

Start free →

Go Pro — unlimited Prova on your own data and a verifiable certificate that proves the skill.

from $15/mo billed yearly

Go Pro →

✓ You're Pro — keep going. The runtime is the judge.

Get new R & Python lessons by email

Practical, reproducible, no spam. Unsubscribe anytime.

Double opt-in. We never share your email.

Share this pageXLinkedInRedditHN