Metadata-Driven ADaM in R with metacore and metatools

Drive and check your admiral derivations from one machine-readable ADaM specification — variable order, labels, lengths, and controlled terminology enforced by the spec, not hand-coded in every program

A complete, runnable tutorial on metadata-driven ADaM programming in R. Learn what metacore and metatools are (metacore holds the ADaM specification as a machine-readable object, built from a Define-XML or a spec spreadsheet; metatools applies and checks a dataset against it), then load the CDISC pilot ADSL specification, inspect its variable metadata and controlled terminology, and validate a derived ADSL with check_variables(), check_ct_data(), drop_unspec_vars(), order_cols(), and sort_by_key() — on public pharmaverse example data, so every line runs.

TipKey takeaways

• The specification is the single source of truth. An ADaM (Analysis Data Model) dataset is defined by a spec — variable names, order, labels, lengths, and controlled terminology (CT) — that the same team also ships as the Define-XML document. Metadata-driven programming reads that spec once and enforces it everywhere, instead of hard-coding it in each program.
• metacore holds the spec; metatools uses it. metacore turns a Define-XML file or a specification spreadsheet into one machine-readable R object. metatools provides the functions that apply the spec to a dataset and check the dataset against it.
• Checks are one line each. check_variables() confirms every specified variable is present (and no extras crept in); check_ct_data() confirms every value sits inside its controlled terminology.
• Conform by construction. drop_unspec_vars() removes off-spec columns, order_cols() puts them in the spec’s order, sort_by_key() sorts rows by the key sequence, and add_labels() applies the spec’s labels — so the derived dataset matches the Define-XML it will be submitted with.
• This is what makes ADaM scale. The admiral derivations you write by hand produce the values; the metacore spec governs the structure. Together they give you an analysis dataset that is correct and conformant across dozens of datasets without re-typing metadata each time.

Introduction

In the earlier lessons of this series you derived analysis datasets by hand: the subject-level ADSL, the laboratory ADLB, and the vital-signs ADVS. Every one of those programs quietly encoded a specification — the exact set of variables, their order, their labels, their lengths, and which values each may take. You typed that structure into the code.

That does not scale. A real submission has dozens of ADaM datasets, and the same specification is also written down in the Define-XML — the machine-readable metadata document that ships with the data to the regulator. Encoding the spec twice (once in your program, once in the Define-XML) is how a label drifts, a variable order diverges, or a value slips outside its controlled terminology (CT) — the permitted code list for a variable, such as M/F for SEX.

Metadata-driven programming removes the duplication: read the specification once into a machine-readable object, then have your program apply and check against it. This lesson uses two pharmaverse packages built for exactly that — metacore and metatools — on the public CDISC (Clinical Data Interchange Standards Consortium) pilot specification, so every line runs as written.

Here is the ADSL specification we will work with, summarized straight from the spec object — 51 variables, grouped by type:

That bar chart was not hand-tallied — it was computed from the specification object itself. By the end of this lesson you will read metadata like this, and use it to check a derived dataset for conformance. If “ADaM”, “Define-XML”, or “CDISC” are new, start with the CDISC standards, which maps where the specification and Define-XML sit in the pipeline.

What metacore and metatools are

The two packages split the work cleanly: one holds the specification, the other acts on it.

metacore (CRAN · docs) reads a specification — a Define-XML file or a specification spreadsheet — into a single R object. That object is a small database of the metadata, organized into seven linked tables:

metacore table	What it holds
ds_spec	Dataset-level information (name, structure, label).
ds_vars	Which variables belong to each dataset, plus order and key sequence.
var_spec	Variable-level metadata: label, type, length, format.
value_spec	Value-level metadata: origin, type, and links to CT and derivations.
derivations	The derivation text for each variable.
codelist	Controlled terminology — the code/decode pairs and permitted values.
supp	Metadata for supplemental qualifier variables.

metatools (CRAN · docs) is the toolbox that reads a metacore object and either builds structure into a dataset or checks a dataset against the spec. These are the functions this lesson uses:

metatools function	What it does
build_from_derived()	Pull “predecessor” columns straight from source datasets to start an ADaM.
check_variables()	Confirm the dataset has exactly the spec’s variables — no missing, no extra.
check_ct_data()	Confirm every value sits inside its controlled terminology.
drop_unspec_vars()	Drop any column not in the spec.
order_cols()	Reorder columns to the spec’s variable order.
sort_by_key()	Sort rows by the spec’s key sequence.
add_labels() / add_variables()	Apply the spec’s labels; add missing spec variables, correctly typed.

The division of labour is the whole idea: metacore is what the dataset should be, metatools makes a dataset become that and proves it did.

Load a metacore specification

In practice you build a metacore object from your own Define-XML or spec spreadsheet. metacore reads a Define-XML with the xml_to_*() reader functions, and a Pinnacle 21-style specification spreadsheet with spec_to_metacore():

# From a Define-XML file (the CDISC metadata exchange format):
library(xml2)
doc <- read_xml("define.xml")
xml_ns_strip(doc)
metacore(
  xml_to_ds_spec(doc), xml_to_ds_vars(doc), xml_to_var_spec(doc),
  xml_to_value_spec(doc), xml_to_derivations(doc), xml_to_codelist(doc)
)

# ...or directly from a specification spreadsheet:
spec_to_metacore("adam_spec.xlsx")

So that this lesson runs anywhere, we use the ready-made ADaM specification object that metacore ships for the CDISC pilot study — the same synthetic trial the admiral lessons derive. It contains the specs for every dataset in the pilot, so the first move is to narrow it to the one we care about, ADSL (the Subject-Level Analysis Dataset — one row per subject), with select_dataset():

library(metacore)
library(metatools)
library(dplyr, warn.conflicts = FALSE)
library(haven)

# The example ADaM spec object for the CDISC pilot (all datasets)
load(metacore_example("pilot_ADaM.rda"))

# Narrow to the ADSL specification
adsl_spec <- metacore %>% select_dataset("ADSL")
adsl_spec

adsl_spec now holds only the ADSL metadata, and printing it shows the dataset structure and the variable count. Every metatools function below takes this subsetted object.

Inspect the specification

Because the spec is an object, not a static document, you can query it. Two views matter most while programming: the per-variable metadata and the controlled terminology.

The variable metadata — label, type, length — lives in var_spec. This is the source of truth for what each column must look like:

adsl_spec$var_spec %>%
  select(variable, type, length, label) %>%
  head(8)

# A tibble: 8 × 4
  variable type    length label                              
  <chr>    <chr>    <int> <chr>                              
1 STUDYID  text        12 Study Identifier                   
2 SUBJID   text         4 Subject Identifier for the Study   
3 SITEID   text         3 Study Site Identifier              
4 SITEGR1  text         3 Pooled Site Group 1                
5 ARM      text        20 Description of Planned Arm         
6 TRT01P   text        20 Planned Treatment for Period 01    
7 TRT01PN  integer      8 Planned Treatment for Period 01 (N)
8 TRT01A   text        20 Actual Treatment for Period 01     

Controlled terminology — the permitted values for a coded variable — comes from get_control_term(). Ask for a variable and it returns the allowed values (a plain vector of permitted values, or a code/decode table for a code list). Here is the CT for SEX, then for RACE:

get_control_term(adsl_spec, SEX)

# A tibble: 2 × 1
  code 
  <chr>
1 M    
2 F    

get_control_term(adsl_spec, RACE)

# A tibble: 5 × 2
  code                                      decode                              
  <chr>                                     <chr>                               
1 WHITE                                     WHITE                               
2 BLACK OR AFRICAN AMERICAN                 BLACK OR AFRICAN AMERICAN           
3 AMERICAN INDIAN OR ALASKA NATIVE          AMERICAN INDIAN OR ALASKA NATIVE    
4 ASIAN                                     ASIAN                               
5 NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER NATIVE HAWAIIAN OR OTHER PACIFIC IS…

SEX may only be M or F; RACE has a five-row code list. These are the exact value sets check_ct_data() will enforce in a moment — the spec defines them once, and the check reads them straight back.

Apply and check a dataset against the spec

Now the payoff. We take a derived ADSL — the finished analysis dataset a program like the ADSL lesson produces — and validate it against the specification. metatools ships a derived ADSL for exactly this purpose:

adsl <- read_xpt(metatools_example("adsl.xpt"))
dim(adsl)

[1] 254  51

Check the variables are all there

check_variables() compares the dataset’s columns against the spec’s variable list. If they match, it says so; if a variable is missing or an unexpected one is present, it tells you which. We surface its message:

adsl <- check_variables(adsl, adsl_spec)

No missing or extra variables

“No missing or extra variables” is the all-clear: the derived ADSL has exactly the columns the spec defines. Had a program forgotten TRTSDT, or left a scratch column behind, the message would name it — and you would fix the derivation (add the variable) or the spec (if the variable genuinely does not belong), so the two agree. The function returns the data unchanged, so it drops straight into a pipeline.

Check the values are in their controlled terminology

check_ct_data() goes value by value: for every variable that has a code list, it confirms each observed value is a permitted one. On a conformant dataset it passes silently-but-for the all-clear:

adsl <- check_ct_data(adsl, adsl_spec, omit_vars = c("AGEGR2", "AGEGR2N"))

✔ All controlled terminology checks passed

(omit_vars skips AGEGR2/AGEGR2N here for a subtle reason: they do have controlled terminology in the spec, but the pilot data stores the decoded labels (e.g. "18-64 years") while check_ct_data() validates against the codelist’s code ("18-64") — a code-vs-decode mismatch in the example data, not a spec gap.) The check passes because every value already matches. To see what a failure looks like — the reason you run the check at all — we deliberately corrupt one value, writing "Male" into SEX where the CT permits only M/F:

adsl_bad <- adsl
adsl_bad$SEX[1] <- "Male"          # not in the SEX code list (M / F)

invisible(check_ct_data(adsl_bad, adsl_spec, omit_vars = c("AGEGR2", "AGEGR2N")))

Warning: ✖ Invalid controlled terminology detected
ℹ Variable: SEX | Codelist: CL.SEX
ℹ Values not permitted 'Male'

The warning names the variable (SEX), the code list (CL.SEX), and the offending value ('Male'). That is the entire value of a metadata-driven check: it does not just say “something is wrong”, it points at the exact cell and the rule it broke — so you fix the derivation that produced "Male" (it should map to M) before the dataset ever reaches the Define-XML.

Conform the structure to the spec

Passing the checks confirms the content is right. Three more metatools functions make the structure conform — and each is one line.

drop_unspec_vars() removes any column the spec does not define. Watch it delete a stray working column:

adsl_extra <- adsl %>% mutate(SCRATCH = "temporary working column")

adsl_clean <- drop_unspec_vars(adsl_extra, adsl_spec)

"SCRATCH" %in% names(adsl_clean)   # dropped -> FALSE

[1] FALSE

order_cols() reorders the columns into the spec’s variable order — the order the Define-XML declares and the reviewer expects:

adsl_ordered <- order_cols(adsl, adsl_spec)
names(adsl_ordered)[1:8]

[1] "STUDYID" "USUBJID" "SUBJID"  "SITEID"  "SITEGR1" "ARM"     "TRT01P" 
[8] "TRT01PN"

sort_by_key() sorts the rows by the spec’s key sequence (for ADSL, USUBJID), so the dataset is in its canonical order:

adsl_final <- sort_by_key(adsl_ordered, adsl_spec)
adsl_final %>%
  select(USUBJID, TRT01P, AGE, SEX, RACE) %>%
  head(5)

# A tibble: 5 × 5
  USUBJID     TRT01P                 AGE SEX   RACE 
  <chr>       <chr>                <dbl> <chr> <chr>
1 01-701-1015 Placebo                 63 F     WHITE
2 01-701-1023 Placebo                 64 M     WHITE
3 01-701-1028 Xanomeline High Dose    71 M     WHITE
4 01-701-1033 Xanomeline Low Dose     74 M     WHITE
5 01-701-1034 Xanomeline High Dose    77 F     WHITE

Two companions finish the metadata: add_labels() applies variable labels from the spec (so the exported dataset carries the Define-XML labels), and add_variables() adds any spec variable a derivation missed, correctly typed and empty — useful as a safety net before the final checks. To start an ADaM from source data, build_from_derived() pulls the “predecessor” columns (variables copied straight from an SDTM — the Study Data Tabulation Model — source) directly out of the source datasets, giving you a typed, labelled skeleton to derive the rest onto.

Why this is the way ADaM scales

Step back and see what changed. In the by-hand lessons, the ADSL structure lived inside the program: you named the variables, ordered them, labelled them, and knew — in your head — which values were legal. Here, that structure lives in one specification object, and the program reads it:

• the derivation (admiral) produces the values;
• the specification (metacore) defines the structure;
• the checks (metatools) prove the values fit the structure — automatically, the same way, for every dataset.

Run the same three checks on ADLB, ADVS, ADAE, and ADTTE and you have submission-wide conformance without re-typing a single label or code list. And because the spec object is built from the same Define-XML you submit, the data and its metadata document cannot silently drift apart — the check would catch it. That is metadata-driven programming: the spec is the single source of truth, and the dataset conforms by construction.

🟢 With an AI agent

Ask Prova “how do I build a metacore object from my own Define-XML and check my derived ADSL against it with metatools?” — it answers grounded in this pillar’s lessons, with runnable metacore/metatools code you can try on the example pharmaverse data. The runtime is the judge. Ask Prova →

Common issues

check_variables() reports a variable missing that you know you derived. The dataset and the spec disagree on the variable name, not its presence — a common cause is a typo or a case difference (trt01p vs TRT01P), or a variable the spec expects that your program renamed. Read the message: it lists exactly which names are missing and which are extra. Fix whichever side is wrong — usually the derivation, occasionally the spec.

check_ct_data() warns about a value that looks correct. Controlled terminology is exact and case-sensitive. "Male" is not "M", "White" is not "WHITE", and a trailing space ("M ") fails too. The warning prints the variable, the code list, and the exact offending value — map that value to its permitted code in the derivation. If the value is genuinely valid and the spec is out of date, update the spec’s code list instead; the point is that one of the two must change so they agree.

A check needs the metacore object narrowed to one dataset first. The metatools check and conform functions expect a spec for a single dataset. If you pass the full multi-dataset object you loaded from Define-XML, subset it first with metacore::select_dataset("ADSL") (as we did with adsl_spec), otherwise the functions cannot tell which dataset’s variables and code lists to use.

Frequently asked questions

NoteWhat is metadata-driven ADaM programming?

It means driving your ADaM derivations from a single machine-readable specification instead of hard-coding variable names, order, labels, lengths, and controlled terminology in each program. You read the spec once (with metacore), derive the values with a tool like admiral, and then apply and check the structure against the spec (with metatools). The specification — the same one shipped as the Define-XML — becomes the single source of truth, so the data and its metadata cannot drift apart.

NoteWhat is the difference between metacore and metatools?

metacore holds the specification: it reads a Define-XML file or a spec spreadsheet into one R object of linked metadata tables (variables, labels, controlled terminology, derivations). metatools acts on that object: its functions apply the spec to a dataset (order_cols(), add_labels(), drop_unspec_vars()) and check a dataset against it (check_variables(), check_ct_data()). In short, metacore is what the dataset should be; metatools makes a dataset become that and proves it did.

NoteHow do I read a Define-XML into R?

Use metacore’s reader functions. Parse the file with xml2::read_xml() and xml_ns_strip(), then call xml_to_ds_spec(), xml_to_ds_vars(), xml_to_var_spec(), xml_to_value_spec(), xml_to_derivations(), and xml_to_codelist() on it and assemble the pieces with metacore(). For a Pinnacle 21-style specification spreadsheet, spec_to_metacore("spec.xlsx") does it in one call. Both give you the same kind of metacore object.

NoteHow do I check controlled terminology in R?

Subset your metacore object to the dataset with metacore::select_dataset(), then call metatools::check_ct_data(data, spec). It compares every coded variable’s values against its code list and warns with the variable name, the code list, and the exact offending value if anything is out of terminology. To see the permitted values for one variable ahead of time, use metacore::get_control_term(spec, VARNAME).

NoteDo I still need admiral if I use metacore and metatools?

Yes — they do different jobs. admiral derives the values (treatment dates, baselines, flags, time-to-event); metacore and metatools govern and check the structure (which variables, in what order, with what labels and controlled terminology). A typical build uses admiral for the derivations and metatools to conform and validate the result against the metacore spec. They are complementary pharmaverse packages, not alternatives.

Test your understanding

ImportantYour turn: check a dataset that is missing a variable

Starting from the derived ADSL in this lesson, drop the TRT01P column to simulate a derivation that forgot it, then run check_variables() against adsl_spec. What does the check report, and which side would you fix?

TipHint

Remove the column with select(-TRT01P), then pass the result to check_variables() together with adsl_spec. By default (strict = FALSE) the check warns on a mismatch rather than stopping, which keeps the message easy to read; pass strict = TRUE to make a mismatch a hard error that halts the run.

TipSolution

adsl_missing <- adsl %>% select(-TRT01P)

check_variables(adsl_missing, adsl_spec, strict = FALSE)

The check flags TRT01P as a missing variable — it is in the specification but not in the dataset. The fix is almost always on the derivation side: add the TRT01P derivation back so the program produces the variable the spec requires. You would only change the spec if TRT01P genuinely should not be in ADSL — which, for planned period-01 treatment, it always should. This is the check earning its keep: a forgotten variable is caught mechanically, before the dataset reaches the Define-XML.

NoteQuick check: what does check_ct_data() validate?

A. That every variable in the spec is present in the dataset B. That every value sits inside its variable’s controlled terminology C. That the columns are in the spec’s order

TipShow answer

B. check_ct_data() validates values against controlled terminology — it warns when a coded variable holds a value outside its code list (like "Male" where only M/F are permitted). A is what check_variables() does, and C is what order_cols() does. The three are separate, one-line steps: variables present, values in terminology, columns in order.

Conclusion

Metadata-driven ADaM programming splits a problem you used to solve in one tangled program into two clean halves: the derivation produces the values, and the specification governs the structure. metacore turns your Define-XML or spec spreadsheet into a queryable object; metatools applies it (drop_unspec_vars(), order_cols(), sort_by_key(), add_labels()) and — most importantly — checks against it (check_variables(), check_ct_data()). Each check is one line, reads the rule straight from the spec, and points at the exact variable or value that breaks it. Do this once and it is a nice tidy-up; do it across every ADaM dataset in a study and it is the difference between hoping your data matches its Define-XML and knowing it does. The specification is the single source of truth, and the dataset conforms by construction.

Note

This lesson is reproducible: every result on this page was produced by the code shown — copy any block and run it to reproduce them. The runtime is the judge.

Related lessons

• Create ADSL in R with admiral — the subject-level derivation whose structure this specification governs; the natural dataset to check first. · Build ADLB and the BDS structure — the parameter-level dataset the same three checks conform. · The CDISC standards: CDASH, SDTM, ADaM, Define-XML — where the specification and Define-XML sit in the CDISC pipeline. · The SAS-to-R shift for clinical programming — why metacore and metatools replace hand-maintained metadata macros.
• Where this fits: the regulatory and CDISC foundations → building ADaM with admiral (ADSL → ADLB → ADVS → ADAE → ADTTE) → governing all of them with one specification (you are here) → producing the tables, listings, and figures every analysis ships. Metadata-driven programming is what keeps that whole chain conformant at scale.

Was this page helpful?

Prove you can do it. Master the whole Clinical Programming in R (pharmaverse) series — track your path, build projects, and earn a certificate.

Start free →

Go Pro — unlimited Prova on your own data and a verifiable certificate that proves the skill.

from $15/mo billed yearly

Go Pro →

✓ You're Pro — keep going. The runtime is the judge.

Get new R & Python lessons by email

Practical, reproducible, no spam. Unsubscribe anytime.

Double opt-in. We never share your email.

Share this pageXLinkedInRedditHN