Schedule for: 22w5184 - Statistical Challenges in the Identification, Validation, & Use of Surrogate Markers

In this talk, I will review the workshop background and motivation, provide an overview of the workshop schedule, describe the goals of the workshop, and describe the structure of the talks, panels, and last-day activities. In addition, I will give a brief overview of the history of surrogate marker work and the current status of available methods and use in practice. I will describe what I think are the open questions in this area, and barriers to practical use of statistical methods to validate and use surrogate markers.

Both humanitarian and commercial considerations have spurred intensive search for methods to reduce the time and cost required to develop new therapies. The identification and use of surrogate endpoints, i.e. measures that can replace or supplement other endpoints in evaluations of experimental treatments or interventions, is a general strategy that has stimulated much enthusiasm. Surrogate endpoints are useful when they can be measured earlier, more conveniently, or more frequently than the "true" endpoints of primary interest. Regulatory agencies around the globe, particularly in the United States, Europe, and Japan, are introducing provisions and policies relating to the use of surrogate endpoints in registration studies. But how can one establish the adequacy of a surrogate, in the sense that treatment effectiveness on the surrogate will accurately predict treatment effect on the intended, and more important, true outcome? What kind of evidence is needed, and what statistical methods portray that evidence most appropriately? The definition of validity, as well as formal sets of criteria, have been proposed, including use of the proportion explained, jointly the within-treatment partial association of true and surrogate responses, and the treatment effect on the surrogate relative to that on the true outcome. In a multi-centre setting, these quantities can be generalized to individual-level and trial-level measures of surrogacy. Consequently, a meta-analytic framework studying surrogacy at both the trial and individual-patient levels has been proposed. A number of variations of this theme have been developed, depending on the type of endpoint for the true and surrogate endpoint and on the focus of the evaluation exercise. The framework commonly used will be sketched, also against the background of alternatives. A perspective will be given on further and ongoing developments.

Context: With the large number of promising new treatments that are currently available for testing, clinical trials need to detect treatment benefits and harms as quickly as possible. In parallel with the need for speed in clinical development, advances in molecular biology, high throughput technologies and imaging techniques provide investigators with an ever growing number of biomarkers which can potentially be used to replace clinical endpoints in the comparison of new treatments with established standards of care. Objective: This talk will discuss the type of statistical evidence required for an intermediate endpoint (possibly based on a biomarker) to be an acceptable surrogate endpoint in clinical trials. [1] Methods: Historically, the first formal definition of surrogacy is due to Ross Prentice. This definition, and the accompanying criteria, have had a huge role in focusing attention on the need for a formal statistical approach to surrogate validation. The validation approach most commonly used currently requires a meta-analysis of several randomized trials to investigate the association between the surrogate and the true endpoint, and the association between treatment effects on these endpoints. An acceptable surrogate must be prognostic for the true endpoint (“individual-level association”), and the treatment effect on the surrogate must be predictive of the treatment effect on the true endpoint (“trial-level association”). Information theory can be used to assess the quality of a potential surrogate at both the individual and trial levels. For the planning of future trials, the “surrogate threshold effect” can be estimated as the minimum effect on the surrogate biomarker that would predict a statistically significant effect on the clinical endpoint. SAS and R software has been developed to implement all these ideas. [2] A very different line of research has evolved from concepts of causal inference, using either principal stratification, or mediation analysis. Causal inference can shed light on statistical associations found in a meta-analysis, and as such the two approaches can complement each other for a full assessment of surrogacy. Results: The potential and limitation of all these approaches will be illustrated using patient level data from clinical trials of treatments for HER2-positive early breast cancer. Conclusion: The search for surrogate endpoints will continue unabated in the future. A rigorous statistical assessment of surrogacy is possible but typically requires access to patient-level data from several (peferably many) randomized clinical trials. References 1. Burzykowski T, Molenberghs G, Buyse M. (eds.) The Evaluation of Surrogate Endpoints. Springer (408 p.), New York, 2005. 2. Alonso A, Bigirumurame T, Burzykowski T, Buyse M, Molenberghs G, Muchene L, Perualila NJ, Shkedy Z, Van der Elst W. Applied Surrogate Endpoint Evaluation Methods with SAS and R. Chapman and Hall/CRC Press, New York, 2017.

In comparative effectiveness research (CER), leveraging short-term surrogates to infer treatment effects on long-term outcomes can guide policymakers in evaluating new treatments. Numerous statistical procedures for identifying surrogates have been proposed for randomized clinical trials (RCTs), but no methods currently exist to evaluate the proportion of treatment effect (PTE) explained by surrogates in real-world data (RWD), which have become increasingly common with the rise of algorithm-derived outcomes. To address this knowledge gap, we propose inverse probability weighted (IPW) and doubly robust (DR) estimators of an optimal transformation of the surrogate and the corresponding PTE measure. We demonstrate that the proposed estimators are consistent and asymptotically normal, and the DR estimator is consistent when either the propensity score model or outcome regression model is correctly specified. Our proposed estimators are evaluated through extensive simulation studies. In two RWD settings, we show that our method can identify and validate surrogate markers for inflammatory bowel disease (IBD).

The use of such surrogates can give rise to paradoxical situations in which the effect of the treatment on the surrogate is positive, the surrogate and outcome are strongly positively correlated, but the effect of the treatment on the outcome is negative, a phenomenon sometimes referred to as the "surrogate paradox." Results are given for consistent surrogates on sufficient conditions that ensure the surrogate paradox is not manifest. It is shown that for the surrogate paradox to be manifest it must be the case that either there is (i) a direct effect of treatment on the outcome not through the surrogate and in the opposite direction as that through the surrogate or (ii) confounding for the effect of the surrogate on the outcome, or (iii) a lack of transitivity so that treatment does not positively affect the surrogate for all the same individuals for whom the surrogate positively affects the outcome. These conditions give rise to criteria under which the use of a surrogate might be considered reasonable.

Endometrial Cancer, or cancer of the uterus, is the most common gynecological cancer in the developed world with incidence increasing due to increasing prevalence of risk factors such as obesity. Risk-reducing interventions to prevent endometrial cancer are being proposed, but waiting to observe the impact on incidence of cancer as a primary endpoint may be too long for needed policy change and action in this disease. I discuss possible surrogates including advantages and disadvantages and propose a novel approach that uses causal learning to burrow surrogate markers of diabetes, a condition highly related to endometrial cancer.

Consider a phase 3 clinical trial that randomizes participants to active vs. control intervention, and follows participants for occurrence of a primary clinical endpoint. Suppose a candidate surrogate endpoint is measured at a fixed time point after randomization. Four causal inference approaches to evaluating the candidate surrogate (e.g. a biomarker) are: (a) Principal Stratification to assess how the treatment effect on the clinical endpoint varies over subgroups defined by the counterfactual biomarker value if assigned active treatment; (b) Static Interventional Controlled Effects to assess controlled direct effects of assigning all participants to active vs. control and to a specific biomarker value; (c) Stochastic Interventional Effects to assess the effect of assigning all participants to active vs. control and drawing the biomarker from specified distributions under each treatment; and (d) Mediation to assess the natural direct and indirect effects of treatment through the biomarker. This talk will discuss how this set of causal approaches may be applied to understand how well -- and how -- the biomarker can be used for making inferences about the clinical treatment effect, with application to the Moderna COVE phase 3 COVID-19 vaccine efficacy trial. The Stochastic Interventional Effects approach is particularly promising for a major application of a surrogate endpoint: transportability/ prediction of the clinical treatment effect for various contexts departing from the phase 3 trial conditions.

Trial level surrogates are useful tools for improving the speed and cost effectiveness of trials, but surrogates that have not been properly evaluated can cause misleading results. This evaluation is often contextual and depends on the type of trial setting. There have been many proposed methods for trial level surrogate evaluation, but none for the specific setting of Bayesian adaptive platform studies. As adaptive studies are becoming more popular, methods for surrogate evaluation using them are needed. These studies also offer a rich data resource for surrogate evaluation that would not normally be possible. However, they also offer a set of statistical issues including heterogeneity of the study population, treatments, implementation, and even potentially the quality of the surrogate. Platform trials often also have a shared control arm and early stopping rules that can lead to interdependence and biased estimation. We propose the use of a hierarchical Bayesian semiparametric model for the evaluation of potential surrogates using nonparametric priors for the true effects based on Dirichlet process mixtures. The motivation for using this method is to flexibly model relationships between the treatment effect on the surrogate and the treatment effect on the outcome and also to identify potential clusters with differential surrogate value in a data-driven manner. In simulations, we find that our proposed method is superior to a simple, but fairly standard, hierarchical Bayesian method. We demonstrate how our method can be used in a simulated illustrative example, in which we are able to identify clusters where the surrogate is and is not useful. Our motivating study is the Probio trial, in which we plan to apply our method in the future (once the trial is completed). This is work with Dr. Michael C Sachs, Dr. Alessio Crippa, and Professor Michael J Daniels.

As immunological mechanisms triggered by vaccination remains only partially understood, gene expression measurements holds a promise of gaining a deeper understanding into molecular processes at play. As more and more transcriptomics data are generated in early phase vaccine trials, there is a question of whether they may be used to capture vaccine effects: gene expression largely determines cellular function, and is thus a promising biomarker for quickly measuring effects of vaccines. Validated gene signatures could dramatically speed up vaccine trials for emerging infectious diseases like Ebola and COVID-19. But because of the high dimension of the gene expression data generated in early phase trials, available methods may not be applicable. Hence, we propose to reduce the dimension of the problem through a selection of the genes that could play the role of first surrogate markers in early trials (providing that an intermediate mid-term surrogate is available such as binding antibodies). The first step is to be able quantify how much of the vaccine effect is mediated by gene expression, and establish if gene expression is suitable for capturing the total vaccine effect. The second step is to construct optimal gene expression signatures for capturing the vaccine effect. Such an approach could be pioneered and applied to the vaccine trials generated by the Vaccine Research Institute against HIV, Ebola and COVID-19.

The work considers the issue of surrogate paradox -- when the surrogate marker gives a measure of treatment effectiveness that is in the reverse direction of the actual treatment effectiveness on the outcome of interest. This is perhaps most serious when the surrogate suggests a treatment will be beneficial when it is in fact harmful, but can also be problematic when effective treatments are rejected. We develop a series of measures for these in settings with multiple trials, where the joint causal effects of treatment and control can be identified, and extend these to settings conditional on covariates, searching for settings where the surrogate paradox might be restricted to population subsets.

As soon as efficacy is established in vaccine trials, identifying the relevant immune response marker associated with the observed protection is of interest to facilitate further steps in the development. For that purpose, a reasonable likelihood to predict clinical benefit (‘vaccine efficacy’) must be shown. Sufficient evidence will have to be generated to address that question and support the validity for further use. Additional question may target identification of a threshold for protection, assessing the strength of surrogacy or how much of the observed effect is explained (or mediated) through a specific marker. Various statistical methods and quantities have been proposed to address these critical questions, based on single as well as multiple trials. However, each come with a specific interpretation, and varying practical relevance within the development. Furthermore the data generated for this objective may be challenged by design choices for evaluation of the primary objective (for e.g. cross-over vaccination in COVID-19 vaccine efficacy trials) or consequences of the results (for e.g. variable efficacy over time due to a changing virus). In this presentation we will present some of these challenges in more detail when targeting these questions and how they can be tackled, thereby employing real examples.

We consider a setting in observational studies in pregnancy where spontaneous abortion can be viewed as a surrogate marker for birth defects, which are often unobserved if the fetus is spontaneously aborted. The common practice to treat these unobserved birth defect outcomes as 'no’ results in bias in the estimated exposure effect. In fact, according to the theory of "terathanasia'', a defected fetus is more likely to be spontaneously aborted, leading to missing not at random. In addition, the typical analysis stratifies on live birth versus spontaneous abortion, which is itself a post-exposure variable and does not give causal interpretation of the stratified results. We develop methods to estimate the average exposure effect as well as the principal effects, making use of the missing data mechanism informed by "terathanasia''. Other complications in the data include left truncation, right censoring, and rare events. The rare events with missing outcomes demand multiple sensitivity analyses. Our findings should shed light on how studies on causal effects of medication or other exposures during pregnancy may be analyzed using state-of-the-art methodologies.

Given recent advances in HIV prevention, future trials of many experimental interventions are likely to be “active-controlled” designs, whereby HIV negative individuals are randomized to the experimental intervention or an active control known to be effective based on a historical trial. The efficacy of the experimental intervention to prevent HIV infection relative to placebo cannot be evaluated directly based on the trial data alone. One approach that has been proposed is to leverage an HIV exposure marker, such as incident rectal gonorrhea which is highly correlated with HIV infection in populations of men who have sex with men (MSM). Assuming we can fit a model associating HIV incidence and incidence of the exposure marker, based on data from multiple historical studies, incidence of the marker in the active-controlled trial population can be used to infer the HIV incidence that would have been observed had a placebo arm been included, i.e. a “counterfactual placebo”, and to evaluate efficacy of the experimental intervention relative to this counterfactual placebo. We formalize this approach and articulate the underlying assumptions, develop an estimation approach and evaluate its performance in finite samples, and discuss the implications of our findings for future development and application of the approach in HIV prevention. Improved HIV exposure markers and careful assessment of assumptions and study of their violation are needed before the approach is applied in practice.

A common practice in clinical trials is to evaluate a treatment effect on an intermediate endpoint when the true outcome of interest would be difficult or costly to measure. We consider how to validate intermediate endpoints in a causally-valid way when the trial outcomes are time-to-event. Using counterfactual outcomes, those that would be observed if the counterfactual treatment had been given, the causal association paradigm assesses the relationship of the treatment effect on the surrogate S with the treatment effect on the true endpoint T. In particular, we propose illness death models to accommodate the censored and semi-competing risk structure of survival data. The proposed causal version of these models involves estimable and counterfactual frailty terms. Via these multistate models, we characterize what a valid surrogate would look like using a causal effect predictiveness plot. We evaluate the estimation properties of a Bayesian method using Markov Chain Monte Carlo and assess the sensitivity of our model assumptions. Our motivating data source is a localized prostate cancer clinical trial where the two survival endpoints are time to distant metastasis and time to death.

Mediation analysis is a useful framework for studying surrogate markers. It has also become extremely popular in genomics. There, its application has raised several interesting new statistical and methodological questions. I will describe a few results in these directions: a surprising property of hypothesis testing in mediation models, and estimation and inference for high-dimensional mediators. My hope is to discuss how these might be useful for assessing and identifying effective surrogate markers.

Many countries have implemented lockdowns to reduce COVID-19 transmission. However, there is no consensus on the optimal timing of these lockdowns to control community spread of the disease. Here we evaluated the relationship between timing of lockdowns, along with other risk factors, and the growth trajectories of COVID-19 across 3,112 counties in the US. We ascertained dates for lockdowns and implementation of various non-pharmaceutical interventions at a county level and merged these data with those of US census and county-specific COVID-19 daily cumulative case counts. We then applied a Functional Principal Component (FPC) analysis on this dataset to generate FPC scores, which were used as a surrogate variable to describe the trajectory of daily cumulative case counts for each county. We then identify risk factors including the timing of lockdown that significantly influenced the surrogate variable.

We used a distributed non-linear lag model to model the non-linear exposure-response delayed effects of Sars-Cov-2 RNA concentrations in the wastewater surveillance system on the hospitalization rate. Our model considered 3 to 15 days delayed effects of both SARs-COV N1 and SARs-COV N2 gene concentrations, and the daily cumulative vaccination rates. We explored the Covid-19 hospitalization records and the wastewater data from Ottawa region. The preliminary analysis for the data suggests that the wastewater virus signals can provide reasonable predictions of the hospitalization rates with the time-varying relationship, where the vaccination rates helps explain this time-varying relationship. SARs-COV N1 and SARs-COV N2 gene concentrations in wastewater seem promising surrogate markers for Covid-19 infection at the population level.

When evaluating the effectiveness of a treatment, policy, or intervention, the desired measure of efficacy may be expensive to collect, not routinely available, or may take a long time to occur. In these cases, it is sometimes possible to identify a surrogate outcome that can more easily, quickly, or cheaply capture the effect of interest. Theory and methods for evaluating the strength of surrogate markers have been well studied in the context of a single surrogate marker measured in the course of a randomized clinical study. However, methods are lacking for quantifying the utility of surrogate markers when the dimension of the surrogate grows. We propose a robust and efficient method for evaluating a set of surrogate markers that may be high-dimensional. Our method does not require treatment to be randomized and may be used in observational studies. Our approach draws on a connection between quantifying the utility of a surrogate marker and the most fundamental tools of causal inference -- namely, methods for robust estimation of the average treatment effect. This connection facilitates the use of modern methods for estimating treatment effects, using machine learning to estimate nuisance functions and relaxing the dependence on model specification. We demonstrate that our proposed approach performs well in simulation and demonstrate connections between our approach and certain mediation effects.

Surrogate markers are often evaluated entirely within the context of a single study design, e.g. a randomized clinical trial. While multiple trials can support surrogacy better than a single trial, additional experimental designs are another approach. Examples include dosing studies, factorial studies, animal studies, or different experimental manipulations of the surrogate. Consistency of the surrogate effect across different designs provides qualitative support for surrogacy which is nice but nebulous. In COVID-19 vaccines, the role of antibody in protective efficacy is of great interest as a surrogate endpoint and a mediator of vaccine effect. In this talk we formalize how a three arm trial of placebo, vaccine, and antibody alone can be used to quantitatively support antibody as a surrogate. The third arm of antibody alone can be used to uncover a formerly cross-world quantity and allow estimation of the proportion mediated. This method eliminates the requirement for several standard assumptions and has substantial face validity. Practical issues in implementation will be discussed. While not always possible, experimental manipulation can be a powerful tool to support surrogacy in some settings.

In survival data analysis, the Cox proportional hazards (PH) model is perhaps the most widely used model to feature the dependence of survival times on covariates. While many inference methods have been developed under such a model or its variants, those models are not adequate for handling data with complex structured covariates. High-dimensional survival data often entail several features: (1) many covariates are inactive in explaining the survival information, (2) active covariates are associated with a network structure, and (3) some covariates are error-contaminated. To handle such kinds of survival data, we propose graphical PH measurement error models and develop inferential procedures for the parameters of interest. Our proposed models significantly enlarge the scope of the usual Cox PH model and have great flexibility in characterizing survival data. Theoretical results are established to justify the proposed methods. Numerical studies are conducted to assess the performance of the proposed methods.

Discovering therapies for neurodegenerative diseases is notoriously difficult, and made worse without accurate disease trajectories to identify when interventions will best prevent or delay irreparable damage. Modeling a disease trajectory is not easy. These diseases progress slowly over decades, and no study covers the full disease course due to time and cost constraints. To compensate, researchers model disease trajectories by piecing together short longitudinal data from patients at different disease stages. The challenge is how to piece together the data to create realistic disease trajectories. One promising way pieces together the short longitudinal data to show changes before and after major events on the disease timeline, like when disease onset occurs. This approach has helped produce realistic disease trajectories, but has shortcomings when the time of the disease event is unknown since without these times, we don't know where to place the data on the disease timeline. To overcome this issue, researchers currently replace all unknown times with predicted times. Despite efforts to predict the time of disease events without bias using various models, the assumptions these models make often do not hold in practice and result in inaccurate predictions. This leads to an incorrect model of the disease trajectory, producing misleading conclusions about how quickly impairments change as the disease advances. We propose a series of estimators to model the disease trajectory around times of disease events without the need to predict times that are unknown. We show that our estimators produce accurate estimates of the trajectory around times of disease events even when we completely misspecified the distribution model of that time of disease event. We apply our methods to studies of Huntington disease where we model trajectories of motor impairments before and after times of major disease events, to help pinpoint when interventions will best prevent or delay irreparable damage.

In studies that require long-term and/or costly follow-up of participants to evaluate a treatment, there is often interest in identifying and using a surrogate marker to evaluate the treatment effect. While several statistical methods have been proposed to evaluate potential surrogate markers, available methods generally do not account for or address the potential for a surrogate to vary in utility or strength by patient characteristics. Previous work examining surrogate markers has indicated that there may be such heterogeneity i.e., that a surrogate marker may be useful (with respect to capturing the treatment effect on the primary outcome) for some subgroups, but not for others. This heterogeneity is important to understand, particularly if the surrogate is to be used in a future trial to replace the primary outcome. In this paper, we propose an approach and estimation procedures to measure the surrogate strength as a function of a baseline covariate W and thus, examine potential heterogeneity in the utility of the surrogate marker with respect to W. Within a potential outcome framework, we quantify the surrogate strength/utility using the proportion of treatment effect on the primary outcome that is explained by the treatment effect on the surrogate. We propose testing procedures to test for evidence of heterogeneity, examine finite sample performance of these methods via simulation, and illustrate the methods using AIDS clinical trial data.

Progression-free survival (PFS) is a surrogate endpoint widely used for overall survival in oncology. It has been routinely used to evaluate the treatment effect in cancer trials. When the non-terminal event such as progression is only assessed periodically, the composite PFS endpoint is subject to a dual censoring scheme involving interval censoring for progression and right censoring for death. We highlight statistical issues associated with the conventional approach of using right endpoint imputation in this setting, explore the determinants of the asymptotic bias and point out the loss of power in detecting treatment effect. We also consider the design of cancer trials aiming at detecting the treatment effect on the composite event of progression-free survival that insures the desired power. Sample size criteria are derived based on an illness-death model that considers cancer progression and death jointly while accounting for the fact that progression is assessed only intermittently.