Ocrevus, PPMS, and the 102 Mystery Patients

The Ocrevus PPMS ORATORIO study was designed for 630 patients. Eventually, though, Genentech ended up recruiting 732. You might think this is a good thing – the larger the study the better, right? Well, read on.

Note: much of the information in this blog post was taken from the CADTH Clinical Review of Ocrevus. CADTH is a Canadian government agency no one’s ever heard of that researches drugs to determine their effectiveness.

To ensure the integrity of a drug trial, the details of how it will be performed are published to regulatory agencies before the trial begins. This is call the study protocol. It should be obvious why this is important – if you’re allowed to continually make changes to an experiment while it’s in progress, you can probably tease any result you want out of it.

For example, imagine I wanted to prove that Houston was hotter than Dallas. I record temperatures every day for a month, but find that Dallas is hotter. I didn’t get the result I wanted, so, I simply extend my study by another month. Eventually, Houston has a heat wave, I conclude my study, and announce I’ve proved what I set out to prove. If, however, I were forced to declare the design of the study ahead of time (“The study will test the temperature at noon each day in the center of each city, in the shade, from January 1 to December 31…”), I would have far less room to distort my experiment to fit the result I’m looking for.

So it’s important to publish the protocol of a drug trial before it starts, and stick to that design. In the original protocol, dated June, 2010, Genentech very specifically described its patient population:

A total of 630 primary progressive MS patients will be enrolled and assigned
(2:1 randomization) to either an ocrelizumab [Ocrevus] arm or a placebo arm, stratified by age and region

Qualified patients are expensive to recruit. In fact, it took Genentech over a year to find its original 630 patient population. After reaching this goal, however, the company did a funny thing. It continued to accept patients, winding up with 732. Genentech attributed these additional patients to a “late screening boost”.

To report changes in the original protocol, drug companies file amended protocols with regulatory agencies. But, according to CADTH, Genentech failed to report the extra patients in its amended protocol.

Here’s why this matters: We know that with the 732 patients, Ocrevus was found to be – just barely – effective enough to merit FDA approval. 6.4% more of the 732 patients were helped by the drug than the placebo, resulting in a p-value of .03 for 12 week confirmed disease progression (CDP). Recall that the threshold for effectiveness is a p-value below .05 .

But Genentech provided another set of data to regulators, one you’ll never see it its advertisements. This data shows the study results if they’d followed the protocol, that is, with just that first 630 patients. The result? Ocrevus helped just 5.2% of people vs placebo. The p-value for this group was .09 – too large to be considered effective by regulatory standards.

In its clinical review, CADTH called out this irregular behavior:

The planned sample size for the ORATORIO trial was 630 patients; however, 732 patients were randomized due to an unexpected increase in screening after the closing of enrolment was announced by the manufacturer. This 16.2% increase in the sample size of the trial was not specified in a protocol amendment and had a measurable impact on the results of the study. A sensitivity analysis demonstrated that the primary end point of the trial would have not have met statistical significance without the enrolment of these additional patients.

There are a number of troubling possibilities here. By the time the 630th patient was recruited, the first patients had been in the study for over a year, and Genentech must have had initial data about the effectiveness of the drug – data which turned out to be less effective than its original protocol projection (30% and 43% 12 week CDP in the Ocrevus and placebo groups respectively). Realizing its drug was ineffective, did Genentech rush to round up a few more patients, hoping this might change the results?

And that brings us to the most disturbing part of the mystery patients – they received an extraordinary (some might say impossible) benefit from the drug versus the protocol group. Let’s look at the numbers. In the official study of 732 patients, 32.3% of those receiving Ocrevus experienced symptom progression, versus 39.3% of the placebo group:

All 732 Patients, 12 week CDPOcrevusPlaceboTotal
Patients487244731
Patients who progressed16096256
Percentages32.9%39.3%

Here’s the data from the 630 original patients – what the study should a have reported, had it stuck to protocol:

Original 630 patients, 12 week CDP OcrevusPlaceboTotal
Patients420209629
Patients who progressed14583228
Percentages34.5%39.7%

Subtracting the data in the second table from the first, we can get the results for of the 102 mystery patients:

Extra 102 mystery patients, 12 week CDP OcrevusPlaceboTotal
Patients6735102
Patients who progressed151328
Percentages22.4%37.1%

Progression for the mystery patients in the placebo group was in-line with what you’d expect from the protocol population: 37.1% vs 39.3%. But progression for the Ocrevus group was 34.5% in the original 630 patients, versus just 22.4% in the mystery group. The patients in the mystery group pushed the p-value of the primary endpoint in the ORATORIO study over the effectiveness threshold, resulting in what will certainly be hundreds of millions of dollars in profits for Roche-Genentech.

What happened? We’ll probably never know. Genentech might just have gotten extraordinarily lucky. But anyone considering a marginally effective drug with serious side effects should consider just how many ways a drug company can tweak its results. If you’re ever trying to prove Dallas is hotter that Houston, or the other way around, the fine folks at Genentech may be able to help.

Ocrevus, PPMS, and the Placebo Effect

In 2017, drug maker Genentech completed a Phase 3 trial with their drug Ocrevus for treating primary progressive multiple sclerosis (PPMS). They named the Study “ORATORIO”. As the name implies, PPMS worsens continually in the patient, without remissions, which are present in the more common relapse/remitting form of MS (RRMS).

For the trial, a group of 244 patients visited medical offices around the world for two treatments every six months. They had been diagnosed with PPMS for an average of 1.5 years, though they’d had symptoms for 5 or 6 years on average.

For each treatment, an IV needle was inserted into the patient’s arm, and medicine dripped for hours into their veins. To pass the time, they read, snoozed, or surfed the web on their cell phones. The lucky ones were served snacks or lunch. Most were given steroids to prevent reactions from the treatment.

They were treated for an average of 2.7 years. At the end of the trial, an astonishing 61% of patients had suffered no symptom progression. Imagine how that 61% must have felt. Hopeful? Lucky? Excited? Ocrevus was a miracle drug!

I’ve left out a small detail. Those 244 patients were the placebo group of the study, and their “drug” infusions were saline solutions.

The patients weren’t aware of this, of course, since the Study was “double blind”, meaning neither they nor their doctors and nurses knew what was in the liquid that they were being injected with.

Now, let’s turn to the group of 487 patients who actually received Ocrevus during the trial. 67% of this group were lucky enough not to encounter disease progression during the Study. So, we can estimate Ocrevus was effective for 6% of those, while the other 61% would not have progressed anyway, just like the placebo group.

Why did so many people fail to progress in both groups? Well, PPMS is in some ways a slow moving disease. It takes 2-3 years, depending on the data set you look at, for disability to increase by one point on the 10-point EDSS (Expanded Disability Status Scale). And for disease progression to count in the Study, a patient must have increased by 1 point on the EDSS if he or she started with an EDSS of less than 5.5, or .5 points if starting with an EDSS above 5.5. The median patient EDSS was 4.5, so the majority of patients had to increase their score by a full point to “qualify” for progression.  

The moral here, is that if you’ve been taking Ocrevus for PPMS for two or three years, and your progression hasn’t increased significantly, it’s worth remembering that 61% of people on the placebo in the ORATORIO study had the same experience. If you’re feeling better, it may well be the Ocrevus. But the Study tells is that odds are it’s just the placebo effect.

Six Reasons Not To Take Ocrevus for PPMS (and Two Reasons You Might Want To)

Read this quote, from the FDA Website, carefully:

There is insufficient information in the application to conclude that ocrelizumab [Ocrevus] is a safe or effective treatment for PPMS. There is only one PPMS trial.

As it is, the trial results count events that may not have occurred, show inconsistencies among important subgroups, and lack independent confirmation.

In women, no beneficial effect balances the potential risk of breast cancer. In addition, there are problems with trial conduct and reasons to suspect the quality of the data.

John Marler, MD, Leader of the FDA Ocrevus PPMS review team.

Whether or not to begin an Ocrevus regimen is a decision many PPMS patients struggle with. Risks, costs, and benefits have to be weighed. While it’s natural to seek medication for such a debilitating disease, Ocrevus may be a bad choice for many. Below are six reasons you may want to say No to this powerful drug.

In the text below, “the Study” is the Phase 3 “ORATORIO” PPMS trial. “CDP” refers to Confirmed Disease Progression. The primary endpoint of the Study was evidence of disease progression, confirmed for at least 12 weeks, or “12 week CDP”.

1. You’re a woman

The primary endpoint of the Study found 32.9% of people in the Ocrevus group had symptom progression, versus 39.3% of those on the placebo:

Unfortunately, as the FDA quote at the top calls out, it had virtually no benefit for women. We while you’ll never see it in their marketing materials, only 1% more women in the Ocrevus group than the placebo group failed to progress:

You might think an important fact like this would be disclosed to patients, but that’s not the way for-profit drug companies work. However, this data was disclosed to the FDA, though you’ll have do dig through their 477 page Medical Reviews doc to find it (I’ll save you the trouble, page 168).

There are ways to spin Ocrevus’ benefit for women in a (slightly) more positive light. For example, Genentech published a hazard ratio (HR), is the relative risk of an event occurring at a point in time, given two sets of results. HRs have their own set of problems (they’re complex statistical regressions that can be manipulated, they suffer from survivorship bias, etc.). However, if you take the female HR Genentech reported for Ocrevus, .94, to be the actual odds that Ocrevus will halt your progression, you have a 6% chance of being helped by the drug. But there’s much data, including Genentech’s own long term studies, to make this blogger skeptical of that number. In any case, that .94 HR is only after Genentech, arguably, chose patients more likely to benefit from Ocrevus and used some creative accounting to make the results look better than they should have (see below). An HR of .76, for all patients was barely enough for FDA approval for Ocrevus. An HR of .94 would never be deemed effective, for any drug.

What caused this lopsided gender result? Apparently nothing in the design of the Study, which had a large population broken down evenly by gender. In 2017, the European Medicines Agency, the drug regulatory authority for the EU, demanded Genentech explain the discrepancy. The best it could come up with was to note slight differences between genders with active lesions (25.7% male vs 29.4% female of those receiving Ocrevus). The EMA was not convinced, stating:

In terms of biological plausibility, the figures on the potential influence of Gd-enhancing lesion distribution on the effect of OCR [Ocrevus] by sex reported in descriptive, univariate and multivariate  analyses  would  not  substantially  explain  the  lower  treatment  effect  observed  in  female  PPMS  patients.  The  same  reduction  of  OCR efficacy  in  females  is  not  seen  in  RMS.

In other words, nobody knows why the fairer sex didn’t respond to Ocrevus. But it’s not surprising. Many drugs affect males and females differently. Women considering this drug should ask their medical team hard questions about the benefits they can expect. And, incidentally, one of those should be about breast cancer – at least seven female patients in the Ocrevus PPMS and RMS studies reported a diagnosis of it, versus none in the placebo groups.

2. The Study didn’t select typical PPMS patients

As the quote from the FDA alludes to, Genentech made a number of suspicious decisions in designing and carrying out the Study – too many for a single blog post, but we’ll highlight a couple here.

Update: you can read more about Genentech’s creative patient accounting here: https://ppms.blog/2019/09/22/ocrevus-ppms-and-the-102-mystery-patients.

It’s important to recall that Genentech didn’t create the Study naively, as it would have a new class of drug. It had previously tested an almost identical molecule, Rituximab, for PPMS, and determined the subgroups for which is was most effective. So in this second study it could choose patients with characteristics similar to those who responded well in the first one. Unfortunately, regulatory agencies don’t dictate drug trial parameters – they’re at the discretion of the drug company.  

For the Rituximab study, the patient age range was 18-65. Other PPMS studies recruited a even older age range. The range for the 2003 Interferon-beta Phase 3 PPMS trial, for example, was 30-65. The age range for the ORATORIO study, however, was 18-56.

This was the same youthful population that responded well to Rituximab, but it is not typical of the average PPMS sufferer. The Cleveland Clinic, which has one of the leading MS practices in the world, notes, for example, that:

Most people start to have symptoms [of PPMS] at about age 50 or older, or about 10 years later than is typical of relapsing forms of MS.

The median age of the Study patient was 46, which means the majority of patients were younger than when the typical person even begins to experience symptoms, per the statement above.

This would be less of a problem if a drug were only allowed to be sold to the types of patients on which it was more effective (countries outside the US do a better job limiting drugs to certain groups). But Genentech markets Ocrevus to everyone. Study details – like effectiveness by age or gender – are buried in footnotes that many long suffering, neurologically compromised patients are unlikely to dig into.

As you might imagine, these younger folks hadn’t suffered from PPMS nearly as long as the typical patient. In fact, the patients in the Study were PPMS newbies, having been diagnosed only 1.5 years prior (median). Given that the median life expectancy for PPMS patients is about 70, the typical PPMS patient alive today has had the disease for more like 10 years. While the median patient in the Study had experienced symptoms for 5 or 6 years, this still doesn’t reflect the actual patient population.

3. Genentech Fudged the Study Results

Reducing the results of a drug study to numbers – p-values, hazard ratios – ignores the messy reality of drug trials. Despite the best efforts of all involved parties, the results are usually less scientifically precise than they appear. For example, trials often span multiple countries, causing variable results. The protocols and equipment used for an MRI in Canada may be very different from those in Croatia, for example.

A key problem in the Study was progression confirmation. Recall that the primary endpoint was confirmed progression. For the progression to be confirmed, each patient had to return to the doctor 12 weeks later. A few people – 9 in the Ocrevus group, 12 in the placebo group – never came back for that follow up appointment.

The conservative, arguably prudent, thing to do here would be to throw out these unconfirmed records, and consider only those who completed the trial. In a clearly effective drug, the removal of just 3% of the data wouldn’t alter the results. But in a marginally effective one, like Ocrevus, every data point counts. In fact, removing the non-confirmations in the Study would have resulted in a p-value of .14 for the primary endpoint, rendering the drug ineffective by FDA standards. So Genentech found another way around the problem. It assumed these patients did in fact return for a second visit, and that at this imaginary appointment, all disabilities were confirmed.

This is not standard practice. The FDA described it as “unusual” it its review of the study. But you can see why Genentech did it – 12 of the 21 non-confirmed patients were in the placebo group, despite there being only half as many total placebo patients. It’s not hard to imagine the many other creative ways the company might have solved this problem if those 12 patients had been on Ocrevus, not the placebo.

4. John Marler, Billy Dunn and the FDA

John Marler, MD, has worked for over a decade as a Clinical Team Leader for the Division of Neurology Products at the FDA. He was one of five physicians deciding whether Ocrevus should be approved for PPMS. After a careful review of the Study, Dr. Marler voted No.

Marler noted that the effectiveness of Ocrevus was marginal at best, and that it only passed the FDA p-value threshold due to Genentech’s creative accounting of the patients without confirmed diagnoses, as noted above.

But Billy Dunn, MD, Director of the Division, disagreed with Dr. Marler, and it was he who made the final approval decision on Ocrevus.

If Marler’s concerns were valid, why did Dr. Dunn overrule them? Dunn made two points repeatedly in the FDA review, which was written in the first person by him. First, it appears he decided to approve the drug simply because there was no other treatment available. Here’s one of many quotes from Dunn’s review:

Regarding product quality, I must reiterate that identified deficiencies would ordinarily preclude approval, but resolution of remaining product quality issues via initial adjustments to the control strategy, appropriate PMCs, continued process verification by the applicant, and continued implementation of corrective and preventive actions, is an appropriate strategy to support the approval of OCR [Ocrevus] given the unmet medical need that it will address.

If this seems like a questionable reason to approve a drug, consider the enormous pressure decision makers like Dr. Dunn face. In 2016, for example, Dunn refused to approve the Muscular Dystrophy drug eteplirsen because the study population in the trial was too small to show effectiveness. How small? Well, the ORATORIO study had 737 participants, and even then, had issues with statistical power. The eteplirsen study had 12.

The decision likely cost its maker, Sarepta Therapeutcs, millions of dollars in profits, and the pro-business Wall Street Journal pounced on the decision. In an editorial titled “Mental Dystrophy at the FDA” that ranted against “government bureaucracy”, it blasted the decision, claiming the FDA should have approved the drug anyway. It even attacked Dunn personally, calling him “arrogant” for suggesting that anecdotal evidence was not enough for FDA approval.

Anti-government rhetoric sells newspapers, boring scientific details don’t, and the Wall Street Journal knows it. And as the long history of snake oil shows, patients want medicine and hope, often whether or not a drug is proven to work.

I’m sure Dr. Dunn made more enemies that day than your or I will in a lifetime. Whatever the stresses you face in your career, having your decisions attacked in the global media probably isn’t one of them. While I have absolutely no proof that Dr. Dunn feels any such pressures when making decisions on marginally effective drugs, I do know how human nature works.

Compare this to the experience one has when approving a drug. From the FDA Ocrevus press release:

“Multiple sclerosis can have a profound impact on a person’s life,” said Billy Dunn, M.D., director of the Division of Neurology Products in the FDA’s Center for Drug Evaluation and Research. “This therapy not only provides another treatment option for those with relapsing MS, but for the first time provides an approved therapy for those with primary progressive MS.”

Billy Dunn. Hero.

The second reason Dunn overruled Marler is that the two disagreed on a key point – that Ocrevus’ effectiveness in treating RRMS should somehow spill over to PPMS. And Ocrevus clearly is effective for RRMS. In two Phase 3 trials, it reduced relapses in RRMS patients by almost 50% (p value < .0001) versus a control group taking another RRMS drug, Rebif, which itself is already an effective RRMS drug.

The problem is that PPMS and RRMS are distinct diseases in many ways. For example, while there were 10 RRMS FDA approved drugs at the time of the Ocrevus review, not a one had been shown to be effective for PPMS. In fact, over 50 trials using various drugs for PPMS have been attempted, all of which have failed.

RRMS, as the name implies, is a disease of relapses – reduce the relapses, and you reduce the progression. It’s also a disease of lesions, readily seen on a brain MRI. The more mysterious PPMS, on the other hand, is a steadily worsening condition resulting in damage mainly to diffuse brain matter. Its pathological mechanisms happen mostly behind, not across, the blood brain barrier, making it exceptionally difficult to treat. It shows up later in life than RRMS, and affects mostly men. RRMS affects mostly women.

The differences between the two MS types are apparent in the Study results as well. As noted above, Ocrevus showed almost no benefit to women with PPMS, but the positive effects on RRMS were just as strong for women as men. Given these differences, assigning a benefit to a PPMS drug based on RRMS success, without evidence, feels like wishful thinking. A flu vaccine won’t prevent the common cold, though the two diseases may have similar symptoms and causes.

5. The Side Effects

Genentech could be accused of misleading advertising in its claims about the benefits of Ocrevus. However, it doesn’t mince words when it comes to the side effects of the drug:

“OCREVUS can cause infusion reactions that can be serious and require you to be hospitalized.”

“OCREVUS increases your risk of getting upper respiratory tract infections, lower respiratory tract infections, skin infections, and herpes infections. ”

“If you have ever had hepatitis B virus infection, the hepatitis B virus may become active again during or after treatment with OCREVUS”

“OCREVUS taken before or after other medicines that weaken the immune system could increase your risk of getting infections.”

It’s not hard to see why these warnings should be taken seriously. Ocrevus has only one effect – to destroy “b-cells”, one of the two main types of white blood cells that protect your body from disease. And that’s certain to weaken your immune system. In fact, Genentech ended its Phase 3 study of Ocrevus for rheumatoid arthritis, despite the fact that the drug was effective at the 500 ML dose, because too many patients reported serious infections. During the ORATORIO study, four Ocrevus patients suffered non-accidental deaths, versus none in the placebo group.

Infusion reactions are another side effect suffered by 40% of Ocrevus patients in the Study (vs 25% in the placebo group). To counteract this, patients are administered powerful steroids during each infusion, and often antihistamines.

Visit any MS internet forum where Ocrevus is discussed and you’ll find a large number of patients who had to stop treatment due to repeated infections, despite the fact the drug has only been on the market since 2017. You’ll also meet plenty of people who tolerated the drug’s effects well. But make no mistake, Ocrevus is a serious antibody you don’t want attacking your immune system without a clear offsetting benefit.

6. You’re Paying

This is more of a philosophical point, since, if you’re an Ocrevus patient, your insurance is almost certainly covering your treatment. But bear with me.

The wholesale price of a year of the drug alone is $65,000, and the total infusion cost runs around $200,000 per year for many.  If I offered you, a PPMS patient, that amount of money, or two Ocrevus infusions, which would you take?

That’s the true test of the usefulness of any drug – whether the benefits are worth the sum of the costs and risks. In the case of very expensive, marginally effective products, it’s a question worth asking. And it’s a question countries with more cost effective health care systems do ask.

Take England. Genentech offered to sell Ocrevus to their National Health Service, the NHS, for about $22,000 per patient per year, versus the $65,000 cost in the US. At this price, the NHS determined the benefits were not worth the cost. After further review, and negotiations to lower the price of the drug even further, the NHS did approve the drug, but only, as noted below, in limited cases where it was proved most effective.

It’s not that the NHS is especially conservative when it comes to MS treatments – in fact, it offers stem cell therapy to some MS patients, which no insurance plan in the US will cover. But, looking at the points above, I hope it’s not hard to see why they found the drug only worth purchasing at modest price points.

Two Reasons to Consider Ocrevus for PPMS

Here are a couple of reasons, despite the arguments above, that you might still consder Ocrevus for PPMS.

1. You’re in a demographic for which Ocrevus is more effective

We know that men respond better to Ocrevus that women – following our hazard ratio logic above, about 11% of male patients will be helped by Ocrevus, and as possibly much as 39%. And we can try to segment that further. For example, the hazard ratio for people 45 and younger was .64 (for both sexes) versus .88 for those over 45. So Ocrevus was exactly three times as effective in younger patients.

Those with active T1 lesions are also better than average candidates for Ocrevus. “Active” means you were injected with a chemical called gadolinium, then had an MRI, and the gadolinium caused the lesions to appear as bright spots on the images. This proves two things: one, that the gadolinium penetrated the “blood/brain” barrier, and two, that blood is flowing through some of your lesions, which labels them “active”. A “T1” MRI is calibrated to highlight active lesions, while “T2” images show both active and inactive ones.  

Anyway, the hazard ratio, regardless of age, was .65 for this group, versus .84 for those without them. This makes sense, as we know b-cells play key roles in MS lesion activity. It’s no wonder that England will approve the drug only for patients with active lesions in the early stages of the disease.

If you’re on the fence with Ocrevus injections, demographics – and lesion activity – may be your best guides. You’ll also want to consider your overall health, beyond PPMS. The weaker your immune system due to other health problems, the more dangerous Ocrevus is probably going to be.

2. You want to roll the dice, and hope you get lucky

Ocrevus isn’t generally effective for women, but it did have a positive effect for a very small number of them. The same is true for patients over 50. You might get lucky and be in the minorities for which it works. You might get lucky a second time and escape serious side effects of the drug.

All this makes a critical assumption, of course: that the patients in the Study had the same response to Ocrevus as the general population would. We can say this confidently about the trial as a whole, but less confidently about the subgroups (age, sex, lesion activity, etc.). This is because the smaller number of patients in each subgroup reduces the predictive power of the results. So you’re going have to get lucky once more.

Ocrevus for PPMS Part One – the Five Ps of Clinical Trials

Is Ocrevus worth trying for PPMS? There’s no simple answer. This is the first of a series of blog posts to help you decide. Before we dig into the pros and cons, you need to be familiar with a few key concepts of drug trials. I call them the Five Ps, and they apply not just to Ocrevus, but any drug. Let’s get started:

Placebos

In a clinical trial, a group of patients are given a drug, then the its effectiveness is measured. In some trials, there’s also a second group that receives a placebo. This appears to be the same drug, but actually does not contain any active ingredients.  This is called a placebo control.

One example is all it should take to convince you of the importance of placebo controls: 49% of patients in the in the Phase 3 (see below) Ocrevus PPMS trial suffered respiratory tract infections. Was this due to the drug? With no similar placebo group, we have nothing to compare it to. Fortunately, this study had a placebo control, and 43% of those had respiratory tract infections. So we know most of the infections can be attributed to other causes.

Well designed studies are randomized, which means the drug company doesn’t get to choose which patient gets the placebo, and which gets their drug. As the name implies, patients are assigned to groups at random.

Usually, more people in a study get the real drug than the placebo, since you can get the information you need from the placebo group with a smaller population. Also, drug trials may switch groups after a period time, giving the placebo group the real drug, and vice versa, to determine if earlier treatment is more effective.

Some drug trials don’t have control groups, since the drug’s side effects make it obvious whether or not you’re getting the real thing. A chemotherapy treatment, for example.   

P-values

A “p-value” is a number that tells us how effective a drug is. Here’s an easy to understand example:

You and your buddy Bob play poker with some less experienced friends every day for a month. Then you each add up your winnings. Bob has won $62 per day on average, and you’ve won $57.

Is Bob a better poker player than you? Or did he just get lucky, by being dealt the right cards?

We can answer that by calculating a p-value – the odds Bob’s larger winnings are due to chance. For example, given these 30 days of poker winnings (in dollars):

You: 45,59,55,45,45,46,60,63,56,61,61,62,84,56,63,50,46,59,55,64,50,51,55,57,75,46,50,53,82,65

Bob: 65,72,62,65,59,68,68,64,61,62,62,59,48,59,47,57,58,60,67,68,68,64,58,70,52,65,68,62,45,71

The p-value is about .025. That means there’s only a 2.5% chance Bob’s higher winnings are due to luck alone. While we can’t say with 100% certainty that Bob is the better poker player, we can be highly confident that he is.

This is exactly the question we ask about the results of Ocrevus versus placebo – is it really an effective drug, or did it just get lucky in single study? What p-value would you need to consider a drug effective?

The requirement for drug approval by the FDA is a p-value of .05 – and it’s a pretty strict standard. In fact, in 2006, the FDA recommended against approving the prostate cancer drug Provenge because the p-value of one of test was .052.

Use common sense when looking a p-values. For example, a drug for a disease you have fails FDA approval with a p-value of .2. The drug is free, and has no side effects. Would you take it? Probably, because there’s an 80% chance it is indeed effective. Would you take it if it cost half your annual income, and came with serious side effects? Then that 20% might become important.

Phases

When a drug company thinks a drug shows promise, it usually tests it in cell cultures or animals first (monkeys in the case of Ocrevus). If the drug is shown to be sufficiently safe and effective there, four types of human trials may be initiated:

  • Phase 1. The goal is here is safety. This usually involves a small group of patients who are given varying dosages of the drug. There is no placebo group.
  • Phase 2. A larger group than Phase 1. Safety, effectiveness, and dosage are tested. Usually no placebo group.
  • Phase 3.  A larger group than Phase 2. A specific primary endpoint (see below) is defined. Placebo controlled. This is the key phase for FDA approval.
  • Phase 4. After regulatory approval. A study to determine the long-term effects of the drug.  

Drug companies like to give their trials fun and distinctive names, usually written in all caps. For example, the four Phase 3 Ocrevus trials for rheumatoid arthritis (RA) were called STAGE, SCRIPT, FILM, and FEATURE. The Phase 3 PPMS study was called ORATORIO.

Sometimes these studies are combined. For example, two studies were carried out to determine the effectiveness of Ocrevus for RA. The results:

  • Combined Phase 1 and 2 study: 237 patients were treated with varying dosages ranging from 10 to 1,000 mg per infusion. The results were promising, and optimal safety and effectiveness were achieved at the 200 mg dosage.
  • Phase 3 study: 1,932 patients. Primary endpoint: 20% improvement in a composite test of RA symptoms. Result: primary endpoint met, p value < .001. Unfortunately, due to a number of opportunistic infections, Genentech suspended development of RA with Ocrevus.

Primary Endpoints

A drug may have a number of effects on a disease. Ocrevus is a perfect example: it might reduce brain lesion load, improve walking times, and increase manual dexterity. At the beginning of a trial, the drug company states the specific benefits it expects the drug to have. These are called trial “endpoints”.

A clinical trial can have many endpoints, but it usually has just one primary endpoint. This is the critical benefit the drug must show for FDA approval. Why just one? It’s a problem of p-values. Here’s how the FDA describes it:

“In a clinical trial with a single endpoint tested at p-value = 0.05, the probability of finding a difference between the treatment group and a control group by chance alone is at most 0.05 (a 5 percent chance).

By contrast, if there are two independent endpoints, each tested at p-value = 0.05, and if success on either endpoint by itself would lead to a conclusion of a drug effect, there is a multiplicity problem. For each endpoint individually, there is at most a 5 percent chance of finding a treatment effect when there is no effect on the endpoint, and the chance of erroneously finding a treatment effect on at least one of the endpoints (a false positive finding) is about 10 percent.”

FDA, Multiple Endpoints in Clinical Trials Guidance for Industry

In other words, defining multiple endpoints weakens the power of any single endpoint, and increases the chance your drug may not be labelled by the FDA as effective for any endpoint at all.

To get around this problem, drug makers can declare “secondary endpoints” to support the success of the primary endpoint. Again, though, the drug almost certainly won’t be approved unless the primary endpoint is met.

It’s worth paying attention to endpoints because drug companies can manipulate them. For example, assume a drug company knows a drug is dangerous, but only after a patient has been taking it for a year. They could define a short study endpoint – say, 12 weeks, and ensure the drug appears safe in the study.

Populations

For a clinical trial, you need a bunch of people, and it’s important to pay attention to how these are chosen. There are three main things to watch for:

  • Quantity. The more subjects in the study, the greater the “power”, or statistical effectiveness, of the results. This usually isn’t a problem in Phase 3 studies, which have sufficiently large populations.  
  • Characteristics. Imagine you’re 60 years old, and have had a rare type of cancer for 10 years. An apparently safe and effective drug is approved for your disease. You’re initially excited, then you look at the trial population and notice something startling – the average age of the study patient is 30, and they’ve only had the disease for an average of two years. What’s safe and effective for this group may not be for you.   
  • Dropouts. In the Ocrevus Phase 3 PPMS study, the primary endpoint was “time to confirmed disease progression”. Here, “confirmed” means the patient was tested again, 12 weeks after the initial progression, to confirm it.

Now imagine a patient who tests positive for progression, but drops out before it can be confirmed 12 weeks later. How to count him?

Or imagine a patient dies midway through the study. Was this due to the drug? How do you count him from a safety standpoint?

In cases where a drug is found to be highly effective, dropouts and patient characteristics are less important. But in a marginal product like Ocrevus, they become major issues, as we’ll see in future blog posts.