10 . 24 . 14 | Dr. Little Shares His Take on Dey’s Book

Dr. Kevin Little, Principal and Founder of Informing Ecological Design, LLC recently penned a review of Kieron Dey’s Competitive Innovation and Improvement: Statistical Design and Control. We had expected the book to be controversial among academics and researchers since it contains no mathematical notation and also explains common errors in experimental work by inappropriate use of mathematical models in both design and analysis. So we were pleasantly surprised to find an independent review by Dr. Kevin Little (PhD is statistics from University of Wisconsin at Madison) an experienced consultant with unusually strong technical skills in this field (as well as many others).

Dr. Little provides a balanced review rightly noting nothing new in the statistical control sections and centering on the parallel use of statistical design and control in unorthodoxly large and diverse business settings, as especially innovative.

The review also notes the case in Chapter 1 alone as being worth the price of the book; this is good insight since it frees healthcare of the widely perceived need to randomize patients to the experiment allowing improvement in live care/disease management operations. It also shows why measurement error is never an issue in problems of this type (unless "cleaned up"!) The case was early in a set of about a dozen similar cases (some of which were of better design or larger improvement) and was chosen in part because it shows how real cases look, imperfections fixed along the way. The dozen cases met with fierce resistance from statistical colleagues which was important to sound design and overcoming similar unfounded concerns in the future. Chapter 1 conveys the sparks caused by pulling it uphill in this way as a feature of good science.

View the full text of Dr. Little's review on his blog: 
http://iecodesign.com/index.php/our-blog/208-statistical-design-and-control-new-book-by-kieron-dey

04 . 03 . 14 | New Book: “Competitive Innovation and Improvement”

When asked why the book was written (and a little about what’s between the covers,) Kieron Dey said:

“I first got the idea about separating tiny signals from large amounts of noise from time spent in radar design and wondered why similar was not much used in industry to solve problems. Where statistical design was used, it tended to be on a small scale and not much in processes involving lots of people.

The idea to combine statistical design and control came from a book on survey sampling. This fusion was controversial for years among professionals, for no reason. Everything used is in the literature. 

“Intent-to-treat" is also used throughout (which means, roughly, allowing an element of laissez-faire, to get real world results, not forced ones that don't hold longer term).

 Simultaneous design (where more than one design runs at once, overlaid) was added in 2012. The simultaneous designs have been important in cross-channel optimization in retail, and in complex healthcare improvements. This was the last addition as the theory was tricky and it finally fell in place in 2011. It found that what had seemed weaknesses (where interactions across designs might be a problem) in fact hid a large strength, which is in Chapter 8 with real cases. The method had to be simplified so that users could apply easily.

Finally, the scientific method is used throughout (which folds nicely into comparative effectiveness research, DMAIC or PDCA etc.) and the book explains what (and how simple) this is. The scientific method allows the same method to be used for existing and new processes: hence the “improvement and innovation” in the title. Innovation becomes less elusive in this way – it can be designed rather than waiting for inspiration. Also, getting back to pure, simple science means using right-brain (creativity) as well as left (analytic) so more people can contribute, valuably for the enterprise (which can be business, industry, research or government).

There is no mathematical notation so that anyone can read and use these well-established methods. Scientists and researchers will find Chapter 8 challenging on scientific method and randomization, so there's something for everyone. Mathematics is used a lot behind the scenes of the book but the real world is used more: to understand how businesses work and make them work better.

There are about 20 exercises peppered through the book, for the reader to accelerate what would be learned in field experience and get started on real business competitive problems.

Surprisingly, it turns out to be a management tool, not one technical people alone can accomplish; it’s not top down though and the book explains why.

The Book is available for pre order on Amazon at: goo.gl/9QSVMB

11 . 11 . 13 | Efficient Orthogonal Design

The March 2013 government publication Efficient Orthogonal Design (AHRQ 13-0024-EF) authored by Mathematica under the direction of Dr. Randy Brown, is now gaining attention from health plans and the healthcare industry as a whole.

The research used in part informal working sessions comingout of the multi-year collaboration between Nobi’s Kieron Dey and Dr. Brown. Citing also published papers by KK Moore, the report is important for its introduction of large, orthogonal designs to test many interventions simultaneously in healthcare, also known as comparative effectiveness studies within the healthcare industry. The paper highlights an example testing about a dozen interventions in a disability study that worked to reduce hospitalizations within a population including both physical and behavioral disabilities.

One of the publication’s strengths is in adapting the original theory into a practical guide that any organization can adopt using analytical expertise already on staff.



See More:
- The Agency for Healthcare Research and Quality
- AHRQ Efficient Orthogonal Design Article Search

04 . 15 . 13 | Broad Based Economic Control

Our previous two posts introduced and explored economic control. The first post, dated March 18th 2013, was an introduction to this principle. The second post, dated April 4th 2013, illustrated in more detail using the healthcare industry. As the third and final part of this series, we’ll explore how economic control can serve any business, and any industry.

Economic control charts find their own level, meaning they find the stable process (“level playing field”) within the instabilities (spikes etc.). The “tramlines” reveal the inherent stable process. There are strict, objective rules in the mathematics for how this is done. Once set up, no changes are made to the limits (“tramlines”) until the process improves, as determined by the chart. New points are added each day, week or month, without changing the limits. Modern software encourages the limits to be recalculated each time but this is wrong (like moving goalposts around). 

The landmark text that introduced economic control in 1931, in a brilliant 300+ page development, used physics, mathematics, statistics and economics. Suddenly (on page 304) appears the seemingly simple formula: process average ± 3 standard deviations. This became known as the control chart. More correctly, it’s an economic control chart [1].

The important element of this deceptively simple formula (used in most industries for nearly 100 years now) is that the control limits are set economically (so that management actions will save and make most money). It is often said that control charts have a false-alarm rate of 0.27% but this is irrelevant. Nowhere in the original text does that number appear. The whole point is the limits are set economically. Further, the 0.27% is inexact in real processes, so it’s a red herring at best.

First impressions of the 3-sigma limits are often that they’re really wide and we should have a tighter standard. In fact they’re not “wide”. They are what they are. This error arises from missing the economic aspect.

It’s also often said that variations inside limits are random. They are not. All fluctuations (large or small) are caused by something(s). Now, the data within limits do often follow random patterns. So when someone says: “What if that point outside limits is just a false alarm?” the answer is there is no such thing. Since all variations have causes, a large one is by definition worth something. The hand-wringing comes because it is thought it may be pure chance. It isn’t. Now the causes that conspired to create a large number may have fallen into a perfect storm by chance, but they’re still real and there’s money to be made. This is a little like serendipity. No-one minds that we stumble into breakthroughs serendipitously. Same here.

The way to see this is to think of variations inside limits as having many causes, unknown. These will be impossible or expensive to figure out. So we use statistical design to do that. Variations outside limits will be economic to figure out and fix, ignore (e.g. month effects) or bake in (if good). In other words their cause(s) are easily found out and exploited for improvement.

Points outside limits (and/or a few other patterns) render the process unstable and are easy to figure out, then remove (or bake in if it’s a good spike), to make the process stable (standardized).

Here we’re not naïve that all processes can be made stable (standardized), such as if weather is involved in outside plant. Then we use standard workarounds. In general though, economic control gives a method to assess and standardize processes and their measurement systems.

If an improvement has just been implemented and a single point then crosses the limit (in the improvement direction) then it is a very big deal. This is where that crude, approximate rule of thumb of 0.27% comes in handy. If we’re plotting data monthly then we’d only expect to cross that limit roughly once every 62 years. So this is like a 62-year flood. (100/(0.27/2) x 12 = 61.7 years.) That information alone adds management fuel to the emerging improvement.

There is a deeper aspect to economic control. All of statistics (in the main techniques statisticians call parametric) is based on distribution theory. Averages, standard deviations, significance tests, regression etc. all are based on this. Certain mathematical requirements precede use of all these things. When we take courses in statistics we see those but read them much like the small-print in a legal contract. Most fundamental in that small-print is that the process be stable. If it is not, distribution theory breaks down. So, for example, even a simple pre-post test to see if something we did improved the process will be wrong if we ignore instabilities. “Wrong” is not strong enough a word – a better one would be arbitrary. Economic control is the only way to adjust so that we find the correct answer. This is profound for competitive advantage in a business.

This surprising comment that all of statistics* breaks down on unstable processes, is in the literature under the heading of analytic statistics. It is not well known even among qualified statisticians.

Industrial processes are almost always unstable, so statistics will not work unless the adjustments are made. The adjustments are simple, fast and follow rigorous rules that cannot be bent.

Anyone can do economic control charts. They are self-teaching devices where experience increases skill in usage. Common sense and a copy of the formulae/rules is enough to get started. Access to an expert  speeds this learning curve and avoids stumbling through common mistakes.

This all happens very fast. Economic control does not slow the business but speeds it. If a business were able to keep up with economic control it would be moving fast indeed. With a little practice, firms will find the economic control charts ready and waiting each morning. Of course they are only used on a few big things that make most money fastest.


* Statisticians use a code here: i.i.d. ~ independent, identically distributed

---

REFERENCES:

1. Shewhart, Walter, A. Economic Control of Quality… Van Nostrand (1931)

04 . 02 . 13 | Healthcare Economic Control

Building on our previous post which introduced the principles of economic control, this set of charts shows the work that preceded the healthcare case described in Case Studies: Healthcare | Health at Home, Not Hospital.

The first of these shows how measurement error (meaning “noise” not “mistake”) was initially unstable but was quickly fixed by removing non-applicable cases. The chart’s spike revealed this flaw in the tracking systems and found it also throughout all the data (not just the spike). That stabilized measurement error (i.e. the data all then fell inside tighter tramlines). Calls for measurement perfection were advised against since it would have been uneconomic (i.e. a severe drain on resources) and often impossible. This economic aspect is one of the most valuable features of economic control.

 

The second chart shows that the nurses (and the patients they cared for) in the largest of 3 simultaneous statistical designs, are homogeneous in terms of chronic health events, similar to the retail stores in the first post. So this assured a “level playing field” for the study that followed.





The next pair of charts are a hybrid. On the top chart, the gray areas are measurement error (i.e. noise) and the outer limits are the process (i.e. chronic events). This gray area offers a simple way to always know measurement error will not get in the way. It is clear the gray portion is not obscuring the chronic events month by month. The gray is about a quarter of distance between the outer dotted lines. Since statisticians use squared (not linear) distances, only about a quarter² = 1/16= 6 ¼% of the process is really obscured. A good rule of thumb here is 25% tops, but 10-15% preferred.

This is surprising, given the visual impression, so the square law clarifies.

The lower chart is also measurement error but looks at precision (i.e. how much measurement error varies). The top chart was measurement accuracy (i.e. how close to the true mark it gets and how well it discriminates process shifts).

 This simple hybrid method allows processes to be improved and all questions about measurement error answered (really pre-empted) in real time, in the months ahead.

This case produced about a third improvement (against experimental prediction of a quarter) in a 3-month study plus a couple of months to solve implementation problems. The implementation population was double the size of the random sample used in the study.

03 . 18 . 13 | Principles of Economic Control

Nobi’s large statistical designs (to test which of 20+ changes improve a business quickly, and quantify by how much) are well understood. Less well known is the novel way we integrate economic control.

Economic control charts just plot the measurement we’re improving, then place limits (that look like tramlines) showing the extremes the process will normally confine to. Some examples will speak best.

The chart above ensured a “level playing field” when randomizing retail stores to a statistical design for testing 11 changes in stores and marketing, bringing a 9.8% “comps” increase from 2 of the 11 (see also homepage Case Studies: Retail Sales | Innovation During a Recession).

The next chart shows an inventory reduction project that started when it did.

Usually there are gaps to close in initial implementation, which typically take 1-2 meetings using the scientific method and economic control. Although little or no adherence monitoring is needed in general, it is used more to close implementation gaps, in order to know what’s been going on at all times and places. A small, random sample of transactions is inspected for the changes being implemented. The work is completed with people in the trenches. This is quite subtle in the details but with good business advantage.

Economic control is essential when using statistical designs for rapid improvement. It’s simple to use, though deceptively clever. It is important to managers in guiding economic decisions, and to scientists in providing objectivity in unstable business data (where the rules of statistics break down).

11 . 29 . 12 | Continued Push Into Healthcare

2012 cases in healthcare numbered roughly a dozen in:

• Reducing hospitalizations
• Reducing re-admits
• Reducing exacerbations for disabled populations
• Increasing engagement in CM/DM and in wellness programs 
• Improving treatments
• Streamlining utilization management
• Predictive modeling for CM/DM selection

In all these studies, the device Nobi pioneered of cluster randomizing (e.g. by nurses, not patients) made the findings easy, fast and pure. This has remained controversial (without reason) but with more researchers adopting, the issue is inching toward more mainstream acceptance.

In most cases, widespread acceptance has required backing out precursors (such as HCC score) analytically, to show the findings do not change (and instead strengthen). No surprise here since the theory has been around since the 1920s but remains notoriously hard to grasp. These more practical exercises of showing users in their own language have been well received.

11 . 19 . 12 | The Control Book

An ingenious management tool as client efforts expand to several statistical designs was suggested to us a while ago by a veteran CEO. It is a seemingly simple report, with one page for each project showing the improvement in the main measurement, annotated for actions taken.

In assembling the first edition for monthly update, the organization reaches agreement on the operational definition of the measurement. Discussions that take place after this discussion are more focused. The Control Book allows executive teams to manage the improvement effort in a short meeting every month. Financial information is also footnoted so that the overall ROI is clear at a glance every month. The idea is simple. Its execution is deceptively difficult but consumes little time. The tool also makes implementation straightforward whereas it can otherwise be elusive.

The reasons this simple tool can be difficult to introduce differ by organization and are best discovered by each client. Once in place and with a monthly forum for managing it, the tool is remarkably powerful and much liked. It is harder to accomplish than the statistical designs, mainly since sustained implementation is the hardest part of improvement.

The Control Book is the most effective way to manage implementation, short and long term.

08 . 01 . 12 | Nobi Presents At ISRN Event

Nobi Speaks at the 2012 Summer Institutes on Evidence-Based Quality Improvement

Following the Network News article (3/20/12) Nobi was invited to speak at the 2012 Summer Institutes on Evidence-Based Quality Improvement Seminar held July 17^th to the 21^st in San Antonio, TX. 

Tracing the origins of the scientific method through the 1923 discovery of statistical design to pioneering work in application to CM/DM generally, Nobi’s Kieron Dey provided case studies and fragments to illustrate a new but long proven method for practical innovation in healthcare generally.

Aspects of management, science, statistics and economics were summarized together with what will cause mainstream adoption from an understanding of its prevention to date.

Insight into organizational dynamics and human nature that are an essential aspect of the scientific method resonated with the 160 clinicians, nurses, physicians, administrators and graduate students in attendance. These elements revealed how implementation (the hardest part) has been made straightforward in delivering and sustaining results predicted by studies. Central to the rapid cycle time for innovation the talk emphasized the ability of orthogonal design to evaluate 20+ changes to treatments or clinical models simultaneously without increasing sample size over RCT norms. The upshot of thereby evaluating over a million potential treatment variants was explained.

Advanced mathematical constructs employed in orthogonal design were simplified. For example the meaning of orthogonality itself (the device which allows cause and effect of 20+ interventions at once to be established) was demonstrated visually with a newspaper. The surprising fact that false alarm rate decreases when testing 20+ interventions was explained and proven with actual data.

Further speeding safe research, the reason why single-shot large orthogonal studies require no refining or validation testing was demonstrated by case example (as is known from a full appreciation of the subtending theory).

Interest continued by correspondence with attendees after the conference including “I thought “yawn” ..but, found [orthogonal design coupled with economic control]...the most interesting and provocative topics at the entire conference.”

07 . 16 . 12 | Disability Study

Nobi is advising on statistical design in a study that began in 2010 to improve health thereby reducing hospitalizations for disabled people. Three health insurance plans are participating, one of which is for disabilities due to severe persistent mental illness (SPMI), by examining the effects of sets of 11 interventions differing by plan somewhat.

The study employs large orthogonal designs with a few dozen nurses (care managers) and the thousands of people they provide telephonic care to. Health outcomes, preventive measures and acute admits are being tracked for analysis.

Typically, these designs have reduced hospitalizations from 5-20% with notable findings such as novel ways to reduce falls. Because each hospitalization costs an average of $10,000 significant savings in the $1-10 million range per study implemented are on file.

Study results are expected in late 2012. Implementation will then follow using the interventions found helpful. Interventions explored range from medications to counseling and screening efforts for the SPMI project, to educating patients on fall risks and changes to the care model for the chronically physically ill patients.

More Information Contact:

Michael Joliat
Tailoj Marketing
m@tailoj.com

03 . 02 . 12 | Randomized Control Trial

Statistical design and the Randomized Control Trial (RCT).

The first recognized RCT in medical research was published in 1948 [1], following earlier work over several years. It remains a mainstay of the industry and rightly so. Statistical design has enormous implications to build on RCT across the healthcare industry:

• It allows studies to be freed of the stifling restriction of randomizing members to nurses (or similar) in Care Management/Disease Management (CM/DM).
•  It allows a treatment, such as a medication, to be optimized by dose, frequency, and other synergies. As well as proving out the basic treatment, and for no increase in sample size of subjects.
• It avoids any “roulette” with the test subjects, in which half the subjects get a placebo. Instead, all are potentially advantaged provided all treatments and other variants are clinically founded.
• It strengthens blinding since every subject is assigned about half of the total interventions tested, and in a way no-one can second-guess or influence until the researcher has analyzed the data.
• It measures what happens in the real world as opposed to one in which the subjects may know they have a 50% chance of a sugar pill or other placebo.
• It uses “intent-to-treat,” rather than a Pyrrhic test of enforced adherence. This of course can be used in an RCT, too.
• Sham studies are considerably strengthened by dropping the device or procedure being tested among an assortment of other treatments and variants.

It is often supposed that while a statistical design offers advantages, the RCT must be more pure. In fact it is the other way: Fisher’s wider basis for induction [2] simply meant that if a treatment worked among so many other things, also varying, then it worked in the real world and not an artificially controlled one.

---

REFERENCES:

1. Marshall, Dr. Geoffrey, et al. (1948) Streptomycin Treatment of Pulmonary Tuberculosis. British Medical Journal.

2. Fisher, R.A. (1935) The Design of Experiments. Oxford University Press (Reprinted 2003) Pages 13 – 26

02 . 05 . 12 | Randomization Distribution

Randomization Distribution | Modernizing An Old Device for CM/DM and Multi-Channel Optimization Generally.

Rampant misunderstandings about multi-intervention studies in care/disease management (CM/DM), other human studies (e.g. sales, education)and non-manufacturing generally are:

1. Differences among test units (e.g. nurses, retail stores, students) will invalidate the study.
2. Adverse selection could contaminate results (e.g. recently hospitalized; new stores, gap students).
3. Influences like regression-to-the-mean, HCC-score, store-trends, size etc. will affect findings.
4. Members (patients) should be randomized to nurses (care managers); ditto students/classes.
5. A 2nd. Study or RCT would increase confidence or “validate”.
6. False-alarm rate will increase with the more things tested.

Since 1926, Fisher [1] hasn’t been well understood by mathematicians. This matters more today in healthcare and complex multi-channels generally. Using also a couple of other devices, we fix (simply for users) all of the above and free of man-made constraints that have held back progress. Using the CM example then closing with the multi-channel sales example to illustrate for all industries:

#1 is completely solved by the simple trick of control-charting pre-study results across nurses from a time-window equal to the planned (randomized!) study period and finding homogeneity (akin to stability in manufacturing). A dry-run pre-study analysis (like a Heckman-Hotz econometric test) is essential or patterns among nurse admit rates can still give spurious findings. This dual homogeneity check has been controversial among PhD statisticians and academics without reason. Users, more correctly, have no trouble with it.

#2 to #4 is solved by correct randomization of nurses to the study. #4 is popular among mathematicians but usually wrong and then would not improve anything or make money. Closed cohort design will re-assure everyone more but isn’t necessary and cannot always be done (e.g. transitional care). Of course long-term validation must have a closed cohort or propensity scored analogy, not open cohort.

#5 is a little like sending a rowing boat out to see if it was safe to sail the ocean liner that just sailed through. Confidence remains about the same off similar results and studying new things for a shorter time trumps replicating, even with one-factor-at-a-time testing [2]. Of course if a 2nd study is conducted, even randomization would not allow it be done on the same sample! A fresh random sample is called for: just like in manufacturing.

#6 goes away at ~20+ interventions: false-alarm is a problem for small studies devoid of scientific context.

Multi-channel optimization (e.g. sales in stores, online and stimulated through media, and call centers; or care by nurses, pharmacists, automated systems and house-calls to the same population) is solved by simultaneous statistical designs provided randomization is correctly deployed and the channel tests set up with a clever new device that’s easy for users (mutual-orthogonality). Education cases to improve learning/careers, then all industries follow the same model.

Optional Note for Professional Statisticians: In a 20 run design, a contrast is simply the calculation to find the effect of one intervention, simply by averaging the 10 tested vs. the other 10 (counterfactual). Comparison to all possible 10 vs. 10 contrasts gives a yardstick to see if the effect is “real” or essentially by chance. The histogram on the home page is of 10,000 random contrasts out of more than 200,000 calculated. There are ²⁰C₁₀ = 184,756 contrasts in total and the extras were run so that every contrast was more likely included. The histogram of contrasts is from untransformed response data.

The average of the 200,000+ contrasts is -0.07337 with maximum at 419.59 and minimum at -410.97.

The true contrast is at -261.39, from a CM/DM case measuring hospitalization rate per thousand people per year.

The histogram shows that a normal approximation is close, as expected. However the calculations are all distribution-free. No original assumption of normality (a “bell-shaped curve”) is needed. The histogram indicates visually how well all possible contrasts approximates normal. The true p-value (from this randomization distribution) is 0.01 vs. a normal approximation (from linear model software) at 0.0033.

Of course the actual contrast will only be absolutely largest among all randomized contrasts (p-value equals 1/184,756= 0.0000054) if the treatment and counterfactual have no overlap in the raw data (as driven by effect size). 

Further insights based on the above reveal why randomization of nurses to treatment combinations is correct but of members to nurses is usually not, as it cannot be often managed that way. Also that modeling nuisance variables (such as prior admit rates, HCC-score, selection criteria etc.) will approximate the same answers as the correct analysis relying on randomization directly. (This back-end analysis first suggested by Neyman in the 1930s is not needed. Were it to differ, one might look for multi-colinearity problems or otherwise check the model.) Of course running such a model to see if randomization “worked” is folly. It will always “work” provided the device is used correctly, which it tends not to be. Further consideration also reveals why a single test unit per combination in the study design is ample (in 20+ intervention designs) and replication is a waste. Finally, that a common mistake is to try for n>30 per combination whereas  n=1 is usually ample.

On misunderstanding #1 (homepage) a trick of analyzing change in admit rate by nurse (since the prior) can be popular. But, this is only valid if that prior is significant and then a covariance analysis is used since the “change” adds noise and can cause errors. 

Power calculations are performed in the usual RCT way (and yield identical sample size requirement) but miss the point that they will be pessimistic as variation usually reduces during large studies. Our earliest case found the standard deviation about 1/3 of the prior. The far larger issue is that single-intervention studies (excluding, say, 19 that could have been included with the same resources) have zero power for all the untested things. 

It has not escaped our attention that performance often improves from day 1 of large studies, rendering them attractive to businesses especially if in urgent need of step-change. Also that large studies stop any accidental roulette with customers, members, students etc.

---

REFERENCES:

 1. Fisher, R.A. (1926). The Arrangement of Field Experiments. J. Min. Agric. G. Br., 33: 503-513 
 
2. Box, G.E.P. (1966). A Simple System of Evolutionary Operation Subject to Empirical Feedback. Technometrics. Vol. 8, No.1