When Bad Science is Torturous: The Narcissist Practicing Poor Methodology Long Before They’ve Mastered the Material of Methodology Part 1
TW: R*pe, torture. When Bad Science is Torturous: The Narcissist Practicing Poor Methodology Long Before They’ve Mastered the Material of Methodology Nevertheless Engaged in Science in “Wrestling With Proteus: Francis Bacon and the ‘Torture’ of Science” and “Protect us from poor-quality medical research.”
Citation: Pesic, P. (1999). Wrestling with Proteus: Francis Bacon and the" torture" of nature. Isis, 90(1), 81-94.
Link: Wrestling with Proteus: Francis Bacon and the" torture" of nature
Citation: "Protect us from poor-quality medical research." Human reproduction 33, no. 5 (2018): 770-776.
Link: Protect us from poor-quality medical research.
Full disclaimer on the unwanted presence of AI codependency cathartics/ AI inferiorists as a particularly aggressive and disturbed subsection of the narcissist population: https://narcissismresearch.miraheze.org/wiki/AIReactiveCodependencyRageDisclaimer
Protect us from poor-quality medical research
Much of the published medical research is apparently flawed, cannot be replicated and/or has limited or no utility.
Much of the published medical research is apparently flawed, cannot be replicated and/or has limited or no utility. This article presents an overview of the current landscape of biomedical research, identifies problems associated with common study designs and considers potential solutions. Randomized clinical trials, observational studies, systematic reviews and meta-analyses are discussed in terms of their inherent limitations and potential ways of improving their conduct, analysis and reporting. The current emphasis on statistical significance needs to be replaced by sound design, transparency and willingness to share data with a clear commitment towards improving the quality and utility of clinical research.
Doctors need methodological training in order to critically appraise the quality of available evidence.
Doctors need methodological training in order to critically appraise the quality of available evidence instead of taking all published literature on trust (Ioannidis et al., 2017).
Credibility and utility problems are found in clinical trials and clinical research; big data and large observational studies; and systematic reviews (SRs) and meta-analyses (MAs).
We recognize upfront that perfectly reliable/credible and useful research is clearly an unattainable utopia. However, there are many ways in which the existing situation can be improved. In the following sections, we overview challenges in credibility and utility that affect medical research at large and then focus on specific challenges that are more specific for some key types of influential studies: clinical trials and clinical research; big data and large observational studies; and systematic reviews (SRs) and meta-analyses (MAs).
Significance results are surprisingly common and when run through more advanced techniques show a Janus phenomenon where different analyses of the same data provide conflicting results to the same question. That said, not all of these analyses are equally well done. This should only be considered an accurate Janus effect when the analyses are both equal.
Although reliability and utility are critical, most research studies primarily aim to obtain and present significant results. Significance itself can be conceptual, clinical and statistical, each carrying a very specific meaning. Statistical significance (typically expressed through P-values obtained from null hypothesis testing) is almost ubiquitous in the biomedical literature. An overwhelming majority of published papers claim to have found (statistically and/or conceptually) significant results. An empirical evaluation of all abstracts published in Medline (1990–2015) reporting P-values showed that 96% reported statistically significant results. An in depth analysis of close to 1 million full-text papers in the same time period identified a similarly high proportion with statistically significant P-values (Chavalarias et al., 2016). Simulation studies have shown that in the absence of a pre-specified protocol and analysis plan, analytical manipulation can produce almost any desired result as a spurious artefact (Patel et al., 2015). Multiple analyses of the same dataset can lead to results which demonstrate variations in both magnitude and direction of effect, occasionally leading to a Janus phenomenon where different analyses of the same data provide conflicting results to the same question (Patel et al., 2015).
P-values have to be in conversation with effect sizes and confidence intervals. A black-and-white “easy P” value problem plagues science as well.
So far we have focused too much on P-values. The P-value suggests a black-and-white distinction that is elusive (Farland et al., 2016). Effect sizes and confidence intervals are to be preferred in studies in the context of clinically relevant questions, biological plausibility, good study design and conduct.
Big data hype can cause bad data to be used and put into action before examining the quality of the big data resulting in disastrous consequences to everyone.
The advent of big data (see below) allows for more ambitious analyses but most available data are of questionable quality and the chance of uncovering genuine effects tends to be relatively low mainly because of high risk of bias. Bias is separate from random error; while random error affects the precision of the signal and big data diminishes the random error, bias may create signals that do not exist or may inflate signals or cause signals in the entirely wrong direction. The availability of big data has been perceived as the dawn of a new paradigm which liberates researchers from some of the more stringent aspects of scientific rigour such as a clear hypothesis, pre-planned analysis, validation and replication, but this is wrong. Hype surrounding new technologies can sway the best academic institutions and innovative entrepreneurs, leading to false expectations about what the new tools and massive datasets can deliver.
Conflicts of interest are not given the diligence they deserve either, even sometimes showing that the conductors of experiments don’t even understand what they are or are willfully blind to them. A moment of such laziness can have profoundly destructive effects on all the adjacent science.
“While recent years have seen major improvements in reporting of conflicts of interest, many continue to go unreported, and there is a growing realization that non-financial conflicts may have a bigger impact than previously imagined. High-level evidence synthesis (e.g. SRs and MAs) and guidelines may help streamline some of the uncertainty surrounding the available evidence and facilitate medical decision-making.”
Guidance exists for randomized trials, diagnostic tests, and meta-analyses that is not taken as seriously as it should be simply because many people don’t understand it and just do what their peers do who also just don’t understand it. Leaders for quality control in science are sparse because most don’t understand what’s at stake and have a profitability addiction.
Reporting guidance exists for randomized trials (CONOSRT), as well as for other types of clinical research, e.g. STARD (for diagnostic test studies), PRISMA (for meta-analyses) and IMPRINT (specifically for fertility trials). These guidance documents aim to improve the quality and completeness of clinical research reports (Glasziou et al., 2014). It is very disturbing that comparisons of protocols with publications in major medical journals revealed that most studies had at least one primary endpoint changed, introduced or omitted (Chan et al., 2004, 2014; Glasziou et al., 2014).
Data entry is viewed as no cause for respect and studies may be subject to corruption at the very ground intake level in the sloppiness and inaccuracy with which data is originally collected and recorded. A good ground level data entry employee can be the unseen factor between a successful and not successful study.
. Inaccuracies in the data can occur due to mistakes in data entry and lack of appropriate checks. Routine data are also likely to contain a minimum set of variables and many key confounders such as body weight, height, smoking status, alcohol intake and socio-economic status may be missing
When data is missing in a non-random fashion, bias becomes clear and a will toward the result willing to sacrifice congruence with the natural world becomes the corrupting influence that can do damage to the scientific “code” as experienced as scientific papers as networks of citation. It does damage to all adjacent material. It is not an isolated corruption.
Finally, data is often missing in a non-random fashion thus introducing the possibility of bias. While some ways of dealing with missing data (Jagsi et al., 2014) are better than others, it may be difficult to address missing data with high confidence.
Informed consent, anonymization, individual risks, and who should be informed of what risks also require precision and diligence.
. These concerns involve lack of informed consent; possible identification of subjects during linkage procedures (even after anonymization); the dilemma of dealing with detected individual risks in an anonymised (rather than anonymous) population, who could potentially be identified and informed; and individuals in very small categories of groups with unusual conditions.
Inflated treatment effects are seen too.
They tend to generate inflated treatment effects even when sophisticated propensity score methods are used (Hemkens et al., 2016b).
Interpretation must therefore be cautious, despite the statistical significance.
Studies based on large datasets can have sample sizes that are so large that they detect very small and clinically unimportant effect sizes. Such studies should be interpreted appropriately according to their clinical significance. Highly statistically significant results may actually be attributable to bias (Peek et al., 2014). With small effects, bias or confounding cannot be excluded. Interpretation must therefore be cautious, despite the statistical significance.
Metaanalyses can direct bias and identify gaps but they are completely disempowered if the original data can’t be trusted or is unreliable.
However, very often the primary data feeding into SRs and MAs are so unreliable that these may have a more important role in detecting bias rather than uncovering the truth. SRs and MAs may also help identify gaps in the use of patient-relevant outcomes where multiple studies exist but outcomes that matter are not addressed.
Many metanalyses are being conducted in China, which has had a peer review network rot detected in it due to grandiosity issues insidious to the primary funder in most cases, even for private capitalist firms still engaged to a large degree in China, the CCP.
The profile of SRs and MAs has changed over the last decade, with increasing numbers of MAs now being generated in China. Most of these MAs are unreliable or misleading (especially the bulkproduced meta-analyses of candidate gene associations). Moreover, there is a new large portfolio of MAs conducted by contractor companies that are commissioned and paid by industry (Schuit and Ioannidis, 2016). Only a small proportion of these MAs are published and publication bias may be related to the results of the MAs and the interests of the sponsor. An online search suggested that over 100 service-offering companies perform SRs and MAs (Schuit and Ioannidis, 2016).
While research fraud (results not reproducible) has been considered up to this point uncommon, the temptation to cut corners prompts many authors to indulge in poor-scientific practices.
A 2016 Nature survey showed that more than twothirds of scientists believed that there is a reproducibility problem (Baker, 2017). Replicability is a benchmark of scientific quality; authors should always try to replicate their own results and provide sufficiently detailed instructions for others to do so. While research fraud is uncommon, the temptation to cut corners prompts many authors to indulge in poor-scientific practices (Tanksalva, 2017)
The ‘publish or perish’ attitude favours hasty, low quality, incomplete research. This may be due to structural funding incompetence surrounding scientists that drives down the quality and incentives corruption. A delicate balance between research quality and quality-support compensation has to be very precisely achieved.
The ‘publish or perish’ attitude favours hasty, low quality, incomplete research with the aim of maximizing the number of papers from a single research project (sometimes known as salami slicing).
Grandiosity and hysteria have no place in science. Sensationalism can do real damage and incentivize inflating statistical significance in truly profoundly dangerous ways to all adjacent studies.
Researchers should confine themselves to their findings and resist the temptation to sensationalize their results. Incentive structures for rewarding research, e.g. publication, funding, promotion and tenure, need to pay more attention to the quality and reproducibility of the work produced.
Involvement of methodological experts, multi-site trials, reporting of standards, transparency with conflicts of interests, and appropriate regulatory oversight can solve these issues.
Investigators can learn from studies that cannot be replicated. Adoption of reporting standards will help, as will multi-site trials, involvement of methodological experts, appropriate regulatory oversight and transparency about conflicts of interest. As gatekeepers, journals can offer high quality peer review (which should include proper statistical/methodological review, as appropriate). Prospective trial registration is not enough, full protocols should also be published, and data should be shared.