One concept that often gets lost when people become enamored with data is the difference between statistical significance and practical significance. Missing this important difference leads to some common problems. The most frequent one is death by a thousand cuts. We find evidence that all kinds of things might influence a project outcome. However, the size of the effect, while statistically significant isn’t practically significant. We minimize one thing and get a one percent quality improvement, but ignore the less understood thing which causes 50% of the quality issues. Having found something that has an effect, regardless of size, we insist on working on it.
There are always more things we could monitor than we should monitor. Knowing the difference between whether the difference is statistically significant vs. practically significant matters.