This is an increasingly familiar question in the face of overwhelming social and political challenges. This question and its variations—“but is that the most important issue?” and “is that worth our efforts?”—suggest a commonsense philosophy: Concentrate on the biggest problems, and don’t sweat the small stuff. A three-percent gender pay gap might matter in theory, but activists must make hard choices about which battles to fight. And when resources are limited, comparatively smaller harms don’t demand immediate attention. However, this popular philosophy is false—and dangerous. Ignoring small harms is a moral mistake, and neglecting small harms can have surprisingly large consequences. From environmental harm to the gender pay gap to racial injustice, large problems caused by cumulative small harms are immunized when we judge that none of the single contributing harms merits attention.

Consider an example of one of these “small” harms. An employee’s boss calls him into her office. “Your drug test has returned a positive result,” she says. The employee has never taken drugs and knows that the result is a false positive. Moreover, the questionable drug test identifies a greater percentage of black employees than white ones. Black employees test positive for cocaine about one-percent more often.

This story reflects the issue in a recent court case involving an employer’s hair follicle drug test that could fail to distinguish between (i) drug ingestion and (ii) environmental exposure to drugs. Employees could have been deemed “drug users” for simply being at a party or concert where their hair was exposed to others’ drug use.

A group of employees sued, arguing that the hair test had a racial-discriminatory impact. Moreover, a more accurate drug test would not have this same racial impact. In this case the employees’ action was successful, but there is a widespread legal debate over the merit of such “small” harms. Some U.S. courts consider similar harms “insignificant” for the purposes of antidiscrimination law: a one-percent harm is too small to be worth addressing.

Similar reasoning surfaces in other contexts. The gender pay gap is often dismissed for its small size. A gender pay gap of twenty percent would be worth prioritizing, some argue, but what’s the big deal with a mere three percent pay gap?

That reasoning embodies a classic “mistake in moral mathematics.” The late philosopher Derek Parfit articulated five of these errors in moral reasoning. One of Parfit’s moral mistakes is ignoring small effects. The harm of a discriminatory drug test may be small. But, as Parfit argued, smallness is not a good reason to entirely disregard an effect or outcome.

To demonstrate the point Parfit invited us to imagine a large number of thirsty persons lying in a desert and the same number of altruists with pints of water several miles away near a water truck. Each altruist who withheld his pint from the water truck would only marginally decrease the amount drunk by each thirsty person. But if such small effects are irrelevant, then there is no objection to every altruist withholding his water. This, Parfit thought, is clearly the wrong conclusion. So small effects are not irrelevant.

But perhaps, one might argue, some of these social problems don’t really involve small harms. Instead they involve small chances. For some employee the job disqualification is a large harm, yet the drug test had only a small chance of unfairly affecting any particular employee. But to dismiss outcomes for their improbability is another of Parfit’s mistakes. This is easily seen: The danger of playing Russian roulette is improbable, but it’s not irrelevant.

Neither smallness nor improbability gives a reason to flatly ignore harms. But are small harms worth much of our (inevitably limited) attention? Surprisingly, yes. Small harms at multiple points in an individual’s life or across a group can result in large cumulative effects. Our social structure includes webs of smaller harms that reinforce each other. When each of numerous small effects is deemed insignificant and not worth addressing, the cumulative stasis has a large total impact.

It’s obvious how some mutually reinforcing small harms can have a large cumulative effect. Consider tiny environmental effects. Some individual actions damage the environment, while others are neutral or even provide benefits. If these effects were randomly distributed, we might have less concern. As it happens, day-to-day activity produces more environment-damaging effects. The significant cumulative impact of these smaller harms is clear.

A similar structure operates more subtly in other domains. Some employment policies seem to harm those who are disabled, female, or of a certain race, while other policies appear neutral or even to benefit those groups. However, often things don’t break even. Instead, these small effects point in the same direction: harm. Certain groups often face broader and deeper networks of cumulative harms. If these harms were distrusted randomly, it’s more tempting to permit treating them as an irrelevant small drop in the bucket. But across some disadvantaged groups, such harm is often one drop from a poisonous tap. The harms are individually small but together toxic.

Focus on only the “biggest” issues can easily overshadow these concerns. To be sure, progress is possible on all fronts—big versus small is often a false dichotomy. But looking behind the big priorities reveals a striking shared experience of cumulative small harms. Even those not personally familiar with such challenges could quickly enter a class exposing them to new sets of small harms. Scientists report that up to 50% of older Americans will develop Alzheimer’s; one in four persons will at some point become disabled; half of Americans would not be able to cover an unexpected expense; and less than a quarter of Americans have a three-month emergency fund. Any one of some very common realities—Alzheimer’s or other disease, accident, layoff—can place you into a new category, be it sick, disabled, or unemployed. In addition to introducing a single big harm, this social categorization also unleashes a new reality of smaller ones.

The severity of these harms is importantly varied among the members of groups and among different groups themselves. But this shared experience provides a foundation for recognizing the significance of small and cumulative harms. No small concern is irrelevant simply because there are “bigger” priorities, and it is crucial to address some of these now, given their cumulative effects.

An antidote of social improvement begins with the refusal to flatly dismiss anyone’s real concern as too small to address. And counterintuitively, it requires devoting some attention to seemingly small problems. To the question “but does that really matter?”, we should reply with a new philosophy: If something matters, it really matters.