Olive anthracnose, caused by Colletotrichum fungi, and the olive fruit fly Bactrocera olea are, respectively, the most important fungal disease and pest affecting olive fruits worldwide, leading to detrimental effects on the yield and quality of fruits and olive oil. This study focuses on the content of hydroxytyrosol (HYT) and its derivatives (the “olive oil polyphenols” health claim) in olive oils extracted from fruits of ‘Galega Vulgar’ and ‘Cobrançosa’ cultivars, naturally affected by olive anthracnose and olive fly. The olives, with different damage levels, were harvested from organic rainfed orchards, located in the center of Portugal, at four harvest times over three years. Galega oils extracted from olives with a higher anthracnose and olive fly incidence showed no conformity for the extra virgin olive oil (EVOO) and virgin olive oil (VOO) categories, presenting high acidity and negative sensory notes accompanied by the disappearance of oleacein. Conversely, no sensory defects were observed in Cobrançosa oils, regardless of disease and pest incidence levels, and quality criteria were still in accordance with the EVOO category. The total HYT and tyrosol (TYR) content (>5 mg/20 g) allows for the use of the “olive oil polyphenols” health claim on the label of all the analyzed Cobrançosa olive oils.