Classifying events with reliable measures of uncertainty is a key scientific challenge. Nuisance parameters are the standard way by which scientists account for “known unknowns” when constructing a mechanistic model of the underlying process without a tractable likelihood. These parameters are usually not observed at inference time nor of direct interest, but must be taken into account to derive trustworthy conclusions. When the distribution over both nuisance parameters and labels changes between train and test data, we refer to this problem as generalized label shift (GLS). In this setting, direct classification using observed data leads to biased predictions and invalid uncertainty estimates. As a solution, we propose a new method that casts classification as a hypothesis testing problem with nuisance parameters. We estimate the classifier’s receiver operating characteristic (ROC) across the entire nuisance parameter space, and devise cutoffs that are invariant to GLS. Our method effectively endows a pre-trained classifier with domain adaptation capabilities and returns valid prediction sets under GLS while maintaining high power. We demonstrate the performance of our method on two challenging scientific problems in biology and astroparticle physics with data from realistic mechanistic models.