We present a broad class of semi-parametric models for time series of random sums of positive variables. Our methodology allows the number of terms inside the sum to be time-varying and is therefore well suited to many examples encountered in the natural sciences. We study the stability properties of the models and provide a valid statistical inference procedure to estimate the model parameters. It is shown that the proposed quasi-maximum likelihood estimator is consistent and asymptotically Gaussian distributed. This work is complemented by simulation results and applied to time series representing growth rates of white spruce (Picea glauca) trees from a few dozen sites in Québec (Canada). This time series spans 41 years, including one major spruce budworm (Choristoneura fumiferana) outbreak between 1968 and 1991. We found significant growth reductions related to budworm-induced defoliation up to two years post-outbreak. Our results also revealed the positive effects of maximum summer temperature, precipitation, and the climate moisture index on white spruce growth. We also identified the negative effects of the climate moisture index in the spring and the maximum temperature of the previous summer. However, the model’s performance on this data set was not improved when the interactions between climate and defoliation on growth were considered. This study represents a major advance in our understanding of budworm–climate–tree interactions and provides a useful tool to project the combined effects of climate and insect defoliation on tree growth in a context of greater frequency and severity of outbreaks coupled with the anticipated increases in temperature.