The advent of abundant Earth observation data enables the development of novel predictive methods for forecasting climate impacts on the state and health of terrestrial ecosystems. Here, we predict the spatial and temporal variations of land surface reflectance and vegetation greenness, measuring the density of green vegetation and active foliage area, conditioned on current and past weather and the local topography. We train two alternative recurrent deep learning models that combine Long Short-Term Memory cells with convolutional layers (ConvLSTM) for forecasting the spatially resolved deviation of surface reflectance across a heterogeneous landscape from a specified initial state. Using data from diverse ecosystems and land cover types across Europe and following a standardized model evaluation framework (EarthNet2021 Challenge), our results indicate increased performance in predicting surface greenness during extreme drought events of the models presented here, compared to currently published benchmarks. This demonstrates how deep learning methods for optical Earth observation time series enable an early-warning of vegetation responses to the impacts of climatic extreme events, such as the drought-related loss of green foliage.