Hello Ben,
Before we even get to the model interpretability or explainability part, I think the philosophy " Let's train a bunch ' o' models and find the best one" is wrong.
IMHO , it just showcases that the data scientist didn't have any intuition whatsoever in selecting the ML algorithm. This is just 'lets throw everything at the wall and see which one sticks' kind of approach. Also selecting models purely based on accuracy metrics like RMSE, AUC etc. is not ideal .