Wouldn't this kind of problem be a perfect match for machine learning? It would have a huge dataset to learn from. Why isn't it happening or what prevents AI tech from forecasting the weather?
It is because there is an understanding, from first principles, of the dynamics that drive weather (e.g. conservation of mass, momentum and energy). The current models are build upon these principles to make predictions, and conform to expectations of how physics operates. The method that these models are based on (finite volume) is efficient and adaptable if modifications need to be made.
Using AI and ML to make predictions about weather will likely not account for the conservation principles and might lead to ridiculous results (in some sense). Creating an accurate AI/ML model of a complex and chaotic system might lead to wrong predictions under extreme circumstances (e.g. predicting the weather >5 days out for an extreme hurricane) or under conditions where some implicit assumption has changed. One can at-least attempt to grapple with these issues when using finite volume. Under AI/ML you just have to hope your model is properly trained.
I could recommend this paper:
Schneider, Tapio, et al. "Earth system modeling 2.0: A blueprint for models that learn from observations and targeted high‐resolution simulations." Geophysical Research Letters 44.24 (2017).
It is, and the research applying ML in this area is starting to ramp up. For example, last year I worked on a project using ML to identify tornado vortex signatures in Doppler weather radar scans. It also turned out that a couple other groups published similar research at the same time. I would say to expect to be much more growth with ML in meteorology hoping it will all eventuality be applied in the field.