Welcome to the modern world!
The biggest down side to mirrorless, IMO, is that the sensor is exposed when changing lenses. I don't think I had to clean the sensor on either DSLR in the family, even after years of use. My Z5 and Z7ii both required regular cleaning, usually with the rocket blower, and occasionally with the swabs. It always made me hesitant to change lenses in the field, and I got used to carrying a couple swabs and fluid in my bag for emergencies in the field. Of course, the sensor cover on the 8 and 9 solves this problem. I've had my Z8 for nearly a year and have yet to need to clean the sensor.
Battery life is the other issue. Having said that, I rarely need to swap batteries in the field. I can easily get 700+ images on a battery, and have often gotten as much as 1K+. I never really felt the need for a spare with the D3400. The only need for a spare I might have had was forgetting to charge the battery before an outing.
Neither of these items come close to offsetting the many advantages of mirrorless, IMO. The "exposure preview" you get with the EVF is particularly valuable to me. I also love the quality of the new Z lenses. l have been extremely happy with the new 180-600 z I got for Christmas. It seems to be a definite upgrade from the venerable 200-500 + FTZ, I was using.
The only other real downside is that you may very well start thinking about switching everything over to mirrorless now. Some folks are happy with a hybrid stable, but a lot of us start thinking about what we can upgrade to mirrorless next as soon as we can afford to, after dipping our toes in the mirrorless water.