Google Asking non-stop to turn on GPS 'High Accuracy'

I've recently started to get a very annoying message every time I turn GPS ONLY on.
I am using HTC M9+ with Android 6.0, I had this phone 2+ years now and something started to happen lately.
I use my GPS a lot but since I tend to use the battery often, I use GPS ONLY (without using internet connection or anything else only GPS) because it uses less battery. Around a month ago, I started receiving this message (It's in Hebrew but I will translate most of it):
Do you want to improve your location?
Let other applications find your location in a quick way using the location of Google. Google may collect data from time to time and use the data anonymously to improve your location...

This service works when the GPS is working and set on 'High Accuracy' or 'Battery Saver' mode.

And then you have the option to Agree or Disagree.

Now the problem is this message shows between couple of seconds and a couple of minutes after turning the GPS only mode. This is really annoying. Even after I agree it pops out after turning to GPS only mode. And after I Disagree, it just spams itself even worse if I just ignore it. I hope there is a solution to this problem because it is very annoying. I thought about disabling Google's permission to my location but I don't know if it's a good idea or not.

I already posted this on AndroidForums, HTCForums and on AndroidCentral but to no avail. Just to note, my phone is not locked to a carrier or has gotten updates from it. I also didn't update anything, this is just started to happen randomly.