I think this is what will happen. That the public internet will become the place you go to seed the data you want to the scrapers and you will use a private internet for everything else. Private sites, private feeds, mesh networks, etc. We're basically going back in time similar to when AOL and friends had their own private networks for their members.
There's a lot of this back-to-basics stuff out there: BBSes, Gopher sites, Gemini protocol, etc. A lot of it is very refreshing in comparison to what the modern web has become.
It's easy to pretend you're human, it's hard to pretend that you have a valid cryptographic signature for Google which attests that your hardware is Google-approved.
Crawling is the price we pay for the web's openness.
Maybe it'll just be cheaper for CDNs or whatever to sell the data they serve directly instead of doing extra steps with scraping