People who have been trained to climb have remarkable ability. Here's Jon Hojer doing something called "front lever", suspending his entire body from the tips of his little fingers, from a training device called a Beast Maker 2000.
If someone one like me can climb a building you'll need to make changes to the building. If someone trained in climbing can get up the building you might need to look at how easy it is for them to do and see if changes can be made, but at some point you're spending ever increasing amounts of money for smaller returns and you need to switch focus to building therapeutic relationships.
Importantly, when I look at a buidling I do not think "Climbing that would be fun", but people trained in climbing do think "I wonder how I'd get up that?". Modifying the building to make it less easy to climb makes it scarier for people like me, but more of a challenge for people like them. You increase the appeal of climbing the building, while simultaneously making it more dangerous to do so.
That image comes from a video which is here: Jon Hojer