UKH

Have Rockfax, BMC or any smart individuals looked into it? Could it be a thing? 

I suspect the technology isn't really there yet. I'm very much not an expert but as far as I can tell, superimposing imagery/information onto the real world requires one or both of:

• An accurate idea of where the viewer is standing and what direction they're looking
• Features the computer can recognise in the camera view

AR apps that display sky maps or highlight particular hills/mountains can rely on GPS location because they're highlighting distant objects so small errors in location don't make much difference to the view. I don't think GPS would be accurate enough for that approach to translate to crags.

I also don't think computers are capable of recognising crag features in the camera view yet, especially bearing in mind the vast range of lighting and viewing angles it would be necessary to cope with. If this kind of computer vision could be pulled off any time in the near future, I think it would be in the realm of Google or Amazon-level technology companies rather than, with no disrespect intended, Rockfax or the BMC.

Why don’t you just go to the crag and read, or look at the pictures, in the f*cking guidebook.

I do realise I’m a dinosaur but augmented reality is not reality.

It wasn't all that long ago that guidebooks had descriptions rather than photo topos and I'm sure there were people who made similar remarks then.

If you lived with Scottish weather you might think augmented reality climbing sounded pretty appealing

I agree, I don't think the tech is good enough yet, an error of a couple of degrees in the direction or a couple of metres in the location could put all the routes in the wrong place.

Something I think could work, and would be good would be an augmented reality tool for viewing walking/running routes through the hills.  To be able to plot a route, and have the phone superimpose a line on the camera screen showing which paths you were taking, which routes you were taking up each hill would be great.  I suspect that would be possible with todays tech but the market probably isn't big enough for it to be made.

Did I say at any point I was in favour or apposed to the idea? Well then, it’d be great if you stopped jumping to conclusions.

It may not be reality and yet here we all are on the internet.

Maybe go practice what you preach, make use some use of this nice weather and look at some guidebooks. 

Many thanks, I was hoping for a answer from someone like yourself who understands the technology better than I do.

Just to clarify to everyone, the idea for the post was something that pasted my mind and an obvious connection between technologies we have and some use.

I’m still taking photos of my guidebooks, even Rockfax ones despite the App, so hopefully that’ll keep all the “back in my day” keyboard warriors hopefully somewhat happier. 

Me in 2027 - "these Rockspex are amazing, I just look at the route and I can see the grade, the FA details, and I can even blink on the 'cheeky beta' option and get full details of the moves and gear. The GPS guides me to the descent and I can log it with full notes, and upload video of me climbing it straight to UKCtube. Oooooh look, they've even put in a virtual coiled turd right on the first stance....."

That turd is because you 'did a Bear' and logged an onsight but used a tr.

You're a wuss for using a guidebook. Just go out and find your own routes! 

Kids these days ... 

Yes. Fatmap, Whymper, OSMaps etc all have AR features. I think you need to ask what you actually want from such an application.

Personally, I think the advantage is where you can't physically position yourself at the same angle from which a particular photo topo was captured. I would find a system that could reconstruct the line of a particular climb from any given angle, using, for example, Photogrammetry and Sensor Fusion, useful on, say, sea cliffs.

Of course you can argue that such technology removes some of the sense of adventure, but no one is under any obligation to use it.

Well it depends what level of AR you're expecting. Mm precision scans of things such as crags are perfectly possible using, say, drone mounted LIDAR but the expense would be prohibitive, and the recreational climbing industry does not quite have the economic weight of, say, oil and gas exploration.

However, coarse-grained 3D reconstruction of crags is possible using photogrammetry (Microsoft has had stuff like this available for years on Photosynth) - and would be all you need to reproduce a photo topo from any angle - something that I think would be genuinely useful.

Putting the case for the dinosaurs:

- trying to use "a system that could reconstruct the line of a particular climb from any given angle, using, for example, Photogrammetry and Sensor Fusion" would be a rilly rilly wild adventure. 

Could you also argue using a guidebook, ever removes some sense of adventure? 

Have a read about microsoft’s Photosynth - the image task isn’t intractable.  It’s something being solved for self driving cars as well.

You and Wintertree seem to be talking about something different from what I understood by AR. A topo that you can adjust the angle of would certainly be useful/interesting and I agree that it's technologically feasible but I'm not sure I would count it as augmented reality.

My interpretation of OP's enquiry was that they wanted to be able to point their smartphone's camera at the crag and have it overlay the guide information onto the live camera view (could obviously be even slicker with some kind of goggles). A detailed 3D image of the crag would be a helpful step towards that goal but not sufficient on its own, I don't think.

The way by which photosynth aligns a new photo to its 3D model of an object is how such an AR app would micro-navigate from the camera view, allowing it to overlay routes and annotations on that view.  The app has some degree of GNSS and compass data available too.  Edit:  I missed planetmarshall’s post - I agree with them.  Although it’s increasingly possible to assembly the required 3D models from stereo imaging (even with just one moving mono camera).  There’s a couple of firms who do this as a service using a UAV.  It may not be prohibitively expensive.

We've looked at 3D topos but tbh the tech isn't ready for that yet never mind AR topos.

There's some cool 3D modals linked on this thread: https://www.ukhillwalking.com/forums/rock_talk/the_future_of_guidebooks-650821...

Good collection here too: https://sketchfab.com/ClimbAssist/collections/big-bend

Sorry, I missed the point you were making with Photosynth. It's certainly fascinating tech, I remember being pretty blown away by it when they first started demo-ing it. I will stand by my point that if it's doable at all with current technology, it's on the level of sophistication that Google/Microsoft/Amazon operate at rather than vastly smaller and rather less tech-focussed organisations like Rockfax and the BMC.

That is really cool.

That would require the ability to view the crag from any PoV, and then overlay the correct, 3D route onto the view. It needs to be able to determine where the viewer is, where the crag is, and how to project the route onto the view of the crag, which needs a 3D understanding of what the camera is viewing, and where the route goes, so it can do the projection to the 2D viewport.

I think existing phone-based 3D photogrammetry is easily good enough to do this; it's good enough to extract 3D models from a number of photos taken around an object.

I imagine this could be done without any new technology using convolutional neural networks. We would just need thousands of different photos of the same climb from different angles, different lighting conditions and from different cameras, all with a marked up version showing the route. The neural network could then be trained to add the route line onto the photo without it.

Or it can be done without any neural networks by using a standard algorithm like SIFT for image registration and using bog standard optimisation/minimisation methods and affine transforms.  

Signed,

An old school grump.

CNNs would not really be the appropriate solution to this problem, which I think is probably best solved using 'traditional' (but still fairly sophisticated) photogrammetry techniques.

What you *can* do with CNNs, is extrapolate what a crag would look like from a certain angle given *incomplete* information. It's a really interesting area of research which is also used, for example, to extrapolate what a person looks like from the front given only information about what they look like from other angles.

I'm not sure why you say it would be inappropriate. I'm pretty sure I could do it. I'd essentially use a GAN to generate the image with topo from an image without, at the same time training a discriminator as training the generative CNN.

As I implied though, and as is usually the case with deep learning, the limiting factor is the amount of data one has to train with. I reckon I could do it from UKC photos of one of the super popular routes, but they'd still need to be annotated by hand

Well you've pretty much answered that question yourself. Because you need a lot of annotated training data which is not available, nor is it likely that a trained CNN will necessarily generalize to other problems (crags) without retraining. Why employ CNNs when this is already a well studied, and largely solved, problem? I don't see what you'd be gaining.

You can tell the press or customers that your technology/product uses “AI”.  

I just keep a little cardboard box with “AI” written on it sat atop my computer.  That way I can tell people “oh yes there’s some AI in there” whilst sticking with nice, hard, quantitative methods that get the job done in a predictable way and give me reliable estimates of their accuracy...

I reckon I can climb E11 in VR.