Google Photorealistic 3D Tiles and Unreal Engine | Hacker News Reader

Google Photorealistic 3D Tiles and Unreal Engine

nilsbakker.nl

644 pointsstijnbakker120 comments

For years I've been dreaming of a driving/pedestrian sandbox game that used Google Maps data to let me drive recklessly around my home town with accurate models of the buildings and roads.

This demo reminds me of that dream.

Me too. The problem seems to be mainly that the models are still awfully approximate and the textures are horribly blurry. They're fantastic for a bird's-eye view or flight simulators, but not what you want for driving.

I'm hoping that at some point in the next 10 years, Google/Microsoft come up with a high-res version produced mainly by Street View-type data or similar drone-supplied data. I suspect that basic photogrammetry is the easy part -- that the challenging part is dealing with changing sun position, hourly/daily/monthly weather/seasons, and all of the pedestrians and moving/parked vehicles.

Fun fact: Google street view cars record video-- the widely spaced street view nodes are a small fraction of the imagery they have stored. It should be pretty easy to do higher res photogrammetry with the raw data.

AI upscaling with diffusion models will probably solve that problem.

That's what I was thinking. The satellite photos are detailed enough to see lane lines, and combine that with the street level photos will get you the road configuration.

But I think whatever data GPS software uses is already detailed enough to know lane config, it tells you exactly what lane to be in when making turns.

photogrammetry is old. Check out NERF. https://youtu.be/DJ2hcC1orc4

LiDAR-mounted drones are available, although a ground-level thingy can also generate neat point clouds in its vicinity: https://enterprise-insights.dji.com/blog/lidar-equipped-uavs

The old 1989 Dos game called Vette let you drive around San Francisco in 3D. To a young kid living in San Francisco with a 286, this was perfection.

I've been looking for a game that lets me drive around real towns ever since.

True Crime modeled LA 1:1 although the game didn’t do that much with it.

Also, GTA V doesn’t model LA 1:1 but holy fuck it’s practically LA.

This was my first job, 22 years ago. Ended up as something called My World, which then became eegeo, and now WRLD

https://en.wikipedia.org/wiki/Project_MyWorld https://www.wrld3d.com

Didn't really end up going anywhere.

The Getaway was the first game that I remember doing this. As a Brit, being able to hare around my capital city like a hooligan was inexhaustible fun:

https://en.wikipedia.org/wiki/The_Getaway_(video_game)

Playing the new Gran Turismo on the PSVR2 setup makes me think this isn't as far off as it feels. Surprisingly immersive game.

YouTube video from the project:

https://www.youtube.com/watch?v=_cziz3qYT4Q

This is amazing - it will be cool when these models in unity can be turned into real buildings where one can design the interiors as well, and maybe use this in Hitman to play real cities.

what if you could own a build and the receive points on either survival or the other version points?

And you can build your own maps - and you set A and B and you win creds based on winner.

Fuck you Morgan Roberts at BLIZZARD you stole my legos

It's super cool.

Does the 3D tile mao extend down to the street level? I would have loved to see flying along the streets between the buildings.

Might it be the extreme chromatic aberration? I once read an article saying how to achieve 'photorealism' in computer graphics, one should add chromatic aberration, as that's what camera lenses naturally produce. I think some people take it too far though!

No lenses except the cheapest Chinese ones made for CCTV cameras exhibit noticeable chromatic aberrations. It's really annoying it's being pushed in video games so much lately.

Yeah but it looks pretty weird and blobby down there, and a lot of the street data is pretty warped and strange looking. I'm playing around with this for a cycling app, I think I'm going to go with sort of a drone follow camera.

Thanks for the video! Really neat technology, seems like a great way to explore. Something about the FOV makes me ill.

What is the music in that video? My Find Sound app is failing.

Love is Empty - Cindy Wang is what Shazaam on android is telling me.

By the way, Google Earth VR (2016) provides a similar experience but in virtual reality. Seing 3D graphics with depth improves a lot the experience.

You also have some nice features such as the possibility to grab the sky to rotate the sunlight.

This demo looks nice though. The post processing has nice colours, though the fake chromatic aberrations may be a bit high.

To this day this is the best experience I had in VR by far. I spent hours roaming the earth, it's truly amazing. It's sad that Google is massively underutilizing this asset.

What should Google be doing with it?

As a complete non-gamer, the ability to walk around the earth with a VR headset would justify buying one to me.

There are so many cities I've always wanted to walk down the streets of!

As the parent comments mentioned, Google Earth VR already exists and it's awesome (disclaimer: I had a hand in creating it). It's as close as you can get to "walking down the streets" with current data. All the available 3D and Street View data is there.

First off, THANK YOU to you and the rest of the Earth VR team. You’ve brought literal joy to many people through your work.

I brought my Quest Pro and a gaming laptop with me to visit a nursing home a few months back and took some of the residents on guided Earth VR tours of their hometowns/places that they remember fondly. I’ll never forget all the tearful smiles I witnessed that day, and it’s all thanks to you and your team.

Quick question: are these new 3D Tiles the same as what’s in Earth VR, or are they higher resolution/newer imagery? The video demo linked upthread looks incredibly crisp, but I’m wondering if that’s just an artifact of seeing it in 2D (the upclose castle textures on flyby looked a lot more like what I’m used to from Earth VR).

This is a great idea. It would be interesting if Google had historical view of towns so in 50+ years I could see what it looked like when I was a kid.

There is a time slider in Street View today https://blog.google/products/maps/go-back-in-time-with-stree...

I think in Google Earth desktop there is (was?) a time slider for aerial imagery as well.

Always great to hear that people enjoy it :) I believe that this API uses the same up-to-date tiles as Google Earth VR and the 3D view in Google Maps. I saw some complaints on Twitter saying that maybe the last and highest level of detail is not available in the API but I haven't verified that myself. I don't currently work at Google and have no insider information.

Team up with microsoft and asobo.

They seem to be doing fine on their own. I prefer to have some competition.

What's wrong with companies teaming up for specific individual products? In this case, Flight Simulator. You license the technology with friendly fees and collab for the life of a single product. Just like all licenses.

(Also, no, MS/Asobo are not doing fine, their photogrammetry is renowned for looking like a molten hellscape, and the bing maps look pretty terrible compared to even plain gmaps, but it's MS so "it has to use Bing" =D)

I thought it was Blackshark.AI doing that specific part of Flight Simulator. AFAIK they are, or at least were looking for to, also selling that capability to other companies; and they are certainly not “vendor locked” with MS’s data..

but what about capitalism?

What about it? Google charges MS/ASobo a "friendly" license fee, and MS/Asobo get a feature they can use to attract more players. All parties stand to gain concrete dollar amounts, the very backbone of capitalism.

Lovely simple Unreal project. It's amazing how much can be accomplished with Blueprints when all the necessary functionality is at your disposal (although it can be hellish when it's not). I have such a love/hate relationship with UE. Sometimes it will blow me away with how some stuff just works with a few blueprints and other times (especially around physics) it just wants to do stuff its way or the highway.

UE looks cool, it's full of features, but it's also full of warts (especially when you realize how poorly designed and implemented certain parts are, and the huge mess they made with their C++). My major issue with blueprints is that they are always abused and turn into an unmaintainable mess in the blink of an eye.

The company I work at has a massive UE-base codebase and we've been working hard for months to purge as much blueprint logic from our it as we can because it's simply impossible to maintain them. Epic is pushing them hard but I just can't see them as more than a way for amateurs to mock up stuff.

Yeah, after my initial honeymoon I'm souring on blueprints fast. Working with them on a team using version control must be a bit of a nightmare. Not having a way to export them into a human usable formal (say a good quality Lua style DSL) means excluding all kinds of tooling that could be built by other parties for handling text based code.

What UE sorely lacks is a good scripting language. It's clear to me that Tim Sweeney wanted to offer a C#-like experience, but using C++ - IMHO UE failed in any possible way to accomplish that - Epic made a sawed-off shotgun in place of a precision weapon. There are just a gazillion ways to introduce concurrency and memory issues in UE, but most of the time it's "fine" because for a videogame some aspects (like proper memory management, safety, ...) are less important than CPU cycles. A game often behaves as if it's the only application running on a machine, and if it gobbles too much RAM the usual answer the developer gives its users is, "buy more". Thankfully this has been changing in the last few years due to mobile gaming and freemium, which forces companies to target not top-notch machines but cheap laptops.

As someone who worked at Realtime Worlds on the MyWorld project, this is very poignant.

https://en.wikipedia.org/wiki/Realtime_Worlds

https://en.wikipedia.org/wiki/Project_MyWorld

Hi Sam,

I worked at Real Time Worlds on that too. Probably before it was called My World, then I think we were calling it Real Driving or something then.

Small world!

Cheers James

:wave emoji:

Interesting!

Do you have some more details you could share, like screenshots or trailers/videos? Sadly the video link to the trailer on YouTube is not working anymore.

This is one of our demos from 2008, which looks very similar (though we aren't using Google Maps data, we're mostly using OS map data). It's also long before an art pass, so the textures, trees, building, road etc are just placeholder.

https://www.youtube.com/watch?v=pfJwX_HnLpU

This is Mike (who did the first lemmings)'s test to se how many lemming-like things you could render in the world. He, myself, and Bill (GTA physics) went on to build a fun game out of this where you could run over them in tanks and blow them up.

https://www.youtube.com/watch?v=2O6WS1VolKA

Blowing them up test:

https://www.youtube.com/watch?v=SJHRdVaUtDs

Very cool! Here's a tutorial from Sebastian Lague on implementing something similar in Unity, which may have served as inspiration for this project:

https://youtu.be/sLqXFF8mlEU

I think the inspiration was the release of a new Google API last week along with support for it via Cesium's Unity and Unreal plugins.

Yeah. Sebastian Lague's game videos are great because of the way he walks through his steps & methodology, but wrt his geographic implementations it's not how you would approach this problem for this level of granularity.

I also tried my hand on integrating the Google Earth Tiles with Cesium.js for GPS Track visualization: https://cubetrek.com/static/highresmode.html

It basically allows you to replay your hikes, running or biking adventures in 3D.

I wonder how much Google's gonna charge once it's out of experimental mode.

Looks like you're already out of credits. I'm getting a "loading assets" error.

Thanks for the heads up

The Google giveth, and the Google taketh away.

Has been HN'd I think!

First "Resource Limit Is Reached", now "Insufficient Storage"...

"file not found" for me, just now.

Looks insanely cool! I also made a simple web-based demo with Deck.GL + shadow lighting + post-processing effects here: https://cheeaun.github.io/photorealistic-3d-deckgl/

There's a limit of 250K "3D Tiles renderer requests per day" so it'll stop serving tiles if exceeded.

For those wondering how it looks like, I have a thread here with lots of screenshots https://twitter.com/cheeaun/status/1657727482492235777

Thanks for the info. This is per API key? How many renderer requests does a user use per minute? I'm just trying to figure out if this is something that can be used in an app at scale.

Limit applies per project if I’m not mistaken. More info here https://developers.google.com/maps/documentation/tile/usage_...

I’m not really sure if there’s a limit per user. It’s still beta and I can’t or don’t dare to increase the quota for now.

    "message": "Quota exceeded for quota metric '3D Tiles root requests' and limit '3D Tiles root requests per day' of service 'tile.googleapis.com' for consumer 'project_number:174664172126'.",

Wow that’s really fast

Okay, turns out it's the "3D Tiles root requests per day" that's exceeded. Max 300.

I remember working in this field about 15-20 years ago. And separately, in my spare time, playing video games like Crysis. It was quite apparent even then, that video games were generally light years ahead at visualisation.

Back then, one could - using professional, multi-thousand-dollar software, and expensive services like land surveyors - painstakingly, manually build up a small-scale 3D scene encompassing an individual owners land. The result would look average/generic, and the rendering performance and navigation limitations would be abysmal.

On that same PC, one could then load up Crysis, marvel at the vastly superior output, navigability, and point-of-view realism - covering massive land areas - and think 'Holy Shiz, if I could just get my data into this, it'd be a revelation, industry-changing'.

It's great to see how things are progressing.

This shows how insanely optimized game engines are, and game development in general.

All enterprise software is optimised in man/hours, as in the maximum amount of features for some given hours of work. Enterprise customers will then invest in the extra hardware, because "developers are more expensive than computers".

Games can't afford that, they need to run in many different kinds of shitty and good hardware, so it is optimised over and over, by very dedicated and underpaid developers. And they mostly code in C++. But games are sold by the millions, so it's a different market.

It could be a great introduction to the metaverse. Imagine adding the interior spaces by making the doors and windows openable. Then allow for more and more interactions. A lot could be done by algorithms (image recognition, etc.), but I think a lot of users would be interested in "building this new world". Adding procedurally generated spaces through portals would be a great next step. We are not far away from being able to "create a castle described by X in Y mixing painting style of Z and sculpture style of Ź". Maybe transforming between these styles and others in time.

I remember toying with Google Earth a year or so back, thinking it'd be amazing if the actual Google Maps API would give access to that level of granularity. Was sort of baffled as to why they wouldn't. Definitely gonna give this a shot.

P.S. For those who haven't visited Google Earth (instead of Google Maps) in a long time, it's alive and well, and still pretty fun to go on a mini excursion with: https://earth.google.com/web/

Just some context - this is not me, but former colleague of mine :-) (even though we share the same surname)

I really appreciate this disclaimer. :)

You can do this stuff with drones and open source tools. I used a cheap 100 dollar intel drone to take photos of my house and processed the pictures using (alice vision?) software that built the 3D mesh. My idea was to package something together for a home remodeling system that would let people measure and look at their houses with a 3D model but it just so happens that I live in the no-drone radius within Washington DC. Cesium also has a pretty nice 3D tiling photogrammetry service.

How long would it take the drone to reach Utrecht?!

I think the beauty of this is the fact that it's now easy to do this for anywhere that has coverage from Google tiles.

I'd love to see a side by side of achieving this with Unreal Vs open source tools though, would give a good indication of the gap between them (if any).

On a side note.. regulation came in quite quick for the no-drone areas in a lot of places, scuppered our farmer unmanned livestock checking startup idea!

There's no way the drone would reach without breaking the law (or with you walking behind your drone the entire time).

Drone pictures are nice but most people aren't allowed to fly them in many places and even if they are, many densely packed cities have restrictions of their own.

These days you could probably get away with a handheld video and NeRF software if you wanted to add a missing place to this thing.

Very impressive work from OP! And cool suggestion from you.

I'd imagine there would be a fair amount of hallucination with ChatGPT generating the Lat/Long pairs (especially for smaller cities/towns)?

Awesome project by Nils Bakker. Linking 3D tiles to Unreal Engine and ChatGPT.

Looks like it got the ol’ HN hug

I’m sure if we all just stay on the site and hammer refresh it will load eventually…

Hehe, by you, you mean? ;)

This is a really fun way to combine something that is being discussed to death with something that is visually beautiful. I'm fairly disinterested in most ChatGPT usages at the moment, but this is a really good way to show its utility. Particularly because it doesn't place _too_ much emphasis on the factual accuracy of the responses ;)

What, you don't think FC Barcelona is the best football club in Europe? ;-)

All I can say is WOW WOW WOW

Very nice!

Throw in some Carmen Sandiego stuff and i'd buy this for my kids

Incredible! The video music is also amazing. (But the slight chromatic aberration was distracting.)

I cannot stand the music, or music in any video of this type. I don't need or want its loaded suggestions. I would love to be able to hear even simulated wind whistling, and sounds from the scenes below.

You're free to mute the video, that's what mute is for.

As for sounds from the scenes below, that's a lot of work. Where do you think people should source that from?

Oh, I know all about mute.

30-ish years ago, DVD players supported alternate soundtracks so people could offer a soundtrack with music and without. I don't think I've ever seen that basic affordance on a web video. That would probably lead to more diegetic sound, and the tools to simulate it.

crazy!! Video here https://www.youtube.com/watch?v=_cziz3qYT4Q&ab_channel=NilsB...

source available somewhere?

It would be really cool if there was more shared than a blog post with a video. What about either a tutorial, source code, or a link or download to use this ourselves?

The post itself has a pretty decent, high-level walkthrough.

This is really superb! Fantastic execution and excellent combination of tools.

Is there a way that we can play with this ourselves, or is it just the demo video?

Either way, it's really great!

If you like this, you'll probably love the current version of Microsoft Flight Simulator.

I just keeping thinking about Terravision and "The Billion Dollar Code" from Netflix while watching the video.

This is amazing! I was mesmerised and watched the whole video hoping it wouldn’t end.

How do we try it for real?

Very cool!

AI should to context fill more detailed textures and fix some geometry issues with the buildings.

I think you're looking for this thread. https://news.ycombinator.com/item?id=36000631

I've moved it thither on the assumption that you're correct.

Thanks for watching out for a fellow user!

404 for me

feels like magic. Those small steps like take me to the biggest city and it magically goes to Barcelona is something from the future.

That's an amazing project *** Bravo!

Could be a fun variation on Geoguessr

Wow that was nauseating, Google really has terrible maps data. Garbage in, garbage out I suppose.

Who has better data, may I ask?

I'd love to know because I've been looking for several years

HOW DO I PLAY?? (serious)

Does anyone know how big the files tend to be for a city? I'd love to make this into a mobile game so everyone can explore the world when bored.

Unfortunately you're not allowed to ship these files with your game. They have to stream directly from Google's servers for each user (caching is allowed as long as you follow standard caching rules).

A chunk of Manhatten I extracted is about 30mb glTF, but 7mb with all compression options in gltf-Transform (https://twitter.com/Omar4ur/status/1657361031763578881). But this is not the highest resolution available.

See terms of use: https://developers.google.com/maps/documentation/tile/polici...

Microsoft Flight Simulator. (serious)

It's a bit the same story as GPT:

Early: Google does primitive tech (Google Earth)

Later: Microsoft does amazing implementation (Flight Simulator / Bing)

Finally: Google tries to copy but fails (Google Maps)

Huh? Google Maps predates Google Earth and neither really relate to MS Flight SIM.

The Google Maps version that we are talking in this thread; aka Google Maps "Photorealistic 3D"

But that has been around for an age as well - it just wasn't available via an API.

Bing came out with 3D buildings after Google did and the coverage was a small fraction of that Google had and hasn't increased very much in the intervening years.

(I hope apps will offer both as data sources and maybe select the one with the best fidelity for a given region)

Very impressive!

For the OP:

Warning: Potential Security Risk Ahead

Firefox detected a potential security threat and did not continue to nilsbakker.nl. If you visit this site, attackers could try to steal information like your passwords, emails, or credit card details.

And:

$ curl https://nilsbakker.nl/portfolio/3d-tiles/ curl: (60) SSL certificate problem: self-signed certificate More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. (hf-course-3.9.13)

My guess is that it's the proxy responding since the server is going down. I got the same error and when I continued I saw "Your domain name is parked.

Hey owner of nilsbakker.nl Your domain name is not linked to your web hosting package, reseller package or virtual/dedicated server. If you need support in connecting your products, you can contact your provider."

No problems here, it displays as a "Let's encrypt" certificate.

These are the fields on the certificate that's being presented to me:

emailAddress = root@dns-redirect001.axc.nl, CN = dns-redirect001.axc.nl, OU = SomeOrganizationalUnit, O = SomeOrganization, L = SomeCity, ST = SomeState, C = --

A self-signed certificate that expired on 6/11/19, 4:35:46 PM GMT+2

It's probably been taken down.

It works for me (again) but I remember that server domain from my old Versio shared hosting account.

They used to be the typical "dirt cheap, low spec, PHP 5.4" hoster of choice for students and people looking to save a buck, but they've since raised their prices significantly (and I don't think they've raised their quality of service much).

Would not recommend. I wouldn't recommend shared hosting in the first place, but if you do, it's worth looking around for better hosts. There's a good chance you'll be better off with Github pages for hosting websites that you expect any visitors to see.

Github pages is excellent and should be the dev’s go to of choice IMO.

I'm getting an error as well, although not a self-signed error. I'm getting a mismatched domain error.

Probably just depends what order your TLS implementation is deciding to complain about the issues: it's not only self-signed but has no DNS SAN and the CN also mismatches.

It's just killed Flight Simulator

Well, except for; y’know…all the ‘Flight Simulation’ parts, uh, sure…?

It's not so impressive when you find out it's just a Unreal engine plugin.

This is a mashup.

Reply on news.ycombinator.com