I recently sat down with two Cemetech administrators: elfprince13, or Thomas Dickerson, and our founder KermMartian, or Christopher Mitchell. Over the past year, they've been busy working together on their startup, Geopipe. Geopipe automatically builds detailed virtual models of the real world, and licensing these immersive models of the world to architects, urban designers, special effects artists, video game designers, and many others. Geopipe leverages proprietary algorithms to automatically generate immersive models of the world from existing sensor data. They take 2D and 3D data like satellite photos, maps, laser scans, terrain contours, and more, and use machine learning to analyze and understand the data. Their algorithms then reconstruct models at configurable levels of detail to meet the needs of a wide variety of customers. Today, they enter the next step in their quest, a 12-week program called Techstars.

Through this interview we'll see how this idea arose as a challenge that they each tackled in separate environments; Thomas in Freebuild and Christopher in SparseWorld. We'll touch on how the similarities of FreeBuild and SparseWorld brought the two of them together and how the ideas behind each of those projects helped spawn Geopipe.

Here's a video that introduces Geopipe and shows Christopher and Thomas onstage presenting their company to investors.


I've summarized our 90-minute conversation below.


Alex: Hey guys, thanks for meeting with me this evening
Christopher & Thomas: Thanks for putting this together.

Alex: Christopher, how long has it been since your obtained your PhD? And in what field?
Christopher: I successfully defended by PhD in Distributed Computing in 2015. Prior to that I got my Bachelor's and Master's degree in Electrical Engineering, and a Master's degree in Computer Science.

Alex: What influenced you to pursue this direction of academia?
Christopher: At a very early age I got into electronics, when I first went to the NYC Transit Museum around the age of 5. They had this workshop with light bulbs and batteries & compasses and magnet wire and I got to learn a little but about circuits. I pursued electronics as a fun hobby throughout my childhood, had many electronics kits, and took apart just about everything to figure out how it worked. I also started to learn programming, in school and on my own. When I went to college I definitely wanted to pursue Electrical Engineering rather than just computer science; I felt electrical engineering was something that I hadn’t had much formal introduction into despite it being closely related to computer programming. Cooper Union offered three EE tracks: signal processing, materials, and computer engineering. I took the signal processing track since I did so much programming on my own, although I also took advantage of computer science courses in operating systems and compiler design and so on while I was there. When I went on to my Ph.D. program, though, I found that the electrical engineering Ph.D. programs were way way too laser focused for me; I wanted to have a field where I could explore projects within a broader range. That was a great experience for me, it allowed me to learn a little bit more about what exactly in computer science particularly excited me and to build up my expertise in those areas, in this case especially distributed systems.

Alex: How do you balance Geopipe with your hobbies such as maintaining the hardware and hacking and programming community Cemetech?
Christopher: Haha, well it’s gotten worse and worse over the years. So while I was doing my Ph.D. program and while I was in college things worked slightly better. I had plenty of time for the community, I was writing a lot of calculator programs, and also doing my academic work as well. I want to say that unfortunately Minecraft impinged on that a little bit as I started my Ph.D. program but I think throughout my Ph.D. program I still did plenty of calculator projects. You know, Doors CSE came out, Tetric A was also during that period, I finally got CALCNet and globalCALCnet working properly around the same time. Starting Geopipe plus various things in life, as life goes on - you know - you have more and more draws on your time, have definitely made it much more challenging to be part of the calculator community these days. I mean I’m certainly still involved, I think more in an advisory role today; guiding Cemetech and our team: our moderator PT_ has been a particularly enthusiastic driver of activity there these days, our great contest administrator jonbush, and quite a lot of other people have taken the reigns from some of the things that I tended to sink all my time into. It’s nice to see the activity has remained even though I don’t get a post count anywhere near what I used to make.

Geopipe is one of those inevitable time sinks, which is a good thing but there’s only so many hours in the day. For me, it’s definitely been an effort of triaging exactly where I’m spending my time and unfortunately beyond advising Cemetech, poking at a few little projects, going to the T3 conference and, some other random bits and pieces around the calculator area that I’ve been working on it has definitely had to fall a little bit by the wayside unfortunately.


Alex: What influenced you to pursue this direction of academia Thomas?
Thomas: My dad is a computer science professor at Middlebury College, and started teaching me computer science skills pretty early. I gave a demonstration on counting in binary to his CS 101 class when I was 7. And at some point I realized I really enjoyed the research process.

Alex: What are some hobbies that you have Thomas? Have they been affected?
Thomas: Rock climbing and Ultimate Frisbee. That’s basically where my non-acedemic, non-Geopipe time goes, as well as Heroes of the Storm.
Christopher: At one point I was expecting you to say LEGO but it seems you haven’t had much time for that lately.
Thomas: No, I actually build LEGO also. I’ve got a set upstairs to build. But, I don’t have many consecutive hours to sit down and build this [points to set behind him]. Haha. It’s the Milano from the second Guardians of the Galaxy with Baby Groot.
Alex: Ohhh, nice!
Thomas: It came with also a mini version of it. The main reason I bought it was because there was a “if you made a purchase over $30 you get a free mini Milano”. So I built that so I’ve got a full size one to build at some point. It’s next to Christopher’s air mattress [that he uses when he visits me] [points again].

Alex: What are some of your hobbies outside of Geopipe and the calculator community, Christopher? And again, have any of them been affected by your work and education?
Christopher: If you had asked me a week ago I might have given you a different answer. [sighs] Let’s see. Certainly video gaming (like Heroes of the Storm with Thomas, Train Simulator, and I play the occasional bit of Minecraft that’s also gotten triaged a little bit to the wayside with Geopipe), I also don’t read quite as much as I used to. Where else does my time go? When I need a bit of fresh air - or a little bit of perspective - taking a nice walk regardless of where I am.

Alex: Has managing and running Cemetech prepared you for managing and running Geopipe?
Christopher: I like to think that it certainly taught me some useful lessons. Being able to manage a lot of different projects with a lot of different input coming from different people, y'know, many of whom give you totally conflicting advice on what they wanted from a product or a tool on Cemetech. Having worked with an awesome staff of people who can take up different pieces when I needed them to; learning a little bit about delegation, that’s certainly something I struggled with because I think it’s not going to get done right if I let other people do parts of it. I think even things like marketing, like as you know we (both Cemetech's projects and my own personal projects) have been on Hackaday, Reddit a number of times, Ars Technica, and republished elsewhere on the internet. I think learning a little more about pitching products and projects in such a way that people get excited about them as well is something that Cemetech taught me in a pretty useful way.

Alex: Thomas, similar question for you: Though you’ve more or less taken a step back from your Administrator position at Cemetech - we miss you! - how has your prior experience helped you run and manage Cemetech?
Thomas: I think the most relevant prior experience outside of a research context was in high school, I started a nonprofit group that did a lot of education around biomass, heating and, local energy economy. But actually before doing that we went through the process of becoming a 501(c)3 and did a lot of work promoting ourselves to state legislators as well as getting tied into the local community; going to a lot of meetings and stuff like that. That sort of communication and organization definitely is similar to running a for-profit business. There’s a lot of ways that it’s different but there’s a lot of similarities as well.
Christopher: We’re both really great at volunteering our time to help other people hopefully, not to pat ourselves on the back too much, but you know that Cemetech is a fully volunteer effort. Our trips to T3 are mostly a labor of love. My books (1, 2) were something I’m extremely proud of but also I did them because I wanted to do it, not because I thought I was going to make a lot of money off it.

Alex: So I had originally thought you guys had two vastly different educational backgrounds. Cause I remember Thomas going on about making a room kinda cold by covering up the windows or somewhere, I forget where it was at --
Thomas: Yeah, that was the summer I was working at the Princeton Physics Lab. It was a room full of physicists and engineers and we were like “Our rooms really hot, it’s acting like a green house maybe we should just raise the albedo.” So we just tin foiled everything, which lowered the temperature after a little while.

Alex: So, continuing with that point though, how do you guys put your two degrees together to work on Geopipe. Getting a head start on the next question, how did Geopipe become an idea between you two?
Thomas: I would say that Christopher has a stronger engineering and building-stuff background; at least his background in systems is really focused on sort of experimentation and getting results and high performance. My background has been much stronger on the sort of mathematical formalisms and theoretical stuff. Even though you can say we both have computer science backgrounds there’s very different tracks that we followed and I think they complement each other pretty well in terms of knowing how to solve different sorts of problems which end up cropping up in a large scale research project like this.
Christopher: Yeah and more specifically for example since, rather actually it’s slightly different, but the areas of computer science that we’ve explored so for me, doing distributed systems that’s really important for the aspect of Geopipe where we’re taking a huge amount of data about the world and organizing that data, we’re processing that data in a distributed setting. But the thing we actually have to do with that data is apply a lot of really clever algorithms to that data and that’s where Thomas’ particular expertise comes in as well as building that system in a way that’s performant, that’s going to be correct, and where Thomas’ theoretical background and his background in simulation and modeling has been extraordinarily helpful.

Alex: How did Geopipe become MORE than an idea?
Thomas: So, there’s a thread on Cemetech, under Cemetech Labs, “Google Earth to LDraw/Minecraft Voxel Deomp” which is this ten-page thread dating to March 2013. That’s basically where Geopipe came from. Christopher and I were floating this idea around for probably 6 months before that thread cropped up. We were both really interested in the game design aspect of being able to interact with the real world in a virtual setting. We were both really interested in these open world sandbox games where you can build a lot of stuff and after talking about for a while we realized that doing the image recognition and sort of dynamic programming to restructure a real building into a LEGO model with actually using windows for the windows and doors for the doors would be pretty hard. So it made a little bit more sense to start with the easier task of just voxelizing it for minecraft.
Christopher: Voxelization being the process of just turning it into a set of full-sized cubes (author's note: A voxel is a pixel with a physical volume).
Thomas: Then let’s see… we started working on that and then I took a computer vision and computational photography class at Brown but I was looking at this sort of automated recognition and decomposition of real world stuff into more iconic representations. Around this time Christopher started realizing that he was getting near the end of his postdoc and needed something to do afterwards, and we always talked about doing a startup of some kind together so this was sort of a natural path. We started trying to find who was actually going to be our customers and pay for this and we really quickly realized that game designers weren’t the people who were going to spend a lot of money on these 3D models.
Christopher: After talks with my [PhD] advisor I naturally contacted Thomas and we explored something more formal of what we’d done with SparseWorld and FreeBuild in the past and it’d be really fun to try and explore that as a start up. So we did an accelerator program last summer, which is when we first incorporated into a company, and started really exploring and doing the customer discovery that Thomas just talked about. And here we are today!

Alex: Very cool! Christopher, so tell me a little bit more about SparseWorld. That was a pretty awesome project. I remember going onto Cemetech and being able to look and interact with a map.
Christopher: That was something that I had kind of wanted to do for a while. I had always kind of had the idea of putting worlds into video games. Even as far back as Blockland, way, way back in the day with Thomas, you, and others. I started working at how I could recreate real-world buildings. For example, I had a project with one of our Blockland members to reconstruct my high school in Blockland as LEGOs. As I started playing Minecraft it was certainly an obvious thing beyond exploring and building virtual worlds, that it’d be extremely cool if we could put the real world into Minecraft and explore playing with the existing buildings or creating a whole new building within New York City (for example).

Then there were some other projects through that, for example, made it easier: a python interface to write minecraft worlds from a program; a project that someone had been working on to take elevation and land cover information from the USGS and turn those into minecraft worlds. I was able to take the work that he’s done and then start looking at things beyond just pretending everything was grass. What if I want to lay a satellite image on top or, what if I wanted to then make 3D buildings out of that, all of those pieces helpfully came together into something that turned into SparseWorld.

With SparseWorld, though, we ran into the problem that we couldn’t actually get 3D models of the real world. Like, we thought “Oh we could just go to Google Earth and they’ll give us all the building models because we see them in Google Earth.” But we found that beyond the 3D Warehouse website where they had special buildings that their members had made, like people had made a really nice Chrysler Building and a really nice Empire State Building, you couldn’t just get any old apartment building or any old house from Google Earth. You can scrape them, but that's illegal, so that wasn't an option for us, plus those models still lack semantic information. If there’s another 3D map provider like Apple Maps or Here Maps or Bing that provide something we can use - maybe? - but in the end we thought that there must be other people who actually need these 3D models to make 3D worlds. What if we stepped back a bit and looked at the raw data to see if we could actually go through the process of creating these models ourselves? And of course as computer scientists we wanted to spend the time making those by hand, we wanted to [naturally] automate this process.

Thomas: I think it's worth bringing up that about precursors like SparseWorld, there's also the fact that Christopher replicated his apartment, the KermCave map, for Blockland then eventually ported over to Freebuild. So we had this idea of wanting to play in the real world for a long time.
Christopher: Yeah, for sure.

Alex: Thomas, can you tell us a little bit more about your Freebuild project?
Thomas: So, before Minecraft there was this game called Blockland. It was basically a game where you can build stuff out of LEGO. The only drawback was that there was tons of drama between the developers. Three main modding groups battled around and really fragmented the community. It was also built on Torque, an open-source 3D engine which got really slow after a few hundred entities. Which, ya know, is a problem when you have models built out of individual LEGO bricks. Another problem was that we couldn't build vehicles, they had to be made in an outside program then imported into Blockland. So I set out to start from scratch to fix all these issues. The problem was that Torque didn't support Mac, my main development platform so I tried developing a game engine from scratch which eventually I fell behind on and never finished it.

Alex: What are some challenges you expected to face before you started Geopipe?
Christopher: Definitely some technical hurdles. As a hobby you can take the time to think about a solution and come back to it. As a company you need the solution as fast as you can find it. So it was also a matter of meeting deadlines and sleepless nights as we collaborated and debated back and forth.
Thomas: It's also my - our - first for-profit business. I was worried that we didn't have the perspective to face the commercial challenges. But we've had tons of support from the programs we've been in and also making decisions as they present themselves.

Alex: What are some challenges you are facing or expect to face?
Christopher: Topology is hard. That's really all I have to say.
Thomas: The technology isn't quite here. The stuff we want to do is stuff that's actively being researched by others so the research papers aren't hard to come by, but when we do the data sets are super manicured so it's easy to understand but then the issue was that these same techniques don't work particularly well on noisy, inaccurate, and un-manicured data. So there is a lot of "How do we get this to work on real-world data?"

Alex: We covered this a bit already but can you go more in-depth about Geopipe's product?
Thomas: We definitely want to construct increasingly detailed models of more areas. Right now we're limited to the areas we have data for. Ultimately we'd love to take a satellite image and build an accurate 3D model from as little data as possible. We'll have to define a specific grammar in the algorithms, such as doors are always on the ground level and build a detailed model based on guesses of where things are. We also want to be able to distinguish a tree from a lamp post in a satellite photo so we can accurately represent the surroundings.
Christopher: We definitely have some verticals [markets] in mind, such as training simulators for the military and even for truck and train drivers, to licensing our models of cities to special effects studios for TV and movies.
Alex: What if you teach your system a new object, like a mailbox. What then? Do you keep any of the data you can't interpret or do you toss it out?
Christopher: We keep the data the system and algorithms don't understand; any data we can't understand we don't place in the world. So if we do teach the system what a mailbox could look like and where it could be, we can reprocess the set of data to find and place those new objects in the world. That saves us from having to re-download everything and reprocess a large set of data all over again.

Alex: I've been following you guys on Facebook and Twitter; you guys have entered a lot of events and competitions. Have you noticed any additional interest after each event?
Christopher: We get tons of business cards so we spend time following up and sending off e-mails to those people. We also get tons of unsolicited e-mails from the form on our website of people trying to offer services.
Alex: Were you ever compelled to follow up with any of those e-mails?
Christopher: Only twice. Haha. But we ultimately didn't continue the venture.

Alex: What sort of waves are you making?
Christopher: We are excited to be a part of various events, programs, and panels like Propellify, the NYC Media Lab Summit, Exploring Future Realities, NYU's $300K Entrepreneurs Competition, and others, and we expect to get a lot of publicity from a prestigious accelerator program that we'll be starting on July 10th. But we've certainly seen a little bit of traction on Twitter over the last year.

Alex: Any closing remarks?
Thomas: We're excited to be entrepreneurs! It's a fun ride and we're looking forward to where it takes us.

Alex: What about Cemetech? Will you be sticking around at all Chris?
Christopher: Definitely. Cemetech is a labor of love and I don't anticipate it falling to the wayside.



Update: July 13, 2017: Fixed an inaccuracy, the organization Thomas started was a 501(c)3, not a 503b. Also added in Thomas' influence towards his career in education.
Lend it to Nintendo for New Donk City, cuz the last screenshot above looks really good.

Seriously though, does it rely on an engine like UE4 to do this?

This product seems really versatile, which could bode well if this is a product you want to lend or sell to the masses.

Good luck on the start-up!
Alex: Thanks for a great article! It was a pleasure to talk to you about Geopipe, and I'm glad we have a chance to share what we're working on (and where our time is going) with the larger community.

xMarminq_ wrote:
Lend it to Nintendo for New Donk City, cuz the last screenshot above looks really good. [...] This product seems really versatile, which could bode well if this is a product you want to lend or sell to the masses.
That's a great idea! We're very excited about the applications to video games.

Quote:
Seriously though, does it rely on an engine like UE4 to do this?
We used Mitsuba to generate the above render, but the model being rendered is purely our own geometry and materials.

Quote:
Good luck on the start-up!
Thanks a lot. Smile
Lots of little details go into these architectural models beyond just the building model that should be pretty fun for you guys to automate:

1) Building material properties (metallic, roughness, specularity) that strongly effect light and realism. A lot could be extracted from comparing high ambient light images (twilight) and high directional light images (and also depend on the angle). I noticed the windows were fairly low-rez/uniform.. I'm guessing these are being established automatically?

2) Trees could be identifiable in many contexts and replaced with a higher detail model. Integrating technology like SpeedTree into the process would be neat.. using existing data to inform the correct procedural tree. I bet those guys would be into it.

3) Identifying sidewalk areas, and then putting little people models around the buildings, ditto for cars and roads.

I'm guessing some of this can be placed in existing architectural tools once imported from geopipe, just seeing a lot of potential improvements beyond the straight capture... the prettier it is right out of the box the easier it will be to sell.
I'm actually really excited to see games use Geopipe, some titles like GTA would really benefit from the models that this would provide. Additionally, I think if glass could be rendered realistically in the high detail models, the models would look even more amazing than they already do.
I'm really excited for people to see Geopipe's use outside of gaming

Agreed. As for trees, in most architectural models they are just represented by blobs (it's a technical term), but perhaps it could generate trees, maybe in a fractal pattern, but with a little bit of added randomness (i.e. make it look like a tree)

This looks like a truly awesome concept, well done!
This is the softest "interview" I've ever read.
Congrats guys! I wish you guys the best of luck with your entrepreneurial adventures!
Congrats! I have one burning question: Can you use the machine learning stuff you've implemented to generate new realistic terrain? That would be the next huge step, in my opinion.
Perhaps some flight simulator in the future will use a technology like GeoPipe to build hyper-realistic cityscapes.
I know this is somewhat of an old topic, but the Cheddar Interview about Geopipe left me with some questions.

They talked about Augmented Reality in the video. Now, this may seem a bit out there, but could Geopipe and AR, sometime in the future, be used in a way to display what an area looked like in the past (using data from the past, of course). I'm thinking about this because it reminds me of this scene from Guardians of the Galaxy. Thoughts?
oldmud0: That's the idea!

123outerme: We've pondered some interesting stuff around that. For example, letting artists sketch a "satellite photo", and then generating the world in question from that "photo".

Michael2_3B: Great question! We have indeed thought about this one as well. It's particularly challenging, because we have limited data from which to work and no ability to get someone to go out and collect new data. However, we can certainly simplify the process if we can get things like building footprints, some photographs, and teach our system a learned model of what X city looks in Y year.
KermMartian wrote:
For example, letting artists sketch a "satellite photo", and then generating the world in question from that "photo".


That sounds interesting. Could you elaborate more on that? Like, could I sketch a neighborhood in pencil from above, as if it were a satellite photo, and your algorithms would interpret that or would I use tools provided by Geopipe to build that neighborhood in software?
I have a few things that could be much better with Geopipe's area downloaded.
1. Please include a tutorial on how to select an area. Just through trial and error, I found out that double clicking started the selection. (The 'tutorial videos' button under the 'Help' drop down menu doesn't work)
2. Personally I would prefer a click-hold and drag selection with a button dedicated to toggling if you are selecting an area or moving the map around.
3.. Please include a 'clear all' button since my final selection was really jagged and I wasn't really happy with it.
4. I made a free account... to upgrade to a free account? And then when I tried to upgrade to the 'premium free account' (I'll call it that) I was required to give a credit card number. I don't have a credit card. Also, why do I need to have a credit card for a free account?

Thanks for reading my complaints, I hope they'll help improve Geopipe. Smile
TheLastMillennial wrote:
I have a few things that could be much better with Geopipe's area downloaded.
These are really helpful, thanks! I'll try addressing a few of these in order:
Quote:
1. Please include a tutorial on how to select an area. Just through trial and error, I found out that double clicking started the selection. (The 'tutorial videos' button under the 'Help' drop down menu doesn't work)
Will do. Would a tutorial video do the trick by itself? What about some kind of in-UI popups or callouts that explain what to do next until you tell it to stop doing so?

Quote:
2. Personally I would prefer a click-hold and drag selection with a button dedicated to toggling if you are selecting an area or moving the map around.
I tried that at one point, and it seemed like it was more confusing than this system to the average user. But I agree that it's not obvious that you need to click multiple times to make a selection.
Quote:
3.. Please include a 'clear all' button since my final selection was really jagged and I wasn't really happy with it.
How about instead of turning into a greyed-out Start button, the Start button turn into a Clear All or Start Over button?
Quote:
4. I made a free account... to upgrade to a free account? And then when I tried to upgrade to the 'premium free account' (I'll call it that) I was required to give a credit card number. I don't have a credit card. Also, why do I need to have a credit card for a free account?
It's our current way of deduplicating users. Because the free account tier has strict usage limits, this ensures that you can't just make 20 accounts to grab a whole city.
I just read this article: https://techcrunch.com/2018/03/14/google-is-using-maps-to-turn-every-video-game-into-an-earth-sized-epic/
How is the Google Earth API going to affect geopipes market?
https://geopi.pe/getmodels

4 km² is 1.54 mi², not 2.49 mi² Smile

The division should be by 2.59 not 1.609.
  
Register to Join the Conversation
Have your own thoughts to add to this or any other topic? Want to ask a question, offer a suggestion, share your own programs and projects, upload a file to the file archives, get help with calculator and computer programming, or simply chat with like-minded coders and tech and calculator enthusiasts via the site-wide AJAX SAX widget? Registration for a free Cemetech account only takes a minute.

» Go to Registration page
Page 1 of 1
» All times are UTC - 5 Hours
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

 

Advertisement