While watching Minecraft videos, I often find designs that I would like to replicate, but no world download is provided with the video. Other times, I would like to do a render with Chunky from a screenshot only. I sometimes decide to painstakingly replicate large builds block-for-block using a video or screenshot.
I think it would be interesting to develop a program to do this automatically.

One possible approach I came up with:

1. Figure out which pixels belong to the UI and should be ignored (Hotbar, inventory, F3, status effects, etc.)
2. Determine rotation and position relative to the block grid
3. If a video or set of screenshots is being used, determine if a block boundary has been passed since the last screenshot.
4. Generate a list of block positions to scan, ordered from closest to farthest, that are within the FOV
5. For each block position, take each potentially visible face and figure out which pixels would be visible if it were the only block in the world
6. Rotate/skew a copy of those pixels so that they form a square shape
7. Determine if there is a "correct" section of this face visible (not blocked by a closer block)
8. If so, determine which block it is most likely to be based on the textures
9. Add that block to the schematic

I haven't started on anything yet; I just wanted to post this concept here and get your feedback.

The one huge major flaw with this is that minecraft is 3d.

The way around that would be to use multiple screenshots from multiple angles, but that would get really messy really fast.

Steps 5-7 are included to compensate for this. As for the multiple screenshots, the first screenshot is assumed to be at the origin, and subsequent screenshots can use a position relative to that if part of a video, where it is easier to figure out the position. If not, the schematics from several screenshots can be combined afterwards to form a single more-complete schematic.

One could look into Photogrammetry techniques to extract a 3D model from a series of frames. It works best if the background doesn't have a lot of detail so it may not work well on Minecraft videos but it could be a decent start.

I agree, although you would have to have the program parse the whole video, as most structures have an inside an an outside, making the program quite large.

Despite that, I really think that this would be awesome. Good luck!

I've never actually seen this technique used for interiors but I suspect it could work. I guess I was thinking more along the lines of those redstone tutorials where they have the different color wool blocks to illustrate different circuits and stuff. Rather than actual buildings. Which, photogrammetry would still work but the interior would probably need a different method.

Photogrammetry works because the program/algorithm assumes all frame references are pointed towards a common center, which isn't easy to do for an interior.