Depth maps -- tools and applications
Facebook 3d Photos: this is one of best ways of showing parallax effects with an image via a depth map -- for publishing your 3D conversion work, or checking depth map retouching etc
For three or four years now you can upload a photo + multi lens/depth sensor data -- to make a depth map based parallax effect image in the cloud from a recent Iphone -- to a Facebook personal feed or Facebook Group:
Or more recently anyone can just upload -- to a Facebook post or Group -- at the same time -- an image (taken with any phone) -- say "xxx.jpg" -- and a depth map you have made - of the same dimensions - named like "xxx_depth,jpg".
The depth map must be coded with depth being lighter in the foreground of the scene, darker in the background. The depth map must be RGB rather than grey scale if it is a png. After a few seconds of cloud processing the Facebook 3d Photo will appear - and you can try it and see if the depth map needs more work.
Facebook 3d Photo workflow
Generally the depth map must look very low in contrast to work well as a Facebook 3d Photo. If there is too much contrast there will be distortions in the parallax animation as the user interacts with it -- at the edges of objects in the scene especially.
btw -- I think this usual requirement of low contrast for good Facebook 3d Photos effect depth maps may possibly have to do with the small separation of the lenses on the phones used to generate (with Facebook's AI processing) the effect - in the original Iphones that could do it -- like for instance with Google's "Stereo Magnification" paper -- with small baselines the depth map from such cameras was inherently low contrast
https://arxiv.org/pdf/1805.09817.pdf
"we explore an intriguing
scenario for view synthesis: extrapolating views from imagery captured by
narrow-baseline stereo cameras, including VR cameras and now-widespread
dual-lens camera phones"
Anyway if your depth map has too much contrast to work well with Facebook 3d Photos you can use Levels (with selections maybe) to reduce the black density and/or increase the white density. Sometimes it helps to turn the image into 16 or 32 bit first before you do this (and then turn it back into 8bit before saving as jpg/png) -- as this can reduce banding in the depth map image from the Levels operation.
Another thing you can do to reduce distortions in your Facebook 3d Photo from too much contrast is to use the "white dot" technique. What you do is this -- look at your depth map. Use the Eyedropper in Photoshop to select the lightest tone in the image. This will now be the Foreground Color. Click on the Foreground Color icon and select a tone a bit lighter than what it is (or pick even pure white). Use the brush or pencil tool to paint a small area in a corner of the depth map with the new brighter tone. Now there will be less distortion in the generated Facebook 3d Photo (and the image will be cropped slightly anyway so you won't see the spot you have made.) Presumably Facebook must look at the entire tonal range of the depth map image before estimating the amount of parallax to generate. This "white dot" process can reduce the depth map "detail" of the effect though and make the Facebook 3d Photo less interesting depth-wise.
Re this "cropping" of the Facebook 3d Photo effect image compared with the original photo/painting etc. This can be very annoying especially at top and bottom of the frame. So often what I do is use Canvas Size to increase the height of the image (and depth map) -- maybe 10%. Then I use Photoshop's new Content-Aware Fill on both image and depth map to smartly add detail in the extended top and bottom areas.
Facebook 3d Photo Groups: the most popular (in terms of number of subscribers) Facebook 3d Photo Group is this one I think --
https://www.facebook.com/groups/3dphotographs
Another Group is this one:
https://www.facebook.com/groups/LetsConvert2DImagesTo3D
... but this is about 3d conversion of 2d images in general .. so there are anaglyph, side by side format etc images as well as Facebook 3d Photos. . Both groups you have to join to post but are publicly accessible otherwise.
Embedding Facebook 3d Photos into blogs etc: it is possible I think -- but maybe it is a bit browser dependent.
For example: https://rostaman.pp.ua/lotus-ode-zhang-liying-1962.html
See: https://developers.facebook.com/blog/post/2018/02/20/3d-posts-facebook/
Alternatives to Facebook 3d Photos for VR viewing: For a year or two after it was introduced it was possible -- by accessing Facebook 3d Photo urls with a VR web browser - with some VR headsets, for example , Oculus Quest - to see them in very impressive 3d in VR. It was like you were standing in front of a large hologram the height of a person. There was some limited amount of 6DOF freedom of movement about the experience. If you moved too close or laterally the picture moved away from you and/or centered itself in front of you again. But then unfortunately Facebook turned off that capability.
Recently though there is something similar in the form of Owl3D which is a (paid) service to make AI depth maps from images and presents them in VR headsets (from an url) in the same sort of way (though it creates what looks more like a 3d reconstruction than a hologram.) https://www.owl3d.ai/
3D conversion of still photographs or paintings: There are a variety of methods, non-AI and increasingly AI, that people use to make 2D pictures 3D - and most of them involve creating depth maps somewhere in the process.
Here is an extensive article on various (mostly non-AI 3d) still conversion techniques.
shorturl.at/csvJ5
(this file opens in document edit mode in Chrome - to make the urls in the article clickable you can go ... View/Mode/Viewing).
Here is a recent Youtube tutorial covering 3d conversion etc AI innovations for stereographers.
https://www.youtube.com/watch?v=LuTEkytsTu4
Boosting Monocular Depth (BMD) -- mid-2021 there was a big advance in AI depth map from images quality in the form of this research:
https://www.youtube.com/watch?v=lDeI17pHlqo
For using the Colab version there are a couple of tutorials:
https://youtu.be/LuTEkytsTu4?t=922
btw he has a bugfix version of the Colab here (I am not sure what bug):
https://colab.research.google.com/drive/18an1b-ffk6K_lz3p2ZIdSfa8Qzhk5r1P?usp=sharing
https://www.youtube.com/watch?v=_-kYlCmIqyA
https://www.youtube.com/watch?v=SCbfV80bZeE
Here are some of my workflow notes with this AI software and its outputs:
Upscaling:
It is important if you want to get very detailed depth maps with BMD that the source image be as large (but still detailed) -- as possible.
So sometimes I use the new Enhance filter in Adobe Raw on the source image (it can 2X upres + denoise). The source image must be a Raw format or DNG file.
For example here is a 3d conversion (anaglyph) from a monoscopic 360 panorama of a biker funeral. The original was 7200*3600 pixels and the upres Enhance version was 14400*7200.
https://www.facebook.com/groups/LetsConvert2DImagesTo3D/posts/3972563566198774/
The stitched panorama original was jpg so I had to convert it to DNG which is non-trivial these days. The only way I have found to do it now is with Topaz Adjust AI which luckily I had.
Here is another Enhance usage example -- depth map with BMD from a Youtube video -- of a urban explorer guy in Vorkuta, Siberia -- 2X from a 4K video frame https://www.facebook.com/groups/3dphotographs/posts/862064171143568/
Panorama conversion and depth map effect previewing: Sometimes BMD does a great job making a depth map from a 360 equirectangular image directly .. but sometimes one needs to split and transform the source image into overlapping wide angle rectilinear images. And then generate depth maps with BMD from each rectilinear image and then stitch the images and the derived depth maps back into equi format. (Using PTGui Pro for these splitting and joining steps). That was the case with this historic 360 panorama of Sydney (1873):
I think with the degraded original source image quality (scans of 10 overlapping vintage prints) and limited vertical view of the historic panorama BMD needed the converging verticals of my downwards tilted extracted wide angle views (the panorama was shot from a tower) to work out the geometry of the scene (presumably BMD - with Midas or Leres AIs -- could use "Manhattan World" AI depth cues with this scene type).
https://www.facebook.com/groups/LetsConvert2DImagesTo3D/posts/4101448026643660/
I have been retouching the depth map of the panoramic scene with the aid of PTViewer3D -- which is a great image+depth 360 3D viewer. Using my 3D TV, which runs as one of my monitors, in squeezed SBS mode (one of the output formats supported by PTViewer3d) ... see my Reddit post here:
https://www.reddit.com/r/depthMaps/comments/pt35si/ptviewer3d_an_imagedepth_map_panorama_player/
Sometimes I use a VR headset (Oculus Rift) instead of - or alternating with -- my 3D 55" TV/monitor (switched in non-headset, direct viewing (with glasses) into squeezed SBS mode) -- as the 3d display for depth map effect previewing and retouching work -- still seated at my desk with my graphics tablet handy -- with Virtual Desktop running in my headset with the squeezed SBS output from PTViewer3d -- with a stereo view in my headset of the image on a virtual screen. In VR I have configured settings in Virtual Desktop (running from Steam) so this virtual screen appears about 1m away, and of a width so it subtends an 80 degree view ie. very large and close (in fact like my real world 3d monitor aka 3D TV view -- only with the depth errors even more visible than on the 3d TV monitor)-- and as I lean in the 3d effect in headtracked VR is heightened in a quasi-6DOF fashion that makes the depth map errors even clearer.
Alternatively, rather than using the stitched 360 panorama/depth map image pair with PTViewer I work with the extracted tilted wide angle (about 75 degrees wide) rectilinear views and their BMD derived depth maps -- where the depth map retouching is often easier as all the buildings in the depth map now have straight lines rather than equirectangular curves.
In this case I use Stereophoto Maker (SPM) and its depth map tools -- for converting from image/depth to outputting squeezed SBS images (or anaglyph) -- for visualizing errors. http://stereo.jpn.org/eng/stphmkr/ . With SPM you can import an image + depth pair as though it was a regular L/R stereo image pair, then then in the Depth Tools section of the Edit Menu you can set the "stereo base" to a high value and output a real squeezed SBS L/R pair with exaggerated depth. And again view that on a virtual screen up close in VR space with Virtual Desktop in a headset.
more lings
https://arxiv.org/pdf/2109.07547.pdf RAFT- Stereo
