Depth from stereo -- depth map retouching

A Facebook 3d Photo ... Newtown Festival

Workflow steps for depth from stereo:
1. Generate "*.exr"  (HDR) format image from Fusion Studio (using KartaVR depth from stereo template). This is 32 bit.
2. Convert it to 16bit tif in Photoshop (or sometimes also with Topaz Studio or HDR Efex Pro 2)
3. Adjust the balance and contrast of the 16bit tif version globally and locally with Photoshop. I often use Viveza plugin  for local density/contrast and "structure" corrections of the depth map at this point. The idea is to have depth map detail visually apparent everywhere you want to see it in the end experience (6dof, Facebook3D  photos etc)
4. Convert the depth map tif to a jpg. Now the depth map looks like this:

You can see at this point there are depth map tonalities varying continuously and pretty evenly from foreground to distance.  You can check it now in 3D with Stereophoto Maker depth tools (in anaglyph or on a 3d TV with squeezed SBS) or browser /headset previews with krpano after converting fisheye image and depth to equirectangular. You only have so much depth representation capability in an 8bit depth map so it has to be used efficiently. You can see that the depth information looks sort of accurate except that the contours of most of the objects need a lot of refinement.

Now currently I am using a "feature image" -guided smoothing software on the depth map  as a preliminary step before detailed contour correction - JointWMF
There is a application download on this page, with various *.bat templates for different filtering applications, and I use a version of the "demo_texture_smooth" one. You can use multiple iterations of the filter with the "t" parameter (no. of repeats). I usually use somewhere from 2 to 8 iterations. At a certain amount of iterations the depth map contours start to no longer improve and density banding (posterisation) starts occuring. The feature image is your source image (one of the original stereo pair), the depth map for that image is the one you are processing with the filter. The improved depth map is the output image. Two iterations take about a minute on my hardware with big files.
The bat file I used looks like this (but I didnt do much testing of possible parameter combination variants  -- you might need to do your own matrix of tests) :

JointWMF -i data1/newtownRdepthmap.jpg -f data1/newtownRsourceimage.jpg -o data1/correctedRdepthmapimage.jpg -r 30 -si 25 -w exp -nF 1028 -t 8

Here is the filtered depth map after JointWMF:

Now the job is standard Photoshop work. I have been working with the image and depth map side by side as a single image but as separate pair works too. Import the single composite image or image/depth image pair into Stereophoto Maker as if it was a standard stereo file (SBS or L/R pair) and then use the Depth Tool to make it into a anaglyph or other format eg. squeezed SBS for 3d TV to check your retouching. SPM will remember your source so you just need to reupload to check your retouching.

The natural thing in Photoshop to do is to work from foreground to background. What I do is this (with my image+depth single composite image). I select a rectangle area  of where I want to fix the depth map from the corresponding section of the source image. Copy that area in register onto the corresponding region of the depth map (use Difference mode etc to fine tune registration). Select a foreground object or figure on the image area. Hide that image layer. Make the depth layer active and invert the selection to fix the depth map contour by cloning from the adjacent background depth map region around the selection. Where the depth map is totally wrong or missing in  a region  eg. the fence railings I select them on the image and fix the depth map by filling with depth gradients (linear or radial ).

Creating and publishing stereoscopic depth map panoramas

Since my previous posts there have been very significant developments with this!

On the publishing side:
Most significantly, krpano,  has released a version with support for image+depth  360 panoramas. It supports anaglyph html output for the desktop -- and WebVR immersive 6DOF or 3DOF stereoscopic panorama viewing - in supporting browsers (eg. Firefox Reality) - on vr headsets eg. Rift, Vive, Go and Quest.

https://krpano.com/forum/wbb/index.php?page=Thread&postID=79218
https://krpano.com/forum/wbb/index.php?page=Thread&postID=79218

On the 360 depth map creation side:
most significantly, Holokilo has released an alpha version of  an application - Deep360 (free) - that uses AI to automatically produce a depth map from monoscopic equirectangular 360 panorama images.
https://holokilo.wikividi.com

The developer Kaj Toet says he is mainly working on a version for 360 mono videos but this works amazingly well for some 360 stills. It outputs depth maps and over-under stereo. (He also has an Android app for any kind of image, not necessarily panoramic -- https://play.google.com/store/apps/details?id=net.kajos.holokilo&hl=en   )

More recent versions of his Windows app have a number of controls:
"Limit Stereo Edge" means fade off the stereo at zenith and nadir (to avoid stereo viewing failing there with the Over-Under output -- a universal issue with ODS (Omnidirectional Stereo) formats generally -- but absent with image+depth panos as correct stereo is generated dynamically in any direction with image+depth pano players)

Other ODS issues are problems when your head is "rolled" and very wide angle, even mildly tilted views. Dersch says ODS is "poison" for good panoramic stereo in one of his articles. (Depth maps are not the only way to avoid being poisoned -- light field technologies eg. Google's Welcome to Light Fields demo, has perfect stereo panoramic viewing in any direction as well as some 6DOF movement capability volume.  https://store.steampowered.com/app/771310/Welcome_to_Light_Fields/  )

"3D strength" means what it says -- it controls the amount of disparity produced -- which corresponds with the depth map to how contrasty it is. I usually have it set to between 3 and 6.

"Dilate filter" controls how much the depth map edges are expanded past the corresponding edges of the source pano image elements ie. how much it overlaps it.  I am not sure why he has this -- it improves the over-under output quality I think. But I usually set it to zero as I am primarily interested in the depth map output and clean contours are more generally useful I think (I can generate stereo formats from image+depth with Stereophoto Maker with one of its depth  tools or with After Effects with the Displacement filter).
http://stereo.jpn.org/eng/

btw ... about ODS rendering: https://developers.google.com/vr/jump/rendering-ods-content.pdf


How does Deep360 do its magic!  I don't know ... but "Deep" means  -- with AI -- of course -- but what kinds of AI? One common AI approach for depth from mono - for one kind of subject - seems to be with the cool-sounding "Manhattan Scene Understanding" concepts:
 http://grail.cs.washington.edu/projects/manhattan/manhattan.pdf

"We focus on the problem of recovering depth maps, as opposed to full object models. The key idea is to replace the smoothness prior used in traditional methods with priors that are more appropriate. To this end we invoke the so-called Manhattan-world assumption [10], which states that all surfaces in the world are aligned with three dominant directions, typically corresponding to the X, Y, and Z axes; i.e., the world is piecewise-axis-alignedplanar."

Other methods of producing depth from monocular 360 images
Depth (or modeling)  from monocular 360 images is a very popular AI research topic but there are few software implementations that just anyone, without machine learning software compilation know how,  can use.

Matterport, the commercial realestate 3d scanning company has 360 image support. They call their technology Cortex AI.  It is for generating models from building interiors with 360 photography.
https://matterport.com/360-cameras/

Everpano, a image-based modeling application, is a sister program for krpano image+depth panorama  publishing. It is rather basic in the sorts of  models it can produce (it is mainly for interiors) -- but it can model directly from  equirectangular panoramas. It also has scripting features (the Navigator Plugin)  for 6dof navigation around the models Everpano makes from 360 pictures --  with krpano.
https://everpano.com/
Krpano has a utility which will  produce a panoramic depth map from Everpano model if your primary output is going to be image+depth. It also has a tool for matching depth map produced scale to the real world .. important for vr exploration.

In the next release, the krpano developer says, with Everpano's Navigator Plugin, we will be able to navigate intuitively in 6Dof, in VR with Quest say, with image+ depth maps alone by clicking on or moving around  multiple panoramas.
https://krpano.com/forum/wbb/index.php?page=Thread&threadID=16984

More depth from monocular AI software:
Masuji Suto's (of Stereophoto Maker) implementation of Google Mannequin Challenge software:
 "Learning the Depths of Moving People by Watching Frozen People."https://www.youtube.com/watch?v=prMk6Znm4Bc
https://github.com/google/mannequinchallenge
http://stereo.jpn.org/jpn/stphmkr/google/indexe.html 
This requires a recent CUDA card to work (I have it working on a PC with a nvidia GTX 1650). This is I think the most basic deep learning capable card it will work on. The output resolution is highly dependent on the amount of video ram on the card.. The maximum it will produce from a panorama with this card (4Gb video memory) is 512*256. (By processing the panorama in sections you might be able to get more resolution but I don't know how well the depth map would stitch.)

Retouching automatic depth from monoscopic panorama depth maps:There are multiple AI depth from mono programs now that are quite specialized in the sort of subjects that they work with. Using different AI for different problem areas in a 360 pano depth map can be an efficient method.

FACES
For instance there is this online service for automatic face models with AI
3D Face Reconstruction from a Single Image     https://cvl-demos.cs.nott.ac.uk/vrn/
https://www.engadget.com/2017/09/19/AI-3d-face-single-photo/?guccounter=1

So you cut out the faces in your mono panorama, upload them, download the models and produce the depth maps. How to produce the depth map? -- you could export a depth channel with After Effects ... or use this workflow with Meshlab (free) -- which has nice controls for the depth map tones ..
https://community.glowforge.com/t/tutorial-creating-a-depth-map-from-a-3d-model-for-3d-engraving/6659
http://www.meshlab.net/
... and then fix the faces in your panoramic depth map image.

Or there are mobile phone apps that do a good job producing  depth maps of  faces and bodies ... popular apps I guess with phone uses whose phone does not do depth maps with hardware .. and who want depth of field control with portraits.
eg. DPTH
"uses AI to predict depth even on single camera phones"
https://play.google.com/store/apps/details?id=ru.roadar.dpth&hl=en
...the paid version lets you save the depth map .. for panoramic images depth map capture you can put the image on a screen and photograph the relevant area with the phone

More generally ..
Often with a panoramic depth map you would want to extract rectilinear views from the area of the source image that needs retouching on the depth map -- with PTGui say, and build a model from that image area, make a depth map from the model and insert that depth map back into the equirectangular depth map image -- again with PTGui.

So, for instance, if you  wanted a depth map of a panorama of  that amazing giant crystal cave in Mexico you could do  it like this in Photoshop -- much more easily in a rectilinear view than in equirectangular:
"Tutorial on manually creating depth-maps in Photoshop for pictures with complex depths."
https://www.youtube.com/watch?v=GUGafT3WWl4

Gimpel3d as a image+depth editor and previewer Gimpel3d supports 360 cylindrical and dome panoramic format input images -- as well as rectilinear. It only runs on one of my PC's (?)-- but it is very interesting but hard to get a handle on documentation-wise. It has SBS export and depth map input and export, and depth brushes. https://sourceforge.net/projects/gimpel3d/ http://www.freeterrainviewer.com/G3DSite/index.html https://www.youtube.com/watch?v=jxnlFqjQIVk https://www.stereoscopynews.com/hotnews/3d-technology/2d-to-3d-conversion/1647-gimpel3d2-conversion-software.html http://3dstereophoto.blogspot.com/2013/10/2d-to-3d-conversion-using-gimpel3d.html It works very well as an interactive anaglyph 6dof explorer player of wide angle image+depth extracts from Deep360 panos -- on the desktop.

https://www.google.com/search?q=depth+from+monoscopic+panoramas:
... one year's (2019) worth of studies:
https://richardt.name/publications/megaparallax/MegaParallax-BertelEtAl-TVCG2019.pdf

https://hal-mines-paristech.archives-ouvertes.fr/hal-
01915197/file/Cinematic_Virtual_Reality_with_Motion_Parallax_from_a_Single_Monoscopic_Omnidirectional_Image.pdf

Post-stitching Depth Adjustment for Stereoscopic Panorama
https://ieeexplore.ieee.org/document/8682724
 "For those regions that still suffer from visible depth error after the global correction, we select control points in those depth anomaly patches and utilize the Thin-Plate-Spline technique to warp those features into their target position."
Thin-Plate-Spline" warping is in ImageJ (Fiji). It is a really fast filter.
https://imagej.net/BigWarp
More than you ever wanted to know on image stitching tactics generally:
A survey on image and video stitching.
https://www.sciencedirect.com/science/article/pii/S2096579619300063

https://www.researchgate.net/publication/336304243_Real-Time_Panoramic_Depth_Maps_from_Omni-directional_Stereo_Images_for_6_DoF_Videos_in_Virtual_Reality
https://www.youtube.com/watch?v=XLOcrAZtc7w
This is a very clear talk. They are working with Suometry -- who have a fast realtime stereo pano capture camera technology. They use stereo panos and normal maps to train realtime depth from mono.

https://web.stanford.edu/~jayantt/data/icme16.pdf
DEPTH AUGMENTED STEREO PANORAMA FOR CINEMATIC VIRTUAL REALITY WITH
HEAD-MOTION PARALLAX
.. a cure for ODS .. "The proposed representation supports head-motion parallax for translations along arbitrary directions."

Motion parallax for 360° RGBD video
http://webdiis.unizar.es/~aserrano/projects/VR-6dof
they have 3 video layers to reduce occlusion errors ... there is a demo:
http://webdiis.unizar.es/~aserrano/projects/VR-6dof.html#downloads

https://www.vr-if.org/wp-content/uploads/Reality-Capture-Present-and-Future.pdf
his prediction 2018-2019:
'“3DOF+” is introduced and begins to gain traction in high-end offline productions, supplanting stereoscopic 360 almost entirely'   mmm..

Depth maps, volumetric experiences and stereo panoramas.

"3D Photos"
Consider this Facebook "3D Photo" scene of mine. If you move your mouse over it it will animate in a sort of 3D fashion.

Facebook first introduced this for some recent phones a few months ago -- phones that have extra camera units embedded that give additional parallax information and permit the construction of depth maps.

https://techcrunch.com/2018/06/07/how-facebooks-new-3d-photos-work/

 Mostly the multi camera/depth map technology on phones is used for shallow depth of focus effects for portraits but these newer "3d" animation style effects are becoming more common, for phones and for desktops.On phones the effect can play back when you tilt the phone up and down or side to side as well as respond to swiping.

More recently  Facebook lets anyone upload a pair of images -- the source image and a depth image with the same name as the source image except with the suffix "_depth" -- and Facebook will calculate and deliver the 3d animation effect in a few seconds.
https://www.oculus.com/blog/introducing-new-features-for-3d-photos-on-facebook/

So what is this "depth map" you speak of? Imagine you are a monochromatic creature, an achromatope:
https://en.wikipedia.org/wiki/Monochromacy
And you are beset with a thick blackish fog. Things will have normal sorts of tones up close, but in the distance everything merges into a black haze.The effect is used by a few CG-based artists- often for gloomy, ethereal themes eg. Kazuki Takamatsu:
https://metropolisjapan.com/wp-content/uploads/2017/11/we-ride-our-decoration-WEB-860x480.jpg
Or the depth encoding can be reversed,with the close things darker, more distant white.

Here is my depth map for the gardens scene above -- with white close (the Facebook 3d Photo convention)

The accuracy of the depth encoding in a depth map depends on the number of bits available, so a 32 bit depth image could have more depth information than an 8 bit depth image. Sometimes color depth maps are used as the extra color channels provide more data room as well as being visually informative.

Acquisition of depth maps:  People with normal vision can discriminate close scenes depth-wise with great precision. According to some evolutionary scientists it is all to do with our fruit hunting tree dwelling ancestors:


But good depth maps are difficult at present. We can see errors in depth map stereo visualisations very clearly, particularly now in virtual reality headsets, but currently this does not translate into a very intuitive feedback loop for better depth map creation and correction.

.... work in progress from this point on in this post:
Depth-aware stereo image editing method ...  (Google)
https://patents.google.com/patent/US20140368506

Depth-aware coherent line drawings
http://juliankratt.info/depth_line_drawing

Depth-aware neural style transfer
https://feedforward.github.io/blog/depth-aware/

Interactive Depth-Aware Effects for Stereo Image Editing
https://scholarsarchive.byu.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=4711&context=etd

Handheld HDR VR180 stereo snapshots -- exposure alignment and deghosting

First some summary notes and links on new AI desktop imaging apps-- and then a detailed discussion of HDR workflow with Aurora HDR and its AI aided exposure blending and deghosting. And then workflow notes on stereo fisheye to aligned equirectangulars (PTGui), vertical disparity corrected (After Effects). depth equalized VR180 (Evil Twin 3D) VR 180 stereo pairs. .

AI desktop imaging
A considerable number of AI plugins and apps for masking, HDR exposure merging, noise reduction, sharpening and resizing, scene analysis and object detection and tracking and depth map extraction have appeared recently:

 eg. Topaz's AI Clear, AI Sharpen, AI Resize, and OnOne's ONI Photo Raw  and Skylum's Luminar and Aurora HDR.

https://photofocus.com/software/why-im-loving-topaz-studio-a-i-clear/
https://petapixel.com/2019/03/04/review-topaz-sharpen-ai-is-amazing/
https://skylum.com/blog/ai-sky-enhancer-a-breakthrough
https://skylum.com/newsroom/aurora-hdr-2019-introduces-aipowered-quantum-hdr-engine
https://www.on1.com/blog/new-ai-quick-mask-tool-on1-photo-raw-2019/

and ...
Matterport has AI powered photogrammetry cloud processing for 360 2D panorama scene capture of interiors. Autodesk's Flame 2020 has added AI-aided human face modeling from 2D video and scene segmentation for depth mapping. Kognak has impressive AI masking of  a large variety of video sequences via scene analysis and  object detection. Arraiy enables realtime AI aided virtual set camera and object tracking.

http://360rumors.com/2019/01/matterport-insta360-ricoh-theta.html
http://www.cgchannel.com/2019/04/autodesk-unveils-flame-2020/
https://www.fxguide.com/quicktakes/flame-embraces-deep-learning/
https://kognat.com/
https://www.fxguide.com/quicktakes/rotobot-bringing-machine-learning-to-roto/
https://www.arraiy.com/
http://vfxvoice.com/the-new-artificial-intelligence-frontier-of-vfx/

AuroraHDR
Some (very technical) notes to self. If you have 3 exposures say, you can choose one to be the reference for deghosting purposes. That exposure will not be changed scale-wise in the blended result. Say the exposures are +2, 0 and -2EV. And -2EV is the deghosting reference. Then other possible blends eg, +2 and -2 and 0 and -2 will similarly be the same dimensionwise as the 3 exposure combo. But with slightly different croppings. By canvas sizing them to the same dimensions you will now have 4 images, the original EV (-2) image,  and the 3 possible exposure blends (-2, 0, 2) (-2, 0) and (2, -2) that are very closely aligned. All can be very exactly aligned  by using the AutoAlign feature of Photoshop as layers, one or two times, -- using the choose align method automatically option. So you have plenty of options for compositng different parts of the blends or original deghost reference exposure.

to be continued ...