Monocular view synthesis asks a model to turn one ordinary RGB photograph into nearby views of the same scene. The hard part is that a single image does not uniquely determine 3D structure: many possible scenes can project to the same pixels. Traditional photogrammetry and many neural rendering pipelines avoid that ambiguity by using multiple posed images, but that makes them unsuitable for interactive conversion of existing photo collections.
SHARP, short for Single-image High-Accuracy Real-time Parallax, demonstrates a feedforward alternative. Given one photograph, it predicts a metric 3D Gaussian representation in one neural-network pass. That representation can then be rendered in real time from nearby camera poses, giving the viewer parallax and depth without per-scene optimization. The published system reports under-one-second synthesis on a standard GPU and more than 100 frames per second for nearby-view rendering.
The key engineering idea is to separate representation generation from rendering. The expensive neural model runs once to regress millions of Gaussian attributes: position, scale, orientation, color, and opacity. After that, a Gaussian splatting renderer handles viewpoint changes efficiently. This is different from diffusion-based view synthesis, where every new view may require an iterative generation process and where temporal or geometric consistency is harder to guarantee.
For AI engineers, the pattern matters beyond this one paper. Many multimodal systems need to convert unstructured perceptual input into an explicit, reusable state representation. SHARP shows one way to do that: combine a pretrained depth backbone, task-specific geometric heads, differentiable rendering losses, and artifact regularizers so the model learns a renderable scene representation rather than just producing another image.
Novel view synthesis generates images of a scene from viewpoints that were not directly captured. Multi-view methods use several posed images to reconstruct geometry and appearance. Single-image methods must infer both visible surfaces and plausible hidden structure from learned priors, making them more ambiguous but much easier to deploy on existing photos.
SHARP narrows the target problem deliberately. It is designed for nearby views, such as natural head motion around a captured photo, not arbitrary exploration far behind occluders. That choice makes the task trainable and useful: the model can preserve photographic sharpness around the input viewpoint while offering enough 3D structure for parallax.
A 3D Gaussian scene represents geometry and appearance as many soft ellipsoidal primitives rather than as a mesh or an implicit neural field. Each primitive carries a 3D mean, scale, orientation, color, and opacity. Rendering projects these Gaussians into the image plane and alpha-composites their contributions in visibility order.
Gaussian splatting is attractive for interactive systems because the scene state is explicit and GPU-friendly. Once the primitives exist, rendering a new view does not require querying a large neural network at every sample point along every camera ray. SHARP uses this property by predicting the Gaussian attributes directly from a single input image.
Monocular depth estimation predicts depth from one image. Metric depth goes further by estimating absolute scale, not just relative near/far ordering. For AR, VR, camera motion, and physical display coupling, metric scale matters: a virtual camera movement of ten centimeters should correspond to the same scale in the reconstructed scene.
SHARP builds on a pretrained Depth Pro backbone, then adapts parts of that depth pipeline for view synthesis. The paper emphasizes that depth is not a fixed truth for this task. Transparent surfaces, reflections, and ambiguous geometry can make a standard monocular depth map harmful unless the downstream view-synthesis objective can reshape it.
The transferable pattern is to make the model emit an intermediate representation with operational semantics. A rendered image is an output; a set of 3D Gaussians is a state that another system can inspect, transform, render from multiple views, cache, stream, or edit.
The network starts with a single RGB image and passes it through a pretrained Depth Pro-style feature encoder. SHARP then uses a modified depth decoder that predicts two depth layers rather than one. The first layer represents primary visible surfaces. The second can help model occluded regions, thin structures, or view-dependent effects that become important when the camera moves.
The authors partially unfreeze the depth stack during training. That is an important design choice: if the monodepth model remains frozen, the rest of the system must accept whatever geometry it produces, even when the depth estimate is incompatible with the image-space rendering objective. Allowing selected depth components to adapt lets the model optimize for view synthesis fidelity.
# Conceptual inference pipeline, not the SHARP implementation.
image = load_rgb_image(path)
features = depth_backbone.encode(image)
depth_1, depth_2 = depth_decoder(features)
base_gaussians = initialize_gaussians(
image=image,
depth_layers=[depth_1, depth_2],
coordinate_space="normalized_camera"
)
attribute_deltas = gaussian_decoder(features, image, depth_1, depth_2)
gaussians = compose_gaussian_attributes(base_gaussians, attribute_deltas)
for camera in nearby_camera_trajectory:
frame = gaussian_renderer.render(gaussians, camera)
display(frame)SHARP initializes base Gaussians by downsampling the input image and adjusted depth maps, unprojecting the depth into normalized space, assigning initial color from the image, and setting default rotation and opacity. This creates a geometric scaffold that is already tied to the visible input pixels.
The Gaussian decoder then predicts refinements for every attribute: position, scale, rotation, color, and opacity. This full-attribute refinement is what turns a rough depth-colored point set into a coherent 3D representation. Position deltas can fix geometry, scale and rotation can align splats to surfaces, color deltas can account for view synthesis artifacts, and opacity can suppress or reveal primitives during compositing.
The paper introduces a learned depth adjustment module inspired by the posterior path in conditional variational autoencoders. During training, this module can use supervision to produce a scale adjustment that resolves monocular depth ambiguity. The adjustment is regularized so it does not become an arbitrary information leak.
At inference time, the depth adjustment path is replaced by identity behavior. This matters operationally: the deployed model still takes only one photograph, but training had a mechanism for teaching the model how depth ambiguities affect view synthesis. The result is a network that is less brittle when the raw depth estimate is not the best geometric explanation for nearby views.
Training supervises the predicted Gaussians in image space. For each training pair, the network predicts Gaussians from an input view, renders both the input and a novel view, and computes losses on the renderings. The loss mix includes pixel reconstruction, perceptual terms for image fidelity, alpha penalties to discourage spurious transparency, depth terms, and regularizers that suppress floaters and extreme Gaussian offsets.
This is a general recipe for representation learning with a renderer in the loop. Instead of directly supervising every primitive attribute, the system supervises the observable consequence of those attributes: the images produced from camera viewpoints. The renderer becomes the differentiable bridge between geometric state and pixel-level training signal.
The most direct application is converting existing photos into shallow 3D scenes for interactive browsing. A viewer can move a phone or headset slightly and see realistic parallax without waiting for per-photo optimization. The method is especially suited to scenes where the desired motion stays near the original camera viewpoint.
Because SHARP predicts metric representations with absolute scale, the output can be coupled to physical camera movements. That makes the technique relevant for AR and VR systems where a synthesized scene must respond consistently to headset pose changes. A non-metric representation could look plausible in isolation but drift or scale incorrectly when attached to a physical device.
A feedforward 3D reconstruction can serve as a conditioning layer for later generative or editing systems. Instead of asking a video model to infer geometry from scratch, a pipeline can first infer a Gaussian scene, render controlled nearby views, and use those views as structured context for animation, editing, or relighting stages.
Single-image view synthesis can create controlled viewpoint variation for vision datasets. The synthesized views should not be treated as ground truth for arbitrary camera moves, but nearby-view augmentation can help train models that need robustness to small camera shifts, display parallax, or pose jitter.
SHARP does not learn monocular geometry from scratch. It starts from a strong depth backbone and adapts it to the downstream rendering objective. This pattern is common in production multimodal systems: use a pretrained model for broad perception, then add task-specific heads and losses that align the representation with the product behavior you actually need.
class ViewSynthesisModel:
def __init__(self, depth_backbone, depth_decoder, gaussian_decoder):
self.depth_backbone = depth_backbone
self.depth_decoder = depth_decoder
self.gaussian_decoder = gaussian_decoder
def trainable_parameter_groups(self):
return [
{"params": self.depth_backbone.low_resolution_encoder()},
{"params": self.depth_decoder.parameters()},
{"params": self.gaussian_decoder.parameters()},
]
def frozen_parameter_groups(self):
return [
{"params": self.depth_backbone.patch_encoder()},
{"params": self.depth_backbone.normalization_layers()},
]The output format should match the renderer’s expectations exactly. SHARP’s released code writes 3D Gaussian splat files, and the README states that these .ply outputs are compatible with public 3D Gaussian renderers. That portability is a design advantage: inference and rendering can evolve separately as long as the representation contract remains stable.
View synthesis systems often fail through visible artifacts: floaters, stretched surfaces, transparency halos, blurry boundaries, or implausible disocclusions. SHARP’s loss configuration includes regularizers aimed at those failure modes. In applied systems, artifact-specific penalties are often as important as the headline reconstruction loss because they encode what users notice.
The depth adjustment design shows a useful training pattern. The model can receive extra signals during training to resolve ambiguity, as long as the deployed inference path does not depend on those signals. This lets the training process teach a compact behavior that the production system can execute from the available input.