Using the NV_blend_equation_advanced extension, your OpenGL programs can use the standard set of blend modes used by 2D compositing applications. NV_path_rendering makes path rendering with blend modes easy. These extensions work on all Fermi- and Kepler-based NVIDIA GPUs.
4. Goal for Accelerating Blend Modes
• Market motivation: 2D, compositing, and path
rendering standards key to smart phones,
tablets, and similar devices
– Motivation is primarily for low-end, power-constrained
devices
• Also part of content creation
– Autodesk Mudbox, Adobe Illustrator, etc. all use blend
modes as their vocabulary for compositing
• Power-efficient hardware support for “blend
modes”
5. Standards Reliant on Blend Modes
Document
Printing and
Exchange
Immersive
Web
Experience
2D Graphics
Programming
Interfaces
Context
Creation
Applications
Java 2D
API
Adobe Illustrator
Flash
QtGui
API
Scalable
Vector
Graphics
Mac OS X
2D API
Open XML
Paper (XPS)
HTML 5
Khronos API
Inkscape
Open Source
10. Blend Modes are Part of
Vocabulary of Digital Media
• Example: $200 course on mastering blend modes
11. Artists have Blend Modes
at their Fingertips
• Literally…
Adobe
Creative
Suite
shortcuts
12. Why the Weird Names?
• A few blend modes have weird names
– Color Dodge
– Color Burn, etc.
• These come from photography
Antiquated process of
dodging while
developing photos
Blend modes provide
“digital” version of same effect
13. Observations
• Blend mode standards decouple blend mode from
fragment coloring
– Jargon of these standards
• Fragment coloring = “paint mode”
– Examples: radial color ramp, linear color ramp, constant color, image
gradient
• Blending = “blend mode”
– Examples: src-over, color-dodge, soft-light, src-atop
– Motivates keeping blend mode & fragment shaders distinct
• Compositing lacks need for multi-render targets
– Multiple render targets (MRT) is a 3D concept primarily for
deferred rendering
– Compositing standards have relatively cheap fragment coloring
+ transparency is common (opaque hiding is rare) so deferred
rendering isn’t applicable
– Comparable concept in blend mode standards is layering, but
layering is an explicit concept
14. Paint independent from
Blend Mode
• OpenGL decouples
• Blend mode
blending from shading
standards also
decouple
Primitive
Rasterization
Path
Rasterization
Fragment
Shading
Paint
Blending
Blend Mode
Paint &
blend mode
are independently
specified!
15. What is a Blend Mode?
• Two inputs
– RGBA color A
– RGBA color B
• One result
RGBA
RGBA
RGBA
Blend Mode
– RGBA color result
• Essentially a function
RGBA
– result = blendMode(colorA, colorB)
• More can be said about the nature of this
blendMode function…
16. Pre-multiplied Alpha
• Blend modes assume pre-multiplied alpha
– Pre-requisite for associative blending
• OpenGL users often naïve about this
– Often use “straight alpha” wrongly
– Examples
• RGBA colors texture filter incorrectly unless stored in pre-multiplied
alpha
• Use GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA for glBlendFunc
– Instead proper mode is GL_ONE,GL_ONE_MINUS_SRC_ALPHA
– Otherwise destination alpha is useless for further compositing
• Major contribution of Porter-Duff paper: pre-multiplied alpha
– 2D compositing people know this and only use pre-multiplied alpha
– Blend modes make sense in terms of pre-multiplied alpha
Pre-multiplied alpha: ( A×R,A×G,A×B,A )
Straight alpha: ( R,G,B,A )
GOOD
BAD
17. Blend Mode Associativity
•
Associativity is useful & common
– srcOver(A, srcOver(B,C)) = srcOver(srcOver(A,B), C)
– srcOver(X,Y) function is Xca + (1-Xa)×Yca
– Notation, assuming pre-multiplied alpha
• Xc = color component (R, G, or B) of X
• Xa = alpha component of X
• Xca = Xc×Xa
•
identical
expansion
•
Math proof
– srcOver(A, srcOver(B,C)
= Aca + (1-Aa)×(Bca+(1-Ba)×Cca)
= Ac×Aa+Bc×Ba+Cc×Ca-Cc×Ca×Ba-Aa×Bc×Ba-Cc×Ca×Aa+Cc×Ca×Ba×Aa
– srcOver(srcOver(A,B), C)
= (Aca+(1-Aa)×Bca)+(1-(Aa+(1-Aa)×Ba))×Cca
= Ac×Aa+Bc×Ba+Cc×Ca-Cc×Ca×Ba-Aa×Bc×Ba-Cc×Ca×Aa+Cc×Ca×Ba×Aa
Interpretation
– “over” compositing of A(BC) is same as (AB)C
– Not all blend modes are associative, but many are
18. Blend Mode Structure
• How is a blend mode structured?
• When color A and B composite on a pixel, there
are four regions
–
–
–
–
A∩B
A ∩ ~B
~A ∩ B
~A ∩ ~B
• Each region has a percentage of coverage
– Notation: cover(X) means % of region X covering the pixel
– 100% = cover(A ∩ B)+cover(A ∩ ~B)+cover(~A ∩ B)+cover(~A ∩ ~B)
• “Whole is the sum of its parts”
– Color contributions should be weighted by coverage percentage
19. Intra-pixel Regions for Compositing
A∩B
A ∩ ~B
~A ∩ B
~A ∩ ~B
Source: SVG Compositing Specification
20. Blend Model Internals
•
Assume uncorrelated overlap within a pixel
– Based on conditional probability
– (We will change this uncorrelated overlap assumption later…)
•
Analytical coverage of each of the 4 regions
–
–
–
–
–
cover(A ∩ B) = Aa×Ba
cover(A ∩ ~B) = Aa×(1-Ba)
cover(~A ∩ B) = (1-Aa)×Ba
cover(~A ∩ ~B) = (1-Aa)×(1-Ba)
Confirm sums to 100%
• Yep, Aa×Ba+Aa×(1-Ba) + (1-Aa)×Ba+(1-Aa)×(1-Ba) = 100%
•
Included color of A ∩ ~B and ~A ∩ B regions is fairly obvious
– color(A ∩ ~B) = Ac
• because this is the region of A that’s not overlapped with B, its color is obviously the
color A (no mixing with B occurs)
– color(~A ∩ B) = Bc
• similarly, because this is the region of B that’s not overlapped with A, its color is
obviously the color B (no mixing with A occurs)
•
Trivial when ~A ∩ ~B
– color(~A ∩ ~B) = (0,0,0,0) since no color here, treat as 0% opaque, no color
•
The interesting region is A ∩ B — this the region where A and B are mixing
22. (f,X,Y,Z) Notation for Blend Modes
• Each blend mode can be specified by quadruple
of state: f, X, Y, and Z
– f is arbitrary function of input colors
– X, Y, and Z are either zero or one
• Together make RGBA result
R
G
B
A
= f(Rs',Rd')*p0(As,Ad) + Y*Rs'*p1(As,Ad) +
= f(Gs',Gd')*p0(As,Ad) + Y*Gs'*p1(As,Ad) +
= f(Bs',Bd')*p0(As,Ad) + Y*Bs'*p1(As,Ad) +
=
X*p0(As,Ad) + Y*
p1(As,Ad) +
Z*Rd'*p2(As,Ad)
Z*Gd'*p2(As,Ad)
Z*Bd'*p2(As,Ad)
Z*
p2(As,Ad)
where p0, p1, and p2 are overlap weighting functions
– Always between 0% and 100%
25. Mixing color(A ∩ B) into Complete
Blend Mode Function
• Mixed region is color(A ∩ B)
• Described by a function f(Ac,Bc)
–
–
–
–
3-component vector function
inputs are two RGB color
output is a mixed RGB color
Mixing means “mixing RGB colors” (alpha isn’t involve)
• Function f has an intuitive mean
– Example: src-over’s f(Ac,Bc) is simply Ac
• Template blend mode function (uncorrelated):
result
= f(Ac,Bc)×Aa×Ba + Ac×Aa×(1-Ba) + Bc×(1-Aa)×Ba
• Example: Substitute template for src-cover case when
f(Ac,Bc)=Ac
result
= Ac×Aa×Ba + Ac×Aa×(1-Ba) + Bc×(1-Aa)×Ba
= Ac×Aa + (1-Aa)×Bc×Ba
= Aca + (1-Aa)×Bca
26. Selecting Coverage Contributions
•
Classic Porter & Duff “Compositing Digital Images” paper
– Says how A and B composite
•
Supports idea of constraining contributions
– Example: A src-in B means “portion of A that is inside B’s region”
• Not interested in contributions from A∩~B and ~A∩B regions
• So zero these contributions
• But src-in’s f(Sc,Dc) function is Sc (just like src-over)
•
Selectable template blend mode template equation:
Rca = f(Ac,Bc)×Aa×Ba + Y×Ac×Aa×(1-Ba) + Z×Bc×(1-Aa)×Ba
Ra = X×Aa×Ba + Y×Aa×(1-Ba) + Z×(1-Aa)×Ba
– Notice the X, Y, and Z selection terms
– X, Y, and Z are either one (meaning include) or zero (meaning exclude)
• Note: X is only used in the alpha version
•
Value of (X,Y,Z) for src-in is (1,0,0), expansion becomes
– Rca
– Ra
= Ac×Aa×Ba + 0%×Ac×Aa×(1-Ba) + 0%×Bc×(1-Aa)×Ba
= Aca×Ba + 0% + 0%
= Aca×Ba
= X×Aa×Ba + Y×Aa×(1-Ba) + Z×(1-Aa)×Ba
= Aa×Ba
30. Porter & Duff Modes Expanded
Operation
f(Ac,Bc)
X
Y
Z
Blend mode
Clear
0
0
0
0
0
Src
Ac
1
1
0
Aca
Dst
Bc
1
0
1
Bca
Src-Over
Ac
1
1
1
Aca+(1-Aa)×Bca
Dst-Over
Bc
1
1
1
Bca+(1-Ba)×Aca
Src-In
Ac
1
0
0
Aca×Ba
Dst-In
Bc
0
1
0
Bca×Aa
Src-out
0
0
1
0
(1-Ba)×Aca
Dst-out
0
0
0
1
(1-Aa)×Bca
Src-atop
Ac
1
0
1
Aca×Ba+(1-Aa)×Bca
Dst-atop
Bc
1
1
0
(1-Ba)×Aca+Aa×Bca
Xor
0
0
1
1
Aca×(1-Ba)+(1-Aa)×Bca
Uncorrelated blend mode expansion of Porter & Duff blend modes
31. Other Blend Overlap Modes
• Uncorrelated is one assumption for how two colors
overlap with a pixel
– Typically treated as conditional probability
– Not the only assumption!
– Ideally, you’d know the exact region within the pixel that colors A
and B cover
• Don’t generally have this knowledge of sub-pixel geometry
• Other reasonable assumptions possible
– Disjoint – means colors A and B avoid overlapping to the
greatest extent possible
• Proper assumption when geometry of A and B is result of a nonoverlapping tessellation
– Conjoint – means colors B overlaps A to the greatest extent
possible
• Proper assumption when geometry of B is known to be “within” the
geometry of A
• Example: smiley face is a yellow circle with black eyes and mouth;
you know the mouth and eyes are fully within the yellow circle
• These assumptions result in different coverage functions
than uncorrelated overlap assumption
32. Example of Overlap Assumption for
Src-Over Blend Mode
Source: “A Realistic 2D Drawing System”
rejected SIGGRAPH 2003(?) paper on Xre before being renamed Cairo
35. Resulting Selectable Template
Blend Mode Equations
Overlap
mode
Selectable template blend mode equation
Uncorrelated
Rca = f(Ac,Bc)×Aa×Ba + Y×Aca×(1-Ba) + Z×Bca×(1-Aa)
Ra = X×Aa×Ba + Y×Aa×(1-Ba) + Z×(1-Aa)×Ba
Disjoint
Rca = f(Ac,Bc)×max(Aa+Ba-1,0) + Y×Ac×min(Aa,1-Ba) + Z×Bc×min(Ba,1-Aa)
Ra = X×max(Aa+Ba-1,0) + Y×min(Aa,1-Ba) + Z×min(Ba,1-Aa)
Ra = Aa+Ba assuming (X,Y,Z) = (0,1,1)
Conjoint
Rca = X×f(Ac,Bc)×max(Aa+Ba-1,0) + Y×Aca + Z×Bc×(max(Ba,Aa)-Sa)
Ra = X×max(Aa+Ba-1,0) + Y×Aa + Z×(max(Ba,Aa)-Aa)
Ra = Aa
Further assumptions:
Disjoint: Aa + Ba ≤ 1
Conjoint: Aa ≥ Ba, Aa ≤ 1
36. Prior Work Developing
Blend Modes
•
Ed Catmull & Alvy Ray Smith invent “alpha component” to accompany RGB (1970s)
–
Called this “integral alpha”
•
•
Tom Porter & Tom Duff develop alpha-based image compositing algebra (1984)
–
•
based on pre-multiplied alpha
Adobe publishes “Transparency in PDF” Technical Note #5407 (2000)
–
–
–
PDF 1.4 includes transparency support
Adobe Illustrator 9 supports PDF 1.4 transparency
Introduces blend modes
•
–
•
Concept incorporated in Pixman / XRender APIs
Flash 8 introduces blend modes (2005)
Adobe’s PixelBender language (2008?) provides fully programmable blending
–
•
Detailed compositing theory developed
Adobe Flash avoids conflation artifacts in blending
–
•
Includes separable & non-separable blend modes
Carl Worth and Keith Packard develop Xr (later Cairo) and explain correlated, disjoint,
and conjoint overlap modes (2002?)
–
•
(alpha integrated with the RGB components, unrelated to integers)
Overly general unfortunately
SVG Compositing specification (2009) proposed (f,X,Y,Z) notion to describe blend
modes
–
Includes layers
37. Advanced Blend Modes
• Adobe for Illustrator introduces blend
modes such as
– ColorDodge, ColorBurn, HardLight, SoftLight,
etc.
– These modes are well-known to digital artists
who intuit their meaning & effect
• In (f,X,Y,Z) notation, (X,Y,Z)=(1,1,1)
• It’s the f function that makes these modes
interesting
39. Still missing some blend modes
• Additional blend modes key to implementing ISO
32000 standard
– Also known as PDF
– Blend modes also in Photoshop, etc.
41. Non-Separable Blend Modes
• Non-separable means color components are
processed differently (so not vector RGB math)
– Color information “crosses” components
• General usage
– Artists works in intuitive Hue, Saturation, Luminance
(HSL) color space
– Wants to get hue from color A but retain color B’s
saturation and luminance
– Not really about compositing (no sense of how the
colors overlap); more about color intuition
• Specific modes
– Hue, Saturation, Luminance, Color
42. Additional Blend Modes
• Some blend modes exist in standards…
– But aren’t expressed in (f,X,Y,Z) notation
– Nor are HSL color space blend modes
• Come from a variety of sources
–
–
–
–
JavaFX
OpenVG
Apple’s Quartz 2D
PhotoShop
• For completeness, these should be supported
– Goal: Make OpenGL a blend mode superset
44. Fragment-rate, Sample-rate, and sRGB
Fragment
Shader
per-fragment processing
linear RGBA source color
per-sample processing (2x, 4x, or 8x fragment rate)
linear RGB destination color
Blend Mode
Blend Mode
Blend Mode
Blend Mode
linear RGB result color
Decode to
Decode to
sRGB
Decode to
sRGB
Decode to
sRGB
sRGB
Encode to
Encode to
sRGB
Encode to
sRGB
Encode to
sRGB
sRGB
sRGB encoded result color
sRGB encoded destination color
Framebuffer
45. OpenGL Support Today
• NV_blend_equation_advanced
– Directly supports standard blend modes efficiently
– Works like standard glBlendEquation, just supporting
the standard blend modes
• Better than frustrating fragment interlock
extensions
– Blend mode should be orthogonal to fragment shader!
– Blend equations more power efficient
• Work better with multisampling, color compression, and
blend optimizations
• Extension support today!
– All Fermi- and Kepler-based NVIDIA GPUs
– Also for Logan-based Tegra
– Introduced with OpenGL 4.4 (Release 326 on)
46. NV_blend_equation_advanced
Example Usage
• Very easy
– glEnable(GL_BLEND);
– glBlendEquation(GL_COLOR_DODGE_NV);
• Yes, it’s that easy *
* However for older hardware, call
glBlendBarrierNV() between rendering of
primitives that double hit color samples
– New hardware: “just works” with blend barriers
• glBlendBarrierNV is simply a no-operation
47. Complete NV_blend_equation_advanced
Blend Mode List (1 of 2)
Blending Equation
----------------GL_ZERO
GL_SRC_NV
GL_DST_NV
GL_SRC_OVER_NV
GL_DST_OVER_NV
GL_SRC_IN_NV
GL_DST_IN_NV
GL_SRC_OUT_NV
GL_DST_OUT_NV
GL_SRC_ATOP_NV
GL_DST_ATOP_NV
GL_XOR_NV
GL_MULTIPLY_NV
GL_SCREEN_NV
GL_OVERLAY_NV
GL_DARKEN_NV
GL_LIGHTEN_NV
GL_COLORDODGE_NV
GL_COLORBURN_NV
GL_HARDLIGHT_NV
GL_SOFTLIGHT_NV
GL_DIFFERENCE_NV
GL_EXCLUSION_NV
OpenVG SVG
PDF Flash JavaFX Quartz2D Qt
XRE
------ ----- ----- ----- ------ -------- ---- ----E
X
X
X
XO
X
X
X
X
X
XO
E
X
X
XO
X
X
X
X
X
X
X
XO
X
X
X
X
XO
X
X
X
X
XO
X
X
X
X
X
XO
E
X
X
X
XO
E
X
X
X
X
XO
E
X
X
X
X
XO
E
X
X
X
XO
E
X
X
X
XO
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
E
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
E
X
X
X
X
X
X
E
X
X
X
X
X
X
E
X
X
X
X
X
X
X
E
X
X
X
X
X
X
E
X
X
X
X
X
X
X
E
X
X
X
X
X
X
X = Supported, E = OpenVG extension, X0 = includes disjoint & conjoint
48. Complete NV_blend_equation_advanced
Blend Mode List (1 of 2)
Blending Equation
OpenVG SVG
PDF Flash JavaFX Quartz2D Qt
XRE
----------------------- ------ ----- ----- ----- ------ -------- ---- ----GL_INVERT
X
GL_INVERT_RGB_NV
GL_LINEARDODGE_NV
E
GL_LINEARBURN_NV
E
GL_VIVIDLIGHT_NV
E
GL_LINEARLIGHT_NV
E
GL_PINLIGHT_NV
E
GL_HARDMIX_NV
E
GL_HSL_HUE_NV
X
X
X
GL_HSL_SATURATION_NV
X
X
X
GL_HSL_COLOR_NV
X
X
X
GL_HSL_LUMINOSITY_NV
X
X
X
GL_PLUS_NV /
GL_PLUS_CLAMPED_NV /
X
X
X
X
X
X
X
GL_PLUS_CLAMPED_ALPHA_NV
GL_PLUS_DARKER_NV
X
GL_MINUS_NV /
E
X
GL_MINUS_CLAMPED_NV
GL_CONTRAST_NV
GL_INVERT_OVG_NV
E
GL_RED_NV
X
GL_GREEN_NV
X
GL_BLUE_NV
X
-
X = Supported, E = OpenVG extension, X0 = includes disjoint & conjoint
49. What is NV_path_rendering?
• Another very powerful
OpenGL extension
– “Stencil, then Cover” (StC)
approach to GPUaccelerated path rendering
– Fastest path rendering
available
• Supports all manner of
2D vector graphics
– Not normally the domain of
OpenGL but now is!
https://developer.nvidia.com/gpu-accelerated-path-rendering
50. NV_path_rendering Interaction
• Works great with NV_blend_equation_advanced
– No need for explicit glBlendBarrierNV commands
when rendering paths with “stencil, then cover”
– Blend modes “just work” on old and new GPUs
• When you “stencil” your path, then “cover” that
automatically provides the blend barrier
– Makes using advanced Blend Modes with
NV_path_rendering super-easy
• Example filling a path with “color burn”:
glBlendEquation(GL_COLORBURN_NV);
glStencilFillPathNV(path_object, GL_COUNT_UP_NV, 0xFF);
glCoverFillPathNV(path_object, GL_CONVEX_HULL_NV);
52. Conclusions
• Blend modes have good working theory
• Used in nearly all Digital Media apps and
standards
– Part of the common vocabulary of digital
artists now
• NVIDIA OpenGL provides these modes
– Both current and future hardware!
– Works hand-in-glove with NV_path_rendering
53. Thanks
• Pat Brown (extension specification
author), Jeff Bolz, Daniel Koch, Tristan
Lorach
– NVIDIA
• Rik Cabanier
– Adobe