## Parametric Approach to OpenGL Using After Effects SDK

This is not something that you’d find in a journal, or a book: this is something I myself have cooked up. But what do I mean by “Parametric Approach to OpenGL”, and by “Using After Effects”?

The above video is an After Effects plugin by yours truly. Everything is based upon one fragment shader (which I got from a pastebin, but it looks like it’s from this repo), which as follows:

#version 330
uniform sampler2D videoTexture;
uniform float sliderVal;
uniform float multiplier16bit;
in vec4 out_pos;
in vec2 out_uvs;
out vec4 colourOut;

#define PI 3.14159265358979323846

float random (in vec2 _st) {
return fract(sin(dot(_st.xy,
vec2(12.9898,78.233)))*
43758.5453123);
}

vec2 truchetPattern(in vec2 _st, in float _index){
_index = fract(((_index-0.5)*2.0));
if (_index > 0.75) {
_st = vec2(1.0) - _st;
} else if (_index > 0.5) {
_st = vec2(1.0-_st.x,_st.y);
} else if (_index > 0.25) {
_st = 1.0-vec2(1.0-_st.x,_st.y);
}
return _st;
}

void main() {
vec2 st = gl_FragCoord.xy/sliderVal;
st /= 10;

vec2 ipos = floor(st);  // integer
vec2 fpos = fract(st);  // fraction

vec2 tile = truchetPattern(fpos, random( ipos ));

float color = 0.0;

color = step(tile.x,tile.y);

gl_FragColor = vec4(vec3(color),1.0);
}



But you may say “so what, what’s so parametric about it?” Before doing anything or saying anything, let’s scrutinize the basis of an After Effects OpenGL plugin.

All AE OpenGL Plugins are based on one example in the SDK: Glator. This example was removed for a large amount of time, maybe 4-5 years, and returned in 2017. Without is, image maninpulation shall be done using Adobe’s own mangled suits, something I wouldn’t wish on my worst enemies. After Effects SDK is based on a few command selectors:

switch (cmd) {
out_data,
params,
output);
break;

case PF_Cmd_GLOBAL_SETUP:
err = GlobalSetup(	in_data,
out_data,
params,
output);
break;

case PF_Cmd_PARAMS_SETUP:
err = ParamsSetup(	in_data,
out_data,
params,
output);
break;

case PF_Cmd_GLOBAL_SETDOWN:
err = GlobalSetdown(	in_data,
out_data,
params,
output);
break;

case  PF_Cmd_SMART_PRE_RENDER:
err = PreRender(in_data, out_data, reinterpret_cast<PF_PreRenderExtra*>(extra));
break;

case  PF_Cmd_SMART_RENDER:
err = SmartRender(in_data, out_data, reinterpret_cast<PF_SmartRenderExtra*>(extra));
break;
}


Which are engulfed in a try...catch statement. You may have noticed some function calls below each command selector. Those are the bread and mead of an After Effects plgi. Our concern is with PreRender() and SmartRender(). PreRender() is not that important:

static PF_Err
PreRender(
PF_InData				*in_data,
PF_OutData				*out_data,
PF_PreRenderExtra		*extra)
{
PF_Err	err = PF_Err_NONE,
err2 = PF_Err_NONE;

PF_ParamDef slider_param;

PF_RenderRequest req = extra->input->output_request;
PF_CheckoutResult in_result;

AEFX_CLR_STRUCT(slider_param);

ERR(PF_CHECKOUT_PARAM(in_data,
GLATOR_SLIDER,
in_data->current_time,
in_data->time_step,
in_data->time_scale,
&amp;slider_param));

ERR(extra->cb->checkout_layer(in_data->effect_ref,
GLATOR_INPUT,
GLATOR_INPUT,
&amp;req,
in_data->current_time,
in_data->time_step,
in_data->time_scale,
&amp;in_result));

if (!err){
UnionLRect(&amp;in_result.result_rect, &amp;extra->output->result_rect);
UnionLRect(&amp;in_result.max_result_rect, &amp;extra->output->max_result_rect);
}
ERR2(PF_CHECKIN_PARAM(in_data, &amp;slider_param));
return err;
}


You can see a call to PF_CHECKOUT_PARAM in the code. “Obtains parameter values, or the source video layer, at a specified time. AfterEffects makes caching decisions based on the checkout state of parameters.”1 This PARAM has been defined before in ParamSetup():

static PF_Err
ParamsSetup (
PF_InData		*in_data,
PF_OutData		*out_data,
PF_ParamDef		*params[],
PF_LayerDef		*output )
{
PF_Err		err		= PF_Err_NONE;
PF_ParamDef	def;

AEFX_CLR_STRUCT(def);

GLATOR_SLIDER_MIN,
GLATOR_SLIDER_MAX,
GLATOR_SLIDER_MIN,
GLATOR_SLIDER_MAX,
GLATOR_SLIDER_DFLT,
SLIDER_DISK_ID);

out_data->num_params = GLATOR_NUM_PARAMS;

return err;
}



You can add as much PARAMs to your program as you wish and later, pass them to your shaders as uniforms. But how? Well, for that, we must see what goes on in SmartRender():

static PF_Err
SmartRender(
PF_InData				*in_data,
PF_OutData				*out_data,
PF_SmartRenderExtra		*extra)
{
PF_Err				err = PF_Err_NONE,
err2 = PF_Err_NONE;

PF_EffectWorld		*input_worldP = NULL,
*output_worldP = NULL;
PF_WorldSuite2		*wsP = NULL;
PF_PixelFormat		format = PF_PixelFormat_INVALID;
PF_FpLong			sliderVal = 0;

AEGP_SuiteHandler suites(in_data->pica_basicP);

PF_ParamDef slider_param;
AEFX_CLR_STRUCT(slider_param);

ERR(PF_CHECKOUT_PARAM(in_data,
GLATOR_SLIDER,
in_data->current_time,
in_data->time_step,
in_data->time_scale,
&amp;slider_param));

if (!err){
sliderVal = slider_param.u.fd.value / 100.0f;
}

ERR((extra->cb->checkout_layer_pixels(in_data->effect_ref, GLATOR_INPUT, &amp;input_worldP)));

ERR(extra->cb->checkout_output(in_data->effect_ref, &amp;output_worldP));

ERR(AEFX_AcquireSuite(in_data,
out_data,
kPFWorldSuite,
kPFWorldSuiteVersion2,
(void**)&amp;wsP));

if (!err){
try
{
// always restore back AE's own OGL context
SaveRestoreOGLContext oSavedContext;

// our render specific context (one per thread)
AESDK_OpenGL::AESDK_OpenGL_EffectRenderDataPtr renderContext = GetCurrentRenderContext();

if (!renderContext->mInitialized) {
AESDK_OpenGL_Startup(*renderContext.get(), S_GLator_EffectCommonData.get());

renderContext->mInitialized = true;
}

renderContext->SetPluginContext();

// - Gremedy OpenGL debugger
// - Example of using a OpenGL extension
bool hasGremedy = renderContext->mExtensions.find(gl::GLextension::GL_GREMEDY_frame_terminator) != renderContext->mExtensions.end();

A_long				widthL = input_worldP->width;
A_long				heightL = input_worldP->height;

AESDK_OpenGL_InitResources(*renderContext.get(), widthL, heightL, S_ResourcePath);

CHECK(wsP->PF_GetPixelFormat(input_worldP, &amp;format));

// upload the input world to a texture
size_t pixSize;
gl::GLenum glFmt;
float multiplier16bit;
gl::GLuint inputFrameTexture = UploadTexture(suites, format, input_worldP, output_worldP, in_data, pixSize, glFmt, multiplier16bit);

// Set up the frame-buffer object just like a window.
ReportIfErrorFramebuffer(in_data, out_data);

glViewport(0, 0, widthL, heightL);
glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
glClear(GL_COLOR_BUFFER_BIT);

// - simply blend the texture inside the frame buffer
RenderGL(renderContext, widthL, heightL, inputFrameTexture, sliderVal, multiplier16bit);

// - we toggle PBO textures (we use the PBO we just created as an input)
ReportIfErrorFramebuffer(in_data, out_data);

glClear(GL_COLOR_BUFFER_BIT);

// swizzle using the previous output
SwizzleGL(renderContext, widthL, heightL, renderContext->mOutputFrameTexture, multiplier16bit);

if (hasGremedy) {
gl::glFrameTerminatorGREMEDY();
}

// - get back to CPU the result, and inside the output world
format, pixSize, glFmt);

glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBindTexture(GL_TEXTURE_2D, 0);
glDeleteTextures(1, &amp;inputFrameTexture);
}
catch (PF_Err&amp; thrown_err)
{
err = thrown_err;
}
catch (...)
{
err = PF_Err_OUT_OF_MEMORY;
}
}

// If you have PF_ABORT or PF_PROG higher up, you must set
// the AE context back before calling them, and then take it back again
// if you want to call some more OpenGL.
ERR(PF_ABORT(in_data));

ERR2(AEFX_ReleaseSuite(in_data,
out_data,
kPFWorldSuite,
kPFWorldSuiteVersion2,
"Couldn't release suite."));
ERR2(PF_CHECKIN_PARAM(in_data, &amp;slider_param));
ERR2(extra->cb->checkin_layer_pixels(in_data->effect_ref, GLATOR_INPUT));

return err;
}


“But Chubak, where do we pass the uniforms?” Patience, Constance dear Patience. We pass them in RenderGL():

void RenderGL(const AESDK_OpenGL::AESDK_OpenGL_EffectRenderDataPtr&amp; renderContext,
A_long widthL, A_long heightL,
gl::GLuint		inputFrameTexture,
PF_FpLong			sliderVal,
float				multiplier16bit)
{
// - make sure we blend correctly inside the framebuffer
// - even though we just cleared it, another effect may want to first
// draw some kind of background to blend with
glEnable(GL_BLEND);
glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);

// view matrix, mimic windows coordinates
vmath::Matrix4 ModelviewProjection = vmath::Matrix4::translation(vmath::Vector3(-1.0f, -1.0f, 0.0f)) *
vmath::Matrix4::scale(vmath::Vector3(2.0 / float(widthL), 2.0 / float(heightL), 1.0f));

glBindTexture(GL_TEXTURE_2D, inputFrameTexture);

glUseProgram(renderContext->mProgramObjSu);

// program uniforms
GLint location = glGetUniformLocation(renderContext->mProgramObjSu, "ModelviewProjection");
glUniformMatrix4fv(location, 1, GL_FALSE, (GLfloat*)&amp;ModelviewProjection);
location = glGetUniformLocation(renderContext->mProgramObjSu, "sliderVal");
glUniform1f(location, sliderVal);
location = glGetUniformLocation(renderContext->mProgramObjSu, "multiplier16bit");
glUniform1f(location, multiplier16bit);

// Identify the texture to use and bind it to texture unit 0
AESDK_OpenGL_BindTextureToTarget(renderContext->mProgramObjSu, inputFrameTexture, std::string("videoTexture"));

// render
glBindVertexArray(renderContext->vao);
glBindVertexArray(0);

glUseProgram(0);
glDisable(GL_BLEND);
}



RenderGL(), like SwizzleGL(), is from a series of functions defined at the top of the file.

So what does it all amounts to? Go back to the top, to the very first listing, and you’ll see a sliderVal amongst the uniforms. That’s what I mean by Parametric OpenGL. Technically, every OpenGL is Parametric, however, this gives us a slider, or a point, or a value (depending on the type of the PARAM, I recommend reading the SDK manual). Ipso facto, parametric here means “something to mess around with”.

There’s a considerable amount of money in After Effects plugin development, and this is, perhaps, the very first blog post about this SDK. I’m not sure. When I was a kid, I spent my own fair share of the money on After Effects plugins. For a brief period in my life, they were my awe, my life and my livelihood. Make an After Effects Plugin, Make a Kid Happy!

I hope you enjoyed this very short post. I have a brouhaha with /r/EnglishLearning. They believed that my prose is, as my friend Tanami puts it, brash. If you believe so, please tell me so I can do something about it. Thank you.

I’ve found another book to mack on while you mack with your paramour. It’s called Fractal Worlds: Grown, Built and Imagined. I’m going to write a fractal generating software based on it, if I don’t die tomorrow. Chubak Out.

• After Effects SDK Manual

## Credits

Since giving credits is more important than anything else, I will do that first. This article is partially based on  two papers: A Survey of Procedural Noise Functions by A. Lagae et al and Efficient Computational Noise in GLSL by McEwan et al. Codes are sto-, ahem, taken from this gist and the table at the beginning of the post is taken from this article. Quotes from the papers are in blue, and are attributed using footnotes. The cover photo is taken directly from A Survey et al.

## What is Perlin Noise?

Perlin Noise was created by Ken Perlin (who won the Academy Award for his achievement) in 1983. See, back then, photorealism was something to be desired by everyone, but people always came up short. Ken Perlin came up with his noise algorithm to battle this wretched “computer-looking” appearance of 3D models. Noise is the random number generator of computer graphics. It is a random and unstructured pattern, and is useful wherever there is a need for a source of extensive detail that is nevertheless lacking in evident structure.1 Perlin noise is a multi-dimensional algorithm used in procedural generation, textures, terrain generation, map generation, surface generation, vertex generation, and so on and so forth. The following table 2 shows the extent of Perlin noise:

 Dimension Raw Noise (Grayscale) Uses 1 Things such as vectors which look hand-drawn 2 Things such as procedural textures, comme feu 3 Minecraft terrain uses Perlin Noise

Perlin gave the following definition for noise: “..noise is an approximation to white noise band-limited to a single octave.3 The formal definition of the noise is:

Where is the noise function. Perlin noise is a procedural noise.The adjective procedural is used in computer science to distinguish entities that are described by program code rather than by data structures. 4 Meaning, something that is generated, not something that is hard-coded. What’s good about using a procedural noise, rather than say, creating the noise ourselves? Well, a procedural noise is compact, meaning it occupies less space. It’s continuous, meaning it’s non-periodic. It’s parametrized, it’s randomly accessible, and many other goodies that make the job of an artist easier… And isn’t that our end goal? To serve the artist? Yesterday, I made an After Effects plugin, which you can see it below this post. I didn’t have the artist in mind, I had my own ego and id in mind, so nobody downloaded it. I learned my lesson: Serve the artist, not yourself. Anyways…

Before Perlin, came Lattice gradient noises. They are geenrated by interpolating between random values, and Perlin noise uses a Cubic Lattice on each vertex and then doing a splined interpolation. The pseudo-random gradient is given by hashing the lattice point and using the result to choose a gradient. 5 These hashes are turned into 12 vectors, and they are interpolated from center to edge using a quintic polynomial. A little difficult to imagine right? No worries. We’ll provide pictures6 and pseudo-codes 7.

And a pseudo-code for Classic Perlin, sans the hash function:

// Function to linearly interpolate between a0 and a1
// Weight w should be in the range [0.0, 1.0]
float lerp(float a0, float a1, float w) {
return (1.0 - w)*a0 + w*a1;

// as an alternative, this slightly faster equivalent formula can be used:
// return a0 + w*(a1 - a0);
}

// Computes the dot product of the distance and gradient vectors.
float dotGridGradient(int ix, int iy, float x, float y) {

// Precomputed (or otherwise) gradient vectors at each grid node

// Compute the distance vector
float dx = x - (float)ix;
float dy = y - (float)iy;

// Compute the dot-product
}

// Compute Perlin noise at coordinates x, y
float perlin(float x, float y) {

// Determine grid cell coordinates
int x0 = int(x);
int x1 = x0 + 1;
int y0 = int(y);
int y1 = y0 + 1;

// Determine interpolation weights
// Could also use higher order polynomial/s-curve here
float sx = x - (float)x0;
float sy = y - (float)y0;

// Interpolate between grid point gradients
float n0, n1, ix0, ix1, value;
n0 = dotGridGradient(x0, y0, x, y);
n1 = dotGridGradient(x1, y0, x, y);
ix0 = lerp(n0, n1, sx);
n0 = dotGridGradient(x0, y1, x, y);
n1 = dotGridGradient(x1, y1, x, y);
ix1 = lerp(n0, n1, sx);
value = lerp(ix0, ix1, sy);

return value;
}


It is worth knowing that “… All noise functions except Perlin noise and sparse convolution noise are approximately band-pass. Perlin noise is only weakly band-pass, which might lead to problems with aliasing and detail loss.” 8 Plus, Perlin noise does not have Gaussian Amplitude DistributionMeaning, the speckles of noise don’t scatter based on Gaussian function, which is outside the scope of this article. There are many things that Perlin fairs well at, but there are things which it performs weakly at. In the following table plate 9 you can witness for yourself

## Perlin Noise in Practice: GLSL Implementations

Well, I said I will talk about Perlin noise in GLSL, and now I’m going to do it.

You can use Perlin noise as a wave, as a diffuse color, as a diffuse material, as a flickering light, and as some speckles on your texture. I personally used it in this instance as a color fizzler.

As I’m writing this, I’m contemplating an After Effects plugin which adds Perlin Noise functionality.

The simplest Perlin Noise can be construed10 as:

float rand(vec2 c){
return fract(sin(dot(c.xy ,vec2(12.9898,78.233))) * 43758.5453);
}

float noise(vec2 p, float freq ){
float unit = screenWidth/freq;
vec2 ij = floor(p/unit);
vec2 xy = mod(p,unit)/unit;
//xy = 3.*xy*xy-2.*xy*xy*xy;
xy = .5*(1.-cos(PI*xy));
float a = rand((ij+vec2(0.,0.)));
float b = rand((ij+vec2(1.,0.)));
float c = rand((ij+vec2(0.,1.)));
float d = rand((ij+vec2(1.,1.)));
float x1 = mix(a, b, xy.x);
float x2 = mix(c, d, xy.x);
return mix(x1, x2, xy.y);
}

float pNoise(vec2 p, int res){
float persistance = .5;
float n = 0.;
float normK = 0.;
float f = 4.;
float amp = 1.;
int iCount = 0;
for (int i = 0; i<50; i++){
n+=amp*noise(p, f);
f*=2.;
normK+=amp;
amp*=persistance;
if (iCount == res) break;
iCount++;
}
float nf = n/normK;
return nf*nf*nf*nf;
}

#define M_PI 3.14159265358979323846

float rand(vec2 co){return fract(sin(dot(co.xy ,vec2(12.9898,78.233))) * 43758.5453);}
float rand (vec2 co, float l) {return rand(vec2(rand(co), l));}
float rand (vec2 co, float l, float t) {return rand(vec2(rand(co, l), t));}

float perlin(vec2 p, float dim, float time) {
vec2 pos = floor(p * dim);
vec2 posx = pos + vec2(1.0, 0.0);
vec2 posy = pos + vec2(0.0, 1.0);
vec2 posxy = pos + vec2(1.0);

float c = rand(pos, dim, time);
float cx = rand(posx, dim, time);
float cy = rand(posy, dim, time);
float cxy = rand(posxy, dim, time);

vec2 d = fract(p * dim);
d = -0.5 * cos(d * M_PI) + 0.5;

float ccx = mix(c, cx, d.x);
float cycxy = mix(cy, cxy, d.x);
float center = mix(ccx, cycxy, d.y);

return center * 2.0 - 1.0;
}

// p must be normalized!
float perlin(vec2 p, float dim) {

/*vec2 pos = floor(p * dim);
vec2 posx = pos + vec2(1.0, 0.0);
vec2 posy = pos + vec2(0.0, 1.0);
vec2 posxy = pos + vec2(1.0);

// For exclusively black/white noise
/*float c = step(rand(pos, dim), 0.5);
float cx = step(rand(posx, dim), 0.5);
float cy = step(rand(posy, dim), 0.5);
float cxy = step(rand(posxy, dim), 0.5);*/

/*float c = rand(pos, dim);
float cx = rand(posx, dim);
float cy = rand(posy, dim);
float cxy = rand(posxy, dim);

vec2 d = fract(p * dim);
d = -0.5 * cos(d * M_PI) + 0.5;

float ccx = mix(c, cx, d.x);
float cycxy = mix(cy, cxy, d.x);
float center = mix(ccx, cycxy, d.y);

return center * 2.0 - 1.0;*/
return perlin(p, dim, 0.0);
}


That is, however, a version of Perlin which was reconstructed in 2002. Visit the Gist to see how to apply Classic Perlin Noise.

Well, that is it for today! Short post, I know, and it was lacking in original content, but I am running out of ideas as of now, because I haven’t read Real-Time Rendering yet. It’s a book that’s chuckfull of concepts and ideas for learning, and teaching. I love this damn book!

Another book I’m currently reading is Fundamentals of Computer Graphics. I got a little stuck reading about Implicit Curves, but I’m going to ask my cousin, who holds a PhD in mathematics, to help me with it. I hope he does, because he has the tendency to forgoe my requests of intellectual assistance. Dunno, really. Aren’t cousins supposed to help each other? I will write him a visualization app in a minute if he asks… Au fait.

Je vous s’exprime quelque chose. S’il vous plait, se introdure mon blog a votre amis, parce que je veux plus visitors, et je veux se attendre les plus gens. Si ils ne voudraient visiter mon blog, leur dit “C’est genial comme un reine qu’est virgo entacta et veut votre putain!”

Dammit, I suck at French, at least I didn’t use Google Translate. Chubak Out.

## Reference

• A Survey of Procedural Noise Functions, Computer Graphics Forum, Volume 29 (2010), Number 8, pp 2379-2600
• Efficient Computational Noise, Journal of Graphics Tools Volume 16, No. 2: 85-94

## Step-by-Step with OpenGL’s Graphics Pipeline

Good Evening everyone! I hope you’re all doing A-Ok. Yesterday I received a PM:

Hey. Do you know anything about game rendering? I’ve seen your blog you must know something about game rendering.

I think this person is referring to Graphic Pipeline, which different APIs dub it as “rendering application”. “The main function of the pipeline is to generate, or render, a two-dimensional image, given a virtual camera,three-dimensional objects, light sources, and more”. 1 Now, SIMD architecture with its many cores makes it possible for parallel calculations, and it is de facto the main reason we have GPUs today. Graphics Pipeline is made up of shaders. Shaders are parallel programs that live in the GPU. OpenGL has five shader stages, and about seven application stages in its Pipeline 2. But what are t he stages of this pipleline? What does pipleline even mean? Read this article and find out.

## The History of the Word “Pipeline”

The word pipeline became popular when gas lamps were invented. “The first Canadian transmission pipeline was built in 1853. A 25 kilometre cast-iron pipe moving natural gas to Trois Rivières, QC. It was the longest pipeline in the world at the time.” 3

What does pipeline mean? Simple. Imagine you wish to transmit something from one stage, to another stage, then another stage, until the output is unrecognizable from the input. For example, in OpenGL, this can be the input:

float vertices[] = {
// first triangle
0.5f,  0.5f, 0.0f,  // top right
0.5f, -0.5f, 0.0f,  // bottom right
-0.5f,  0.5f, 0.0f,  // top left
// second triangle
0.5f, -0.5f, 0.0f,  // bottom right
-0.5f, -0.5f, 0.0f,  // bottom left
-0.5f,  0.5f, 0.0f   // top left
};


And this can be the output:

Some stages of the OpenGL are mandatory, some stages are optional. You can see this pipeline in the following picture.

Based on two books, The OpenGL Red Book 9th Edition and Real-Time Rendering 4th Edition I wish to explain to you this so-called Pipeline. Turn off your phones, and pay attention!

## 1. Vertex Data

We saw an example of vertices attributes in the first code in this very post. In OpenGL, vertices are recorded in Homogeneous Coordinate System, which is a complicated 4-point coordinate system. I’ll talk about this coordinate system in another post, very soon. Keep in mind that vertices can be hardcoded in the shader, but most of the times, it is passed to the vertex shader as a Vertex Buffer Object.

In Vertex Shader, three matrices are multiplied by the outgoing gl_Position. One is the Model Matrix. the other is the View Matrix, and at the end, we have the Projection Matrix. You can clearly see them in the following picture:

The model matrix deals with the object. The view matrix deals with the world around it. The projection matrix deals with the camera. One day, I’ll talk in detail about various camera projections.

“Imagine you have a bouncing ball object.If you represent it with a single set of triangles, you can run into problems with quality or performance. Your ball may look good from 5 meters away, but up close the individual triangles, especially along the silhouette, become visible. If you make the ball with more triangles to improve quality, you may waste considerable processing time and memory when the ball is far away and covers only a few pixels on the screen.With tessellation, a curved surface can be generated with an appropriate number of triangles.” 4

Tessellation is basically the process of breaking higher order primitives, such as cubes, into many small sub-primitives in order to create a more detailed geometry. In OpenGL, instead of cubes, we use patches. TCS is the first of two Tessellation Shaders, the other one being…

Ones the Tessellation Engine does its job, it produces a number of geometry vertices which are responsible for adding detail to the given patch. Tessellation Evaluation Shader invokes them. This shader runs for each generated vertex, and adds a lot of overhead. That’s why programmers shouldn’t be thrifty with TCS.

Geometry Shader looks like Vertex Shader, and uses gl_Position, but it’s responsible for creating multiple instances of a primitive through EmitVertex() and EndPrimitive(). “Each time we call EmitVertex the vector currently set to gl_Position is added to the primitive. Whenever EndPrimitive is called, all emitted vertices for this primitive are combined into the specified output render primitive. By repeatedly calling EndPrimitive after one or more EmitVertex calls multiple primitives can be generated. This particular case emits two vertices that were translated by a small offset from the original vertex position and then calls EndPrimitive, combining these two vertices into a single line strip of 2 vertices.” 5

### 6. Primitive Assembly

Primitive Assembly is a short stage. It is basically grouping of of primitives into lines and triangles. You may ask, well, what about points? They can’t be assembled? The question to your answer is, that yes, it does happen for points, but it’s redundant.

### 7. Clipping

After gl_Position is converted to Cartesian coordinates, and gets normalized, meaning it is carried out between -1 and 1, it is clipped for screen. Meaning, the primitives which are not in the viewport propogated by the projection matrix are discarded.  This stage is very important for overall performance reasons.

### 8. Culling

This is another step taken for performance. Each 3D primitive has its face to the camera, and its back is out of the view of the camera. So it’s entirely useless to render the back of the primitive also. Culling is the process of discarding the back in favour of the face of each primitive.

### 9. Rasterization

Rasterization is the process of converting 3D objects in computer’s memory to 2D objects to be displayed in the monitor, or rendered to a file. In other words, rasterization is the process of determining which fragments might be covered by a triangle or a line. Rasterization has two stages: trangle setup, and triangle traversal. In the former, “…. edge equations, and other data for the triangle are computed. These data may be used for triangle traversal, as well as for interpolation of the various shading data produced by the geometry stage. Fixed function hardware is used for this task. 6. In the latter, each pixel, or a sample, is covered by a triangle, and a fragment is generated for  each pixel, or sample. If the number of samples are low, aliasing happens. Triangles are interpolated, and pixels are sent to the next stage, which is the most important stage: Fragment shader.

Creme de la creme, it is the last programmable stage in the pipeline, and does operations such as coloring, texturing, shadowing, antialisiang, raytracing (if possible), etc.Perhaps the most important thing done in this stage is texturing.

If you are interested in learning more about fragment shaders and what you can do in them, you can read The Book of Shaders, a free internet book that, although outdated, teaches a lot of tricks in the trade.

## 11. Z-Buffer, Stencil Buffer, and Color Buffer

Z-Buffer is the depth buffer of the program, which is propagated by an FBO. An FBO can also be a stencil buffer, which as the name suggest, creates a black mask aroudn the screen. An FBO can also be a color buffer, which is simply the background color of the program.

## 12. Compute Shaders, and Beyond!

Compute Shaders are a part of GPGPU, or General-Purpose GPU. GPUs, with their SIMD architecture, are the bee’s knees for high-performance computing. So in modern OpenGL, Kronos has provided us with a way to harness the power of GPU using shaders. Compute Shaders use GLSL, and are good for thigns like image processing. However, Compute Shaders are not a part of the Pipeline, and in fact, a Shader Program sannot contain both type of shaders. However, a regular program can contain both.

There are many things to do after Fragment Shader. Such as collision detection, animation, physics, music, etc. But they are not a part of OpenGL.

So that’s it! I hope you’ve enjoyed this post. Since there was a brouhaha last night about me not citing the sources, this time I made sure to cite everyone. People were right, I should have cited. Thanks people!

Please share my post with your interested friends. It goes a long way. Thank you, Chubak Out.

## SMAA: Enhanced Subpixel Morphological Antialiasing

The following article is based on a journal by Jorge Jimenez, Jose J. Echevarria, Tiago Sousa and Diego Gutierrez.

You can view their SMAA demo here. It runs fine on my GTX 960 2GB.

## Old Methods of Antialiasing

For years, MSAA (Multisampling Antialiasing) and SSAA (Supersampling Antialiasing) have been de facto the methods of antialiasing. In fact, these two still retain the highest quality amongst the various modern antialiasing methods. As we know, aliasing is caused by the lack of samples, in spatial level (jagged lines) and in temporal level (flickering), usually around the edges and high/low contrast regions of the picture. To battle these, we have two main ways that were once the only way around, Supersampling, and Multisampling:  In Supersampling, we blow up the picture, then downsampling for the final resolution. It works fine because, as my uncle puts it, it’s a pincer attack, because it covers every basi s of the problem and surrounds it. Multisampling is similarly pincer-ish. In this method, each sample gets duplicated based on the given coefficient. In today’s high resolutions, it would require a rather fiendish graphics card to achieve this. Therefore, we need new methods of antialiasing, both in spatial, and temporal level. All these methods rely on one algorithm to do their job: edge detection algorithm. But they rely on other things as well.

## Modern Ways of Antialiasing

There are many modern filter-based methods for antialiasing which all, although inferior to the former and latter, do their job. FXAA,  DEAA, GPAA, GBAA, CSAA, EQAA, DLAA… In this article, we’ll talk about SMAA, and its predecessor, MLAA. These modern filter-based methods have their own problems:

• Most edge detection methods, which are the basis of these methods, only take into account numerical differences between pixels, ignoring how they appear to the viewer.
• The original shape of the object is not always preserved, an overall rounding of corners is most of the times clearly visible in text, sharp corners, and subpixel features.
• Most approaches are designed to handle horizontal or vertical patterns only, ignoring the vericals.
• Real subpixel features and subpixel motion are not properly handled. Specular and shading aliasing is not completely removed.

You’ve guessed right: We raise these issues because aim to decimate them.

## Morphological Antialiasing (MLAA)

MLAA tries to estimate the coverage of the original geometry. To accurately rasterize an anitalised triangle, the coverage area for each pixel inside the triangle must be calculated to blend it properly with the background. MLAA begins the image without antialiasing, and it reverses the process by re-vectorizing the silhouettes, in order to estimate such coverage areas. Then, since the background cannot be known after rasterization, MLAA blends with a neighbor, assuming that its value is similar with the original background. In other words, The algorithm detects borders (either using color or depth information) and then finds specific patterns in these. Anti-aliasing is achieved by blending pixels in the borders intelligently. MLAA has implementations in DirectX 10 and Mono Game (XNA). Games such as Fable II use it faithfully. From the creators of MLAA, comes SMAA, or Enhanced Subpixel Morphological Antialiasing, which is the main point of our post.

## Enhanced Subpixel Morphological Antialiasing (SMAA)

SMAA offers reliable edge detection, and a simple and effective way to handle sharp geometric features and diagonal lines. Besides, SMAA doesn’t change the shape of the geometry, as many other methods do.

SMAA builds on MLAA pipeline, improving or redefining at every step. In particular, the edge detection is improved by using color information with local contrast adaptation for cleaner edges. It extends the number of patterns handled for preservation of sharp geometric features and diagonals. And lastly, it shows how morphological antialiasing can be accurately combined with multisampling or supersampling and temporal reprojection.

## Edge Detection

Edge detection is vital, because undetected edges remain aliased. On the other hand, too many filtered edges can reduce the quality of the antialiased image. Different information can be used to detect he edge: Chroma, luma, depth, surface normal, or a combination of them. For four reasons, SMAA uses luma:

1.   Less artifacts.
2. Luma is always visible.
3. It can handle shading aliasing.
4. And finally, it’s faster than chroma.

Have this image in mind. Here’s how edge detection works: the final calculated value is a boolean called left edge boundary. Boolean values for the top edge is similarly calculated. The formula is:

All the c values are called contrast deltas.

## Pattern Handling

SMAA pattern detection allows to preserve sharp geometric features like corners, deals with diagonal and enables accurate distance searches.

Sharp Geometric Features: The re-vectorization of silhouettes in MLAA tensd to round corners. To avoid this, SMAA makes the observation that crossing the edges in contour lines have a maximum size of oen pixel, whereas for sharp corners this length will most likely be longer. Thusly, SMAA fetches two-pixel-long crossing edges instead, this allows less aggressive corners processing.

Diagonal Patterns: We introduce a novel diogonal pattern detection. It consists of the following steps:

1. Search for the diagonal distance and to the left and the right of the diagonal lines.
2. Fetch the crossing edges and .
3. Use this input information, defining the specific diagonal patter, to access the precomputed area texture, yielding the areas and and .

If the diagonal pattern detection fails, then the orthogonal detection is triggered.

Accurate Distance Search: Key to pattern detection and classification is obtaining accurate edge distance (lengths to both end of the lien) MLAA makes extensive use of hardware interpolation to speed up this process. Hardware bilinear filtering can be used as a way of fetching and encoding up to four different values with a single memory access. This linear interpolation of two binary values (that is, bilinear) producing a single floating point value is shown as:

Where and are two binary values (either 0 or 1) and x is the interpolation value.

## Results

MLAA works with a single sample per pixel. This translates to subsampling, which makes it impossible to recover real subpixel features.

SMAA, however, works in the subpixel level. This results in:

• Local contrast
• Diagonal pattern detection
• Sharp geometric features
• Accurate searches

You can view all these in the following image, with these features compared to other methods. In fact, SMAA can produce results close to SSAA 16x.

The overhead produced by each of the solutions is negligible. In particular, local contrast adaptation is only 0.08ms, the sharp geometric features detection adn accurate distance takes 0.01ms, and diagonals processing produces an overhead of 0.12ms. In short, SMAA is rather fast, slower than SSAA and MSAA, but more fruitful, and less resource-intensive.

Well, thanks for reading the article! And thanks for the writers of the journal which I used for the majority of the article. I hope you guys have a good day, and also, go on reading scientific articles on your own. It’s simple, just head to libgen.io and search for what you like — not necessarily graphics programming. Read, read and read! Don’t watch too many Youtube tutorials, it kills your senses. I can’t stress that enough. I don’t intend to tell you what to do, these are all just suggestions. I’m currently studying Structured Computer Organization and enjoying it very, very much. I recommend it for everyone, even as bedtime reading.

Please, please, please tweet my blog, introduce it to your friends, and share it to people whom you want to enjoy life, and learn about graphics programming. Thank you, thanks a lot. Chubak out.

## The Programming Behind Cel Shading

As the inquisitive reader may have guessed, or simply, googled my name, I hail from Persia, where Prince of Persia (as most people here put it, pe-rance of per-sh-ee-a)  games and the movie are set. Some people complained that Persians are brown, and the fact that in the movie had hasted a Nordic actor to play the role of the Prince was rather insulting and racist, however, what these warriors choose to ignore is that Persians, and most other Iranic tribes at the time, such as Parthians (Parthia, where I live) and Medians were, in fact white. Let’s not beat around the bush here. For nationalistic reasons, I never played the reboot, or the trilogy, I just played the 2D platformer on a Sega Genesis emulator. Anyways, I thought the reboot was the first instance of cel shading in computer games, however, I was wrong. Ten years before, a Dreamcast game by the name of Jetset Radio (pictured above) had set the trend. I’m too young for Dreamcast and I’ve never seen one even. But it must have been helluva game judging by the videos I’ve seen of it.

But what is cel shading or as Graham Sellers puts it, cell shading [sic]? It’s basically the process of using tricks in the render pipelien to make everything appear as if they were drawn by hand. The alternative approach for naming this technique is toon shading.

To find out which games use this effect, I headed to my trusty hangout, TVTropes.org, a wiki created using PMWiki and serving astray, bored media lovers for years… and in its cel shading page, it said:

Cel Shading applies first and foremost to the way the lighting is rendered.

This layman explanation is exactly, how I would describe it. Indeed, cel shading is rendering of the diffuse light channel. Texturing, and rendering. I know that light is a component of the material, and not something to be rendered, but using LUTs, we can achieve exactly this.

Anyways, what is  LUT?

#### Color LUT

If you have played around in Da Vinci Resolve, you’ll know what LUT is. It’s short for Look Up Table. Imagine you wish to change a series of colors to another series of colors. Each color must correspond to another. This is where Color LUT comes into play.

In OpenGL, each LUT is a 1D array that corresponds to a number between 0.0f and 1.0f. This number is the intensity of the diffuse component of the Phong lighting system (remember the last post? Phong lighting is different from Phong material). Imagine this, if you will.

is normal vector, is the light vector. the LUT maps each color to each intensity using the formula. is dependent on your code.

Let’s put it all together and see what happens.

Note: These are taken from OpenGL Superbible 7ed, by Graham Sellers.

First, our front-end, at least, a part of it:

Listing 1: OpenGL Cel Shading Front End

//declare our LUT
static const GLubyte toon_tex_data[] =
{
0x44, 0x00, 0x00, 0x00,
0x88, 0x00, 0x00, 0x00,
0xCC, 0x00, 0x00, 0x00,
0xFF, 0x00, 0x00, 0x00
};

glGenTextures(1, &amp;tex_toon); //Generate texture
glBindTexture(GL_TEXTURE_1D, tex_toon); //Bind texture, a 1D texture
glTexStorage1D(GL_TEXTURE_1D, 1, GL_RGB8, sizeof(toon_tex_data) / 4);
glTexSubImage1D(GL_TEXTURE_1D, 0,
0, sizeof(toon_tex_data) / 4,
GL_RGBA, GL_UNSIGNED_BYTE,
toon_tex_data); //Pass the data
glTexParameteri(GL_TEXTURE_1D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_1D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_1D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); //Conditions


Note that all the codes are explained in comments, and this is an introductory article, and not a tutorial, so I don’t see fit to explain the code in detail. You can buy OpenGL Superbible and see for yourself.

#version 420 core

uniform mat4 mv_matrix;
uniform mat4 proj_matrix;

layout (location = 0) in vec4 position;
layout (location = 1) in vec3 normal;

out VS_OUT
{
vec3 normal;
vec3 view;
} vs_out;

void main(void)
{
vec4 pos_vs = mv_matrix * position;

// Calculate eye-space normal and position
vs_out.normal = mat3(mv_matrix) * normal;
vs_out.view = pos_vs.xyz;
// Send clip-space position to primitive assembly
gl_Position = proj_matrix * pos_vs;
}


#version 420 core

layout (binding = 0) uniform sampler1D tex_toon;

uniform vec3 light_pos = vec3(30.0, 30.0, 100.0);

in VS_OUT
{
vec3 normal;
vec3 view;
} fs_in;

out vec4 color;

void main(void)
{
// Calculate per-pixel normal and light vector
vec3 N = normalize(fs_in.normal);
vec3 L = normalize(light_pos - fs_in.view);
// Simple N dot L diffuse lighting
float tc = pow(max(0.0, dot(N, L)), 5.0);

// Sample from cell shading texture
color = texture(tex_toon, tc) * (tc * 0.8 + 0.2);
}


If we load a donut model into the program, this is what we’ll get:

Well, hope you enjoyed it! I’m currently learning After Effects. And I may write a book about it. What do you think?

## Creating a Procedural Texture Using Gizeh

Gizeh is a small library for Python which you can install using pip. It doesn’t have any official documentation, so you have to suffice to the small documentation on it’s Github Page. Before starting this tutorial, read the README file carefully, as you may confused later on. Now, let’s see what we want to create. A procedural texture of a textile. Maybe a denim, maybe another pattern. I don’t know. Right now we can’t afford to be picky. The output is entirely randomized, and if you aren’t happy with the texture, you can just run the code again.

Not happy? Let’s run the code again (I’m using Pycharm to write and run this code, teaching you about setting up Python and a good IDE is beyond the scope of this tutorial):

But how? Simple:

1- A blue background.

2- 200 different groups of 50 circles that increase in alpha as they decrease in radius. colors are from rgb(200, 200, 200) to rgb(250, 250, 250).

3- Vertical and horizontal lines spaced together based on a coefficient of 0.01 and tilted by a random number between 1 and 8.

That’s it!

Let’s start. First, let’s setup a window, and a surface:

import gizeh as gz
import random

w = 640
h = 480

surface = gz.Surface(width=w, height=h, bg_color=(21/255, 96/255,189/255))


You can change w and h to change the height and the width. Now, let’s generate our radial gradient:

def circle_generator(x, y, r, c):
i = r
a = 0
while i > 0:
circle = gz.circle(r = i, xy=[x, y], fill=(c/255, c/255, c/255, a*0.70))
i -= 1
a += 1/2000

circle.draw(surface)


Besides the location of the circles, the function accepts two other parameters: the radius, which decreases in each while loop, and the color, which we divide by 255 since in Gizeh, rgb is between 0 and 1. We increase the alpha (a) by adding 1/2000 to it in each iteration.  At the end, we draw the circle.

Now, let’s make a line generator:

def generate_lines(x1, y1, x2, y2, s, a):
line = gz.polyline(points=[(x1, y1), (x1, y1), (x2, y2), (x2, y2)], stroke_width=s,
stroke=(0, 0, 0, a/255), fill=(0, 0, 0, a/255))

line.draw(surface)


This one just generates one line, we’re actually drawing a polygon were its origin is (x1, y1) and ends at (x2, y2). If we add a randomized number to the second coordinates, we can have a mangled shape, which will come to use if we want to generate some other pattern. s is stroke, and a is alpha. We set the fill to black, but it’s useless, as we don’t have a fill.

for i in range(200):
x = random.randint(0, w)
y = random.randint(0, h)
r = random.randint(20, 100)
c = random.randint(200, 255)
circle_generator(x, y, r, c)



we choose entirely random locations, a radius between 20 and 100, and a color between very light gray and white. Then we call the function.

Now, the lines!

s = random.randint(1, 3)
a = random.randint(50, 200)
coeff = 0.01

for i in range(w):
tilt = random.randint(1, 8)
generate_lines(w * (i * coeff), 0, w * (i * coeff) + tilt, h, s, a)

for i in range(h):
tilt = random.randint(1, 8)
generate_lines(0, h * (i * coeff), w, h  * (i * coeff) + tilt, s, a)


First, we choose a random opacity. and a random stroke. Then we set the coefficient to 0.01. You can experience with other coefficient values, but be warned that if you randomize it, it sometimes may take you 10 runs to get a right pattern. We then create a new random number in each iteration to use it as a tilt. We then generate the horizontal lines w times and vertical lines h times.

At the end, we save the file as a PNG. This is why you should use PyCharm, it shows the picture and updates it in a different window. Other IDEs don’t do that.

surface.write_to_png("only_bg.png")


Well, that’s it! You can experiment with it, use diagonal lines, creates woven textiles, et al. Hope you enjoyed it.

## Pixel Perfect Crosshair with Processing

Have you ever wondered how is it possible to create a pixel-perfect crosshair? Well, wonder no more. Processing is here.

Well, no time to lose. Let us start off by writing the setup() function.

void setup(){
size(500, 500);
noLoop();
}


It simply creates a 500*500 image, and sets the loop to noLoop, so draw() function won’t go on indefinitely. Now, the draw function. Before that, save the code. So the image will be saved in the same folder as your code. Alright, let’s have a looksie.

void draw(){
noFill();
strokeWeight(10);
arc(width / 2, height / 2, 300, 300, QUARTER_PI, PI);
strokeWeight(8);
arc(width / 2, height / 2, 250, 250, QUARTER_PI, PI);
strokeWeight(6);
arc(width / 2, height / 2, 200, 200, 0, QUARTER_PI);
strokeWeight(4);
arc(width / 2, height / 2, 200, 200, 0, 3);
strokeWeight(6);
arc(width / 2, height / 2, 250, 250, PI, TWO_PI);
fill(0);
ellipse(width / 2, height / 2, 30, 30);
strokeWeight(12);
line(width * 0.25, height * 0.76, width / 2, height / 2);
line(width * 0.75, height * 0.75, width / 2, height / 2);
line(width * 0.25, height * 0.26, width / 2, height / 2);
line(width * 0.75, height * 0.25, width / 2, height / 2);

saveFrame("crosshair.png");
}


noFill() will disable the color fill, so we’ll only have strokes. strokeWeight() which we use religiously sets the size of the stroke in pixels. And arc() creates an arc. This is the main part of this short tutorial, the arc function goes if defined as:

arc(postion_x, position_x, radius_x, radius_y, start_angle, end_angle);


start_angle and end_angle are in Radians. You can use QUARTER_PI (45d), HALF_PI (perpendicular), PI (180d), and PI_TWO (circular) like:

You can add these together to create PI/3 (35d) and 2*PI/3 (275d).

At the end, we add four lines like so:

line(x1, y1, x2, y2);


We use math to access quarter and a third of the screen. All x2 and y2s are the center of the screen.

At the end, we save the frame.

### Coming up next…

I have prepared a Processing and Illustrator tutorial for creating two types of planets. Stay tuned!