Tessellation Tutorial for OpenGL 4.3

So you’re curious about tessellation… Well, you’ve clicked on the right link. In my last tutorial I explained in broad terms what it means, but for those of you who want to plunge on further into this shader’s secrets, here you go. Hope you’re grateful :D.

Tessellation is the process of taking a high-order primitive (aka polygon or patch – from now on we will use the former name) and breaking it down into smaller parts, such as triangles, in order to provide scalable level-of-detail. Logically, the tessellation process is placed between the vertex and geometric shaders.


Rendering Pipeline (OpenGL Superbible 6th ed.)

 The tessellation shader is actually composed of 3 separate parts:

  • Tessellation Control Shader (TCS) takes input from the VS (vertex shader) and is responsible for 2 tasks: determining the tessellation levels to be sent to the TE (Tessellation Engine) and generating data to be passed to the TES
  • Tessellation Engine (TE) which does the actual tessellation, by breaking down the patch according to the levels set in the TCS
  • Tessellation Evaluation Shader is the last link in the process, taking each new vertex created by the tessellation and giving it a position within the world

Among these 3 parts, only the first and third ones can be programmed. But before we start discussing what can be done with them, here’s what you need to do in the main program to create and use the shader.

First of all, if you are not using triangles as your primitive of choice, then the glPatchParameteri(GLenum pname, Glint value) function must be called, where pname is the name of the parameter to be set (can be GL_PATCH_VERTICES, GL_PATCH_DEFAULT_OUTER_LEVEL or GL_PATCH_DEFAULT_INNER_LEVEL) and value is, of course, the value of that parameter. In this case, you need to call it with GL_PATCH_VERTICES and the vertex number of your primitive, which needs to be less than GL_MAX_PATCH_VERTICES. As for the inner and outer level values, they can be set before the shader is called, thus saving some processing power.

Another thing which is crucial for something to actually show up on your screen when running the program is for the first parameter in the glDrawArrays to be GL_PATCHES. Trust me, I spent a lot of time staring blankly at the screen trying to understand why it wasn’t working.

The Tessellation Control Shader (TCS)

The most basic TCS must set the levels of inner and outer tessellation (in case you didn’t set them using glPatchParameteri) and pass the vertex position to the next entity. Here’s the example:

#version 430 core
//you need to specify the OpenGL version, of course
//specify the number of vertices per patch
layout (vertices = 3) out;
void main(void){
    if (gl_InvocationID == 0){
        gl_TessLevelInner[0] = 7.0;
        gl_TessLevelOuter[0] = 2.0;
        gl_TessLevelOuter[1] = 3.0;
        gl_TessLevelOuter[2] = 7.0;

//in case of quad, you have to specify both gl_TessLevelInner[1] and //gl_TessLevelOuter[3]
    gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;

Now that we’ve finally exemplified through code what must be done, it’s time to explain what the deal is with the inner and outer levels, for both triangles and quads.

But first, here are some examples (inner levels are in and outer levels are out so:

  • in0   = horizontal tessellation,
  • in1   = vertical tessellation ,
  • out0 = edge 0-3
  • out1 = edge 2-3
  • out2 = edge 1-2
  • out3 = edge 0-1):

Tessellation with quads


Tessellation with quads

We should start with the outer levels, because they’re just easier to explain. Each edge is associated with one level, determining the number of equal segments that edge will be divided in.

You may be a bit confused about the outer levels and the logic behind the association between level and edge. Let’s say that you pass the quad’s vertices to the Vertex Shader in the order 0-1-2-3. It would seem logical that, given this order of vertices and the 4 outer levels, level 0 would be correlated with edge 0-1, level 1 with edge 1-2 etc. WRONG! It’s the exact reverse. You have to go through the vertices in reverse order to get the correct associations. Thus, level 0 is with edge 0-3, level 1 with edge 3-2, etc.

Moving on to inner levels, gl_TessLevelInner[0] sets the level horizontally and gl_TessLevelInner[1] vertically. That’s all there is to it. If you look closely at the examples, you will quickly observe how these are implemented.

Now moving on to triangles…

Further on, I will refer to edges which are opposite to vertices. To clarify what I mean, here’s an example:

Edge BC is opposite vertex A

Edge BC is opposite vertex A

The most important part of this process is determining the desired tessellation levels, both inner and outer. The inner level is set by just one variable and as can be seen in the image below, it determines the number of inner triangles which will be created inside the patch. Observe the difference between an odd and an even level. With every level increase, the shader will attempt to create another triangle inside the last created triangle, followed by creating additional triangles between the last 2 created in relation to the outer levels set. In the case of an odd level, the center of the patch will be a triangle, while in an even level, it will be the triangle’s center of gravity (intersection of all the medians).

The outer level determines the number of segments each edge will be divided into. Here, the order in which you pass the vertices to the vertex shader matters, because the first outer level pertains to the opposite edge of the first vertex of the triangle, the second level with the second vertex, etc. For example, out3 is the level of vertex 3 and its effects are seen on edge 1-2, opposite to vertex 3.

After setting the desired levels, you have to pass along the patch vertices’ positions to the TES:

gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;

gl_out and gl_in are arrays of vertices which are used for receiving and transmitting them from shader to shader and are built in the language. gl_InvocationID is a parameter used to identify each vertex. gl_Position, of course, returns the position of the vertex in a vec4.

Various inner and outer levels

Various inner and outer levels

The Tessellation Evaluation Shader (TES)

This final part of the tessellation process is responsible for giving the newly created vertices their position within the world, receiving from the TCS the patch vertices’ positions and the tessellation coordinates (each instance of the TES deals with only one new vertex). Knowing these values, we will be able to determine the exact position of the new vertex.

For quads, we use standard bilinear interpolation to determine the position. For those of you who don’t know what interpolation is, here’s a little example:

Linear interpolation

Linear interpolation

Let’s say you have segment AB, on the x axis (y is constant). You know the coordinates of A and B, you know that C is at 0.4 (if A is 0 and B is 1). To find out x3, you have to use linear interpolation:


Bilinear interpolation is just an extrapolation of the linear one.

Bilinear interpolation

Bilinear interpolation

The process is simple. First you do a linear interpolation on the x axis, to determine the x coordinate and then on that vertical line which goes through that coordinate you do a y interpolation.

Now, armed with this knowledge, let’s take a look at the code for the TES with quads:

#version 430 core
layout (quads) in;
void main(void){

vec4 p1 = mix(gl_in[1].gl_Position,gl_in[0].gl_Position,gl_TessCoord.x);
vec4 p2 = mix(gl_in[2].gl_Position,gl_in[3].gl_Position,gl_TessCoord.x);
gl_Position = mix(p1, p2, gl_TessCoord.y);

First of all, let’s briefly discuss the mix function. It just executes basic linear interpolation between the first 2 values, using the third as the weight. First we calculate the 2 points from the upper and lower edges (horizontal interpolation) and knowing those, we do the vertical one, using the tessellation coordinates as weights.

For triangles, the TES is a bit harder to understand, mostly because first you have to know about barycentric coordinates.

An illustration of barycentric coordinates (source here)

An illustration of barycentric coordinates (source here)

Basically, you use a tridimensional vector in which each position can be 1 only in one of the vertices (the other 2 will be 0). The rule is that the sum of all 3 positions of the vector must always be 1. On an edge between 2 vertices (excluding the vertices themselves) only the positions associated with the 2 vertices can be positive.

There is a lot of theory behind barycentric coordinates, but for the time this will suffice. The code for the TES using triangles:

#version 430 core
layout (triangles, equal_spacing, cw) in;
void main(void){ 

By multiplying the barycentric coordinates with the positions of each vertex of the patch, the nex vertex’s position will be determined.

These are the basic principles of tessellation. There are other things you can do with the tessellation shader (one example is my previous article, about Terrain LoD and stitching), however this article is not the place to present more advanced concepts. This article was meant only as a basic tessellation tutorial, to give you a basic understanding of the concept. I hope it has been eloquent and informative enough to do just that.

Tessellate away, mates!


I'm an engineer, currently employed at a financial software company. My interests include gaming, LPing and, of course, reviewing, but also game dev and graphics. Also, in the past I've dabbled in amateur photography, reviewing movies and writing short stories and blog posts. I am also a huge Song of Ice and Fire fan, but that's beside the point. Youtube Channel, Deviantart , Google + , Twitter

blog comments powered by Disqus