Face Morphing

Part 1 - Defining Correspondences

Correspondences are essential for good image morphing so that we know how the shape of things in the image change.

I used the very nice tool provided by the course staff to define correspondences for this project. I made a few modifications, however. Here is a link to my version of the tool.

Changes to the staff-provided tool:

• Changed the mode showing correspondence numbers to show it on both images. This makes it easier to verify points.
• Added a new "Move" mode that allows you to click on a correspondence point and move it to a new location. Hit M to enter this mode.
• Added a new option to add a correspondence point to the right image when the left one is clicked. This combined with the move mode makes create correspondences much faster because you can batch your work, and it eliminates mouse moves significantly (hey, I'm an HCI grad student, I care about stuff like that).

I specified 98 correspondence points. I initially had about 60, but after getting further in the project I noticed a few places where points could be added to improve the animation. The points can be viewed in this JSON file, which can also be loaded into the tool with the two images below.

A halfway point was computed for each pair of correspondence points and a Delaunay Triangulation was computed for the set of points. The midway point was chosen to not bias the triangulation to one image. The triangulation was applied to the correspondence points for each image to create the images below.

Part 2 - Computing the "Mid-Way Face"

Before computing the morph between the two faces, we compute it for one face. We first get the morphed geometry (points) at the halfway point. This is done with a simple cross dissolve method that is used in several parts of this project.

  def cross_dissolve(pts1, pts2, alpha):
    return (1 - alpha) * pts1 + alpha * pts2
          

For each triangle in the triangulation, we compute an Affine Transformation that converts from the start image to the mid-way, and the end image to the mid-way. The affune transformation from some point p to a point q is defined as follows:

This system of equations can be rewritten into the following form by expanding out the dot products (left as an exercise for the reader).

This new matrix can be inverted and applied to both sides of this equation, giving us the solutions for the affine parameters.

To find the pixel values for each triangle, the inverse of the affine transformation is applied to each pixel in the target triangle (which was computed using sk.draw.polygon). No interpolation was used when looking up pixel values from the source images.

Each image was then blended together using the same cross_disolve method, but now on the pixel values instead of correspondence points.

Part 3 - The Morph Sequence

The previous step was abstracted into a method and computed for a bunch of different midway points. A range of alpha values were used to compute 30 different mid-way faces, allowing us to transition smoothly between the two faces. This animation was created by reversing and stacking the blend back to back.

Part 4 - The "Mean Face" of a Population

We can compute the "average face" of a population of people given enough pictures of people from that population. For this we use the IMM Dataset of Danish people (the initially released dataset of 37 faces).

Like before we can compute an average geometry for each of the faces by taking the mean position of each correspondence point (which is provided by the dataset).

Using this mean geometry, we can then morph each of the faces in the dataset into the mean face. Here is one example.

We can see that there is quite a bit of warping along the top of the head. I wanted to fix this, and I realized that the issue was due to the triangulation stretching the triangles in that area. I inserted a few "fake correspondence points" into each image into the top of the image to prevent elongated triangles stretching over the faces. This changes the mean triangulation.

We can also look at the before and after of the previous picture.

Much better! We will also look at how this affects some later results. For now, lets look at a few more morphed Dane examples.

Here is a gif file I made showing all of the morphed faces

If we take the average of all of these morphed faces, we get the mean face in the IMM dataset. Lets look at both versions (the "fixed" triangulation and the original) to see if the alteration had any effect.

Looking at this result, it doesn't actually change the value of the mean face much because all of those errors would average out. Oh well, maybe it will have an effect on the next experiment.

I took a picture of myself and created a correspondence with the points in the dataset. This was done using the same tool as before. I loaded the triangulation image from above and clicked on each point in the order they were defined in the dataset (this was verified manuall). Then I moved the points to match my face. Here is a link to the JSON file with my correspondence points, which can be loaded in the tool with the triangulation image from above and the picture of me from below.

Now we morph my face to the mean Danish face geometry. Also we morph the mean Danish face to my geometry. The artifacts on the top of the head are due to the fact that there are no correspondences in the dataset along the top of the head, so the long triangles present on the top of the image lead to artifacts.

Did the fixed triangulation have any effect here? Let's look at both morphs of my face.

The results here are significalty better. This implies that for individual results and experiments, better triangulations are more important because imperfections don't get averaged away.

Part 5 - Caricatures: Extrapolating from the mean

For the previous result we morphed my face to the mean Danish face geometry. We can also morph in the opposite direction, emphasizing the differences between my face and the mean. I'm sure this won't be cursed.

Part 6 - Bells and Whistles: Gender Change

We can explore how to use these images to change my gender. We will present 3 approaches: One with the geometry of my face changed to match the geometry of the female face. One which is a blend of the appearences of the two faces. One which is the morph between the two faces.

To blend the geometry, we compute a triangulation of the average points of my face and the spanish female face (correspondence file here, computed using the above 2 images). Then we do a morph with no appearence blending, fully morphing the geometry of my face to that of the female.

To blend only the appearence, first the spanish female's geometry is morphed to my face geometry. Then an appearence blend of 0.5 is applied to the two images.

To morph both, a mid-way face is computing by finding the mid-way geometry and mid-way appearence (geometry and appearence alpha of 0.5).