Exploring Face Detection with OpenCV

by Bryan Chung

Both OpenCV and Processing have new releases after the publication of my book, Pro Processing for Images and Computer Vision with OpenCV. An interesting new feature in OpenCV is face detection through a new module that is named face. In this post, I would like to introduce a few features of the face module with the use of the CVImage library.

The latest release for OpenCV is 3.4.2, and for Processing is 3.4. A corresponding new, pre-built version of the CVImage library that was developed in Chapter 8 of my book can be found on my website. The new version of CVImage includes the optflow module described in my book, and the new face module from OpenCV. 

Chapter 7 of the book describes face detection using the objdetect module with a Haar Cascade file. We can achieve a similar result using the face module by invoking the following function: 

     Face.getFacesHAAR(image, faces, dataPath(faceFile));

Note that image is a Mat variable for detection, faces is a MatOfRect containing all the face rectangles detected, and faceFile is a string variable for the Haar Cascade file.

In addition to face detection, the face module can detect face landmarks (i.e., a person’s individual facial features.) Face landmark detection is done via the Facemark class. Methods in the class can return the detailed information about a subject’s face, eyebrows, eyes, nose, and mouth as shown in Figure 1.

^{fig 1. Face landmarks.}

The following lines show the code to obtain face landmarks from an input image of type Mat, held in the variable i. Function getFacesHAAR is what is described in the previous paragraph to detect a face in front of the camera. The Facemark function fm.fit is where the magic happens.

     ArrayList<MatOfPoint2f> shapes = new ArrayList<MatOfPoint2f>();
     MatOfRect faces = new MatOfRect();
     Face.getFacesHAAR(i, faces, dataPath(faceFile));
     if (!faces.empty()) {
          fm.fit(i, faces, shapes);
     }

After this code executes, the variable shapes will contain all the face landmark information that has been detected. The variable fm in the example is an instance of the Facemark class. Create the variable in the beginning of the program as follows:

     fm = Face.createFacemarkKazemi();
     fm.loadModel(dataPath(“face_landmark_model.dat”);

The ArrayList is an array of 68 elements. Each element corresponds to a feature point in the face according to the following scheme.

• 0-16 face
• 17-21 left eyebrow
• 22-26 right eyebrow
• 27-35 nose
• 36-41 left eye
• 42-47 right eye
• 48-67 mouth

We can make use of face landmark points to divide a person’s facial area using the Delaunay triangulation. That triangulation can be done by the Subdiv2D class in the imgproc module, and Figure 2 illustrates conceptually how the division occurs. The left side of the figure shows the individual triangles. The right side of the figure shows the mask of the entirety of the triangulated areas.

^{fig 2. The Delaunay Triangulation.}

The following code segment will obtain the Delaunay triangles given an array of Point in the variable pts:

     ArrayList<int []> tri = new ArrayList<int []>();
     Subdiv2D subdiv = new Subdiv2D(r);
     for (Point p : pts) {
          if (r.contains(p))
               subdiv.insert(p);
      }
     MatOfFloat6 triangleList = new MatOfFloat6();
     subdiv.getTriangleList(triangleList);
     float [] triangleArray = triangleList.toArray();

The output variable triangleArray contains the vertices of the triangles. That array is organized into groups of six floating point numbers. Each group represents the three pair of x, y positions for the three vertices of a triangle. 

We can perform the Delaunay triangulation on two images, and subsequently warp the first image into the second one. In the end, we can implement a Processing version of the face swap effect, as demonstrated in the sample program, sample_face_swapping.cpp, in the OpenCV distribution. Figure 3 is an example of swapping my kindergarten photo image onto my live image from a webcam.

^{fig 3. Swapping face images.}

For more information on face landmarks and face swapping, including examples and source code, please visit my website. There you’ll find many posts with examples showing how to use the OpenCV’s face module, as well as other examples involving OpenCV and image processing.

About the Author

Bryan WC Chung is an interactive media artist and design consultant. He was the Grand Prize winner of the 19th Japan Media Art Award, Art Division, 2015. In 2009, his consultation work on the Coca-Cola Happy Whistling Machine won the Media Kam Fan Advertising Award. Bryan's works have been exhibited at the World Wide Video Festival, Multimedia Art Asia Pacific, Stuttgart Film Winter Festival, Microwave International New Media Arts Festival, and the China Media Art Festival. In the former Shanghai Expo 2010, he provided interactive design consultancy to industry leaders in Hong Kong. Bryan also develops software libraries for the open source programming language Processing. He is the author of the book Multimedia Programming with Pure Data. Currently, he is Associate Professor in the Academy of Visual Arts, Hong Kong Baptist University, where he teaches subjects on interactive art, computer graphics, and multimedia. 

This article was contributed by Bryan Chung, author of Pro Processing for Images and Computer Vision with OpenCV.