You can use the multimedia analytics framework in a Big Data environment to perform facial recognition in videos and images.
The multimedia analytics feature of Oracle Big Data Spatial and Graph provides a framework for processing video and image data in Apache Hadoop. The framework enables distributed processing of video and image data. Features of the framework include:
APIs to process and analyze video and image data in Apache Hadoop
Scalable, high speed processing, leveraging the parallelism of Apache Hadoop
Built-in face recognition using OpenCV
Ability to install and implement custom video/image processing (for example, license plate recognition) to use the framework to run in Apache Hadoop
The video analysis framework is installed on Oracle Big Data Appliance if Oracle Spatial and Graph is licensed, and you can install it on other Hadoop clusters.
The multimedia analytics feature comes with built-in face recognition. Face recognition uses OpenCV libraries, available with the product. This chapter describes using this face recognition functionality.
Face recognition has two steps:
“Training” a model with face images. This step can be run in any Hadoop client or node.
Recognizing faces from input video or images using the training model. This step is a MapReduce job that runs in a Hadoop cluster.
The training process creates a model stored in a file. This file is used as input for face recognition from videos or images.
Training is done using the Java program
OrdFaceTrainer, which is part of part of
ordhadoop_multimedia_analytics.jar. Inputs to this program are a set of images and a label mapping file that maps images to labels. The output is a training model that is written to a file. (You must not edit this file.)
To train the multimedia analytics feature to detect (recognize) faces, follow these steps.
Create a parent directory and subdirectories to store images that are to be recognized.
Each subdirectory should contain one or more images of one person. A person can have images in multiple subdirectories, but a subdirectory can have images of only one person. For example, assume that a parent directory named
images exists where one subdirectory (
d1) contains images of a person named Andrew, and two subdirectories (
d3) contain images of a person named Betty (such as pictures taken at two different times in two different locations). In this example, the directories and their contents might be as follows:
images/d1 contains five images of Andrew.
images/d2 contains two images of Betty.
images/d3 contains four images of Betty.
Create a mapping file that maps image subdirectories to labels.
A “label” is a numeric ID value to be associated with a person who has images for recognition. For example, Andrew might be assigned the label value 100, and Betty might be assigned the label value 101. Each record (line) in the mapping file must have the following structure:
d1,100,Andrew d2,101,Betty d3,101,Betty
Set the required configuration properties:
oracle.ord.hadoop.ordfacemodel oracle.ord.hadoop.ordfacereader oracle.ord.hadoop.ordsimplefacereader.dirmap oracle.ord.hadoop.ordsimplefacereader.imagedir
For information about the available properties, see Configuration Properties for Multimedia Analytics.
Set the CLASSPATH. Include the following in the Java CLASSPATH definition. Replace each asterisk (*) with the actual version number.
$MMA_HOME/lib/ordhadoop-multimedia-analytics.jar $MMA_HOME/opencv_3.0.0/opencv-300.jar $HADOOP_HOME/hadoop-common-*.jar $HADOOP_HOME/hadoop-auth-*.jar $HADOOP_HOME/commons-lang*.jar $HADOOP_HOME/commons-logging-*.jar $HADOOP_HOME/commons-configuration-*.jar $HADOOP_HOME/commons-collections-*.jar $HADOOP_HOME/guava-*.jar $HADOOP_HOME/slf4j-api-*.jar $HADOOP_HOME/slf4j-log4j12-*.jar $HADOOP_HOME/log4j-*.jar $HADOOP_HOME/commons-cli-*.jar $HADOOP_HOME/protobuf-java-*.jar $HADOOP_HOME/avro-*.jar $HADOOP_HOME/hadoop-hdfs-*.jar $HADOOP_HOME/hadoop-mapreduce-client-core-*.jar
Create the training model. Enter a command in the following general form:
java -classpath <…> oracle.ord.hadoop.recognizer.OrdFaceTrainer <training_config_file.xml>
$MMA_HOME/examplehas a set of sample files. It includes scripts for setting the Java
CLASSPATH. You can edit the example as needed to create a training model.
Images used to create the training model should contain only the face, with as little extra detail around the face as possible. The following are some examples, showing four images of the same man’s face with different facial expressions.
The selection of images for training is important for accurate matching. The following guidelines apply:
The set of images should contain faces with all possible positions and facial movements, for example, closed eyes, smiles, and so on.
Try to avoid including images that are very similar.
If it is necessary to recognize a person with several backgrounds and light conditions, include images with these backgrounds.
The number of images to include depends on the variety of movements and backgrounds expected in the input data.
To detect (recognize) faces in videos, you have the following options for video processing software to transcode video data:
OrdOpenCVFaceRecognizerMulti as the frame processor, along with any of the frontal face cascade classifiers available with OpenCV.
Haarcascade_frontalface_alt2.xml is a good place to start. You can experiment with the different cascade classifiers to identify a good fit for your requirements.
Use third-party face recognition software.
To perform recognition, follow these steps:
Copy the video files (containing video in which you want to recognize faces) to HDFS.
Copy these required files to a shared location accessible by all nodes in the cluster:
Generated training model
Mapping file that maps image subdirectories to labels
Cascade classifier XML file
Create the configuration file.
Required configuration parameters:
oracle.ord.hadoop.inputtype: Type if input data (
oracle.ord.hadoop.outputtypes: Format of generated results (
oracle.ord.hadoop.ordframegrabber: Get a video frame from the video data. You can use the Java classes available with the product or you can provide an implementation for the abstraction.
oracle.ord.hadoop.ordframeprocessor: Processor to use on the video frame to recognize faces. You can use the Java classes available with the product or you can provide an implementation for the abstraction.
oracle.ord.hadoop.recognizer.classifier: Cascade classifier XML file.
oracle.ord.hadoop.recognizer.labelnamefile: Mapping file that maps image subdirectories to labels.
Optional configuration parameters:
oracle.ord.hadoop.frameinterval: Time interval (number of seconds) between frames that are processed. Default: 1.
oracle.ord.hadoop.numofsplits: Number of splits of the video file on the Hadoop cluster, with one split analyzed on each node of the Hadoop cluster. Default: 1.
oracle.ord.hadoop.recognizer.cascadeclassifier.scalefactor: Scale factor to be used for matching images used in training with faces identified in video frames or images. Default: 1.1 (no scaling)
oracle.ord.hadoop.recognizer.cascadeclassifier.minneighbor: Determines size of the sliding window to detect face in video frame or image. Default: 1.
oracle.ord.hadoop.recognizer.cascadeclassifier.flags: Determines type of face detection.
oracle.ord.hadoop.recognizer.cascadeclassifier.minsize: Smallest bounding box used to detect a face.
oracle.ord.hadoop.recognizer.cascadeclassifier.maxsize: Largest bounding box used to detect a face.
oracle.ord.hadoop.recognizer.cascadeclassifier.maxconfidence: Maximum allowable distance between the detected face and a face in the model.
oracle.ord.hadoop.ordframeprocessor.k2: Key class for the implemented class for
oracle.ord.hadoop.ordframeprocessor.v2: Value class for the implemented class for
Set the HADOOP_CLASSPATH.
Ensure that HADOOP_CLASSPATH includes the files listed in Training to Detect Faces
Run the Hadoop job to recognize faces. Enter a command in the following format:
$ hadoop jar $MMA_HOME/lib/orhadoop-multimedia-analytics.jar -conf <conf file> <hdfs_input_directory_containing_video_data> <hdfs_output_directory_to_write_results>
The accuracy of detecting faces depends on a variety of factors, including lighting, brightness, orientation of the face, distance of the face from the camera, and clarity of the video or image. You should experiment with the configuration properties to determine the best set of values for your use case. Note that it is always possible to have false positives (identifing objects that are not faces as faces) and false recognitions (wrongly labeling a face).
$MMA_HOME/examplehas a set of sample files. It includes scripts for setting the Java CLASSPATH. You can edit as needed to submit a job to detect faces.
To detect faces in images, copy the images to HDFS. Specify the following property:
<property> <name>oracle.ord.hadoop.inputtype</name> <value>image</value> </property>
Several examples and training materials are provided to help you get started detecting faces.
$MMA_HOME contains these directories:
video/ (contains a sample video file in mp4 and avi formats) facetrain/ analytics/
facetrain/ contains an example for training.
facetrain/config/ contains the sample configuration files, and
facetrain/faces/ contains images to create the training model, and the mapping file that maps labels to images.
trainface.sh provide the choice of using a Makefile or a bach script to run the training step.
You can create the training model as follows:
The training model will be written to
For detecting faces in videos, analytics/ contains an example for running a Hadoop job to detect faces in the input video file. This directory contains conf/ with configuration files for the example.
trainface.sh provide the choice of using a Makefile or a batch script to submit the Hadoop job.
You can run the job as follows (includes copying the video file to HDFS directory
The output of the job will be in the HDFS directory
The multimedia analytics framework uses the standard methods for specifying configuration properties in the
hadooop command. You can use the
–conf option to identify configuration files, and the
-D option to specify individual properties. This topic presents reference information about the configuration properties.
Some properties are used for specific tasks. For example, training properties include:
The following are the available configuration properties, listed in alphabetical order. (All names start with
oracle.ord.hadoop.). For each parameter the parameter name is listed, then information about the parameter.
String.Timestamp interval (in seconds) to extract frames for processing. Allowable values: positive integers and floating point numbers. Default value: 1. Example:
<property> <name>oracle.ord.hadoop.frameinterval</name> <value>1</value> </property>
InputFormat class name in the framework, which represents the input file type in the framework. Default value:
<property> <name>oracle.ord.hadoop.inputformat</name> <value>oracle.ord.hadoop.OrdVideoInputFormat</value> </property>
String. Type of input data:
<property> <name>oracle.ord.hadoop.inputtype</name> <value>video</value> </property>
Positive integer. Number of the splits of the video files on the Hadoop cluster, with one split able to be analyzed in each node of the Hadoop cluster. Recommended value: the number of nodes/processors in the cluster. Default value: 1. Example:
<property> <name>oracle.ord.hadoop.numofsplits</name> <value>1</value> </property>
String. Name of the file that stores the model created by the training. Example:
<property> <name> oracle.ord.hadoop.ordfacemodel </name> <value>ordfacemodel_bigdata.dat</value> </property>
String. Name of the Java class that reads images used for training the face recognition model. Example:
<property> <name> oracle.ord.hadoop.ordfacereader </name> <value> oracle.ord.hadoop.OrdSimpleFaceReader </value> </property>
String. File containing additional configuration properties for the specific application. Example:
<property> <name> oracle.ord.hadoop.ordfacereaderconfig </name> <value>config/ordsimplefacereader_bigdata.xml</value> </property>
String. Name of the Java class that decodes a video file. This is the implemented class for
OrdFrameGrabber, and it is used by the mapper to decode the video file. Available installed implementations with the product:
oracle.ord.hadoop.OrdJCodecFrameGrabber (the default) and
oracle.ord.hadoop.OrdFFMPEGFrameGrabber (when FFMPEG is installed by the user). You can add custom implementations. Example:
<property> <name>oracle.ord.hadoop.ordframegrabber</name> <value>oracle.ord.hadoop.OrdJCodecFrameGrabber</value> </property>
String. Name of the implemented Java class of interface OrdFrameProcessor, which is used by the mapper to process the frame and recognize the object of interest. Default value: oracle.ord.hadoop.mapreduce.OrdOpenCVFaceRecognizerMulti. Example:
<property> <name>oracle.ord.hadoop.ordframeprocessor </name> <value>oracle.ord.hadoop.mapreduce.OrdOpenCVFaceRecognizerMulti</value> </property>
String. Java class name, output key class of the implemented class of interface
OrdFrameProcessor. Default value:
<property> <name>oracle.ord.hadoop.ordframeprocessor.k2</name> <value>org.apache.hadoop.io.Text</value> </property>
String. Java class name, output value class of the implemented class of interface
OrdFrameProcessor . Default value:
<property> <name>oracle.ord.hadoop.ordframeprocessor.v2 </name> <value>oracle.ord.hadoop.mapreduce.OrdImageWritable</value> </property>
String. Only only relevant for custom (user-specified) plug-ins: name of the implemented Java class of interface
OrdOutputProcessor that processes the key-value pair from the map output in the reduce phase. Example:
<property> <name>oracle.ord.hadoop.ordframeprocessor</name> <value>mypackage.MyOutputProcessorClass</value> </property>
String. Mapping file that maps face labels to directory names and face images. Example:
<property> <name> oracle.ord.hadoop.ordsimplefacereader.dirmap </name> <value>faces/bigdata/dirmap.txt</value> </property>
String. File system directory containing faces used to create a model. This is typically in a local file system. Example:
<property> <name> oracle.ord.hadoop.ordsimplefacereader.imagedir </name> <value>faces/bigdata</value> </property>
String. Name of the OutputFormat class, which represents the output file type in the framework. Default value:
<property> <name>oracle.ord.hadoop.outputformat</name> <value> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; </value> </property>
String. Format of output that contains face labels of identified faces with the time stamp, location, and confidence of the match: must be
<property> <name>oracle.ord.hadoop.outputtype</name> <value>json</value> </property>
String. File containing additional configuration properties for the specific job. Example:
<property> <name>oracle.ord.hadoop.parameterfile </name> <value>oracle_multimedia_face_recognition.xml</value> </property>
String. Use this property to select the type of object detection. Must be
CASCADE_FIND_BIGGEST_OBJECT (look only for the largest face), or
CASCADE_DO_ROUGH_SEARCH. . Default:
CASCADE_SCALE_IMAGE | CASCADE_DO_ROUGH_SEARCH. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.flags</name> <value>CASCADE_SCALE_IMAGE</value> </property>
Floating point value. Specifies how large the distance (difference) between a face in the model and a face in the input data can be. Larger valuse will give more matches but might be less accurate (more false positives). Smaller values will give fewer matches, but be more accurate. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.maxconfidence</name> <value>200.0</value> </property
String, specifically a pair of values. Specifies the maximum size of the bounding box for the object detected. If the object is close by, the bounding box is larger; if the object is far away, like faces on a beach, the bounding box is smaller. Objects with a larger bounding box than the maximum size are ignored. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.maxsize</name> <value>(500,500)</value> </property>
Integer. Determines the size of the sliding window used to detect the object in the input data. Higher values will detect fewer objects but with higher quality. Default value: 1. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.minneighbor</name> <value>1</value> </property>
String, specifically a pair of values. Specifies the minimum size of the bounding box for the object detected. If the object is close by, the bounding box is larger; if the object is far away, like faces on a beach, the bounding box is smaller. Objects with a smaller bounding box than the minimum size are ignored. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.minsize</name> <value>(100,100)</value> </property>
Floating pointnumber. Scale factor to be used with the mapping file that maps face labels to directory names and face images. A value of 1.1 means to perform no scaling before comparing faces in the run-time input with images stored in subdirectories during the training process. Example:
<property> <name> oracle.ord.hadoop.recognizer.cascadeclassifier.scalefactor</name> <value>1.1</value> </property>
String. XML file containing classifiers for face. The feature can be used with any of the frontal face pre-trained classifiers available with OpenCV. Example:
<property> <name> oracle.ord.hadoop.recognizer.classifier</name> <value>haarcascade_frontalface_alt2.xml</value> </property>
String. Mapping file that maps face labels to directory names and face images. Example:
<property> <name> oracle.ord.hadoop.recognizer.labelnamefiler</name> <value>haarcascade_frontalface_alt2.xml</value> </property>
String. File containing the model generated in the training step. The file must be in a shared location, accessible by all cluster nodes. Example:
<property> <name> oracle.ord.hadoop.recognizer.modelfile</name> <value>myface_model.dat</value> </property>
You can implement and install custom modules for multimedia decoding and processing.
You can use a custom video decoder in the framework by implementing the abstract class
oracle.ord.hadoop.decoder.OrdFrameGrabber. See the Javadoc for additional details. The product includes two implementations of the video decoder that extend
OrdFrameGrabber for JCodec and FFMPEG (requires a separate installation of FFMPEG).
You can use custom multimedia analysis in the framework by implementing two abstract classes.
oracle.ord.hadoop.mapreduce.OrdFrameProcessor<K1,V1,K2,V2>. The extended class of
OrdFrameProcessor is used in the map phase of the MapReduce job that processes the video frames or images. (K1, V1) is the input key-value pair types and (K2, V2) is the output key-value pair type. See the Javadoc for additional details. The product includes an implementation using OpenCV.
oracle.ord.hadoop.mapreduce.OrdOutputProcessor<K1,V1,K2,V2>. The extended class of
OrdFrameProcessor is used in the reducer phase of the MapReduce job that processes the video frames or images. (K1, V1) is the input key-value pair types and (K2, V2) is the output key-value pair type. See the Javadoc for additional details. Most implementations do not require implementing this class.
An example of framework configuration parameters is available in
If the output is displayed as images, oracle.ord.hadoop.OrdPlayImages can be used to display all the images in the output HDFS directory. This will display the image frames marked with labels for identified faces. For example:
$ java oracle.ord.hadoop.demo.OrdPlayImages –hadoop_conf_dir $HADOOP_CONF_DIR –image_file_dir voutput