morzel.net

.net, js, html, arduino, java... no rants or clickbaits.

Detecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 2

OVERVIEW

In Part 1 you have learned what OpenCV is, what is the role of Emgu CV wrapper and how to create a Visual Studio 2017 C# project that utilizes the two libraries. In this part I will show you how to loop through frames captured from video file. Check the first part to watch demo video and find information about sample project (all the interesting stuff is inside Program.cs  - keep this file opened in separate tab as its fragments will be shown in this post)...

 

STEP 1: Capturing video from file

Before any processing can happen we need to obtain a frame from video file. This can be easily done by using VideoCapture class (many tutorials mention Capture class instead but it is not available in recent Emgu versions).

Check the Main method from our sample project:

private const string BackgroundFrameWindowName = "Background Frame";
// ...
private static Mat backgroundFrame = new Mat(); // Frame used as base for change detection
// ...

static void Main(string[] args)
{
    string videoFile = @"PUT A PATH TO VIDEO FILE HERE!";

    using (var capture = new VideoCapture(videoFile)) // Loading video from file
    {
        if (capture.IsOpened)
        {
            // ...

            // Obtaining and showing first frame of loaded video (used as the base for difference detection)
            backgroundFrame = capture.QueryFrame();
            CvInvoke.Imshow(BackgroundFrameWindowName, backgroundFrame);

            // Handling video frames (image processing and contour detection)
            VideoProcessingLoop(capture, backgroundFrame);
        }
        else
        {
            Console.WriteLine($"Unable to open {videoFile}");
        }
    }
}

VideoCapture has four constructor versions. The overload we are using takes string parameter that is a path to video file or video stream. Other versions allow us to connect to cameras. If you design your program right, switching from file input to a webcam might be as easy as changing new VideoCapture call!

Once VideoCapture instance is created we can confirm if opening went fine by accessing IsOpened property (maybe path is wrong or codecs are missing?).

VideoCapture offers few ways of acquiring frames but the one I find most convenient is by call to QueryFrame method. This method returns Mat class instance (you know it already from part 1) and moves to next frame. If next frame cannot be found then null is returned. We can use this fact to easily loop through video. 

 

STEP 2: Loading and presenting background frame

Our drone detection project is based on finding the difference between background frame and other frames. The assumption is that we can treat the first frame obtained from the video as the background, hence the call to QueryFrame right after creating VideoCapture object:

 backgroundFrame = capture.QueryFrame();

After background is loaded we can check how it looks with a call to Imshow method (you know it from part 1 too):

CvInvoke.Imshow(BackgroundFrameWindowName, backgroundFrame);

Is finding a (meaningful!) difference in a video always as easy as subtracting frames? No, it isn't. First of all the background might not be static (imagine that drone was flying in front of threes moved by wind or if lighting in a room was changing significantly). The second challenge might come from movements of the camera. Having a fixed background and camera position keeps our drone detection task simple enough for beginner's OpenCV tutorial plus it's not completely unrealistic. Video detection/recognition is often used in fully controlled environment such as part of factory... OpenCV is capable of handling more complex scenarios - you can read about background subtraction techniques and optical flow to get a hint...

 

STEP 3: Looping through video frames

We know that we can use QueryFrame to get single frame image (Mat instance) and progress to next frame and we know that QueryFrame returns null if it can't go any further. Let's use this knowledge to build a method that goes through frames in a loop:

private static void VideoProcessingLoop(VideoCapture capture, Mat backgroundFrame)
{
    var stopwatch = new Stopwatch(); // Used for measuring video processing performance

    int frameNumber = 1;
    while (true) // Loop video
    {
        rawFrame = capture.QueryFrame(); // Getting next frame (null is returned if no further frame exists)

        if (rawFrame != null) 
        {
            frameNumber++;

            stopwatch.Restart();
            ProcessFrame(backgroundFrame, Threshold, ErodeIterations, DilateIterations);
            stopwatch.Stop();

            WriteFrameInfo(stopwatch.ElapsedMilliseconds, frameNumber);
            ShowWindowsWithImageProcessingStages();

            int key = CvInvoke.WaitKey(0); // Wait indefinitely until key is pressed

            // Close program if Esc key was pressed (any other key moves to next frame)
            if (key == 27)
                Environment.Exit(0);
        }
        else
        {
            capture.SetCaptureProperty(CapProp.PosFrames, 0); // Move to first frame
            frameNumber = 0;
        }
    }
}

In each loop iteration a frame is grabbed from video file. It is then passed to ProcessFrame method which does image difference, noise removal, contour detection and drawing (it will be discussed in detail in the next post)... Call to ProcessFrame is surrounded with System.Diagnostics.Stopwatch usage - this way we can measure video processing performance. It took my laptop only about 1.5ms to fully handle each frame - I've told you OpenCV is fast! :)

If QueryFrame returns null then program moves back to first frame by calling SetCaptureProperty method on VideoCapture instance (video will be processed again).

WriteFrameInfo puts a text in the frame's upper-left corner with information about it's number and how long it took to process it. ShowWindowsWithImageProcessingStages ensures that we can see current (raw) frame, background frame, intermediate frames and final frame in separate windows... Both methods will be shown in next post.

The while loop is going to spin forever unless program execution is stopped by Escape key being pressed in any of the windows that show frames (not the console window!). If 0 is passed as WaitKey argument then program waits until some key is pressed. This let's you look at each frame as long as you want. If you pass other number to WaitKey then the program will wait until key is pressed or a delay elapses. You might use it to automatically play video at specified frame rate:

int fps = (int)capture.GetCaptureProperty(CapProp.Fps);
int key = CvInvoke.WaitKey(1000 / fps); // 40ms delay

Warning: One thing you might notice while processing videos is that moving through a file is not always as easy as setting CapProp.PosFrame to desired number. Your experience might vary from format to format. This is because video files are optimized for playing forward at natural speed and frames might not be simply kept as sequence of images. Full HD (1920x1080) movie has over 2 million pixels in each frame. Now let's say we have an hour of video at 30 FPS ->  3600 * 30 * 2,073,600 = 223,948,800,000. Independent frame compression is not enough to crush that number! No wonder some people need to dedicate their scientific/sofware careers to video compression...

Ok, enough for now - next part coming soon!

Detecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 1

INTRO

Emgu CV is a .NET wrapper for OpenCV (Open Source Computer Vision Library) which is a collection of over 2500 algorithms focused on real-time image processing and machine learning. OpenCV lets you write software for:

  • face detection,
  • object identification,
  • motion tracking,
  • image stitching,
  • stereo vision
  • and much, much more...

Open CV is written in highly optimized C/C++, supports multi-core execution and heterogeneous execution platforms (CPU, GPU, DSP...) thanks to OpenCL. The project was launched in 1999 by Intel Research and is now actively developed by open source community members and contributors from companies like Google, Microsoft or Honda...

My experience with Emgu CV/OpenCV comes mostly from working on paintball turret project (which I use to have a break from "boring" banking stuff at work). I'm far from computer vision expert but I know enough to teach you how to detect a mini quadcopter flying in a room:

In the upper-left corner you can see frame captured from video file, following that is the background frame (static background and camera makes our task simpler)... Next to it are various stages of image processing run before drone (contour) detection is executed. Last frame shows original frame with drone position marked. Job done! Oh, and if you are wondering what is the "snow" seen in the video: these are some particles I put to make the video a bit more "noisy"...

I assume that you know a bit about C# programming but are completely new to Emgu CV/OpenCV.

By the end of this tutorial you will know how to:

  • use Emgu CV 3.2 in C# 7 (Visual Studio 2017) application (most tutorials available online are quite outdated!),
  • capture frames from video,
  • find changes between images (diff and binary threshold),
  • remove noise with erode and dilate (morphological operations),
  • detect contours,
  • draw and write on a frame

Sounds interesting? Read on!

 

THE CODE

I plan to give detailed description of the whole program (don't worry: it's just about 200 lines) but if you would like to jump straight to the code visit this GitHub repository: https://github.com/morzel85/blog-post-emgucv. It's a simple console app - I've put everything into Program.cs so you can't get lost!

Mind that because Emgu CV/OpenCV binaries are quite large these are not included in the repo. This should not be a problem because Visual Studio 2017 should be able to automatically download (restore) the packages...

Here you can download the video I've used for testing: http://morzel.net/download/emgu_cv_drone_test_video.mp4 (4.04 MB, MPEG4 H264 640x480 25fps).

 

STEP 0: Crating project with Emgu CV

To start lets use Visual Studio Community 2017 to create new console application:

New project... Click to enlarge...

Now we need to add Emgu CV to our project. The easiest way to do it is to use Nuget to install Emgu CV package published by Emgu Corporation. To do so run "Install-Package Emgu.CV" command in Package Manager Console or utilize Visual Studio UI:

Adding Nuget package... Click to enlarge...

If all goes well package.config and DLL references should look like this (you don't have to worry about ZedGraph):

Packages and references... Click to enlarge...

Now we are ready to test if OpenCV's magic is available to us through Emgu CV wrapper library. Let's do it by creating super simple program that loads an image file and shows it in a window with obligatory "Hello World!" title:

using Emgu.CV; // Contains Mat and CvInvoke classes

class Program
{
    static void Main(string[] args)
    {
        Mat picture = new Mat(@"C:\Users\gby\Desktop\Krakow_BM.jpg"); // Pick some path on your disk!
        CvInvoke.Imshow("Hello World!", picture); // Open window with image
        CvInvoke.WaitKey(); // Render image and keep window opened until any key is pressed
    }
}

Run it and you should see a window with the image you've selected. Here's what I got - visit Kraków if you like my picture :)

Window with image... Click to enlarge...

Above code loads picture from a file into Mat class instance. Mat is a n-dimensional dense array containing pointer to image matrix and a header describing this matrix. It supports a reference counting mechanism that saves memory if multiple image processing operations act on same data... Don't worry if it sounds a bit confusing. All you need to know now is that we can load images (from files, webcams, video frames etc.) into Mat objects. If you are curious read this nice description of Mat.

The other interesting thing you can see in the code is the CvInvoke class. You can use it to call OpenCV functions from your C# application without dealing with complexity of operating native code and data structures from managed code - Emgu the wrapper will do it for you through PInvoke mechanism.

Ok, so now you have some idea on what Emgu CV/OpenCV libraries are and how to bring them into your application. Next post coming soon...