Miłosz Orzeł

.net, js, html, arduino, java... no rants or clickbaits.

Resizing All ag-Gird (React) Columns

TL;DR

You can use measureText method to calculate widths of text and then use setColumnWidth to adjust columns. This the demo and this is the repo. It's quite likely that you don't need this technique (suppressColumnVirtualisation might be enough), but read on if your grid has huge amount of columns and users need high data density...

 

VIRTUALIZATION VS RESIZE

I've been porting some old UI with grids based on HTML table to ag-Grid. Users appreciate the features added by ag-Grid (such as filters, pinning, sorting, resizing, reordering, grouping...) but noticed one change for worse: not all columns were automatically sized to take the smallest amount of space possible. In enterprise applications data density is really important (not recognizing that is a recipe for failure). Too much white space means that people might miss important information or have to waste time scrolling...

ag-Grid is is capable of handing huge amount of rows and columns thanks to virtualization. The gird creates elements only for rows and columns which are currently visible (plus some buffer) to avoid producing bloated DOM (too many elements severely degrade performance). Virtualization means that the API methods designed to automatically resize columns are only capable of resizing the rendered columns. ag-Grid has suppressColumnVirtualisation option but in my case girds can have as much as 300 columns so turning virtualization off is not possible for performance reasons.

 

THE SOLUTION

So is it possible to resize all columns while enjoying the benefits of virtualization? If your grid cells contain only text and you know the font then yes, it's possible.

You can calculate the width of text for each grid cell and collect the minimal width needed for each of the gird columns. ag-Grid API might then be used to set desired column sizes. Text width calculation could be done efficiently with measureText method on Canvas.

Here's a module that can collect minimal width needed for each column (it doesn't have any dependency on React):

const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');

const collectMaxWidth = (text, group, maxWidths) => {
    const width = context.measureText(text).width;

    const maxWidth = maxWidths.get(group);

    if (maxWidth === undefined || width > maxWidth) {
        maxWidths.set(group, width);
    }
};

const collectMaxWidthCached = (text, group, maxWidths, widthsCache) => {
    const cachedWidth = widthsCache.get(text);

    let width;

    if (cachedWidth === undefined) {
        width = context.measureText(text).width;
        widthsCache.set(text, width);
    } else {
        width = cachedWidth;
    }

    const maxWidth = maxWidths.get(group);

    if (maxWidth === undefined || width > maxWidth) {
        maxWidths.set(group, width);
    }
};

const calculateColumnWidths = config => {
    console.time('Column widths calculation');

    const maxWidths = new Map()

    if (config.measureHeaders) {
        context.font = config.headerFont;

        config.columnDefs.forEach(column => {
            collectMaxWidth(column.headerName, column.field, maxWidths);
        });
    }

    context.font = config.rowFont;

    config.rowData.forEach(row => {
        config.columnDefs.forEach(column => {
            if (config.cache) {
                collectMaxWidthCached(row[column.field], column.field, maxWidths, config.cache);
            } else {
                collectMaxWidth(row[column.field], column.field, maxWidths);
            }
        });
    });

    const updatedColumnDefs = config.columnDefs.map(cd => ({
        ...cd,
        width: Math.ceil(maxWidths.get(cd.field) + config.padding)
    }));

    console.timeEnd('Column widths calculation');

    return updatedColumnDefs;
};

export default calculateColumnWidths;

The module creates canvas element and then a 2d context is retrieved from it. The context is used in collectMaxWidth function to measure size of provided text by invoking context.measureText(text).width

The module also has collectMaxWidthCached function which can offer a performance improvement based on the fact that grid data is often repetitive. If some string was already measured, there's no need to use canvas API again - taking the value from JavaScript Map is super quick. So unless you are extremely worried about memory limits, use the cached version.

The module exports calculateColumnWidths function which takes config object with following properties:

  • columnDefs - Array used to specify ag-Grid columns (limit this if you have hidden columns).
  • rowData - Array of grid records (limit this if you use paging or filtering).
  • measureHeaders - If true then column headers should be taken into account (caveat: header icons are ignored). 
  • headerFont - Determines column header font.
  • rowFont - Determines normal grid cell font.
  • padding - Additional width added to measured text (you need to choose it experimentally, mind varying header icons).
  • cache - JS Map used for speed boost (length of each unique non-header text is measured only once), pass null to skip caching.

Below is an example of button's click handler from my demo app what uses calculateColumnWidths:

const handleResizeWithCustomClick = () => {
    console.time('Resize all columns (including widths calculation)');

    if (gridApi && gridColumnApi) {
        // Here ALL columns and rows are used because there are no hidden columns
        // and the the grid has neither paging nor filtering enabled!
        const updatedColumDefs = calculateColumnWidths({
            columnDefs,
            rowData,
            measureHeaders: true,
            headerFont: 'bold 12px Arial',
            rowFont: 'normal 12px Arial',
            padding: 30,
            cache: useWidthsCache ? textWidthsCache : null
        });

        // Setting width by setColumnWidth has the advantage of preserving column
        // changes done by user such as sorting or filters. The disadvantage is that
        // initial resize might be slow if the grid was scrolled towards later columns
        // before resizing was invoked (bug in the gird?).
        // Resize by gridApi.setColumnDefs(updatedColumDefs) or setColumnDefs(updatedColumDefs)
        // should be faster but columns settings could be reset (mind deltaColumnMode)...            
        updatedColumDefs.forEach(def => gridColumnApi.setColumnWidth(def.field, def.width));
    }

    console.timeEnd('Resize all columns (including widths calculation)');
};

Notice how setColumnWidth from ag-Grid Column API is used to apply calculated width. Mind the comments about paging/filtering and the difference between using setColumnWidth and calling Grid API setColumnDefs or updating state bound to grid's columnDefs property...

 

DEMO APP

Live demo: https://morzel85.github.io/blog-post-ag-grid-full-resize 
Source code on GitHub: https://github.com/morzel85/blog-post-ag-grid-full-resize

The app uses React 16.13.1 and ag-Grid Community 23.0.1 (I've tested it in Chrome 80, Firefox 74, Edge 44).
Clone the repo, do npm install and npm start to run the app locally (just like with any other thing started with create-react-app)...

Usage:
Click on "Generate data (100 rows with 300 columns)" button to create 30K grid cells. You should then click the "Resize columns with columnApi.autoSizeColumns" button and scroll to the right to notice that only visible columns were resized:

Resizing by autoSizeColumns... Click to enlarge...


Now reload the page, click data generation again and press "Resize columns with custom text measure" and scroll the gird horizontally. You should notice that all columns were resized:

Resizing by measureText... Click to enlarge...


Clicking resizing button the the second time should give you better performance because of text measure caching and because ag-Grid itself has less work to do. These are the timings from Chrome: 

Resizing perfromance in Chrome... Click to enlarge...
I would say that 100ms for 30K cell measure is quite fast! With caching it drops to 5ms! You may also notice that most of the time is not really spent in my text measuring function but in the ag-Grid which handles columns resize. 

I've noticed one weird performance issue: if you scroll to the right of the grid before clicking the resize button, the time spent by ag-Grid handling columnApi.autoSizeColumns calls increases significantly. Doing bulk resize with gridApi.setColumnDefs or by updating array used in for columnDefs property solves the performance issue at the cost of column settings reset (exact behavior depends on deltaColumnMode)...

ag-Grid API Access with Hooks

TL;DR

Utilize useState or useRef hooks and gridReady event to maintain access to Grid API and Column API. Check here for code samples.

 

AG-GRID IN REACT 16.8.0+

ag-Grid has an impressive set of features, matched by unusually comprehensive documentation. The issue is that at this moment (February 2020), most of the React-specific examples are written for class components. Because of this, using the grid inside a function (functional) component could be slightly challenging. One of the things that are not so obvious is how to access Grid API and Column API...

Grid and Column APIs have over 150 methods that let you manipulate everything from cell focus to column width. The docs suggest obtaining and saving references to these APIs inside gridReady event handler:

class GridExample extends Component {
  // Irrelevant parts removed for brevity... 

  onGridReady = params => {
    this.gridApi = params.api;
    this.gridColumnApi = params.columnApi;
  }

  render() {
    return (
		<AgGridReact          
            onGridReady={this.onGridReady}            
        />  
    );
  }	
}

Such approach works nicely in a class component but doesn't work when component is "just" a function. Worry not! useState and useRef hooks (part of React 16.8.0) can help you access the APIs in a function component and it's all quite easy

I've asked ag-Grid team if they have a preference between useState and useRef and they don't, so I will show you the two options and you can pick the one that suites you.

Here's a component that uses selectAll method from Grid API and moveColumnByIndex method from Column API with the help of useState hook:

import React, { useState } from "react";
import { AgGridReact } from "ag-grid-react";

import { columnDefs } from "./columnDefs";
import { rowData } from "./rowData";

import "ag-grid-community/dist/styles/ag-grid.css";
import "ag-grid-community/dist/styles/ag-theme-balham.css";

const AgGridWithUseState = () => {
  console.log("AgGridWithUseState Render");

  const [gridApi, setGridApi] = useState();
  const [columnApi, setColumnApi] = useState();

  return (
    <div className="example">
      <h1>API access with useState</h1>
      <div>
        <button
          onClick={() => gridApi && gridApi.selectAll()}>
          Grid API selectAll()
        </button>
        <button 
          onClick={() => columnApi && columnApi.moveColumnByIndex(0, 1)}>
          Column API moveColumnByIndex(0, 1)
        </button>
      </div>
      <div className="ag-theme-balham">
        <AgGridReact
          columnDefs={columnDefs}
          rowData={rowData}
          rowSelection="multiple"
          onGridReady={params => {
            console.log("AgGridWithUseState Grid Ready");
            setGridApi(params.api);
            setColumnApi(params.columnApi);
          }}
        />
      </div>
    </div>
  );
};

export default AgGridWithUseState;

and below is the exact same thing done with useRef:

import React, { useRef } from "react";
import { AgGridReact } from "ag-grid-react";

import { columnDefs } from "./columnDefs";
import { rowData } from "./rowData";

import "ag-grid-community/dist/styles/ag-grid.css";
import "ag-grid-community/dist/styles/ag-theme-balham.css";

const AgGridWithUseRef = () => {
  console.log("AgGridWithUseRef Render");

  const gridApi = useRef();
  const columnApi = useRef();

  return (
    <div className="example">
      <h1>API access with useRef</h1>
      <div>
        <button
          onClick={() => gridApi.current && gridApi.current.selectAll()}>
          Grid API selectAll()
        </button>
        <button
          onClick={() => columnApi.current && columnApi.current.moveColumnByIndex(0, 1)}>
          Column API moveColumnByIndex(0, 1)
        </button>
      </div>
      <div className="ag-theme-balham">
        <AgGridReact
          columnDefs={columnDefs}
          rowData={rowData}
          rowSelection="multiple"
          onGridReady={params => {
            console.log("AgGridWithUseRef Grid Ready");
            gridApi.current = params.api;
            columnApi.current = params.columnApi;
          }}
        />
      </div>
    </div>
  );
};

export default AgGridWithUseRef;

Check the function assigned to onGridReady property to find out how to retrieve and keep references to the APIs. Button's onClick handlers show you how to use the APIs.

 

USEREF VS USESTATE (DEMO)

Live demo: https://morzel85.github.io/blog-post-ag-grid-api-access-with-hooks 
Source code on GitHub: https://github.com/morzel85/blog-post-ag-grid-api-access-with-hooks

The app uses React 16.12.0 and ag-Grid Community 22.1.1 (I've tested it in Chrome 80, Firefox 73, Edge 44).
Clone the repo, do npm install and npm start to run the app locally (just like with any other thing started with create-react-app)...

Demo shows access to selectAll method from Grid API and moveColumnByIndex method from Column API. This application has two grids that differ only in the way the APIs are accessed (one uses useState hook, the other goes with useRef hook).

What's the difference between useState and useRef version?

It boils down to the way hooks are designed to work. useState causes component rerender and useRef doesn't. You can see the difference if you open the console and check the log:

useState vs useRef - render calls

Notice two "AgGridWithUseState Render" lines after "AgGridWithUseState Grid Ready".

This happens because Grid and Column APIs are set by two calls to state hook setters and that causes component rerenders. Sounds bad? Remember that in React, a component rerender doesn't necessary mean any DOM change has to be applied. ag-Grid is also smart enough to avoid unnecessary work. useState triggers rendering by design and it lets you write the component in more declarative way. For example you could keep rowData and columnDefs in state and have the grid update itself without explicit API calls...

How about the useRef approach? Use it if you don't need to run any action as a result of API reference being set.

CSS Grid - Site for Rapid Experimentation

 

PAGE LAYOUT

Making website layout used to be hard... Frames, tables, floats, display: table, vertical-align, negative margins, absolute positioning... you name it! People & frameworks had to abuse HTML/CSS features just to center a div (let alone do a responsive page setup)... Fortunately the dark ages are over, Flexbox is supported for quite a while and about two years ago mayor browsers implemented Grid. It took 30 years, but website layout problem is solved (almost, still waiting for Subgrid).

If you haven't heard about CSS Grid here's a couple of great resources:

 

QUICK EXPERIMENTS

Above sites (and myriad of others) are doing a great job of describing grid features, so I'm not going to do that here. Instead I will share the page I made while learning grid: https://morzel85.github.io/css-grid-playground (ups, I've just noticed that https://www.cssgridplayground.com exists, the more playgrounds the better?)

My site is not a grid generator but rather a place were you can quickly test effects of various CSS Grid properties. Play with properties and immediately see the result (you may also want to open browser devtools to see how styles are actually applied to container and items)... If you are lost don't worry, just reload the page to get to initial state.

Here's how the site looks:

CSS Grid Playground - Rapid Experimentation... Click to enlarge...

On left-hand side you can control properties of grid container (div with grid-container class). In the middle you can set the amount of grid items (divs with grid-item class) and see the resulting grid. If you click an item, and additional panel opens where you can control specific grid item properties. If you are not sure what a property does, hover over it to see a brief description and example values.

I've tested the page in Chrome 75, Firefox 68 and Edge 44.

 

SUPPORTED PROPERTIES

CSS Grid is quite complex, but if think that my tool does a decent job of demonstrating its most important features. These are the supported properties:

grid-template-rows
Defines the amount and sizes of explicit grid rows.

grid-template-columns
Defines the amount and sizes of explicit grid columns.

grid-auto-rows
Defines sizes of implicit grid rows. Implicit rows are created when amount of items exceeds the amount of explicit rows (rows defined by template) or when an item placement goes outside the range of explicit rows.

grid-template-areas
Defines named grid areas which can be associated with particular grid items by using placement properties: grid-row-start, grid-row-end, grid-column-startgrid-column-end (or their shorthands: grid-row, grid-column and grid-area). Areas for each row are specified between quotes, white-space might be added to increase readability.

grid-auto-columns
Defines sizes of implicit grid columns. Implicit columns are created when amount of items exceeds the amount of explicit columns (columns defined by template) or when an item placement goes outside the range of explicit columns.

grid-auto-flow
Defines placement algorithm for items which are not specifically positioned (for example by using grid-column or grid-area).

row-gap
Defines space (gutter) size between grid rows. Older implementations used grid-row-gap name. gap shorthand property can be used to specify row and column gutters at the same time.

column-gap
Defines space (gutter) size between grid columns. Older implementations used grid-column-gap name. gap shorthand property can be used to specify row and column gutters at the same time.

justify-items
Defines placement of items in grid cells along inline (main) axis (horizontally, except for vertical writing mode). Value set on a container can be overridden by justify-self on an item. place-items shorthand property can be used to specify both align and justify at the same time.

align-items
Defines placement of items in grid cells along block (cross) axis (vertically, except for vertical writing mode). Value set on a container can be overridden by aling-self on an item. place-items shorthand property can be used to specify both align and justify at the same time.

justify-content
Defines placement of grid cells along inline (main) axis (horizontally, except for vertical writing mode). The setting will not have any effect if there is no space left on inline axis (for example when some column is defined as 1fr)! place-content shorthand property can be used to specify both align and justify at the same time.

align-content
Defines placement of grid cells along block (cross) axis (vertically, except for vertical writing mode). The setting will not have any effect if there is no space left on block axis (for example when some row is defined as 1fr)! place-content shorthand property can be used to specify both align and justify at the same time.

grid-row
A shorthand property that controls item row placement by setting grid-row-start, and grid-row-end values respectively. If placement goes outside of explicitly defined grid rows, implicit rows will be created.

grid-column
A shorthand property that controls item column placement by setting grid-column-start, and grid-column-end values respectively. If placement goes outside of explicitly defined grid columns, implicit columns will be created.

grid-area
A shorthand property that controls item placement by setting grid-row-startgrid-column-start, grid-row-end and grid-column-end values respectively. The setting is especially helpful when grid-template-areas "ascii-art" is used. If placement goes outside of explicitly defined grid, implicit row or columns will be created.

justify-self
Places the item inside grid cell along inline (main) axis (horizontally, except for vertical writing mode). The setting takes precedence over justify-items specified on container.

align-self
Places the item inside grid cell along block (cross) axis (vertically, except for vertical writing mode). The setting takes precedence over align-items specified on container.

order
Allows reordering of automatically placed item. By default items have order 0. The higher the order the later the item will appear. If other items have default order, setting to 1 will move the item to the end, and setting to -1 will move the item to the beginning.

Legacy Apps - Dealing with IFRAME Mess (Window.postMessage)

It's October 2018 so I should probably write something about React 16.5, Angular 7.0 or Blazor 0.6... But... How about some fun with iframe-infested legacy application instead? ;)

TL;DR

You can pass a message to embedded iframe with:

someIframe.contentWindow.postMessage({ a: 'aaa', b: 'bbb' }, '*');

and to parent of an iframe like this:

window.parent.postMessage({ a: 'aaa', b: 'bbb' }, '*');

This is how you can receive the message:

window.addEventListener('message', function (e) {
    // Do something with e.data...         
});  

Mind the security! See my live example and its code on GitHub or go for framebus library if you want some more features.

THE IFRAMES

Unless you work for a startup, chances are that part of your duties is to keep some internal web application alive and this app remembers the glorious era of IE 6. It’s called work for a reason, right? ;) In the old days iframes were used a lot. Not only for embedding content from other sites, cross domain ajax or hacking an overlay that covered selects but also to provide boundaries between page zones or mimic desktop-like windows layout…

So let’s assume that you have site with nested iframes where you need to modify state of one iframe based on action that happened in another iframe:

Page with many iframes...

In the example above, top iframe 0 has a text field and if Update User Name button is clicked we should modify User Name labels in nested iframe 1a and iframe 1b. When Update Account Number button is pressed Account Number in deeply nested iframe 2a should change. Clicking on Update News should modify text in iframe 2b. That last iframe contains a Clear News button, and when it's clicked a notification should be passed to top iframe 0...

DIRECT ACCESS (THE BAD WAY)

One way of implementing interactions between iframes is through direct access to nested/parent iframe's DOM elements (if same-origin policy allows). State of element in nested iframe can modified be such code:

document.getElementById('someIframe').contentWindow.document.getElementById('someInput').value = 'test';

and reaching element in parent can be done with:

window.parent.document.getElementById('someInput').value = 'test';

The problem with this approach is that it tightly couples iframes and that’s unfortunate since the iframes were likely used to provide some sort of encapsulation. Direct DOM access has another flaw: it gets really nasty in case of deep nesting: window.parent.parent.parent.document...

MESSAGING (THE GOOD WAY)

Window.postMessage metod was introduced into browsers to enable safe cross-origin communication between Window objects. The method can be used to pass data between iframes. In this post I’m assuming that the application with iframes is old but it can be run in Internet Explorer 11, which is the last version that Microsoft released (in 2013). From what I’ve seen it’s often the case that IE has to be supported but at least it’s the latest version of it. Sorry if that assumption doesn’t work for you, I’ve suffered my share of old IE support... 

Thanks to postMessage method it’s very easy to create a mini message bus so events triggered in one iframe can be handled in another if the target iframe chooses to take an action. Such approach reduces coupling between iframes as one frame doesn't need to know any details about elements of the other...

Take a look at an example function that can send a messages down to all directly nested iframes:

const sendMessage = function (type, value) {
     console.log('[iframe0] Sending message down, type: ' + type + ', value: ' + value);

     var iframes = document.getElementsByTagName('iframe');
     for (var i = 0; i < iframes.length; i++) {
         iframes[i].contentWindow.postMessage({ direction: 'DOWN', type: type, value: value }, '*');
     }
};

In the code above, iframes are found with document.getElementsByTagName and then a message is sent to each of them through contentWindow.postMessage call. First parameter of postMessage method is the message (data) we want to pass. Browser will take care of its serialization and its up to you to decide what needs to be passed. I've chosen to pass an object with 3 properties: first designate in which direction message should go (UP or DOWN), second states the message type (UPDATE_USER for example) and the last one contains the payload of the message. In the case of our sample app it will be a text user put into input and which should affect elements in nested iframes. The '*' value passed to contentWindow method determines how browser dispatches the event. Asterisk means no restrictions - it's ok for our code sample but in real world you should consider providing an URI as the parameter value so browser will be able to restrict the event based on scheme, host name and port number. This is a must in case you need to pass sensitive data (you don't want to show it to any site that got loaded into iframe)!

This is how sendMessage function can be used to notify nested iframes about the need to update user info:

document.getElementById('updateUserName').addEventListener('click', function (event) {
    sendMessage('UPDATE_USER', document.getElementById('textToSend').value);
});

Code shown above belongs to iframe 0 which contains two nested iframes: 1a and 1b. Below is the code from iframe 1b which can do two things: handle a message in case it is interested in it or just pass it UP or DOWN:

window.addEventListener('message', function (e) {
    console.log('[iframe1b] Message received');

    if (e.data.type === 'UPDATE_USER') {
        console.log('[iframe1b] Handling message - updating user name to: ' + e.data.value);
        document.getElementById('userName').innerText = e.data.value;
    } else {
        if (e.data.direction === 'UP') {
            console.log('[iframe1b] Passing message up');
            window.parent.postMessage(e.data, '*');
        } else {
            console.log('[iframe1b] Passing message down');
            document.getElementById('iframe2b').contentWindow.postMessage(e.data, '*');
        }
    }               
});

You can see that messages can be captured by listening to message event on window object. Passed message is available in event's data field, hence the check for e.data.type is done to see if code should handle the message or just pass it. Passing UP is done with window.parent.postMessage, passing DOWN works with contentWindow.postMessage called on an iframe element. 

iframe 2b has a button with following click handler:

document.getElementById('clearNews').addEventListener('click', function () {
   document.getElementById('news').innerText = '';

   console.log('[iframe2b] News cleared, sending message up, type: NEWS_CLEARED');
   window.parent.postMessage({ direction: 'UP', type: 'NEWS_CLEARED' }, '*');
);

It clears news text and sends notification to parent window (iframe). This message will be received by iframe 1b and passed up to iframe 0 which will handle it by displaying 'News cleared' text: 

window.addEventListener('message', function (e) {
    console.log('[iframe0] Message received');

    if (e.data.type === 'NEWS_CLEARED') {
        console.log('[iframe0] Handling message - notifying about news clear');
        document.getElementById('newsClearedNotice').innerText = 'News cleared!';
    }
});

Notice that this time message handler is quite simple. This is because in the case of top iframe 0 we don't want to pass received messages. 

EXAMPLE

That's it. Here's a working sample of iframe "rich" page. Open browser console to see how messages fly around. Check the repo to see the code, it's vanilla JS with no fancy features since we assumed that IE 11 has to be directly supported (checked also in Firefox 62, Chrome 69 and Edge 42).

Detecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 3

OVERVIEW

Part 1 introduced you to OpenCV and its Emgu CV wrapper library plus showed the easiest way to create Emgu project in Visual Studio 2017. Part 2 was all about grabbing frames from video file. The third (and last) episode focuses on image transformations and contour detection...

If case you forgot: here's the complete code sample on GitHub (focus on Program.cs file as it contains all image acquisition and processing code). This is the app in action:

 

STEP 4: Difference detection and noise removal

In previous post you've seen that VideoProcessingLoop method invoked ProcessFrame for each frame grabbed from video, here's the method:

// Determines boundary of brightness while turning grayscale image to binary (black-white) image
private const int Threshold = 5;

// Erosion to remove noise (reduce white pixel zones)
private const int ErodeIterations = 3;

// Dilation to enhance erosion survivors (enlarge white pixel zones)
private const int DilateIterations = 3;

// ...

// Containers for images demonstrating different phases of frame processing 
private static Mat rawFrame = new Mat(); // Frame as obtained from video
private static Mat backgroundFrame = new Mat(); // Frame used as base for change detection
private static Mat diffFrame = new Mat(); // Image showing differences between background and raw frame
private static Mat grayscaleDiffFrame = new Mat(); // Image showing differences in 8-bit color depth
private static Mat binaryDiffFrame = new Mat(); // Image showing changed areas in white and unchanged in black
private static Mat denoisedDiffFrame = new Mat(); // Image with irrelevant changes removed with opening operation
private static Mat finalFrame = new Mat(); // Video frame with detected object marked

// ...

private static void ProcessFrame(Mat backgroundFrame, int threshold, int erodeIterations, int dilateIterations)
{
    // Find difference between background (first) frame and current frame
    CvInvoke.AbsDiff(backgroundFrame, rawFrame, diffFrame);

    // Apply binary threshold to grayscale image (white pixel will mark difference)
    CvInvoke.CvtColor(diffFrame, grayscaleDiffFrame, ColorConversion.Bgr2Gray);
    CvInvoke.Threshold(grayscaleDiffFrame, binaryDiffFrame, threshold, 255, ThresholdType.Binary);

    // Remove noise with opening operation (erosion followed by dilation)
    CvInvoke.Erode(binaryDiffFrame, denoisedDiffFrame, null, new Point(-1, -1), erodeIterations, BorderType.Default, new MCvScalar(1));
    CvInvoke.Dilate(denoisedDiffFrame, denoisedDiffFrame, null, new Point(-1, -1), dilateIterations, BorderType.Default, new MCvScalar(1));

    rawFrame.CopyTo(finalFrame);
    DetectObject(denoisedDiffFrame, finalFrame);
}

AbsDiff and CvtColor

First the current frame (kept in rawFrame) is compared to background with CvInvoke.AbsDiff. In other words: current frame is subtracted from background frame pixel by pixel (absolute value is used to avoid negative results). After that the difference image is converted into grayscale with CvInvoke.CvtColor call. We care only about overall pixel difference (not it's individual blue-green-red color components). The whiter the pixel is the more it color has changed... Take a look at below picture showing background frame, current frame and the grayscale difference:

Frames difference... Click to enlarge...

 

Threshold

Grayscale image is changed into and image with only black and white (binary) pixels with the use of CvInvoke.Threshold. Our intention is to mark changed pixels as white. Threshold value allows as to control change detection sensitivity. Below you can see how different thresholds produce various binary results:

Threshold difference... Click to enlarge...

First image (left) was produced with Threshold = 1, so even the slightest change got marked - such image is not a suitable input for contour detection. Image in the middle used Threshold = 5. Drone position is clearly marked and smaller white pixel zones can be removed with erosion... Last image (right) is the result of setting Threshold to = 200. This time sensitivity was too low and we got just a couple of white pixels. 

Erode and Dilate

It's hard to find a threshold that gives desired difference detection for each video frame. If threshold is too low then too many pixels are white, if it's too high then the drone might not be detected at all... The best is to pick a threshold which marks the change we need even if we get a bit of undesired white spots as these can be safely removed with erosion followed by dilation. When combined, these two operations create so called opening operation which can be treated as a noise removal method. Opening is a type of morphology transformation (read this article to learn more about them). These operations work by probing pixel neighborhood with a structuring element (aka kernel) and deciding pixel value based on the values found in the neighborhood...

CvInvoke.Erode is meant to simulate a physical process in which object area is reduced due to destructive effects of its surroundings. Detailed behavior depends on the parameters passed (structuring element, anchor, border type - never mind, this is a beginners guide, right?) but the general idea is like this: if pixel is white but has a black pixel around it then it should become black too. The more erode iterations are run the more white pixel zones get eaten away. Here's an example of image erosion:

Erosion... Click to enlarge...

On the left is the input image and on the right we have the result of erosion which used structuring element in the shape of a 3x3 square (this is the default value used when null is passes for element parameter in Erode invocation). The value of output pixel was decided by probing all neighboring pixels and checking for their minimal value. If a pixel was white in the input image but had at least one black pixel in its immediate surroundings then it became black in the output image.

If erosion is used wisely we can get rid of irrelevant white pixels. But there's a catch: the white pixel zone that marks the drone is reduced too. Don't worry: dilation is going to help us! Just like pupils in your eyes are enlarged (dilated) when it gets dark the white pixel zones that survived erosion can get enlarged too... Again details vary by CvInvoke.Dilate parameters but generally speaking: if pixel is black but has while neighbor it gets white too. The more iterations are run the more white zones are enlarged. This is an example of dilation:

Dilation... Click to enlarge...

On the left we have the same input image as used in erosion example and on the right we can see the result of single call to Dilate method (again with 3 by 3 kernel matrix). Notice how pixel obtains the maximal value of its surroundings (if any neighboring pixel is white it becomes white too)...

Erosion followed by dilation is such a common transformation that OpenCV has methods that combine the two into one opening operation but using separate Erode and Dilate calls gives you a bit more control. Below you can see how opening cleared the noise and enhanced white spot that marks drone position:

Opening... Click to enlarge...

 

STEP 5: Contour detection

Once all above image preparation steps are done we have a binary image which is suitable input for contour detection provided by DetectObject method:

private static void DetectObject(Mat detectionFrame, Mat displayFrame)
{
    using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
    {
        // Build list of contours
        CvInvoke.FindContours(detectionFrame, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);

        // Selecting largest contour
        if (contours.Size > 0)
        {
            double maxArea = 0;
            int chosen = 0;
            for (int i = 0; i < contours.Size; i++)
            {
                VectorOfPoint contour = contours[i];

                double area = CvInvoke.ContourArea(contour);
                if (area > maxArea)
                {
                    maxArea = area;
                    chosen = i;
                }
            }

            // Draw on a frame
            MarkDetectedObject(displayFrame, contours[chosen], maxArea);
        }
    }
}

The method takes binary difference image (detectionFrame) on which contours will be detected and another Mat instance (displayFrame) on which detected object will be marked (it's a copy of unprocessed frame)...

CvInvoke.FindContours takes the image and runs contour detection algorithm to find boundaries between black (zero) and white (non-zero) pixels on 8-bit single channel image - our Mat instance with the result of AbsDiff->CvtColor->Threshold->Erode->Dilate suites it just fine! 

A contour is a VectorOfPoint, because image might have many contours we keep them inside VectorOfVectorOfPoint. In case many contours get detected we want to pick the largest of them. This is easy thanks to a CvInvoke.ContourArea method...

Read the docs about contour hierarchy and approximation methods if you are curious about RetrType.List, ChainApproxMethod.ChainApproxSimple enum values seen in CvInvoke.FindContours call. This is a good read too...

 

STEP 6: Drawing and writing on a frame

Once we've found the drone (that is we have a contour that marks its position) it would be good to present this information to the user. This is done by MarkDetectedObject method:

private static void MarkDetectedObject(Mat frame, VectorOfPoint contour, double area)
{
    // Getting minimal rectangle which contains the contour
    Rectangle box = CvInvoke.BoundingRectangle(contour);

    // Drawing contour and box around it
    CvInvoke.Polylines(frame, contour, true, drawingColor);
    CvInvoke.Rectangle(frame, box, drawingColor);

    // Write information next to marked object
    Point center = new Point(box.X + box.Width / 2, box.Y + box.Height / 2);

    var info = new string[] {
        $"Area: {area}",
        $"Position: {center.X}, {center.Y}"
    };

    WriteMultilineText(frame, info, new Point(box.Right + 5, center.Y));
}

The method uses CvInvoke.BoundingRectangle to find minimal box (Rectangle) that surrounds the entire contour. Box is later drawn with a call to CvInvoke.RectangleThe contour itself is plotted by CvInvoke.Polylines method which takes a list of points that describe the line. You can notice that both drawing methods receive drawingColor parameter, it is an instance of MCvScalar defined this way:

private static MCvScalar drawingColor = new Bgr(Color.Red).MCvScalar;

Bgr structure constructor can take 3 values that define its individual color components or it can take a Color structure like in my example.

Important: Point, Rectangle and Color structures come from System.Drawing assembly which by default is not included in new console application template so you need to add reference to System.Drawing.dll yourself.

Information about detected object location is written by WriteMultilineText helper method (the same method is used to print info about frame number and processing time). This is the code:

private static void WriteMultilineText(Mat frame, string[] lines, Point origin)
{
    for (int i = 0; i < lines.Length; i++)
    {
        int y = i * 10 + origin.Y; // Moving down on each line
        CvInvoke.PutText(frame, lines[i], new Point(origin.X, y), FontFace.HersheyPlain, 0.8, drawingColor);
    }
}

In each invocation of CvInvoke.PutText method y coordinate of the line is increased so lines are not colliding with each other...

This is how frame captured from video looks like after drawing and writing is applied:

Drone marking... Click to enlarge...

 

STEP 7: Showing it all

In part 1 you've seen that CvInvoke.Imshow method can be used to present a window with an image (Mat instance). Below method is called for every video frame so the user has a chance to see various stages of image processing and the final result:

private static void ShowWindowsWithImageProcessingStages()
{
    CvInvoke.Imshow(RawFrameWindowName, rawFrame);
    CvInvoke.Imshow(GrayscaleDiffFrameWindowName, grayscaleDiffFrame);
    CvInvoke.Imshow(BinaryDiffFrameWindowName, binaryDiffFrame);
    CvInvoke.Imshow(DenoisedDiffFrameWindowName, denoisedDiffFrame);
    CvInvoke.Imshow(FinalFrameWindowName, finalFrame);
}

Displaying intermediate steps is a great debugging aid for any image processing application (I didn't show separate windows for erosion and dilation because only 6 windows of my test video fit on full HD screen):

All windows... Click to enlarge...

 

SUMMARY

This three part series assumed that you were completely new to image processing with OpenCV/Emgu CV. Now you have some idea what these libraries are and how to use them in Visual Studio 2017 project while following a coding approach recommended for version 3 of the libs...

You've learned how to grab frames from video and how to prepare them for contour detection using fundamental image processing operations (difference, color space conversion, thresholding and morphological transformations). You also know how to draw shapes and text on an image. Good job!

Computer vision is a complex yet very interesting topic (its importance is constantly increasing), you've just made a first step in this field - who knows, maybe one day I will ride in autonomous vehicle powered by your software? :)