Page 1 of 1

Advanced - Deepstack/AI Tool + Blue Iris

Posted: Thu Mar 25, 2021 10:36 pm
by varghesesa
Introduction

Deepstack (deepstack.cc) was a private AI company (December 2018) located in Nigeria that recently open sourced their work (March 2019). The value of AI in surveillance is it improves accuracy to detect objects and reduces false alerts. In the example below, I show the alert list. The top three trigger alerts had Deepstack AI activated. Deepstack was turned off for the bottom two trigger alerts. With Deepstack, objects detected by Deepstack will have the following additional information:
  • The object detected with the confidence level, e.g. Car (92%)
  • Flagged in the clip list to draw attention to those alerts
alert list.PNG
alert list.PNG (137.6 KiB) Viewed 1660 times

Benefits
AI reduces false alerts because the AI will cancel alerts that were once triggered by the wind, changes in sunlight, flies, birds, squirrels, rain etc.
AI improves accuracy as well, because it is possible for the AI to detect objects that would otherwise be missed by the human eye, for example at night.

References
Before jumping into an activity, it is best to do some preliminary research to get an overview of the topic and the tasks at hand. Below are some resources I reviewed which you may find helpful as well.
Architecture
Architecture.png
Architecture.png (23.42 KiB) Viewed 1665 times
From the diagram above there are three components to the Deepstack integration: BlueIris, AI Tool and Deepstack. Each component is a standalone server (process) and communication happens via API calls across the network. My implementation has all three services running on one machine but this solution could be easily scaled to work across many servers.

AI Tool
The AI Tool was first developed by a user named GentlePumpkin. However, due to lack of free time by his own admission, another developer Chris, aka VorlonCD added many useful features and seems to be the popular branch now for the code. GentlePumpkin seems to approve Chris' efforts based on public posts as well. I have listed both GitHub repositories below. The general consensus seems to be the VorlonCD fork has the latest and greatest, so I chose to install from the VorlonCD repository.
Deepstack AI
Deepstack AI seems to be a robust AI open source community. They provide edge AI computing on many platforms including many flavors of Ubuntu, Linux, MacOS, Windows etc. For Windows alone there is the option to install a CPU only version, a GPU (NVidia) version and a dockers container version!

Because I have an Intel based server with an NVidia GTX 1070 Ti graphics card, I decided to choose the GPU install. From that stats, it seems my inference time is 50-60 ms per frame which seems plenty good to me. I would suspect CPU only solutions would be in the 1-2 s per inference, but need to confirm.

Installation

Deepstack AI
Depending on your machine, you can choose the many Windows installation options. As stated earlier, I decided to use the GPU option because of my NVidia GPU. Below is the link to the Windows installation page.

https://docs.deepstack.cc/windows/

Other useful Deepstack resources; According to the documentation, the GPU installation also requires installation CUDA 10.1 and cuDNN. The latest version of CUDA is 11.2. I took a chance with CUDA 11.2 and it seems to be working fine for me. I will share issues if I come across any. For those who are curious, CUDA is an API created by NVidia to make parallel programming on NVidia GPUs easier.

Once installed, it is easy to start/run deepstack (see below). All that is needed is a port number to run the service.

Code: Select all

deepstack --VISION-DETECTION True --PORT ###
It is easy to check whether Deepstack is running as well. From a browser go to the corresponding port:
deepstack.PNG
deepstack.PNG (80 KiB) Viewed 1662 times
AI Tool
  • Backup your current AITOOLS folder. For me, it is G:\BlueIris\AITOOL-VORLONCD. I chose to place executable next to my BI storage.
    BI Directory.PNG
    BI Directory.PNG (9.13 KiB) Viewed 1656 times
  • Download code from github repository: https://github.com/VorlonCD/bi-aidetection
  • Download the ZIP, right-click > Properties > UNBLOCK.
  • Unzip to existing folder, overwriting any files.
  • Most if not all settings should migrate.
Starting AI Tool is simple. Double click on the AI Tool executable in the installation directory.

AI Tool Settings
I am going to reference the blog post: AI Object Detection for Blue Iris – New Method (link below) for guidance on connecting AI Tool to a BI Camera. The Settings and Camera Setup section under the AI Tool section provides detailed instructions.

https://familytechexec.com/blog/ai-moti ... -and-free/

I have one enhancement to the AI Tool Settings recommendations provided by the author. Depending on your scene and what you are trying to accomplish, you may need to enable Masking. I will discuss the importance of Masking further below.

Summary
At this point you should have Blue Iris, Deepstack and AI Tool installed on your machine.
  • To start Deepstack:

    Code: Select all

    deepstack --VISION-DETECTION True --PORT ###
  • To start AI Tool: Double click on the AI Tool executable in the installation directory.
Blue Iris / Deepstack Integration

BI Settings
I am again going to reference the blog post: AI Object Detection for Blue Iris – New Method for guidance on setting up BI. The Blue Iris Setup section provides detailed instructions.

The above article recommends to uncheck “Use secure session keys and login page" in Global settings -> Web server -> Advanced dialog. I am not sure that is necessary. In my local computer deployment it is not necessary.

Overview
DeepstackIntegration.png
DeepstackIntegration.png (18.54 KiB) Viewed 1652 times
The blog post again documents all the setup details.

First let me dissect how the integration works:
  • Blue Iris triggers an alert at 8:26:09 PM on 2021-Mar-25. I have a 5 sec pre-trigger buffer, so alert actually triggered at 8:26:14 PM.
    BI trigger.PNG
    BI trigger.PNG (29.18 KiB) Viewed 1651 times
  • Record settings tell BI to create a JPEG image every 1s in the AI folder which AI Tool starts processing.
    ai-tool-images_optimized.png
    ai-tool-images_optimized.png (89.2 KiB) Viewed 1649 times
  • Deepstack recognizes car at 8:26:23 -> AI tool tells BI to continue trigger.
    deepstack-car-recognized_optimized.png
    deepstack-car-recognized_optimized.png (131.72 KiB) Viewed 1648 times
  • BI finally stops trigger at 8:26:31 which is also the last image sent to AI Tool.
AI and Static Object Gotchas

Static objects are objects that are stationary in a scene for a long time. The most obvious example is a parked car. If you have car recognition activated on your garage camera facing the driveway, if a car is parked in the driveway, then every trigger will be an alert because of the parked car. It is even worse with the Deepstack integration because the current implementation will result in a continuous loop until the car moves.
  • As soon as a car parks, the next trigger on that camera will result in the first alert. The logic then tells BI to continue triggering for a said period of time.
  • The parked car has not removed so extend the trigger for another period of time.
  • This leads to the software continually extending the trigger until the car is eventually moved.
Below is the endless loop I ran into due to the parked car.
endless loop.PNG
endless loop.PNG (224.48 KiB) Viewed 1646 times
Incredibly, the developers have already come out with a masking tool to solve the problem of static objects such as parked cars. I don't know all the details yet, but a tool in the toolkit to keep in mind depending on your scene.
masking.PNG
masking.PNG (12.32 KiB) Viewed 1646 times
In addition, great debugging support is also provided to understand if masks are mistakenly suppressing alerts.
mask debugging.PNG
mask debugging.PNG (140.39 KiB) Viewed 1640 times

Final thoughts

I am very impressed how far AI Tool has come in such a short period of time. BI with AI Tool and Deepstack provide an improved way to deliver more accurate alerts.

To work well together, you still need to be aware of the BI motion settings and do some tweaking. For example, with AI, Zone crossings are not so relevant, but defining a Zone A or the area of interest is important to reduce false alerts. Below is my area interest set in BI, again trying to reduce noise by masking out the clock and the road.
bi-masking_optimized.png
bi-masking_optimized.png (86.6 KiB) Viewed 1640 times
  • Some settings may not be as much needed as in the past. For example, I found the Object travels setting could be turned off and rely more on the AI to filter out the noise more accurately. Now if the AI turns out to be less accurate in other scenes, then you may need to rely on Object travels to reduce false alarms.
  • I was no longer as concerned about the directionality of the motion so I switched back to the Simple algorithm.
  • A learning, however, is the importance of contrast at different times of day due to lighting. In the example below, while most cars were being caught the car below was not, no matter how sensitive I made the contrast or object size.
    car-missed_optimized.png
    car-missed_optimized.png (75.46 KiB) Viewed 1640 times
Overall, it's great to have BI motion detection with AI to deliver the best results.