DeepStack Gotchas

We created a lot of DeepStack self-help content.
The DeepStack + Blue Iris article talks about the integration and proper setup.
The DeepStack Gotchas article is learnings from past tickets.
Understanding the DeepStack analysis (.dat) functionality is very useful when trying to understand what DeepStack is doing.
See Fine Tuning Settings article.
Post Reply
varghesesa
Posts: 90
Joined: Thu Jul 11, 2019 9:52 pm

DeepStack Gotchas

Post by varghesesa »

Introduction

Gotchas are issues that have been discovered via tickets from past customers. These articles help users resolve their own issues based on past learnings.

Gotchas

Gotcha 15: DeepStack version 2022.01.01
:idea: Intel from a user. 20220-Jan-28

A new version of DeepStack was just released. There is an issue with the way Blue Iris starts up the new version of DeepStack. The issue is DS version 2021.09.01 had two of the same files just named differently in the folder "C:\DeepStack\server". One was named deepstack.exe and the other was named server.exe. Blue Iris is trying to start up server.exe.

DeepStack version 2022.01.1 only has one file in folder "C:\DeepStack\server" named deepstack.exe. I did a temporary fix by copying deepstack.exe in the same folder with the file name server.exe.

Link: https://github.com/johnolafenwa/DeepSta ... /2022.01.1

Engineering states 5.5.5.1 will have the fix.

Gotcha 14: DeepStack timeouts

Issue: DeepStack keeps timing out. Customer comments
I have been successfully using Deep Stack for some time. However on <date> it ceased working. The system would no longer flag clips for recognized items.

It appears to identify objects but times out.

deepstack gotchas_timeout errors.png
deepstack gotchas_timeout errors.png (173.34 KiB) Viewed 12410 times

Fix:
Unfortunately we do not know the root cause. Users would re-install DeepStack. User comments below.

I think I have solved the problem by fully uninstalling DeepStack, manually deleting the leftover folder, and reinstalling it. I have now properly gotten a detection with object detection turned on
Everything has been working really well now and I’m really enjoying the fully integrated deepstack functionality.

1) Deleting the leftover folder: After running the uninstaller from windows (IE window’s remove programs) for deep stack, I noticed that there was still a C:\DeepStack folder with files inside, so I manually deleted this (which uncovered there were still things running such as “server.exe” which prevented the deletion of the folder until I manually stopped those processes).

2) I had the latest version installed, however, when I went to the windows remove programs, I noticed that it showed two versions of deepstack installed, the old version I had from 2020 and the latest one. I uninstalled them both. The old install was the CPU version, the new install was the GPU.

3) GPU: nvidia 970

I would also note that the GPU install of deepstack is much more difficult than your average user would be willing to go through. You have to sign up for NVidia’s developer program and answer questions for why you are downloading the AI libraries, and then manually move files into the appropriate installation directories. I understand that deepstack installation is somewhat outside of the scope of BlueIris software, however the difference in delays is significant if you are running a video system that is already heavily loading the CPU. Maybe some helpful links to the deepstack install guides (especially for GPU) would be something useful in the documentation.

Gotcha 13: Network traffic

Issue: I've been monitoring my BlueIris setup for a while and found some pretty strange behavior with the DeepStack intregration.
I have DeepStack installed on a Jetson nano doing the AI detection. I wanted to monitor the duration of the communication between BI server and the Jetson nano using WireShark, and I've noted that the v1/vision/detection call is sent very often to the DS server even if no motion is triggered, producing network traffic and CPU load. Actually, I've even logged the network traffic with the 0 profile and the shield disarmed. Still a lot of calls to the DS server.

Resolution: The software does not call DS when it's not required.
Engineering has seen this traffic as well. It appears that the various DS modules are in constant contact with each other. I am not sure how to stop it or if it's required for it's proper operation.

Gotcha 12: Microsoft Process Explorer

See Microsoft Process Explorer section to download/install Process Explorer.

Process Explorer is a good way to see if DeepStack is running correctly on your machine.
Below is an example from an actual ticket.

Issue: For some reason, DeepStack stopped working with one camera.
I have the latest software and am using the 1st CPU version of DeepStack. Everything has been working but recently one of the two Front Cameras which are in the same Group, stopped working using DeepStack. A car just went by so I am showing you that Side-Front shows the Snapshot (size) and then jpg Group and car:63% but the camera below called Front shows none of that. It was working until the last few updates. Does any of this ring a bell on what caused my main Hikvision to stop working with DeepStack or with the other group.

Update: As I was typing this, I decided to hit the Start Now of deepstack on the AI tab and it started working. Strange that one camera would work with the DeepStack settings and the other wouldn’t until I hit Start Now.
Support Response:
Check the logs for DeepStack errors.
Run Windows Process Explorer to make sure you have just have the one set of DS executables showing beneath BlueIris.exe.
IF others exist, they may be abandoned by a crash, causing issues with CPU and RAM over time.

Process Explorer Screenshot

deepstack gotcha_process explorer_480.png
deepstack gotcha_process explorer_480.png (146.82 KiB) Viewed 13548 times

Resolution:
When Blue Iris starts/stops DS the conhost, server and python processes will be listed as children of BlueIris.exe.
It's safe to kill those other ones (or restart Windows).



Gotcha 11: DeepStack is confirming alerts even though it is outside my area of interest.

deepstack gotcha 11 alert.png
deepstack gotcha 11 alert.png (21.96 KiB) Viewed 14828 times

Issue: Defined an area of interest (Zone B).

deepstack-gotcha-11-zones_optimized.png
deepstack-gotcha-11-zones_optimized.png (55.82 KiB) Viewed 14828 times

Told BI to only trigger based on Zone B.

deepstack-gotcha-11-object detection.png
deepstack-gotcha-11-object detection.png (16.58 KiB) Viewed 14828 times

Received an Alert due to motion outside Zone B.

deepstack-gotcha-11-alert image.png
deepstack-gotcha-11-alert image.png (213.97 KiB) Viewed 14828 times

The problem is DeepStack will observe the entire visible image.

Fix:
Two options.

1. Add more intelligence to Object Crossing logic.

deepstack-gotcha-11-object detection fix.png
deepstack-gotcha-11-object detection fix.png (17 KiB) Viewed 14828 times

By improving the logic, BI object tracking will only trigger if there is movement from A-B or vice versa.
The camera would never trigger if there was motion just in B, for example a piece of paper was blowing in the wind.
Thus BI would never have triggered and DeepStack would never have the opportunity to analyze the frame where the car was driving by on the road.

2. Obscure the road completely by redefining Zone A.
By obscuring the road, it is completely out of the picture. DeepStack will never analyze the street.
Probably leads to less CPU/GPU resources for DeepStack as well.

deepstack-gotcha-11-zone a new.png
deepstack-gotcha-11-zone a new.png (250.26 KiB) Viewed 14828 times

Gotcha 10: DeepStack temp directory filling up.
For some reason, certain deployments (CPU) result in DeepStack filling up the temp directory over a few days as the server is running.
c:\…AppData\Local\Temp\DeepStack

deepstack temp directory.png
deepstack temp directory.png (43.12 KiB) Viewed 15086 times


We still do not know the root cause. Please share with Support if you know the issue.
Below is a work around shared by one user.
Unfortunately what I have isn’t so much a fix as it is a workaround. It’s just using Windows Task Scheduler to call a batch file that runs ‘del C:\DeepStack\redis\temp-*.rdb’ every hour.

Gotcha 9: Server Error 100
deepstack server error 100.png
deepstack server error 100.png (20.62 KiB) Viewed 15285 times

See DeepStack: Server Error 100 section in Status Log Errors article for details.
Gotcha 8: DeepStack appears to be running but no images are being processed.

DeepStack is on (or believed to be on). However, no AI alerts appear in clip list. And no errors in the Status->Log!
We are getting reports that users have DeepStack running based on the DeepStack splash page popping up. However, NO Deepstack annotations appear when they should. It's as if DeepStack is not turned on. Clips list looks like below. Only motion annotations

no deepstack cliplist.png
no deepstack cliplist.png (43.42 KiB) Viewed 15321 times

Once DeepStack is activated, you should receive AI annotations. Even with no objects, the "nothing found" AI annotation appears!

nothing found.PNG
nothing found.PNG (29.18 KiB) Viewed 15321 times

If DeepStack were turned on, the clip list would look like below when objects were found.

deepstack running.png
deepstack running.png (69.3 KiB) Viewed 15321 times


First confirm the BI server and the DeepStack server are communicating. Global settings -> AI tab.
Also confirm you have DeepStack turned ON for the camera you are debugging! Camera settings -> Trigger tab -> Artificial Intelligence. Remember, if you have multiple Profiles, DeepStack needs to be set correctly (i.e. on or off) for each Profile.

Big gotchas:
  • If you uncheck Object detection from the Advanced section of Camera settings -> Trigger tab -> Motion sensor Configure, you have unbeknownst to yourself, turned off DeepStack because object detection is needed for DeepStack to work.
    deepstack object detection gotcha.PNG
    deepstack object detection gotcha.PNG (51.21 KiB) Viewed 15321 times
  • If all the trigger sources are inactive in the Alerts tab, you have unbeknownst to yourself, turned off DeepStack.
    If you want to turn off all you alerts, best do so by deleting all your entries in the action map or disabling all your entries in the action map.
    The default setting is to have all your trigger sources active. At a minimum keep the Motion Zones trigger sources enabled so DeepStack is not affected.
    deepstack trigger sources gotcha.PNG
    deepstack trigger sources gotcha.PNG (28.41 KiB) Viewed 15321 times
Assuming your settings are correct, run DeepStack manually to see if requests are being sent to DeepStack from BI.
  • Uncheck Auto start/stop in Global settings -> AI.
  • Restart Windows to make sure no DeepStack processes are still running. DeepStack could be unstable if trying to do so manually with all the additional processes that DeepStack spawns (server.exe, python.exe, redis-server.exe).

    deepstack processes.PNG
    deepstack processes.PNG (6.46 KiB) Viewed 15321 times
  • Start DeepStack manually from the command line. "deepstack --VISION-DETECTION True --PORT 82". Port 82 is used by default but if you chose a different port (Global settings -> AI tab), specify the same number in command line.
From the console, every time you trigger the camera or camera triggers on its own, observe whether a new request appears in the command window.
deepstack log.PNG
deepstack log.PNG (146.93 KiB) Viewed 15321 times
Gotcha 7: 2021-May-19 Update:

Deepstack is running but sometimes BI responds server not running

The below logs show a successful Deepstack response at 1:09:15. Then at 1:09:46 BI states Deepstack is unreachable, however user confirmed Deepstack is still running.

Code: Select all

5/12/2021 1:09:46 PM Oby1Cam: DeepStack: Not running or unreachable (84)
5/12/2021 1:09:45 PM Oby1Cam: MOTION (79)
5/12/2021 1:09:15 PM Ofy1Cam: DeepStack: Alert cancelled (nothing found) (83)
5/12/2021 1:09:09 PM Ofy1Cam: MOTION (75)
The fix:
First confirm all Software Security Exemptions (e.g. Windows Defender) have been applied. The Windows Tuning Article provides details.
https://blueirissoftware.com/forum/view ... =12&t=2080

The unreachable message occurs if DS does not respond on the IP:port that you specify. If this is not due to issues with security software, perhaps DS is having other issues. I do see an occasional DS state where it is not responding.

BI is now detecting Deepstack not running or unreachable states and restarts DeepStack behind the scenes (assuming BI is set to start/stop DeepStack). This should fix the issue without any user interaction needed.


Gotcha 6: BI Status Log Error - Deepstack not running or unreachable

If BI cannot communicate with Deepstack, BI will let you know.
Assuming you have Auto start/stop with BI enabled, simple fix is stop Deepstack if it's running and restart BI.
BI cannot talk to Deepstack.png
BI cannot talk to Deepstack.png (6.37 KiB) Viewed 15321 times

Gotcha 5: High CPU usage with Deepstack.

AI / Deepstack is very computationally heavy. If your CPU usage jumps and you don't have a GPU the best you can do is throttle the calls to Deepstack.
  • Turn on Deepstack only for outdoor cameras. BI motion settings are usually fine for indoor cameras.
  • Reduce number of images sampled per alert.
  • Tighten your BI motion sensor settings so cameras do not trigger an alert so frequently.
Gotcha 4: Deepstack missing obvious objects.

Shout out to our users for sharing issues and solutions!

For some reason, the BlueIris software is ignoring what the Deepstack server says in terms of confidence and the label. This seems to happen during the second, third or sometimes fourth image that Deepstack analyzes. I have confirmed with a packet capture that despite these additional images being sent back to the BlueIris software with a successful response, BlueIris ignores it and goes to cancel the alert. I confirm these images meet the criteria of the minimum confidence percentage and "to confirm". I would like to also mention that if DeepStack responds back with a prediction that is within the criteria on the first image, it works OK.

Fix: I just needed to change the keyframe interval to be the same value as the FPS so it is equal to 1.00 (as displayed from Blue Iris). The detection is working much better now. See Camera stream optimization article for details.

Alternative fix: If for some reason you cannot adjust the key frame interval on the camera, identify the current key frame interval (Status -> Cameras tab). If the value is 0.25, i.e. 1/4, i.e. 1 key frame every 4s, then set the pre-trigger buffer (Camera settings -> Record tab) greater than or equal to 4s. This will guarantee BI at least one key frame to process.


Gotcha 3: Deepstack always returning nothing found. (Another shout out to the community for making everyone smarter)

This could imply the Deepstack server is returning server errors and BI is just reporting back "nothing found".
BI will eventually report the Deepstack errors in the Status->log to make users aware of the issue easier.

Until done so, one user creatively used Postman to identify the issue. If you are a developer, you are probably familiar with Postman. It's a very popular tool that helps test and debug APIs. If new to Postman, lots of content on the internet how to use it. By using Postman, the user discovered object detection requests to Postman were returning 500 error codes. 500 errors mean the server had a problem processing the request.

FYI, 400 errors mean the request to Deepstack is not correct. For those new to Postman, a 400 error means the request you created in Postman to send to Deepstack is not correct. You need to first make sure you are sending correct requests to Deepstack.

Once the user realized their Deepstack installation was not working correctly, they reinstalled Deepstack. Deepstack does provide error logs, "C:\Users\[Username]\AppData\Local\DeepStack\logs" (%LOCALAPPDATA%\DeepStack\logs). Understanding the logs were challenging so best course of action was to try reinstalling Deepstack.

User also noted uninstalling Deepstack may not flush out all the files. The user went back into the installation directory (C:\DeepStack) and manually deleted any remaining files.

Furthermore, user noted, it's good to go into task manager and make sure any old Deepstack processes are not running. You can go into Task Manager and kill any of the following processes: server.exe, python.exe, redis-server-exe if they exist.

FYI, installing / using AI Tool for debugging may be worthwhile as BI continues to build out the debugging capabilities. In AI Tool, error messages will pop up right away if old Deepstack versions are still running.
ai tool error.PNG
ai tool error.PNG (4.89 KiB) Viewed 15319 times

Another clue that Deepstack is not running properly is by running Deepstack manually and observing the response time in the console. If Deepstack is taking 1m to respond, there is probably something wrong.
deepstack response time.PNG
deepstack response time.PNG (28.81 KiB) Viewed 15319 times
Gotcha 2: Front door camera

Front door view has the road and therefore a lot of unnecessary alerts. All you really want are people coming to the house or property.

front door.jpg
front door.jpg (154.09 KiB) Viewed 15321 times

So you create a zone in front of your house to reduce the noise from the road.

b_optimized.png
b_optimized.png (158.66 KiB) Viewed 15321 times

No alerts! The problem is the camera will trigger when a person's feet crosses Zone B. The image being processed by Deepstack is just the feet! So Deepstack will often return nothing found.

The fix! Make sure to always define a Zone A which is the overall area of interest on the camera. The image processed by Deepstack is the overlap of all the zones, thus the entire person is sent to Deepstack for processing and the appropriate alert is sent.

a_optimized.png
a_optimized.png (125.22 KiB) Viewed 15321 times


One important point that I forgot to mention in the Webinar but did come up in the Q&A is how to tell BI to only be concerned with motion in Zone B??? Simple, see below. By setting "Object crosses zones" to B, I am telling BI to only worry about motion in Zone B. Notice with AI, I uncheck "Object travels" and "Object size exceeds" settings. With AI, the motion sensor settings can usually be simplified!

Zone B setting.PNG
Zone B setting.PNG (12.79 KiB) Viewed 15321 times

Post Reply