This article is focused on the Clips and archiving tab, i.e. Storage. If you are new and are trying to understand how camera recordings tie into storage on your file system (Clips and archiving tab), the Storage and Recordings section of the Checklist is the place to start. The webinar walks through the Help files and provides the foundation of how Recordings (from cameras) tie into Storage (Clips and archiving tab).
This article provides a framework for troubleshooting storage issues and documents learnings from past tickets (Gotchas section).
If you were directed here, see Next steps / Submitting a ticket section for needed information to move forward. Simply stating you have an issue will not lead to resolution.
Troubleshooting
Providing just a description of the issue will not resolve the issue.
Follow steps below to address storage issues.
Basics
Make sure you followed best practices. FYI, reviewing entire Windows tuning article is always beneficial.
Self-correct
Double check your Global settings -> Clips and archiving tab settings. Confirm your cameras are recording files to the New folder or other folders based on your settings.
Seek guidance from the Help button. Watch the Storage and recordings webinar.
BI stats and warnings
Self-correct BI warnings and errors. Status button -> Clip storage tab. Help button provides details.
- storage_warning.png (570 Bytes) Viewed 32638 times
Similar error when seen in the log files.
Code: Select all
2 2/7/2022 11:59:19.456 PM Clips C:: 1427, 1.46T/1.95T (-26.3G)
- storage_NAS errors.png (13.56 KiB) Viewed 32638 times
User went through best practices listed above (Basics section) to resolve the issue:
After above fixes:
- storage_NAS fix.png (27.62 KiB) Viewed 32638 times
Notice how both drives have remaining green (unused) space. This is a good practice to NOT allocate the ENTIRE drive to BI.
Logs
Archive thread
The archive thread is responsible for moving files based on your Clips and archiving settings. If you have Log clip move/delete details selected in Global settings -> Clips and archiving tab, then below log entries every 5 minutes indicates the thread is running properly.
The actual details will be different based on your Clips and archiving setup. The 0 code event starting with Delete:, however, are the identifiers associated with archive thread events.
Log files
The logs will tell you if BI is having issues with your Storage. In order to know which errors to look for, we are documenting all the Status log errors. The ones concerning storage are in the Storage category and Recordings category. Each log error provides details and possible fixes to self-correct.
The Log files article explains how to easily deal with log files using excel.
The NAS Case Study article is a case study on how to leverage the logs to get to Storage issues.
Log tab
Might be easier to look for errors in the Logs tab if you check logs frequently. See Routine maintenance guidance for best practices. Keep in mind the Log tab is a snapshot in time of the recent events. Just because Storage and Recording errors do not pop up in the Log tab does NOT mean Storage and Recording errors do NOT exist.
The archive thread is responsible for moving files based on your Clips and archiving settings. If you have Log clip move/delete details selected in Global settings -> Clips and archiving tab, then below log entries every 5 minutes indicates the thread is running properly.
Code: Select all
0 1/1/2022 12:07:22.765 AM New Delete: nothing to do [719/720 hrs, 11.0G/100.0G, 297.1G free] 1 locked
0 1/1/2022 12:07:22.766 AM Alerts Delete: nothing to do [167/168 hrs, 50.2M/20.0G, 297.1G free]
Log files
The logs will tell you if BI is having issues with your Storage. In order to know which errors to look for, we are documenting all the Status log errors. The ones concerning storage are in the Storage category and Recordings category. Each log error provides details and possible fixes to self-correct.
The Log files article explains how to easily deal with log files using excel.
The NAS Case Study article is a case study on how to leverage the logs to get to Storage issues.
Log tab
Might be easier to look for errors in the Logs tab if you check logs frequently. See Routine maintenance guidance for best practices. Keep in mind the Log tab is a snapshot in time of the recent events. Just because Storage and Recording errors do not pop up in the Log tab does NOT mean Storage and Recording errors do NOT exist.
- storage_storage error.png (47.29 KiB) Viewed 32631 times
Many tech saavy users have identified storage access errors or hard drive issues using the following tools. Understanding how to use each tool is up to you.
windirstat
CrystalDiskInfo
procmon
sfc: Windows tool. Scans and verifies the integrity of all protected system files.
Next steps / Submitting a ticketwindirstat
CrystalDiskInfo
procmon
sfc: Windows tool. Scans and verifies the integrity of all protected system files.
See Storage ticket for details.
Gotchas
This section documents key learnings from past tickets.
Clips list / Alerts list
Issue: My clips list / alerts list is blank.
It's very unlikely BI deleted your recordings. Check the filesystem where you told BI to save your recordings. For example my New folder saves to G:\BlueIris\New folder. Global settings -> Clips and archiving tab.
And when I look at the content in the folder, I can confirm my recordings are there.
As you can see, recordings are saved based on the camera short name, e.g. Cam6. Since my short name Cam6 is assigned to an existing camera in BI, the recordings and alerts appear in the Clips list and Alerts list
If I change the short name to something else, e.g. Cam50, the Clips/Alerts list will be empty. Make note, the recordings and alerts are NOT deleted. The association between the camera short name and entries in the database, alerts and clips, are now broken. As soon as I change the camera short name back to Cam6, all my alerts and recordings reappear in the UI.
So, if your clips list and alerts list is suddenly empty, first confirm the recordings exist. Second, make sure the camera short name for one of your cameras did not change.
NAS
- storage troubleshooting_database empty.png (4.77 KiB) Viewed 31890 times
It's very unlikely BI deleted your recordings. Check the filesystem where you told BI to save your recordings. For example my New folder saves to G:\BlueIris\New folder. Global settings -> Clips and archiving tab.
- storage troubleshooting_new.png (35.7 KiB) Viewed 31890 times
And when I look at the content in the folder, I can confirm my recordings are there.
- storage troubleshooting_folder.png (5.81 KiB) Viewed 31890 times
As you can see, recordings are saved based on the camera short name, e.g. Cam6. Since my short name Cam6 is assigned to an existing camera in BI, the recordings and alerts appear in the Clips list and Alerts list
- storage troubleshooting_camera short name.png (5.41 KiB) Viewed 31890 times
If I change the short name to something else, e.g. Cam50, the Clips/Alerts list will be empty. Make note, the recordings and alerts are NOT deleted. The association between the camera short name and entries in the database, alerts and clips, are now broken. As soon as I change the camera short name back to Cam6, all my alerts and recordings reappear in the UI.
So, if your clips list and alerts list is suddenly empty, first confirm the recordings exist. Second, make sure the camera short name for one of your cameras did not change.
The category is NAS but this section can apply to RAID storage as well. Basically any non-internal drives.
USB drives may also apply.
Gotcha 1: Playback
Many permutations of the symptoms, but basically playback is NOT working and your storage includes non-internal drives.
New folder and database folder
If you are using the external storage for the New folder and/or database folder, move those folders back to your local hard drive, e.g. C: drive.
Create a small New folder on your local drive (c:) (via Clips and archiving) and delete files after 1 day so as not to fill up your c: drive.
If your database folder is NOT local, move the database folder as well.
See if playback starts working?
Then you know the issue is with your storage. This is how you identify the root cause by turning features off or simplifying your setup.
Windows
Confirm you setup storage correctly so Windows is NOT in conflict.
Double check you connected the Storage device correctly.
It's good to review the checks in the entire article.
Troubleshooting external storage
Instead of writing directly to external storage (New and Database folders on external storage), see if moving files to Stored from New works?
Can you now playback from the raid drive? Example of simplifying the setup and observing if functionality works.
Logs
Check the logs for Record and Storage errors. Log files are NOT the same as the Log tab.
Any time there are problems with NAS devices, it almost always has to do with permission issues or the NAS stopped communicating with the BI server. Unfortunately, the issue manifests itself with the drive containing the New folder (often the C: drive) filling up and crashing the server.
Common issues/fixes:
- Database -> Repair / Regenerate may resolve the issue
- Double check your Anti-virus/Firewall settings.
The logs will show the write errors well before filling up the C: drive, but this requires users paying attention to the Status button when it turns red or occasionally monitoring their logs for errors.
The tell tale errors are MoveFile Errors as seen in the below log snippet. Routinely check logs for Storage and/or Record errors.
Code: Select all
0 7/19/2021 7:32:01.679 PM New Move: 2 items 3.66G [602.1G/600.0G, 326.2G free] 126 locked
0 7/19/2021 7:32:01.788 PM Stored Delete: 2 items 4.24G [2.93T/2.92T, 1.99T free]
0 7/19/2021 7:32:01.791 PM Alerts Delete: nothing to do [1439/4800 hrs, 895.5M/100.0G, 329.9G free]
1 7/19/2021 7:32:01.860 PM Clips MoveFile Error 2: F:\BlueIris\New\FrontDoor.20210719_080000.bvr
1 7/19/2021 7:32:01.925 PM Clips MoveFile Error 2: F:\BlueIris\New\FrontDoorStation.20210719_080000.bvr
The Fix:
Use the exact same Windows username/password on your NAS device to provide the easiest chance for BI to access your NAS storage.
Check the Storage tab in the Status button to see whether BI recognizes your NAS device. There should be no RED regions.
Second, you need to make sure your NAS is working properly. Pay attention to log errors as mentioned above. The Log Files - Storage/Record Issues article documents how to approach the log files in order to understand the issue. In this case, the user's NAS would run correctly for 21 straight days before denying access to BI and causing BI to crash a few days later.
Below is a quote from another user that decided to replace his NAS storage.
Troubleshooting your NAS device is beyond BI support.I went to the Status page, which showed the overallocated storage, and clicked on Repair DB. Then slowly the data migrated to NAS storage. Maybe it lost contact with the NAS and needed a kick to restart it. All is well, now.
I'll download the NAS logs and give you more information shortly.
I tried to dump the logs on the NAS, but it hung up and now won't boot. Probably too much log. So it looks like the NAS just wasn't responding like it should.
The forum may provide insight.