Change Export Speed ?

[Cire3]

The client I have this installed on needs a full 24 hours of video exported. I selected the date, and it's exporting. However It's been almost a day and it's @ roughly 350Gb so far.

From the traffic I see daily , it takes in 1.5Tb. Any way to export this faster ? The system is only running @ 30%ish.

I'm unsure how to make this dedicate more resources to the export ? Anything I can do to get it exported faster ? I set all clips to export as MPEG4. He needs full res for needed compliance.

I have more horsepower, just unsure how to make it happen. It's been almost a day, and I have roughly 350Gb so far.

Exporting to a non-used sata drive on the system. And shutting down cameras is sadly not an option.

Thoughts ?

[YrbkMgr]

First thing that comes to mind is the export profile being used. Take a look there - here's an example of Export Encoder Profiles. The intended use of the exported MP4 drives the extent to which the problem is addressed by encoding optimization. Other workarounds include auto flag/export Alerts instead of the full 24 hours - if necessary they can be combined post export, and a variety of other methods to relieve the bottleneck. But start with profiles and let us know if you need more help.

[Cire3]

Wow, passed right over that. So I should be on default / veryfast ?

I also noticed I had no hardware acceleration set, as I'm sure that didn't help at all. So, these settings are just for export and don't affect how anything operate normally correct?

Now if I change it, it asks if I want to over write it ? It's been 3 days and it's only @ 557Gb, out of what I think will be near 1.5Tb as that's what my daily download from cameras seems to be. Will this re-start the process, and if so will it be much faster ? I don't want to dump 3 days of progress if at all possible. Didn't think it would be this slow at all.

By the way, system is only running @ 24-26% CPU, 18.6 of 32Gb memory, and 2% GPU while this is going on.

Specs are: i9-7980XE, 32Gb memory, and NVidia Quadro K2200.

40 Cameras spanned over 7 X 10Tb drives. (Few cameras per drive, no raid) And a dedicated 10Tb drive for export, etc...

Thanks for the help, exactly the route I was hunting for.

[YrbkMgr]

Wow, passed right over that. So I should be on default / veryfast ?

Can't really say. I have no stick time with it and only export precious few videos. That said the help files says

p116
The preset defaults to superfast. When combined with the zero frame rate latency option, this provides the least amount of latency for video encoding. Changing this may have the possibility to increase overall image quality during re-encoding, but always at the expense of more CPU resource. Also, latency translates to “blackness” at the beginning of the video for one or more frames as the encoder must setup a “pipeline” and does immediately produce output.

I also noticed I had no hardware acceleration set, as I'm sure that didn't help at all. So, these settings are just for export and don't affect how anything operate normally correct?

I don't have an appreciation for the details of hardware acceleration. From what I have gleaned HWA carries a risk/reward factor depending on a bunch of factors. As I understand it tasks that will benefit from HWA are computer graphics and audio editing. For example, you can make video conversion run faster in some apps. The downside is that sometimes there are unintended consequences requiring disabling HWA. I have it enabled on all of my cameras (3), but I can't testify to its benefit. That's a long winded way to say, I don't know for sure, but most people enable it.

Now if I change it, it asks if I want to over write it ? It's been 3 days and it's only @ 557Gb, out of what I think will be near 1.5Tb as that's what my daily download from cameras seems to be. Will this re-start the process, and if so will it be much faster ? I don't want to dump 3 days of progress if at all possible. Didn't think it would be this slow at all.

My advice is don't interrupt the current conversion. I'm curious about your strategy - mostly because I've not heard the complaint before. I'm no expert but if a process were choking performance to the degree you're experiencing, something's wrong. And if I were a betting man, I'd wager there's a better way to accomplish the goal.

I don't have any idea what you're configuration is now, but I bet it's sub optimal. And without knowing the use case for the MP4 exports, the following is just spitballin'. You can decrease demand on resources by using the Combine/Cut feature on the recording tab. Break up the recordings into manageable chunks for export processing. Processing three 8 hour files is far more efficient and less prone to loss/corruption than one single behemoth file. Further, letting Clips & Archiving manage when and where the files are exported makes life a whole lot easier. You can set up a dedicated location and configure retention policies to convert.

Further, if you change Cut/Combine on the record tab settings for one camera, you can easily copy that tab to every camera or cameras of your choosing. Take a peek at this post: Can you copy and paste alert settings?

In your shoes, I'd clone a camera and test that one to see how well it does or doesn't work, before committing to the other cameras.

[MikeBwca]

When you export, do you have 'Re-encode' checked?
If so, it does not need to be, as long as the 'Output format' is 'MP4.

I've found this to be extremely fast.

However, if you do 'Re-Encode' the resulting MP4 will be quite a bit smaller.
For my small exports up to 10-30 minutes, I 'Re-Encode'. Larger exports I uncheck that option.

Re-Encode checked, I get 150 fps. I tested by exporting a 24-hour bvr. Taking too long, and have up.
Re-encode Unchecked & '2-pass; checked, I get 850-900 fps.

[MikeBwca]

Ran a test the other day. It took 34 minutes to export a 33gig 24 hours bvr. Given the 1.5tb your exporting, that would be around 26 hours I think.

[Cire3]

Ok, all great info ! And thanks for your time on this.

Hopefully one last question. How do I see if it's still exporting? Any way to view a queue? It's been 4 days now and I'm ready to try the different settings. As of now, I'm going back in every few hours to see if it's growing in size to determine if it's finished. I would like to just stop it, and start over.

Again thanks, all great info !!!

[YrbkMgr]

Love the discussion. If your strategy impedes the goal, change the strategy - work with smaller bits. I have a lot of experience with batch processing of very large files and numbers of files. My observation is that your workload is choking your system - it all boils down to that. I think you should pause your export processing as soon as the one processing now finishes, clone one camera, make the bvr's smaller - cut every 8 hours. Manually export clips after 24 hours and evaluate performance.

I assume that the reason for one file = one day relates to your cataloging and retrieval strategy - and you don't want to change it. If that's true, you can still get there from here. But test it on a clone or one camera first - don't address 1 file = 1 day yet.

I am genuinely curious though, why are you exporting a full day of video to MP4? I'm not suggesting there's anything wrong with that - I only export on an as needed basis, so I'm curious about why one would do that.

[MikeBwca]

...
How do I see if it's still exporting? Any way to view a queue?
...

From the BI manual on page 127...

You can display a list of clips waiting for batch export by right-clicking in the clips window
and selecting View—>Convert/Export queue. It should be possible to remove items from this
queue if necessary by selecting and using the Del key.