Browsed by
Tag: distributed computing

Distributed AV1 Video Encoding

Distributed AV1 Video Encoding

Hi everyone,

Yesterday, I began working on writing code to encode video files to AV1 in a distributed computing fashion using my private BOINC server. I wrote some PHP code and developed a Windows Forms software that will submit the video files to the distributed computing server so that it can send jobs to connected machines.

The encoding software I’m using is rav1e. This software is currently single-threaded, which means that encoding videos will take some time, especially when using the –speed 0 flag, which is the slowest mode available nut gives the best quality.

My solution at this time consists on splitting the video files into 1-second segments and then launching a rav1e encoding task for each part, each in one CPU thread. This significantly speeds up the encoding process because it can simultaneously encode several parts depending on the CPU being used. For example, if we split a video into a 1-second segment, and you have an AMD Ryzen 7 2700X CPU, this means that you can process 16 seconds at a time.

Preparing the input video file

As I mentioned at the start, I wrote a Windows Forms software that will handle the video submission process. The software will first call ffmpeg to split the video into 1-second segments, extract the audio file and encode it to Opus using the desired bitrate, and then it will submit the 1-second segment video files to the BOINC server, generating a task for each file:

av1_task_submitter

I’m submitting the video files using a quantizer setting of 175 and a speed of 0.

Once the tasks are generated, comes the processing.

AV1 encoding:

The BOINC client is the software responsible for contacting the server to retrieve the splitted files and initiate the tasks. It will download the rav1e executable from the BOINC server, the input video file, and will start the encoder. This process takes about 2 hours and 30 minutes on an AMD Ryzen 7 2700X CPU while for my Intel i7-3610QM and 4700MQ it took more than 3 hours. Still, the advantage is that I get to encode multiple segments at once, speeding the encoding of the source video significantly.

Here, you can see the encoding tasks from 4 of my machines:

rav1e BOINC Tasks
rav1e BOINC Tasks

Result files:

Once the encoding tasks are finished, they are sent back to the BOINC server which will mark the task as completed and will move the results to a defined folder:

.ifv segments
.ifv segments

Merging the files:

The final step is to merge the result files and also merge the audio file. For this, I wrote a simple bash script that calls ffmpeg’s concat function and then merges the concatenated file with the audio to generate one final .webm file. WebM is s container compatible with AV1 video files as well as Opus audio tracks:

Script to merge the .ifv files into the final .webm file
Script to merge the .ifv files into the final .webm file

The script takes 3 arguments. The first one is the list of files to concatenate, the second is the Opus audio file and the third and final one is the name for the output file.

When we run it, we should see the following at the end:

The merging script after running
The merging script after running

The final file:

We can see the final encoded .webm file at the location we typed when we ran the above script:

The final .webm file
The final .webm file

We can also check the file using MediaInfo:

Mediainfo
Mediainfo

And that’s it! BOINC is really useful in this case as I can encode multiple files and the server will handle all of the processing. The only manual part is where we run the merging script to generate the output file. However, it really speeds encoding by a lot. This is, until rav1e becomes multithreading, at which point, he BOINC server can still be used by changing the app to a multi-threading one.

Hope you enjoyed reading this post!

World Community Grid Statistics – September 26, 2018 – AM Report

World Community Grid Statistics – September 26, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid contribution from the morning of September 26, 2018.

I’ve returned 127 work units so far. I’ve also generated 38,335 points and have run a total of 14 days with 8 hours and 11 minutes:

WCG 2018-09-26 AM 1

Here are the stats per project:

WCG 2018-09-26 AM 2

Here are my average stats:

WCG 2018-09-26 AM 3

And the historical graphs:

WCG 2018-09-26 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here are my stats for the past 14 days:

WCG 2018-09-26 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 25, 2018 – AM Report

World Community Grid Statistics – September 25, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid contribution from the morning of September 25, 2018.

I’ve returned 78 work units so far. I’ve also generated 30,037 points and have run a total of 8 days with 5 hours and 25 minutes:

WCG 2018-09-25 AM 1

Here are the stats per project:

WCG 2018-09-25 AM 2

Here are my average stats:

WCG 2018-09-25 AM 3

And the historical graphs:

WCG 2018-09-25 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here are my stats for the past 14 days:

WCG 2018-09-25 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 24, 2018 – PM Report

World Community Grid Statistics – September 24, 2018 – PM Report

Hi everyone,

Here’s my PM report for my World Community Grid contribution on September 24, 2018.

I’ve returned 192 work units so far. I’ve also generated 73,853 points and have run a total of 24 days with 17 hours and 45 minutes:

WCG 2018-09-24 PM 1

Here are the stats per project:

WCG 2018-09-24 PM 2

Here are my average stats:

WCG 2018-09-24 PM 3

And the historical graphs:

WCG 2018-09-24 PM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here are my stats for the past 14 days:

WCG 2018-09-24 PM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 23, 2018 – AM Report

World Community Grid Statistics – September 23, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid contribution from the morning of September 23, 2018.

I’ve returned 49 work units so far. I’ve also generated 17,149 points and have run a total of 7 days with 8 hours and 46 minutes:

WCG 2018-09-23 AM 1

Here are the stats per project:

WCG 2018-09-23 AM 2

Here are my average stats:

WCG 2018-09-23 AM 3

And the historical graphs:

WCG 2018-09-23 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here are my stats for the past 14 days:

WCG 2018-09-23 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 22, 2018 – PM Report

World Community Grid Statistics – September 22, 2018 – PM Report

Hi everyone,

Here’s my PM report for my World Community Grid on September 22, 2018.

I’ve returned 92 work units. I’ve generated 36,505 points and have run a total of 12 days with 9 hours and 31 minutes:

WCG 2018-09-22 PM 1

Here are the stats per project:

WCG 2018-09-22 PM 2

Here are my average stats:

WCG 2018-09-22 PM 3

And the historical graphs:

WCG 2018-09-22 PM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here are my stats for the past 14 days:

WCG 2018-09-22 PM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 21, 2018 – AM Report

World Community Grid Statistics – September 21, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid contribution on the morning of September 21, 2018.

I’ve returned 98 work units so far. I’ve also generated 36,592 points and have ran a total of 10 days with 11 hours and 48 minutes:

WCG 2018-09-21 AM 1

Here’s the stats per project:

WCG 2018-09-21 AM 2

Here’s my average stats:

WCG 2018-09-21 AM 3

And the historical graphs:

WCG 2018-09-21 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here’s my stats for the past 14 days:

WCG 2018-09-21 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 20, 2018 – AM Report

World Community Grid Statistics – September 20, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid contribution from the morning of September 20, 2018.

I’ve returned 145 work units so far. I’ve also generated 55.960 points and have ran a total of 13 days with 2 hours and 44 minutes:

WCG 2018-09-20 AM 1

Here’s the stats per project:

WCG 2018-09-20 AM 2

Here’s my average stats:

WCG 2018-09-20 AM 3

And the historical graphs:

WCG 2018-09-20 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here’s my stats for the past 14 days:

WCG 2018-09-20 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 19, 2018 – PM Report

World Community Grid Statistics – September 19, 2018 – PM Report

Hi everyone,

Here’s my PM report for my World Community Grid contribution on September 19, 2018.

I’ve returned 283 work units. I’ve generated 103,838 points and have ran a total of 26 days with 12 hours and 40 minutes:

WCG 2018-09-19 PM 1

Here’s the stats per project:

WCG 2018-09-19 PM 2

Here’s my average stats:

WCG 2018-09-19 PM 3

And the historical graphs:

WCG 2018-09-19 PM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here’s my stats for the past 14 days:

WCG 2018-09-19 PM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.

World Community Grid Statistics – September 19, 2018 – AM Report

World Community Grid Statistics – September 19, 2018 – AM Report

Hi everyone,

Here’s my report for my World Community Grid from the morning of September 19, 2018.

I’ve returned 137 work units so far. I’ve also generated 52,353 points and have ran a total of 13 days with 4 hours and 19 minutes:

WCG 2018-09-19 AM 1

Here’s the stats per project:

WCG 2018-09-19 AM 2

Here’s my average stats:

WCG 2018-09-19 AM 3

And the historical graphs:

WCG 2018-09-19 AM 4

Currently, I am contributing to the following World Community Grid subprojects:

These are running on a variety of machines.

Here’s my stats for the past 14 days:

WCG 2018-09-19 AM 5

See my currently-running BOINC tasks by clicking here.

Also, see my Finished BOINC Tasks by clicking here.