Browsed by
Category: BOINC

Distributed AV1 Video Encoding

Distributed AV1 Video Encoding

Hi everyone,

Yesterday, I began working on writing code to encode video files to AV1 in a distributed computing fashion using my private BOINC server. I wrote some PHP code and developed a Windows Forms software that will submit the video files to the distributed computing server so that it can send jobs to connected machines.

The encoding software I’m using is rav1e. This software is currently single-threaded, which means that encoding videos will take some time, especially when using the –speed 0 flag, which is the slowest mode available nut gives the best quality.

My solution at this time consists on splitting the video files into 1-second segments and then launching a rav1e encoding task for each part, each in one CPU thread. This significantly speeds up the encoding process because it can simultaneously encode several parts depending on the CPU being used. For example, if we split a video into a 1-second segment, and you have an AMD Ryzen 7 2700X CPU, this means that you can process 16 seconds at a time.

Preparing the input video file

As I mentioned at the start, I wrote a Windows Forms software that will handle the video submission process. The software will first call ffmpeg to split the video into 1-second segments, extract the audio file and encode it to Opus using the desired bitrate, and then it will submit the 1-second segment video files to the BOINC server, generating a task for each file:

av1_task_submitter

I’m submitting the video files using a quantizer setting of 175 and a speed of 0.

Once the tasks are generated, comes the processing.

AV1 encoding:

The BOINC client is the software responsible for contacting the server to retrieve the splitted files and initiate the tasks. It will download the rav1e executable from the BOINC server, the input video file, and will start the encoder. This process takes about 2 hours and 30 minutes on an AMD Ryzen 7 2700X CPU while for my Intel i7-3610QM and 4700MQ it took more than 3 hours. Still, the advantage is that I get to encode multiple segments at once, speeding the encoding of the source video significantly.

Here, you can see the encoding tasks from 4 of my machines:

rav1e BOINC Tasks
rav1e BOINC Tasks

Result files:

Once the encoding tasks are finished, they are sent back to the BOINC server which will mark the task as completed and will move the results to a defined folder:

.ifv segments
.ifv segments

Merging the files:

The final step is to merge the result files and also merge the audio file. For this, I wrote a simple bash script that calls ffmpeg’s concat function and then merges the concatenated file with the audio to generate one final .webm file. WebM is s container compatible with AV1 video files as well as Opus audio tracks:

Script to merge the .ifv files into the final .webm file
Script to merge the .ifv files into the final .webm file

The script takes 3 arguments. The first one is the list of files to concatenate, the second is the Opus audio file and the third and final one is the name for the output file.

When we run it, we should see the following at the end:

The merging script after running
The merging script after running

The final file:

We can see the final encoded .webm file at the location we typed when we ran the above script:

The final .webm file
The final .webm file

We can also check the file using MediaInfo:

Mediainfo
Mediainfo

And that’s it! BOINC is really useful in this case as I can encode multiple files and the server will handle all of the processing. The only manual part is where we run the merging script to generate the output file. However, it really speeds encoding by a lot. This is, until rav1e becomes multithreading, at which point, he BOINC server can still be used by changing the app to a multi-threading one.

Hope you enjoyed reading this post!

I found a prime number! PrimeGrid BOINC project

I found a prime number! PrimeGrid BOINC project

Hi everyone,

Today, I got an automated private message in the PrimeGrid BOINC project telling me I found a prime number in their AP27 subproject.

The message I got is the following,

Primegrid prime number notice

Here it is in plaintext:

(This is an automated message. Replies to this message will not be read.)

Dear AP finder,

Congratulations! Our records indicate that a computer registered by you has found a unique Arithmetic Progression of primes of length 21. This computer is assigned to the AP27 project. Since primes found in this subproject are not large enough to report to the Top 5000 Primes List, your AP21 sequence is visible immediately.

Workunit 574020495 : 108096152238162469+6453505523#n for n=0..20

You can select the minimum length of AP for which to receive these notifications in your PrimeGrid preferences.

If you have any questions or concerns, please contact us and we will surely resolve any problems.

Once again, congratulations on your find! Thank you for participating in PrimeGrid.

Sincerely,
The PrimeGrid Staff

Workunit details:

Primegrid Workunit

Checking the task sent to my machine, it was found by my AMD Ryzen 7 2700X machine with its Nvidia Geforce GTX 1060 3GB GPU:

Primegrid Host Details

Yay!

Today also, the PrimeGrid challenge ended:

PrimeGrid challenge results

This time, I ended in position #106:

PrimeGrid challenge position 106

My Signature:

Primegrid challenge signature

That’s all! Proud to see my machines doing useful stuff 😁

Improvements to my Finished BOINC Tasks section in my website and today’s Cosmology@Home rank

Improvements to my Finished BOINC Tasks section in my website and today’s Cosmology@Home rank

Hi everyone,

Today afternoon, I worked on setting up a sorting option allowing you to sort my finished BOINC tasks by date and by host, or one of them in the Finished BOINC Tasks section of my website.

The result looks like this:

BOINC Finished Tasks 1

You can, for example, select the tasks that’s been processed today, for my AMD Ryzen machine, and you’ll get this:

BOINC Finished Tasks 2

Note that I’m not an expert at HTML or CSS, but I’ve managed to bring this functionality to my site for all of you to enjoy.

This new code is available at the BOINCToWeb Repo that you can see by clicking here.

Position #8 at the Cosmology@Home challenge!

That’s right! I’m already at position #8 in the Cosmology@Home challenge:

Cosmology@Home Rank

And that’s why I also made the Finished BOINC Tasks section at my website, so everyone can keep track of my completed BOINC Tasks 😁, as I sometimes get asked what hardware I use to crunch tasks that allow me to go up in the leaderboards 🙂.

That’s all!

New Gridcoin BOINC Challenge: Cosmology@Home – Installing VirtualBox and configuring the project

New Gridcoin BOINC Challenge: Cosmology@Home – Installing VirtualBox and configuring the project

Hi everyone,

Today, a new BOINC challenge for the project Cosmology@Home started. This challenge was issued by Team Gridcoin:

Cosmology@Home Challenge 1

I went ahead and added my machines to this project, only to realize I need VirtualBox installed:

BOINC Virtualbox not installed

So in order to crunch tasks for this project, I went to install VirtualBox as well as configure the project. Here I detail you the installation and optional configuration process, which is really simple:

Installing VirtualBox

1. First, we go to the VirtualBox Website, https://virtualbox.org and click the Download VirtualBox button:

Downloading Virtualbox 1

2. Because I’m using Windows, I’ll download the Windows version:

Downloading Virtualbox 2

3. We’ll click Save:

Downloading Virtualbox 3

4. And after the download finishes, we will click on Run:

Downloading Virtualbox 4

5. We press Next:

Installing Virtualbox 1

6. We’ll leave this as it is and press Next:

Installing Virtualbox 2

7. We’ll press Next again:

Installing Virtualbox 3

8. Now, we’ll click Yes:

Installing Virtualbox 4

9. Now on Install:

Installing Virtualbox 5

10. It will ask us to install some driver software. We’ll click Install:

Installing Virtualbox 6

11. When the installation finishes, uncheck the Start VirtualBox checkbox and press Finish:

Installing Virtualbox 7

Configuring the project

I recommend doing these steps if you have a really bad lag or unresponsiveness in your PC. I had to apply a customized version of the configuration posted in the Cosmology@Home FAQ section as the VM would get stuck in my AMD Ryzen machine, so I configured it to use half the PC cores.

Here’s how to configure it:

1. Go to the Cosmology@Home Limit CPU section in the FAQ and copy and paste the app_config.xml template:

Configuring Cosmology@Home 1

2. Open your favorite Text Editor and copy the contents you copied recently. I used Notepad++ here. Then, change the <avg_ncpus> section to specify the amount of threads you want the VM to use. I first limited it to 4 cores and kept going up until I could maximize my PC resources while keeping it stable. In my Intel machine, I tried this configuration to test it:

Configuring Cosmology@Home 2

3. Save the config in the Cosmology@Home folder. If you installed BOINC and left the directories at their default location, then this directory/folder is C:\ProgramData\BOINC\projects\www.cosmologyathome.org. Save the file as app_config.xml:

Configuring Cosmology@Home 3

4. You’ll see the syntax is highlighted. It should be like this meaning it saved as an XML file:

Configuring Cosmology@Home 4

And that’s it! We’ve finished the configuration.

Testing the installation and configuration.

Now, we need to launch BOINC and perform a project update. For this, exit BOINC if it is already running and launch it again. Then select the Cosmology@Home project and press Update:

Running Cosmology@Home 1

You should see in the log that it is downloading tasks:

Running Cosmology@Home 3

And that they are running:

Running Cosmology@Home 3

You can also check VirtualBox and see that the task VM is running:

Running Cosmology@Home 4

And here’s how the CPU of one of my Intel machines looks like:

CPU Threads used by Cosmology@Home

Eventually, I upped the CPU threads to 8 so that it can use every resource in it. The only exception is my AMD Ryzen 7 2700X machine which is limited to 8 threads out of its 16.

If you followed this guide and are running Cosmology@Home tasks, congrats! And I wish you luck in the competition!

I’m at position #27 right now 😁:

Cosmology@Home Challenge Position 27

As a bonus, I made a section on my personal website where you can see the tasks I’ve finished processing:

BOINC Finished Tasks 1

This required some coding in my BOINCToWeb software I wrote to accomplish this. I’ll talk about the code changes and additions in another post.

I also remember all of you that you can see the tasks I’m crunching by clicking here:

BOINC Running Tasks

That’s all!

Running BOINC Tasks in my website updated! Scripts and tool updated

Running BOINC Tasks in my website updated! Scripts and tool updated

Repository:

https://github.com/moisespr123/BOINCToWeb

Hi everyone,

In the weekend, I worked to improve the BOINC Tasks section of my website:

BOINC Statistics Website Update 1

Instead of manually updating this page, I’ve updated the PHP files in charge of retrieving the tasks and showing it. This file was custom made and was not in the GitHub repo. Instead, the file that was in the repo was the one that looks like this:

BOINC Statistics Website Update 2

Because that one shows all of the tasks from all of the hosts, I made another PHP file to replicate the format I used in my website but instead of having to manually type each machine there, I made the php file use some functions and loops to retrieve the machines and tasks from the database. The result is the following:

BOINC Statistics Website Update 3
BOINC Statistics Website Update 4

Looks better, right?

Let’s take a look at the code:

First, I made a separate file called config.php to keep the MySQL details there. I’m also using a global mysqli object. Because of this change, I modified the original BOINC Tasks PHP script:

BOINC Statistics Website Update 5

After doing that, I wrote the next file, which I called boinc_tasks_per_machine.php:

BOINC Statistics Website Update 6

There’s 3 functions in this file:

BOINC Statistics Website Update 7

function printMachinesLinks($mysqli)

This function gets the machines from the MySQL database and it’s used to print the machine list, which is printed with a clickable link that goes to the machine table that shows the BOINC Tasks

function getTasksFromMachines($mysqli)

After getting and printing the machine list, we call this function which will then call the next function I’ll mention below.

function getTasksPerMachine($mysqli, $machine)

This function will get the BOINC tasks that a specific machine is running. It generates the table and prints the header as well.

Those are the functions. Next, we call them to print the neatly formatted document with these 2 lines:

printMachinesLinks($mysqli); 
getTasksFromMachines($mysqli);

The first one calls the printMachinesLinls($mysqli) function to print the machine list and the second line is the one that calls the function that generates the tables.

The software that populates the database:

BOINC Statistics Website Update 8

I also did some updates to this software. First, I made TruncateTables a function which will return True if it was successful truncating the table, and False if it wasn’t:

BOINC Statistics Website Update 9

Also, I added a NotifyIcon element to hide the software in the system tray and added a function to run the software when we pass the -s argument. This way, the software can be programmed for the Task Scheduler tool so it is launched in a minimized way and start retrieving the tasks:

BOINC Statistics Website Update 10

Then, I also made the Fetch button a Stop button. Clicking it after the software is running and is fetching the tasks will stop the process:

BOINC Statistics Website Update 11

Also, I refactored some code:

BOINC Statistics Website Update 12

And that’s it for the software changes. Basically I improved it in some ways. The page is now dinamically updated instead of manually updating which is a real plus for me. No more coding in the page! I just need to add and alter the machine list in the software and that’s it! The page will get updated with the machines listed in the software which it was able to connect and retrieve the tasks successfully.

Because the PHP files are very simple and straightforward, you can copy the file content into another page or add some styling. In my case, I integrated it with Drupal by making a PHP content page there.

PHP Files:

The PHP Files shown in this post can be found here.

Integration example:

You can see how the script looks integrated into WordPress here.

What next should I do with this? I’ve yet to know 🤔

Hope you like this!

Previous posts:

  1. Introductory post
  2. v1.1 changes
  3. v1.2 changes
Participating in the yoyo@home BOINC challenge

Participating in the yoyo@home BOINC challenge

Hi everyone,

It’s been 2 days since the yoyo@home BOINC challenge started. As team Gridcoin is participating in this one, I’m also participating as well 😁:

Participating in the yoyo@home BOINC challenge 1

yoyo@home is a project that has several distributed computing projects and brings them to the BOINC environment. Some projects that are currently running there are the Siever, distributed.net’s OGR challenge, Perfect Cuboids, and more.

I’m running this project in my machines and here are some screenshots of BOINC with the project attached. They are all running Windows 10 Insider Preview:

You can see the amount of cores processing tasks and this is why I’m proud to announce that I’m currently in position #33 in the challenge started:

Participating in the yoyo@home BOINC challenge 5

These are my statistics in the project:

Participating in the yoyo@home BOINC challenge 6

You can see I have 2 badges as well.

I also have several tasks that are currently pending validation:

Participating in the yoyo@home BOINC challenge 7

So, expect me to continue going up in the challenge leaderboard!

As with the past challenges, I’ll keep all of you updated 😁

The GPUGrid Competition results are in!

The GPUGrid Competition results are in!

Hi everyone,

Yesterday night, the Compete with team Gridcoin! GPUGRID BOINC competition ended, and today I’m sharing my results.

I reached position #80!

GPUGrid 5-18-2018 - 1

And up to today, I got 33 tasks validated. Yay!

GPUGrid 5-18-2018 - 2

I’m very happy for my contribution toward the project and the competition, and I’ll keep running this project to keep my GPUs busy while doing useful research.

Credits:

Here, you can see my credits up to this point:

GPUGrid 5-18-2018 - 3

What’s next?

There’s another GPU competition coming, but unfortunately, Team Gridcoin isn’t part of it, so that means that while I can continue contributing toward the project, my stats will not be shown there because of the team not participating. However, there’s a yoyo@home competition which starts in 4 more days, in which Team Gridcoin is going to participate 😁

GPUGrid 5-18-2018 - 4

My general stats:

It’s been a long time since I posted about my stats in general, so here you can see my credits by project and how much credits I’ve been granted per project:

GPUGrid 5-18-2018 - 5

That’s basically it for this post! In 4 days, a new challenge starts and I’m ready 🙂!

GPUGrid Statistics on May 16, 2018

GPUGrid Statistics on May 16, 2018

Hi everyone,

It’s time for today’s GPUGrid stats update. We are 2 days and an hour from ending this competition in which I’m proudly participating and I’d like to share how I’m coming along in it:

GPUGrid 5-16-2018 - 1

So far, I’ve gotten 17 validated tasks in this project:

GPUGrid 5-16-2018 - 2

Now, unfortunately, I went down to position #92. Still, what matters is the contribution I’m doing to the project and even if I’m 92, I’m still listed on the first page.

GPUGrid 5-16-2018 - 3

2 more days remain in this competition, and I’ll make those 2 days count by continue crunching GPU tasks on my Nvidia GPUs 🙂.

My GPUGrid Statistics – 5/14/2018

My GPUGrid Statistics – 5/14/2018

Hi everyone,

Today, I’ll show you my statistics for my contribution to GPUGrid up until this moment.

As you have read in my previous post, I’ve been participating at the “Compete with team Gridcoin! GPUGRID” BOINC challenge. Today, I’m at position #136:

GPUGrid Statistics - 2018-05-14 - 1

Project Stats:

These are my credits granted for the project up to this point:

GPUGrid Statistics - 2018-05-14 - 2

Valid tasks:

I already have 6 tasks validated:

GPUGrid Statistics - 2018-05-14 - 3

In progress:

Currently, 2 tasks are being processed. One task per machine:

GPUGrid Statistics - 2018-05-14 - 4

Hosts:

Here are more details of my 2 PCs which are crunching tasks for this project. Both have an Nvidia Geforce 1060 GPU except that one has a 3GB version and the other has a 6GB GPU version:

AMD Ryzen 7 2700X + Nvidia Geforce GTX 1060 3GB:

GPUGrid Statistics - 2018-05-14 - 5

Yeah… I haven’t changed the name of the PC. Previously, it had an AMD Ryzen 7 1700 CPU.

Intel i7-7700HQ + Nvidia Geforce GTX 1060 6GB:

GPUGrid Statistics - 2018-05-14 - 6

So there you have it. Some stats of the project and the progress made so far 😁.

Keep looking forward for more stats!

(Screenshots taken with my Samsung Galaxy S9+ plugged into my Samsung Dex Pad.📱)