0

I want to have two youtube-dl processes (or as much as possible )to run in parallel. Please show me how. thanks in advance.

#!/bin/bash #package: youtube-dl axel #file that contains youtube links FILE="/srv/backup/temp/youtube.txt" #number of lines in FILE COUNTER=`wc -l $FILE | cut -f1 -d' '` #download destination cd /srv/backup/transmission/completed if [[ -s $FILE ]]; then while [ $COUNTER -gt 0 ]; do #get video link URL=`head -n 1 $FILE` #get video name NAME=`youtube-dl --get-filename -o "%(title)s.%(ext)s" "$URL" --restrict-filenames` #real video url vURL=`youtube-dl --get-url $URL` #remove first link sed -i 1d $FILE #download file axel -n 10 -o "$NAME" $vURL & #update number of lines COUNTER=`wc -l $FILE | cut -f1 -d' '` done else break fi
Improve this question
2
  • 1
    stackoverflow.com/q/6384013/139010 Commented Feb 15, 2013 at 17:54
  • @MattBall what about NAME and vURL initialization? how can I run them in parallel? Commented Feb 15, 2013 at 18:14

4 Answers 4

Reset to default
2

This ought to work with GNU Parallel:

cd /srv/backup/transmission/completed parallel -j0 'axel -n 10 -o $(youtube-dl --get-filename -o "%(title)s.%(ext)s" "{}" --restrict-filenames) $(youtube-dl --get-url {})' :::: /srv/backup/temp/youtube.txt

Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Solution

You need to run your command in a subshell, i.e. put your command into ( cmd ) &.

Definition

A shell script can itself launch subprocesses. These subshells let the script do parallel processing, in effect executing multiple subtasks simultaneously.

Code

For you it will look like this I guess (I add quote to $vURL) :

( axel -n 10 -o "$NAME" "$vURL" ) &
Improve this answer

6 Comments

what about NAME and vURL initialization? how can I run them in parallel?
I think the trick here is not subshell, but & which makes your command run in background
@tictacbum I mean, the script works fine. but I think I can make NAME and vURL run in parallel as well. I've tried a couple of ways. they didn't work out.
Yes, you can, but unless those commands take more than seconds to execute, why make things more complex for so little gain?
@William what is the correct syntax with $() command substitution ?
|
0

I don't know if it is the best way, you can define a function and then call it in background

something like this:

#!/bin/bash #package: youtube-dl axel #file that contains youtube links FILE="/srv/backup/temp/youtube.txt" # define a function download_video() { sleep 3 echo $1 } while read -r line; do # call it in background, with & download_video $line & done < $FILE

script ends quick but function still runs in background, after 3 seconds it will show echos

also used read and while loop to simplify the file reading

Improve this answer

2 Comments

You don't need cat here. I know it isn't essential to the answer, but you should do while read -r line ; do download_video $ line & ; done < $FILE. Or lines=$(cat $FILE) if you insist on using cat
@tictacbum nice. this structure is better. but it still doesn't have NAME and vURL in parallel. This youtube-dl command takes the longest time. that's why I wanna have both of them in parallel
0

Here's my take on it. By avoiding several commands you should see some minor improvement in speed though it might not be noticeable. I did add error checking which can save you time on broken URLs.

#file that contains youtube links FILE="/srv/backup/temp/youtube.txt" while read URL ; do [ -z "$URL" ] && continue #get video name if NAME=$(youtube-dl --get-filename -o "%(title)s.%(ext)s" "$URL" --restrict-filenames) ; then #real video url if vURL=$(youtube-dl --get-url $URL) ; then #download file axel -n 10 -o "$NAME" $vURL & else echo "Could not get vURL from $URL" fi else echo "Could not get NAME from $URL" fi done << "$FILE"

By request, here's my proposal for paralleling the vURL and NAME fetching as well as the download. Note: Since the download depends on both vURL and NAME there is no point in creating three processes, two gives you about the best return. Below I've put the NAME fetch in its own process, but if it turned out that vURL was consistently faster, there might be a small payoff in swapping it with the NAME fetch. (That way the while loop in the download process won't waste even a second sleeping.) Note 2: This is fairly crude, and untested, it's just off the cuff and probably needs work. And there's probably a much cooler way in any case. Be afraid...

#!/bin/bash #file that contains youtube links FILE="/srv/backup/temp/youtube.txt" GetName () { # URL, filename if NAME=$(youtube-dl --get-filename -o "%(title)s.%(ext)s" "$1" --restrict-filenames) ; then # Create a sourceable file with NAME value echo "NAME='$NAME'" > "$2" else echo "Could not get NAME from $1" fi } Download () { # URL, filename if vURL=$(youtube-dl --get-url $1) ; then # Wait to see if GetName's file appears timeout=300 # Wait up to 5 minutes, adjust this if needed while (( timeout-- )) ; do if [ -f "$2" ] ; then source "$2" rm "$2" #download file if axel -n 10 -o "$NAME" "$vURL" ; then echo "Download of $NAME from $1 finished" return 0 else echo "Download of $NAME from $1 failed" fi fi sleep 1 done echo "Download timed out waiting for file $2" else echo "Could not get vURL from $1" fi return 1 } filebase="tempfile${$}_" filecount=0 while read URL ; do [ -z "$URL" ] && continue filename="$filebase$filecount" [ -f "$filename" ] && rm "$filename" # Just in case (( filecount++ )) ( GetName "$URL" "$filename" ) & ( Download "$URL" "$filename" ) & done << "$FILE"
Improve this answer

3 Comments

NAME and vURL still run in sequence. is it possible to run them in parallel?
You could wrap the part from #get video name to the last fi in a function and run that in the background: ( myfunc "$URL" ) & (Replace instances of $URL with $1 in the function, of course.) Personally, I'd rather get those values and know they are valid before spawning the download, but if they are time consuming it might be worthwhile.
OK, I see what you're asking now. You want those two individually paralleled. The short answer is, as far as I know, yes, but it isn't pretty. They can't directly set NAME and vRUL in that case so you'd have to jump through some hoops to retrieve those values.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.