Parallel script execution

Jow Forums I need a way to start multiple scripts on GNU +Linux.

These are bullshit scripts like copy past etc.

I was using GNU parallel.
The problem is...
its GNU parallel.

How do I explain this to someone who never used it.

Picture extremely relevant.

The number of scripts I can start is limited by the number of PROCESOR threads I have on my computer.

I think you see the problem here
>You can only start 8 things at the same time

Not only this I think every bullshit script gets complete priority or some shit because the core will max out.
Picture extremely relevant.

So what do I use to have multiple (BASH) scripts started by a different (BASH) script?

GNU parallel works however its...
Picture extremely relevant.

Attached: parallel.png (1221x651, 1.24M)

Other urls found in this thread:

youtube.com/watch?v=56ucT_Hw4bg
youtube.com/watch?v=XyNlqQId-nk
youtube.com/watch?v=E9U9xS4thxU
youtube.com/watch?v=6ryjB9v8OQU
youtube.com/watch?v=hY7m5jjJ9mM
youtube.com/watch?v=kcKM00lmrjQ
youtube.com/watch?v=i-AXImNxCAE
youtube.com/watch?v=iRXJXaLV0n4
youtube.com/watch?v=rNSnfXl1ZjU
youtube.com/watch?v=opYts0nFOvs
youtube.com/watch?v=ZxMl1SDJ7po
youtube.com/watch?v=p1V4IGFMunk
youtube.com/watch?v=R9NCI-naXbU
youtube.com/watch?v=FX5jFhUyabM
youtube.com/watch?v=DXUAyRRkI6k
youtube.com/watch?v=i8nLrvcNcrg
pastebin.com/yV3fhQm8
pastebin.com/XEbeBAVh
pastebin.com/piWtS8ra
youtube.com/watch?v=56u
youtube.com/watch?v=XyN
youtube.com/watch?v=E9U
youtube.com/watch?v=6ry
youtube.com/watch?v=hY7
twitter.com/SFWRedditGifs

Python, subprocess.Popen()

I need this in hash however.

*bash
I need this in bash however.

Is there a specific reason you can't use a Python script to start your bash scripts? I guess it's for a company?

> I guess it's for a company?
No its personal.
>Is there a specific reason

Be me.
Use youtube-dl to download videos online.
Some video services decide to limit video download to some thin ass limit.
I simply have a pree-made file full of
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate


Copy URL to the last line and create a list of bash commands super easy.
After finishing simply use CRTL+END+SHIFT after the last command to delete the useless and empty commands.

Use another script file to execute every command at the same time.
parallel --no-notice -a videos.sh


Turns out its limited by the number of cores.
Internet connection is not even at 30%.
All cores are used.
All cores are maxing out.
I-i-i-i-i-i-t werks in a way...

MFW OP-picture + this

Attached: 1509392017382.png (654x539, 525K)

I don't see why the execution script needs to be bash if all it's doing is launching another script.
Maybe you could try backgrounding the scripts in a loop without waiting if they don't need to start at exactly the same time. Simplify the command script so it only runs one instance and provide it the URL as an argument, or something like that.
Can't help further since I don't know bash that well.

> Python script to start your bash scripts
Can you make it super easy like I'm doing in
(I don't want to create 1 file for 1 video)
Can you tell me how exactly do do this?

Also I don't like piling shit loads of programing languages on top of one another.
Can bash seriously not executer all scripts at the same time?
I don't even give a shit about collision because they will never collide and its irrelevant (1 script works only on 1 separate file).

something like this?
for i in $(seq NUM_PROCESSES); do ./your-script-here & disown; done

>I don't see why the
The most important part is my convenience in copy pasting the videos.
I like copy pasting it into the videos.sh script.

Something like that. You can pass it the title and so on from the command line so that it can be reusable for any video.

Do you even bash?

Myparallelexecution.sh
> call.sh &
> call.sh &
> ...
> wait

Each call gets executed in its own subshell and executed in the background (that's what the & does)

Optionally you could redirect output into a log file or everything will interleaved but its not requir d.

At the end you wait until all processes finished.

import os
os.system("echo 'your bash command here'")

What if call.sh fails?

OK I'm trying this I get an error
>NUM_PROCESSES
is simply not recognized.
for i in $(seq NUM_PROCESSES); do ./x.sh & disown; done

Then the call fails but others still get executed. If you want to know which fails and conditionally rexecute things get ugly in bash :(

You forgot the $

Nice.
Thanks!

It works!
However is there a way for me to skip having the need to add & at the end?
Because I simply want to copy past URLs with minimal interrupts.

Is there a way for another script file to execute every command like it was ending with & ?

Attached: tenor(1).gif (320x270, 289K)

Sorry bro.
Its a direct copy past of

Its more me not wanting to turn this into a FPS game where I need to precisely click in a specific place to CTRL+V the URL.

hitting the end of a script is not that hard.

>Each call gets executed in its own subshell and executed in the background (that's what the & does)
The console emulator is displaying stuff youtube-dl is saying however
(This is a good thing and i like this)
>Optionally you could redirect output into a log file or everything will interleaved but its not requir d.
>At the end you wait until all processes finished.
Na its cool I like it working in one window even if its multiple process talking over one another.
This shows me its working.

How can I make this
for i in $(seq NUM_PROCESSES); do ./your-script-here & disown; done

Work?

I just meant it as a template for you to fill out based on your needs

"NUM_PROCESSES" is a positive integer for however many instances of the script you want to run, and "./your-script-here" is the path to the script you want to run

so for example if you want to run 9 instances of "/opt/bin/turds.sh", you could say
for i in $(seq 9); do /opt/bin/turds.sh & disown; done

>NUM_PROCESSES" is a positive integer for however many instances of the script you want to run
Ow shit.
This is a problem.
because I have no idea how many scripts will be in the file.
Sometime I take 20 videos and other times 40~50.

Its not like i want to count how many URLs I copy pasted into the videos.sh. script.

So that would work like this:
> #! /bin/bash
> for command in "$@"
> do
> ${command} &
> done
> wait

You can test this with:
> time yourscript.sh 'sleep 5' 'sleep 8' 'sleep 1'
time taken will be (very close to) 8 sec.

Keep in mind that you almost always should quote the params.

$(seq $(wc -l file-with-one-url-per-line))

Thanks for this.
However this is failing to execute for me
for i in $(seq $(wc -l x.sh)); do ./x.sh & disown; done


x.sh is the one with the youtube-dl commands

whoops sorry i meant this:
$(seq $(cat file-with-one-url-per-line | wc -l))


looks like wc will also print the file name if you dont pipe into stdin

Sure, because ./x.sh it's not a valid file, you have to get the i line

hmm dude?
./x.sh
is one of the text files I created and I'm working with as we speak.
This is actual production code.

see

?
In the script you are trying to execute the entire x.sh for each x.sh line, that's not what you want right?
You want to execute each line of x.sh separately with the &

>whoops sorry i meant this:
No problem.

Also this code
for i in $(seq $(cat x.sh | wc -l)); do ./x.sh & disown; done

1)All my cores are maxing out.
Also if I close the GUI console window.
2)Internet connection is maxed out

3)looks painfully linear if executed.
Meaning that I don't see the files created in the folder while starts shitting multiple files in the folder after start meaning its downloading multiple files at the same time.


Are you certain there are not other problems with the code?
In like its getting something ridicules like
for i in $(seq 2354567567

OP, your problem is I/O bound. Bash is not the right tool for the job. Use something else.

xargs

x.sh has literally something like
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=56ucT_Hw4bg
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=XyNlqQId-nk
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=E9U9xS4thxU
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=6ryjB9v8OQU
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=hY7m5jjJ9mM
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=kcKM00lmrjQ
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=i-AXImNxCAE
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=iRXJXaLV0n4
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=rNSnfXl1ZjU
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=opYts0nFOvs
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=ZxMl1SDJ7po
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=p1V4IGFMunk
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=R9NCI-naXbU
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=FX5jFhUyabM
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=DXUAyRRkI6k
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=i8nLrvcNcrg
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=56ucT_Hw4bg

in it and its 100% of the file

i don't think we're on the same page

in the op you said you wanted to launch multiple bash scripts from a single bash script, and that's what the line i gave you does

are you looking to run a different script on each iteration, or the same one? if it's the former, get rid of the "seq" and loop over each line of the file with the links, and pass "$i" as an argument to your youtube-dl command.

THE FUCK!
Why is Jow Forums adding
in my code!
pastebin.com/yV3fhQm8
Here is 100% of x.sh

Exactly, try to figure this out, for each line of this file execute this file
If you have n lines you execute n^2 commands

Me again, I fiddled with termux (phoneposting) and found a better solution
Here is a working single line:

> awk '{system($0 "&")}' x.sh

^this use awk, the most underrated terminal program

>in the op you said you wanted to launch multiple bash scripts from a single bash script
Yes I still want this.
>are you looking

And I'm telling you
executing these 2 files in 1 folder
x.sh
pastebin.com/yV3fhQm8

and
a.sh
pastebin.com/XEbeBAVh

Only creates 1 file in the folder while downloading contrast this with running this script
pastebin.com/piWtS8ra

Who will create multiple files after seconds.
you can see this in the file browser.

Like run the scripts yourself.

>executing these 2 files in 1 folder
I'm only doing
./a.sh
BTW.

WOW it looks like its working.
Thanks.
I see the results my self after the test finishes.

>^this use awk, the most underrated terminal program
Thanks if only i knew what it is doing.

Ah I see. x.sh+a.sh do a different thing from the third pastebin you linked.

I assumed you wanted to launch the same script multiple times, and what is happening is that all the commands in x.sh are getting run over and over again, multiple times each.

The other pastebin runs each command in x.sh once.

But yeah, if you wanted to do that in the first place, then is the best approach

a.txt:
youtube.com/watch?v=56ucT_Hw4bg
youtube.com/watch?v=XyNlqQId-nk
youtube.com/watch?v=E9U9xS4thxU
youtube.com/watch?v=6ryjB9v8OQU
youtube.com/watch?v=hY7m5jjJ9mM


a.sh
#!/bin/sh

for url in `cat a.txt`; do
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate $url &
done

I wanted to download multiple videos using youtube-dl with the minimal level of difficulty for me copy pasting the URLs.

No problem, I'm happy that it works.
From what I've read, you are competent enough to use Linux and copy paste snippet from the web, but not to create your own.
If you have time I suggest you to learn some basic programming, maybe in python, it's fun and useful.

Sweat thanks!
A lot.
>I suggest you to learn some basic programming
I'm no master programmer.
However I know C and other languages.

However C is not something that is for system automation, you know the nice stuff like using youtube-dl.
And bash is mostly chaining together other programs.

I wanted to know how to start something at the same time goggled it and the results where all for GNU parallel then look at OP picture to see my frustration.

Is the best solution I think
After learning that & exists I was thinking about a similar solution however I wanted to ask before I start trying to write my own abomination.

For what is GNU parallel actually used?
I'm interested.

Attached: handheart.jpg (500x334, 82K)

Use -j 0?

I would be surprised if youtube didn't throttle you to oblivion no matter how parallel you download videos. Don't expect to get any gain from this, I would imagine if youtube recognizes a bot it throws captchas at it, that would block youtube-dl.

Fun fact.
>youtube
Gives you max download speed all the time.
> Don't expect to get any gain from this
Other services who throttle are tricked by this and this is a fact!
>recognizes a bot it throws captchas at it
Nope they are not detecting this.

Feel free to test it yourself and enjoy the best solution given by
You are all welcome.

Attached: 1509037644056.png (576x432, 603K)

Also,m if you're still here, here's an extract from the parallel man page:

--jobs N
-j N
--max-procs N
-P N Number of jobslots. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which
will run one job per CPU core.

If --semaphore is set default is 1 thus making a mutex.

Ye thanks however the problem was that my CPU cores where maxed out all the time.

Thanks for this.
Also for what exactly is GNU parallel used?

Same as what you were using it for. Running tasks in parallel.

Why did my cores max out if I run it?
>Same as what you were using it for. Running tasks in parallel.
bullshit youtube-dl ?
Or some protein folding crap?

>The number of scripts I can start is limited by the number of PROCESOR threads I have on my computer.
Meanwhile AmigaOS was fully multitasking on a single core in 80s.

I doubt biology scientists would use bash for their computing needs. My bet is its target audience is people like you, automating tasks to make their workflow easier.

>Why did my cores max out if I run it?
I don't know. I didn't use parallel a lot (I generally use perl not bash) but when I did, I didn't have any issues. Are your cores maxing out with bash's &?

I know.
Every OS was multitasking I was only curious how to make it work in bash.
Because this is what I got from GNU parallel.

Do you think the picture is acuter?

>Are your cores maxing out with bash's &?
There is a spike and hill in all cores if I start & however after this its normal.

See picture.
I'm runing this in the terminal emulator and the PC is on some load now.
However even if they spike I did not see system lags.

GNU parallel was maxing out all the time.
>automating tasks to make their workflow easier.
Then why the fuck is it limited to the number of my processor threads?
By default?
Who's brilliant idea was this?

I understand if it gives complete priority to every script etc.
However this...

Attached: 2018-04-26.png (1801x223, 74K)

i think you've misunderstood what parallel is for
if you want to just open a bunch of shit, do this
thing &
otherthing &
anotherthing &
etc &

also, you can tell parallel how many jobs to run at once, obviously, since job management is it's entire purpose

What am I looking at?
Elaborate this to the given examples:

>Then why the fuck is it limited to the number of my processor threads?
Cause 8 cars usually dont fit down a 4 lane highway

>i think you've misunderstood what parallel is for
I was thinking the same is it for some protein folding crap?

echo -e {1..100}'\n' | xargs -n1 -P5 -I{} echo {}

will run 5 instances of whatever command you want in parallel, sequenitally iterating over the input range
killing the xargs will kill the subprocesses so it's a bit better than & imo

Also thanks a lot for the good examples however other have given the same examples before you (not that I expect everyone to read every post)
I'm only curious for hat exactly GNU parallel is for.

it's purpose is to take a large list of commands, and run them in parallel, typically one per core, since that makes sense in most situations, which is why it defaults to that
it can also handle jobs on remote machines and other fancy shit
as an example, lets say you want to resize a folder of 1000 pictures, it may not be possible to do them all at once, due to memory constraints, and if you have only 4 cores (and not 1000), it won't be any faster than doing 4 at a time, which is where parallel comes in, it will go through all 1000, but only 4 at any one time, when one finishes, it moves to the next

-- and a working example (transcoding a bunch of audio files);
$ time for i in *.flac; do ffmpeg -i "$i" -c:a libopus -b:a 112k "${i%.flac}".opus; done
...
real 0m41.579s
user 0m49.216s
sys 0m0.600s
$ time parallel ffmpeg -i "{}" -c:a libopus -b:a 112k "{.}".opus ::: *.flac
...
real 0m12.806s
user 0m51.855s
sys 0m0.848s

Hmm.
I don't quite think I understand.
Why not make these things sequentially?

For example GIMP has a batch function to make massive photo editing in one go on multiple files.

The stuff I wanted it for is bullshit to circumvent the limitations on some streaming services.
I can use the bash & to string together other bullshit like.
Updating youtube-dl other system clean ups etc.

>only 4 cores
Why is it limited per core? And not some custom number you give it?

I can understand if I have thousands commands however the per core and core maxing out suggested some dedication of one core.

>Why is it limited per core? And not some custom number you give it?
it isn't, it /defaults/ to one job per core, for the reason i explained
you can specify an arbitrary number of jobs with "-j"
in most cases, this is used to run jobs which max out one core, though you could for example, tell it to run half as many 2-threaded jobs, or run more jobs than cpus if the jobs are limited in another way, such as by a network

>this is used to run jobs which max out one cor
I'm interested in examples of these jobs.
For example you did talk about audio.

most things are serial in nature, most of the time, jobs that use multiple cpus are just a set of serial jobs running in parallel, at least to a degree
the reason is simple, splitting a job into multiple pieces requires that the resulting pieces don't depend on each other
for example, in video encoding, you can't encoding multiple consecutive frames in parallel (except in the case of I-frame-only video), as most frames depend on the output of the previous frame(s), think of it like a production line, each step depends on the last step to be completed, so you can't skip anything

speaking of audio, i *could* split one audio track into multiple jobs, to encode a single audio file in parallel
most (if not all) audio codecs work like video formats, where the state of the decoder resets periodically, to make seeking easier (if it didn't, you'd have to decode an hour of audio to seek to the 1 hour mark, for example)
each time this happens, you remove the need to know what any previous state was, making it possible to process each of these chunks in parallel

>I'm interested in examples of these jobs.
im lazy as fuck so i just write single threaded things and parallel them.

Op here I was trying to rewrite this script so that it executes everything line by line
a.sh
#!/bin/sh

for com in `cat x.sh`; do
$com &
done


x.sh
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=56u
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=XyN
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=E9U
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=6ry
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=hY7


However I'm getting error on line 4.
Anyone has ideas why?

PS:
The reason for this split is me experimenting and wanting to get the ability to run these coands
1)
Sequentially
Simply
./s.sh

2)
All at once
./a.sh

3)
or even use GNU parallel (if there is shit loads of videos and stuff) by swamping or changing
a.sh

>./s.sh
was about to write
./x.sh

#!/bin/env bash
while read line; do
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate -- "$line"
done

I wanted the
youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate youtube.com/watch?v=XyN

Inside of x.sh
while a.sh is
#!/bin/sh

for com in `cat x.sh`; do
$com &
done


Or something like this.

Just make a cron job lmao

while read line; do
eval "$line" &
done

why would you want that? His version is much cleaner.

Simple see:
The reason for this split is me experimenting and wanting to get the ability to run these commands
1)
Sequentially
Simply
./x.sh

2)
All at once
./a.sh

3)
or even use GNU parallel (if there is shit loads of videos and stuff) by swamping or changing
a.sh

I have a hard on for changing the ways I run things and not locking myself into one solution.

1)
parallel -j1 -a links.txt youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate {}
2)
parallel -j100 -a links.txt youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate {}

3)
parallel -a links.txt youtube-dl -ci -o "%(title)s-%(id)s.%(ext)s" --no-check-certificate {}

(yes, i realise 2 isn't "all at once", but it's really not a good idea to truly have no limit, imagine spinning up 2000 instances of youtube-dl at once, not only will it choke your machine, it'll use a tons of ram, kill network reponse times, and generally just be completely useless)

This is why I'm using 2 files.
One with the commands(x.sh) and another to initialize the commands(a.sh).

Sometime its a good idea to run it sequentially because some video is fucked up(unavailable in your country, removed for offensive content) and you want to see the output.

Thanks a lot the best solution!
You all rock.

let no one say that Jow Forums is useless.

Basically in my setup I can change the "Parallelization engine" (I made this word up) to whatever I want.

And I'm not locked into one solution,

Software modularity is a good thing.

>Internet connection is not even at 30%.
>All cores are used.
>All cores are maxing out.

>implying you could max out your internet connection
nigga you getting jewed by the servers, even if you didn't have a fisher price CPU, you'd still be far from filling that pipe

bumb

parallel is easy to use.
parallel youtube-dl ::: url1 url2 url3 url4 etc
# Or
parallel youtube-dl < file_with_urls_on_each_line