How to download files from text file

user1757774

6/26/23, 12:48 PM

I have multiple text files such as a0001.txt, a0002.txt...upto a0900.txt containing direct image links https://example.com/photos/example.jpg

I want to batch download all links from a0001.txt to a0001 folder and so on in respective folder names as text file.

Links must be download in order so Images renamed with prefix. For sorting purposes.

What I tried was below. But it downloading one file at a time. Super slow. Any other way to do it in parallel?

#! /usr/bin/env
bashif [ -z $1 ] || egrep -qv '^https?:\/\/[^[:space:]]+' $1; then
  echo "Usage: $0 FILE"
  echo "FILE must be a newline-separated list of URLs."
  exit
fi

INPUT_FILE=$1
OUTPUT_DIRECTORY="${INPUT_FILE%.*}"

mkdir -p $OUTPUT_DIRECTORY

i=1
while read URL; do
  FILENAME="${URL##*/}"
  curl -L $URL > "${OUTPUT_DIRECTORY}/${i}_${FILENAME}"
  i=$(($i+1))
done < $INPUT_FILE

0 + 0

command-line

bash

scripts

terdon

6/26/23, 3:29 PM

Is your shebang line really `#! /usr/bin/env` or did you mean `#! /usr/bin/env bash`?

waltinator

6/26/23, 4:39 PM

Always paste your script into `https://shellcheck.net`, a syntax checker, or install `shellcheck` locally. Make using `shellcheck` part of your development process.

FedKad

6/26/23, 7:17 PM

You can start the `curl` processes in parallel by appending `&` to the end of the command. But you should not start too many processes. For example in every ten iterations, you can run a `wait` command.

waltinator

6/26/23, 9:06 PM

Read `man seq` for generating the "`A00001, ..., A0900` strings. Append "`&`" to your `curl`" command.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to download files from text file

TH: วิธีดาวน์โหลดไฟล์จากไฟล์ข้อความ

RO: Cum să descărcați fișiere din fișier text

RU: Как скачать файлы из текстового файла

VI: Cách tải tệp từ tệp văn bản

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.