0

I want to download files with wget in a bash script. The url's look like this: https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r..

Problem is the $ doller sign in the url

When I download the file with double quotes I get a 403 because the $ sign is probably interpreted.

wget "https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r.."

When I single quote the url and download the file everything goes well:

wget 'https://xxx.blob.core.windows.net/somefolder/$somefile.webm?sv=xxx&sp=r..'

But the url should come from a line in a text file. So I read the file and pass the lines as url:

files=($(< file.txt))

# Read through the url.txt file and execute wget command for every filename
while IFS='=| ' read -r param uri; do 
    for file in "${files[@]}"; do
        wget "${file}"
    done
done < file.txt

I get the 403 here as well and don't know how to prevent the termimal from interpreting the dollar sign. How can I achieve this?

Marcel Hoekstra
  • 1,334
  • 12
  • 19
  • 1
    Use single quotes, not double quotes, to surround a string wherein a `$` should be literal. – Charles Duffy Jan 31 '23 at 19:26
  • 1
    The `$` should not be interpreting through a variable. Can you print the url contents? (change `wget` to `echo`). That should make the issue more obvious. – Jason Jan 31 '23 at 19:27
  • You're not using `param` and `uri` for anything? – Benjamin W. Jan 31 '23 at 19:27
  • 2
    Beyond that, `files=( $(...anything...) )` is inherently buggy; see [BashPitfalls #50](https://mywiki.wooledge.org/BashPitfalls#hosts.3D.28_.24.28aws_.2BICY.29_.29) -- if you want to populate an array with lines from a file, use `readarray -t files – Charles Duffy Jan 31 '23 at 19:27
  • That said, your script itself _doesn't_ run `wget "..."` the way your examples show (those examples briefly led me to mistakenly close the question as a duplicate, before taking a closer look and realizing they were irrelevant to your real problem): `"$file"` passes the variable's exact contents with no further interpretation, unlike the interpretation you would get if you put the variable's contents inside those same quotes in the source code. – Charles Duffy Jan 31 '23 at 19:28
  • Run `set -x` to enable logging, and compare actual and intended behaviors. If you have a working wget command you can run by hand, compare what `set -x` logs for that working command to what it logs when you run `bash -x yourscript`. – Charles Duffy Jan 31 '23 at 19:29
  • See: [Difference between single and double quotes in bash](http://stackoverflow.com/q/6697753/3776858) – Cyrus Jan 31 '23 at 20:38

1 Answers1

3

But the url should come from a line in a text file.

If you have file with 1 URL per line or are able to easily alter your file to hold 1 URL per line, then you might use -i file option, from wget man page

-i file

--input-file=file

Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from a file literally named -.)

If this function is used, no URLs need be present on the command line. If there are URLs both on the command line and in an input file, those on the command lines will be the first ones to be retrieved. If --force-html is not specified, then file should consist of a series of URLs, one per line.(...)

So if you have single file, say urls.txt you might use it like so

wget -i urls.txt

and if you have few files you might concat them and shove through standard input like so

cat urls1.txt urls2.txt urls3.txt | wget -i -

If file(s) contain additional data then remember to process them so GNU wget will get only URLs.

Daweo
  • 31,313
  • 3
  • 12
  • 25