Some Apache servers do activate the DirectoryIndex so you can easily go through the directories listing over HTTP and download some files. It’ s often used for giving possibility to people to download multiple releases for a package/software.
But in some cases, it can be useful to download the content of a full tree view.
This can be done by using a simple tool available by default on Linux (can be downloaded on Windows/Mac): wget
Let’s take an example!
We want to download all the files on https://pkg.mywebsite.eu/releases/ which contains all the latest releases of our preferred soft.
wget -r -np -nH --cut-dirs=1 -l 15 -R index.html https://pkg.mywebsite.eu/releases/
Here, we are using many options to say to wget what we want to do:
- -r : Turn on recursive retrieving (max 5)
- -np : Do not ever ascend to the parent directory when retrieving recursively
- -nH –cut-dirs=1 : Disable generation of host-prefixed directories, and remove 1 level (here, remove “pkg.mywebsite.eu” and create a “releases” dir)
- -l 15 : Change default depth for downloading to 15 levels (default is 5)
- -R index.html : Specify comma-separated lists of file name suffixes or patterns to reject
Let’s now add a complexity, this server is only available through a proxy to protect the content. The proxy is available on your localhost (127.0.0.1) on port 3128 (default port for Squid Proxy):
wget -r -np -nH --cut-dirs=1 -l 15 -R index.html -e use_proxy=yes -e http_proxy=127.0.0.1:3128 https://pkg.mywebsite.eu/releases/
Here, we just added 2 options to be able to send exactly the same request but using proxy:
- -e use_proxy=yes : Execute command to enable proxy
- -e http_proxy=127.0.0.1:3128 : Execute command to give URL to use for HTTP proxy