How to use wget on windows - Доктор Windows - ваш надежный помощник

What is wget?

Wget is a free GNU command-line utility tool used to download files from the internet. It retrieves files using HTTP, HTTPS, and FTP protocols.

It serves as a tool to sustain unstable and slow network connections. If a network problem occurs during a download, this helpful software can resume retrieving the files without starting from scratch.

Another important aspect is its capability of recursive downloads, with which it mirrors websites. It transfers parts of a website by following links and directory structure, thus creating local versions of webpages.

The wget command is also highly flexible and can be used in terminals, scripts, and cron jobs. During the download, the user does not have to be active nor logged in. As wget is non-interactive, it can independently run in the background.

Read this article to learn how to use some of the most common wget commands.

Tutorial on how to use wget commands with examples.

How to Check if wget is Installed?

Most likely, the wget package is already on your system as it now comes pre-installed on most Linux distributions.

To check, open the terminal window and type in:

wget

If you have the wget software, the output tells you that the wget command is missing a URL, as shown in the image below:

wget Command Not Found

If the output displays wget command not found you need to download and install the tool manually. Below you will find the installation instructions for Ubuntu/Debian, CentOS, and Windows.

How to Install wget on Ubuntu/Debian?

To install wget on Ubuntu or Debian releases, use the command:

sudo apt-get install wget

How to Install wget on CentOS/Fedora?

To install wget on CentOS or Fedora, type the following command:

sudo yum install wget

How to Install wget on Windows?

To install and configure wget for Windows:

Download wget for Windows and install the package.
Add the wget bin path to environment variables (optional). Configuring this removes the need for full paths, and makes it a lot easier to run wget from the command prompt:
- Open the Start menu and search for “environment.”
- Select Edit the system environment variables.
- Select the Advanced tab and click the Environment Variables button.
- Select the Path variable under System Variables.
- Click Edit.
- In the Variable value field add the path to the wget bin directory preceded by a semicolon (;). If installed in the default path, add C:Program Files (x86)GnuWin32bin.
Open the command prompt (cmd.exe) and start running wget commands.

Introduction to wget Syntax

The wget syntax has the following pattern:

wget [option][URL]

Each [option] has its long and short form which are conveniently interchangeable. This attribute specifies what to do with the URL that follows.

[URL] is the address of the file or directory you wish to download.

Download File from Web

To download a file from the web use:

wget [URL]

For example, to install Tomcat 9, first you need to download the package with wget using the command:

wget http://apache.cs.utah.edu/tomcat/tomcat-9/v9.0.20/bin/apache-tomcat-9.0.20.tar.gz

Download File and Save Under Specific Name

To download a file and save it under a specified name run:

wget -O [file_name] [URL]

The wget command allows you to rename files prior to downloading them on your computer.

For instance, you may want to install Terraform. To download the package and rename it terraform.zip use the following command:

wget -O terraform.zip https://releases.hashicorp.com/terraform/0.12.2/terraform_0.12.2_linux_amd64.zip

Download File to Specific Directory

By default wget downloads a file in the directory the user is in. To save the file in a different location, add the -P option:

wget -P [wanted_directory] [URL]

For example, while installing Git on Ubuntu, you can download the package in the /temp directory with the command:

wget -P /temp https://github.com/git/git/archive/master.zip

Set Download Speed

You can set the download speed when downloading a big file, so it does not use the full available bandwidth. The download speed is defined in kilobytes (k) and megabytes (m). Use the command:

wget --limit-rate [wanted_speed] [URL]

For example, if you are installing NVIDIA TESLA drivers on Linux and want to limit the download speed to 1 megabyte, would use the command:

wget --limit-rate 1m http://us.download.nvidia.com/tesla/396.37/nvidia-diag-driver-local-repo-ubuntu1710-396.37_1.0-1_amd64.deb

Continue Download After Interruption

Instead of having to start from scratch, wget can resume downloading where it stopped before the interruption. This is a useful feature if there is a lost of connection while downloading a file.

wget -c [URL]

For instance, you may want to install a Mumble Server on Linux and suddenly lose internet connection while downloading the installation file. To continue downloading, type in the command:

wget -c https://github.com/mumble-voip/mumble/releases/download/1.2.19/murmur-static_x86-1.2.19.tar.bz2

Download Multiple Files

wget allows downloading multiple files at the same time using the command:

wget -i [file_name]

To do so, follow the steps outlined below:

1. First, create and open a file under the name MultipleDownloads.txt (or a name of your choice), using a text editor. In this case, we used Nano:

nano MultipleDownloads.txt

2. Once in the editor, add the URLs of the packages you want to download, one per line.

3. Save and exit the file.

4. Run the following wget command in the terminal window:

wget -i MultipleDownloads.txt

This prompts wget to download from each URL in the text file.

Download Web page (Mirror Web page)

With wget you can download an entire website from the internet, using the -m option. It prompts wget to create a mirror of the specified webpage. The basic command for doing so is:

wget -m [URL]

For example:

wget -m <a href="https://phoenixnap.com/" target="_blank" rel="noreferrer noopener">https://phoenixnap.com</a>

Download via FTP

To download via FTP, type in the username and password of the FTP server, followed by the ftp address:

wget --ftp-user=[ftp_username] --ftp-password=[ftp_password] ftp://...

For instance:

wget --ftp-user=sofiftp --ftp-password=TopSecretPassword ftp://123.456.7890

Download in Background

You can download in the background, a practical feature when dealing with a large file:

wget -b [URL]

You can check the status of the download with the command:

tail -f wget -log

To download the RPM package manager in the background, type:

wget -b http://some_website/sample_file.rpm

Increase Retry Attempts

You can set how many times wget attempts to download a file after being interrupted by a bad network with:

wget --tries=[number_of_tries] [URL]

By default, the number of retry attempts is set to 20.

You can also set the number to infinity with the values 0 or inf, as in the following example:

wget --tries=inf http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo

Skip Certificate Check

By default, wget checks whether the server has a valid SSL/TLS certificate. If it does not identify an authentic certificate, it refuses to download.

The --no-check-certificate option is used to avoid certificate authorities checking for a server certificate. However, utilize it only if you are sure of the website’s credibility or are not worried about security issues it may cause.

wget --no-check-certificate [URL]

If http://enteratonerisk.com has an untrusted certificate, but will not harm the system, you can download it with:

wget --no-check-certificate http://enteratonerisk.com

Change User Agent

When downloading a webpage, wget essentially emulates a browser. In some cases, the output might say you don’t have permission to access the server, or that the connection is forbidden. This may be due to a website blocking client browsers that have a specific “User-Agent.”

“User-Agent” is a header field that the browser sends to the server it wants to access. Therefore, to download from a server that is refusing to connect, try to modify the user agent.

Find a database of all user agents online, search for the one you need and run the command:

wget --user-agent="User Agent Here" "[URL]"

wget -U "User Agent Here" "[URL]"

For example, to emulate Chrome (version 74), you would change the user agent with the command:

wget --user-agent=" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "https://phoenixnap.com"

Note: Make sure you always download from a trusted source because wget can instruct a download of a script from a malicious source. We outline this and many other dangerous commands in our article 14 Dangerous Linux Terminal Commands.

Conclusion

This article sums up why wget is such a powerful tool for downloading files over the internet. It also serves as a good reference for beginners with its list of 12 essential wget commands and examples.

Источник

WGET is a free tool to crawl websites and download files via the command line.

In this wget tutorial, we will learn how to install and how to use wget.

What is Wget?

Wget is free command-line tool created by the GNU Project that is used todownload files from the internet.

It lets you download files from the internet via FTP, HTTP or HTTPS (web pages, pdf, xml sitemaps, etc.).
It provides recursive downloads, which means that Wget downloads the requested document, then the documents linked from that document, and then the next, etc.
It follows the links and directory structure.

It lets you overwrite the links with the correct domain, helping you create mirrors of websites.

How to Install Wget

Check if Wget is installed

Open Terminal and type:

If it is installed, it will return the version.

If not, follow the next steps to download wget on either Mac or Windows.

Download Wget on Mac

The recommended method to install wget on Mac is with Homebrew.

First, install Homebrew.

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then, install wget.

Download Wget on Windows

To install and configure wget for Windows:

Download wget for Windows and install the package.

Copy the wget.exe file into your C:WindowsSystem32 folder.
Open the command prompt (cmd.exe) and run wget to see if it is installed.

Here is a quick video showing you how to download wget on windows 10.

Wget Basics

Let’s look at the wget syntax, view the basic commands structure and understand the most important options.

Wget Syntax

Wget has two arguments: [OPTION] and [URL] .

wget [OPTION]... [URL]...

[OPTION] tells what to do with the [URL] argument provided after. It has a short and a long-form (ex: -V and --version are doing the same thing).
[URL] is the file or the directory you wish to download.
You can call many OPTIONS or URLs at once.

View WGET commands

To view available wget commands, use wget -h.

14 Wget Commands to Extract Web Pages

Here are the 11 best things that you can do with Wget:

Download a single file
Download a files to a specific directory
Rename a downloaded files
Define User Agent
Extract as Googlebot
Extract Robots.txt when it changes
Convert links on a page
Mirror a single page
Extract Multiple URLs from a list
Limit Speed
Number of attempts
Use Proxies
Continue Interrupted Downloads
Extract Entire Website

Download a single file with Wget

$ wget https://example.com/robots.txt

Download a File to a Specific Output Directory

Here replace <YOUR-PATH> by the output directory location where you want to save the file.

$ wget ‐P <YOUR-PATH> https://example.com/sitemap.xml

Rename Downloaded File when Retrieving with Wget

To output the file with a different name:

$ wget -O <YOUR-FILENAME.html> https://example.com/file.html

Define User Agent in WGET

Identify yourself. Define your user-agent.

$ wget --user-agent=Chrome https://example.com/file.html

$ wget --user-agent="Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/path

Let’s extract robots.txt only if the latest version in the server is more recent than the local copy.

First time that you extract use -S to keep a timestamps of the file.

$ wget -S https://example.com/robots.txt

Later, to check if the robots.txt file has changed, and download it if it has.

$ wget -N https://example.com/robots.txt

Wget command to Convert Links on a Page

Convert the links in the HTML so they still work in your local version. (ex: example.com/path to localhost:8000/path)

$ wget --convert-links https://example.com/path

Mirror a Single Webpage in Wget

To mirror a single web page so that it can work on your local.

$ wget -E -H -k -K -p --convert-links https://example.com/path

Add all urls in a urls.txt file.

https://example.com/1
https://example.com/2
https://example.com/3

To be a good citizen of the web, it is important not to crawl too fast by using --wait and --limit-rate.

--wait=1: Wait 1 second between extractions.
--limit-rate=10K: Limit the download speed (bytes per second)

Define Number of Retry Attempts in Wget

Sometimes the internet connection fails, sometimes the attempts it blocked, sometimes the server does not respond. Define a number of attempts with the -tries function.

$ wget -tries=10 https://example.com

How to Use Proxies With Wget?

To use proxies with Wget, we need to update the ~/.wgetrc file located at /etc/wgetrc.

You can modify the ~/.wgetrc in your favourite text editor

$ vi ~/.wgetrc # VI
$ code ~/.wgetrc # VSCode

And add these lines:

use_proxy = on
http_proxy =  http://username:password@proxy.server.address:port/
https_proxy =  http://username:password@proxy.server.address:port/

Then, by running any wget command, you’ll be using proxies.

Alternatively, you can use the -e command to run wget with proxies without changing the environment variables.

wget -e use_proxy=yes -e http_proxy=http://proxy.server.address:port/ https://example.com

How to remove the Wget proxies?

When you don’t want to use the proxies anymore, update the ~/.wgetrc to remove the lines that you added or simply use the command below to override them:

Continue Interrupted Downloads with Wget

When your retrieval process is interrupted, continue the download with restarting the whole extraction using the -c command.

$ wget -c https://example.com

Recursive mode extract a page, and follows the links on the pages to extract them as well.

This is extracting your entire site and can put extra load on your server. Be sure that you know what you do or that you involve the devs.

$ wget --recursive --page-requisites --adjust-extension --span-hosts --wait=1 --limit-rate=10K --convert-links --restrict-file-names=windows --no-clobber --domains example.com --no-parent example.com

Command	What it does
–recursive	Follow links in the document. The maximum depth is 5.
–page-requisites	Get all assets (CSS/JS/images)
–adjust-extension	Save files with .html at the end.
–span-hosts	Include necessary assets from offsite as well.
–wait=1	Wait 1 second between extractions.
–limit-rate=10K	Limit the download speed (bytes per second)
–convert-links	Convert the links in the HTML so they still work in your local version.
–restrict-file-names=windows	Modify filenames to work in Windows.
–no-clobber	Overwrite existing files.
–domains example.com	Do not follow links outside this domain.
–no-parent	Do not ever ascend to the parent directory when retrieving recursively
–level	Specify the depth of crawling. `inf` is used for infinite.

$ wget --spider -r https://example.com -o wget.log

Wget VS Curl

Wget’s strength compared to curl is its ability to download recursively. This means that it will download a document, then follow the links and then download those documents as well.

Use Wget With Python

Wget is strictly command line, but there is a package that you can import the wget package that mimics wget.

import wget
url = 'http://www.jcchouinard.com/robots.txt'
filename = wget.download(url)
filename

Debug Wget Command Not Found

If you get the -bash: wget: command not found error on Mac, Linux or Windows, it means that the wget GNU is either not installed or does not work properly.

Go back and make sure that you installed wget properly.

About Wget

👑 Wget was developed by	Hrvoje Nikšić
🏠 Wget is Maintained by	Tim Rühsen and al.
🛎️ Wget Supported Protocols	HTTP(S), FTP(S)
👛 Wget was Created In	January 1996
🍀 Installing Wget	brew install wget
📙 Wget Command	wget [`option`]…[`URL`]…

Detail table about WGET

Wget FAQs

What is Wget Used For?

Wget is used to download files from the Internet without the use of a browser. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

How Does Wget Work?

Wget is non-interactive and allows to download files from the internet in the background without the need of a browser or user interface. It works by following links to create local versions of remote web sites, while respecting robots.txt.

What is the Difference Between Wget and cURL?

Both Wget and cURL are command-line utilities that allow file transfer from the internet. Although, Curl generally offers more features than Wget, wget provide features such as recursive downloads.

Can you Use Wget With Python?

Yes, you can run wget get in Python by installing the wget library with $pip install wget

Does Wget Respect Robots.txt?

Yes, Wget respects the Robot Exclusion Standard (/robots.txt)

Is Wget Free?

Yes, GNU Wget is free software that everyone can use, redistribute and/or modify under the terms of the GNU General Public License

What is recursive download?

Recursive download, or recursive retrieval, is the capacity of downloading documents, follow the links within them and finally downloading those documents until all linked documents are downloaded, or the maximum depth specified is reached.

How to specify download location in Wget?

Use the -P or –directory-prefix=PREFIX. Example: $ wget -P /path <url>

Conclusion

This is it.

You now know how to install and use Wget in your command-line.

SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. In a quest to programmatic SEO for large organizations through the use of Python, R and machine learning.

Источник

Wget proves that “new” doesn’t always work better. Since its release in 1996, this application has become one of the most popular download managers in the world. Almost all Linux operating systems have included Wget as the default application. The simple and simple way to use is able to perform various forms of downloading activities, such as downloading single files , entire website files , videos , large files , and so on.

This application is very simple, so for new users it doesn’t take long to use it. Just learn a few commands below to download.

In this guide we will discuss what Wget is and examples of commonly used Wget commands on Windows and Linux. Hopefully after following this guide you can use Wget to the full. Moreover, this application has many functions and benefits.

What is Wget Command?

Wget is a download manager tool that is quite simple but very useful. Developed by the GNU Project, this application is one of the best download managers at the moment. Even though it is intended for Linux, you can now use it on various operating systems: Linux, Windows, and so on.

Related to the name, Wget comes from World Wide Web Get. Of course this is in accordance with its function to download various types of files on the internet. You can also use this application offline (connected to a local network) to exchange data between computer devices.

How to Install Wget

1. Linux

How to install Wget on Linux is quite easy. You just simply open Terminal and type the following command:

Ubuntu
$ sudo apt install wget
CentOS:
$ sudo yum install wget
Fedora:
$ sudo install install wget

2. Windows

There are several steps to install Wget on Windows, namely:

Download the Wget installation file for Windows
Perform the installation process
Check the use of Wget
(Optional) Direct the path to the program

Step 1 — Download the installation file

You can do Wget in Windows by downloading the installation file first. You can download it via the SourceForge page. You can also download it from other sites.

Step 2 — Perform the installation process

The second step is to install Wget on Windows. Just double click the installation file and follow the instructions given.

The Wget installation window on Windows.

Step 3 — Check the usage of Wget

Until this process, you can actually use Wget. But you have to access the folder that Wget program files use. The trick is to use Command Prompt. For example, when you download a file via a URL, the command is used like this:

“C: Program Files (x86) GnuWin32 bin wget.exe” [addressURL]

* AkunURL is replaced with the URL of the file to be downloaded.

Note : Until this step, you have finished following the guide on how to install Wget on Windows. But if you want convenience when running the download process, use step 4 to set the Environment Variable .

Step 4 — (Optional) Direct the path to the program

So that you don’t need to direct the directory to the program when running the application, it is necessary to do environment variable settings first.

Access “ View Advance System Settings ” “ Environment Variable ” then edit the path to add Wget in it.

Wget Configuration on Windows.

The steps above will make it easier for you when you want to use Wget. After following step 4, you can already use Wget without having to enter into the program file folder first.

Wget testing after PATH settings.

At this point, you can complete the Wget installation on Linux or Windows. The next review is how to use the Wget command and its various functions.

Example of Using Wget Command

Wget is not only used to download files, but also folders and can also be used to find broken links. The examples of using Wget command below are just a few examples, besides this there are still many Wget functions that you can still learn.

Here are some examples of using Wget command:

Saving Files with Different Names You can change the default file name of the downloaded file. In the case below, the original file is named ‘ source.zip ’. After downloading and storing it on the local computer, the saved file name is ‘ saved_name.zip ’. All this is because there is an “-O” option when downloading.

$ wget -O saved_name.zip https://example.com/source.zip
Saving Files in a Special Directory Besides changing the file name, you can also point to the folder where the file will be saved. In the example below, the source.zip file is directed to be stored in the data folder in the document folder . The direction of this storage directory uses the “-P” option.

$ wget -P document / data / https://example.com/source.zip
Downloading a Single File Downloading files that are usually done is downloading only one file. You don’t need to add an option, just type the Wget command followed by the URL address where the file is as below.

$ wget https://example.com/source.zip
Downloading Multiple Direct Files Besides downloading one file, you can also download several files directly. First , create a file with any name (I use example.txt) . Second , in the file enter a list of any URLs that will be downloaded. For example:

https://example.co.id/source.zip
https://example.com/source.zip
https://sampel.com/yourphoto.png

Then to execute the download process, use the option “-i” so that Wget downloads all the files above.

$ wget -i example.txt
Looking for Broken Links Besides being used to download files and the like, Wget can also be used to check web pages that are not accessible (damaged). The following commands you can use.

$ wget -o wget-log -r -1 5 –spider http://example.com
Limiting Download Speed Wget uses existing bandwidth to download. Sometimes this will interfere with access to other applications. You can limit the download speed with the “-limit-rate” option followed by the maximum download that can be done.

$ wget -limit-rate = 500k https://wordpress.org/latest.zip

Those are some examples of using Wget commands that you can use. Below there are some tricks to using Wget to make it easier and maximal to use.

Some Wget Tricks

There are several options offered by Wget for the download process. If you want to see some other functions or features from Wget, use the “ man wget ” command in Terminal or “ wget -help ” in Windows. The command will display a complete guide to using Wget.

There are several Wget usage options, such as:

You can do the download process in the background by using the “-b” option when running the program.
You can use the “-t” option followed by a number (x). This option is used to conduct retry experiments up to x times if a problem occurs. This option you can use when the website or domain that is accessed is sometimes problematic.

Here are some other Wget options that you can use:

-V	-Version	Display the version of Wget and exit.
-h	-Help	Showing help.
-b	-Background	Go to the background after starting.
-d	— debug	Show lots of debugging information.
-q	-Quiet	Shut up (no output).
-r	-Recursive	Specify to download recursively.
	-No-check-certificate	Don’t validate the server certificate.

In addition, you can also use Wget to transfer data from one hosting server to another hosting server. Of course it will simplify the process of transferring data.

Conclusion

Through this article, you can already use some examples of using Wget command. This application, not only for downloading files, but also can be used to check the health of the website. In addition to its easy use, this application also uses little disk space.

Installing Wget on Windows is a little complicated compared to Linux. This is because Wget is an application built on Linux architecture. Just run one installation command and you can use it to download files from other devices.

Take Your Knowledge to the Next Level

Find out what an ISO is and how to burn an ISO to DVD on Windows, Mac and Linux, the main burning programs. All the information here!
So, you want to know how to update this Widevine Content Decryption module of Chrome component and how to fix it in Windows 10? Keep reading.
You are listening to some of your favorite songs on Spotify, the music stops and you see the error: “Spotify can’t play this right now”.
As you explore Reddit, you’ll also find videos that you might want to save. Here are some methods to download Reddit videos on phone or PC.
Know How to Troubleshoot YouTube App If It Not Working on Android Mobile or iOS Device. Also checkout the alternatives for YouTube App.
There are several ways to check the health of your hard drive on Windows 10. This article explains the best methods. check out the guide..
Google Authenticator can be helpful, but it’s annoying that Google hasn’t yet created an official desktop app. Let’s explore the ways to do..
One of the most common errors in Google Chrome is the dns_probe_finished_nxdomain or DNS Probe Finished NXDomain. Know how to fix this easily

Hopefully this article is enough to help you find out what Wget is and examples of using Wget command.

Источник

What is wget?

How to Check if wget is Installed?

wget Command Not Found

How to Install wget on Ubuntu/Debian?

How to Install wget on CentOS/Fedora?

How to Install wget on Windows?

Introduction to wget Syntax

Download File from Web

Download File and Save Under Specific Name

Download File to Specific Directory

Set Download Speed

Continue Download After Interruption

Download Multiple Files

Download Web page (Mirror Web page)

Download via FTP

Download in Background

Increase Retry Attempts

Skip Certificate Check

Change User Agent

What is Wget?

How to Install Wget

Check if Wget is installed

Download Wget on Mac

Download Wget on Windows

Wget Basics

Wget Syntax

View WGET commands

14 Wget Commands to Extract Web Pages

Download a single file with Wget

Download a File to a Specific Output Directory

Rename Downloaded File when Retrieving with Wget

Define User Agent in WGET

Wget command to Convert Links on a Page

Mirror a Single Webpage in Wget

Define Number of Retry Attempts in Wget

How to Use Proxies With Wget?

Continue Interrupted Downloads with Wget

Wget VS Curl

Use Wget With Python

Debug Wget Command Not Found

About Wget

Wget FAQs

Conclusion

What is Wget Command?

How to Install Wget

1. Linux

2. Windows

Step 1 — Download the installation file

Step 2 — Perform the installation process

Step 3 — Check the usage of Wget

Step 4 — (Optional) Direct the path to the program

Example of Using Wget Command

Some Wget Tricks

Conclusion

Take Your Knowledge to the Next Level

Вот еще несколько интересных статей: