.\" Process this file with .\" groff -man -Tascii mget.1 .\" .TH MGET 1 "February, 2002" "GNU Linux/Solaris/FreeBSD" "Users's Manual" .SH NAME mget \- Multithreaded commandline web-download manager .SH SYNOPSIS .B mget -f URL .B [-n number_segments] .B [-O Output_file_name] .B [-t timeout_seconds] .B [-R referrer] .B [-p port] .B [-r rollback bytes] .B [-C] .B [-v] .B [-V] .B [-F, --use_ftp_proxy=] .B [-x] .B [-H, --use_http_proxy=] .B [-h] .SH DESCRIPTION .B mGet is a command line download manager. It splits the file into a number of segments and uses several separate threads to download each segment. It can handle proxies and can tunnel through them to get files blocked by proxies. Currently mGet only handles HTTP downloads directly, while FTP downloads must go through a proxy. .SH USAGE file_name denotes the full URL of the file to be downloaded .TP .I mget just run mget to get the usage .TP .I mget -f[file_name] default number of threads=1 .TP .I mget -n10 -f[file_name] -o[out_file_name] .TP .I mget -n10 -H[http_proxy] -f[file_name] .TP .I mget -n10 -F -f[file_name] there is an environment variable ftp_proxy .B getting value from environment variable does not work in SunOS due to a problem in their getopt() implementation .TP .I mget --use_http_proxy=[http_proxy] -f[file_name] -t[timeout value] .TP .I mget -f[file_name] -R[referrer URL] .SH OPTIONS Nearly all the options (except \-f) is optional. If unspecified, the default values for some of the options are given in the curly braces. .TP .B \-n number_of_segments{1} get file in .I number_of_segments parts. (default value is 1 when unspecified) .TP .B \-f URL the complete URL of the file to be downloaded. It can contain the value of the port. .TP .B \-H[http_proxy], \-\-use_http_proxy=[http_proxy] use http proxy. If there is no argument .I http_proxy specified then the value is taken from the envirinment variable http_proxy (if set). The argument can be of form .I proxy_host:proxy_port or if the port is not specified, default value of 80 is assumed. See .B NOTES for Solaris usage. .TP .B \-H[ftp_proxy], \-\-use_ftp_proxy=[ftp_proxy] use ftp proxy. If there is no argument .I ftp_proxy specified then the value is taken from the envirinment variable ftp_proxy (if set). The argument can be of form .I proxy_host:proxy_port or if the port is not specified, default value of 80 is assumed. See .B NOTES for Solaris usage. .TP .B \-p port{80} specify the port for the host from which to download file. This is needed if the URL argument in option -f contains no port value and port value is different from 80. The default value of port is 80. Only use this option if sure, else donot use it. In all general cases it works even without this option. .TP .B \-O output_file_name Generally the name of the file on disk is derived from the URL. Use this option to save the file in the name of .I output_file_name .TP .B \-C Use tunnelling to get the file. See .I draft-luotonen-web-proxy-tunneling-01.txt .TP .B \-t timeout_seconds{60} default value for timeout option (if unspecified) is 60 seconds. .TP .B \-R referrer use .I referrer in the Referrer header for requests. .TP .B \-r rollback specifies the amount of bytes to rollback during resending requests after timeout. Default is zero. .TP .B \-v verbose output. .TP .B \-V version information and usage The proxy options are optional and generally the -p option is not required since most servers use the default port. The maximum number of threads supported are 10 (max. 10 segments can be downloaded at a time.) On specifying the proxy options proxy authentication is asked for and the password is masked and encoded using the standard Base64 encoding (RFC1341). (Unlike wget, which shows the username password as a part of process, mGet kee[s this as internal variables]) .SH NOTES .TP The output is very wide (maximum 10 threads etc.) and so a wide terminal is preferable (rather necessary). About 140 columns wide would be fine ('tput cols' gives the number of columns). If the terminal is not wide enough then a single line may be wrapped by the terminal, and thus the output may not not be as desired. .TP To avoid very small sized files being downloaded in large number of segments, the program chacks the desired numner of parts against the size of the file. About 50 Kb per segment with additional 5 Kb is specified. (:( To change this and make suitable to your network, you need to look in the source code). .TP The Range field in the http request is in "bytes". This is the most common but not necessary, so the program won't work on sites who donot specify Content-Range in bytes. (This can be easily changed.) .TP mGet can download files only from HTTP servers. (It follows HTTP/1.1 ) .TP The HTTP headers are assumed to be within 1024 bytes in length. It is very rare to find longer headers (I haven't seen any). .TP The file is downloaded in the directory from which mGet is run. So any existing file with the same name is overwritten. .TP In case the complete_file_url doesnot specify a file (like www.iitk.ac.in or www.iitk.ac.in/) then the downloaded file is named 'index.html'. However, this practice is not recommended and should be avoided; the URL should point to a total file. .TP In case the URL is redirected by the server to some other URL, currently mGet only reports the redirected URL to the user. .TP The file to be downloaded must be given as complete_url (relative urls cannot be specified). .TP Due to the different implementation of getopt() in Solaris, reading the values of proxy from environment variables is not supported i.e. use -H instead of only -H with value of environment variable http_proxy set. .TP Long options (e.g. --use_http_proxy etc. those starting with '--') are not supported in Solaris. .TP On some machines, '-n 10' works and on some '-n10' works (note the space). .SH TODO .TP Configuration file support, at least username and password for proxy. .TP Other common modes of authentication support like Basic Authentication. .TP Trying to fetch different segments from different sources. .TP Control to individual threads, so that some specific connection can be forced to stop and reconnect. .TP Some log file so that downloads can be stopped and resumed later. .TP In case of redirection, get the redirected URL and download the file from the redirected URL. .TP Can't recall now :-D. .SH SEE ALSO wget .SH COPYRIGHT mget is Copyright (c) 2001-2002 Debajyoti Bera. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA. .SH BUGS There are a number of known bugs (and surely a few unknown :( bugs). Some of the bugs are described in NOTES section. Report bugs to dbera@cse.iitk.ac.in. Mention the version of mget used, the command-line and core (if segfaulted). .SH THANKS Special thanks must go to my friends whose usage of mGet encouraged me to work on it and enhance it features. Of them, I must mention the name of Utkarsh Srivastava, Ankur Gupta who have pointed you bugs and desirable features many a times. Thanks must go to Joseph for taking the initiative of writing the man-page in place of the README file. Also I must thank Sidar and Martynas for providing me with accounts on FreeBSD machines. .SH AUTHORS Debajyoti Bera .VN http://www.cse.iitk.ac.in/users/dbera/mget.html http://www.cse.iitk.ac.in/users/dbera/mget.html .VE