Home | History | Annotate | Download | only in ftp
History log of /src/usr.bin/ftp/version.h
RevisionDateAuthorComments
 1.99  29-Nov-2024  lukem ftp: exit non-zero if short http transfer when filesize is known

If a http file size is known and the fetch finishes with less bytes
transferred, exit non-zero.

Bump version to 20241129.

PR bin/54713
PR bin/58281
 1.98  16-May-2023  lukem branches: 1.98.2;
ftp 20230516 for openssl 3.0 compat SSL_OP_IGNORE_UNEXPECTED_EOF
 1.97  05-May-2023  lukem add timeout for ssl connect

Implement a timeout for SSL connection setup, using -q QUITTIME,
defaulting to 60 seconds.
SSL_connect(3) (unlike connect(2)) doesn't timeout by default.

Adapt ssl error messages destination: if unexpected error
from local API, use warn()/warnx() to stderr;
if expected error from a network operation (e.g., timeouts),
use fprintf to ttyout (which might be stdout).

Consistently use ftp_poll() instead of select();
ssl.c (using select()) was added 7 years after the
previous uses of select() were converted to poll().

Check EAGAIN as well as existing EINTR error from ftp_poll(),
for portability.
 1.96  25-Feb-2023  mlelstv Add option sslnoverify to control validation of SSL certificates.
Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.
Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.
 1.95  22-Sep-2022  lukem branches: 1.95.2;
update ftp version to 20220911

PR/57003: Handle relative URLs (patch by kim@)
 1.94  26-Aug-2021  lukem update ftp version to 20210826

Version bump for PASV/LPSV address validation
 1.93  03-Jun-2021  lukem use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219
 1.92  06-Jan-2021  lukem branches: 1.92.4;
ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.

ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857
 1.91  18-Jul-2020  lukem ftp: add -? for help. improve synopsis

Add -? to display usage synopsis and help to stdout.
This allows for "ftp -? | less", which is more user friendly.
Errors still show usage to stderr.
Consistency improvements in some usage text.
 1.90  11-Jul-2020  lukem ftp.c: improve signal handler restoration

Only invoke the old signal handler if it's a real signal handler
and not SIG_IGN, SIG_DFL, SIG_HOLD, or SIG_ERR, using new static
function issighandler().
Avoids an intermittent race condition with a null pointer
dereference via (*SIG_DFL)().
Bug class reported by Joyu Liao from Juniper Networks.

Use SIG_ERR instead of NULL as the indicator that a signal handler
hasn't been changed, so that SIG_DFL (equivalent to NULL)
will be restored.
 1.89  08-Jun-2020  lukem ftp: exit if lostpeer invoked by a signal

lostpeer() calls too many async-unsafe functions (both directly
and indirectly) to close and cleanup the remote connections,
so just exit after the cleanup if invoked by a signal.

Reported in private mail by Qi Hou.
May also resolve a crash reported by Thomas Klausner.
 1.88  26-Feb-2020  lukem update ftp version to 20190622
 1.87  12-Sep-2015  wiz branches: 1.87.8; 1.87.16; 1.87.18;
Bump version for SNI support.
 1.86  23-Apr-2015  lukem Add -x xferbufsize to set xferbuf size.

Implement -x xferbufsize set the socket send and receive buffer size,
as per 'xferbuf' in interactive mode.

Patch from Nicholas Mills (via private mail), with minor adjustment by me.
 1.85  31-Oct-2014  lukem Version 20141026

Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.84  05-May-2013  lukem branches: 1.84.6;
Version 20130220: restart fix, SYST response parsing improvement
 1.83  06-Feb-2013  christos merry belated x-mas.
 1.82  05-Jun-2010  lukem branches: 1.82.6; 1.82.8; 1.82.12; 1.82.14; 1.82.16;
In ftpvis(), prevent incomplete escape sequences at end of dst,
and ensure NUL-termination of dst. Also tweak for readibility.
Fix from Uwe Stuehler and Stefan Sperling, via Marc Balmer.
 1.81  05-Mar-2010  lukem ftp 20100305: fix http date parsing
 1.80  15-Nov-2009  lukem Crank version to 20090915:
* rename internal getline() to get_line() to avoid conflict with libc
* avoid NULL deref in verbose output in an error path
* improve man page markup
 1.79  24-Apr-2009  lukem Revert incorrect change made as part of the WARNS=4.
Fixes "ftp host [port]" on the CLI.
Noted by Geoff Wing.
 1.78  12-Apr-2009  lukem Fix numerous WARNS=4 issues (-Wcast-qual -Wsign-compare).
 1.77  13-Aug-2008  lukem branches: 1.77.2; 1.77.4; 1.77.6; 1.77.12;
Use AF_INET instead of AF_UNSPEC as the default family if !defined(INET6).
Avoids problem reported by Matthias Scheler <tron@NetBSD.org>.
 1.76  10-May-2008  skd Bump version string, as per lukem@.
 1.75  05-May-2008  lukem branches: 1.75.2;
Only attempt to el_parse() a command unknown by the default parser
if editing is enabled.
Fixes PR bin/38589
 1.74  28-Apr-2008  martin Remove clause 3 and 4 from TNF licenses
 1.73  22-Apr-2008  lukem Use the service name to getaddrinfo() (along with the host name), so that
features such as DNS Service Discovery have a better chance of working.
Suggested by David Young <dyoung>.

Display the service name in various status & error messages.

Don't getservbyname() the :port component of a URL; RFC 3986 says it's
just an unsigned number, not a service name.
 1.72  21-Jan-2008  lukem branches: 1.72.4;
Turn off the alarmtimer before resetting the SIGALRM handler back to SIG_DFL.
Fix from dieter roelants <dieter.NetBSD@pandora.be>
Should fix PR 35630.
 1.71  05-Dec-2007  lukem Rename HAVE_STRUCT_SOCKADDR_SA_LEN to HAVE_STRUCT_SOCKADDR_IN_SIN_LEN
to accurately reflect the structure member being used.
 1.70  22-Aug-2007  lukem branches: 1.70.2;
Improve parsing of chunked transfer chunks per RFC2616:
* more stringent chunk-size parsing
* ignore optional trailing ';chunk-ext' stuff, instead of barfing
* detect EOF before final \r\n.
 1.69  06-Aug-2007  lukem branches: 1.69.2;
Update version for:
Replace HAVE_SOCKADDR_SA_LEN with defined(HAVE_STRUCT_SOCKADDR_SA_LEN
 1.68  05-Jun-2007  lukem Enforce restriction that (http) proxied URL fetchs don't support
being restarted at this time.
PR #28697.
 1.67  24-May-2007  lukem Display times in RFC2822 form rather than using ctime(3), since
the former is more explicit about the timezone offset.
 1.66  22-May-2007  lukem * main: call tzset() to ensure TZ is setup for other <time.h> functions.
* remotemodtime(): use strptime() to parse the reply.
* fetch_url(): ensure struct tm is zeroed before calling strptime().
 1.65  15-May-2007  lukem * Modify parse_url() to consistently strip the leading `/' off ftp URLs.
Fixes PR 17617.
* Use 'RFCnnnn' (with leading 0) instead of 'RFC nnnn', to be
consistent with the style in the RFC index.
* Refer to RFC3916 instead of 1738 or 2732.
* Expand the list of supported RFCs in ftp(1) to contain the document
name as well.
 1.64  10-May-2007  lukem Implement copy_bytes() to copy bytes from one fd to another via the
provided buffer, with optional rate-limiting and hash-mark printing,
using one loop and handle short writes.
Refactor sendrequest() and recvrequest() to use copy_data().
Addresses PR 15943.
 1.63  18-Apr-2007  lukem Clean up use of confirm() and mbort() so that the current operation
is always passed in (instead of depending upon the 'mname' global).
For confirm(), if the second argument is NULL print the "Continue with <cmd>"
prompt. This fixes up the the display of interrupted prompts.
 1.62  17-Apr-2007  lukem * Implement -s srcaddr; uses srcaddr as the local IP address for all
connections.
Based on code in the version of ftp that FreeBSD had before they
replaced it with lukemftp.
* Move error message handling into ftp_connect() rather than in the
caller, so that more specific error reporting can occur.
* Improve consistency of various warning and error messages.
 1.61  12-Apr-2007  lukem Various fixes from Nicholas Marriott sent to openbsd-tech and in private email:
* Prevent segfault if a .netrc token is too long.
* Prevent segfault when using a macro from an empty macdef.
* Check more return values.
 1.60  11-Apr-2007  lukem Suppress printing non-COMPLETE reply strings from EPSV and EPRT, as we're
going to fall back to PASV / PORT (respectively) if the former fail,
and this avoids printing a failure reply followed by a success reply.
Should fix a problem with the emacs ftp wrapper.
 1.59  11-Apr-2007  lukem getpass() can return NULL upon error in some implementations
(as documented in older standards documents, before the API was obsoleted).
Problem observed in tnftp on Solaris by Emil Mikulic.
 1.58  26-Jul-2006  lukem If a file upload (via -u) fails, return an non-zero exit value based on the
index of the file that caused the problem (a la auto-fetch retrieval).
Problem noted by A P Garcia in private email.
 1.57  21-Aug-2005  lukem Don't clear the trailing character on the auth_url() username;
we now use getline() and that newline strips for us.
Problem found & fixed by Mark Davies.
 1.56  19-Jul-2005  lukem Revert back to using an int (instead of size_t) for holding a value that may
be negative.
Fixes progressbar display on terminals <43 columns wide.
Bug noted and solution suggested by Gavan Fantom.
 1.55  10-Jun-2005  lukem Implement:
int getline(FILE *stream, char *buf, size_t buflen, const char **errormsg)
Read a line from the FILE stream into buf/buflen using fgets(), so up
to buflen-1 chars will be read and the result will be NUL terminated.
If the line has a trailing newline it will be removed.
If the line is too long, excess characters will be read until
newline/EOF/error.
Various -ve return values indicate different errors, and errormsg
will be changed to an error description if it's not NULL.

Convert to use getline() instead of fgets() whenever reading user input
to ensure that an overly long input line doesn't leave excess characters
for the next input operation to accidentally use as input.

Zero out the password & account after we've finished with it.

Consistently use getpass(3) (i.e, character echo suppressed) when
reading the account data. For some reason, historically the "login"
code suppressed echo for Account: yet the "user" command did not!

Display the hostname in the "getaddrinfo failed" warning.

Appease some -Wcast-qual warnings. Fixing all of these requires
significant code refactoring. (mmm, legacy code).
 1.54  01-Jun-2005  lukem * Only print the "Trying <address>..." message if verbose and
there's more than one struct addrinfo in the getaddrinfo() result.
* Don't use non-standard "u_int".
 1.53  29-May-2005  lukem * fetch_ftp(): preserve 'anonftp' across a disconnect() so that multiple
ftp auto-fetches on the same command line login automatically.
* auto_fetch(): use an initialized volatile int to appease IRIX cc.
 1.52  27-May-2005  lukem formatbuf(): fix %m and %M to use the hostname, not the username.
 1.51  26-May-2005  lukem Improve method used in fileindir() to determine if `file' is in or under `dir':
realpath(3) on non-NetBSD systems may fail if the target filename doesn't
exist, so instead use realpath(3) on the parent directory of `file'.
Per discussion with Todd Eigenschink.
 1.50  14-May-2005  lukem Fix some cast issues highlighted by Scott Reynolds using gcc 4 on OSX.4
 1.49  13-May-2005  lukem * Correct the "optlen" argument passed to getsockopt(3) and setsockopt(3)
in various places. Fixes a problem noted by Allen Briggs.
* Improve warning printed when connect(2) for the data channel fails.
 1.48  11-May-2005  lukem Use socklen_t instead of int as the 5th argument to getsockopt().
Improve invocation of setsockopt() and associated failure messages.
 1.47  10-May-2005  lukem Prevent an overly-long input line causing a core dump when editing is enabled.
Issue noted by Ryoji Kanai in FreeBSD Problem Report # 77158.
 1.46  11-Apr-2005  lukem Implement a timeout on the accept(2) in dataconn() and the
connect(2) in xconnect() by temporarily setting O_NONBLOCK
on the socket and using xpoll() to wait for the operation
to succeed.
The timeout used is the '-q quittime' argument (defaults to
60s for accept(2), and the system default for connect(2)).
Idea inspired by discussion with Chuck Cranor.
This may (indirectly) fix various problems with timeouts
in active mode through broken firewalls.

Implement xpoll() as a wrapper around poll(2), to make it
easier to replace on systems without a functional poll(2).
Unconditionally use xpoll() instead of conditionally using
select(2) or poll(2).
 1.45  10-Apr-2005  lukem In fetch_url(), don't call freeaddrinfo(res0) too early, as we use pointers
to its contents later in the function.
Problem found by Onno van der Linden.
 1.44  12-Jan-2005  lukem branches: 1.44.2;
Fix ftp url reget when globs are being used.
Provided by Mathieu Arnold <mat@FreeBSD.org>.
 1.43  03-Jan-2005  lukem Forbid filenames returned from mget that aren't in (or below) the
current directory.
The previous behaviour (of trusting the remote server's response when
retrieving the list of files to mget with prompting disabled) has been
in ftp ~forever, and has been a "known issue" for a long time.
Recently an advisory was published by D.J. Bernstein on behalf of
Yosef Klein warning of the problems with the previous behaviour, so
to alleviate concern I've fixed this with a sledgehammer.

Remember the local cwd after any operation which may change it.
Use "remotecwd" instead of "remotepwd".
 1.42  10-Dec-2004  lukem * Always decode %xx in a url's user & pass components.
* Only remember {WWW,Proxy}-Authenticate "Basic" challenges; no point
in tracking any others since ftp doesn't support them.
* Improve the parsing of HTTP responses.
 1.41  08-Aug-2004  lukem Don't base64 encode the trailing NUL in the HTTP basic auth response.
Problem noted by Eric Haszlakiewicz.
 1.40  21-Jul-2004  lukem Slightly rework SIGINT handling; if we're exiting the auto-fetch stuff
and sigint_raised is non-zero, reset the handler for SIGINT to SIG_DFL
and raise(SIGINT) so that the appropriate wait(3) status is setup.
Based on solution proposed by Ognyan Kulev.
This should really fix PR [pkg/26351].
 1.39  20-Jul-2004  lukem Improve parsing of HTTP response headers to be more RFC2616 compliant, and
skip LWS (linear white space; CR, LF, space, tab) and the end of lines and
between the field name and the field value. This still isn't 100% compliant,
since we don't support "multi line" responses at this time.
This should fix PR [bin/22611] from TAMURA Kent (although I can't easily
find a http server to reproduce the problem against.)

Fix a minor memory leak when parsing HTTP response headers.
 1.38  20-Jul-2004  lukem Ensure that "mname" is set in ls() and mls() so that an aborted confirm()
prints the correct name.
Problem highlighted & suggested fix from PR [bin/17766] by Steve McClellan.
 1.37  20-Jul-2004  lukem If an ftp auto-fetch transfer is interrupted by SIGINT (usually ^C),
exit with 130 instead of 1 (or rarely, 0).
This allows an ftp auto-fetch in a shell loop to correctly terminate the loop.
Should fix PR [pkg/26351], and possibly others.
 1.36  15-Jul-2004  lukem Fix bug in Christos' recent cleanup which broke the "help" and "rhelp" commands.
 1.35  10-Apr-2004  lukem If connect(2) in xconnect() fails with EINTR, call select(2) on the socket
until it's writable or it fails with something other than EINTR.
This matches the behaviour in SUSv3, and prevents the problem when
pressing ^T (SIGINFO, which is marked as restartable) during connection
setup would cause ftp to fail with EADDRINUSE or EALREADY when the
second connect(2) was attempted on the same socket.
Problem found and solution provided by Maxime Henrion <mux@freebsd.org>.
 1.34  10-Dec-2003  lukem Don't warn about "ignored setsockopt" failures unless debugging is
enabled. Suggested by Todd Vierling.

Allow empty passwords in ftp://user:@host/file auto-fetch URLs,
per RFC 1738. Requested by Simon Poole.

Update version.
 1.33  31-Jul-2003  lukem crank version for:
* Work around broken ftp servers (notably ProFTPd) that can't even follow
RFC 2389, and skip any amount of whitespace before a FEATure response.
The RFC says 'single space' yet ProFTPd puts two.
Noted by DervishD <raul@pleyades.net>.
* Improve formatting of features[] debug dump.
* Invalidate remote directory completion cache if any command which
may change the remote contents completes successfully, including:
del, mdel, ren, mkdir, rmdir, quote, and all upload commands
Patch from Yar Tikhiy <yar@freebsd.org>.
 1.32  31-Jul-2003  lukem * $FTPUSERAGENT overrides the HTTP User-Agent header.
Based on patch from Douwe Kiela.
* Add about:tnftp
* Fix URL in about:netbsd
* Crank version
 1.31  15-Jun-2003  lukem Don't coredump when printing '%n' in the prompt if there's no username yet.
Fix from Maxim Konovalov <maxim at freebsd dot org>.
 1.30  28-Feb-2003  lukem crank version
 1.29  05-Jun-2002  lukem - when showing the final progress bar, replace "00:00 ETA" with the
elapsed time. (suggested by simonb)
- actually display transfer stats after a URL fetch. (bug introduced a
*long* time ago)
- update copyright & version
 1.28  06-May-2002  lukem - Only send port number in HTTP/1.1 Host: request if port != 80.
Fixes [bin/15415] from Takahiro Kambe <taca@sky.yamashina.kyoto.jp>
- Fix bad mode passed by mls() to recvrequest().
Fixes [bin/16642] from <steve.mcclellan@radisys.com>
 1.27  26-Dec-2001  lukem update copyrights
 1.26  23-Dec-2001  lukem Add -4 to force IPv4 and -6 to force IPv6 address usage.
From Hajimu UMEMOTO, via Mike Heffner of FreeBSD.

(FreeBSD has imported NetBSD's ftp as their ftp client;
Mike is sending back some of their local changes).
 1.25  20-Dec-2001  lukem Large file ASCII mode support by using fseeko() instead of fseek().
From Andrey A. Chernov of FreeBSD, via Mike Heffner.
 1.24  15-Dec-2000  lukem invoke cmdtab.c_handler()s with argv[0] == c_name instead of the
supplied name. that way the full (unambiguous) name is displayed in
error messages and usage strings.
 1.23  24-Nov-2000  itojun cope with 2553bis getnameinfo (always attach scope id)
getnameinfo error check.
 1.22  15-Nov-2000  lukem - implement "mreget"; as per "mget" but uses "reget" instead of "get"
- add -N netrc and $NETRC, as methods to select an alternative .netrc file
- cache local user name and home directory for further use
- in mget(), use docase() instead of a local version to do the case
conversion.
 1.21  28-Sep-2000  lukem explicitly use SOCK_STREAM with socket() instead of res->ai_socktype,
because it appears that linux with glibc doesn't set the latter
correctly after one of getaddrinfo() or getnameinfo().
 1.20  06-Aug-2000  lukem * implement parseport(), which takes a string and attempts to convert
it to a numeric port number
* use parseport() in parse_url() and hookup()
* don't try and lookup the port number using getaddrinfo(), as it's too hard
to separate a failed host name lookup from a failed service name lookup.
this was causing lossage on systems that don't have `http' in services(5)
(such as solaris), but only crept in when we started using getaddrinfo()
unconditionally.
 1.19  01-Aug-2000  lukem - rename NO_QUAD to NO_LONG_LONG, QUAD* -> LL* and add ULL* (unsigned)
equivalents. name change suggested by Klaus Klein <kjk@netbsd.org>
- change defined(BSD4_4) || HAVE_SIN_LEN tests into HAVE_SOCKADDR_SA_LEN,
and set the latter if BSD4_4 exists
 1.18  31-Jul-2000  lukem - we can't just rename BSD4_4 -> HAVE_SIN_LEN, since bsd systems define BSD4_4;
change tests to test for either defined(BSD4_4) or HAVE_SIN_LEN
- more KNF
 1.17  30-Jul-2000  lukem clean up NO_QUAD support: create helper #defines and use as appropriate:
#define NOQUAD ! NOQUAD
------- ------ - ------
QUADF "%ld" "%lld"
QUADFP(x) "%" x "ld" "%" x "lld"
QUADT long long long
STRTOL(x,y,z) strtol(x,y,z) strtoll(x,y,z)
 1.16  28-Jul-2000  lukem rename "opts" to "remopts", so people used to "o host" don't get bitten...
 1.15  18-Jul-2000  lukem add support for FEAT and OPTS commands with `features' and `opts'.
(from RFC 2389).

add support for MLST & MLSD (machine parseble listings) with 'mlst', 'mlsd'
and 'pmlsd' (mlsd |$PAGER) commands. (from draft-ietf-ftpext-mlst-11)

rename remotesyst() to getremoteinfo(), and modify to parse the result from
FEAT (if supported), and take into account the support for the various
extensions such as MDTM, SIZE, REST (STREAM), MLSD, and FEAT/OPTS.
put each feature into one of the following categories:
- known to work (explicit FEAT)
- unknown but assume works until explicit failure, when it's
then tagged as `known not to work'.
- known not to work (FEAT succeeded but didn't return anything,
or was unknown and then explicit failure)
assign results into features[] matrix.

add support to getreply() so that an optional callback will be called
for each line received from the server except for the first and last.
this is used in FEAT (and MLST) parsing.

modify various commands to check if REST (STREAM), MDTM and SIZE are
explicitly or implicitly supported before using.

fix `syst' when verbose is off.

minor knf (indent goto labels by one space, etc).

simply various command usage handlers by assuming that argv != NULL except
for quit() and disconnect().
 1.14  15-Jun-2000  lukem * migrate the SYST parsing from setpeer() into a separate remotesyst().
call remotesyst() only when login has been successful
some servers don't let you run SYST until you've successfully logged in.
* in fetch_ftp(), always call setpeer() with autologin disabled, and use
the following ftp_login() to DTRT.
this prevents ftp from trying to login a second time if the first autologin
fails when connecting to a remote site anonymously using autofetch.
* reset unix_proxy and unix_server in cleanuppeer()
* missed a function conversion in the KNF sweep...
 1.13  11-Jun-2000  lukem from itojun: better fix for previous (doesn't need in_addr_t or u_int32_t)
 1.12  11-Jun-2000  lukem portability fixes for lukemftp:
* initconn(): use in_addr_t instead of u_int32_t when manipulating IPv6
addresses (and assume anything with ipv6 has in_addr_t; if not, i'll
add an autoconf test for it)
* ai_unmapped(): not all systems have sin_len; so only set #ifdef BSD4_4
* fix some lint
 1.11  05-Jun-2000  lukem - fix ai_unmapped() to be a no-op in the !def INET6 case
- display `(-INET6)' at the end of the version string if !def INET6
- clarify in the man page that IPv6 support may not be present (for lukemftp :)
 1.10  31-May-2000  lukem Add support for 'ftp -u url file ...', to upload a list of files to given url.
Mostly based on [bin/10019] by Scott Aaron Bamford <sab@ansic.net>
 1.9  28-May-2000  lukem Change `ls' to use the `LIST' and not `NLST' FTP protocol command.
Now that after many years on not caring we find certain popular
ftp servers are starting to obey RFC959 to the letter of the law
and will only return a list of filenames (not directories or
other filetypes) in the output of `NLST', then `LIST' is more useful
in this case. (Note that the aforementioned pedanticness means that
filename completion isn't as useful as it could be...)
Fixes [bin/8937] by David A. Gatwood <dgatwood@deepspace.mklinux.org>
 1.8  01-May-2000  lukem branches: 1.8.2;
* Add support for `fget localfile', which reads a list of filenames to
retrieve from localfile. Based on work by Darren Reed.
* Crank version.
* Update copyright dates.
 1.7  13-Apr-2000  lukem crank version
 1.6  13-Apr-2000  lukem s/strtoq/strtoll/ (the latter is standardised)
 1.5  31-Jan-2000  lukem crank
 1.4  26-Jan-2000  lukem crank version
 1.3  25-Jan-2000  lukem crank version
 1.2  12-Dec-1999  lukem branches: 1.2.2;
crank version
 1.1  05-Dec-1999  lukem move version into separate file to reduce recompilation after version crank.
 1.2.2.1  27-Dec-1999  wrstuden Pull up to last week's -current.
 1.8.2.1  23-Jun-2000  minoura Sync w/ netbsd-1-5-base.
 1.44.2.13  28-Aug-2005  tron Pull up following revision(s) (requested by lukem in ticket #715):
usr.bin/ftp/version.h: revision 1.57
usr.bin/ftp/fetch.c: revision 1.164
Don't clear the trailing character on the auth_url() username;
we now use getline() and that newline strips for us.
Problem found & fixed by Mark Davies.
 1.44.2.12  24-Jul-2005  tron Pull up revision 1.56 (requested by lukem in ticket #606):
Revert back to using an int (instead of size_t) for holding a value that may
be negative.
Fixes progressbar display on terminals <43 columns wide.
Bug noted and solution suggested by Gavan Fantom.
 1.44.2.11  24-Jul-2005  tron Pull up revision 1.55 (requested by lukem in ticket #606):
Implement:
int getline(FILE *stream, char *buf, size_t buflen, const char **errormsg)
Read a line from the FILE stream into buf/buflen using fgets(), so up
to buflen-1 chars will be read and the result will be NUL terminated.
If the line has a trailing newline it will be removed.
If the line is too long, excess characters will be read until
newline/EOF/error.
Various -ve return values indicate different errors, and errormsg
will be changed to an error description if it's not NULL.
Convert to use getline() instead of fgets() whenever reading user input
to ensure that an overly long input line doesn't leave excess characters
for the next input operation to accidentally use as input.
Zero out the password & account after we've finished with it.
Consistently use getpass(3) (i.e, character echo suppressed) when
reading the account data. For some reason, historically the "login"
code suppressed echo for Account: yet the "user" command did not!
Display the hostname in the "getaddrinfo failed" warning.
Appease some -Wcast-qual warnings. Fixing all of these requires
significant code refactoring. (mmm, legacy code).
 1.44.2.10  24-Jul-2005  tron Pull up revision 1.54 (requested by lukem in ticket #606):
* Only print the "Trying <address>..." message if verbose and
there's more than one struct addrinfo in the getaddrinfo() result.
* Don't use non-standard "u_int".
 1.44.2.9  24-Jul-2005  tron Pull up revision 1.53 (requested by lukem in ticket #606):
* fetch_ftp(): preserve 'anonftp' across a disconnect() so that multiple
ftp auto-fetches on the same command line login automatically.
* auto_fetch(): use an initialized volatile int to appease IRIX cc.
 1.44.2.8  24-Jul-2005  tron Pull up revision 1.52 (requested by lukem in ticket #606):
formatbuf(): fix %m and %M to use the hostname, not the username.
 1.44.2.7  24-Jul-2005  tron Pull up revision 1.51 (requested by lukem in ticket #606):
Improve method used in fileindir() to determine if `file' is in or under `dir':
realpath(3) on non-NetBSD systems may fail if the target filename doesn't
exist, so instead use realpath(3) on the parent directory of `file'.
Per discussion with Todd Eigenschink.
 1.44.2.6  24-Jul-2005  tron Pull up revision 1.50 (requested by lukem in ticket #606):
Fix some cast issues highlighted by Scott Reynolds using gcc 4 on OSX.4
 1.44.2.5  18-May-2005  snj Pull up revision 1.49 (requested by lukem in ticket #301):
* Correct the "optlen" argument passed to getsockopt(3) and setsockopt(3)
in various places. Fixes a problem noted by Allen Briggs.
* Improve warning printed when connect(2) for the data channel fails.
 1.44.2.4  18-May-2005  snj Pull up revision 1.48 (requested by lukem in ticket #318):
Use socklen_t instead of int as the 5th argument to getsockopt().
Improve invocation of setsockopt() and associated failure messages.
 1.44.2.3  11-May-2005  tron Pull up revision 1.47 (requested by lukem in ticket #289):
Prevent an overly-long input line causing a core dump when editing is enabled.
Issue noted by Ryoji Kanai in FreeBSD Problem Report # 77158.
 1.44.2.2  09-May-2005  tron Pull up revision 1.46 (requested by lukem in ticket #265):
Implement a timeout on the accept(2) in dataconn() and the
connect(2) in xconnect() by temporarily setting O_NONBLOCK
on the socket and using xpoll() to wait for the operation
to succeed.
The timeout used is the '-q quittime' argument (defaults to
60s for accept(2), and the system default for connect(2)).
Idea inspired by discussion with Chuck Cranor.
This may (indirectly) fix various problems with timeouts
in active mode through broken firewalls.
Implement xpoll() as a wrapper around poll(2), to make it
easier to replace on systems without a functional poll(2).
Unconditionally use xpoll() instead of conditionally using
select(2) or poll(2).
 1.44.2.1  09-May-2005  tron Pull up revision 1.45 (requested by lukem in ticket #264):
In fetch_url(), don't call freeaddrinfo(res0) too early, as we use pointers
to its contents later in the function.
Problem found by Onno van der Linden.
 1.69.2.2  06-Aug-2007  lukem Update version for:
Replace HAVE_SOCKADDR_SA_LEN with defined(HAVE_STRUCT_SOCKADDR_SA_LEN
 1.69.2.1  06-Aug-2007  lukem file version.h was added on branch matt-mips64 on 2007-08-06 03:41:15 +0000
 1.70.2.2  23-Mar-2008  matt sync with HEAD
 1.70.2.1  09-Jan-2008  matt sync with HEAD
 1.72.4.1  18-May-2008  yamt sync with head.
 1.75.2.2  18-Sep-2008  wrstuden Sync with wrstuden-revivesa-base-2.
 1.75.2.1  23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.77.12.1  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1929):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.77.6.1  13-May-2009  jym Sync with HEAD.

Third (and last) commit. See http://mail-index.netbsd.org/source-changes/2009/05/13/msg221222.html
 1.77.4.1  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1929):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.77.2.1  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1929):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.82.16.1  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1180):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.82.14.1  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1180):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.82.12.2  23-Jun-2013  tls resync from head
 1.82.12.1  25-Feb-2013  tls resync with head
 1.82.8.3  27-Aug-2016  bouyer Apply patch, requested by nonaka in ticket #1375:
src/usr.bin/ftp/cmds.c: patch
src/usr.bin/ftp/fetch.c: patch
src/usr.bin/ftp/ftp.1: patch
src/usr.bin/ftp/ftp.c: patch
src/usr.bin/ftp/ftp_var.h: patch
src/usr.bin/ftp/main.c: patch
src/usr.bin/ftp/ssl.c: patch
src/usr.bin/ftp/ssl.h: patch
src/usr.bin/ftp/version.h: patch
Update ftp(1) to version 20150912, adding https via proxy support.
 1.82.8.2  03-Nov-2014  msaitoh Pull up following revision(s) (requested by lukem in ticket #1180):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.82.8.1  17-Dec-2013  bouyer Apply patch, requested by tron in ticket #997:
usr.bin/ftp/Makefile patch
usr.bin/ftp/cmds.c patch
usr.bin/ftp/cmdtab.c patch
usr.bin/ftp/extern.h patch
usr.bin/ftp/fetch.c patch
usr.bin/ftp/ftp.1 patch
usr.bin/ftp/ftp.c patch
usr.bin/ftp/ftp_var.h patch
usr.bin/ftp/main.c patch
usr.bin/ftp/progressbar.c patch
usr.bin/ftp/ssl.c patch
usr.bin/ftp/ssl.h patch
usr.bin/ftp/util.c patch
usr.bin/ftp/version.h patch

Add HTTPS support to ftp(1).
 1.82.6.1  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.84.6.2  05-Nov-2015  riz Pull up following revision(s) (requested by wiz in ticket #981):
usr.bin/ftp/ftp.1: revision 1.135
usr.bin/ftp/ssl.c: revision 1.3
usr.bin/ftp/ssl.c: revision 1.4
usr.bin/ftp/ssl.h: revision 1.3
usr.bin/ftp/version.h: revision 1.86
usr.bin/ftp/version.h: revision 1.87
usr.bin/ftp/fetch.c: revision 1.207
usr.bin/ftp/main.c: revision 1.123
Add -x xferbufsize to set xferbuf size.
Implement -x xferbufsize set the socket send and receive buffer size,
as per 'xferbuf' in interactive mode.
Patch from Nicholas Mills (via private mail), with minor adjustment by me.
Add Server Name Indication (SNI) support for https.
Needed for e.g. some github URLs.
Bump version for SNI support.
servername cannot be NULL here.
Noted by joerg@.
 1.84.6.1  31-Oct-2014  martin Pull up following revision(s) (requested by lukem in ticket #172):
usr.bin/ftp/version.h: revision 1.85
Version 20141026
Ignore special characters unless they're from the command line.
Fixes CVE-2014-8517
 1.87.18.6  12-Sep-2022  martin Catch up to current, requested by christos in ticket #1523

usr.bin/ftp/Makefile up to 1.39
usr.bin/ftp/fetch.c up to 1.235
usr.bin/ftp/ftp.1 up to 1.147
usr.bin/ftp/ftp_var.h up to 1.86
usr.bin/ftp/main.c up to 1.128
usr.bin/ftp/ssl.c up to 1.12
usr.bin/ftp/util.c up to 1.164
usr.bin/ftp/version.h up to 1.94

PR 57003: Support relative redirects.
 1.87.18.5  12-Sep-2022  martin Back out ticket #1523 for now - trust anchor validation is not solved.
 1.87.18.4  12-Sep-2022  martin Catch up to current, requested by christos in ticket #1523

usr.bin/ftp/Makefile up to 1.39
usr.bin/ftp/fetch.c up to 1.235
usr.bin/ftp/ftp.1 up to 1.147
usr.bin/ftp/ftp_var.h up to 1.86
usr.bin/ftp/main.c up to 1.128
usr.bin/ftp/ssl.c up to 1.11
usr.bin/ftp/util.c up to 1.164
usr.bin/ftp/version.h up to 1.94

PR 57003: Support relative redirects.
 1.87.18.3  14-Jun-2021  martin Pull up following revision(s) (requested by lukem in ticket #1295):

usr.bin/ftp/ssl.c: revision 1.10
usr.bin/ftp/ssl.h: revision 1.5
usr.bin/ftp/version.h: revision 1.93
usr.bin/ftp/Makefile: revision 1.39

use fetch_*() for I/O with SMALLPROG / !WITH_SSL builds

Adapt the SMALLPROG / -UWITH_SSL build to also use the fetch_*()
methods from ssl.c, instead of using stdio, as stdio isn't robust
when using interruptable signals.

Disable ssl-specific support in the fetch_*() methods if WITH_SSL
isn't defined, so SMALLPROG still doesn't have ssl support (as expected).

The resulting SMALLPROG binary is slightly larger than before
(e.g., 157KiB vs 153KiB on amd64).

Set version to 20210603 for this fix and the SO_KEEPALIVE fix for PR 56129.

PR install/56219
 1.87.18.2  14-Jun-2021  martin Pull up following revision(s) (requested by lukem in ticket #1290):

usr.bin/ftp/version.h: revision 1.90
usr.bin/ftp/ftp.c: revision 1.170

ftp.c: improve signal handler restoration

Only invoke the old signal handler if it's a real signal handler
and not SIG_IGN, SIG_DFL, SIG_HOLD, or SIG_ERR, using new static
function issighandler().

Avoids an intermittent race condition with a null pointer
dereference via (*SIG_DFL)().

Bug class reported by Joyu Liao from Juniper Networks.

Use SIG_ERR instead of NULL as the indicator that a signal handler
hasn't been changed, so that SIG_DFL (equivalent to NULL)
will be restored.
 1.87.18.1  29-Jan-2021  martin Pull up following revision(s) (requested by lukem in ticket #1190):

usr.bin/ftp/progressbar.c: revision 1.24
usr.bin/ftp/ssl.c: revision 1.9
usr.bin/ftp/progressbar.h: revision 1.9
usr.bin/ftp/ftp.c: revision 1.171
usr.bin/ftp/version.h: revision 1.92

ftp: don't use restartable signals

Refactor to not rely upon restartable signals (SA_RESTART),
possibly fixing intermittent failures with -q QUITTIME.
ftp transfers: handle EINTR/EAGAIN in copy_bytes(),
instead of relying upon restartable signals.

http/https transfers: Explicitly print an error similar to
progressmeter() when timing-out for -Q QUITTIME in fetch_wait(),
and set errno to ETIMEDOUT so that the warn() in fetch_url()
prints a more accurate error message.

PR/55857
 1.87.16.1  08-Apr-2020  martin Merge changes from current as of 20200406
 1.87.8.3  12-Sep-2022  martin Catch up to current, requested by christos in ticket #1763:

usr.bin/ftp/Makefile up to 1.39
usr.bin/ftp/cmds.c up to 1.141
usr.bin/ftp/complete.c up to 1.47
usr.bin/ftp/domacro.c up to 1.23
usr.bin/ftp/extern.h up to 1.82
usr.bin/ftp/fetch.c up to 1.235
usr.bin/ftp/ftp.1 up to 1.147
usr.bin/ftp/ftp.c up to 1.174
usr.bin/ftp/ftp_var.h up to 1.86
usr.bin/ftp/main.c up to 1.128
usr.bin/ftp/progressbar.c up to 1.24
usr.bin/ftp/progressbar.h up to 1.9
usr.bin/ftp/ssl.c up to 1.12
usr.bin/ftp/ssl.h up to 1.5
usr.bin/ftp/util.c up to 1.164
usr.bin/ftp/version.h up to 1.94

ftp(1): validate address from PASV and LPSV response.
ftp(1): use raw write(2) instead of fwrite(3) to avoid stream
corruption because of the progress bar interrupts.
Fixes for PR 56219 and PR 55857.
PR 57003: Support relative redirects.
 1.87.8.2  12-Sep-2022  martin Backout ticket #1763 for now - trust anchors are not solved.
 1.87.8.1  12-Sep-2022  martin Catch up to current, requested by christos in ticket #1763:

usr.bin/ftp/Makefile up to 1.39
usr.bin/ftp/cmds.c up to 1.141
usr.bin/ftp/complete.c up to 1.47
usr.bin/ftp/domacro.c up to 1.23
usr.bin/ftp/extern.h up to 1.82
usr.bin/ftp/fetch.c up to 1.235
usr.bin/ftp/ftp.1 up to 1.147
usr.bin/ftp/ftp.c up to 1.174
usr.bin/ftp/ftp_var.h up to 1.86
usr.bin/ftp/main.c up to 1.128
usr.bin/ftp/progressbar.c up to 1.24
usr.bin/ftp/progressbar.h up to 1.9
usr.bin/ftp/ssl.c up to 1.11
usr.bin/ftp/ssl.h up to 1.5
usr.bin/ftp/util.c up to 1.164
usr.bin/ftp/version.h up to 1.94

ftp(1): validate address from PASV and LPSV response.
ftp(1): use raw write(2) instead of fwrite(3) to avoid stream
corruption because of the progress bar interrupts.
Fixes for PR 56219 and PR 55857.
PR 57003: Support relative redirects.
 1.92.4.1  06-Jun-2021  cjep sync with head
 1.95.2.3  02-Dec-2024  martin Pull up following revision(s) (requested by lukem in ticket #1021):

usr.bin/ftp/ftp.c: revision 1.178
usr.bin/ftp/version.h: revision 1.99
usr.bin/ftp/ruserpass.c: revision 1.35
usr.bin/ftp/main.c: revision 1.134
usr.bin/ftp/main.c: revision 1.135
usr.bin/ftp/progressbar.c: revision 1.27
usr.bin/ftp/util.c: revision 1.165
usr.bin/ftp/cmds.c: revision 1.144
usr.bin/ftp/extern.h: revision 1.84
usr.bin/ftp/fetch.c: revision 1.242
usr.bin/ftp/ftp.1: revision 1.160

s/bninary/binary/ in comment.

extract duplicate code into a function.

Check bounds when copying to destination.

Remove const where the const string ended up being overwritten.

use unsigned when doing shifts.

remove const when string gets overwritten.

ftp: exit non-zero if short http transfer when filesize is known
If a http file size is known and the fetch finishes with less bytes
transferred, exit non-zero.
Bump version to 20241129.
PR bin/54713
PR bin/58281

ftp: help improvements
Document -? as a separate mode.
Document -H HEADER in the usage.
Clarify units for -b and -x.
Consistent argument names between ftp -? and ftp(1).

ftp: order getopt Upper before lower
Consistently order options in getopt and the switch
with the upper case option before the lower case option.
This makes it easier to cross-reference with -? and ftp(1).
No functional change.
 1.95.2.2  16-May-2023  martin Pull up following revision(s) (requested by lukem in ticket #171):

usr.bin/ftp/ssl.c: revision 1.15
usr.bin/ftp/util.c: revision 1.167
usr.bin/ftp/ftp.c: revision 1.175
usr.bin/ftp/version.h: revision 1.97

add timeout for ssl connect

Implement a timeout for SSL connection setup, using -q QUITTIME,
defaulting to 60 seconds.

SSL_connect(3) (unlike connect(2)) doesn't timeout by default.
Adapt ssl error messages destination: if unexpected error
from local API, use warn()/warnx() to stderr;
if expected error from a network operation (e.g., timeouts),
use fprintf to ttyout (which might be stdout).

Consistently use ftp_poll() instead of select();
ssl.c (using select()) was added 7 years after the
previous uses of select() were converted to poll().

Check EAGAIN as well as existing EINTR error from ftp_poll(),
for portability.
 1.95.2.1  16-May-2023  martin Pull up following revision(s) (requested by lukem in ticket #170):

usr.bin/ftp/ssl.c: revision 1.13
usr.bin/ftp/fetch.c: revision 1.236
usr.bin/ftp/util.c: revision 1.166
usr.bin/ftp/main.c: revision 1.129
usr.bin/ftp/extern.h: revision 1.83
usr.bin/ftp/ftp.1: revision 1.148
usr.bin/ftp/cmdtab.c: revision 1.53
usr.bin/ftp/version.h: revision 1.96

Add option sslnoverify to control validation of SSL certificates.

Add netrc processing to fetch-mode (URL on command line) to enable options and autologin
via netrc.

Fix SSL cleanup in some error paths.

Certificate validation is now enabled by default. Set FTPSSLNOVERIFY=1 in environment
or configure a corresponding init macro via netrc to not validate certs (required if
you haven't installed a required CA certificate for OpenSSL).

Discussed with lukem@ on icb.
 1.98.2.1  02-Aug-2025  perseant Sync with HEAD

RSS XML Feed