Home | History | Annotate | Download | only in make
History log of /src/usr.bin/make/str.c
RevisionDateAuthorComments
 1.106  28-Jun-2025  rillig make: error out on an ":M" modifier whose pattern ends with backslash

This edge case is not expected to occur intentionally in practice.
 1.105  07-Jul-2024  rillig make: only generate code for cleanup functions in CLEANUP mode
 1.104  06-Jul-2024  rillig make: error out on a matching malformed matching pattern '[['
 1.103  14-Apr-2024  rillig make: make string matching platform-independent

Previously, whether the character range '[a-รค]' matched, depended on the
signedness of the plain 'char' type. Since make operates on byte
strings and does not support UTF-8 or other multi-byte character
encodings, this edge case is not expected to occur in practice.

No change in the unit tests as this edge case is not covered by tests.
 1.102  05-Jan-2024  rillig make: miscellaneous cleanups
 1.101  17-Dec-2023  rillig make: speed up pattern matching in the ':M' modifier

In the common patterns where '*' is followed by a regular character,
such as in the patterns '*.c' or '*.mk', search the next possible
matching position in a small loop, instead of repeatedly comparing the
first remaining pattern character to all special characters.
 1.100  16-Dec-2023  rillig make: eliminate a local variable in Str_Match

No functional change.
 1.99  23-Jun-2023  rillig make: reduce indentation in pattern matching code

No functional change.
 1.98  23-Jun-2023  rillig make: warn about malformed patterns in ':M', ':N' and '.if make(...)'

These patterns shouldn't occur in practice, as their results are tricky
to predict. Generate a warning for now, and maybe an error later.

Reviewed by sjg@.
 1.97  22-Jun-2023  rillig make: unclutter string matching code
 1.96  22-Jun-2023  rillig make: rename variables in string matching, remove redundant code

No functional change.
 1.95  22-Jun-2023  rillig make: speed up pattern matching in the ':M' and ':N' modifiers

In the code coverage report, the highest count for Str_Match goes from
5,298,924 down to 79,646.
 1.94  07-Dec-2022  rillig make: clean up comments
 1.93  11-Jun-2022  rillig make: clean up comments for string splitting and string matching

No binary change.
 1.92  11-Jun-2022  rillig make: condense Str_Match

The test for '\\' followed by '\0' was redundant since at that point,
*str is guaranteed to be not '\0', which takes the next 'return false'.

No functional change.
 1.91  13-May-2022  rillig make: document platform dependency in string pattern matching

No unit test for this edge case since all other unit tests are platform-
independent.

To reproduce:
$ make clean
$ make -s PROG=s-make NOMAN=yes USER_CFLAGS=-fsigned-char
$ make clean
$ make -s PROG=u-make NOMAN=yes USER_CFLAGS=-funsigned-char
$ make clean
$ range=$(lua -e 'print(("[%c-%c]"):format(0xe4, 0x61))')
$ ./s-make -V "\${:UM:M$range}\${:UN:N$range}"
M
$ ./u-make -V "\${:UM:M$range}\${:UN:N$range}"
N
 1.90  13-May-2022  rillig make: clean up low-level comments, eliminate common subexpression

No functional change.
 1.89  03-Mar-2022  rillig make: make code for string matching syntactically more consistent

No functional change.
 1.88  15-Dec-2021  rillig make: in CLEANUP mode, free interned strings at the very end

Noticed by sjg.
 1.87  13-Dec-2021  rillig make: fix memory leak for filenames in .for loops (since 2013-06-18)

Previously, each time a .for directive pushed its buffer on the input
file stack, the current filename was duplicated. This was a waste of
memory.

The name of a file is typically only used while it is read in. There is
one situation when the filename is needed for longer, which is when a
target is defined.

Since .for loops are implemented as a special form of included files,
each .for loop duplicated the current filename as well.

$ cat << EOF > for.mk
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.for i in 1 2 3 4 5 6 7 8 9 0
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor
.endfor

all:
@ps -o rsz -p ${.MAKE.PID}
EOF

$ make-2021.12.13.03.55.16 -r -f for.mk
RSZ
10720

$ ./make -r -f for.mk
RSZ
1716

The difference is 8 MB, which amounts to 1 million .for loops.
 1.86  21-Jun-2021  rillig make: eliminate local variable in Substring_Words

No functional change.
 1.85  30-May-2021  rillig make: inline str_concat4

This function is only ever used for forming strings of the form
"archive(member)".

No functional change.
 1.84  11-Apr-2021  rillig branches: 1.84.2;
make: avoid unnecessary calls to strlen when evaluating modifiers

No functional change.
 1.83  03-Apr-2021  rillig make: backport to C90

In the past few months I had accidentally used C99 features in the make
code. According to tools/README, tools that are used in the build
system should restrict themselves to C90.

This allows make to build with GCC's options "-pedantic
-Wno-system-headers -Dinline= -Wno-error=cast-qual".

I didn't notice anyone actively complaining though, I just wanted to see
how much work this backporting would be. The identifier __func__ is
still used, as in other tools.

No functional change.
 1.82  03-Apr-2021  rillig make: use C99 bool type instead of defining its own

No functional change.
 1.81  01-Feb-2021  rillig make: document necessary tests for Str_Match
 1.80  01-Feb-2021  rillig make: replace pre-increment with post-increment or simple addition

The rest of the code already prefers post-increment if there is no
actual difference.
 1.79  19-Jan-2021  rillig make(1): remove do-not-format markers from comments

These markers had been used inconsistently. Furthermore the source code
had not been formatted automatically before 2020 at all, otherwise there
wouldn't have been any trailing whitespace left.
 1.78  10-Jan-2021  rillig make(1): make a few more bool expressions more precise

The previous version of lint(1) from a few hours ago didn't catch all
occurrences. And even the current one doesn't catch everything.
Function arguments and return types still need some work. The "return
quietly" from shouldDieQuietly still implicitly converts from int to
_Bool.

No functional change.
 1.77  10-Jan-2021  rillig make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set. If Boolean is an ordinary
integer type (the default), some high bits may get lost. But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.
 1.76  30-Dec-2020  rillig make(1): format multi-line comments
 1.75  12-Dec-2020  rillig make(1): remove redundant call to strlen in Str_Words
 1.74  16-Nov-2020  rillig make(1): use postfix increment where possible
 1.73  15-Nov-2020  rillig make(1): clean up make.h, meta.c, parse.c, str.c

The main changes are in the comments, which have been shortened and
corrected.

Some local variables changed their names.

In ParseErrorInternal, the scope of va_start is now narrower.

In ParseDoDependency, the type of tOp has been fixed.

ParseGetLine doesn't take flags anymore but instead a parsing mode.
Previously, the flags had not been combined anyway.

At the beginning of Parse_File, fatals is already guaranteed to be 0, and
even if not, it would be wrong to just discard the fatal errors.
 1.72  07-Nov-2020  rillig make(1): add pp_skip_hspace to skip horizontal whitespace during parsing
 1.71  07-Nov-2020  rillig make(1): clean up code stylistically

* Replace character literal 0 with '\0'.
* Replace pointer literal 0 with NULL.
* Remove redundant parentheses.
* Parentheses in multi-line conditions are not redundant at the
beginning of a line.
* Replace a few !ptr with ptr == NULL.
* Replace a few ptr with ptr != NULL.
* Replace (expr & mask) == 0 with !(expr & mask).
* Remove redundant braces for blocks in cases where the generated code
stays the same. (Assertions further down in the code would get
different line numbers.)
* Rename parameters in CondParser_String to reflect the data flow.
* Replace #ifdef notdef with #if 0.

The generated code stays exactly the same, at least with GCC 5.5.0 on
NetBSD 8.0 amd64 using the default configuration.
 1.70  24-Oct-2020  rillig make(1): remove macros MIN and MAX

These macros typically evaluate one of their arguments twice. Until
2020-08-31, they had not parenthesized their arguments properly. They
are only used in a few places, therefore it doesn't hurt much to have
them expanded.
 1.69  22-Oct-2020  rillig make(1): remove redundant type casts

This mainly affects the void pointers in callback functions for lists.
These had been necessary once when the parameter type was still
ClientData instead of void pointer.
 1.68  05-Oct-2020  rillig make(1): revert previous commit

It had accidentally reverted all the work from the past few days.
 1.67  05-Oct-2020  rillig make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25. In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath. Dir_ClearPath takes full ownership of the given list and
empties it. This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table. This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes. This by itself
was not a problem since the hash table would be freed afterwards. But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again. Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.
 1.66  03-Oct-2020  rillig make(1): inline Str_FindSubstring in JobOutput
 1.65  13-Sep-2020  rillig make(1): clean up RCSID blocks

These blocks mostly consisted of redundant structure, following the same
#ifndef pattern over and over, with only minimal variation.

It's easier to maintain if the common structure is only written once and
encapsulated in a macro.

To avoid "defined but unused" warnings from GCC in the case where
MAKE_NATIVE is not defined, I had to add volatile. Adding
MAKE_ATTR_UNUSED alone would not preserve the rcsid variable in the
resulting binary.
 1.64  30-Aug-2020  rillig make(1): replace brk_string with Str_Words

The API is much simpler, and there is less detail that is exposed by
default and fewer punctuation to type on the caller's side. To see that
there is some memory to be freed, one would have to look into the
struct. Having part of the return value as the actual return value and
the rest in output parameters was unnecessarily asymmetrical.
 1.63  29-Aug-2020  rillig make(1): allow for strict type checking for Boolean

Having Boolean aliased to int creates ambiguities since int is widely
used. Allow to occasionally compile make with -DUSE_DOUBLE_BOOLEAN to
check that the type definitions still agree.
 1.62  23-Aug-2020  rillig make(1): make brk_string return size_t for the number of words
 1.61  23-Aug-2020  rillig make(1): prepare str.c for WARNS=6

The next step is to make brk_string return size_t instead of int.
 1.60  11-Aug-2020  rillig make(1): add str_concat4 to make the other code simpler

There's no need for arch.c to call strlen when there is a high-level API
for the same purpose.
 1.59  10-Aug-2020  rillig make(1): replace str_concat with str_concat2 and str_concat3

The new functions have a simpler interface, and str_concat3 is even more
general-purpose, since the middle string is no longer required to be
exactly of length 1.
 1.58  10-Aug-2020  rillig make(1): fix parameter name of str_concat

The previous documentation mentioned Str_Concat, but str_concat has been
written in lowercase for years. The "flags" are not flags since they
cannot be combined, not even when they are written in hex.
 1.57  10-Aug-2020  rillig make(1): fix integer type in str_concat
 1.56  09-Aug-2020  rillig make(1): format code in str.c consistently
 1.55  03-Aug-2020  rillig make(1): no declaration-after-statement anymore

NetBSD make is intended to be maximally portable, therefore it uses only
C89. This was not declared in the Makefile before.

There are still a few places in parse.c and metachar.c that use
end-of-line comments. These will be fixed in a follow-up commit.
 1.54  01-Aug-2020  rillig make(1): reduce scope of local variables in brk_string

This also removes the unused assignment to words_cap.
 1.53  26-Jul-2020  rillig make(1): document the circumstances in which brk_string returns NULL
 1.52  19-Jul-2020  rillig make(1): move SysV string matching to var.c

This kind of string matching is only used in variable modifiers, and only
if this feature is enabled by SYSVVARSUB.
 1.51  03-Jul-2020  rillig make(1): refactor Str_Match

- use shorter variable names to avoid line breaks
- eliminate c2 variable
- change return type to Boolean
 1.50  03-Jul-2020  rillig make(1): clean up Str_Match

- removed unnecessary parentheses
- uniform test for end of string
 1.49  03-Jul-2020  rillig make(1): clean up documentation of Str_Match
 1.48  15-Jun-2020  rillig make(1): fix performance problem in specially crafted :M modifier

This fix was previously suspected to make the vax build fail. The next
build succeeded though, and it started 2 hours before this fix was
reverted.
 1.47  14-Jun-2020  rillig usr.bin/make: revert performance improvement

That change might be the cause of a build failure for vax.
http://releng.netbsd.org/builds/HEAD/202006131940Z/ builds fine.
http://releng.netbsd.org/builds/HEAD/202006141020Z/ doesn't.

The build fails with:
/home/source/ab/HEAD/src/external/gpl3/gcc/dist/gcc/machmode.h:524:28:
error: 'mode_size_inline' was not declared in this scope
 1.46  13-Jun-2020  rillig usr.bin/make: make Str_Match faster for repeated asterisks

Conceptually related to https://en.wikipedia.org/wiki/ReDoS.
 1.45  13-Jun-2020  rillig usr.bin/make: remove redundant parentheses around return
 1.44  13-Jun-2020  rillig usr.bin/make: consistently use ++ for incrementing pointers
 1.43  13-Jun-2020  rillig usr.bin/make: fix typo in comment
 1.42  06-May-2020  christos Behave like gmake: empty string does not match % pattern
 1.41  03-May-2020  rillig usr.bin/make: refactor brk_string

The variables are renamed to reflect to which memory region each pointer
belongs.

The variable "curlen" was always zero.

The type of "ch" has changed to char, and its scope is now limited to
its actual use.

Comparisons of pointers are now consistently written as p != NULL
instead of !p, like character comparisons are written as ch != '\0'.

The "store_words_buf" is updated when the function returns, not before.
 1.40  25-Apr-2020  christos - Percent on the rhs is special only if the lhs has one too.
- If the rhs does not have percent, but the lhs has, the result is the rhs
This behavior matches gmake
 1.39  01-Dec-2019  rillig Fix out-of-bounds read in Str_Match.
 1.38  21-Apr-2017  sjg branches: 1.38.12;
Str_Match: fix closure tests for [^] and add unit-test.
 1.37  11-Apr-2017  sjg Str_Match: allow [^a-z] to behave as expected.
 1.36  06-Apr-2016  gson branches: 1.36.2; 1.36.4;
Fully avoid the nonstandard types u_char, u_int, and u_long, which
were only used inconsistently in a few places anyway.
 1.35  12-Feb-2014  sjg Reduce the calls to realloc from brk_string by basing initial argmax guess
on length of input.

Reviewed by: christos
 1.34  03-Mar-2012  dholland branches: 1.34.2;
Note in comment that Str_Match() does not detect or report invalid
patterns. This is not critical, since for the moment both possible
cases (unclosed [], and \ at the end of the string) are handled more
or less sanely.
 1.33  25-Feb-2009  sno branches: 1.33.6;
replace c++ style comment by a c-style one
 1.32  17-Jan-2009  dsl branches: 1.32.2;
Change 'Buffer' so that it is the actual struct, not a pointer to it.
Saves having to malloc/free a fixed size structure.
Buf_Init() now takes ptr to Buffer to initialiase.
Change Buf_Destroy() to return ptr to string when not freed.
Remove large number of casts to (Byte) and (Byte *) - 'Byte' is 'char' here.
Buf_AddByte[s] guarantees that the data is 0 termininated, so never add '\0'.
Keep 'count' not 'left' and 'inPtr', code is simplier with only one update.
Fix fallou, no functional change.
 1.31  19-Dec-2008  christos PR/32154: Roland Illig: brk_string fails to report errors in unquoted strings.
 1.30  19-Dec-2008  christos PR/34138: Roland Illig: In the non-expand case, fix brk_string not to eat
quotes.
 1.29  06-Oct-2008  joerg Don't use emalloc and friends directly, but call them consistently
bmake_malloc and friends. Implement them via macros for the native case
and provide fallback implementations otherwise. Avoid polluting the
namespace by not defining enomem globally. Don't bother to provide
strdup and strndup, they were only used for the estrdup and estrndup
comapt code.

This addresses the presence of emalloc in system libraries on A/UX and
resulted strange issues as reported by Timothy E. Larson.
 1.28  15-Feb-2008  christos back all changes out until I fix it properly.
 1.27  14-Feb-2008  christos - use pid_t/size_t as appropriate instead of int.
- use %ld to print pids.
- fix a bit of lint.
- WARNS=4
 1.26  22-Dec-2006  erh Correctly handle '\' at the end of a variable in the non-expanding case.
 1.25  11-Aug-2006  christos branches: 1.25.2; 1.25.4;
If a word is "" eat the last quote since we ate the first.
 1.24  08-Aug-2005  christos From Max Okumoto:
- Remove casts to NULL.
- Remove space between cast and object.
 1.23  16-Feb-2005  christos PR/29203, PR/29204: Max Okumoto: KNF changes to make [no functional changes]
 1.22  07-May-2004  ross Simplify build, no functional changes.

Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().

It's now possible to build make on some hosts with: cc *.c */*.c
 1.21  27-Sep-2003  sjg branches: 1.21.2;
Implement :[] modifier to allow picking a range of words out of a variable.
Also :tW and a W flag to :C and :S to allow treating value as a single word.
Add unit tests for the above, and fix some corner cases.
Based on patches supplied by Alan Barrett <apb@cequrux.com>
 1.20  07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22365, verified by myself.
 1.19  14-Jul-2003  christos Pass WARNS=3
 1.18  15-Jun-2002  wiz Remove !__STDC__ stuff, de-__P(), ANSIfy, and de-register.
 1.17  06-Nov-1998  christos - full prototypes
- more conservative printf
- char -> unsigned char
 1.16  26-Mar-1998  christos PR/5210: Hauke Fath: make core dumps with .SHELL
Unfortunately this revealed a deeper problem with the brk_string code.
To fix it:
- remove sharing of the buffer between brk_string invocations
- change the semantics of brk_string so that the argument array
starts with 0, and return the buffer where the strings are
stored
 1.15  28-Sep-1997  lukem branches: 1.15.2;
wrap #include <sys/cdefs.h>, __RCSID(...) stuff in #ifndef MAKE_BOOTSTRAP
 1.14  01-Jul-1997  christos Add WARNS=1
RCSID police
 1.13  06-Nov-1996  christos - Merge in FreeBSD and Lite2 changes.
- Fix bug where a non-archive target with a .a suffix would always
be considered to be out of date, since it does not have a TOC.
 1.12  29-Mar-1996  jtc branches: 1.12.4;
Added erealloc() function that calls enomem() if the allocation fails
(this is like the existing emalloc() function). Changed all realloc()
calls to erealloc().
 1.11  30-Oct-1995  ws Always ensure space for trailing NULL in argv
 1.10  14-Jun-1995  christos branches: 1.10.2;
- $NetBSD$ rcsids
- Fixed so that .[A-Z]* targets that do not match keywords are ignored as
Posix mandates
- Added .PHONY target keyword
 1.9  04-Feb-1995  christos str.c: Don't free NULL; does not work with non posix frees.
var.c: Minor memory leak plugged.
suff.c: Don't add extra sources on the null suffix if it has dependency
lines or commands attached to it [POSIX says so]
 1.8  16-Jun-1994  jtc Christos' fix for quoting variable extraction properly.
 1.7  06-Jun-1994  jtc Fixes from Christos Zoulas, who used purify, objectcenter and testcenter
to find memory leaks and illegal memory accesses.
 1.6  17-May-1994  jtc Properly parse null command line arguments in makefile rules.
Fix from Christos Zoulas.
 1.5  23-Mar-1994  jtc Fixes from Christos Zoulas:
The following two patches fix a couple of problems with make(1)

1. Null Suffixes were not being copied, but they were being free'd
This caused rules of the form:

.c:
${CC} ...

to access invalid memory and potentially core dump..
[That was always broken; I did not break that one :-)]

2. My recent fixes to parse ${VAR:%.foo=%.bar} handled the null string
case incorrectly (${VAR:=.c} was broken).
 1.4  05-Mar-1994  cgd fixes/improvements from Christos Zoulas <christos@deshaw.com>.
 1.3  13-Jan-1994  jtc Include appropriate header files to bring prototypes into scope.
 1.2  01-Aug-1993  mycroft Add RCS identifiers.
 1.1  21-Mar-1993  cgd branches: 1.1.1;
Initial revision
 1.1.1.2  28-Dec-1996  tls Import 4.4BSD-Lite2 sources onto CSRG branch (already merged at head)
 1.1.1.1  21-Mar-1993  cgd initial import of 386bsd-0.1 sources
 1.10.2.1  30-Oct-1995  jtc Update from trunk: Always ensure space for trailing NULL in argv
 1.12.4.1  26-Jan-1997  rat Update make(1) from trunk, by request from Christos Zoulas. Fixes many bugs.
 1.15.2.1  08-May-1998  mycroft Sync with trunk, per request of christos.
 1.21.2.1  10-May-2004  tron Pull up revision 1.22 (requested by sjg in ticket #282):
Simplify build, no functional changes.
Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().
It's now possible to build make on some hosts with: cc *.c */*.c
 1.25.4.1  03-Sep-2007  wrstuden Sync w/ NetBSD-4-RC_1
 1.25.2.1  05-Jun-2007  bouyer Apply patch (requested by tron in ticket #696):
usr.bin/make/compat.c patch
usr.bin/make/cond.c patch
usr.bin/make/dir.c patch
usr.bin/make/for.c patch
usr.bin/make/main.c patch
usr.bin/make/make.1 patch
usr.bin/make/make.c patch
usr.bin/make/make.h patch
usr.bin/make/nonints.h patch
usr.bin/make/parse.c patch
usr.bin/make/str.c patch
usr.bin/make/targ.c patch
usr.bin/make/util.c patch
usr.bin/make/var.c patch

Synchronize make(1) with HEAD branch to increase perfomance and
improve stability.
 1.32.2.1  13-May-2009  jym Sync with HEAD.

Third (and last) commit. See http://mail-index.netbsd.org/source-changes/2009/05/13/msg221222.html
 1.33.6.2  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.33.6.1  17-Apr-2012  yamt sync with head
 1.34.2.1  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.36.4.1  21-Apr-2017  bouyer Sync with HEAD
 1.36.2.1  26-Apr-2017  pgoyette Sync with HEAD
 1.38.12.1  08-Apr-2020  martin Merge changes from current as of 20200406
 1.84.2.1  31-May-2021  cjep sync with head

RSS XML Feed