Home | History | Annotate | Download | only in makemandb
History log of /src/usr.sbin/makemandb/Makefile
RevisionDateAuthorComments
 1.13  13-Nov-2024  riastradh makemandb(8): Fix static build with PROGDPLIBS.

XXX This shouldn't have to mention transitive dependencies -- we
should have a mechanism for upstream build goo for libarchive to
transmit the information automatically to downstream build goo for
makemandb.
 1.12  12-Nov-2020  nia Revert addition of pthread dependency on sqlite.

It is less trivial than expected and introduced some surprising breakage.
 1.11  09-Nov-2020  martin Fix copy&pasto in previous
 1.10  08-Nov-2020  nia sqlite: do not build without multithreading support

at least a few pkgsrc packages avoid base sqlite because it fails
this check, and it's probably a surprising performance penalty for
unsuspecting users
 1.9  18-Jun-2017  abhinav Add a custom tokenizer which does not stem certain keywords.

Which keywords should not be stemmed is specified in the nostem.txt file.
(Right now I have taken all the man page names, split them if they had
underscores, removed common English words and converted everything to
lowercase.)

The tokenizer itself is based on the Porter stemming tokenizer shipped with
Sqlite. The code in custom_apropos_tokenizer.c is copy of that code with
some modifications to prevent stemming keywords specified in nostem.txt.

Additionally, it now uses underscore `_' also as a token delimiter. Therefore,
now it's possible to do query for `lwp' and all `_lwp_*' man page names
will be matched. Or the query can be `unconst' and `__UNCONST' will be matched.
This was not possible earlier, because underscore was not a delimiter and therefore
the index would have __UNCONST as a key rather than UNCONST.

The tokenizer needs fts3_tokenizer.h file, which is not shipped with the
amalgamation build of Sqlite, therefore it needs to be added here (unless
we decide there is a better place for it).

To enforce using the new tokenizer, a schema version bump is needed

Since the tokenization is done both at the indexing time (via makemandb) and
also while query time (via apropos or whatis), it will be needed to bump
the schema version everytime nostem.txt is modified. Otherwise the
index will consist of old tokens and desired changes will not be seen with
apropos.

This should also fix the issue reported in PR bin/46255. Similar suggestion was
also made on tech-userlevel@ recently:
<http://mail-index.netbsd.org/tech-userlevel/2017/06/08/msg010620.html>

Thanks to christos@ for multiple rounds of reviews of the tokenizer code.
 1.8  21-May-2017  riastradh Remove MKCRYPTO option.

Originally, MKCRYPTO was introduced because the United States
classified cryptography as a munition and restricted its export. The
export controls were substantially relaxed fifteen years ago, and are
essentially irrelevant for software with published source code.

In the intervening time, nobody bothered to remove the option after
its motivation -- the US export restriction -- was eliminated. I'm
not aware of any other operating system that has a similar option; I
expect it is mainly out of apathy for churn that we still have it.
Today, cryptography is an essential part of modern computing -- you
can't use the internet responsibly without cryptography.

The position of the TNF board of directors is that TNF makes no
representation that MKCRYPTO=no satisfies any country's cryptography
regulations.

My personal position is that the availability of cryptography is a
basic human right; that any local laws restricting it to a privileged
few are fundamentally immoral; and that it is wrong for developers to
spend effort crippling cryptography to work around such laws.

As proposed on tech-crypto, tech-security, and tech-userlevel to no
objections:

https://mail-index.netbsd.org/tech-crypto/2017/05/06/msg000719.html
https://mail-index.netbsd.org/tech-security/2017/05/06/msg000928.html
https://mail-index.netbsd.org/tech-userlevel/2017/05/06/msg010547.html

P.S. Reviewing all the uses of MKCRYPTO in src revealed a lot of
*bad* crypto that was conditional on it, e.g. DES in telnet... That
should probably be removed too, but on the grounds that it is bad,
not on the grounds that it is (nominally) crypto.
 1.7  21-Apr-2017  christos libarchive now needs crypto
 1.6  21-Jul-2016  christos Add -lz to makefile to fix the build.
 1.5  14-Jan-2013  christos branches: 1.5.12;
Fix the #ifdef NOTYET, namely use the terminal standout mode to highlight the
search terms.
 1.4  14-Jan-2013  christos Since mdocml decided to name headers that conflict with system ones (term.h)
move the header inclusion one up.
 1.3  06-Oct-2012  wiz Make mandb path configurable. makemandb (and related tools) use
the path from the _mandb variable from man.conf now.

Set _mandb in man.conf to same value as was used before.

From Abhinav Upadhyay <er.abhinav.upadhyay@gmail.com>.
 1.2  16-Feb-2012  joerg branches: 1.2.2; 1.2.4;
Add support for compressed man pages in all the usual formats.
 1.1  07-Feb-2012  joerg branches: 1.1.2;
Import the new apropos/whatis.

This code has been developed by Abhinav Upadhyay as part of Google's Summer
of Code 2011. It uses libmandoc to parse man pages and builds a Full
Text Index in a SQLite database. The combination of indexing the full
manual page, filtering out stop words and ranking individual matches
based on the section gives a much improved user experience.

The old makewhatis and friends are kept under MKMAKEMANDB=no for now.
 1.1.2.1  18-Feb-2012  riz Pull up following revision(s) (requested by joerg in ticket #4):
usr.sbin/makemandb/apropos.c: revision 1.5
external/bsd/mdocml/dist/read.c: revision 1.6
external/bsd/mdocml/dist/mandoc.h: revision 1.2
usr.sbin/makemandb/makemandb.c: revision 1.3
usr.sbin/makemandb/makemandb.c: revision 1.4
usr.sbin/makemandb/makemandb.c: revision 1.5
usr.sbin/makemandb/Makefile: revision 1.2
etc/man.conf: revision 1.30
usr.sbin/makemandb/apropos.1: revision 1.3
external/bsd/mdocml/Makefile.inc: revision 1.15
Be a bit more friendly to man pages using the roff .so command by
changing the current directory to the parent of the man -p entry, e.g.
/usr/share/man for /usr/share/man1.
Also handle hyphen replacement if it was used as plain input and no
backslash sequence was used at all in the line.
Add support for apropos -s like in the old apropos. Fix capitalisation.
Add note about optional quotation. From Abhinav Upadhyay.
Merge mandoc.h r1.99 and read.c 1.28 from upstream:
Add mparse_readmem, which allows application to run the parser directly
from memory, e.g. after de-compressing a document.
Add support for compressed man pages in all the usual formats.
Use mmap when dealing with plain files.
Add entries for bzip2 and xz.
 1.2.4.2  25-Feb-2013  tls resync with head
 1.2.4.1  20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.2.2.4  23-Jan-2013  yamt sync with head
 1.2.2.3  30-Oct-2012  yamt sync with head
 1.2.2.2  17-Apr-2012  yamt sync with head
 1.2.2.1  16-Feb-2012  yamt file Makefile was added on branch yamt-pagecache on 2012-04-17 00:09:49 +0000
 1.5.12.2  26-Apr-2017  pgoyette Sync with HEAD
 1.5.12.1  26-Jul-2016  pgoyette Sync with HEAD

RSS XML Feed