1 Maintaining Autosetup in the SQLite Tree 2 ======================================================================== 3 4 This document provides some tips and reminders for the SQLite 5 developers regarding using and maintaining the [Autosetup][]-based 6 build infrastructure. It is not an [Autosetup][] reference. 7 8 **Table of Contents**: 9 10 - [Autosetup API Reference](#apiref) 11 - [API Tips](#apitips) 12 - [Ensuring TCL Compatibility](#tclcompat) 13 - [Design Conventions](#conventions) 14 - Symbolic Names of Feature Flags 15 - Do Not Update Global Shared State 16 - [Updating Autosetup](#updating) 17 - ***[Patching Autosetup for Project-local changes](#patching)*** 18 - [Branch-specific Customization](#branch-customization) 19 20 21 ------------------------------------------------------------------------ 22 23 <a name="apiref"></a> 24 Autosetup API Reference 25 ======================================================================== 26 27 The Autosetup API is quite extensive and can be read either in 28 the [files in the `autosetup` dir](/dir/autosetup) or using: 29 30 > 31 ``` 32 $ ./configure --reference | less 33 ``` 34 35 That will include any docs from any TCL files in the `./autosetup` dir 36 which contain certain (simple) markup defined by autosetup. 37 38 This project's own configuration-related TCL code is spread across the 39 following files: 40 41 - [proj.tcl][]: project-agnostic utility code for autosetup-driven 42 projects. This file is designed to be shared between this project, 43 other projects managed under the SQLite/Hwaci umbrella 44 (e.g. Fossil), and personal projects of SQLite's developers. It is 45 essentially an amalgamation of a decade's worth of autosetup-related 46 utility code. 47 - [sqlite-config.tcl][]: utility code which is too project-specific 48 for `proj.tcl`. We split this out of `auto.def` so that it can be 49 used by both `auto.def` and... 50 - [auto.def][]: the primary driver for the `./configure` process. 51 When we talk about "the configure script," we're technically 52 referring to this file, though it actually contains very little 53 of the TCL code. 54 - [autoconf/auto.def][]: the main driver script for the "autoconf" 55 bundle's configure script. It is essentially a slightly trimmed-down 56 version of the main `auto.def` file. The `autoconf` dir was ported 57 from the Autotools to Autosetup in the 3.49.0 dev cycle but retains 58 the "autoconf" name to minimize downstream disruption. 59 60 61 <a name="apitips"></a> 62 Autosetup API Tips 63 ======================================================================== 64 65 This section briefly covers only APIs which are frequently useful in 66 day-to-day maintenance and might not be immediately recognized as such 67 from a casual perusal of the relevant TCL files. The complete docs of 68 those with `proj-` prefix can be found in [proj.tcl][] and those with 69 an `sqlite-` prefix are in [sqlite-config.tcl][]. The others are part 70 of Autosetup's core packages and are scattered around [the TCL files 71 in ./autosetup](/dir/autosetup). 72 73 In (mostly) alphabetical order: 74 75 - **`file-isexec filename`**\ 76 Should be used in place of `[file executable]`, as it will also 77 check for `${filename}.exe` on Windows platforms. However, on such 78 platforms it also assumes that _any_ existing file is executable. 79 80 - **`get-env VAR ?default?`**\ 81 Will fetch an "environment variable" from the first of either: (1) a 82 KEY=VALUE passed to the configure script or (2) the system's 83 environment variables. Not to be confused with `getenv`, which only 84 does the latter and is rarely, if ever, useful in this tree. 85 - **`proj-get-env VAR ?default?`**\ 86 Works like `get-env` but will, if that function finds no match, 87 look for a file named `./.env-$VAR` and, if found, return its 88 trimmed contents. This can be used, e.g., to set a developer's 89 local preferences for the default `CFLAGS`.\ 90 Tip: adding `-O0` to `.env-CFLAGS` reduces rebuild times 91 considerably at the cost of performance in `make devtest` and the 92 like. 93 94 - **`proj-fatal msg`**\ 95 Emits `$msg` to stderr and exits with non-zero. Its differences from 96 autosetup's `user-error` are purely cosmetic. 97 98 - **`proj-if-opt-truthy flag thenScript ?elseScript?`**\ 99 Evals `thenScript` if the given `--flag` is truthy, else it 100 evals the optional `elseScript`. 101 102 - **`proj-indented-notice ?-error? ?-notice? msg`**\ 103 Breaks its `msg` argument into lines, trims them, and emits them 104 with consistent indentation. Exactly how it emits depends on the 105 flags passed to it (or not), as covered in its docs. This will stick 106 out starkly from normal output and is intended to be used only for 107 important notices. 108 109 - **`proj-opt-truthy flag`**\ 110 Returns 1 if `--flag`'s value is "truthy," i.e. one of (1, on, 111 enabled, yes, true). 112 113 - **`proj-opt-was-provided FLAG`**\ 114 Returns 1 if `--FLAG` was explicitly provided to configure, 115 else 0. This distinction can be used to determine, e.g., whether 116 `--with-readline` was provided or whether we're searching for 117 readline by default. In the former case, failure to find it should 118 be treated as fatal, where in the latter case it's not.\ 119 Unlike most functions which deal with `--flags`, this one does not 120 validate that `$FLAG` is a registered flag so will not fail fatally 121 if `$FLAG` is not registered as an Autosetup option. 122 123 - **`proj-val-truthy value`**\ 124 Returns 1 if `$value` is "truthy," See `proj-opt-truthy` for the definition 125 of "truthy." 126 127 - **`proj-warn msg`**\ 128 Emits `$msg` to stderr. Closely-related is autosetup's `user-notice` 129 (described below). 130 131 - **`sqlite-add-feature-flag ?-shell? FLAG...`**\ 132 Adds the given feature flag to the CFLAGS which are specific to 133 building libsqlite3. It's intended to be passed one or more 134 `-DSQLITE_ENABLE_...`, or similar, flags. If the `-shell` flag is 135 used then it also passes its arguments to 136 `sqlite-add-shell-opt`. This is a no-op if `FLAG` is not provided or 137 is empty. 138 139 - **`sqlite-add-shell-opt FLAG...`**\ 140 The shell-specific counterpart of `sqlite-add-feature-flag` which 141 only adds the given flag(s) to the CLI-shell-specific CFLAGS. 142 143 - **`sqlite-configure BUILD-NAME {script}`**\ 144 This is where all configure `--flags` are defined for all known 145 build modes ("canonical" or "autoconf"). After processing all flags, 146 this function runs `$script`, which contains the build-mode-specific 147 configuration bits, and then runs any finalization bits which are 148 common to all build modes. The `auto.def` files are intended to contain 149 exactly two commands: 150 `use sqlite-config; sqlite-configure BUILD-NAME {script}` 151 152 - **`user-notice msg`**\ 153 Queues `$msg` to be sent to stderr, but does not emit it until 154 either `show-notices` is called or the next time autosetup would 155 output something (it internally calls `show-notices`). This can be 156 used to generate warnings between a "checking for..." message and 157 its resulting "yes/no/whatever" message in such a way as to not 158 spoil the layout of such messages. 159 160 161 <a name="tclcompat"></a> 162 Ensuring TCL Compatibility 163 ======================================================================== 164 165 One of the significant benefits of using Autosetup is that (A) this 166 project uses many TCL scripts in the build process and (B) Autosetup 167 comes with a TCL interpreter named [JimTCL][]. 168 169 It is important that any TCL files used by the configure process and 170 makefiles remain compatible with both [JimTCL][] and the canonical 171 TCL. Though JimTCL has outstanding compatibility with canonical TCL, 172 it does have a few corners with incompatibilities, e.g. regular 173 expressions. If a script runs in JimTCL without using any 174 JimTCL-specific features, then it's a certainty that it will run in 175 canonical TCL as well. The opposite, however, is not _always_ the 176 case. 177 178 When [`./configure`](/file/configure) is run, it goes through a 179 bootstrapping process to find a suitable TCL with which to run the 180 autosetup framework. The first step involves [finding or building a 181 TCL shell](/file/autosetup/autosetup-find-tclsh). That will first 182 search for an available `tclsh` (under several common names, 183 e.g. `tclsh8.6`) before falling back to compiling the copy of 184 `jimsh0.c` included in the source tree. i.e. it will prefer to use a 185 system-installed TCL for running the configure script. Once it finds 186 (or builds) a TCL shell, it then runs [a sanity test to ensure that 187 the shell is suitable](/file/autosetup/autosetup-test-tclsh) before 188 using it to run the main autosetup app. 189 190 There are two simple ways to ensure that running of the configure 191 process uses JimTCL instead of the canonical `tclsh`, and either 192 approach provides equally high assurances about configure script 193 compatibility across TCL implementations: 194 195 1. Build on a system with no `tclsh` installed in the `$PATH`. In that 196 case, the configure process will fall back to building the in-tree 197 copy of JimTCL. 198 199 2. Manually build `./jimsh0` in the top of the checkout with:\ 200 `cc -o jimsh0 autosetup/jimsh0.c`\ 201 With that in place, the configure script will prefer to use that 202 before looking for a system-level `tclsh`. Be aware, though, that 203 `make distclean` will remove that file. 204 205 **Note that `./jimsh0` is distinctly different from the `./jimsh`** 206 which gets built for code-generation purposes. The latter requires 207 non-default build flags to enable features which are 208 platform-dependent, most notably to make its `[file normalize]` work. 209 This means, for example, that the configure script and its utility 210 APIs must not use `[file normalize]`, but autosetup provides a 211 TCL-only implementation of `[file-normalize]` (note the dash) for 212 portable use in the configure script. Contrariwise, code-generation 213 scripts invoked via `make` may use `[file normalize]`, as they'll use 214 `./jimsh` or `tclsh` instead of `./jimsh0`. 215 216 217 Known TCL Incompatibilities 218 ------------------------------------------------------------------------ 219 220 A summary of known incompatibilities in JimTCL 221 222 - **CRNL line endings**: prior to 2025-02-05 `fconfigure -translation ...` 223 was a no-op in JimTCL, and it emits CRNL line endings by default on 224 Windows. Since then, it supports `-translation binary`, which is 225 close enough to `-translation lf` for our purposes. When working 226 with files using the `open` command, it is important to use mode 227 `"rb"` or `"wb"`, as appropriate, so that the output does not get 228 CRNL-mangled on Windows. 229 230 - **`file copy`** does not support multiple source files. See 231 [](/info/61f18c96183867fe) for a workaround. 232 233 - **Regular expressions**: 234 235 - Patterns treat `\nnn` octal values as back-references (which it 236 does not support). Those can be reformulated as demonstrated in 237 [](/info/aeac23359bb681c0). 238 239 - `regsub` does not support the `\y` flag. A workaround is demonstrated 240 in [](/info/c2e5dd791cce3ec4). 241 242 243 <a name="conventions"></a> 244 Design Conventions 245 ======================================================================== 246 247 This section describes the motivations for the most glaring of the 248 build's design decisions, in particular how they deviate from 249 historical, or even widely-conventional, practices. 250 251 Symbolic Names of Feature Flags 252 ------------------------------------------------------------------------ 253 254 Historically, the project's makefile has exclusively used 255 `UPPER_UNDERSCORE` form for makefile variables. This build, however, 256 primarily uses `X.y` format, where `X` is often a category label, 257 e.g. `CFLAGS`, and `y` is the specific instance of that category, 258 e.g. `CFLAGS.readline`. 259 260 When the configure script exports flags for consumption by filtered 261 files, e.g. [Makefile.in][] and the generated 262 `sqlite_cfg.h`, it does so in the more conventional `X_Y` form because 263 those flags get exported as as C `#define`s to `sqlite_cfg.h`, where 264 dots are not permitted. 265 266 The `X.y` convention is used in the makefiles primarily because the 267 person who did the initial port finds that considerably easier on the 268 eyes and fingers. In practice, the `X_Y` form of such exports is used 269 exactly once in [Makefile.in][], where it's translated from `@X_Y@` 270 into into `X.y` form for consumption by [Makefile.in][] and 271 [main.mk][]. For example: 272 273 > 274 ``` 275 LDFLAGS.shobj = @SHOBJ_LDFLAGS@ 276 LDFLAGS.zlib = @LDFLAGS_ZLIB@ 277 LDFLAGS.math = @LDFLAGS_MATH@ 278 ``` 279 280 (That first one is defined by autosetup, and thus applies "LDFLAGS" as 281 the suffix rather than the prefix. Which is more legible is a matter 282 of taste, for which there is no accounting.) 283 284 285 Do Not Update Global Shared State 286 ------------------------------------------------------------------------ 287 288 In both the legacy Autotools-driven build and common Autosetup usage, 289 feature tests performed by the configure script may amend global flags 290 such as `LIBS`, `LDFLAGS`, and `CFLAGS`[^as-cflags]. That's 291 appropriate for a makefile which builds a single deliverable, but less 292 so for makefiles which produce multiple deliverables. Drawbacks of 293 that approach include: 294 295 - It's unlikely that every single deliverable will require the same 296 core set of those flags. 297 - It can be difficult to determine the origin of any given change to 298 that global state because those changes are hidden behind voodoo 299 performed outside the immediate visibility of the configure script's 300 maintainer. 301 - It can force the maintainers of the configure script to place tests 302 in a specific order so that the resulting flags get applied at 303 the correct time and/or in the correct order.\ 304 (A real-life example: before the approach described below was taken 305 to collecting build-time flags, the test for `-rpath` had to come 306 _after_ the test for zlib because the results of the `-rpath` test 307 implicitly modified global state which broke the zlib feature 308 test. Because the feature tests no longer (intentionally) modify 309 shared global state, that is not an issue.) 310 311 In this build, cases where feature tests modify global state in such a 312 way that it may impact later feature tests are either (A) very 313 intentionally defined to do so (e.g. the `--with-wasi-sdk` flag has 314 invasive side-effects) or (B) are oversights (i.e. bugs). 315 316 This tree's [configure script][auto.def], [utility APIs][proj.tcl], 317 [Makefile.in][], and [main.mk][] therefore strive to separate the 318 results of any given feature test into its own well-defined 319 variables. For example: 320 321 - The linker flags for zlib are exported from the configure script as 322 `LDFLAGS_ZLIB`, which [Makefile.in][] and [main.mk][] then expose as 323 `LDFLAGS.zlib`. 324 - `CFLAGS_READLINE` (a.k.a. `CFLAGS.readline`) contains the `CFLAGS` 325 needed for including `libreadline`, `libedit`, or `linenoise`, and 326 `LDFLAGS_READLINE` (a.k.a. `LDFLAGS.readline`) is its link-time 327 counterpart. 328 329 It is then up to the Makefile to apply and order the flags however is 330 appropriate. 331 332 At the end of the configure script, the global `CFLAGS` _ideally_ 333 holds only flags which are either relevant to all targets or, failing 334 that, will have no unintended side-effects on any targets. That said: 335 clients frequently pass custom `CFLAGS` to `./configure` or `make` to 336 set library-level feature toggles, e.g. `-DSQLITE_OMIT_FOO`, in which 337 case there is no practical way to avoid "polluting" the builds of 338 arbitrary makefile targets with those. _C'est la vie._ 339 340 341 [^as-cflags]: But see this article for a detailed discussion of how 342 autosetup currently deals specifically with CFLAGS: 343 <https://msteveb.github.io/autosetup/articles/handling-cflags/> 344 345 346 <a name="updating"></a> 347 Updating Autosetup 348 ======================================================================== 349 350 Updating autosetup is, more often than not, painless. It requires having 351 a checked-out copy of [the autosetup git repository][autosetup-git]: 352 353 > 354 ``` 355 $ git clone https://github.com/msteveb/autosetup 356 $ cd autosetup 357 # Or, if it's already checked out: 358 $ git pull 359 ``` 360 361 Then, from the top-most directory of an SQLite checkout: 362 363 > 364 ``` 365 $ /path/to/autosetup-checkout/autosetup --install . 366 $ fossil status # show the modified files 367 ``` 368 369 Unless the upgrade made any incompatible changes (which is exceedingly 370 rare), that's all there is to it. After that's done, **apply a patch 371 for the change described in the following section**, test the 372 configure process, and check it in. 373 374 <a name="patching"></a> 375 Patching Autosetup for Project-local Changes 376 ------------------------------------------------------------------------ 377 378 The autosetup files require the following patches after updating 379 from their upstream sources: 380 381 ### `--debug` flag 382 383 Autosetup reserves the flag name **`--debug`** for its own purposes, 384 and its own special handling of `--enable-...` flags makes `--debug` 385 an alias for `--enable-debug`. As this project has a long history of 386 using `--enable-debug`, we patch autosetup to use the name 387 `--autosetup-debug` in place of `--debug`. That requires (as of this 388 writing) four small edits in 389 [/autosetup/autosetup](/file/autosetup/autosetup), as demonstrated in 390 [check-in 3296c8d3](/info/3296c8d3). 391 392 If autosetup is upgraded and this patch is _not_ applied the invoking 393 `./configure` will fail loudly because of the declaration of the 394 `debug` flag in `auto.def` - duplicated flags are not permitted. 395 396 ### Fail on `malloc()` error 397 398 See [check-in 72c8a5b94cdf5d](/info/72c8a5b94cdf5d). 399 400 401 <a name="branch-customization"></a> 402 Branch-specific Customization 403 ======================================================================== 404 405 Certain vendor-specific branches require slight configure script 406 customization. Rather than editing `sqlite-config.tcl` for this, 407 which frequently leads to merge conflicts, the following approach 408 is recommended: 409 410 In the vendor-specific branch, create a file named 411 `autosetup/sqlite-custom.tcl`. 412 413 That file should contain the following content... 414 415 If flag customization is required, add: 416 417 > 418 ``` 419 proc sqlite-custom-flags {} { 420 # If any existing --flags require different default values 421 # then call: 422 options-defaults { 423 flag-name new-default-value 424 ... 425 } 426 # ^^^ That will replace the default value but will not update 427 # the --help text, which may lead to some confusion: 428 # https://github.com/msteveb/autosetup/issues/77 429 430 return { 431 {*} { 432 new-flag-name => {Help text} 433 ... 434 } 435 }; #see below 436 } 437 ``` 438 439 That function must return either an empty string or a list in the form 440 used internally by [sqlite-config.tcl][]'s `sqlite-configure`. 441 442 Next, define: 443 444 > 445 ``` 446 proc sqlite-custom-handle-flags {} { 447 ... do any custom flag handling here ... 448 } 449 ``` 450 451 That function, if defined, will be called relatively late in the 452 configure process, before any filtered files are generated but after 453 all other significant processing. 454 455 456 [Autosetup]: https://msteveb.github.io/autosetup/ 457 [auto.def]: /file/auto.def 458 [autoconf/auto.def]: /file/autoconf/auto.def 459 [autosetup-git]: https://github.com/msteveb/autosetup 460 [proj.tcl]: /file/autosetup/proj.tcl 461 [sqlite-config.tcl]: /file/autosetup/sqlite-config.tcl 462 [Makefile.in]: /file/Makefile.in 463 [main.mk]: /file/main.mk 464 [JimTCL]: https://msteveb.github.io/jimtcl/ 465
Indexes created Wed Apr 29 00:23:26 UTC 2026