refactor: split out importer code from searchix/web

refactor: move SetupIndex and indexing progress to importer

fix: missing content of last index run time

feat: improve handling of long option descriptions

feat: widen body

fix: block single-character queries

feat: improve display of indexing times and durations

feat: link directly to version changelog in footer

build: show version in development

feat(js): allow opening details in new tab/window

docs: add JSDoc comments to javascript functions

docs: change status to beta

feat: promote prefix/exact name/attr matches

Fixes: https://todo.sr.ht/~alanpearce/searchix/18

fix: don't attempt to load more than configured number of results

chore: release 0.1.31

fix: remove rendering from search timeout restriction

perf: iterate over results without keeping the set in memory

feat(nixos-module): allow setting environment variables

feat(nixos-module): allow setting environment variables

fix: package programs displayed off-centre

feat: demote NUR results in combined search

feat: promote results with literal or prefix name matches

feat: make list of source links dynamic

feat: enable NUR package import

feat: wrap search form in semantic <search> element

build: prefix branded version with 'v'

feat: show last/next/current indexing run time

feat: shorten shutdown timeout in development

fix: footer spacing

fix: detach version from rest of footer text

docs: add build status badge

fix: wrong pagination links for combined results

Fixes: https://todo.sr.ht/~alanpearce/searchix/15

feat: Convert templ components to gomponents

chore: release 0.1.30

feat: show version number and link in footer

build: update base css

feat: split compound words in names into n-grams

Implements: https://todo.sr.ht/~alanpearce/searchix/9

chore: use nixpkgs-unstable as flake input

chore: release 0.1.29

fix: nil pointer deference in nixpkgs-channel fetcher

chore: don't reindex on every dev start

fix: build error (incomplete commit)

chore: add build task to justfile

fix: build error caused by incomplete commit

refactor: ensure errors have stack traces

chore: release 0.1.28

fix: js error due to missing class on nav menu item

build: remove lorri from direnv configuration

chore: fix nixos module default config link not updating on release

chore: release 0.1.27

docs: use link to current version's defaults.toml

fix: sources disabled by default in NixOS module

Fixes: https://todo.sr.ht/~alanpearce/searchix/12

chore: release 0.1.26

fix: nixos import fails with default settings

Fixes: https://todo.sr.ht/~alanpearce/searchix/14

chore: release 0.1.25

fix: panic when source order > number of sources

Fixes: https://todo.sr.ht/~alanpearce/searchix/13

chore: update flake inputs, go and templ

feat: link to referenced projects

build: add justfile alias for projectdo

chore: release 0.1.24

refactor: remove unnecessary source lookup

fix(js): switching source after reset uses old query

chore: release 0.1.23

fix: remove duplicate link to source code

chore: release 0.1.22

feat: default to combined search on root path

refactor: interface{}->any

chore: release 0.1.21

feat: prioritise exact matches on programs/attr/name field

feat: improve relevance with compound words

Fixes: https://todo.sr.ht/~alanpearce/searchix/8
Fixes: https://todo.sr.ht/~alanpearce/searchix/10

refactor: remove unused analyser

build(dev): automatically re-index on indexer.go changes

chore: update bleve overlay

refactor: use pointer to sourceMeta

refactor: split importer args

fix: re-index refusing to delete data directory with programs.db

chore: release 0.1.20

fix: build failure (bad rebase)

chore: release 0.1.19

fix: import slowdown when querying programs database

perf: enable support for cpu profiling

build: allow passing flags to just {reindex,update}

perf: pass context to importer for better tracing

chore: release 0.1.18

feat: allow HTTP request logging to be disabled

feat(sentry): trace all transactions

chore: release 0.1.17

build: don't try to run non-existent tests

fix(sentry): report correct HTTP path in traces

refactor: trace 10% of requests

fix: use importer configured timeout for sentry check-ins

chore: release 0.1.16

feat: enable searching via program names for multi-program packages

implements: https://todo.sr.ht/~alanpearce/searchix/6

fix(dev): fix CSP not working in dev mode behind reverse proxy

chore: release 0.1.15

feat: offer to show all results on one page (if < 10000)

Implements: https://todo.sr.ht/~alanpearce/searchix/5

fix(dev): overwrite previous asset hash on change

refactor: extract pagination into module

chore: release 0.1.14

fix: boost non-stemmed matches over stemmed matches

https://todo.sr.ht/~alanpearce/searchix/7

feat: show scores in results table in dev mode

chore: release 0.1.13

feat: increase results per page

References: https://todo.sr.ht/~alanpearce/searchix/5

feat: de-rank packages on 'all' search

Implements: https://todo.sr.ht/~alanpearce/searchix/4

refactor: improve type hinting of setField

build: workaround nix build error with go 1.23

chore: update dependencies

chore: release 0.1.12

fix: use UTC time for fetch/import timer

fix: enable using module in pure evaluation mode

fix: enable using module in pure evaluation mode

chore: release 0.1.11

feat: apply stemming to search index terms

chore: disable integer overflow warnings

docs: update default nix-darwin importer config

chore: release 0.1.10

fix: search all missing expected results

Fixes: https://todo.sr.ht/~alanpearce/searchix/1

chore: release 0.1.9

fix: don't error when item not found

build: remove shell.nix to force lorri to use flakes

fix: tell browsers not to capitalise search input

chore: release 0.1.8

build: update nix package version number automatically

fix: nil panics rendering last page of search results

fix: panic rendering error pages

refactor: ignore unused variable

chore: release 0.1.8

fix: opensearch searches not using input query

refactor: remove unnecessary DocumentData struct

feat: enable combined search

fix: extra data after error parsing query string

fix: panic rendering search results

feat: more structured logging

fix: panic on search error

chore: release 0.1.7

build: add command to cut a release

build: generate correct changelog for next version

build: stamp binary correctly

refactor: rename go module

build: remove formatter for go template files

build: remove formatter for go template files

feat: add message for users with javascript disabled

fix: blank modal when using browser navigation

fix: duplicated results tables via JS search

fix: panic rendering results

fix: incorrect formatting of options results headers

style: remove vertical scrollbar on firefox-dev

refactor: use encoding/xml for OpenSearchDescription

refactor: switch to templ for HTML templates

build: use modd instead of watchexec

style: just use system fonts

chore: update dependencies

chore: release 0.1.6

style: increase information density

chore: update simple-css

feat: add low-memory mode

refactor: use a struct to configure index options

feat: enable configuration of flags as environment variables

fix: create index metadata when index exists and meta doesn't

build: add emacs formatting configuration

feat: display results in a table, showing details on click

feat: add detail pages for packages/options

chore: reduce tracing sample rate

feat: use stale-while-revalidate instead of immutable asset paths

feat: serve assets via immutable paths

fix: moving through search history shows no results

build: re-enable asset reloading in dev

feat: make security headers stricter

feat: enable sub-resource integrity for assets

refactor: extract default config to own file

docs: move and clarify (achieved!) relevancy goal to "Goals"

fix: nil pointer exception when (re-)indexing

chore: release 0.1.5

build: add command to generate changelog

feat: improve search relevance by boosting keyword matches

fix: missing alternate forms of packages with the same name

e.g. for git: git, gitFull, gitMinimal, etc..

fix: incorrect (switched) behaviour of --update/--replace

fix: re-enable exit --update/--replace

regression after extracting the command-line handling to
cmd/searchix-web

chore: update dependencies

build: add task to update dependencies

build: fix flake error

feat: support cleartext HTTP/2 (i.e. from a reverse proxy)

refactor: don't log request Host, it's irrelevant to this server

build: just use nix version as internal version

build: enable non-flake import

refactor: split server cmd and module

It should now be possible to run the server from inside another go
application by importing the main module and running its Start() function

fix: check for nil pointers in NewMux

chore: release v0.1.4

refactor: use a single value for per-source import timeouts

feat: stream files directly from fetcher to importer

Use IndexMeta to store the information relevant to making conditional
updates in future runs.

fix: abort import of source on batch processing errors

refactor: clarify log message when starting import

fix(web): fetch host from request, not header, where it's expected

docs: expand readme, move dev docs to subfolder

docs: document how to run an instance

build: fix pre-commit check exclude/ignore patterns

fix(nix): generate default baseURL from listen address and port

refactor(nix): rename module attribute to allow for other modules

build: narrow overly-broad linting exclusions

I started this project using `niv`, which creates `nix/sources.nix`,
which is all I really needed to exclude as it's generated by niv.

Now there's generated code only in `nix/overlays`, so the rest of the
files under `nix/` should be formatted by nixpkgs-fmt and analysed by
statix and deadnix.

fix: make fetcher check on creation that it supports Source.Importer

fix(importer): abort import of current source in case of error

The switch to logger.Error is to make clear that the operation is
aborted, whereas the remaining warning (fetching the git revision) is
not so important so it remains a warning and does not skip the current
iteration

perf: make channel-nixpkgs importer only fetch necessary files

i.e. don't fetch options.json when Source.Type = "packages"

refactor: make repository URLs by Repo.Type (currently no change)

"currently no change" because only GitHub repos are supported, but
more ready to support other types

refactor: use enum for Repository.Type

refactor: make declaration/definition URLs from Source.Repo

build: fix `prepare` task not working after flake migration

feat: add --update flag to force-update index and exit

style: use double-dash arguments to wgo

docs: clarify help text for --replace

docs: add comments to default configuration

feat(config): use enums' string representations to print defaults

docs: add link to default config file in nix module

build: check defaults.toml wasn't modified pre-commit

fix: nix cannot read date/time TOML values from default config

feat(config): print durations and URLs with human values

build: store default configuration in repo and ensure freshness

feat: show default configuration with --print-default-config

fix: duplicate error logging on config parse error

chore: release 0.1.3

feat(ui): enable resetting form by clicking current source link

feat: render markdown in packages' LongDescription

feat: use attribute instead of name for packages

fix(css): pagination styles not applied

fix(nix): missing version variable in build

chore(release): release 0.1.0

build: enable versioning of package

chore: add convco configuration

feat: display more information about packages

fix: render errors as HTML given correct content-type

fix: missing last page of search results

fix: store only data from current entity in index

fix: nil pointer when updating all sources

feat: error on non-matching configuration entries

refactor: reduce debug logging

docs: update default sources in nixos module

feat: automatically add/remove sources to/from index

fix: don't panic when re-indexing if fetcher did not update

build: make `just dev` watch config.toml

feat: support searching packages

feat(index): enable storage of package data

fix(css): support browsers without css nesting

feat: improve accessibility and browser behaviour

feat: exit after rebuilding index if `--replace` is specified

refactor: export functions for parsing enum types

perf: flush index batch in groups of 10000

fix(logging): track whether headers are written properly

feat: decode brotli-compressed files

refactor: move general results template code into results template

refactor: separate fetch and import logic

fix: rebuild index when --replace is specified

refactor: remove unnecessary and incorrect bleve config option

feat: extract and store package set (e.g. nodePackages)

refactor: use interface to unify Options and Packages

refactor: remove unused interface

feat(importer): process packages from nixpkgs package.json

refactor: remove toml special cases

refactor(importer): move functions to utils file

feat: add caching header (except on error pages)

feat: monitor import jobs via sentry

fix: bad git conflict resolution

feat: add browser search engines via opensearch description

https://developer.mozilla.org/en-US/docs/Web/OpenSearch

style: use consistent naming for config value

feat: print clickable URL on startup in dev mode

refactor: remove log prefix

refactor: default to development Environment outside nix module

feat: render HTML error pages

refactor(templates): render partials with less hackiness

build: remove duplicate debug log on dev file watcher update

perf: move scripts to head to enable fetching in parallel

build: bake git hash into build for headers and --version

build: use a nixpkgs-approved version scheme

build: improve --help output

fix(flake): remove overlay export

feat(module): add default for Web.Environment

fix(importer): don't create duplicate nested directories

feat: add DownloadOptions importer (fetches pre-built options.json)

feat: log on startup if index doesn't exist

refactor(importer): add context and reduce log level

feat: rely on system to show log times in production

fix(css): barely-visible text in dark mode

fix(module): include nix in path

style: make port a string for easier nix integration

fix(config): show read URL on parse failure

refactor: combine import and web server into one binary

build: fix wrong arguments in justfile

refactor(config): simplify configuration

commit 8225dbdb692c99b39dcafe2e5ec6ddc4daf08fb5
Author: Alan Pearce <alan@alanpearce.eu>
Date:   Mon May 13 19:18:26 2024 +0200

    refactor: consolidate configuration to reduce command-line options

commit 5616d4c5a9bc6c0c14f744f812fa6609f859dc34
Author: Alan Pearce <alan@alanpearce.eu>
Date:   Mon May 13 17:41:58 2024 +0200

    refactor: move config file parsing to program entry points

refactor: move Source/Repository types to config package

fix(module): allow setting sentry DSN

build: workaround nix weirdness

build: export nixos module and overlay

build: switch to flakes

feat: allow adding extra HTML to the body via configuration

feat: enable using channel URLs if unable to resolve via NIX_PATH

build: limit files passed to nix

fix: don't panic on TOML decode error

refactor: move Source type

fix: linking to wrong declaration URLs

feat: nix module

feat: clicking another source re-uses query

refactor: merge search and results pages

refactor: extract index deletion function

fix(import): don't remove index if there are errors

now that we can update the index, this is no longer reasonable

build: add extra command to index without replacing existing index

feat(js): save and restore input state with history

fix(js): make reload after error execute attempted search

refactor(js): use async/await, it's 2024

build: optimise and fix handing of csp headers in dev mode

fix: js error when no results found

refactor: split http.Server and http.ServeMux creation

build: make `just dev` reload on go file changes

refactor: don't explain searches

fix: missing menu on search pages

style: remove extraneous <p> tags around option descriptions

fix: no results for home-manager search

build: ignore local configuration file

fix(config): add default channel name for home-manager

feat: version search index mapping schema and warn if outdated

refactor: move sources input data to $data_root/sources

refactor: deduce index path automatically from config.DataPath

refactor: split up NewIndexer function

build: enable building with nix with embedded frontend

build: optionally embed frontend files

style: highlight current search menu item

fix: missing menu on search page

style: use title-cased source names

docs: change links to point to sourcehut

refactor: clean up flags and environment variables

build: add reindex command

build: pass config file to dev command

fix: pointer type mismatch

feat: dynamically link to sources

feat: format server listen address as URL

refactor: pass Source object to templates

style: rename Option.Option to Option.Name

style: reformat with golines and enforce on commit

feat: enable index path to be configured

feat(importer): make timeout configurable via command-line

feat(importer): add --replace flag to overwrite existing index

feat: allow configuration of log level

feat: enable setting config file path via command line

docs: embed defaults in app

refactor: make config entries kebab-case

feat: import sources from configuration in go code and index options

feat: make configuration optional

fix(js): pushstate also on errors, so that the user can refresh

refactor: enable access to bleve DocumentMatch structs

refactor: reduce new variables when looping over hits

refactor: store searched documents directly in the search index

refactor: log index loading progress at startup

feat: improve search by using a custom tokeniser for option names

fix(js): null value error for pagination

fix: incorrect default/example values stored after processing

style: fix overflowing <pre> tags

fix: incorrect data shown for default and example option keys

feat: paginate search results

perf: create search indexes in parallel with server startup

feat: search multiple sources

style: show pages for different sources

style: make page wider and header shorter

style: add more text to homepage and footer

style: split homepage and search page

feat: search one set of options

build: workaround wgo CPU usage

refactor: rely on html/template functionality more

refactor: extract templating into own file

refactor: make data path (partially) configurable

fix: panic on importing darwin options

feat: link to exact commits in NixOS/nixpkgs

feat: render markdown examples

build: fetch darwin and home-manager options

feat: link to nixpkgs on github for declarations

fix: use options exposed via user's <nixpkgs> channel instead of ours

feat: auto-linkify URLs in markdown

style: increase contrast

style: remove border-radius

style: improve layout and formatting of code samples

style: add a footer

feat: render markdown in option descriptions

refactor: only reload templates when such files are updated

style: remove unused variable

style: add a small gap between search bar and button

fix(css): first grid row too tall

feat: browser history management

chore: watch all frontend files

feat: show metadata in grid

feat: frontend search implementation

chore: remove unnecessary shell variable

chore: use prettier with plugin to format go templates

feat: create nixos options.json importer

feat: add basic input frontend

chore: revert to blocklist-based gitignore

feat: add css

feat: watch and live reload in development

feat: serve a very basic html template

chore: initial commit