feat: stream files directly from fetcher to importer

Use IndexMeta to store the information relevant to making conditional
updates in future runs.

fix: abort import of source on batch processing errors

refactor: clarify log message when starting import

fix(web): fetch host from request, not header, where it's expected

docs: expand readme, move dev docs to subfolder

docs: document how to run an instance

build: fix pre-commit check exclude/ignore patterns

fix(nix): generate default baseURL from listen address and port

refactor(nix): rename module attribute to allow for other modules

build: narrow overly-broad linting exclusions

I started this project using `niv`, which creates `nix/sources.nix`,
which is all I really needed to exclude as it's generated by niv.

Now there's generated code only in `nix/overlays`, so the rest of the
files under `nix/` should be formatted by nixpkgs-fmt and analysed by
statix and deadnix.

fix: make fetcher check on creation that it supports Source.Importer

fix(importer): abort import of current source in case of error

The switch to logger.Error is to make clear that the operation is
aborted, whereas the remaining warning (fetching the git revision) is
not so important so it remains a warning and does not skip the current
iteration

perf: make channel-nixpkgs importer only fetch necessary files

i.e. don't fetch options.json when Source.Type = "packages"

refactor: make repository URLs by Repo.Type (currently no change)

"currently no change" because only GitHub repos are supported, but
more ready to support other types

refactor: use enum for Repository.Type

refactor: make declaration/definition URLs from Source.Repo

build: fix `prepare` task not working after flake migration

feat: add --update flag to force-update index and exit

style: use double-dash arguments to wgo

docs: clarify help text for --replace

docs: add comments to default configuration

feat(config): use enums' string representations to print defaults

docs: add link to default config file in nix module

build: check defaults.toml wasn't modified pre-commit

fix: nix cannot read date/time TOML values from default config

feat(config): print durations and URLs with human values

build: store default configuration in repo and ensure freshness

feat: show default configuration with --print-default-config

fix: duplicate error logging on config parse error

chore: release 0.1.3

feat(ui): enable resetting form by clicking current source link

feat: render markdown in packages' LongDescription

feat: use attribute instead of name for packages

fix(css): pagination styles not applied

fix(nix): missing version variable in build

chore(release): release 0.1.0

build: enable versioning of package

chore: add convco configuration

feat: display more information about packages

fix: render errors as HTML given correct content-type

fix: missing last page of search results

fix: store only data from current entity in index

fix: nil pointer when updating all sources

feat: error on non-matching configuration entries

refactor: reduce debug logging

docs: update default sources in nixos module

feat: automatically add/remove sources to/from index

fix: don't panic when re-indexing if fetcher did not update

build: make `just dev` watch config.toml

feat: support searching packages

feat(index): enable storage of package data

fix(css): support browsers without css nesting

feat: improve accessibility and browser behaviour

feat: exit after rebuilding index if `--replace` is specified

refactor: export functions for parsing enum types

perf: flush index batch in groups of 10000

fix(logging): track whether headers are written properly

feat: decode brotli-compressed files

refactor: move general results template code into results template

refactor: separate fetch and import logic

fix: rebuild index when --replace is specified

refactor: remove unnecessary and incorrect bleve config option

feat: extract and store package set (e.g. nodePackages)

refactor: use interface to unify Options and Packages

refactor: remove unused interface

feat(importer): process packages from nixpkgs package.json

refactor: remove toml special cases

refactor(importer): move functions to utils file

feat: add caching header (except on error pages)

feat: monitor import jobs via sentry

fix: bad git conflict resolution

feat: add browser search engines via opensearch description

https://developer.mozilla.org/en-US/docs/Web/OpenSearch

style: use consistent naming for config value

feat: print clickable URL on startup in dev mode

refactor: remove log prefix

refactor: default to development Environment outside nix module

feat: render HTML error pages

refactor(templates): render partials with less hackiness

build: remove duplicate debug log on dev file watcher update

perf: move scripts to head to enable fetching in parallel

build: bake git hash into build for headers and --version

build: use a nixpkgs-approved version scheme

build: improve --help output

fix(flake): remove overlay export

feat(module): add default for Web.Environment

fix(importer): don't create duplicate nested directories

feat: add DownloadOptions importer (fetches pre-built options.json)

feat: log on startup if index doesn't exist

refactor(importer): add context and reduce log level

feat: rely on system to show log times in production

fix(css): barely-visible text in dark mode

fix(module): include nix in path

style: make port a string for easier nix integration

fix(config): show read URL on parse failure

refactor: combine import and web server into one binary

build: fix wrong arguments in justfile

refactor(config): simplify configuration

commit 8225dbdb692c99b39dcafe2e5ec6ddc4daf08fb5
Author: Alan Pearce <alan@alanpearce.eu>
Date:   Mon May 13 19:18:26 2024 +0200

    refactor: consolidate configuration to reduce command-line options

commit 5616d4c5a9bc6c0c14f744f812fa6609f859dc34
Author: Alan Pearce <alan@alanpearce.eu>
Date:   Mon May 13 17:41:58 2024 +0200

    refactor: move config file parsing to program entry points

refactor: move Source/Repository types to config package

fix(module): allow setting sentry DSN

build: workaround nix weirdness

build: export nixos module and overlay

build: switch to flakes

feat: allow adding extra HTML to the body via configuration

feat: enable using channel URLs if unable to resolve via NIX_PATH

build: limit files passed to nix

fix: don't panic on TOML decode error

refactor: move Source type

fix: linking to wrong declaration URLs

feat: nix module

feat: clicking another source re-uses query

refactor: merge search and results pages

refactor: extract index deletion function

fix(import): don't remove index if there are errors

now that we can update the index, this is no longer reasonable

build: add extra command to index without replacing existing index

feat(js): save and restore input state with history

fix(js): make reload after error execute attempted search

refactor(js): use async/await, it's 2024

build: optimise and fix handing of csp headers in dev mode

fix: js error when no results found

refactor: split http.Server and http.ServeMux creation

build: make `just dev` reload on go file changes

refactor: don't explain searches

fix: missing menu on search pages

style: remove extraneous <p> tags around option descriptions

fix: no results for home-manager search

build: ignore local configuration file

fix(config): add default channel name for home-manager

feat: version search index mapping schema and warn if outdated

refactor: move sources input data to $data_root/sources

refactor: deduce index path automatically from config.DataPath

refactor: split up NewIndexer function

build: enable building with nix with embedded frontend

build: optionally embed frontend files

style: highlight current search menu item

fix: missing menu on search page

style: use title-cased source names

docs: change links to point to sourcehut

refactor: clean up flags and environment variables

build: add reindex command

build: pass config file to dev command

fix: pointer type mismatch

feat: dynamically link to sources

feat: format server listen address as URL

refactor: pass Source object to templates

style: rename Option.Option to Option.Name

style: reformat with golines and enforce on commit

feat: enable index path to be configured

feat(importer): make timeout configurable via command-line

feat(importer): add --replace flag to overwrite existing index

feat: allow configuration of log level

feat: enable setting config file path via command line

docs: embed defaults in app

refactor: make config entries kebab-case

feat: import sources from configuration in go code and index options

feat: make configuration optional

fix(js): pushstate also on errors, so that the user can refresh

refactor: enable access to bleve DocumentMatch structs

refactor: reduce new variables when looping over hits

refactor: store searched documents directly in the search index

refactor: log index loading progress at startup

feat: improve search by using a custom tokeniser for option names

fix(js): null value error for pagination

fix: incorrect default/example values stored after processing

style: fix overflowing <pre> tags

fix: incorrect data shown for default and example option keys

feat: paginate search results

perf: create search indexes in parallel with server startup

feat: search multiple sources

style: show pages for different sources

style: make page wider and header shorter

style: add more text to homepage and footer

style: split homepage and search page

feat: search one set of options

build: workaround wgo CPU usage

refactor: rely on html/template functionality more

refactor: extract templating into own file

refactor: make data path (partially) configurable

fix: panic on importing darwin options

feat: link to exact commits in NixOS/nixpkgs

feat: render markdown examples

build: fetch darwin and home-manager options

feat: link to nixpkgs on github for declarations

fix: use options exposed via user's <nixpkgs> channel instead of ours

feat: auto-linkify URLs in markdown

style: increase contrast

style: remove border-radius

style: improve layout and formatting of code samples

style: add a footer

feat: render markdown in option descriptions

refactor: only reload templates when such files are updated

style: remove unused variable

style: add a small gap between search bar and button

fix(css): first grid row too tall

feat: browser history management

chore: watch all frontend files

feat: show metadata in grid

feat: frontend search implementation

chore: remove unnecessary shell variable

chore: use prettier with plugin to format go templates

feat: create nixos options.json importer

feat: add basic input frontend

chore: revert to blocklist-based gitignore

feat: add css

feat: watch and live reload in development

feat: serve a very basic html template

chore: initial commit