Code search now matches case-insensitive literal substrings like GitHub,
in every script. Word tokenizers can't do this: the ES pattern dropped
every non-ASCII word, and the `standard` tokenizer keeps `console.log`
as one token.
- Elasticsearch indexes positional trigrams, plus 1-2 grams for short
terms
- Bleve indexes distinct 1-3 grams as a candidate filter and verifies
candidates against the content before paging, because phrase queries
over trigrams are quadratic in repeated grams
- Snippet offsets come from the original content, so `<em>` in sources
and width-changing lowercase no longer misplace them
- `search_mode=fuzzy` requests behave like Words, the UI never offered
fuzzy for code search
- The ES code indexer test now runs in CI, it was skipped before
There are no UI changes. Both code indexes rebuild after upgrade.
Fixes: https://github.com/go-gitea/gitea/issues/40027
Fixes: https://github.com/go-gitea/gitea/issues/16158
Fixes: https://github.com/go-gitea/gitea/issues/21359
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The bleve path token filter added in
https://github.com/go-gitea/gitea/pull/32210 never generated the full
path of a file, so searching a file by its path (e.g. `potato/ham`)
found nothing. It also gave the path tokens made-up offsets instead of
their position in the path.
The filter is replaced by a tokenizer that emits the raw path suffixes
starting at each segment and word (`potato/ham.md`, `ham.md`). Paths
below the root now match too, as do names containing `-` or spaces and
dotfiles. An exact file name also ranks above files inside a directory
with the same name.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
