Browse Source

added ICUFoldingFilter so Ökonomie sorts with Okonomie

main
Paul Pound 2 weeks ago
parent
commit
4b55eccea4
  1. 21
      README.md
  2. 17
      src/EventSubscriber/SolrConfigSetSubscriber.php

21
README.md

@ -113,10 +113,10 @@ field types do not have to precede the fields that use them.
<fieldType name="string_sort_ignore_articles" class="solr.TextField" sortMissingLast="true" omitNorms="true">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="^[^\p{L}\p{N}]+" replacement="" replace="all"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="^(?i)(a\s+|an\s+|the\s+)" replacement="" replace="all"/>
<filter class="solr.PatternReplaceFilterFactory" pattern="^[^\p{L}\p{N}]+" replacement="" replace="all"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
</fieldType>
@ -135,7 +135,13 @@ zip:
Keep the comment markers — they are what makes the block findable and cleanly
removable later.
Both blocks load unchanged on Solr 8.11 and 9.x.
Both blocks load unchanged on Solr 8.11 and 9.x. `ICUFoldingFilterFactory`
needs the ICU analysis jar, but any core running the stock `search_api_solr`
config set already has it — every `collated_*` field type is a
`solr.ICUCollationField` from that same jar, and the core would not load at
all without it. On Solr 8 the config set loads it through the
`contrib/analysis-extras` `<lib>` directives; on Solr 9 it comes from
`SOLR_MODULES` including `analysis-extras`.
Reload the core:
@ -162,8 +168,15 @@ search view.
## Verifying
At `/solr/#/<core>/analysis`, field type `string_sort_ignore_articles` with
input `The !Matrix` should produce the single token `matrix`.
At `/solr/#/<core>/analysis`, field type `string_sort_ignore_articles` should
reduce each of these to a single token:
| Input | Token | Checks |
| --- | --- | --- |
| `The !Matrix` | `matrix` | article and punctuation both stripped |
| `A Tale of Two Cities` | `tale of two cities` | one token, not one per word |
| `Theory of Everything` | `theory of everything` | a word starting with an article is left alone |
| `Ökonomie` | `okonomie` | diacritics folded, so it sorts under O |
The field itself can be confirmed with the Schema API:

17
src/EventSubscriber/SolrConfigSetSubscriber.php

@ -94,15 +94,22 @@ class SolrConfigSetSubscriber implements EventSubscriberInterface {
<analyzer>
<!-- 1. Keep the entire string as a single token. -->
<tokenizer class="solr.KeywordTokenizerFactory"/>
<!-- 2. Strip leading punctuation, symbols and spaces.
<!-- 2. NFKC normalise, case fold and strip diacritics, so 'Matrix',
'matrix' and 'Mátrix' sort together instead of the accented forms
landing after every ASCII letter. This also normalises a no-break
space to a plain one, so step 4 can see the article boundary.
Needs the ICU analysis jar, which is already loaded: the stock
search_api_solr schema types every collated_* field as
solr.ICUCollationField from the same jar. -->
<filter class="solr.ICUFoldingFilterFactory"/>
<!-- 3. Strip leading punctuation, symbols and spaces.
[^\p{L}\p{N}] is "anything that is NOT a letter or a number". -->
<filter class="solr.PatternReplaceFilterFactory" pattern="^[^\p{L}\p{N}]+" replacement="" replace="all"/>
<!-- 3. Remove a leading 'A ', 'An ' or 'The ' (case-insensitive). -->
<!-- 4. Remove a leading 'A ', 'An ' or 'The '. Step 2 has already lower
cased the token; (?i) is kept so this still holds on its own. -->
<filter class="solr.PatternReplaceFilterFactory" pattern="^(?i)(a\s+|an\s+|the\s+)" replacement="" replace="all"/>
<!-- 4. Strip leading punctuation again, to catch "The !Matrix". -->
<!-- 5. Strip leading punctuation again, to catch "The !Matrix". -->
<filter class="solr.PatternReplaceFilterFactory" pattern="^[^\p{L}\p{N}]+" replacement="" replace="all"/>
<!-- 5. Lowercase so 'Matrix' and 'matrix' sort together. -->
<filter class="solr.LowerCaseFilterFactory"/>
<!-- 6. Trim any accidental leading/trailing whitespace. -->
<filter class="solr.TrimFilterFactory"/>
</analyzer>

Loading…
Cancel
Save