[Bug 20073] New: Move Elasticsearch settings to configuration files

classic Classic list List threaded Threaded
77 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] New: Move Elasticsearch settings to configuration files

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

            Bug ID: 20073
           Summary: Move Elasticsearch settings to configuration files
 Change sponsored?: ---
           Product: Koha
           Version: unspecified
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5 - low
         Component: Searching - Elasticsearch
          Assignee: [hidden email]
          Reporter: [hidden email]

There are a lot of settings in the code making customization difficult. These
need to be moved to configuration files that can be overridden as necessary.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|[hidden email]-commun |[hidden email]
                   |ity.org                     |
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #1 from Ere Maijala <[hidden email]> ---
Created attachment 70827
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70827&action=edit
Bug 20073: Move Elasticsearch configs to yaml files and improve the default
settings.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #2 from Ere Maijala <[hidden email]> ---
The patch contains the following changes:
- Index settings moved from code to
etc/searchengine/elasticsearch/index_config.yaml. An alternative can be
specified in koha-conf.xml.
- Field settings moved from code to
etc/searchengine/elasticsearch/field_config.yaml. An alternative can be
specified in koha-conf.xml.
- Default settings have been improved to remove punctuation from phrases used
for sorting etc.
- State variables are used for storing configuration to avoid parsing it
multiple times.
- A possibility to reset the fields too has been added to the reset operation
of mappings administration.
- mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch.
- An stdno field type has been added for standard identifiers.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #70827|0                           |1
        is obsolete|                            |

--- Comment #3 from Ere Maijala <[hidden email]> ---
Comment on attachment 70827
  --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70827
Bug 20073: Move Elasticsearch configs to yaml files and improve the default
settings.

Oops, I omitted the required DB and UI changes.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #4 from Ere Maijala <[hidden email]> ---
Created attachment 70836
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70836&action=edit
Bug 20073: Move Elasticsearch configs to yaml files and improve the default
settings.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #5 from Ere Maijala <[hidden email]> ---
Created attachment 70837
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70837&action=edit
Bug 20073: Add database and mappings UI support for isbn and stdno field types.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #6 from Ere Maijala <[hidden email]> ---
Ok, so I'm fumbling a bit with git-bz here, but the current patches should do
it.  Note that the isbn field type has been added in anticipation of a future
patch that adds a bit of special handling for ISBNs so that they're indexed in
all forms.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |20244


Referenced Bugs:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20244
[Bug 20244] Elasticsearch - Indexing improvements
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Fridolin SOMERS <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
                   |                            |m

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nick Clemens <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nicolas Legrand <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #7 from Nicolas Legrand <[hidden email]> ---
How could I test it?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nick Clemens <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |In Discussion

--- Comment #8 from Nick Clemens <[hidden email]> ---
There are some stray sections related to payment systems left in the
/etc/koha-conf.xml file and some whitespace errors.


This seems to work otherwise, need a test plan, it might be easier to separate
the move to files and the updates to mappings to two separate bugs for simpler
tests plans, or can be comprehensive here.

Should this depend on the improvement to mappings reset provided else where?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #9 from Ere Maijala <[hidden email]> ---
Thanks for the comments. I'll work on adding a test plan and rebasing the
patches. This doesn't necessarily require Bug 20248, though testing is more
difficult without it.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Séverine Queune <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|In Discussion               |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #70836|0                           |1
        is obsolete|                            |
  Attachment #70837|0                           |1
        is obsolete|                            |

--- Comment #10 from Ere Maijala <[hidden email]> ---
Created attachment 73054
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73054&action=edit
Bug 20073 - Move Elasticsearch configs to yaml files and improve the default
settings.

Improvements:
1) Index settings moved from code to
etc/searchengine/elasticsearch/index_config.yaml. An alternative can be
specified in koha-conf.xml.
2) Field settings moved from code to
etc/searchengine/elasticsearch/field_config.yaml. An alternative can be
specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch. An alternative can be specified in
koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used
for sorting etc.
5) State variables are used for storing configuration to avoid parsing it
multiple times.
6) A possibility to reset the fields too has been added to the reset operation
of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the
mappings properly, including ISBN and other standard number fields.
5) Index some records
6) Verify that you can find the records
7) Put
<elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to
koha-conf.xml
8) Verify that
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make
elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nicolas Legrand <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Failed QA

--- Comment #11 from Nicolas Legrand <[hidden email]> ---
Hey Ere,

my install (Unimarc, ES 5.2.8) failed at step 6.

I can index and find books on master. With 20073 applied The index seems to
work, I don't have deprecated messages as on master. But I can't find anything,
my index is empty:

$ curl http://localhost:9200/_cat/indices?v
health status index                  uuid                   pri rep docs.count
docs.deleted store.size pri.store.size
yellow open   koha_robin_biblios     oDy7ppLWTkS4uOeta1KijA   5   1          0
          0       810b           810b
yellow open   koha_robin_authorities TFRTgnFYTCusa9pTuTI35Q   5   1       3865
          0      6.9mb          6.9mb

On master, it is filled as it should:

$ curl http://localhost:9200/_cat/indices?v
health status index                  uuid                   pri rep docs.count
docs.deleted store.size pri.store.size
yellow open   koha_robin_authorities 0vd2mF0aTWCyvsKtQlrkhw   5   1       3865
          0      4.9mb          4.9mb
yellow open   koha_robin_biblios     hto7dbqNSSSN3Y4IG0FK0Q   5   1        856
          0       10mb           10mb

Or maybe I am missing something?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #12 from Ere Maijala <[hidden email]> ---
Hi Nicolas,

The test step 5 was probably quite bad. You should make sure to run
misc/search_tools/rebuild_elastic_search.pl long enough for it to commit at
least one set of records before it creates the index with the correct settings.
Otherwise indexing will quietly fail (this would be the subject of another
bug). So, something like

misc/search_tools/rebuild_elastic_search.pl -d -b -v -v

should do the trick if you make sure that you don't try to save any records in
the cataloguing UI before the first commit. Otherwise you'll probably find only
errors in /var/log/elasticsearch/elasticsearch.log. Bug 20248 contains some
improvements to the indexing script.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #13 from Nicolas Legrand <[hidden email]> ---
I have a lot of:

[2018-03-19T15:55:21,033][DEBUG][o.e.a.b.TransportShardBulkAction] [0aw4dLe]
[koha_robin_biblios][2] failed to execute bulk item (index) BulkShardRequest
[[ko
ha_robin_biblios][2]] containing [21] requests

I'm also on a dev install and not on a kohadevbox. I try to make and make
install again, to take in account the etc changes, but it didn't do any good...
Is is working for you?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Failed QA                   |Needs Signoff

--- Comment #14 from Ere Maijala <[hidden email]> ---
We did a bit of testing but couldn't quite figure the issue out yet. It's
working for me with MARC 21 and Unimarc. If anyone else is up to it, please
feel free to test and provide feedback.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #15 from Ere Maijala <[hidden email]> ---
Nicolas kindly provided me with test set of Unimarc records. I was able to
import and index them without issues. Perhaps the issue something specific to
ES version or something. I'm running 5.6.8.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #73054|0                           |1
        is obsolete|                            |

--- Comment #16 from Ere Maijala <[hidden email]> ---
Created attachment 73138
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73138&action=edit
Bug 20073 - Move Elasticsearch configs to yaml files and improve the default
settings.

Improvements:
1) Index settings moved from code to
etc/searchengine/elasticsearch/index_config.yaml. An alternative can be
specified in koha-conf.xml.
2) Field settings moved from code to
etc/searchengine/elasticsearch/field_config.yaml. An alternative can be
specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch. An alternative can be specified in
koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used
for sorting etc.
5) State variables are used for storing configuration to avoid parsing it
multiple times.
6) A possibility to reset the fields too has been added to the reset operation
of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the
mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with
misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put
<elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to
koha-conf.xml
8) Verify that
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make
elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

--- Comment #17 from Ere Maijala <[hidden email]> ---
I managed to reproduce the problem and found a stupid typo in
field_config.yaml. The latest patch should work.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nicolas Legrand <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Signed Off

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nicolas Legrand <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #73138|0                           |1
        is obsolete|                            |

--- Comment #18 from Nicolas Legrand <[hidden email]> ---
Created attachment 73144
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73144&action=edit
Bug 20073 - Move Elasticsearch configs to yaml files and improve the default
settings.

Improvements:
1) Index settings moved from code to
etc/searchengine/elasticsearch/index_config.yaml. An alternative can be
specified in koha-conf.xml.
2) Field settings moved from code to
etc/searchengine/elasticsearch/field_config.yaml. An alternative can be
specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch. An alternative can be specified in
koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used
for sorting etc.
5) State variables are used for storing configuration to avoid parsing it
multiple times.
6) A possibility to reset the fields too has been added to the reset operation
of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the
mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with
misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put
<elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to
koha-conf.xml
8) Verify that
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make
elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

Signed-off-by: Nicolas Legrand <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nick Clemens <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Signed Off                  |Passed QA

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20073] Move Elasticsearch settings to configuration files

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20073

Nick Clemens <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #73144|0                           |1
        is obsolete|                            |

--- Comment #19 from Nick Clemens <[hidden email]> ---
Created attachment 73383
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73383&action=edit
Bug 20073 - Move Elasticsearch configs to yaml files and improve the default
settings.

Improvements:
1) Index settings moved from code to
etc/searchengine/elasticsearch/index_config.yaml. An alternative can be
specified in koha-conf.xml.
2) Field settings moved from code to
etc/searchengine/elasticsearch/field_config.yaml. An alternative can be
specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch. An alternative can be specified in
koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used
for sorting etc.
5) State variables are used for storing configuration to avoid parsing it
multiple times.
6) A possibility to reset the fields too has been added to the reset operation
of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to
etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the
mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with
misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put
<elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to
koha-conf.xml
8) Verify that
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make
elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to
admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

Signed-off-by: Nicolas Legrand <[hidden email]>

Signed-off-by: Nick Clemens <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
123