[Bug 20664] New: Optimize retrieval of biblio and item data

classic Classic list List threaded Threaded
52 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] New: Optimize retrieval of biblio and item data

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

            Bug ID: 20664
           Summary: Optimize retrieval of biblio and item data
 Change sponsored?: ---
           Product: Koha
           Version: master
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5 - low
         Component: Architecture, internals, and plumbing
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]

There are several inefficiencies in the current process of retrieving biblios
and embedding items in the MARC data. These affect especially batch processes
such as exporting of records and the OAI-PMH provider.

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|[hidden email]-commun |[hidden email]
                   |ity.org                     |

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #1 from Ere Maijala <[hidden email]> ---
Created attachment 74873
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74873&action=edit
Bug 20664: Optimize retrieval of biblio and item data

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are three
ways this is accomplished:

1.) Use state variables to hold prepared SQL statements so that the preparation
is done only once.
2.) Create optimized method for fetching item fields for MARC embedding.
3.) Use the cache service more and where repeated calls are made.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #2 from Ere Maijala <[hidden email]> ---
These changes improve performance also for other processes that call e.g.
GetMarcBiblio repeatedly.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #3 from Ere Maijala <[hidden email]> ---
My test results show and improvement from about 30 seconds to about 10 seconds
with the example export in the test plan.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Frédéric Demians <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #74873|0                           |1
        is obsolete|                            |

--- Comment #4 from Ere Maijala <[hidden email]> ---
Created attachment 75968
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=75968&action=edit
Bug 20664: Optimize retrieval of biblio and item data

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are three
ways this is accomplished:

1.) Use state variables to hold prepared SQL statements so that the preparation
is done only once.
2.) Create optimized method for fetching item fields for MARC embedding.
3.) Use the cache service more and where repeated calls are made.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #5 from Ere Maijala <[hidden email]> ---
Latest version fixes a mistake in the itemnumber check. Also rebased on master.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Lari Taskula <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Lari Taskula <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Failed QA

--- Comment #6 from Lari Taskula <[hidden email]> ---
Fails t/db_dependent/Items.t

    #   Failed test 'For OPAC, the pref OpacHiddenItems should have been take
into account. Only items with homebranch ne CPL should have been embedded'
    #   at t/db_dependent/Items.t line 631.
    #          got: '8'
    #     expected: '2'

    #   Failed test 'For OPAC, If all items are hidden, no item should have
been embedded'
    #   at t/db_dependent/Items.t line 642.
    #          got: '8'
    #     expected: '0'
    # Looks like you failed 2 tests of 7.

#   Failed test 'C4::Biblio::EmbedItemsInMarcBiblio'
#   at t/db_dependent/Items.t line 649.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.
Use of uninitialized value in string ne at /home/koha/kohaclone/C4/Items.pm
line 1729.

    #   Failed test 'items.barcode is not mapped anymore, so the DB column has
not been updated'
    #   at t/db_dependent/Items.t line 726.
    #          got: undef
    #     expected: 'a barcode'
    # Looks like you failed 1 test of 4.
t/db_dependent/Items.t .. 11/14
#   Failed test 'C4::Items::_build_default_values_for_mod_marc'
#   at t/db_dependent/Items.t line 731.
# Looks like you failed 2 tests of 14.
t/db_dependent/Items.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/14 subtests

Test Summary Report
-------------------
t/db_dependent/Items.t (Wstat: 512 Tests: 14 Failed: 2)
  Failed tests:  10-11
  Non-zero exit status: 2

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #7 from Lari Taskula <[hidden email]> ---
(In reply to Lari Taskula from comment #6)
> Fails t/db_dependent/Items.t
This is due to caching parsed YAML from OpacHiddenItems. Cache does not cleared
between tests when OpacHiddenItems changes.

Instead of additionally caching parsed YAML of this system preference, would it
be better for Koha to handle YAML system preferences as parsed by default? This
would require some changes to Koha core but I guess it would benefit all
features using system preferences represented in YAML.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #8 from Ere Maijala <[hidden email]> ---
Thanks for the feedback. I think I'll spin that off as a separate bug and
remove the caching thing here.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Lari Taskula <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.koha-community
                   |                            |.org/bugzilla3/show_bug.cgi
                   |                            |?id=20930

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #75968|0                           |1
        is obsolete|                            |

--- Comment #9 from Ere Maijala <[hidden email]> ---
Created attachment 76045
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76045&action=edit
Bug 20664: Optimize retrieval of biblio and item data

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are three
ways this is accomplished:

1.) Use state variables to hold prepared SQL statements so that the preparation
is done only once.
2.) Create optimized method for fetching item fields for MARC embedding.
3.) Use the cache service more and where repeated calls are made.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Failed QA                   |Needs Signoff

--- Comment #10 from Ere Maijala <[hidden email]> ---
Latest patch removes the custom YAML caching and fixes tests to properly flush
all caches after changing the settings. Additionally fixes warnings surfaced by
the use of Modern::Perl.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #76045|0                           |1
        is obsolete|                            |

--- Comment #11 from Ere Maijala <[hidden email]> ---
Created attachment 76046
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76046&action=edit
Bug 20664: Optimize retrieval of biblio and item data

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are three
ways this is accomplished:

1.) Use state variables to hold prepared SQL statements so that the preparation
is done only once.
2.) Create optimized method for fetching item fields for MARC embedding.
3.) Use the cache service more and where repeated calls are made.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #76046|0                           |1
        is obsolete|                            |

--- Comment #12 from Ere Maijala <[hidden email]> ---
Comment on attachment 76046
  --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76046
Bug 20664: Optimize retrieval of biblio and item data

I need to come up with a better mechanism for avoiding unnecessary query
preparation since statements cached in state variables may become stale.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.koha-community
                   |                            |.org/bugzilla3/show_bug.cgi
                   |                            |?id=19365

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Owen Leonard <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |ASSIGNED

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #13 from Ere Maijala <[hidden email]> ---
Created attachment 76669
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76669&action=edit
Bug 20664: Optimize retrieval of biblio and item data

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are two
changes to accomplish this:

1.) Create optimized method for fetching item fields for MARC embedding.
2.) Use the cache service more and where repeated calls are made.

Also the now-unnecessary frameworkcode parameter to _strip_item_fields() has
been removed.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #14 from Ere Maijala <[hidden email]> ---
The latest patch is more simple and avoids any attempts at caching prepared
statements. Fortunately MySQL prepares quickly, so the performance improvement
is still good.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |ASSIGNED

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.koha-community
                   |                            |.org/bugzilla3/show_bug.cgi
                   |                            |?id=21006

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]-c
                   |                            |ommunity.org

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Ere Maijala <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #76669|0                           |1
        is obsolete|                            |

--- Comment #15 from Ere Maijala <[hidden email]> ---
Created attachment 78868
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78868&action=edit
Bug 20664: Optimize retrieval of biblio and item data

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

Optimizes embedding of item data in MARC and fixes several bottlenecks
encountered while profiling OAI-PMH and exporting of records. There are two
changes to accomplish this:

1.) Create optimized method for fetching item fields for MARC embedding.
2.) Use the cache service more and where repeated calls are made.

Also the now-unnecessary frameworkcode parameter to _strip_item_fields() has
been removed.

Test plan:

1.) Before applying the patch, time an export_records.pl run for a set of
biblios that also have items. Run it a couple of times to account for initial
slowness and possible fluctuations. For example:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename unoptimized.xml

2.) Apply the patch.

3.) Time the export process again with a different output file:

time misc/export_records.pl --record-type bibs --starting_biblionumber 960000
--ending_biblionumber 965000 --format xml --filename optimized.xml

4.) Verify that the optimized process is faster.

5.) Compare the resulting export files to make sure they're identical:

diff -u unoptimized.xml optimized.xml

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #16 from Ere Maijala <[hidden email]> ---
Created attachment 78869
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78869&action=edit
Bug 20664: Add unit tests for GetMarcItem

To test:
prove -v t/db_dependent/Items.t

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20664] Optimize retrieval of biblio and item data

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664

--- Comment #17 from Ere Maijala <[hidden email]> ---
Created attachment 78870
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78870&action=edit
Bug 20664: Unit tests for GetMarcItemFields

To test:
prove -v t/db_dependent/Items.t

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
12