[Bug 17196] New: Move marcxml out of the biblioitems table

classic Classic list List threaded Threaded
158 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] New: Move marcxml out of the biblioitems table

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

            Bug ID: 17196
           Summary: Move marcxml out of the biblioitems table
 Change sponsored?: ---
           Product: Koha
           Version: master
          Hardware: All
                OS: All
            Status: ASSIGNED
          Severity: enhancement
          Priority: P5 - low
         Component: Architecture, internals, and plumbing
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |17197


Referenced Bugs:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17197
[Bug 17197] misc/batchupdateISBNs.pl is certainly no longer in use
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |10455


Referenced Bugs:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10455
[Bug 10455] remove redundant 'biblioitems.marc' field
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |17193


Referenced Bugs:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17193
[Bug 17193] C4::Search::SearchAcquisitions is not used
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #1 from Jonathan Druart <[hidden email]> ---
Created attachment 54910
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54910&action=edit
Bug 17196: Move marcxml out of the biblioitems table

Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html

There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
  SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY
publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats
(marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).

Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
    * misc/cronjobs/delete_records_via_leader.pl
    * misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result

Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #2 from Jonathan Druart <[hidden email]> ---
Created attachment 54911
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54911&action=edit
Bug 17196: Fix query builder for item search

The item search needs to join on biblio_metadata to allow search on
marcxml field

Test plan:
Launch complex item searches (using marc fields).

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #3 from Jonathan Druart <[hidden email]> ---
Created attachment 54912
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54912&action=edit
Bug 17196: Update misc/maintenance/sanitize_records.pl

Test plan:
Use the sanitize_records.pl maintenance script with the --auto-search
option

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #4 from Jonathan Druart <[hidden email]> ---
Created attachment 54913
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54913&action=edit
Bug 17196: Update switch_marc21_series_info.pl

Not sure if this script is still used, could someone confirm?

Test plan:
If you know how to test it, please do

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #5 from Jonathan Druart <[hidden email]> ---
Created attachment 54914
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54914&action=edit
Bug 17196: DB Changes

Remove biblioitems.marcxml and deletedbiblioitems.marcxml
and adds 2 table biblio_metadata and deletedbiblio_metadata

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #54914|0                           |1
        is obsolete|                            |

--- Comment #6 from Jonathan Druart <[hidden email]> ---
Created attachment 54918
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54918&action=edit
Bug 17196: DB Changes

Remove biblioitems.marcxml and deletedbiblioitems.marcxml
and adds 2 table biblio_metadata and deletedbiblio_metadata

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Bob Birchall <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Patch doesn't apply

--- Comment #7 from Mason James <[hidden email]> ---
Hi Jonathan

I attempted to test this patchset, but I had a big problem trying to 'git bz
apply' the  1st patch, on master (0254fd7)

0254fd7 Bug 17135 - DBRev 16.06.00.028

would you take another look at the patchset please?

---------------

mason@xen1:~/g/k/Koha$ git bz apply 17196
Bug 17196 - Move marcxml out of the biblioitems table

54910 - Bug 17196: Move marcxml out of the biblioitems table
54911 - Bug 17196: Fix query builder for item search
54912 - Bug 17196: Update misc/maintenance/sanitize_records.pl
54913 - Bug 17196: Update switch_marc21_series_info.pl
54918 - Bug 17196: DB Changes

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 17196: Move marcxml out of the biblioitems table
fatal: sha1 information is lacking or useless (C4/Biblio.pm).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 17196: Move marcxml out of the biblioitems table
The copy of the patch that failed is found in:
   /home/mason/g/k/Koha/.git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in
/tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-jCnYTS.patch

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Patch doesn't apply         |Needs Signoff

--- Comment #8 from Jonathan Druart <[hidden email]> ---
(In reply to Mason James from comment #7)
> Hi Jonathan
>
> I attempted to test this patchset, but I had a big problem trying to 'git bz
> apply' the  1st patch, on master (0254fd7)
>
> 0254fd7 Bug 17135 - DBRev 16.06.00.028
>
> would you take another look at the patchset please?

Everything is applying ok for me, please check the dependencies.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #9 from Owen Leonard <[hidden email]> ---
> > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > apply' the  1st patch, on master (0254fd7)

git bz didn't apply this successfully when using the auto-inclusion of
dependent patches, but I don't know why. Applying the patches manually works
without error.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #10 from Mason James <[hidden email]> ---
(In reply to Jonathan Druart from comment #8)

> (In reply to Mason James from comment #7)
> > Hi Jonathan
> >
> > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > apply' the  1st patch, on master (0254fd7)
> >
> > 0254fd7 Bug 17135 - DBRev 16.06.00.028
> >
> > would you take another look at the patchset please?
>
> Everything is applying ok for me, please check the dependencies.

afaik (In reply to Jonathan Druart from comment #8)

> (In reply to Mason James from comment #7)
> > Hi Jonathan
> >
> > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > apply' the  1st patch, on master (0254fd7)
> >
> > 0254fd7 Bug 17135 - DBRev 16.06.00.028
> >
> > would you take another look at the patchset please?
>
> Everything is applying ok for me, please check the dependencies.

i have all listed dependencies applied on commit 56da7be (10455 17193 17197)

the patch still fails for me

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Patch doesn't apply

--- Comment #11 from Mason James <[hidden email]> ---
(In reply to Mason James from comment #10)

> (In reply to Jonathan Druart from comment #8)
> > (In reply to Mason James from comment #7)
> > > Hi Jonathan
> > >
> > > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > > apply' the  1st patch, on master (0254fd7)
> > >
> > > 0254fd7 Bug 17135 - DBRev 16.06.00.028
> > >
> > > would you take another look at the patchset please?
> >
> > Everything is applying ok for me, please check the dependencies.
>
> afaik (In reply to Jonathan Druart from comment #8)
> > (In reply to Mason James from comment #7)
> > > Hi Jonathan
> > >
> > > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > > apply' the  1st patch, on master (0254fd7)
> > >
> > > 0254fd7 Bug 17135 - DBRev 16.06.00.028
> > >
> > > would you take another look at the patchset please?
> >
> > Everything is applying ok for me, please check the dependencies.
>
> i have all listed dependencies applied on commit 56da7be (10455 17193 17197)
>
> the patch still fails for me

here's the transcript...

mason@xen1:~/g/k/Koha$ git log --oneline'
56da7be Bug 17352 - Patron search type is hard coded to 'contain' in
circ/circulation.pl
5eb4b06 Bug 16273 - DBRev 16.06.00.031
3d4dc58 Bug 16273: Add the ability not to display the login details for newly
registered patrons
8d0b412 Bug 16273: Add the new pref PatronSelfRegistrationPrefillForm
d85e80b Bug 17301: Follow-up - Standardize headings

mason@xen1:~/g/k/Koha$ git bz apply 10455
Bug 10455 - remove redundant 'biblioitems.marc' field
55844 - Bug 10455: Remove biblioitems.marc - DB changes
55845 - Bug 10455: Remove biblioitems.marc
55846 - Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl
55847 - Bug 10455: Remove biblioitems.marc - update
delete_records_via_leader.pl
55848 - Bug 10455: (follow-up) Remove biblioitems.marc
55849 - Bug 10455: (follow-up 2) Remove biblioitems.marc
55850 - Bug 10455: (follow-up 3) Remove biblioitems.marc
55851 - Bug 10455: Remove MARC21_utf8_flag_fix.pl
55852 - Bug 10455: Fix tests
55853 - Bug 10455 (QA Followup)

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 10455: Remove biblioitems.marc - DB changes
Applying: Bug 10455: Remove biblioitems.marc
Applying: Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl
Applying: Bug 10455: Remove biblioitems.marc - update
delete_records_via_leader.pl
Applying: Bug 10455: (follow-up) Remove biblioitems.marc
Applying: Bug 10455: (follow-up 2) Remove biblioitems.marc
Applying: Bug 10455: (follow-up 3) Remove biblioitems.marc
Applying: Bug 10455: Remove MARC21_utf8_flag_fix.pl
Applying: Bug 10455: Fix tests
Applying: Bug 10455 (QA Followup)
mason@xen1:~/g/k/Koha$
mason@xen1:~/g/k/Koha$ git bz apply 17193
Bug 17193 - C4::Search::SearchAcquisitions is not used

54962 - [PASSED QA] Bug 17193: Remove C4::Search::SearchAcquisitions

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 17193: Remove C4::Search::SearchAcquisitions
Using index info to reconstruct a base tree...
M       C4/Search.pm
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
mason@xen1:~/g/k/Koha$ git bz apply 17197
Bug 17197 - misc/batchupdateISBNs.pl is certainly no longer in use

55478 - [PASSED QA] Bug 17197: Remove misc/batchupdateISBNs.pl

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 17197: Remove misc/batchupdateISBNs.pl
Using index info to reconstruct a base tree...
A       misc/batchupdateISBNs.pl
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
mason@xen1:~/g/k/Koha$

mason@xen1:~/g/k/Koha$ git log --oneline'
4f6b1ee Bug 10455 (QA Followup)
98af430 Bug 10455: Fix tests
a5e6f98 Bug 10455: Remove MARC21_utf8_flag_fix.pl
fbb280d Bug 10455: (follow-up 3) Remove biblioitems.marc
81a0fb2 Bug 10455: (follow-up 2) Remove biblioitems.marc
4e3520d Bug 10455: (follow-up) Remove biblioitems.marc
339d2bc Bug 10455: Remove biblioitems.marc - update
delete_records_via_leader.pl
d572063 Bug 10455: Remove biblioitems.marc - rm MARC21_parse_test.pl
dccc79a Bug 10455: Remove biblioitems.marc
558f6b1 Bug 10455: Remove biblioitems.marc - DB changes
56da7be Bug 17352 - Patron search type is hard coded to 'contain' in
circ/circulation.pl
5eb4b06 Bug 16273 - DBRev 16.06.00.031

mason@xen1:~/g/k/Koha$ git bz apply 17196
Bug 17196 - Move marcxml out of the biblioitems table

54910 - Bug 17196: Move marcxml out of the biblioitems table
54911 - Bug 17196: Fix query builder for item search
54912 - Bug 17196: Update misc/maintenance/sanitize_records.pl
54913 - Bug 17196: Update switch_marc21_series_info.pl
54918 - Bug 17196: DB Changes

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 17196: Move marcxml out of the biblioitems table
fatal: sha1 information is lacking or useless (C4/Biblio.pm).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 17196: Move marcxml out of the biblioitems table
The copy of the patch that failed is found in:
   /home/mason/g/k/Koha/.git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in
/tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-HBdRv1.patch

mason@xen1:~/g/k/Koha$ git apply
/tmp/Bug-17196-Move-marcxml-out-of-the-biblioitems-tabl-HBdRv1.patch
error: patch failed: C4/Biblio.pm:42
error: C4/Biblio.pm: patch does not apply
error: patch failed: C4/ILSDI/Services.pm:406
error: C4/ILSDI/Services.pm: patch does not apply
error: patch failed: misc/cronjobs/delete_records_via_leader.pl:68
error: misc/cronjobs/delete_records_via_leader.pl: patch does not apply

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #12 from Mason James <[hidden email]> ---
(In reply to Owen Leonard from comment #9)
> > > I attempted to test this patchset, but I had a big problem trying to 'git bz
> > > apply' the  1st patch, on master (0254fd7)
>
> git bz didn't apply this successfully when using the auto-inclusion of
> dependent patches, but I don't know why. Applying the patches manually works
> without error.

hi Owen, would you paste your command workflow to the ticket, please?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Patch doesn't apply         |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #54910|0                           |1
        is obsolete|                            |
  Attachment #54911|0                           |1
        is obsolete|                            |
  Attachment #54912|0                           |1
        is obsolete|                            |
  Attachment #54913|0                           |1
        is obsolete|                            |
  Attachment #54918|0                           |1
        is obsolete|                            |

--- Comment #13 from Jonathan Druart <[hidden email]> ---
Created attachment 55856
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55856&action=edit
Bug 17196: Move marcxml out of the biblioitems table

Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html

There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
  SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY
publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats
(marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).

Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
    * misc/cronjobs/delete_records_via_leader.pl
    * misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result

Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #14 from Jonathan Druart <[hidden email]> ---
Created attachment 55857
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55857&action=edit
Bug 17196: Fix query builder for item search

The item search needs to join on biblio_metadata to allow search on
marcxml field

Test plan:
Launch complex item searches (using marc fields).

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #15 from Jonathan Druart <[hidden email]> ---
Created attachment 55858
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55858&action=edit
Bug 17196: Update misc/maintenance/sanitize_records.pl

Test plan:
Use the sanitize_records.pl maintenance script with the --auto-search
option

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #16 from Jonathan Druart <[hidden email]> ---
Created attachment 55859
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55859&action=edit
Bug 17196: Update switch_marc21_series_info.pl

Not sure if this script is still used, could someone confirm?

Test plan:
If you know how to test it, please do

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

--- Comment #17 from Jonathan Druart <[hidden email]> ---
Created attachment 55860
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55860&action=edit
Bug 17196: DB Changes

Remove biblioitems.marcxml and deletedbiblioitems.marcxml
and adds 2 table biblio_metadata and deletedbiblio_metadata

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #55856|0                           |1
        is obsolete|                            |

--- Comment #18 from Mason James <[hidden email]> ---
Created attachment 55904
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55904&action=edit
Bug 17196: Move marcxml out of the biblioitems table

Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html

There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
  SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY
publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats
(marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).

Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
    * misc/cronjobs/delete_records_via_leader.pl
    * misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result

Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.

Signed-off-by: Mason James <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #55857|0                           |1
        is obsolete|                            |

--- Comment #19 from Mason James <[hidden email]> ---
Created attachment 55905
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55905&action=edit
Bug 17196: Fix query builder for item search

The item search needs to join on biblio_metadata to allow search on
marcxml field

Test plan:
Launch complex item searches (using marc fields).

Signed-off-by: Mason James <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #55858|0                           |1
        is obsolete|                            |

--- Comment #20 from Mason James <[hidden email]> ---
Created attachment 55906
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55906&action=edit
Bug 17196: Update misc/maintenance/sanitize_records.pl

Test plan:
Use the sanitize_records.pl maintenance script with the --auto-search
option

Signed-off-by: Mason James <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #55859|0                           |1
        is obsolete|                            |

--- Comment #21 from Mason James <[hidden email]> ---
Created attachment 55907
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55907&action=edit
Bug 17196: Update switch_marc21_series_info.pl

Not sure if this script is still used, could someone confirm?

Test plan:
If you know how to test it, please do

Signed-off-by: Mason James <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 17196] Move marcxml out of the biblioitems table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17196

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #55860|0                           |1
        is obsolete|                            |

--- Comment #22 from Mason James <[hidden email]> ---
Created attachment 55908
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55908&action=edit
Bug 17196: DB Changes

Remove biblioitems.marcxml and deletedbiblioitems.marcxml
and adds 2 table biblio_metadata and deletedbiblio_metadata

Signed-off-by: Mason James <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
1234 ... 6