[Bug 20377] New: Various speed tweaks to 'remove_unused_authorities.pl' script

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] New: Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

            Bug ID: 20377
           Summary: Various speed tweaks to 'remove_unused_authorities.pl'
                    script
 Change sponsored?: ---
           Product: Koha
           Version: master
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5 - low
         Component: Command-line Utilities
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]
                CC: [hidden email]

Here is a patch to make 'remove_unused_authorities.pl' a bit faster
i get a 5x speedup after this patch

fyi: i pinched most of this patch from bulkmarcimport.pl


1/ import a bunch of authority record into a Koha, and index

2/ run script, note execution time
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m50s

3/ apply patch

4/ run script again, and notice it's a bit faster
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m10s

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |Needs Signoff

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

--- Comment #1 from Mason James <[hidden email]> ---
Created attachment 72727
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=72727&action=edit
Bug 20377: Various speed tweaks to 'remove_unused_authorities.pl' script

Here is a patch to make 'remove_unused_authorities.pl' a bit faster
i get a 5x speedup after this patch

fyi: i pinched most of this patch from bulkmarcimport.pl

1/ import a bunch of authority record into a Koha, and index

2/ run script, note execution time
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m50s

3/ apply patch

4/ run script again, and notice it's a bit faster
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m10s

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Brendan Gallagher <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Needs Signoff               |Signed Off

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Brendan Gallagher <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #72727|0                           |1
        is obsolete|                            |

--- Comment #2 from Brendan Gallagher <[hidden email]> ---
Created attachment 72775
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=72775&action=edit
Bug 20377: Various speed tweaks to 'remove_unused_authorities.pl' script

Here is a patch to make 'remove_unused_authorities.pl' a bit faster
i get a 5x speedup after this patch

fyi: i pinched most of this patch from bulkmarcimport.pl

1/ import a bunch of authority record into a Koha, and index

2/ run script, note execution time
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m50s

3/ apply patch

4/ run script again, and notice it's a bit faster
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m10s

Signed-off-by: Brendan Gallagher <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Marcel de Rooy <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
         QA Contact|[hidden email]-communit |[hidden email]
                   |y.org                       |
   Patch complexity|---                         |Small patch
           Assignee|[hidden email]-commun |[hidden email]
                   |ity.org                     |

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

--- Comment #3 from Marcel de Rooy <[hidden email]> ---
Note: My first impression about your use of $commit was that we should not use
it that way (I would interpret it as a confirm flag like in other cron jobs),
but bulkmarcimport is using it like you do here..

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Marcel de Rooy <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Signed Off                  |Passed QA

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Marcel de Rooy <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #72775|0                           |1
        is obsolete|                            |

--- Comment #4 from Marcel de Rooy <[hidden email]> ---
Created attachment 73166
  -->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73166&action=edit
Bug 20377: Various speed tweaks to 'remove_unused_authorities.pl' script

Here is a patch to make 'remove_unused_authorities.pl' a bit faster
i get a 5x speedup after this patch

fyi: i pinched most of this patch from bulkmarcimport.pl

1/ import a bunch of authority record into a Koha, and index

2/ run script, note execution time
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m50s

3/ apply patch

4/ run script again, and notice it's a bit faster
  $ time perl ./misc/migration_tools/remove_unused_authorities.pl
  real    0m10s

Signed-off-by: Brendan Gallagher <[hidden email]>

Signed-off-by: Marcel de Rooy <[hidden email]>

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

--- Comment #5 from Marcel de Rooy <[hidden email]> ---
Combining -test and -commit is kind of confusing but could be considered
outside the scope? Would welcome a confirm flag instead of having to add -test.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Passed QA                   |Failed QA
                 CC|                            |[hidden email]-c
                   |                            |ommunity.org

--- Comment #6 from Jonathan Druart <[hidden email]> ---
- Please improve bug title and commit messages (there are no various tweaks,
just using a transaction)
- specify default value in POD
- maybe we should remove 'c' to avoid the ambiguity with confirm?
- transaction must be done using $schema->txn_do

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Various speed tweaks to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

--- Comment #7 from Jonathan Druart <[hidden email]> ---
+        DelAuthority({ authid => $data->{'authid'}, skip_merge => 1 }) unless
$test;

Checking $test must be done on the commit statements

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Add transaction and skip_merge to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Various speed tweaks to     |Add transaction and
                   |'remove_unused_authorities. |skip_merge to
                   |pl' script                  |'remove_unused_authorities.
                   |                            |pl' script

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 20377] Add transaction and skip_merge to 'remove_unused_authorities.pl' script

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20377

Mason James <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email],
                   |                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/