Help testing marc staging performance issues

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Help testing marc staging performance issues

barton
Hi, All!

At ByWater, we've had a few reports of

"Home › Tools › Stage MARC records for import" timing out on large import
batches, with the message "there was an error."

When this happens, there are typically two batches under

Home › Tools › Manage staged MARC records

The first batch will show a number of records, and the second batch will
show 0 records.

I believe this typically happens under plack. I'm having a *very* difficult
time re-creating the error, so I don't have any plack error logs, I don't
have any good numbers on how many records cause the problem.

One of our libraries told me that the problem seems to be worse when using
matching rules.

If you've seen issues like this, I would love to know

1) What version of Koha you're using
2) How many records you were importing
3) If you were doing any sort of record matching
4) Any other relevant details
5) If you have access to the server error logs, I'd like to know what
messages you see there.

I'd like to file a bug, but without being able to replicate the problem, I
can't get any traction there.

Thanks,

--Barton
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Help testing marc staging performance issues

Jonathan Druart
Hi Barton,

> I believe this typically happens under plack.

These scripts should not run under plack, check your apache config file
(search for "ProxyPass" in apache-shared-intranet-plack.conf)

Cheers,
Jonathan

On Wed, 19 Sep 2018 at 19:17 Barton Chittenden <[hidden email]>
wrote:

> Hi, All!
>
> At ByWater, we've had a few reports of
>
> "Home › Tools › Stage MARC records for import" timing out on large import
> batches, with the message "there was an error."
>
> When this happens, there are typically two batches under
>
> Home › Tools › Manage staged MARC records
>
> The first batch will show a number of records, and the second batch will
> show 0 records.
>
> I believe this typically happens under plack. I'm having a *very* difficult
> time re-creating the error, so I don't have any plack error logs, I don't
> have any good numbers on how many records cause the problem.
>
> One of our libraries told me that the problem seems to be worse when using
> matching rules.
>
> If you've seen issues like this, I would love to know
>
> 1) What version of Koha you're using
> 2) How many records you were importing
> 3) If you were doing any sort of record matching
> 4) Any other relevant details
> 5) If you have access to the server error logs, I'd like to know what
> messages you see there.
>
> I'd like to file a bug, but without being able to replicate the problem, I
> can't get any traction there.
>
> Thanks,
>
> --Barton
> _______________________________________________
> Koha mailing list  http://koha-community.org
> [hidden email]
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Help testing marc staging performance issues

barton
On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart <
[hidden email]> wrote:

> Hi Barton,
>
> > I believe this typically happens under plack.
>
> These scripts should not run under plack, check your apache config file
> (search for "ProxyPass" in apache-shared-intranet-plack.conf)
>

You're exactly right, those are excluded:

        ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!"
        ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"

All, please disregard what I said about plack, apparently my brain was
playing tricks with me when I wrote that. The rest of the message stands:
If you've seen errors importing large batches of Marc records and gotten
the message "there was an error.", I'd like more information so that I can
file a decent bug report.

Thanks,

--Barton
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Help testing marc staging performance issues

Tomas Cohen Arazi
We should really do a two-pass processing, so the first thing is
calculating the batch size. That way Ajax will not timeout.

El mié., 19 de sep. de 2018 20:04, Barton Chittenden <
[hidden email]> escribió:

> On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart <
> [hidden email]> wrote:
>
> > Hi Barton,
> >
> > > I believe this typically happens under plack.
> >
> > These scripts should not run under plack, check your apache config file
> > (search for "ProxyPass" in apache-shared-intranet-plack.conf)
> >
>
> You're exactly right, those are excluded:
>
>         ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!"
>         ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
>
> All, please disregard what I said about plack, apparently my brain was
> playing tricks with me when I wrote that. The rest of the message stands:
> If you've seen errors importing large batches of Marc records and gotten
> the message "there was an error.", I'd like more information so that I can
> file a decent bug report.
>
> Thanks,
>
> --Barton
> _______________________________________________
> Koha mailing list  http://koha-community.org
> [hidden email]
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io <http://theke.io/>)
✆ +54 9351 3513384
GPG: B2F3C15F
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Help testing marc staging performance issues

barton
Tomas,

Do you know if there's a bug open for this? If not, would you mind filing
one?

Thanks,

--Barton

On Wed, Sep 19, 2018 at 7:57 PM, Tomas Cohen Arazi <[hidden email]>
wrote:

> We should really do a two-pass processing, so the first thing is
> calculating the batch size. That way Ajax will not timeout.
>
> El mié., 19 de sep. de 2018 20:04, Barton Chittenden <
> [hidden email]> escribió:
>
>> On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart <
>> [hidden email]> wrote:
>>
>> > Hi Barton,
>> >
>> > > I believe this typically happens under plack.
>> >
>> > These scripts should not run under plack, check your apache config file
>> > (search for "ProxyPass" in apache-shared-intranet-plack.conf)
>> >
>>
>> You're exactly right, those are excluded:
>>
>>         ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!"
>>         ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
>>
>> All, please disregard what I said about plack, apparently my brain was
>> playing tricks with me when I wrote that. The rest of the message stands:
>> If you've seen errors importing large batches of Marc records and gotten
>> the message "there was an error.", I'd like more information so that I can
>> file a decent bug report.
>>
>> Thanks,
>>
>> --Barton
>> _______________________________________________
>> Koha mailing list  http://koha-community.org
>> [hidden email]
>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>
> --
> Tomás Cohen Arazi
> Theke Solutions (https://theke.io <http://theke.io/>)
> ✆ +54 9351 3513384
> GPG: B2F3C15F
>
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Help testing marc staging performance issues

barton
So, looking around, I see some bugs that seem to cover the same ground:

Bug 17240 - "Allow processes that rely on background jobs run in Plack mode"

Bug 15032 - [Plack] Scripts that fork (like stage-marc-import.pl) don't
work as expected

Bug 20342 - Fix table locking issue, progress bar not updating, and allow
use of plack for Stage MARC Import


On Thu, Sep 20, 2018 at 5:02 PM Barton Chittenden <
[hidden email]> wrote:

> Tomas,
>
> Do you know if there's a bug open for this? If not, would you mind filing
> one?
>
> Thanks,
>
> --Barton
>
> On Wed, Sep 19, 2018 at 7:57 PM, Tomas Cohen Arazi <[hidden email]>
> wrote:
>
>> We should really do a two-pass processing, so the first thing is
>> calculating the batch size. That way Ajax will not timeout.
>>
>> El mié., 19 de sep. de 2018 20:04, Barton Chittenden <
>> [hidden email]> escribió:
>>
>>> On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart <
>>> [hidden email]> wrote:
>>>
>>> > Hi Barton,
>>> >
>>> > > I believe this typically happens under plack.
>>> >
>>> > These scripts should not run under plack, check your apache config file
>>> > (search for "ProxyPass" in apache-shared-intranet-plack.conf)
>>> >
>>>
>>> You're exactly right, those are excluded:
>>>
>>>         ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!"
>>>         ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
>>>
>>> All, please disregard what I said about plack, apparently my brain was
>>> playing tricks with me when I wrote that. The rest of the message stands:
>>> If you've seen errors importing large batches of Marc records and gotten
>>> the message "there was an error.", I'd like more information so that I
>>> can
>>> file a decent bug report.
>>>
>>> Thanks,
>>>
>>> --Barton
>>> _______________________________________________
>>> Koha mailing list  http://koha-community.org
>>> [hidden email]
>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>
>> --
>> Tomás Cohen Arazi
>> Theke Solutions (https://theke.io <http://theke.io/>)
>> ✆ +54 9351 3513384
>> GPG: B2F3C15F
>>
>
>
_______________________________________________
Koha mailing list  http://koha-community.org
[hidden email]
https://lists.katipo.co.nz/mailman/listinfo/koha