[Bug 22417] New: Delegate background jobs execution

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] New: Delegate background jobs execution

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

            Bug ID: 22417
           Summary: Delegate background jobs execution
 Change sponsored?: ---
           Product: Koha
           Version: unspecified
          Hardware: All
                OS: All
            Status: ASSIGNED
          Severity: enhancement
          Priority: P5 - low
         Component: Architecture, internals, and plumbing
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]
  Target Milestone: ---

This is related to:
Bug 15032 - [Plack] Scripts that fork (like stage-marc-import.pl) don't work as
expected
Bug 1993 - Task Scheduler Needs Re-write

And many others.

The goal here would be to remove the background execution in real time we are
doing right now and delegate the jobs to an external service (handling several
workers, load balancing, etc.)

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Fridolin SOMERS <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
                   |                            |m

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Jonathan Druart <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |In Discussion

--- Comment #1 from Jonathan Druart <[hidden email]> ---
I first tried Minion but found problematic issues for us (Koha):
- The mysql backend requide Mojo::mysql which requires DBD::mysql 4.042
However Jessie has 4.028 and Stretch 4.041
- We could have used the Postgres backend, but we would have require
MySQL/MariaDB and Postgres, which is not acceptable.

I then gave a try to RabbitMQ (https://www.rabbitmq.com/documentation.html),
which is an implementation of AMQP in Erlang.
If you want more information about it I let you follow the link.
The result is quite good, I managed to have a working POC, adapt to our uses,
in a single day.

Test plan:
0. Prerequisite:
% sudo apt install libanyevent-rabbitmq-perl rabbitmq-server
% sudo cpanm Net::RabbitFoot # Quid of the debian package?

Then make sure the RabbitMQ server is running:
% sudo service rabbitmq-server status
and start it if needed with
% sudo service rabbitmq-server start

Then create, or make sure you have a MARC Modification Template, with
template_id=1 (like add new field 123$z=foo)

1. To test this patch set you should first understand how the enqueue/consume
process work:
 % perl new_koha_job.pl # Launch it 1+ times
 => The job must have been enqueued (no error in the output)
 % perl koha_worker.pl # Will consume the 1+ tasks you enqueued
 Do not close this worker and start another one in another console, then
enqueue more new jobs.

Enjoy :)
(yes it should be fun)

2. Watch the history of the jobs
Hit admin/background_jobs.pl (logged in as superlibrarian)
This is an interface (to improve) to the list of pending/running and finished
processes

3. Use this whole stuff in a real Koha world:
Play with the "Batch record modification" tool (with biblios)
Notice the different in the last (report) step and click on the "View detail of
the enqueued job" link

Discussions:
1. Please grep TODO and FIXME in the patchset, there are lot to do/fix.
2. Even if we do not go for RabbitMQ I think we could start moving the code
from pl to Koha::BackgroundJob::*pm
It will be "easy" (nothing is easy, right?) to replace it with something else.
3. With this base we will want to answer different needs:
  * background jobs
  * task scheduler
  * reindex process (?)

Going further:
1. Implement Koha::BackgroundJob for the different background jobs
2. Provide a koha-worker script to consume to jobs
3. Rewrite the task scheduler with Koha::BackgroundJob (1993)
4. Provide a configuration file to prioritize or postpone jobs (for instance no
batch modification on Wednesday between 9-17) - Or maybe we won't need that (?)
5. What would be your needs?

Code is at https://gitlab.com/joubu/Koha/commits/bug_22417 (as the code will
evolve in the next days I will not pollute here).
Switched to 'In Discussion' to collect thoughts.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #2 from Jonathan Druart <[hidden email]> ---
And the screenshots:

https://snag.gy/WVeqDf.jpg
  What you get once the job has been enqueued

https://snag.gy/1GgnN5.jpg
  Detail of a non started job

https://snag.gy/ySnlaF.jpg
  Detail of a finished job

https://snag.gy/s5dF7n.jpg
  List of jobs

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Tomás Cohen Arazi <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Bob Birchall <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

David Cook <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from David Cook <[hidden email]> ---
I've taken a brief look at the code and the screenshots and it's looking good.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #4 from David Cook <[hidden email]> ---
(In reply to Jonathan Druart from comment #1)

> Discussions:
> <snip>
> 2. Even if we do not go for RabbitMQ I think we could start moving the code
> from pl to Koha::BackgroundJob::*pm
> It will be "easy" (nothing is easy, right?) to replace it with something
> else.

I totally agree; it would nice to be as message queue implementation agnostic
as possible.


> Going further:
> 1. Implement Koha::BackgroundJob for the different background jobs
> 2. Provide a koha-worker script to consume to jobs
> 3. Rewrite the task scheduler with Koha::BackgroundJob (1993)

It's been a couple years since I looked at RabbitMQ. Is there a way to
cancel/remove/delete a message once it is enqueued? I think this would be
useful for a task scheduler. I built that into the POE-based task scheduler I
built for https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662.
(Actually, I also built-in the ability for in-progress jobs to be canceled in
the event a big batch job was started and a person realized it was a mistake.)


> 4. Provide a configuration file to prioritize or postpone jobs (for instance
> no batch modification on Wednesday between 9-17) - Or maybe we won't need
> that (?)

Fridolin mentioned something like this to me when I was talking about
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with him at
Kohacon18 in Portland.

He pointed out that for a multi-tenant server you might not want all the
different Koha instances on the server to run their background jobs at the same
time due to restraints like performance, rate-limiting, connection limits, etc.

Off the top of my head, I don't have a solution for this one though, although
in the case of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662
I use separate task schedulers for every Koha instance on the server.

In this case, we could potentially use 1 RabbitMQ server, and X number of
workers... although we'd have to have a way of setting the KOHA_CONF for each
task.

That might be the best way to handle multi-tenant scenarios.

Alternatively, I think we'd need to have unique queues for each Koha instance
and workers for those unique queues. Requires a bit of thought in any case.

> 5. What would be your needs?
>

Ideally, it would be nice to replace my home-made task scheduler in
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with this one.
I like that mine can add/start/pause/stop/remove tasks, but I haven't seen a
mainstream task scheduler that is capable of doing that, so that might be too
difficult to do here.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

[hidden email] <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
                   |                            |om

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Julian Maurice <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #5 from Jonathan Druart <[hidden email]> ---
I have added a commit to add the ability to cancel a job.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Josef Moravec <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #6 from Josef Moravec <[hidden email]> ---
I just tried it and it works like a charm! Great job Jonathan!

I think just to polish this functionality would be great start and big progress
from what we have now.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #7 from David Cook <[hidden email]> ---
(In reply to Jonathan Druart from comment #5)
> I have added a commit to add the ability to cancel a job.

Thanks, Jonathan!

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #8 from David Cook <[hidden email]> ---
(In reply to Josef Moravec from comment #6)
> I think just to polish this functionality would be great start and big
> progress from what we have now.

Actually, I think that's a really good point. I think in the community we
sometimes have a tendency to debate/discuss things to death... which stifles
innovation.

I think it could be good to get something polished into Koha (maybe just 1
background job using RabbitMQ to start) and then if that is working well then
we add more background jobs.

I suppose it might be more efficient to have it fully specified before
implementing in Koha rather than doing iterative development but...

--

Oh, but I think my concerns about multi-tenant environments are important. At a
glance, this implementation seems like it would only allow 1 Koha instance talk
to 1 RabbitMQ instance. But I think in practice there are many people who run
multiple Koha instances on the same server and it would be good if they could
share 1 RabbitMQ instance.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Marcel de Rooy <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #9 from Marcel de Rooy <[hidden email]> ---
Do we really need the background jobs ? Or should we refactor these scripts in
the direction of AJAX API calls in JavaScript ?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Katrin Fischer <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #10 from Katrin Fischer <[hidden email]> ---
(In reply to Marcel de Rooy from comment #9)
> Do we really need the background jobs ? Or should we refactor these scripts
> in the direction of AJAX API calls in JavaScript ?

I am not sure if this would also allow reimplementing the scheduler with
scheduled reports? In my understanding some jobs might not be triggered by the
GUI.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #11 from Marcel de Rooy <[hidden email]> ---
(In reply to Katrin Fischer from comment #10)
> (In reply to Marcel de Rooy from comment #9)
> > Do we really need the background jobs ? Or should we refactor these scripts
> > in the direction of AJAX API calls in JavaScript ?
>
> I am not sure if this would also allow reimplementing the scheduler with
> scheduled reports? In my understanding some jobs might not be triggered by
> the GUI.

OK That's true. I was focused on the forking scripts.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #12 from Tomás Cohen Arazi <[hidden email]> ---
(In reply to Marcel de Rooy from comment #11)

> (In reply to Katrin Fischer from comment #10)
> > (In reply to Marcel de Rooy from comment #9)
> > > Do we really need the background jobs ? Or should we refactor these scripts
> > > in the direction of AJAX API calls in JavaScript ?
> >
> > I am not sure if this would also allow reimplementing the scheduler with
> > scheduled reports? In my understanding some jobs might not be triggered by
> > the GUI.
>
> OK That's true. I was focused on the forking scripts.

IMHO, in either case we need a way to throttle concurrency and server loads. So
a task queue is needed.

My only doubt is if we should really model the task queue with a message queue.
I tried the same Jonathan tried but using AnyEvent::Task and the results where
similar, and it only required a unix socket to communicate client and server
processes. My suggestion was to use zeromq. But lets see how this goes.
Specially in the integration step.

I agree with David that this of course would require a single RabbitMQ server
so questions: Is there a concept of namespaces? can any Koha instance access
other instance's messages?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #13 from Jonathan Druart <[hidden email]> ---
(In reply to Tomás Cohen Arazi from comment #12)
> I agree with David that this of course would require a single RabbitMQ
> server so questions: Is there a concept of namespaces? can any Koha instance
> access other instance's messages?

I really would like you guys to apply, read and try the patches, read the code,
then discuss :)
There are a lot of FIXME and TODO.

This one answers your question:

 16 sub connect {
 17     my ( $self );
 18     my $conn = Net::RabbitFoot->new()->load_xml_spec()->connect(
 19         host => 'localhost', # TODO Move this to KOHA_CONF
 20         port => 5672,
 21         user => 'guest',
 22         pass => 'guest',
 23         vhost => '/',
 24     );
 25
 26     return $conn;
 27 }

So different vhost, user and pass can be used.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #14 from Jonathan Druart <[hidden email]> ---
There is also:

"Going further:
2. Provide a koha-worker script to consume the jobs"

So one worker per instance.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #15 from David Cook <[hidden email]> ---
(In reply to Jonathan Druart from comment #13)

> (In reply to Tomás Cohen Arazi from comment #12)
> > I agree with David that this of course would require a single RabbitMQ
> > server so questions: Is there a concept of namespaces? can any Koha instance
> > access other instance's messages?
>
> I really would like you guys to apply, read and try the patches, read the
> code, then discuss :)
> There are a lot of FIXME and TODO.
>
> This one answers your question:
>
>  16 sub connect {
>  17     my ( $self );
>  18     my $conn = Net::RabbitFoot->new()->load_xml_spec()->connect(
>  19         host => 'localhost', # TODO Move this to KOHA_CONF
>  20         port => 5672,
>  21         user => 'guest',
>  22         pass => 'guest',
>  23         vhost => '/',
>  24     );
>  25
>  26     return $conn;
>  27 }
>
> So different vhost, user and pass can be used.

I haven't tried the patches but I have read your code, and that's why I've
voiced my concerns about multi-tenancy. There wasn't enough information there
to understand, so that's why I asked the questions.

Your reply almost answers the question, but not quite, so now I'm just reading
about RabbitMQ.

If I understand correctly, each Koha instance would get its own RabbitMQ
"vhost", and within each "vhost" there would be named queues (which at the
moment are hard-coded but could be configurable I suppose).

Cool. That removes my concerns about multi-tenancy. Thanks for that, Jonathan.

(Maybe in your example you could use something like "vhost => '/kohademo'" or
maybe include some code comments just so it's more obvious how it works.)

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #16 from David Cook <[hidden email]> ---
(In reply to Tomás Cohen Arazi from comment #12)
> My only doubt is if we should really model the task queue with a message
> queue. I tried the same Jonathan tried but using AnyEvent::Task and the
> results where similar, and it only required a unix socket to communicate
> client and server processes. My suggestion was to use zeromq. But lets see
> how this goes. Specially in the integration step.

Yeah, for #10662, I used POE::Component::JobQueue and a Unix socket. It was
very effective, although my bespoke implementation had a number of limitations.
I figured allowing TCP sockets also might be a good idea just to allow for more
distributed architectures.

When I was working on #10662, I was considering both RabbitMQ and ZeroMQ, but
ended up doing my own job queue. I'm not necessarily sold on RabbitMQ yet, but
I figure if Jonathan is willing to do it, I'm happy to see how this goes too.
I'm not willing to put in the time to make it happen :/.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #17 from Jonathan Druart <[hidden email]> ---
(In reply to David Cook from comment #15)
> If I understand correctly, each Koha instance would get its own RabbitMQ
> "vhost", and within each "vhost" there would be named queues (which at the
> moment are hard-coded but could be configurable I suppose).

I imaged one koha-worker script per Koha instance, with as well one vhost.
The queues would be named 'batch_record_modification', 'manage-marc-import',
'batch_item_modification', etc.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #18 from Jonathan Druart <[hidden email]> ---
TODO, remove that:

Bug 15032: UpdateDB entry
+    # $dbh->do( "ALTER TABLE biblio ADD COLUMN badtaste int" );

Bug 22417: Add the ability to cancel a job
+        sleep(5);

:)

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Fridolin SOMERS <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugs.koha-community
                   |                            |.org/bugzilla3/show_bug.cgi
                   |                            |?id=22481

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #19 from David Cook <[hidden email]> ---
I was thinking about this work this morning. I'm really busy at the moment, but
still think this is a good thing to be working on.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #20 from Julian Maurice <[hidden email]> ---
I played a little with this patchset, and also with Minion. Some remarks:

- Net::RabbitFoot says it's compatible with version 0-8 of the AMQP
specification. RabbitMQ website says it implements version 0-9-1. Are we sure
these two are compatible ?
- Net::RabbitFoot is quite easy to package for Debian stretch/stable (all
dependencies are already packaged - I did not try on oldstable)
- If rabbitmq-server is not started, new_koha_job.pl and koha_worker.pl do not
work. Is there a way to enqueue job when rabbitmq-server is not started ?
- If koha_worker.pl is not started, and I enqueue jobs, and restart
rabbitmq-server and then start koha_worker.pl, the worker does nothing.
Potential loss of jobs here ?

Now for the Minion part,
- the mysql backend requires a recent version of Minion (which requires a more
recent version of Mojolicious than the one packaged in debian.k-c.org). It
might be complicated to package.
- the sqlite backend however have some versions that can be packaged and can
work with our version of Mojolicious (requires packaging Minion 6.04,
Mojo::SQLite 2.002, and Minion::Backend::SQLite 0.009)
- it works similarly to the rabbitmq code (a worker that run continuously, some
code to enqueue jobs, where all jobs are stored in database) but doesn't
require to have something (like rabbitmq-server) listening all the time to
work. The worker just fetches queued jobs from its backend when it starts (and
also regularly while it's running)
- from what I have seen, it requires less code (see for example:
https://mojolicious.io/blog/2018/12/10/minion-stands-alone/)
- workers can be "paused" (meaning they won't accept new jobs, but active jobs
will continue to run)

I'm a little concerned about the potential loss of jobs with the RabbitMQ
option. Is there any way to avoid this problem ?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #21 from Jonathan Druart <[hidden email]> ---
(In reply to Julian Maurice from comment #20)
> I'm a little concerned about the potential loss of jobs with the RabbitMQ
> option. Is there any way to avoid this problem ?

Using Monit to make sure the service is always up? :)

But indeed we should return an error if the server has not been reached, and so
the job not enqueued.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #22 from Julian Maurice <[hidden email]> ---
(In reply to Jonathan Druart from comment #21)
> (In reply to Julian Maurice from comment #20)
> > I'm a little concerned about the potential loss of jobs with the RabbitMQ
> > option. Is there any way to avoid this problem ?
>
> Using Monit to make sure the service is always up? :)
>
> But indeed we should return an error if the server has not been reached, and
> so the job not enqueued.

That solves one problem, but not the other: when RabbitMQ already has enqueued
jobs but stops before koha_worker.pl had the chance to process them.

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 22417] Delegate background jobs execution

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #23 from Jonathan Druart <[hidden email]> ---
But then the job will be in the DB table, so we will not lose it, right?

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
12