Taking patron images out of db into a directory

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Taking patron images out of db into a directory

Mengu Yazicioglu
Hi all,

We'd like to write a patch to move patron images out of db into a
directory in the server.

By this change, we'd like to satisfy following.

1. The size of database backup will be decreased, by this way one may
store db backups more.

2. Database performance will increase

3. It will be possible to put patron images in different disk in the server

4. Easily put images gathered from other systems like (university
system, personnel system, etc)

What is your idea?

--
Mengü Yazıcıoğlu
Devinim Yazılım Eğitim Danışmanlık
Reşit Galip Cad. No: 29/6
Çankaya/ANKARA

Tel: (312) 446 86 68
Cep: (532) 701 90 27

_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Owen Leonard-4
> We'd like to write a patch to move patron images out of db into a
> directory in the server.

I think this is a good idea. There's no reason not to offer it as a
choice if it's possible to do both. I think an argument could be made
that it shouldn't be a choice presented to the librarian and therefore
should be a setting in Koha's configuration file rather than, say, a
system preference.

   -- Owen

--
Web Developer
Athens County Public Libraries
(740) 737-6006
https://www.myacpl.org
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mark Tompsett
In reply to this post by Mengu Yazicioglu
Greetings,

> We'd like to write a patch to move patron images out of db into a
> directory in the server.

Mmmm... no. People doing backups might forget to supplement
the default Koha backup with that specific directory, and come a
system failure and a restore later... they discover the photos are
no more.


> 1. The size of database backup will be decreased, by this way one
> may store db backups more.

Only if the pictures are on a different server.


> 2. Database performance will increase

Theoretically, yes. But I don't see how one tiny thing would speed it up
that much, unless ...
(goes to update his branches and see the schema)
Oh... okay... yes, perhaps patron images should be in its own linked table,
so
that borrower.* won't be hindered with patronimage data.


> 3. It will be possible to put patron images in different disk in the
> server
> 4. Easily put images gathered from other systems like
>      (university system, personnel system, etc)

Putting the files in another collection that gets backed up differently
would be okay.


> What is your idea?

I prefer the notion of in DB or externally controlled by system preferences.
I think system preferences, because librarians may not have access to the
technical koha-conf.xml configuration settings file. And patron images would
likely fall under the responsibility of librarians, not IT people.

GPML,
Mark Tompsett

_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Katrin Fischer-2
Hi Mark and all,

I kind of disagree with a few points. I see no harm in storing the
patron images optionally outside of the database as suggested by Owen. I
don't think the database is a good place to store files actually,
storing them outside makes a lot of sense to me.

Patron images are currently stored in their own linked table called
patronimage.

The setting of paths is usually done in the koha-conf.xml file so I
think that would be the place for configuration of patron image storage
too. Afaik we already have an option to include index files and uploaded
files in the backups - I think it should not be hard to add another
option. We could make the switch between database and directory storage
a pref maybe.

Katrin

On 10.04.20 21:49, Mark Tompsett wrote:

> Greetings,
>
>> We'd like to write a patch to move patron images out of db into a
>> directory in the server.
>
> Mmmm... no. People doing backups might forget to supplement
> the default Koha backup with that specific directory, and come a
> system failure and a restore later... they discover the photos are
> no more.
>
>
>> 1. The size of database backup will be decreased, by this way one
>> may store db backups more.
>
> Only if the pictures are on a different server.
>
>
>> 2. Database performance will increase
>
> Theoretically, yes. But I don't see how one tiny thing would speed it
> up that much, unless ...
> (goes to update his branches and see the schema)
> Oh... okay... yes, perhaps patron images should be in its own linked
> table, so
> that borrower.* won't be hindered with patronimage data.
>
>
>> 3. It will be possible to put patron images in different disk in the
>> server
>> 4. Easily put images gathered from other systems like
>>      (university system, personnel system, etc)
>
> Putting the files in another collection that gets backed up
> differently would be okay.
>
>
>> What is your idea?
>
> I prefer the notion of in DB or externally controlled by system
> preferences.
> I think system preferences, because librarians may not have access to the
> technical koha-conf.xml configuration settings file. And patron images
> would
> likely fall under the responsibility of librarians, not IT people.
>
> GPML,
> Mark Tompsett
> _______________________________________________
>
> Koha mailing list  http://koha-community.org
> [hidden email]
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Tomas Cohen Arazi
In reply to this post by Mengu Yazicioglu
El jue., 9 abr. 2020 a las 11:41, Mengu Yazicioglu (<[hidden email]>)
escribió:

> Hi all,
>
> We'd like to write a patch to move patron images out of db into a
> directory in the server.
>
> By this change, we'd like to satisfy following.
>
> 1. The size of database backup will be decreased, by this way one may
> store db backups more.
>

And they would take far less time to finish. Also, using incremental
backups would be possible.


> 2. Database performance will increase
>

Not sure about that. But the DB usually uses caching for the rows, and with
a limited space, so it *could* improve caching.

3. It will be possible to put patron images in different disk in the server
>

Yes, taking them out of the DB comes with this advantages.

4. Easily put images gathered from other systems like (university
> system, personnel system, etc)
>

It should definitely be using Koha::UploadedFIles. We really want to keep
the consistency between the DB and the images. Command-line utilities could
be developed to sync new batches, check nothing is missing, etc.

Good idea!


--
Tomás Cohen Arazi
Theke Solutions (http://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Owen Leonard-4
In reply to this post by Mark Tompsett
On Fri, Apr 10, 2020 at 3:50 PM Mark Tompsett <[hidden email]> wrote:

> Mmmm... no.

This kind of comment is completely counterproductive to meaningful
cooperation. Please be constructive without being dismissive.

  -- Owen

--
Web Developer
Athens County Public Libraries
(740) 737-6006
https://www.myacpl.org
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Eric Bégin-2
In reply to this post by Mengu Yazicioglu
Mengu,

We did something similar for one clients.  It's patrons pictures were on an external server using the borrowernumber as filename (they have  a unified id for all of their services)

So their is definitively use cases for such a feature.

In our case, we just "hacked" the picture URL with their server URL.

I think we should have a syspref for the patron pictures base URL, and maybe the key to use as filename (cardnumber, borrowernumber or even an patron attribute)

Hope that help !

⁣Eric Bégin
Solutions inLibro inc.​

Le 9 avr. 2020 à 10:42, à 10:42, Mengu Yazicioglu <[hidden email]> a écrit:
>
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mark Tompsett
In reply to this post by Owen Leonard-4
Greetings,

I wrote:
> Mmmm... no.

Owen replied:
> This kind of comment is completely counterproductive to meaningful
> cooperation. Please be constructive without being dismissive.

If that had been all I wrote, then your comment would be valid in my mind.

However, you didn't include the rest of the comment which explained why no:
"People doing backups might forget to supplement
the default Koha backup with that specific directory, and come a
system failure and a restore later... they discover the photos are
no more."

I also said:
"Putting the files in another collection that gets backed up differently
would be okay."

So... over all, I do not think I was dismissive, and apologize if people
read it that way.

I was sharing worries, because I believe someone (probably me and others),
somewhere, somehow will forget to configure something, and then something
bad will happen, because they forgot or didn't realize. After all, the
original poster did ask, "What is your idea?" which I took to mean, "What do
you think?" Sharing worries and gut reactions are valid answers to the
question, though as Owen noted not useful for trying to find a good
direction.

+1 to Tomas' comment about command line utilities to keep sync.


Katrin wrote:
> Afaik we already have an option to include index files and uploaded
> files in the backups - I think it should not be hard to add another
> option.

If we do, that's great. If not, we have a functionality gap to solve.


Additionally, system level things are usually beyond the access of
librarians and only IT.

Katrin addressed my concern:
> We could make the switch between database and directory storage a syspref
> maybe.

+1 to this idea.


Hope this clarifies.


Peace,
Mark Tompsett

_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mengu Yazicioglu
In reply to this post by Eric Bégin-2
Thanks for everyone's comments.

We've developed this kind of feature first for 3.20 version with more
than 1,600,000 users in the past.

Some of our new customers also wanted this feature for new versions and
with your comments, we'll start to write it soon.

Thanks everyone !

Mengü Yazıcıoğlu
Devinim Yazılım Eğitim Danışmanlık
Reşit Galip Cad. No: 29/6
Çankaya/ANKARA

Tel: (312) 446 86 68
Cep: (532) 701 90 27

On 11.04.2020 03:56, Eric Bégin wrote:

> Mengu,
>
> We did something similar for one clients.  It's patrons pictures were
> on an external server using the borrowernumber as filename (they have 
> a unified id for all of their services)
>
> So their is definitively use cases for such a feature.
>
> In our case, we just "hacked" the picture URL with their server URL.
>
> I think we should have a syspref for the patron pictures base URL, and
> maybe the key to use as filename (cardnumber, borrowernumber or even
> an patron attribute)
>
> Hope that help !
>
> Eric Bégin
> Solutions inLibro inc.
> Le 9 avr. 2020, à 10:42, Mengu Yazicioglu <[hidden email]
> <mailto:[hidden email]>> a écrit:
>
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Tomas Cohen Arazi
The first step: write the tests!

El sáb., 11 de abril de 2020 17:47, Mengu Yazicioglu <[hidden email]>
escribió:

> Thanks for everyone's comments.
>
> We've developed this kind of feature first for 3.20 version with more
> than 1,600,000 users in the past.
>
> Some of our new customers also wanted this feature for new versions and
> with your comments, we'll start to write it soon.
>
> Thanks everyone !
>
> Mengü Yazıcıoğlu
> Devinim Yazılım Eğitim Danışmanlık
> Reşit Galip Cad. No: 29/6
> Çankaya/ANKARA
>
> Tel: (312) 446 86 68
> Cep: (532) 701 90 27
>
> On 11.04.2020 03:56, Eric Bégin wrote:
> > Mengu,
> >
> > We did something similar for one clients.  It's patrons pictures were
> > on an external server using the borrowernumber as filename (they have
> > a unified id for all of their services)
> >
> > So their is definitively use cases for such a feature.
> >
> > In our case, we just "hacked" the picture URL with their server URL.
> >
> > I think we should have a syspref for the patron pictures base URL, and
> > maybe the key to use as filename (cardnumber, borrowernumber or even
> > an patron attribute)
> >
> > Hope that help !
> >
> > Eric Bégin
> > Solutions inLibro inc.
> > Le 9 avr. 2020, à 10:42, Mengu Yazicioglu <[hidden email]
> > <mailto:[hidden email]>> a écrit:
> >
> _______________________________________________
>
> Koha mailing list  http://koha-community.org
> [hidden email]
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mark Tompsett
Greetings,

Tomas Cohen Arazi wrote:
> The first step: write the tests!
+1000

By writing the tests first, you know what you are coding towards, and have
full test coverage for what you are going to provide a patch for. This makes
everything easier for reviewing. The test plan becomes: run the tests, they
have full code coverage.

GPML,
Mark Tompsett

P.S.

El sáb., 11 de abril de 2020 17:47, Mengu Yazicioglu <[hidden email]>
escribió:
> We've developed this kind of feature first for 3.20 version with more
> than 1,600,000 users in the past.

Ouch! We were stuck on that version for the longest time. May you have
success in coding and implementing your patch!

_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mengu Yazicioglu
Hi,
> El sáb., 11 de abril de 2020 17:47, Mengu Yazicioglu
> <[hidden email]>
> escribió:
>> We've developed this kind of feature first for 3.20 version with more
>> than 1,600,000 users in the past.
>
> Ouch! We were stuck on that version for the longest time. May you have
> success in coding and implementing your patch!

Why do you think that we may not? We're supporting Koha since v.2, and
we also did similar thing in 16.11. I just wanted to give the above
number of persons

to show that taking pictures out of db is a must in some cases. In any
size, taking pictures out of db will give more benefits than saving in
database.

Regards.


> _______________________________________________
>
> Koha mailing list  http://koha-community.org
> [hidden email]
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Tomas Cohen Arazi
Mengu, I read Mark as saying he's been there as well, stuck on a version
for a long time due to some customization. I don't think he's talking about
your ability to implement it at all!

El lun., 13 abr. 2020 a las 5:57, Mengu Yazicioglu (<[hidden email]>)
escribió:

> Hi,
> > El sáb., 11 de abril de 2020 17:47, Mengu Yazicioglu
> > <[hidden email]>
> > escribió:
> >> We've developed this kind of feature first for 3.20 version with more
> >> than 1,600,000 users in the past.
> >
> > Ouch! We were stuck on that version for the longest time. May you have
> > success in coding and implementing your patch!
>
> Why do you think that we may not? We're supporting Koha since v.2, and
> we also did similar thing in 16.11. I just wanted to give the above
> number of persons
>
> to show that taking pictures out of db is a must in some cases. In any
> size, taking pictures out of db will give more benefits than saving in
> database.
>
> Regards.
>
>
> > _______________________________________________
> >
> > Koha mailing list  http://koha-community.org
> > [hidden email]
> > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
> _______________________________________________
>
> Koha mailing list  http://koha-community.org
> [hidden email]
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>


--
Tomás Cohen Arazi
Theke Solutions (http://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F
_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

Mark Tompsett
In reply to this post by Mengu Yazicioglu
Greetings,

I wrote:
> Ouch! We were stuck on that version for the longest time. May you have
> success in coding and implementing your patch!

To which Mengu Yazicioglu  replied:
> Why do you think that we may not?
> We're supporting Koha since v.2, and we also did similar thing in 16.11.

I apologize, I misread. I thought you were still on 3.20 for that patch.
Upgrading patches is not always easy. I was offering best wishes and good
will
if that was the case.

I also apologize for clearly not expressing myself well enough, such that
I keep being understood as saying mean things. I'll go back to being silent
now.

Grace, Peace, Mercy, and Love to everyone...
Mark Tompsett

_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

BRANNON, CHRISTOPHER
In reply to this post by Mengu Yazicioglu
I would also like to suggest while you are doing this, please add a way to manage this directory from Koha.  I would see it as a huge asset if an administrator could browse and manage these images, along with images for other things, such as cover images, or even an online repository an archive.

___________________________
Christopher Brannon
IT Coordinator

702 E Front Ave
Coeur d'Alene, ID 83814
(208)769-2315 Ext. 456
[hidden email]
www.cdalibrary.org



_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Reply | Threaded
Open this post in threaded view
|

Re: Taking patron images out of db into a directory

dcook
In reply to this post by Mengu Yazicioglu
Recently, I've been re-thinking the scalability of Koha, so it's interesting to hear about this from you, Mengu, as the person who I think runs the biggest Koha installation in the world.

Originally, I thought putting images in the database was a bad idea, but it is an easy way to provide that image data over a network to multiple nodes and is part of an established community-supported application backup strategy.

But fetching arbitrary blobs from a database could be slow and it could inflate Koha's metadata/transaction database. Storing image data in the database seems convenient - especially for small implementations - but is more problematic for large implementations.

The probable solution to me for a large implementation is a network file share or object storage (e.g. AWS S3, Ceph, Minio, OpenStack Swift, etc). But that puts more of a burden on system administrators to make sure things are backed up. Not a problem for large implementations but it would be for small implementations.

The ultimate solution to me would be a standard API (Koha::UploadedFiles I presume) in Koha with different storage backends. This is a common approach used by other FOSS systems. You can have drivers for database storage, file system storage, object storage - and you can choose the driver best suited to your situation with the default option being the most convenient for small implementations (ie probably database storage).

Recently, we've locally provided an object storage integration to provide people with a robust low-cost method for storing documents in Koha, which seems to be working well.

Using a caching layer would be nice too, so loading the same page 2 times doesn't equal 2 high cost trips to the data storage. I've been using the IIIF Image Server Loris backed by Fedora Commons backed by AWS S3 and caching makes a world of difference for end users.

Anyway, that's my 2 cents. Ideally, the work done on this would be generalized, so that it could be used for things like biblio images, documents, etc. (I suppose storing URIs might be a useful way to do it. That way you could use file:///local/disk, file:///network/disk, s3://mybucket... maybe mysql://..)

(I'd be interested too to hear about optimizations people have for serving image data. Starman won't perform this task as well as Nginx or Apache, but I think we need to have this data behind an authorization layer, which currently means Perl. But that's really a completely different discussion.)

We might also want to move this from koha to koha-devel listserv?

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-----Original Message-----
Date: Thu, 9 Apr 2020 17:40:17 +0300
From: Mengu Yazicioglu <[hidden email]>
To: Koha <[hidden email]>
Subject: [Koha] Taking patron images out of db into a directory
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=utf-8; format=flowed

Hi all,

We'd like to write a patch to move patron images out of db into a directory in the server.

By this change, we'd like to satisfy following.

1. The size of database backup will be decreased, by this way one may store db backups more.

2. Database performance will increase

3. It will be possible to put patron images in different disk in the server

4. Easily put images gathered from other systems like (university system, personnel system, etc)

What is your idea?

--
Mengü Yazıcıoğlu
Devinim Yazılım Eğitim Danışmanlık
Reşit Galip Cad. No: 29/6
Çankaya/ANKARA

Tel: (312) 446 86 68
Cep: (532) 701 90 27



_______________________________________________

Koha mailing list  http://koha-community.org
[hidden email]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha

signature.asc (495 bytes) Download Attachment