[Bug 25603] New: Add 001 and 003 to biblio table

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] New: Add 001 and 003 to biblio table

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

            Bug ID: 25603
           Summary: Add 001 and 003 to biblio table
 Change sponsored?: ---
           Product: Koha
           Version: master
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5 - low
         Component: MARC Bibliographic data support
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]

Koha itself does not care too much about these fields, but sometimes they are
used when importing/syncing in records from external sources, and then it can
be work intensive to match on WHERE ExtractValue(
metadata,\'//controlfield[@tag="001"]\' ). This would be much easier if the 001
and 003 fields from the record had their own database columns we could match
on.

Similar to: Bug 11529 - Add subtitle, medium and part fields to biblio table

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

Andreas Jonsson <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Andreas Jonsson <[hidden email]> ---

It would be nice to have one table for control fields and one table for
data-fields.  We could make it configurable which fields that should be indexed
in these tables if storage space would be an issue.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

David Cook <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #2 from David Cook <[hidden email]> ---
(In reply to Andreas Jonsson from comment #1)
> It would be nice to have one table for control fields and one table for
> data-fields.  

I don't think that's ever going to happen, as we're (slowly) trying to get away
from MARC.

> We could make it configurable which fields that should be
> indexed in these tables if storage space would be an issue.

We wouldn't make the database schema configurable, and it would be the database
indexing that would be valuable here. Indexing database columns for identifiers
like 001 and 003 would be very reasonable though.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #3 from David Cook <[hidden email]> ---
(In reply to Magnus Enger from comment #0)
> Koha itself does not care too much about these fields, but sometimes they
> are used when importing/syncing in records from external sources, and then
> it can be work intensive to match on WHERE ExtractValue(
> metadata,\'//controlfield[@tag="001"]\' ). This would be much easier if the
> 001 and 003 fields from the record had their own database columns we could
> match on.
>
> Similar to: Bug 11529 - Add subtitle, medium and part fields to biblio table

I'm trying to wrap my head around how we'd do this in practice. In theory, Koha
should be resetting the 001 and 003, but it doesn't do that currently.

I guess you'd be relying on that? Trying to match off the original 001 and 003
to improve the ease of matching records from external sources. I can see the
logic there for sure.

Personally, I wish that we'd move the 001 and 003 to the 035 during import,
although that would present its own matching problems, plus the 035 is a
repeatable field.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #4 from David Cook <[hidden email]> ---
I'm trying to think how we could do this in a more extensible way... and I
might have an idea.

MARC:
001 - Control Number
003 - Control Number Identifier
005 - Date and Time of Latest Transaction

OAI-PMH:
identifier
baseURL
datestamp

These two schemas actually overlap pretty much perfectly, and those are the 3
criteria I use when I'm matching for my OAI-PMH imports.

It's basically a record number, a record origin, and a last modified date. When
matching, if the record number and origin match and the inbound record is
newer, update the record. Easy.

Thinking about other metadata schemas...
- Unqualified Dublin Core wouldn't fit that pattern, but that's partially
because it's too simplistic of a schema. Qualified Dublin Core could.
- I don't know BIBFRAME well enough. Looking at
https://www.loc.gov/bibframe/implementation/register.html, I see a
bf:changeDate. I suppose maybe their "Control Number Identifier" or "baseURL"
could be empty since the BIBFRAME URI should be globally unique. Although
BIBFRAME is just a whole other ball game anyway...
-MODS has a recordIdentifier for 001, and recordIdentifier "source" attribute
for 003, and recordChangeDate for 005...

I suppose the reality is that we just support MARC for the time being, and we
have no idea what a post-MARC world will actually look like, so maybe we should
just develop against a MARC world.

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #5 from David Cook <[hidden email]> ---
But in any case... I think there's a difference between the Koha 001 and 003...
and the original 001 and 003.

I think we abuse the 001 and 003 in Koha, and I'm not sure we should develop
functionality to further entrench that abuse.

But I would be interested in a table that tracks record imports better, so that
it would be easier to find a local Koha identifier (ie biblionumber / 999$c)
that was created during an import using the 001 and 003 (or OAI-PMH identifier
and baseUrl).

However, how likely is it for source MARC data to have the 001 and 003 properly
filled out?

Looks like import_biblios has a control_number field, but cleanup_database.pl
deletes all the records from the import_* tables periodically, so that's not
useful.

Anyway, I'll stop spamming now.

I think that I'm probably just being too idealistic.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #6 from Magnus Enger <[hidden email]> ---
(In reply to David Cook from comment #5)
> But in any case... I think there's a difference between the Koha 001 and
> 003... and the original 001 and 003.

Could you expand on this? I thought Koha left these alone, pretty much?

> However, how likely is it for source MARC data to have the 001 and 003
> properly filled out?

In the case of Sweden and Norway: very. And I bet the Germans do it properly
too?

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #7 from Katrin Fischer <[hidden email]> ---
> > However, how likely is it for source MARC data to have the 001 and 003
> > properly filled out?
>
> In the case of Sweden and Norway: very. And I bet the Germans do it properly
> too?

For libraries participating in a union catalog (a lot) 001 and 003 will be
filled out. A lot of libraries not participating still use records from union
catalogs in their cataloguing, so there is probably a high percentage there as
well.

--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

Hans PĆ„lsson <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #8 from Katrin Fischer <[hidden email]> ---
Hm, using ExtractValue is slow, especially on big databases. As Magnus says in
lot of use cases a quick access to 001 and 003 for matching is really helpful.
We have started to map the biblioitems.ean to 001 for use in reports etc. as
the 001 is so important to us. The column is indexed and not mapped by default
in MARC21 - but it remains a bit of a workaround and I'd be happy about a real
controlnumber column.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #9 from David Cook <[hidden email]> ---
(In reply to Magnus Enger from comment #6)
> (In reply to David Cook from comment #5)
> > But in any case... I think there's a difference between the Koha 001 and
> > 003... and the original 001 and 003.
>
> Could you expand on this? I thought Koha left these alone, pretty much?
>

It does but it shouldn't šŸ¤£.

> > However, how likely is it for source MARC data to have the 001 and 003
> > properly filled out?
>
> In the case of Sweden and Norway: very. And I bet the Germans do it properly
> too?

I'm sure that's true, but I suppose I'm thinking about all the libraries (like
all the ones I support) that don't use it.

But I suppose if it's unused that doesn't really create a problem. And if the
database columns were there, maybe we would more highly recommend them to use
the 001 and 003...

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

[Bug 25603] Add 001 and 003 to biblio table

bugzilla-daemon
In reply to this post by bugzilla-daemon
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25603

--- Comment #10 from David Cook <[hidden email]> ---
(In reply to Katrin Fischer from comment #8)
> Hm, using ExtractValue is slow, especially on big databases. As Magnus says
> in lot of use cases a quick access to 001 and 003 for matching is really
> helpful. We have started to map the biblioitems.ean to 001 for use in
> reports etc. as the 001 is so important to us. The column is indexed and not
> mapped by default in MARC21 - but it remains a bit of a workaround and I'd
> be happy about a real controlnumber column.

Hmm, ok, I think I'm convinced.

We're still living in a MARC centric world, so there's no reason not to add it.

And I doubt Koha will ever use the 001 instead of the 999$c, so even if it
isn't ideal from a MARC perspective, it does seem like a practical move.

I know I hate using ExtractValue in databases in the hundreds of thousands or
millions of records O_O.

--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/