Repack git repo

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Repack git repo

Jonathan Druart
Hi devs,

Some of you may know, we got an issue with our Koha gitlab repo, it
was not syncing because of too big (no detail here, but that did not
really make sense anyway). It seems that it is syncing now but we
removed all the branches that are not protected (so only master)

I had a new try at splitting our repo, removing the po files from the
core repo (https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking).
It's not a trivial process as it requires several changes, but I
noticed that:

After a git clone: `git count-objects -vH` returns 3.67/4.23 GiB
(cannot remember the correct number, but high!)
After a repack [1]: ~755 MiB

The command I used: `git gc --aggressive --prune=now`

I think we should schedule that command on the git server (it took
around 3 hours on a small VPS) to reduce its size.

I am available if you need any helps for that task.

Cheers,
Jonathan

PS: you may need to increase the windowMemory value with: `git config
--global pack.windowMemory 256m`

[1] https://git-scm.com/docs/git-repack

Spoiler: After removing the po file, and a repack, the repo index size is 127MiB
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Repack git repo

Liz Rea-2
Ok, what time of day in UTC do you like for this?

On Tue, Apr 28, 2020 at 3:19 AM Jonathan Druart <[hidden email]> wrote:
Hi devs,

Some of you may know, we got an issue with our Koha gitlab repo, it
was not syncing because of too big (no detail here, but that did not
really make sense anyway). It seems that it is syncing now but we
removed all the branches that are not protected (so only master)

I had a new try at splitting our repo, removing the po files from the
core repo (https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking).
It's not a trivial process as it requires several changes, but I
noticed that:

After a git clone: `git count-objects -vH` returns 3.67/4.23 GiB
(cannot remember the correct number, but high!)
After a repack [1]: ~755 MiB

The command I used: `git gc --aggressive --prune=now`

I think we should schedule that command on the git server (it took
around 3 hours on a small VPS) to reduce its size.

I am available if you need any helps for that task.

Cheers,
Jonathan

PS: you may need to increase the windowMemory value with: `git config
--global pack.windowMemory 256m`

[1] https://git-scm.com/docs/git-repack

Spoiler: After removing the po file, and a repack, the repo index size is 127MiB
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Repack git repo

Jonathan Druart
My feeling would be to:
0. stop pushing to the repo, so wait for the RM and RMaints to confirm
they are aware of the task
1. copy the original repo directory (backup, just in case)
2.`git gc --aggressive --prune=now`

I personally don't have preference about the time.


Le mar. 28 avr. 2020 à 17:28, Liz Rea <[hidden email]> a écrit :

>
> Ok, what time of day in UTC do you like for this?
>
> On Tue, Apr 28, 2020 at 3:19 AM Jonathan Druart <[hidden email]> wrote:
>>
>> Hi devs,
>>
>> Some of you may know, we got an issue with our Koha gitlab repo, it
>> was not syncing because of too big (no detail here, but that did not
>> really make sense anyway). It seems that it is syncing now but we
>> removed all the branches that are not protected (so only master)
>>
>> I had a new try at splitting our repo, removing the po files from the
>> core repo (https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking).
>> It's not a trivial process as it requires several changes, but I
>> noticed that:
>>
>> After a git clone: `git count-objects -vH` returns 3.67/4.23 GiB
>> (cannot remember the correct number, but high!)
>> After a repack [1]: ~755 MiB
>>
>> The command I used: `git gc --aggressive --prune=now`
>>
>> I think we should schedule that command on the git server (it took
>> around 3 hours on a small VPS) to reduce its size.
>>
>> I am available if you need any helps for that task.
>>
>> Cheers,
>> Jonathan
>>
>> PS: you may need to increase the windowMemory value with: `git config
>> --global pack.windowMemory 256m`
>>
>> [1] https://git-scm.com/docs/git-repack
>>
>> Spoiler: After removing the po file, and a repack, the repo index size is 127MiB
>> _______________________________________________
>> Koha-devel mailing list
>> [hidden email]
>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> website : http://www.koha-community.org/
>> git : http://git.koha-community.org/
>> bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Repack git repo

Mark Tompsett
In reply to this post by Jonathan Druart
Greetings,

I checked out the wiki page. Thank you for typing that up, Jonathan.

I vote for keeping tags (hey GB to MB -- we can keep tags), and I really like:

It's still pretty clean in terms of getting all of Koha.

GPML,
Mark Tompsett

_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Repack git repo

Jonathan Druart
The wiki page is outdated, I will try and update it soon. I managed to
get rid of po files and l kept the history and tags (using BFG instead
of git filter-branch).

Le mar. 28 avr. 2020 à 20:02, Mark Tompsett <[hidden email]> a écrit :

>
> Greetings,
>
> I checked out the wiki page. Thank you for typing that up, Jonathan.
>
> I vote for keeping tags (hey GB to MB -- we can keep tags), and I really like:
> https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking#Howto_use_Koha_main_repo_with_localization_repo
>
> It's still pretty clean in terms of getting all of Koha.
>
> GPML,
> Mark Tompsett
> _______________________________________________
> Koha-devel mailing list
> [hidden email]
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Fwd: Repack git repo

Liz Rea-2

Hi all,

As the release is now done, yesterday ~3pm Central Standard Time I ran a repack on the Koha repo. The repo is now 1.03G in size, down from 4.6G.

There is more work to be done on that server - namely it's got a old and busted (and cantankerous!) OS, and thusly an old and busted (and cantankerous!) git version. It will need an upgrade at least, and it may end up being a server move.

I don't have a timeline for any of that, but will aim to make it as unimpactful as I can when it happens, and will let everybody know closer to the time.

Cheers,
Liz



On Tue, Apr 28, 2020 at 4:57 PM Jonathan Druart <[hidden email]> wrote:
The wiki page is outdated, I will try and update it soon. I managed to
get rid of po files and l kept the history and tags (using BFG instead
of git filter-branch).

Le mar. 28 avr. 2020 à 20:02, Mark Tompsett <[hidden email]> a écrit :
>
> Greetings,
>
> I checked out the wiki page. Thank you for typing that up, Jonathan.
>
> I vote for keeping tags (hey GB to MB -- we can keep tags), and I really like:
> https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking#Howto_use_Koha_main_repo_with_localization_repo
>
> It's still pretty clean in terms of getting all of Koha.
>
> GPML,
> Mark Tompsett
> _______________________________________________
> Koha-devel mailing list
> [hidden email]
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Repack git repo

Tomas Cohen Arazi
Awesome, Liz!

El mié., 3 jun. 2020 a las 14:54, Liz Rea (<[hidden email]>) escribió:

Hi all,

As the release is now done, yesterday ~3pm Central Standard Time I ran a repack on the Koha repo. The repo is now 1.03G in size, down from 4.6G.

There is more work to be done on that server - namely it's got a old and busted (and cantankerous!) OS, and thusly an old and busted (and cantankerous!) git version. It will need an upgrade at least, and it may end up being a server move.

I don't have a timeline for any of that, but will aim to make it as unimpactful as I can when it happens, and will let everybody know closer to the time.

Cheers,
Liz



On Tue, Apr 28, 2020 at 4:57 PM Jonathan Druart <[hidden email]> wrote:
The wiki page is outdated, I will try and update it soon. I managed to
get rid of po files and l kept the history and tags (using BFG instead
of git filter-branch).

Le mar. 28 avr. 2020 à 20:02, Mark Tompsett <[hidden email]> a écrit :
>
> Greetings,
>
> I checked out the wiki page. Thank you for typing that up, Jonathan.
>
> I vote for keeping tags (hey GB to MB -- we can keep tags), and I really like:
> https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking#Howto_use_Koha_main_repo_with_localization_repo
>
> It's still pretty clean in terms of getting all of Koha.
>
> GPML,
> Mark Tompsett
> _______________________________________________
> Koha-devel mailing list
> [hidden email]
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/


--
Tomás Cohen Arazi
Theke Solutions (http://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F

_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Repack git repo

Jonathan Druart
In reply to this post by Liz Rea-2
Thanks Liz!
I git clone the "new" repo, then redo the git gc, and tadaaa

$ git gc --aggressive --prune=now
Counting objects: 590241, done.
Compressing objects: 100% (571643/571643), done.
Writing objects: 100% (590241/590241), done.
Total 590241 (delta 459721), reused 130510 (delta 0)
Checking connectivity: 590241, done.

$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 590241
packs: 1
size-pack: 759.32 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

So yes, we must upgrade the server to get a new version of git (if
that makes the difference).

How can we help? Who can provide a server for that?

Le mer. 3 juin 2020 à 19:53, Liz Rea <[hidden email]> a écrit :

>
>
> Hi all,
>
> As the release is now done, yesterday ~3pm Central Standard Time I ran a repack on the Koha repo. The repo is now 1.03G in size, down from 4.6G.
>
> There is more work to be done on that server - namely it's got a old and busted (and cantankerous!) OS, and thusly an old and busted (and cantankerous!) git version. It will need an upgrade at least, and it may end up being a server move.
>
> I don't have a timeline for any of that, but will aim to make it as unimpactful as I can when it happens, and will let everybody know closer to the time.
>
> Cheers,
> Liz
>
>
>
> On Tue, Apr 28, 2020 at 4:57 PM Jonathan Druart <[hidden email]> wrote:
>>
>> The wiki page is outdated, I will try and update it soon. I managed to
>> get rid of po files and l kept the history and tags (using BFG instead
>> of git filter-branch).
>>
>> Le mar. 28 avr. 2020 à 20:02, Mark Tompsett <[hidden email]> a écrit :
>> >
>> > Greetings,
>> >
>> > I checked out the wiki page. Thank you for typing that up, Jonathan.
>> >
>> > I vote for keeping tags (hey GB to MB -- we can keep tags), and I really like:
>> > https://wiki.koha-community.org/wiki/Git_Splitting_and_Shrinking#Howto_use_Koha_main_repo_with_localization_repo
>> >
>> > It's still pretty clean in terms of getting all of Koha.
>> >
>> > GPML,
>> > Mark Tompsett
>> > _______________________________________________
>> > Koha-devel mailing list
>> > [hidden email]
>> > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> > website : http://www.koha-community.org/
>> > git : http://git.koha-community.org/
>> > bugs : http://bugs.koha-community.org/
>> _______________________________________________
>> Koha-devel mailing list
>> [hidden email]
>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> website : http://www.koha-community.org/
>> git : http://git.koha-community.org/
>> bugs : http://bugs.koha-community.org/
>
> _______________________________________________
> Koha-devel mailing list
> [hidden email]
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
[hidden email]
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/