[git] configuring Hudson build job for multiple Git repositories

[Matthew Heitz]

Hi,

I need some help configuring Hudson to build my project after its
migration from CVS to Git.

We migrated from one CVS repository to several Git repositories, thanks to
statements like "Typically, one Git repository (.git) is created for each
logical unit, or grouping of code -- a plugin, a connector, a component,
and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git . I want the
code from all of the Git repos to be compiled together by one Hudson job,
but I haven't succeeded yet.

Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
http://wiki.eclipse.org/The_Hudson_Book that "Providing multiple
repositories only makes sense when they are clones or instances off the
same repository, since they will be used for merging together the content
prior to the build." That's not what I'm trying to do. Each repository
contains a distinct set of plugins.

In order for my build to work, the files from all repos must be extracted
to the same directory in the Hudson workspace. But Hudson is removing the
previous contents of the directory each time it fetches files from a repo.
I end up with only the files from the last repo.

I tried telling Hudson to put each repo's files into a different place,
but then javac couldn't find them.

I appreciate any help you can give me.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/

[Eric Gwin]

As I understand it, there'd be a separate build job for each repo. The Jobs should rely upon each other in the needed build order using "spawn another build" or "start after X project". Then the base build kicks off, spawning higher level projects (from a dependency point of view) as it goes.

That's the theory anyway. I'm not sure what other teams do, and I haven't gotten to the point of splitting the builds yet (we needed to maintain all history, so I pretty much fully migrated from svn to Git (1->1), and will be pruning and adding additional repos shortly.

I did do some experimenting though and linking builds seems to work ok. Though I haven't yet figured out how to share workspaces (so test can directly access the built artifact for instance).

HTH

Eric

Hi,
I need some help configuring Hudson to build my project after its migration from CVS to Git.
We migrated from one CVS repository to several Git repositories, thanks to statements like "Typically, one Git repository (.git) is created for each logical unit, or grouping of code -- a plugin, a connector, a component, and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git. I want the code from all of the Git repos to be compiled together by one Hudson job, but I haven't succeeded yet.
Hudson's build config page allows you to specify as many Git repos as you'd like, but I just read in the Hudson Book http://wiki.eclipse.org/The_Hudson_Bookthat "Providing multiple repositories only makes sense when they are clones or instances off the same repository, since they will be used for merging together the content prior to the build." That's not what I'm trying to do. Each repository contains a distinct set of plugins.
In order for my build to work, the files from all repos must be extracted to the same directory in the Hudson workspace. But Hudson is removing the previous contents of the directory each time it fetches files from a repo. I end up with only the files from the last repo.
I tried telling Hudson to put each repo's files into a different place, but then javac couldn't find them.
I appreciate any help you can give me.
-Matt
--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/
_______________________________________________
git mailing list
http://dev.eclipse.org/mailman/listinfo/git

[Matthew Heitz]

Thanks Eric. That solution had occurred to me but I'm not thrilled about
the additional complexity.

Can somebody tell me the benefit of having many Git repositories vs. just
one? My team was working just fine with only one CVS repository.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/






From: Eric Gwin <***@oracle.com>
To: "Git related discussions, specifcally migration help for
projects at Eclipse.org" <***@eclipse.org>,
Date: 07/30/2012 11:25 AM
Subject: Re: [git] configuring Hudson build job for multiple Git
repositories
Sent by: git-***@eclipse.org



As I understand it, there'd be a separate build job for each repo. The
Jobs should rely upon each other in the needed build order using "spawn
another build" or "start after X project". Then the base build kicks off,
spawning higher level projects (from a dependency point of view) as it
goes.

That's the theory anyway. I'm not sure what other teams do, and I haven't
gotten to the point of splitting the builds yet (we needed to maintain all
history, so I pretty much fully migrated from svn to Git (1->1), and will
be pruning and adding additional repos shortly.

I did do some experimenting though and linking builds seems to work ok.
Though I haven't yet figured out how to share workspaces (so test can
directly access the built artifact for instance).

HTH

Eric

On 30/07/2012 10:49 AM, Matthew Heitz wrote:
Hi,

I need some help configuring Hudson to build my project after its
migration from CVS to Git.

We migrated from one CVS repository to several Git repositories, thanks to
statements like "Typically, one Git repository (.git) is created for each
logical unit, or grouping of code -- a plugin, a connector, a component,
and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git . I want the
code from all of the Git repos to be compiled together by one Hudson job,
but I haven't succeeded yet.

Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
http://wiki.eclipse.org/The_Hudson_Book that "Providing multiple
repositories only makes sense when they are clones or instances off the
same repository, since they will be used for merging together the content
prior to the build." That's not what I'm trying to do. Each repository
contains a distinct set of plugins.

In order for my build to work, the files from all repos must be extracted
to the same directory in the Hudson workspace. But Hudson is removing the
previous contents of the directory each time it fetches files from a repo.
I end up with only the files from the last repo.

I tried telling Hudson to put each repo's files into a different place,
but then javac couldn't find them.

I appreciate any help you can give me.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/

[David Carver]

It really comes down to how big your project is, and how it can be built
and used by your community. Anything that is core to the project should
probably be in the same repo. Anything note core, could be moved into
separate repos.

One reason to split things up is if you don't need to build everything
together.

Not having everything together gives you some added flexibility with
your builds, but it also adds complication which your project and
community may not need.

Dave

Post by Matthew Heitz
Thanks Eric. That solution had occurred to me but I'm not thrilled
about the additional complexity.
Can somebody tell me the benefit of having many Git repositories vs.
just one? My team was working just fine with only one CVS repository.
-Matt
--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/
To: "Git related discussions, specifcally migration help for
Date: 07/30/2012 11:25 AM
Subject: Re: [git] configuring Hudson build job for multiple Git
repositories
------------------------------------------------------------------------
As I understand it, there'd be a separate build job for each repo. The
Jobs should rely upon each other in the needed build order using
"spawn another build" or "start after X project". Then the base build
kicks off, spawning higher level projects (from a dependency point of
view) as it goes.
That's the theory anyway. I'm not sure what other teams do, and I
haven't gotten to the point of splitting the builds yet (we needed to
maintain all history, so I pretty much fully migrated from svn to Git
(1->1), and will be pruning and adding additional repos shortly.
I did do some experimenting though and linking builds seems to work
ok. Though I haven't yet figured out how to share workspaces (so test
can directly access the built artifact for instance).
HTH
Eric
Hi,
I need some help configuring Hudson to build my project after its
migration from CVS to Git.
We migrated from one CVS repository to several Git repositories,
thanks to statements like "Typically, one Git repository (.git) is
created for each logical unit, or grouping of code -- a plugin, a
connector, a component, and so on." in
_http://wiki.eclipse.org/Git/Migrating_to_Git_. I want the code from
all of the Git repos to be compiled together by one Hudson job, but I
haven't succeeded yet.
Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
_http://wiki.eclipse.org/The_Hudson_Book_that "Providing multiple
repositories only makes sense when they are clones or instances off
the same repository, since they will be used for merging together the
content prior to the build." That's not what I'm trying to do. Each
repository contains a distinct set of plugins.
In order for my build to work, the files from all repos must be
extracted to the same directory in the Hudson workspace. But Hudson
is removing the previous contents of the directory each time it
fetches files from a repo. I end up with only the files from the last
repo.
I tried telling Hudson to put each repo's files into a different
place, but then javac couldn't find them.
I appreciate any help you can give me.
-Matt
--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer_
__http://www.eclipse.org/edt/__
__http://www.ibm.com/developerworks/rational/products/rbde/_
_______________________________________________
git mailing list
_http://dev.eclipse.org/mailman/listinfo/git_
_______________________________________________
git mailing list
http://dev.eclipse.org/mailman/listinfo/git
_______________________________________________
git mailing list
http://dev.eclipse.org/mailman/listinfo/git

[John Arthorne]

This is off your topic, but I think the suggestion on that wiki page of
having one Git repository per plugin is far too fine-grained. There are
trade-offs either way and I suspect opinions differ. I have added more
information to the wiki page on what those trade-offs are, to help people
make more informed decisions.

John





Matthew Heitz <***@us.ibm.com>
Sent by: git-***@eclipse.org
07/30/2012 10:49 AM
Please respond to
"Git related discussions, specifcally migration help for projects at
Eclipse.org" <***@eclipse.org>


To
***@eclipse.org
cc

Subject
[git] configuring Hudson build job for multiple Git repositories






Hi,

I need some help configuring Hudson to build my project after its
migration from CVS to Git.

We migrated from one CVS repository to several Git repositories, thanks to
statements like "Typically, one Git repository (.git) is created for each
logical unit, or grouping of code -- a plugin, a connector, a component,
and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git . I want the
code from all of the Git repos to be compiled together by one Hudson job,
but I haven't succeeded yet.

Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
http://wiki.eclipse.org/The_Hudson_Book that "Providing multiple
repositories only makes sense when they are clones or instances off the
same repository, since they will be used for merging together the content
prior to the build." That's not what I'm trying to do. Each repository
contains a distinct set of plugins.

In order for my build to work, the files from all repos must be extracted
to the same directory in the Hudson workspace. But Hudson is removing the
previous contents of the directory each time it fetches files from a repo.
I end up with only the files from the last repo.

I tried telling Hudson to put each repo's files into a different place,
but then javac couldn't find them.

I appreciate any help you can give me.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/

[Matthew Heitz]

John,

The new version of the wiki page says very large repositories may be
cumbersome to work with. What's your definition of "very large"?

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/






From: John Arthorne <***@ca.ibm.com>
To: "Git related discussions, specifcally migration help for
projects at Eclipse.org" <***@eclipse.org>,
Date: 07/30/2012 12:57 PM
Subject: Re: [git] configuring Hudson build job for multiple Git
repositories
Sent by: git-***@eclipse.org



This is off your topic, but I think the suggestion on that wiki page of
having one Git repository per plugin is far too fine-grained. There are
trade-offs either way and I suspect opinions differ. I have added more
information to the wiki page on what those trade-offs are, to help people
make more informed decisions.

John




Matthew Heitz <***@us.ibm.com>
Sent by: git-***@eclipse.org
07/30/2012 10:49 AM

Please respond to
"Git related discussions, specifcally migration help for projects
at Eclipse.org" <***@eclipse.org>


To
***@eclipse.org
cc

Subject
[git] configuring Hudson build job for multiple Git repositories








Hi,

I need some help configuring Hudson to build my project after its
migration from CVS to Git.

We migrated from one CVS repository to several Git repositories, thanks to
statements like "Typically, one Git repository (.git) is created for each
logical unit, or grouping of code -- a plugin, a connector, a component,
and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git . I want the
code from all of the Git repos to be compiled together by one Hudson job,
but I haven't succeeded yet.

Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
http://wiki.eclipse.org/The_Hudson_Book that "Providing multiple
repositories only makes sense when they are clones or instances off the
same repository, since they will be used for merging together the content
prior to the build." That's not what I'm trying to do. Each repository
contains a distinct set of plugins.

In order for my build to work, the files from all repos must be extracted
to the same directory in the Hudson workspace. But Hudson is removing the
previous contents of the directory each time it fetches files from a repo.
I end up with only the files from the last repo.

I tried telling Hudson to put each repo's files into a different place,
but then javac couldn't find them.

I appreciate any help you can give me.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/

[John Arthorne]

Well I guess that depends on your definition of "cumbersome" :) If
someone wants to contribute a fix they generally need to clone the entire
repository to their local desktop. So what's a reasonable amount of data
we can expect someone to transfer to their desktop to work on the project?
For example the current Eclipse Platform repositories range from 20MB to
200MB in total size. Now these have 10+ years of history in them so that's
probably larger than typical for their LOC. I think these sizes are ok,
but if you had repositories of 500+ MB I think it would become more
painful to use.

On the flip side, having code changes that span multiple repositories is a
pain to work with. You lose the ability to capture the entire set of
changes in a single URL, the ability to easily cherry-pick/transfer to
other branches, etc. After a couple of years of heavy Git use I'm
definitely in the "less repositories is better" camp.

John




Matthew Heitz <***@us.ibm.com>
Sent by: git-***@eclipse.org
07/30/2012 01:40 PM
Please respond to
"Git related discussions, specifcally migration help for projects at
Eclipse.org" <***@eclipse.org>


To
"Git related discussions, specifcally migration help for projects at
Eclipse.org" <***@eclipse.org>
cc

Subject
Re: [git] configuring Hudson build job for multiple Git repositories






John,

The new version of the wiki page says very large repositories may be
cumbersome to work with. What's your definition of "very large"?

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/






From: John Arthorne <***@ca.ibm.com>
To: "Git related discussions, specifcally migration help for
projects at Eclipse.org" <***@eclipse.org>,
Date: 07/30/2012 12:57 PM
Subject: Re: [git] configuring Hudson build job for multiple Git
repositories
Sent by: git-***@eclipse.org



This is off your topic, but I think the suggestion on that wiki page of
having one Git repository per plugin is far too fine-grained. There are
trade-offs either way and I suspect opinions differ. I have added more
information to the wiki page on what those trade-offs are, to help people
make more informed decisions.

John



Matthew Heitz <***@us.ibm.com>
Sent by: git-***@eclipse.org
07/30/2012 10:49 AM

Please respond to
"Git related discussions, specifcally migration help for projects
at Eclipse.org" <***@eclipse.org>



To
***@eclipse.org
cc

Subject
[git] configuring Hudson build job for multiple Git repositories










Hi,

I need some help configuring Hudson to build my project after its
migration from CVS to Git.

We migrated from one CVS repository to several Git repositories, thanks to
statements like "Typically, one Git repository (.git) is created for each
logical unit, or grouping of code -- a plugin, a connector, a component,
and so on." in http://wiki.eclipse.org/Git/Migrating_to_Git . I want the
code from all of the Git repos to be compiled together by one Hudson job,
but I haven't succeeded yet.

Hudson's build config page allows you to specify as many Git repos as
you'd like, but I just read in the Hudson Book
http://wiki.eclipse.org/The_Hudson_Book that "Providing multiple
repositories only makes sense when they are clones or instances off the
same repository, since they will be used for merging together the content
prior to the build." That's not what I'm trying to do. Each repository
contains a distinct set of plugins.

In order for my build to work, the files from all repos must be extracted
to the same directory in the Hudson workspace. But Hudson is removing the
previous contents of the directory each time it fetches files from a repo.
I end up with only the files from the last repo.

I tried telling Hudson to put each repo's files into a different place,
but then javac couldn't find them.

I appreciate any help you can give me.

-Matt

--------------
Matt Heitz
Developer, Eclipse EGL Development Tools and IBM Rational Business
Developer
http://www.eclipse.org/edt/
http://www.ibm.com/developerworks/rational/products/rbde/

_______________________________________________
git mailing list
***@eclipse.org
http://dev.eclipse.org/mailman/listinfo/git