split source build bits ...

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

split source build bits ...

michael meeks
Hi guys,

        I thought I had sent this, but looking again, I can't see the mail -
apologies if this is a duplicate.

        I'm currently playing with split source builds: by which I mean that we
can slice OO.o into a number of smaller pieces (following kendy's
suggested split), having split the source into chunks with the attached
src-pack2. I also attach my notes on how we're going about this so far.

        Matthias asked how best Sun could help out with this - and of course,
there are a number of somewhat disruptive changes that are hard for us
to test externally pwrt. Sun's internal build system.

        Firstly - scp2: - this module is built very late in the build system,
but - we need the results from it to install ~all our libraries during
the split build. ie. if we build the 'ure' then in order to install the
libs in the right place; we need the scp2 to hand.

        So - request 1: can we split the scp2 into each project ? perhaps into
prj/ such that each module can install itself into the system, as it is
built. [ NB. we could perhaps re-use this to make OO.o run-able from the
solver ;-]. For the straight through build, hopefully the scp2 can be
simply re-aggregated later to run make_installer on it.

        Secondly - i18n: multiplying the already large number of lang-packs by
16 or so is not a good idea: all linux distros go to quite some lengths
to re-aggregate separated i18n packages. If we split the source, we need
to re-aggregate all our i18n work. Of course, currently that is
scattered all over the code-base which (for a split) is not ideal.

        So - request 2: can we move to a more centralised i18n system - whereby
we install un-translated src/hrc files to the solver, then run a
complete translation pass later, to build a single i18n package ? That
might have the added benefit that we can build a standalone
translation-pack, to make it (perhaps) simpler to build localisations -
though this is not my area :-)

        As a 3rd request - we need to patch the settings / target pieces fairly
hard to cope with build from (effectively) two solvers - one in the
local module set and one in the system. Comments on & help getting
changes like that up-stream appreciated - we have a number of (on the
face of them trivial) piece-* patches to tweak things there.

        Thanks,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

src-pack2 (5K) Download Attachment
split.txt (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

Eike Rathke
Hi Michael,

On Friday, 2008-07-18 11:18:20 +0100, Michael Meeks wrote:

> Secondly - i18n: multiplying the already large number of lang-packs by
> 16 or so is not a good idea: all linux distros go to quite some lengths
> to re-aggregate separated i18n packages. If we split the source, we need
> to re-aggregate all our i18n work. Of course, currently that is
> scattered all over the code-base which (for a split) is not ideal.
>
> So - request 2: can we move to a more centralised i18n system - whereby
> we install un-translated src/hrc files to the solver, then run a
> complete translation pass later, to build a single i18n package ? That
> might have the added benefit that we can build a standalone
> translation-pack, to make it (perhaps) simpler to build localisations -
> though this is not my area :-)
Assming you're talking of l10n, not i18n ;-)  effort for
http://qa.openoffice.org/issues/show_bug.cgi?id=79750
is ongoing in
http://eis.services.openoffice.org/EIS2/cws.ShowCWS?Path=DEV300%2Fl10ncleanup

That seems to be a good starting point.

  Eike

--
 OOo/SO Calc core developer. Number formatter stricken i18n transpositionizer.
 SunSign   0x87F8D412 : 2F58 5236 DB02 F335 8304  7D6C 65C9 F9B5 87F8 D412
 OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
 Please don't send personal mail to the [hidden email] account, which I use for
 mailing lists only and don't read from outside Sun. Use [hidden email] Thanks.

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

michael meeks
Hi Eike,

On Wed, 2008-07-23 at 12:19 +0200, Eike Rathke wrote:
> Assming you're talking of l10n, not i18n ;-)

        ;-)

>   effort for
> http://qa.openoffice.org/issues/show_bug.cgi?id=79750
> is ongoing in
> http://eis.services.openoffice.org/EIS2/cws.ShowCWS?Path=DEV300%2Fl10ncleanup

        Wow - yes, that seems ideal.

        Thanks,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

Kay Ramme - Sun Germany - Hamburg
In reply to this post by michael meeks
Hi Michael,

Michael Meeks wrote:
>
> I'm currently playing with split source builds: by which I mean that we
> can slice OO.o into a number of smaller pieces (following kendy's
> suggested split), having split the source into chunks with the attached
> src-pack2. I also attach my notes on how we're going about this so far.
I am going to take a look.
>
> Matthias asked how best Sun could help out with this - and of course,
> there are a number of somewhat disruptive changes that are hard for us
> to test externally pwrt. Sun's internal build system.
OK
>
> Firstly - scp2: - this module is built very late in the build system,
> but - we need the results from it to install ~all our libraries during
> the split build. ie. if we build the 'ure' then in order to install the
> libs in the right place; we need the scp2 to hand.
I know ... ;-)

>
> So - request 1: can we split the scp2 into each project ? perhaps into
> prj/ such that each module can install itself into the system, as it is
> built. [ NB. we could perhaps re-use this to make OO.o run-able from the
> solver ;-]. For the straight through build, hopefully the scp2 can be
> simply re-aggregated later to run make_installer on it.

That is what we have in mind as well ... just give me some more days (I
am on vacation the next two weeks :-), I am than going to present what I
have in mind with a little prototype.

>
> Secondly - i18n: multiplying the already large number of lang-packs by
> 16 or so is not a good idea: all linux distros go to quite some lengths
> to re-aggregate separated i18n packages. If we split the source, we need
> to re-aggregate all our i18n work. Of course, currently that is
> scattered all over the code-base which (for a split) is not ideal.
>
> So - request 2: can we move to a more centralised i18n system - whereby
> we install un-translated src/hrc files to the solver, then run a
> complete translation pass later, to build a single i18n package ? That
> might have the added benefit that we can build a standalone
> translation-pack, to make it (perhaps) simpler to build localisations -
> though this is not my area :-)
>
> As a 3rd request - we need to patch the settings / target pieces fairly
> hard to cope with build from (effectively) two solvers - one in the
> local module set and one in the system. Comments on & help getting
> changes like that up-stream appreciated - we have a number of (on the
> face of them trivial) piece-* patches to tweak things there.

I think others are better prepared to comment on your other points :*)
>
> Thanks,
>
> Michael.

Kind regards


        Kay

>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

michael meeks
Hi Kay,

        I missed you reply I believe; please CC me ;-)

On Wed, 2008-07-23 at 12:49 +0200, Kay Ramme - wrote:
> Michael Meeks wrote:
> > I'm currently playing with split source builds: by which I mean that we
> > can slice OO.o into a number of smaller pieces (following kendy's
> > suggested split), having split the source into chunks with the attached
> > src-pack2. I also attach my notes on how we're going about this so far.
>
> I am going to take a look.

        So - in fact this now works quite nicely for me; I have a set of RPMs
that build separately & run much of the OO.o.

        http://svn.gnome.org/viewvc/ooo-build/trunk/patches/dev300/

        the piece-* patches are they, those piece-* can all be JCA'd etc. but
are likely only to be useful as pointers in their current form.

> > Firstly - scp2: - this module is built very late in the build system,
> > but - we need the results from it to install ~all our libraries during
> > the split build. ie. if we build the 'ure' then in order to install the
> > libs in the right place; we need the scp2 to hand.
>
> I know ... ;-)

        So - having just banged through make_installer.pl and co. once again -
I'm really beginning to loathe it ;-) Anyhow.

        Here is what I suggest we do, as a way ahead: kill several birds with
one stone:

        * merge a variant of the above patches
        * split the scp2 into the individual modules
!! * kill half of 'deliver.pl'
                + split the solver into two:
                        + install-set layout image
                        + solver/ layout image
                + make the install-set layout image from the scp2 &
                  an (accelerated) make_installer.pl

                [ this is effectively what I do ]

        This has a number of rather obvious benefits:
                + we can easily split the code any-way we like later
                + we have a "run-able" OO.o even in all-the-way through
                  OO.o: should save space, and pain for developers.
                + we can trivially build pieces against an
                  installed OO.o

        Failing that, it's necessary to take something like the piece-*
patches, but re-work each of them, and add a lot of conditionals to make
them work both in all-the-way-through mode and also in built-vs-system
mode - which is pain, complexity and so on.

        So - the question is: is there any good reason why we would not want
the 'live' half of the solver to look like an install set ? in the
abstract - if we wanted it to look like something else of course it
should be easy enough to do that just by tweaking the include paths in
foootherproduct.lst such that make_installer.pl can build that too.

        HTH,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

stephan.bergmann
In reply to this post by michael meeks
On 08/15/08 15:59, Michael Meeks wrote:

> Here is what I suggest we do, as a way ahead: kill several birds with
> one stone:
>
> * merge a variant of the above patches
> * split the scp2 into the individual modules
> !! * kill half of 'deliver.pl'
> + split the solver into two:
> + install-set layout image
> + solver/ layout image
> + make the install-set layout image from the scp2 &
>  an (accelerated) make_installer.pl

Not sure I understand you.  Is it the following?
- Have two solver parts (solverA, solverB) instead of single solver.
- In deliver, copy those files that will be part of an OOo installation
into solverA (at the same relative path as in an OOo installation), and
copy other files (i.e., those that are only needed during later stages
of building OOo) into solverB.
- After a build of OOo, solverA contains an OOo installation.

-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

michael meeks
Hi Stephan,

On Fri, 2008-08-15 at 16:35 +0200, Stephan Bergmann wrote:
> Not sure I understand you.  Is it the following?

        Yes - correct :-) is there something obviously silly about this ?

> - Have two solver parts (solverA, solverB) instead of single solver.
> - In deliver, copy those files that will be part of an OOo installation
> into solverA (at the same relative path as in an OOo installation), and
> copy other files (i.e., those that are only needed during later stages
> of building OOo) into solverB.
> - After a build of OOo, solverA contains an OOo installation.

        Right. This is basically what I'm doing now, and with:

solverA: /opt/openoffice-3.0 and
solverB: /opt/lib/somewhere-not-easy-to-find-to-please-mhu/solver ;-)

        for solverB ;-) it turns into a build against the system, and with:

solverA: /home/scratch/ooo300-m1/solver-inst
solverB: /home/scratch/ooo300-m1/solver

        it can be an 'old-style' straight through build suitable for Win32,
people-with-lots-of-space-and-CPU etc. :-)

        HTH,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

stephan.bergmann
On 08/15/08 17:03, Michael Meeks wrote:
> Hi Stephan,
>
> On Fri, 2008-08-15 at 16:35 +0200, Stephan Bergmann wrote:
>> Not sure I understand you.  Is it the following?
>
> Yes - correct :-) is there something obviously silly about this ?

No, sounds very reasonable to me.  (For example, we can get rid of
setting LD_LIBRARY_PATH or equivalent in the build environment, then.)

-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

michael meeks

On Fri, 2008-08-15 at 17:12 +0200, Stephan Bergmann wrote:
> > On Fri, 2008-08-15 at 16:35 +0200, Stephan Bergmann wrote:
> >> Not sure I understand you.  Is it the following?
> >
> > Yes - correct :-) is there something obviously silly about this ?
>
> No, sounds very reasonable to me.  (For example, we can get rid of
> setting LD_LIBRARY_PATH or equivalent in the build environment, then.)

        Sure - and, I guess as a side-effect - the more we build up the install
set as we go, the more easily we can have unit tests that run nicely
against a partially working install.

        The last sticking point for something like that is the monolithic
services.rdb build; I hate small scattered files - but, IMHO there is a
good argument for a very small, simple text file per module (or group of
modules) that expresses the service info in a simple and more efficient
fashion; and that (if necessary) can be catted together. Personally I
find this sort of thing rather amusing (in a black sort of a way):

$ regview services.rdb > services.txt # ...
$ ls -lh services.*
-r--r--r-- 1 michael users 2.4M 2008-08-28 10:30 services.rdb
-r--r--r-- 1 michael users 540K 2008-08-28 10:30 services.rdb.gz
-rw-r--r-- 1 michael users 638K 2008-08-28 10:31 services.txt
-rw-r--r-- 1 michael users  46K 2008-08-28 10:31 services.txt.gz

        And of course - this is after the not-merged-up-stream size
optimisations we do in store/ - the up-stream wastage is far worse here.
It would be far nicer & smaller to have components register themselves
via a simple flat text file IMHO.

        HTH,

                Michael.

--
 [hidden email]  <><, Pseudo Engineer, itinerant idiot


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: split source build bits ...

stephan.bergmann
In reply to this post by michael meeks
On 08/28/08 11:59, Michael Meeks wrote:

> The last sticking point for something like that is the monolithic
> services.rdb build; I hate small scattered files - but, IMHO there is a
> good argument for a very small, simple text file per module (or group of
> modules) that expresses the service info in a simple and more efficient
> fashion; and that (if necessary) can be catted together. Personally I
> find this sort of thing rather amusing (in a black sort of a way):
>
> $ regview services.rdb > services.txt # ...
> $ ls -lh services.*
> -r--r--r-- 1 michael users 2.4M 2008-08-28 10:30 services.rdb
> -r--r--r-- 1 michael users 540K 2008-08-28 10:30 services.rdb.gz
> -rw-r--r-- 1 michael users 638K 2008-08-28 10:31 services.txt
> -rw-r--r-- 1 michael users  46K 2008-08-28 10:31 services.txt.gz
>
> And of course - this is after the not-merged-up-stream size
> optimisations we do in store/ - the up-stream wastage is far worse here.
> It would be far nicer & smaller to have components register themselves
> via a simple flat text file IMHO.

Yes, any---compatible---simplifications in this area are very welcome, IMO.

-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]