New Build System Requirements

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

New Build System Requirements

Björn Michaelsen-2

Hi List,

I started a new wikipage here:
http://wiki.services.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements

collecting the major requirements for a new build system and what needs
to be done to implement these requirements with GNU make and CMake
(those two currently seem to be the only serious contestants). If you
find additional requirements, feel free to add them to the page.

Best Regards,

Bjoern

--
===========================================================================
 Sitz der Gesellschaft:
 Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
 Amtsgericht München: HRB 161028
 Geschäftsführer: Thomas Schröder, Wolfgang Engels
 Vorsitzender des Aufsichtsrates: Martin Häring
===========================================================================


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Jussi Pakkanen-2
On Mon, Feb 1, 2010 at 3:09 PM, bjoern michaelsen - Sun Microsystems -
Hamburg Germany <[hidden email]> wrote:

> collecting the major requirements for a new build system and what needs
> to be done to implement these requirements with GNU make and CMake
> (those two currently seem to be the only serious contestants). If you
> find additional requirements, feel free to add them to the page.

There are some inconsistencies with the page. I'll post them here
rather than directly to the page to prevent an editing war.

Firstly under dependencies it says "[CMake] is not usually available
on the default install of many platforms." and then a bit further down
"One might consider using Python as a tool for _all_ non-standard
build tasks. Python is easy to distribute and install on all
platforms." This is somewhat conflicting: CMake is just as available
and easy to install as Python on every (meaningful) platform.

Another point is "However, there are many cases where the current
build process depends on a large set of external tools like bash, awk,
findutils, coreutils." Does the actual building require these or is it
more of a case of "need these, because dmake/build.pl needs these"? In
my OOo CMake experiment I built idlc, generated urd, Java etc all
without needing the tools listed. Perhaps someone could post an
example from current build where these are used.

If one is going to go the CMake route, the long term smart thing to do
is probably to convert these existing things into CMake's internal
syntax. It has quite a rich library of stuff. Yes, this is more work
but the good news is that you don't have to do it at once. First you
can replicate the old build system by calling the awk scripts and what
have you and then replace them one by one. With CMake macros and some
discipline this is straightforward.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

stephan.bergmann
On 02/02/10 10:05, Jussi Pakkanen wrote:
> Another point is "However, there are many cases where the current
> build process depends on a large set of external tools like bash, awk,
> findutils, coreutils." Does the actual building require these or is it
> more of a case of "need these, because dmake/build.pl needs these"? In
> my OOo CMake experiment I built idlc, generated urd, Java etc all
> without needing the tools listed. Perhaps someone could post an
> example from current build where these are used.

One example that comes to mind is solenv/inc/tg_shl.mk calling
solenv/bin/addsym.awk on Linux and solenv/bin/addsym-macosx.sh on Mac OS
X, each needed to adjust the generic linker version map files checked
into the OOo code base to the specific needs of those respective platforms.

> If one is going to go the CMake route, the long term smart thing to do
> is probably to convert these existing things into CMake's internal
> syntax. It has quite a rich library of stuff. Yes, this is more work
> but the good news is that you don't have to do it at once. First you
> can replicate the old build system by calling the awk scripts and what
> have you and then replace them one by one. With CMake macros and some
> discipline this is straightforward.

Sounds good.

-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Björn Michaelsen-2
In reply to this post by Jussi Pakkanen-2
On Tue, 02 Feb 2010 11:05:26 +0200
Jussi Pakkanen <[hidden email]> wrote:

> There are some inconsistencies with the page. I'll post them here
> rather than directly to the page to prevent an editing war.
>
> Firstly under dependencies it says "[CMake] is not usually available
> on the default install of many platforms." and then a bit further down
> "One might consider using Python as a tool for _all_ non-standard
> build tasks. Python is easy to distribute and install on all
> platforms." This is somewhat conflicting: CMake is just as available
> and easy to install as Python on every (meaningful) platform.
Did you miss the line "However, it is also a very fat dependency and
thus it is questionable, if it would be worth the effort just to
get rid of cygwin (or another Unix environment on windows)." just
following the sentence? ;)
I am not too comfortable to introduce Python as a dep for the build
system, but I would consider it as an option, if one can get completely
rid of the dep on the unix environment currently used (including sh,
awk, sed, find ...). Python would be the toolset not the build system
itself. (see discussion below)

> If one is going to go the CMake route, the long term smart thing to do
> is probably to convert these existing things into CMake's internal
> syntax. It has quite a rich library of stuff. Yes, this is more work
> but the good news is that you don't have to do it at once. First you
> can replicate the old build system by calling the awk scripts and what
> have you and then replace them one by one. With CMake macros and some
> discipline this is straightforward.
There are some things that are currently done in the build that will
not be translatable to cmake syntax (sed/awk are rather expressive and
powerful). Stephan gave you an example, but those are certainly not
the only ones. The OOo build is huge and you can be certain that there
are some "Unknown Unknowns" hidden in it.

As for the toolset the following combinations are thinkable:
GNU make + Unix Tools (sh, awk, sed ... - cygwin on windows)
GNU make + Python (native on windows/unix)
CMake + Unix Tools (sh, awk, sed ... - cygwin on windows)
CMake + Python (native on windows/unix)
CMake + custom tools (written in C/C++)

Currently we have:
Perl + build.pl and friends (like mkout.pl) + dmake + Unix tools +
custom tools (like rscdep)

There is a wide agreement, that we want to get rid of as much
"homegrown tools" as possible and that pretty much rules out the
"custom tools" idea.

BTW, _If_ one depends on Python anyway, other options become available
too. The waf build system for example:
http://code.google.com/p/waf/
But as with CMake, one has to ask oneself if there are any real life
benefits making it worth to deviate from the conservative choice (which
currently certainly is GNU make).


Best Regards,

Bjoern


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Jussi Pakkanen-2
On Tue, Feb 2, 2010 at 1:08 PM, bjoern michaelsen - Sun Microsystems -
Hamburg Germany <[hidden email]> wrote:

>> Firstly under dependencies it says "[CMake] is not usually available
>> on the default install of many platforms." and then a bit further down
>> "One might consider using Python as a tool for _all_ non-standard
>> build tasks. Python is easy to distribute and install on all
>> platforms." This is somewhat conflicting: CMake is just as available
>> and easy to install as Python on every (meaningful) platform.
>
> Did you miss the line "However, it is also a very fat dependency and
> thus it is questionable, if it would be worth the effort just to
> get rid of cygwin (or another Unix environment on windows)." just
> following the sentence? ;)

No I did not, but despite that it somewhat harsh to imply that CMake
is hard to get or install. Getting it is roughly as hard as getting a
compiler (buying full registered versions of MSVC notwithstanding).

> As for the toolset the following combinations are thinkable:
> GNU make + Unix Tools (sh, awk, sed ... - cygwin on windows)
> GNU make + Python (native on windows/unix)
> CMake + Unix Tools (sh, awk, sed ... - cygwin on windows)
> CMake + Python (native on windows/unix)
> CMake + custom tools (written in C/C++)

There is also "Pure CMake". But as I said earlier, it is probably
smarter to first go "CMake + Unix" and then transition to pure CMake.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Björn Michaelsen-2
On Tue, 02 Feb 2010 16:21:24 +0200
Jussi Pakkanen <[hidden email]> wrote:
> No I did not, but despite that it somewhat harsh to imply that CMake
> is hard to get or install. Getting it is roughly as hard as getting a
> compiler (buying full registered versions of MSVC notwithstanding).
On unix/linux this point is indeed pretty much moot as those all have a
decent package management, making the installation of all prerequisites
a simple oneliner. As always, Windows is still in need of special care.
Just take a look at:
http://wiki.services.openoffice.org/wiki/Documentation/Building_Guide/Building_on_Windows#software_requirements
The amount of click-trough installers needed to get a build environment
on Windows is disgusting. Yes, CMake is trivial to install on current
platforms, but this stuff keeps adding up. Of course, currently cygwin
makes the installation of prerequisites quite easy.
And then there is the point of portability: While CMake is portable,
GNU make will be on any platform you can dream up ASAP as it is part
of the GNU toolchain and gcc is build with it.

> There is also "Pure CMake". But as I said earlier, it is probably
> smarter to first go "CMake + Unix" and then transition to pure CMake.
You are severely underestimating the dark magic done in the build
system (mostly concentrated in a few "ugly" modules"), if you believe
all current ops can be done in "Pure CMake". There are some pretty
mystical textprocessing operations done using sed, awk, perl etc.
throughout the build. Going for "Pure CMake" is going to be a lot of
work and is spoiled if it cant be completed as there is little worth in
"mostly pure CMake". To request resources for such a project on the
prospect that it "will probably work out" is unlikely to work out
with any of the big supporters of OOo.

BR,

Bjoern

--
===========================================================================
 Sitz der Gesellschaft:
 Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
 Amtsgericht München: HRB 161028
 Geschäftsführer: Thomas Schröder, Wolfgang Engels
 Vorsitzender des Aufsichtsrates: Martin Häring
===========================================================================


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Mathias Bauer
In reply to this post by Jussi Pakkanen-2
Jussi Pakkanen wrote:

> Another point is "However, there are many cases where the current
> build process depends on a large set of external tools like bash, awk,
> findutils, coreutils." Does the actual building require these or is it
> more of a case of "need these, because dmake/build.pl needs these"? In
> my OOo CMake experiment I built idlc, generated urd, Java etc all
> without needing the tools listed. Perhaps someone could post an
> example from current build where these are used.

By chance ;-) I have collected a list of modules that currently leave
the simple path of "just some c++/Java/src/idl compiling and that's it".
I think we should investigate their current makefiles and see how they
translate into systems using one of the possible candidates. Tomorrow I
will add this list to the wiki page Björn has started.

Regards,
Mathias

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "[hidden email]".
I use it for the OOo lists and only rarely read other mails sent to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Jussi Pakkanen-2
In reply to this post by Björn Michaelsen-2
On Tue, Feb 2, 2010 at 5:32 PM, bjoern michaelsen - Sun Microsystems -
Hamburg Germany <[hidden email]> wrote:

>> There is also "Pure CMake". But as I said earlier, it is probably
>> smarter to first go "CMake + Unix" and then transition to pure CMake.
>
> You are severely underestimating the dark magic done in the build
> system (mostly concentrated in a few "ugly" modules"), if you believe
> all current ops can be done in "Pure CMake". There are some pretty
> mystical textprocessing operations done using sed, awk, perl etc.
> throughout the build.

I want to note out the following so no-one will think that I am a
religious zealot:

CMake *will* take more work up front than GNU Make (which, as I
understand, already works).

It may be a lot of work, it may be a sh*tload of work.

The question is, whether it is worth it. This is a question of
analysis and (especially) testing.

> Going for "Pure CMake" is going to be a lot of
> work and is spoiled if it cant be completed as there is little worth in
> "mostly pure CMake". To request resources for such a project on the
> prospect that it "will probably work out" is unlikely to work out
> with any of the big supporters of OOo.

I don't think you can ever get rid of Cygwin completely. At the very
least you need Flex and Bison and the easiest way to install them is
to use Cygwin. Then you might as well install other tools as well.
"Pure CMake" was mostly listed for completeness. I guess it should
have said that transitioning to it can be done as an independent step,
at a later time, and only if it is deemed necessary at the time. And
most importantly: you get CMake's benefits even if you never do it at
all.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Mathias Bauer
Jussi Pakkanen wrote:

> On Tue, Feb 2, 2010 at 5:32 PM, bjoern michaelsen - Sun Microsystems -
> Hamburg Germany <[hidden email]> wrote:
>
>>> There is also "Pure CMake". But as I said earlier, it is probably
>>> smarter to first go "CMake + Unix" and then transition to pure CMake.
>>
>> You are severely underestimating the dark magic done in the build
>> system (mostly concentrated in a few "ugly" modules"), if you believe
>> all current ops can be done in "Pure CMake". There are some pretty
>> mystical textprocessing operations done using sed, awk, perl etc.
>> throughout the build.
>
> I want to note out the following so no-one will think that I am a
> religious zealot:
>
> CMake *will* take more work up front than GNU Make (which, as I
> understand, already works).

It works for some modules, yes. Amongst others for "sw", so we have more
or less covered everything that you can find in "normal" modules.
We still have to do a lot of work for all the modules that are
"different". Please see my mail about the module build targets
investigation.

> It may be a lot of work, it may be a sh*tload of work.

The advantage of CMake is that we have to code only for one "platform":
CMake. With GNUMake we surely have to do more "porting". The advantage
of GNU Make is in the troubleshooting.

>> Going for "Pure CMake" is going to be a lot of
>> work and is spoiled if it cant be completed as there is little worth in
>> "mostly pure CMake". To request resources for such a project on the
>> prospect that it "will probably work out" is unlikely to work out
>> with any of the big supporters of OOo.
>
> I don't think you can ever get rid of Cygwin completely. At the very
> least you need Flex and Bison and the easiest way to install them is
> to use Cygwin. Then you might as well install other tools as well.
> "Pure CMake" was mostly listed for completeness. I guess it should
> have said that transitioning to it can be done as an independent step,
> at a later time, and only if it is deemed necessary at the time. And
> most importantly: you get CMake's benefits even if you never do it at
> all.

I tend to agree that - whatever we will do - we probably won't get rid
of cygwin. Where we have to get rid of it is the building of the
"normal" modules.

As an example, if a developer is going to work on "sw", he might want to
build sw and all c++ libraries it depends on (well, without the external
ones). Being able to do that in Visual Studio would be a tremendeous
achievement. But using Visual Studio as a "launcher" for cygwin shells
is probably not what we want here. So the build of "normal" C++
libraries should run inside VS completely.

Building other modules like extras, helpcontent etc. with Cygwin won't
hurt in this scenario. But it might have some negative influence on the
performance of a complete build if vcbuild is used as the "launcher" for
several cygwin shells that we will need to build the "special" modules.

Regards,
Mathias

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "[hidden email]".
I use it for the OOo lists and only rarely read other mails sent to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Mathias Bauer
In reply to this post by Mathias Bauer
Mathias Bauer wrote:

> Jussi Pakkanen wrote:
>
>> Another point is "However, there are many cases where the current
>> build process depends on a large set of external tools like bash, awk,
>> findutils, coreutils." Does the actual building require these or is it
>> more of a case of "need these, because dmake/build.pl needs these"? In
>> my OOo CMake experiment I built idlc, generated urd, Java etc all
>> without needing the tools listed. Perhaps someone could post an
>> example from current build where these are used.
>
> By chance ;-) I have collected a list of modules that currently leave
> the simple path of "just some c++/Java/src/idl compiling and that's it".
> I think we should investigate their current makefiles and see how they
> translate into systems using one of the possible candidates. Tomorrow I
> will add this list to the wiki page Björn has started.

Done.

I hope that by investigating the makefiles of the modules at the top of
the table we can identify the "dark magic" Björn has mentioned.

Regards,
Mathias

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "[hidden email]".
I use it for the OOo lists and only rarely read other mails sent to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

stephan.bergmann
In reply to this post by Mathias Bauer
On 02/03/10 15:38, Mathias Bauer wrote:

> I tend to agree that - whatever we will do - we probably won't get rid
> of cygwin. Where we have to get rid of it is the building of the
> "normal" modules.
>
> As an example, if a developer is going to work on "sw", he might want to
> build sw and all c++ libraries it depends on (well, without the external
> ones). Being able to do that in Visual Studio would be a tremendeous
> achievement. But using Visual Studio as a "launcher" for cygwin shells
> is probably not what we want here. So the build of "normal" C++
> libraries should run inside VS completely.

Not sure what your scenario is where the developer "might want to build
[just] sw and all c++ libraries it depends on," but doing all of that
strictly inside VS would not work if any of those C++ libraries in turn
depended on something not buildable strictly inside VS (e.g., thinking
of things like flex/bison/idlc/cppumaker/whatever invocations).

-Stephan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Mathias Bauer
Stephan Bergmann wrote:

> On 02/03/10 15:38, Mathias Bauer wrote:
>> I tend to agree that - whatever we will do - we probably won't get rid
>> of cygwin. Where we have to get rid of it is the building of the
>> "normal" modules.
>>
>> As an example, if a developer is going to work on "sw", he might want to
>> build sw and all c++ libraries it depends on (well, without the external
>> ones). Being able to do that in Visual Studio would be a tremendeous
>> achievement. But using Visual Studio as a "launcher" for cygwin shells
>> is probably not what we want here. So the build of "normal" C++
>> libraries should run inside VS completely.
>
> Not sure what your scenario is where the developer "might want to build
> [just] sw and all c++ libraries it depends on," but doing all of that
> strictly inside VS would not work if any of those C++ libraries in turn
> depended on something not buildable strictly inside VS (e.g., thinking
> of things like flex/bison/idlc/cppumaker/whatever invocations).

Well, "all" was perhaps too much. I was aiming on what you usually work
with. A lot of common build scenarios require nothing special, just the
"usual suspects". Many modules could be build with Visual Studio.

That's nothing for a complete build, but very useful for the daily work
of the developers. That would definitely be a benefit.

Regards,
Mathias

--
Mathias Bauer (mba) - Project Lead OpenOffice.org Writer
OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS
Please don't reply to "[hidden email]".
I use it for the OOo lists and only rarely read other mails sent to it.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New Build System Requirements

Jussi Pakkanen-2
In reply to this post by Mathias Bauer
On Wed, Feb 3, 2010 at 4:43 PM, Mathias Bauer <[hidden email]> wrote:

> I hope that by investigating the makefiles of the modules at the top of
> the table we can identify the "dark magic" Björn has mentioned.

According to a cursory look most of those seem to be of the type "call
some command with a list of files". The external command can be
external (such as zip) or an internal script file, usually in Perl.

These are straightforward to port to CMake, though will obviously take
some work. The calls are similar to invocations of Flex or Bison or a
self-compiled exe such as idlc.

The most complex thing I found was the versioning thing explained in
cli_ure/readme.txt. A couple of iterations through the text and
makefiles did not make it clear to me what it is supposed to do.
However:

1) CMake has support for library versioning. It may or may not be the
same thing:

http://cmake.org/cmake/help/cmake-2-8-docs.html#command:set_target_properties

2) If this dark magic is contained in an external script and
makefile.mk only instructs when and how to invoke it, this is the same
thing as discussed above.

If the versioning is something completely different, then that needs
to be examined.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]