Automated gbuild -> SCons convertibility now at 88.57%

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Automated gbuild -> SCons convertibility now at 88.57%

Damjan Jovanovic
Hi

In the scons-build branch, I've just pushed a set of 11 commits that
theoretically get 93 out of 105 gbuild modules (88.57%) automatically
converting to gbuild.

The "gotoSCons" converter and the SCons infrastructure in that branch have
now been developed to such a level that a module can be automatically
converted from gbuild to SCons, from where it can use SCons for all of the
following:
Building C/C++ objects
Linking shared libraries, static libraries, and executables
Building JUnit tests and running them
Building Google Tests and running them
Building .component files with XSLT
Running Ant sub-builds
Delivering "package" files such as headers
Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
merged .src -> .srs -> .res for each language).

I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but
I think only Jar and Zip are worth implementing automatic conversion for,
as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in
only 2 places, which makes manual conversion for them easier. The hardest
conversions are already done.

Where does this leave us?

The gotoSCons converter can't support a number of features, like
non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A
module can only be automatically converted if it doesn't use the
unsupported features. Currently, 35 modules use only supported features,
and can be converted automatically (this should increase to 39 modules when
I add Jar and Zip conversion).

Another 58 modules use non-deterministic constructs or custom make rules.
Converting those 58 could be done through a semi-automated process, which
involves editing the gbuild files to remove the unsupported features,
running the automated conversion on what is left, then manually patching
what was removed into the conversion results. Sometimes this is quick and
easy, at other times probably not.

The final 12 modules use unsupported targets requiring a longer and mostly
manual conversion to SCons, though even there the supported targets could
be converted automatically.

The SCons infrastructure does require some cleanup, as I was learning while
developing, and we still need library naming conversions, tests on
Linux/WIndows/Mac, etc.

The more I've used SCons, the more I've liked it. I've even started using
it in my own projects at work now. I've found a way to solve every problem
I've encountered, and the SCons developers have been helpful when I asked
them questions. Complex functionality like header dependency scanning,
automatic directory creation for output files, using @responsefile for long
command lines when necessary, and other features gbuild implements
manually, all work in SCons automatically. In 1816 lines of code, our SCons
infrastructure implements what took gbuild 9418 lines, and SCons is far
more readable and maintainable (even in its current messy state).

The plan isn't to merge this to trunk any time soon. Rather the idea is to
develop the converter even further, then when it's complete enough, convert
as many gbuild modules to SCons. Then measure performance of building those
modules en-masse with SCons alone - if there are performance problems at
that stage, they are only going to get worse with more modules.

The real test however is converting the other 78 dmake modules to SCons. 37
of them are 3rd party "externals" like jpeg and zlib, which have their own
build systems that we just call, so conversion is relatively easy. The
other 41 modules are hard to convert, but gbuild is one of the reasons that
they were hard, and where SCons is expected to make the greatest
difference. Only once we are building without dmake, without gbuild,
without build.pl, without Cygwin, on all platforms, then it would be the
right time to merge to trunk.

If at some stage in this process we are unhappy with SCons, and some better
build system can be found, it shouldn't be difficult to change to it. The
converter could output files for that other build system instead; at
present it's only 498 lines of code in 1 file, that are involved in writing
SCons build files, all we would need is a similar file for that other build
system.

Build systems are not the most exciting part of development, but bad build
systems make development painful, and as a large multi-platform project, we
build a lot. We answer build-related questions on the mailing lists too
often, and new contributors are put off by the current build system. A lot
of what we want to do with AOO, such as ports to Win64, AArch64, newer MSVC
versions, and so on, also involve build-related changes.

Damjan
Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Patricia Shanahan
In the new build system, what is the status of automatic symbol
generation, needed for easy debug.

It is badly broken in 4.1.7, with most modules not getting symbols
generated despite --enable-symbols in the configure parameters. This has
cost me weeks of work on a debug project.

On 7/4/2020 11:44 AM, Damjan Jovanovic wrote:

> Hi
>
> In the scons-build branch, I've just pushed a set of 11 commits that
> theoretically get 93 out of 105 gbuild modules (88.57%) automatically
> converting to gbuild.
>
> The "gotoSCons" converter and the SCons infrastructure in that branch have
> now been developed to such a level that a module can be automatically
> converted from gbuild to SCons, from where it can use SCons for all of the
> following:
> Building C/C++ objects
> Linking shared libraries, static libraries, and executables
> Building JUnit tests and running them
> Building Google Tests and running them
> Building .component files with XSLT
> Running Ant sub-builds
> Delivering "package" files such as headers
> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
> merged .src -> .srs -> .res for each language).
>
> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but
> I think only Jar and Zip are worth implementing automatic conversion for,
> as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in
> only 2 places, which makes manual conversion for them easier. The hardest
> conversions are already done.
>
> Where does this leave us?
>
> The gotoSCons converter can't support a number of features, like
> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A
> module can only be automatically converted if it doesn't use the
> unsupported features. Currently, 35 modules use only supported features,
> and can be converted automatically (this should increase to 39 modules when
> I add Jar and Zip conversion).
>
> Another 58 modules use non-deterministic constructs or custom make rules.
> Converting those 58 could be done through a semi-automated process, which
> involves editing the gbuild files to remove the unsupported features,
> running the automated conversion on what is left, then manually patching
> what was removed into the conversion results. Sometimes this is quick and
> easy, at other times probably not.
>
> The final 12 modules use unsupported targets requiring a longer and mostly
> manual conversion to SCons, though even there the supported targets could
> be converted automatically.
>
> The SCons infrastructure does require some cleanup, as I was learning while
> developing, and we still need library naming conversions, tests on
> Linux/WIndows/Mac, etc.
>
> The more I've used SCons, the more I've liked it. I've even started using
> it in my own projects at work now. I've found a way to solve every problem
> I've encountered, and the SCons developers have been helpful when I asked
> them questions. Complex functionality like header dependency scanning,
> automatic directory creation for output files, using @responsefile for long
> command lines when necessary, and other features gbuild implements
> manually, all work in SCons automatically. In 1816 lines of code, our SCons
> infrastructure implements what took gbuild 9418 lines, and SCons is far
> more readable and maintainable (even in its current messy state).
>
> The plan isn't to merge this to trunk any time soon. Rather the idea is to
> develop the converter even further, then when it's complete enough, convert
> as many gbuild modules to SCons. Then measure performance of building those
> modules en-masse with SCons alone - if there are performance problems at
> that stage, they are only going to get worse with more modules.
>
> The real test however is converting the other 78 dmake modules to SCons. 37
> of them are 3rd party "externals" like jpeg and zlib, which have their own
> build systems that we just call, so conversion is relatively easy. The
> other 41 modules are hard to convert, but gbuild is one of the reasons that
> they were hard, and where SCons is expected to make the greatest
> difference. Only once we are building without dmake, without gbuild,
> without build.pl, without Cygwin, on all platforms, then it would be the
> right time to merge to trunk.
>
> If at some stage in this process we are unhappy with SCons, and some better
> build system can be found, it shouldn't be difficult to change to it. The
> converter could output files for that other build system instead; at
> present it's only 498 lines of code in 1 file, that are involved in writing
> SCons build files, all we would need is a similar file for that other build
> system.
>
> Build systems are not the most exciting part of development, but bad build
> systems make development painful, and as a large multi-platform project, we
> build a lot. We answer build-related questions on the mailing lists too
> often, and new contributors are put off by the current build system. A lot
> of what we want to do with AOO, such as ports to Win64, AArch64, newer MSVC
> versions, and so on, also involve build-related changes.
>
> Damjan
>

--
This email has been checked for viruses by AVG.
https://www.avg.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Damjan Jovanovic
Given how I've only developed on FreeBSD so far, anything Windows is
probably at negative infinity :).

Do you have some example modules with that symbols problem, or at least
know whether they are gbuild or dmake modules?

On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan <[hidden email]> wrote:

> In the new build system, what is the status of automatic symbol
> generation, needed for easy debug.
>
> It is badly broken in 4.1.7, with most modules not getting symbols
> generated despite --enable-symbols in the configure parameters. This has
> cost me weeks of work on a debug project.
>
> On 7/4/2020 11:44 AM, Damjan Jovanovic wrote:
> > Hi
> >
> > In the scons-build branch, I've just pushed a set of 11 commits that
> > theoretically get 93 out of 105 gbuild modules (88.57%) automatically
> > converting to gbuild.
> >
> > The "gotoSCons" converter and the SCons infrastructure in that branch
> have
> > now been developed to such a level that a module can be automatically
> > converted from gbuild to SCons, from where it can use SCons for all of
> the
> > following:
> > Building C/C++ objects
> > Linking shared libraries, static libraries, and executables
> > Building JUnit tests and running them
> > Building Google Tests and running them
> > Building .component files with XSLT
> > Running Ant sub-builds
> > Delivering "package" files such as headers
> > Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
> > merged .src -> .srs -> .res for each language).
> >
> > I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget,
> but
> > I think only Jar and Zip are worth implementing automatic conversion for,
> > as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget
> in
> > only 2 places, which makes manual conversion for them easier. The hardest
> > conversions are already done.
> >
> > Where does this leave us?
> >
> > The gotoSCons converter can't support a number of features, like
> > non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules,
> etc. A
> > module can only be automatically converted if it doesn't use the
> > unsupported features. Currently, 35 modules use only supported features,
> > and can be converted automatically (this should increase to 39 modules
> when
> > I add Jar and Zip conversion).
> >
> > Another 58 modules use non-deterministic constructs or custom make rules.
> > Converting those 58 could be done through a semi-automated process, which
> > involves editing the gbuild files to remove the unsupported features,
> > running the automated conversion on what is left, then manually patching
> > what was removed into the conversion results. Sometimes this is quick and
> > easy, at other times probably not.
> >
> > The final 12 modules use unsupported targets requiring a longer and
> mostly
> > manual conversion to SCons, though even there the supported targets could
> > be converted automatically.
> >
> > The SCons infrastructure does require some cleanup, as I was learning
> while
> > developing, and we still need library naming conversions, tests on
> > Linux/WIndows/Mac, etc.
> >
> > The more I've used SCons, the more I've liked it. I've even started using
> > it in my own projects at work now. I've found a way to solve every
> problem
> > I've encountered, and the SCons developers have been helpful when I asked
> > them questions. Complex functionality like header dependency scanning,
> > automatic directory creation for output files, using @responsefile for
> long
> > command lines when necessary, and other features gbuild implements
> > manually, all work in SCons automatically. In 1816 lines of code, our
> SCons
> > infrastructure implements what took gbuild 9418 lines, and SCons is far
> > more readable and maintainable (even in its current messy state).
> >
> > The plan isn't to merge this to trunk any time soon. Rather the idea is
> to
> > develop the converter even further, then when it's complete enough,
> convert
> > as many gbuild modules to SCons. Then measure performance of building
> those
> > modules en-masse with SCons alone - if there are performance problems at
> > that stage, they are only going to get worse with more modules.
> >
> > The real test however is converting the other 78 dmake modules to SCons.
> 37
> > of them are 3rd party "externals" like jpeg and zlib, which have their
> own
> > build systems that we just call, so conversion is relatively easy. The
> > other 41 modules are hard to convert, but gbuild is one of the reasons
> that
> > they were hard, and where SCons is expected to make the greatest
> > difference. Only once we are building without dmake, without gbuild,
> > without build.pl, without Cygwin, on all platforms, then it would be the
> > right time to merge to trunk.
> >
> > If at some stage in this process we are unhappy with SCons, and some
> better
> > build system can be found, it shouldn't be difficult to change to it. The
> > converter could output files for that other build system instead; at
> > present it's only 498 lines of code in 1 file, that are involved in
> writing
> > SCons build files, all we would need is a similar file for that other
> build
> > system.
> >
> > Build systems are not the most exciting part of development, but bad
> build
> > systems make development painful, and as a large multi-platform project,
> we
> > build a lot. We answer build-related questions on the mailing lists too
> > often, and new contributors are put off by the current build system. A
> lot
> > of what we want to do with AOO, such as ports to Win64, AArch64, newer
> MSVC
> > versions, and so on, also involve build-related changes.
> >
> > Damjan
> >
>
> --
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Patricia Shanahan
On 7/4/2020 12:24 PM, Damjan Jovanovic wrote:
> Given how I've only developed on FreeBSD so far, anything Windows is
> probably at negative infinity :) >
> Do you have some example modules with that symbols problem, or at least
> know whether they are gbuild or dmake modules?

c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx
does not have symbols.

c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does
have symbols generated.


> On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan <[hidden email]> wrote:
>
>> In the new build system, what is the status of automatic symbol
>> generation, needed for easy debug.
>>
>> It is badly broken in 4.1.7, with most modules not getting symbols
>> generated despite --enable-symbols in the configure parameters. This has
>> cost me weeks of work on a debug project.
>>
>> On 7/4/2020 11:44 AM, Damjan Jovanovic wrote:
>>> Hi
>>>
>>> In the scons-build branch, I've just pushed a set of 11 commits that
>>> theoretically get 93 out of 105 gbuild modules (88.57%) automatically
>>> converting to gbuild.
>>>
>>> The "gotoSCons" converter and the SCons infrastructure in that branch
>> have
>>> now been developed to such a level that a module can be automatically
>>> converted from gbuild to SCons, from where it can use SCons for all of
>> the
>>> following:
>>> Building C/C++ objects
>>> Linking shared libraries, static libraries, and executables
>>> Building JUnit tests and running them
>>> Building Google Tests and running them
>>> Building .component files with XSLT
>>> Running Ant sub-builds
>>> Delivering "package" files such as headers
>>> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
>>> merged .src -> .srs -> .res for each language).
>>>
>>> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget,
>> but
>>> I think only Jar and Zip are worth implementing automatic conversion for,
>>> as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget
>> in
>>> only 2 places, which makes manual conversion for them easier. The hardest
>>> conversions are already done.
>>>
>>> Where does this leave us?
>>>
>>> The gotoSCons converter can't support a number of features, like
>>> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules,
>> etc. A
>>> module can only be automatically converted if it doesn't use the
>>> unsupported features. Currently, 35 modules use only supported features,
>>> and can be converted automatically (this should increase to 39 modules
>> when
>>> I add Jar and Zip conversion).
>>>
>>> Another 58 modules use non-deterministic constructs or custom make rules.
>>> Converting those 58 could be done through a semi-automated process, which
>>> involves editing the gbuild files to remove the unsupported features,
>>> running the automated conversion on what is left, then manually patching
>>> what was removed into the conversion results. Sometimes this is quick and
>>> easy, at other times probably not.
>>>
>>> The final 12 modules use unsupported targets requiring a longer and
>> mostly
>>> manual conversion to SCons, though even there the supported targets could
>>> be converted automatically.
>>>
>>> The SCons infrastructure does require some cleanup, as I was learning
>> while
>>> developing, and we still need library naming conversions, tests on
>>> Linux/WIndows/Mac, etc.
>>>
>>> The more I've used SCons, the more I've liked it. I've even started using
>>> it in my own projects at work now. I've found a way to solve every
>> problem
>>> I've encountered, and the SCons developers have been helpful when I asked
>>> them questions. Complex functionality like header dependency scanning,
>>> automatic directory creation for output files, using @responsefile for
>> long
>>> command lines when necessary, and other features gbuild implements
>>> manually, all work in SCons automatically. In 1816 lines of code, our
>> SCons
>>> infrastructure implements what took gbuild 9418 lines, and SCons is far
>>> more readable and maintainable (even in its current messy state).
>>>
>>> The plan isn't to merge this to trunk any time soon. Rather the idea is
>> to
>>> develop the converter even further, then when it's complete enough,
>> convert
>>> as many gbuild modules to SCons. Then measure performance of building
>> those
>>> modules en-masse with SCons alone - if there are performance problems at
>>> that stage, they are only going to get worse with more modules.
>>>
>>> The real test however is converting the other 78 dmake modules to SCons.
>> 37
>>> of them are 3rd party "externals" like jpeg and zlib, which have their
>> own
>>> build systems that we just call, so conversion is relatively easy. The
>>> other 41 modules are hard to convert, but gbuild is one of the reasons
>> that
>>> they were hard, and where SCons is expected to make the greatest
>>> difference. Only once we are building without dmake, without gbuild,
>>> without build.pl, without Cygwin, on all platforms, then it would be the
>>> right time to merge to trunk.
>>>
>>> If at some stage in this process we are unhappy with SCons, and some
>> better
>>> build system can be found, it shouldn't be difficult to change to it. The
>>> converter could output files for that other build system instead; at
>>> present it's only 498 lines of code in 1 file, that are involved in
>> writing
>>> SCons build files, all we would need is a similar file for that other
>> build
>>> system.
>>>
>>> Build systems are not the most exciting part of development, but bad
>> build
>>> systems make development painful, and as a large multi-platform project,
>> we
>>> build a lot. We answer build-related questions on the mailing lists too
>>> often, and new contributors are put off by the current build system. A
>> lot
>>> of what we want to do with AOO, such as ports to Win64, AArch64, newer
>> MSVC
>>> versions, and so on, also involve build-related changes.
>>>
>>> Damjan
>>>
>>
>> --
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

--
This email has been checked for viruses by AVG.
https://www.avg.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Damjan Jovanovic
scripting is a dmake module with no symbols
package is a gbuild module with symbols

So yes, as we move off dmake, more and more modules should have working
debug symbols.

This is the case on *nix too, where dmake doesn't provide full paths to
filenames, breaking debugging.

On Sat, Jul 4, 2020 at 9:35 PM Patricia Shanahan <[hidden email]> wrote:

> On 7/4/2020 12:24 PM, Damjan Jovanovic wrote:
> > Given how I've only developed on FreeBSD so far, anything Windows is
> > probably at negative infinity :) >
> > Do you have some example modules with that symbols problem, or at least
> > know whether they are gbuild or dmake modules?
>
> c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx
>
> does not have symbols.
>
> c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does
> have symbols generated.
>
>
> > On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan <[hidden email]> wrote:
> >
> >> In the new build system, what is the status of automatic symbol
> >> generation, needed for easy debug.
> >>
> >> It is badly broken in 4.1.7, with most modules not getting symbols
> >> generated despite --enable-symbols in the configure parameters. This has
> >> cost me weeks of work on a debug project.
> >>
> >> On 7/4/2020 11:44 AM, Damjan Jovanovic wrote:
> >>> Hi
> >>>
> >>> In the scons-build branch, I've just pushed a set of 11 commits that
> >>> theoretically get 93 out of 105 gbuild modules (88.57%) automatically
> >>> converting to gbuild.
> >>>
> >>> The "gotoSCons" converter and the SCons infrastructure in that branch
> >> have
> >>> now been developed to such a level that a module can be automatically
> >>> converted from gbuild to SCons, from where it can use SCons for all of
> >> the
> >>> following:
> >>> Building C/C++ objects
> >>> Linking shared libraries, static libraries, and executables
> >>> Building JUnit tests and running them
> >>> Building Google Tests and running them
> >>> Building .component files with XSLT
> >>> Running Ant sub-builds
> >>> Delivering "package" files such as headers
> >>> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
> >>> merged .src -> .srs -> .res for each language).
> >>>
> >>> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget,
> >> but
> >>> I think only Jar and Zip are worth implementing automatic conversion
> for,
> >>> as SdiTarget and UnoApi are only used in 5 places each, and
> WinResTarget
> >> in
> >>> only 2 places, which makes manual conversion for them easier. The
> hardest
> >>> conversions are already done.
> >>>
> >>> Where does this leave us?
> >>>
> >>> The gotoSCons converter can't support a number of features, like
> >>> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules,
> >> etc. A
> >>> module can only be automatically converted if it doesn't use the
> >>> unsupported features. Currently, 35 modules use only supported
> features,
> >>> and can be converted automatically (this should increase to 39 modules
> >> when
> >>> I add Jar and Zip conversion).
> >>>
> >>> Another 58 modules use non-deterministic constructs or custom make
> rules.
> >>> Converting those 58 could be done through a semi-automated process,
> which
> >>> involves editing the gbuild files to remove the unsupported features,
> >>> running the automated conversion on what is left, then manually
> patching
> >>> what was removed into the conversion results. Sometimes this is quick
> and
> >>> easy, at other times probably not.
> >>>
> >>> The final 12 modules use unsupported targets requiring a longer and
> >> mostly
> >>> manual conversion to SCons, though even there the supported targets
> could
> >>> be converted automatically.
> >>>
> >>> The SCons infrastructure does require some cleanup, as I was learning
> >> while
> >>> developing, and we still need library naming conversions, tests on
> >>> Linux/WIndows/Mac, etc.
> >>>
> >>> The more I've used SCons, the more I've liked it. I've even started
> using
> >>> it in my own projects at work now. I've found a way to solve every
> >> problem
> >>> I've encountered, and the SCons developers have been helpful when I
> asked
> >>> them questions. Complex functionality like header dependency scanning,
> >>> automatic directory creation for output files, using @responsefile for
> >> long
> >>> command lines when necessary, and other features gbuild implements
> >>> manually, all work in SCons automatically. In 1816 lines of code, our
> >> SCons
> >>> infrastructure implements what took gbuild 9418 lines, and SCons is far
> >>> more readable and maintainable (even in its current messy state).
> >>>
> >>> The plan isn't to merge this to trunk any time soon. Rather the idea is
> >> to
> >>> develop the converter even further, then when it's complete enough,
> >> convert
> >>> as many gbuild modules to SCons. Then measure performance of building
> >> those
> >>> modules en-masse with SCons alone - if there are performance problems
> at
> >>> that stage, they are only going to get worse with more modules.
> >>>
> >>> The real test however is converting the other 78 dmake modules to
> SCons.
> >> 37
> >>> of them are 3rd party "externals" like jpeg and zlib, which have their
> >> own
> >>> build systems that we just call, so conversion is relatively easy. The
> >>> other 41 modules are hard to convert, but gbuild is one of the reasons
> >> that
> >>> they were hard, and where SCons is expected to make the greatest
> >>> difference. Only once we are building without dmake, without gbuild,
> >>> without build.pl, without Cygwin, on all platforms, then it would be
> the
> >>> right time to merge to trunk.
> >>>
> >>> If at some stage in this process we are unhappy with SCons, and some
> >> better
> >>> build system can be found, it shouldn't be difficult to change to it.
> The
> >>> converter could output files for that other build system instead; at
> >>> present it's only 498 lines of code in 1 file, that are involved in
> >> writing
> >>> SCons build files, all we would need is a similar file for that other
> >> build
> >>> system.
> >>>
> >>> Build systems are not the most exciting part of development, but bad
> >> build
> >>> systems make development painful, and as a large multi-platform
> project,
> >> we
> >>> build a lot. We answer build-related questions on the mailing lists too
> >>> often, and new contributors are put off by the current build system. A
> >> lot
> >>> of what we want to do with AOO, such as ports to Win64, AArch64, newer
> >> MSVC
> >>> versions, and so on, also involve build-related changes.
> >>>
> >>> Damjan
> >>>
> >>
> >> --
> >> This email has been checked for viruses by AVG.
> >> https://www.avg.com
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
> --
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Patricia Shanahan
On 7/4/2020 12:53 PM, Damjan Jovanovic wrote:
> scripting is a dmake module with no symbols
> package is a gbuild module with symbols
>
> So yes, as we move off dmake, more and more modules should have working
> debug symbols.

I have suddenly become a convert to moving off dmake.

>
> This is the case on *nix too, where dmake doesn't provide full paths to
> filenames, breaking debugging.
>
> On Sat, Jul 4, 2020 at 9:35 PM Patricia Shanahan <[hidden email]> wrote:
>
>> On 7/4/2020 12:24 PM, Damjan Jovanovic wrote:
>>> Given how I've only developed on FreeBSD so far, anything Windows is
>>> probably at negative infinity :) >
>>> Do you have some example modules with that symbols problem, or at least
>>> know whether they are gbuild or dmake modules?
>>
>> c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx
>>
>> does not have symbols.
>>
>> c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does
>> have symbols generated.
>>
>>
>>> On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan <[hidden email]> wrote:
>>>
>>>> In the new build system, what is the status of automatic symbol
>>>> generation, needed for easy debug.
>>>>
>>>> It is badly broken in 4.1.7, with most modules not getting symbols
>>>> generated despite --enable-symbols in the configure parameters. This has
>>>> cost me weeks of work on a debug project.
>>>>
>>>> On 7/4/2020 11:44 AM, Damjan Jovanovic wrote:
>>>>> Hi
>>>>>
>>>>> In the scons-build branch, I've just pushed a set of 11 commits that
>>>>> theoretically get 93 out of 105 gbuild modules (88.57%) automatically
>>>>> converting to gbuild.
>>>>>
>>>>> The "gotoSCons" converter and the SCons infrastructure in that branch
>>>> have
>>>>> now been developed to such a level that a module can be automatically
>>>>> converted from gbuild to SCons, from where it can use SCons for all of
>>>> the
>>>>> following:
>>>>> Building C/C++ objects
>>>>> Linking shared libraries, static libraries, and executables
>>>>> Building JUnit tests and running them
>>>>> Building Google Tests and running them
>>>>> Building .component files with XSLT
>>>>> Running Ant sub-builds
>>>>> Delivering "package" files such as headers
>>>>> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
>>>>> merged .src -> .srs -> .res for each language).
>>>>>
>>>>> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget,
>>>> but
>>>>> I think only Jar and Zip are worth implementing automatic conversion
>> for,
>>>>> as SdiTarget and UnoApi are only used in 5 places each, and
>> WinResTarget
>>>> in
>>>>> only 2 places, which makes manual conversion for them easier. The
>> hardest
>>>>> conversions are already done.
>>>>>
>>>>> Where does this leave us?
>>>>>
>>>>> The gotoSCons converter can't support a number of features, like
>>>>> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules,
>>>> etc. A
>>>>> module can only be automatically converted if it doesn't use the
>>>>> unsupported features. Currently, 35 modules use only supported
>> features,
>>>>> and can be converted automatically (this should increase to 39 modules
>>>> when
>>>>> I add Jar and Zip conversion).
>>>>>
>>>>> Another 58 modules use non-deterministic constructs or custom make
>> rules.
>>>>> Converting those 58 could be done through a semi-automated process,
>> which
>>>>> involves editing the gbuild files to remove the unsupported features,
>>>>> running the automated conversion on what is left, then manually
>> patching
>>>>> what was removed into the conversion results. Sometimes this is quick
>> and
>>>>> easy, at other times probably not.
>>>>>
>>>>> The final 12 modules use unsupported targets requiring a longer and
>>>> mostly
>>>>> manual conversion to SCons, though even there the supported targets
>> could
>>>>> be converted automatically.
>>>>>
>>>>> The SCons infrastructure does require some cleanup, as I was learning
>>>> while
>>>>> developing, and we still need library naming conversions, tests on
>>>>> Linux/WIndows/Mac, etc.
>>>>>
>>>>> The more I've used SCons, the more I've liked it. I've even started
>> using
>>>>> it in my own projects at work now. I've found a way to solve every
>>>> problem
>>>>> I've encountered, and the SCons developers have been helpful when I
>> asked
>>>>> them questions. Complex functionality like header dependency scanning,
>>>>> automatic directory creation for output files, using @responsefile for
>>>> long
>>>>> command lines when necessary, and other features gbuild implements
>>>>> manually, all work in SCons automatically. In 1816 lines of code, our
>>>> SCons
>>>>> infrastructure implements what took gbuild 9418 lines, and SCons is far
>>>>> more readable and maintainable (even in its current messy state).
>>>>>
>>>>> The plan isn't to merge this to trunk any time soon. Rather the idea is
>>>> to
>>>>> develop the converter even further, then when it's complete enough,
>>>> convert
>>>>> as many gbuild modules to SCons. Then measure performance of building
>>>> those
>>>>> modules en-masse with SCons alone - if there are performance problems
>> at
>>>>> that stage, they are only going to get worse with more modules.
>>>>>
>>>>> The real test however is converting the other 78 dmake modules to
>> SCons.
>>>> 37
>>>>> of them are 3rd party "externals" like jpeg and zlib, which have their
>>>> own
>>>>> build systems that we just call, so conversion is relatively easy. The
>>>>> other 41 modules are hard to convert, but gbuild is one of the reasons
>>>> that
>>>>> they were hard, and where SCons is expected to make the greatest
>>>>> difference. Only once we are building without dmake, without gbuild,
>>>>> without build.pl, without Cygwin, on all platforms, then it would be
>> the
>>>>> right time to merge to trunk.
>>>>>
>>>>> If at some stage in this process we are unhappy with SCons, and some
>>>> better
>>>>> build system can be found, it shouldn't be difficult to change to it.
>> The
>>>>> converter could output files for that other build system instead; at
>>>>> present it's only 498 lines of code in 1 file, that are involved in
>>>> writing
>>>>> SCons build files, all we would need is a similar file for that other
>>>> build
>>>>> system.
>>>>>
>>>>> Build systems are not the most exciting part of development, but bad
>>>> build
>>>>> systems make development painful, and as a large multi-platform
>> project,
>>>> we
>>>>> build a lot. We answer build-related questions on the mailing lists too
>>>>> often, and new contributors are put off by the current build system. A
>>>> lot
>>>>> of what we want to do with AOO, such as ports to Win64, AArch64, newer
>>>> MSVC
>>>>> versions, and so on, also involve build-related changes.
>>>>>
>>>>> Damjan
>>>>>
>>>>
>>>> --
>>>> This email has been checked for viruses by AVG.
>>>> https://www.avg.com
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>> --
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

--
This email has been checked for viruses by AVG.
https://www.avg.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Peter Kovacs-3
In reply to this post by Damjan Jovanovic
Awesome Damjan,

I had a talk with SCons on freenode IRC. they liked the plan to move to
SCons a lot.

And did support us right a way. :) So I am poretty confidant that SCons
will do the trick.

How did you talk to SCon people?


I think there is are some more tools that we need to examine before we
can drop Cygwin.

http://www.openoffice.org/tools/build_env_tools.html


I have to look at the SCons. I plan to build GSICheck also with SCons,
even if it is a total simple tool.


All the best.

Peter


Am 04.07.20 um 20:44 schrieb Damjan Jovanovic:

> Hi
>
> In the scons-build branch, I've just pushed a set of 11 commits that
> theoretically get 93 out of 105 gbuild modules (88.57%) automatically
> converting to gbuild.
>
> The "gotoSCons" converter and the SCons infrastructure in that branch have
> now been developed to such a level that a module can be automatically
> converted from gbuild to SCons, from where it can use SCons for all of the
> following:
> Building C/C++ objects
> Linking shared libraries, static libraries, and executables
> Building JUnit tests and running them
> Building Google Tests and running them
> Building .component files with XSLT
> Running Ant sub-builds
> Delivering "package" files such as headers
> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
> merged .src -> .srs -> .res for each language).
>
> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but
> I think only Jar and Zip are worth implementing automatic conversion for,
> as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in
> only 2 places, which makes manual conversion for them easier. The hardest
> conversions are already done.
>
> Where does this leave us?
>
> The gotoSCons converter can't support a number of features, like
> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A
> module can only be automatically converted if it doesn't use the
> unsupported features. Currently, 35 modules use only supported features,
> and can be converted automatically (this should increase to 39 modules when
> I add Jar and Zip conversion).
>
> Another 58 modules use non-deterministic constructs or custom make rules.
> Converting those 58 could be done through a semi-automated process, which
> involves editing the gbuild files to remove the unsupported features,
> running the automated conversion on what is left, then manually patching
> what was removed into the conversion results. Sometimes this is quick and
> easy, at other times probably not.
>
> The final 12 modules use unsupported targets requiring a longer and mostly
> manual conversion to SCons, though even there the supported targets could
> be converted automatically.
>
> The SCons infrastructure does require some cleanup, as I was learning while
> developing, and we still need library naming conversions, tests on
> Linux/WIndows/Mac, etc.
>
> The more I've used SCons, the more I've liked it. I've even started using
> it in my own projects at work now. I've found a way to solve every problem
> I've encountered, and the SCons developers have been helpful when I asked
> them questions. Complex functionality like header dependency scanning,
> automatic directory creation for output files, using @responsefile for long
> command lines when necessary, and other features gbuild implements
> manually, all work in SCons automatically. In 1816 lines of code, our SCons
> infrastructure implements what took gbuild 9418 lines, and SCons is far
> more readable and maintainable (even in its current messy state).
>
> The plan isn't to merge this to trunk any time soon. Rather the idea is to
> develop the converter even further, then when it's complete enough, convert
> as many gbuild modules to SCons. Then measure performance of building those
> modules en-masse with SCons alone - if there are performance problems at
> that stage, they are only going to get worse with more modules.
>
> The real test however is converting the other 78 dmake modules to SCons. 37
> of them are 3rd party "externals" like jpeg and zlib, which have their own
> build systems that we just call, so conversion is relatively easy. The
> other 41 modules are hard to convert, but gbuild is one of the reasons that
> they were hard, and where SCons is expected to make the greatest
> difference. Only once we are building without dmake, without gbuild,
> without build.pl, without Cygwin, on all platforms, then it would be the
> right time to merge to trunk.
>
> If at some stage in this process we are unhappy with SCons, and some better
> build system can be found, it shouldn't be difficult to change to it. The
> converter could output files for that other build system instead; at
> present it's only 498 lines of code in 1 file, that are involved in writing
> SCons build files, all we would need is a similar file for that other build
> system.
>
> Build systems are not the most exciting part of development, but bad build
> systems make development painful, and as a large multi-platform project, we
> build a lot. We answer build-related questions on the mailing lists too
> often, and new contributors are put off by the current build system. A lot
> of what we want to do with AOO, such as ports to Win64, AArch64, newer MSVC
> versions, and so on, also involve build-related changes.
>
> Damjan
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Damjan Jovanovic
On Sun, Jul 5, 2020 at 7:27 AM Peter <[hidden email]> wrote:

> Awesome Damjan,
>
>
Thank you :)


> I had a talk with SCons on freenode IRC. they liked the plan to move to
> SCons a lot.
>
> And did support us right a way. :) So I am poretty confidant that SCons
> will do the trick.
>
> How did you talk to SCon people?
>
>
That's great.

We talked on the [hidden email] mailing list.



>
> I think there is are some more tools that we need to examine before we
> can drop Cygwin.
>
> http://www.openoffice.org/tools/build_env_tools.html
>
>
That list is missing idlc, if not others. And what does that mean? Those
tools are built by the "Executable" targets which we already support.

With the changes I just pushed, AllLangResTarget from main/uui built fine
in "cmd" on my first try (with Python and SCons installed outside Cygwin),
though I do want to check that the paths passed to rsc.exe are correct. My
MSVC is broken, so I can't test the binary targets.


>
> I have to look at the SCons. I plan to build GSICheck also with SCons,
> even if it is a total simple tool.
>
>
Ok great.


>
> All the best.
>
> Peter
>
>
You too
Damjan


>
> Am 04.07.20 um 20:44 schrieb Damjan Jovanovic:
> > Hi
> >
> > In the scons-build branch, I've just pushed a set of 11 commits that
> > theoretically get 93 out of 105 gbuild modules (88.57%) automatically
> > converting to gbuild.
> >
> > The "gotoSCons" converter and the SCons infrastructure in that branch
> have
> > now been developed to such a level that a module can be automatically
> > converted from gbuild to SCons, from where it can use SCons for all of
> the
> > following:
> > Building C/C++ objects
> > Linking shared libraries, static libraries, and executables
> > Building JUnit tests and running them
> > Building Google Tests and running them
> > Building .component files with XSLT
> > Running Ant sub-builds
> > Delivering "package" files such as headers
> > Even doing the impossibly difficult 5 step "AllLangResTarget" (.src ->
> > merged .src -> .srs -> .res for each language).
> >
> > I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget,
> but
> > I think only Jar and Zip are worth implementing automatic conversion for,
> > as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget
> in
> > only 2 places, which makes manual conversion for them easier. The hardest
> > conversions are already done.
> >
> > Where does this leave us?
> >
> > The gotoSCons converter can't support a number of features, like
> > non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules,
> etc. A
> > module can only be automatically converted if it doesn't use the
> > unsupported features. Currently, 35 modules use only supported features,
> > and can be converted automatically (this should increase to 39 modules
> when
> > I add Jar and Zip conversion).
> >
> > Another 58 modules use non-deterministic constructs or custom make rules.
> > Converting those 58 could be done through a semi-automated process, which
> > involves editing the gbuild files to remove the unsupported features,
> > running the automated conversion on what is left, then manually patching
> > what was removed into the conversion results. Sometimes this is quick and
> > easy, at other times probably not.
> >
> > The final 12 modules use unsupported targets requiring a longer and
> mostly
> > manual conversion to SCons, though even there the supported targets could
> > be converted automatically.
> >
> > The SCons infrastructure does require some cleanup, as I was learning
> while
> > developing, and we still need library naming conversions, tests on
> > Linux/WIndows/Mac, etc.
> >
> > The more I've used SCons, the more I've liked it. I've even started using
> > it in my own projects at work now. I've found a way to solve every
> problem
> > I've encountered, and the SCons developers have been helpful when I asked
> > them questions. Complex functionality like header dependency scanning,
> > automatic directory creation for output files, using @responsefile for
> long
> > command lines when necessary, and other features gbuild implements
> > manually, all work in SCons automatically. In 1816 lines of code, our
> SCons
> > infrastructure implements what took gbuild 9418 lines, and SCons is far
> > more readable and maintainable (even in its current messy state).
> >
> > The plan isn't to merge this to trunk any time soon. Rather the idea is
> to
> > develop the converter even further, then when it's complete enough,
> convert
> > as many gbuild modules to SCons. Then measure performance of building
> those
> > modules en-masse with SCons alone - if there are performance problems at
> > that stage, they are only going to get worse with more modules.
> >
> > The real test however is converting the other 78 dmake modules to SCons.
> 37
> > of them are 3rd party "externals" like jpeg and zlib, which have their
> own
> > build systems that we just call, so conversion is relatively easy. The
> > other 41 modules are hard to convert, but gbuild is one of the reasons
> that
> > they were hard, and where SCons is expected to make the greatest
> > difference. Only once we are building without dmake, without gbuild,
> > without build.pl, without Cygwin, on all platforms, then it would be the
> > right time to merge to trunk.
> >
> > If at some stage in this process we are unhappy with SCons, and some
> better
> > build system can be found, it shouldn't be difficult to change to it. The
> > converter could output files for that other build system instead; at
> > present it's only 498 lines of code in 1 file, that are involved in
> writing
> > SCons build files, all we would need is a similar file for that other
> build
> > system.
> >
> > Build systems are not the most exciting part of development, but bad
> build
> > systems make development painful, and as a large multi-platform project,
> we
> > build a lot. We answer build-related questions on the mailing lists too
> > often, and new contributors are put off by the current build system. A
> lot
> > of what we want to do with AOO, such as ports to Win64, AArch64, newer
> MSVC
> > versions, and so on, also involve build-related changes.
> >
> > Damjan
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Automated gbuild -> SCons convertibility now at 88.57%

Peter Kovacs-3

Am 05.07.20 um 08:00 schrieb Damjan Jovanovic:

>
>> I think there is are some more tools that we need to examine before we
>> can drop Cygwin.
>>
>> http://www.openoffice.org/tools/build_env_tools.html
>>
>>
> That list is missing idlc, if not others. And what does that mean? Those
> tools are built by the "Executable" targets which we already support.
>
> With the changes I just pushed, AllLangResTarget from main/uui built fine
> in "cmd" on my first try (with Python and SCons installed outside Cygwin),
> though I do want to check that the paths passed to rsc.exe are correct. My
> MSVC is broken, so I can't test the binary targets.
Do you need the MSVC Installation media? We can share that for reinstall
if you like.


All the best

Peter




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]