gotoSCons: automated gbuild to SCons conversion

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

gotoSCons: automated gbuild to SCons conversion

Damjan Jovanovic
Hi

gbuild has become an unmaintainable nightmare. While there are only 37
internal dmake modules left (+ another 37 external modules), I cannot
motivate myself to convert even 1 more. At my development speed of about 2
lines of code every 8 hours, gbuild is a disaster that wastes unbelievable
amounts of time, and it's really not a development I can say I am proud of.
It has the ugliest syntax ever, it can only be debugged through
experimentation, and nobody really understands it. Also, it doesn't fully
work, eg. a lot of the newer targets I've added don't get cleaned on "make
clean", CustomTarget fails to deliver files sometimes, etc.

To that end, I've restarting playing with an old idea: to switch to the
SCons build system, using a tool to automatically convert build files from
gbuild to SCons.

I got my previous scons build (of 1 module) up and running very quickly,
and it still works. Then I continued development of my gbuild parser and
automated converter, which I've called gotoSCons, and got it to parse most
of the build instructions in Module_xxx.mk, Library_xxx.mk,
Executable_xxx.mk and Package_xxx.mk and convert them to a SCons build file
(SConscript).

gotoSCons is relatively simple, about 1200 lines of Java, and I can provide
it to interested parties, or put it in a Git branch. It already found 3
bugs in these gbuild files:
1. main/xmloff/Package_inc.mk duplicates this line:
$(eval $(call
gb_Package_add_file,xmloff_inc,inc/xmloff/XMLTextShapeImportHelper.hxx,xmloff/XMLTextShapeImportHelper.hxx))
I've committed a fix in 85bfc14eebba4af4847075b1cf1eaecfa87bcfc4
2. main/bean/Module_bean.mk adds no targets.
3. main/testgraphical/Module_testgraphical.mk does nothing.

The gotoSCons parser is very strict, it tries hard to guarantee a correct
conversion, and will immediately fail on non-deterministic sections, such
as "ifeq ($(OS),WNT)". It will also immediately fail on deliverable types
that I haven't implemented yet, such as AllLangResTarget_, StaticLibrary_
etc., and commands within targets that I haven't implemented yet such as
gb_Executable_set_targettype_gui and gb_Library_use_externals. Out of our
total of 105 gbuild modules, 53 use non-deterministic sections, and will
require some degree of manual conversion, but the converter should
eventually be able to handle the other 52 modules. Even with the limited
deliverable/command support at present, it can successfully convert 19
modules. The conversion would be straightforward even if done manually, as
both gbuild and SCons are high level build systems with similar concepts:
includes, defines, cflags, libraries to link to, and so on (dmake is the
difficult low-level tool).

I love SCons. It's been easy integrating it into our build. It just works.
It has beautiful clear syntax. It's understandable and debuggable. It makes
development fun again. It's a well established build system, with a 20 year
history, supporting Python 2 and 3, supporting almost every platform out
there including OS/2, supporting every version of MSVC including Visual
Studio 2019, supporting advanced features like ELF sonames, library version
symlinks and library name prefixes and suffixes, flex and bison targets
(which we all need), automatically generating cleaning targets for every
build target, can use checksums instead of timestamps to avoid unnecessary
rebuilds, automatic header dependency extraction, can build a module by
itself instead of the whole project, parallelizes building at a file level
(across module boundaries). It can build at least some AOO modules without
Cygwin. It is easy to add custom targets, unlike gbuild where it's almost
impossible. It is equal to or better than gbuild in just about everything,
and we don't have to maintain it going forward like we do with gbuild: its
developers develop it, and we just use it.

If someone would like to get involved, please let me know soon, so we can
decide on an API, file layout, etc. before I've begun committing any
changes (and relax, they would be committed to a separate branch until we
are convinced it works well).

I was the major force that led us into more gbuild, but now I think it's
time to leave it. It's already coming apart at the seams, and I don't see
it being developed further. It's based on ($eval), an optional extra added
quite late, as an afterthought for special cases, probably not how GNU make
was intended to be used at scale, and a feature absent in other "make"s.
The only other project in the world that uses gbuild is LO, and I recall
reading how it took them an enormous amount of work to migrate to it fully.
Let's learn from their mistake?

Regards
Damjan
Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Yury
My opinion, which might be not worth much, being the opinion of an occasional
dabbler, is the current build system looks unworkable, and that you
shouldn't wait for some communal sanction (which might never come), but just
go ahead and convert to some workable state whatever is there to convert.
Whatever build system Libreoffice guys currently have is at least capable of
doing 'make' and 'make clean' (I can even almost make sense of their
makefiles).



--
Sent from: http://openoffice.2283327.n4.nabble.com/Development-f2916443.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Peter Kovacs-3
In reply to this post by Damjan Jovanovic
Hi Damjan,

Let's try it. But I suggest to push to an own branch. There is no worth in trying and getting stuck in the same spot.

Merge is done quickly. And it is great if others can have a look, too.

All the best
Peter


Am 1. Februar 2020 13:36:42 MEZ schrieb Damjan Jovanovic <[hidden email]>:

>Hi
>
>gbuild has become an unmaintainable nightmare. While there are only 37
>internal dmake modules left (+ another 37 external modules), I cannot
>motivate myself to convert even 1 more. At my development speed of
>about 2
>lines of code every 8 hours, gbuild is a disaster that wastes
>unbelievable
>amounts of time, and it's really not a development I can say I am proud
>of.
>It has the ugliest syntax ever, it can only be debugged through
>experimentation, and nobody really understands it. Also, it doesn't
>fully
>work, eg. a lot of the newer targets I've added don't get cleaned on
>"make
>clean", CustomTarget fails to deliver files sometimes, etc.
>
>To that end, I've restarting playing with an old idea: to switch to the
>SCons build system, using a tool to automatically convert build files
>from
>gbuild to SCons.
>
>I got my previous scons build (of 1 module) up and running very
>quickly,
>and it still works. Then I continued development of my gbuild parser
>and
>automated converter, which I've called gotoSCons, and got it to parse
>most
>of the build instructions in Module_xxx.mk, Library_xxx.mk,
>Executable_xxx.mk and Package_xxx.mk and convert them to a SCons build
>file
>(SConscript).
>
>gotoSCons is relatively simple, about 1200 lines of Java, and I can
>provide
>it to interested parties, or put it in a Git branch. It already found 3
>bugs in these gbuild files:
>1. main/xmloff/Package_inc.mk duplicates this line:
>$(eval $(call
>gb_Package_add_file,xmloff_inc,inc/xmloff/XMLTextShapeImportHelper.hxx,xmloff/XMLTextShapeImportHelper.hxx))
>I've committed a fix in 85bfc14eebba4af4847075b1cf1eaecfa87bcfc4
>2. main/bean/Module_bean.mk adds no targets.
>3. main/testgraphical/Module_testgraphical.mk does nothing.
>
>The gotoSCons parser is very strict, it tries hard to guarantee a
>correct
>conversion, and will immediately fail on non-deterministic sections,
>such
>as "ifeq ($(OS),WNT)". It will also immediately fail on deliverable
>types
>that I haven't implemented yet, such as AllLangResTarget_,
>StaticLibrary_
>etc., and commands within targets that I haven't implemented yet such
>as
>gb_Executable_set_targettype_gui and gb_Library_use_externals. Out of
>our
>total of 105 gbuild modules, 53 use non-deterministic sections, and
>will
>require some degree of manual conversion, but the converter should
>eventually be able to handle the other 52 modules. Even with the
>limited
>deliverable/command support at present, it can successfully convert 19
>modules. The conversion would be straightforward even if done manually,
>as
>both gbuild and SCons are high level build systems with similar
>concepts:
>includes, defines, cflags, libraries to link to, and so on (dmake is
>the
>difficult low-level tool).
>
>I love SCons. It's been easy integrating it into our build. It just
>works.
>It has beautiful clear syntax. It's understandable and debuggable. It
>makes
>development fun again. It's a well established build system, with a 20
>year
>history, supporting Python 2 and 3, supporting almost every platform
>out
>there including OS/2, supporting every version of MSVC including Visual
>Studio 2019, supporting advanced features like ELF sonames, library
>version
>symlinks and library name prefixes and suffixes, flex and bison targets
>(which we all need), automatically generating cleaning targets for
>every
>build target, can use checksums instead of timestamps to avoid
>unnecessary
>rebuilds, automatic header dependency extraction, can build a module by
>itself instead of the whole project, parallelizes building at a file
>level
>(across module boundaries). It can build at least some AOO modules
>without
>Cygwin. It is easy to add custom targets, unlike gbuild where it's
>almost
>impossible. It is equal to or better than gbuild in just about
>everything,
>and we don't have to maintain it going forward like we do with gbuild:
>its
>developers develop it, and we just use it.
>
>If someone would like to get involved, please let me know soon, so we
>can
>decide on an API, file layout, etc. before I've begun committing any
>changes (and relax, they would be committed to a separate branch until
>we
>are convinced it works well).
>
>I was the major force that led us into more gbuild, but now I think
>it's
>time to leave it. It's already coming apart at the seams, and I don't
>see
>it being developed further. It's based on ($eval), an optional extra
>added
>quite late, as an afterthought for special cases, probably not how GNU
>make
>was intended to be used at scale, and a feature absent in other
>"make"s.
>The only other project in the world that uses gbuild is LO, and I
>recall
>reading how it took them an enormous amount of work to migrate to it
>fully.
>Let's learn from their mistake?
>
>Regards
>Damjan
Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Damjan Jovanovic
Hi

I've just pushed a "scons-build" branch.

The build infrastructure is in main/scons. Python and SCons have to be
installed system-wide and available in $PATH. Currently we require Python
3, but that's easy to change.

So far main/fileaccess has been converted to SCons as a test. Its gbuild
files are still there; prj/makefile.mk determines whether to use gbuild or
scons. SCons is only used at a module level, build.pl is still the
launcher. You can build main/fileaccess in isolation by running "scons"
inside main/, or "scons -u" inside main/fileaccess, and clean by adding
"-c" (or "--clean") to those flags. It will correctly build the .cxx file,
link it into a library, install the library, run xsltproc on the .component
file, install the transformed .component file, and install the .xml file.
Everything can also be successfully cleaned.

At present FreeBSD has been tested, and I will test Windows soon. Other
platforms don't exist, and still have to be added in
main/site_scons/platform.

The converter is in gotoGBuild/, at the same level as main/ and test/. You
build it with Maven by running "mvn package". Then:

java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
parsingAnalysis ../main
Attempts to parse each gbuild module, printing out errors for those that
couldn't be, and a summary of which could be parsed.
What is also useful is:
java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
parsingAnalysis ../main 2>&1 | grep java.lang.Exception >
/tmp/errorsByModule.csv
Then open /tmp/errorsByModule.csv in AOO, use # as the field separator, and
you get a table of modules that failed and a reason for each one. Sort by
column B, and you can see how often a reason occurs, for example 21 modules
need AllLangResTarget implemented. That can inform further development.

To actually convert a module to SCons, use one of the modules that previous
results said could be parsed, eg. io, and run:
java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
parseModule ../main/io/Module_io.mk
It will print the converted SCons file to standard output.

Converting library names is currently broken. In main/fileaccess and
main/site_scons I've begun with dmake's way of naming libraries, like in
main/solenv/inc/libs.mk. GBuild re-did library naming by using 10
layer-specific rules, and then having tons of exceptions in
main/RepositoryFixes.mk; I am not sure which is worse. Maybe we should give
up the pretense, and just have a table in a CSV file, with platforms as
column headers, and library names as row headers, with the
platform-specific name in each field, and parse it with Python's built-in
CSV package on build startup?



On Sat, Feb 1, 2020 at 8:10 PM Peter Kovacs <[hidden email]> wrote:

> Hi Damjan,
>
> Let's try it. But I suggest to push to an own branch. There is no worth in
> trying and getting stuck in the same spot.
>
> Merge is done quickly. And it is great if others can have a look, too.
>
> All the best
> Peter
>
>
> Am 1. Februar 2020 13:36:42 MEZ schrieb Damjan Jovanovic <
> [hidden email]>:
> >Hi
> >
> >gbuild has become an unmaintainable nightmare. While there are only 37
> >internal dmake modules left (+ another 37 external modules), I cannot
> >motivate myself to convert even 1 more. At my development speed of
> >about 2
> >lines of code every 8 hours, gbuild is a disaster that wastes
> >unbelievable
> >amounts of time, and it's really not a development I can say I am proud
> >of.
> >It has the ugliest syntax ever, it can only be debugged through
> >experimentation, and nobody really understands it. Also, it doesn't
> >fully
> >work, eg. a lot of the newer targets I've added don't get cleaned on
> >"make
> >clean", CustomTarget fails to deliver files sometimes, etc.
> >
> >To that end, I've restarting playing with an old idea: to switch to the
> >SCons build system, using a tool to automatically convert build files
> >from
> >gbuild to SCons.
> >
> >I got my previous scons build (of 1 module) up and running very
> >quickly,
> >and it still works. Then I continued development of my gbuild parser
> >and
> >automated converter, which I've called gotoSCons, and got it to parse
> >most
> >of the build instructions in Module_xxx.mk, Library_xxx.mk,
> >Executable_xxx.mk and Package_xxx.mk and convert them to a SCons build
> >file
> >(SConscript).
> >
> >gotoSCons is relatively simple, about 1200 lines of Java, and I can
> >provide
> >it to interested parties, or put it in a Git branch. It already found 3
> >bugs in these gbuild files:
> >1. main/xmloff/Package_inc.mk duplicates this line:
> >$(eval $(call
>
> >gb_Package_add_file,xmloff_inc,inc/xmloff/XMLTextShapeImportHelper.hxx,xmloff/XMLTextShapeImportHelper.hxx))
> >I've committed a fix in 85bfc14eebba4af4847075b1cf1eaecfa87bcfc4
> >2. main/bean/Module_bean.mk adds no targets.
> >3. main/testgraphical/Module_testgraphical.mk does nothing.
> >
> >The gotoSCons parser is very strict, it tries hard to guarantee a
> >correct
> >conversion, and will immediately fail on non-deterministic sections,
> >such
> >as "ifeq ($(OS),WNT)". It will also immediately fail on deliverable
> >types
> >that I haven't implemented yet, such as AllLangResTarget_,
> >StaticLibrary_
> >etc., and commands within targets that I haven't implemented yet such
> >as
> >gb_Executable_set_targettype_gui and gb_Library_use_externals. Out of
> >our
> >total of 105 gbuild modules, 53 use non-deterministic sections, and
> >will
> >require some degree of manual conversion, but the converter should
> >eventually be able to handle the other 52 modules. Even with the
> >limited
> >deliverable/command support at present, it can successfully convert 19
> >modules. The conversion would be straightforward even if done manually,
> >as
> >both gbuild and SCons are high level build systems with similar
> >concepts:
> >includes, defines, cflags, libraries to link to, and so on (dmake is
> >the
> >difficult low-level tool).
> >
> >I love SCons. It's been easy integrating it into our build. It just
> >works.
> >It has beautiful clear syntax. It's understandable and debuggable. It
> >makes
> >development fun again. It's a well established build system, with a 20
> >year
> >history, supporting Python 2 and 3, supporting almost every platform
> >out
> >there including OS/2, supporting every version of MSVC including Visual
> >Studio 2019, supporting advanced features like ELF sonames, library
> >version
> >symlinks and library name prefixes and suffixes, flex and bison targets
> >(which we all need), automatically generating cleaning targets for
> >every
> >build target, can use checksums instead of timestamps to avoid
> >unnecessary
> >rebuilds, automatic header dependency extraction, can build a module by
> >itself instead of the whole project, parallelizes building at a file
> >level
> >(across module boundaries). It can build at least some AOO modules
> >without
> >Cygwin. It is easy to add custom targets, unlike gbuild where it's
> >almost
> >impossible. It is equal to or better than gbuild in just about
> >everything,
> >and we don't have to maintain it going forward like we do with gbuild:
> >its
> >developers develop it, and we just use it.
> >
> >If someone would like to get involved, please let me know soon, so we
> >can
> >decide on an API, file layout, etc. before I've begun committing any
> >changes (and relax, they would be committed to a separate branch until
> >we
> >are convinced it works well).
> >
> >I was the major force that led us into more gbuild, but now I think
> >it's
> >time to leave it. It's already coming apart at the seams, and I don't
> >see
> >it being developed further. It's based on ($eval), an optional extra
> >added
> >quite late, as an afterthought for special cases, probably not how GNU
> >make
> >was intended to be used at scale, and a feature absent in other
> >"make"s.
> >The only other project in the world that uses gbuild is LO, and I
> >recall
> >reading how it took them an enormous amount of work to migrate to it
> >fully.
> >Let's learn from their mistake?
> >
> >Regards
> >Damjan
>
Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Damjan Jovanovic
I've implemented static libraries, added linker version scripts, and made
some cleanups, and 22 modules can now be converted to SCons (up from 19 a
week ago).

AllLangResTarget is the next big target, and it seems exceptionally
complicated, with 3 layers of intermediate targets: .src --(transex3)-->
.src --(rsc)--> .src --(cat)--> .srs --(rsc)--> .res per language in
AllLangResTarget, with transex3 being skipped when not building
translations. Since the .src can #include C headers, they will need a
custom dependency scanner, in addition to the custom builder that all
custom targets need, plus of course delivery rules. Fortunately, as it's
Python, we have some flexibility as to how to implement it.

On Sun, Feb 2, 2020 at 2:31 AM Damjan Jovanovic <[hidden email]> wrote:

> Hi
>
> I've just pushed a "scons-build" branch.
>
> The build infrastructure is in main/scons. Python and SCons have to be
> installed system-wide and available in $PATH. Currently we require Python
> 3, but that's easy to change.
>
> So far main/fileaccess has been converted to SCons as a test. Its gbuild
> files are still there; prj/makefile.mk determines whether to use gbuild
> or scons. SCons is only used at a module level, build.pl is still the
> launcher. You can build main/fileaccess in isolation by running "scons"
> inside main/, or "scons -u" inside main/fileaccess, and clean by adding
> "-c" (or "--clean") to those flags. It will correctly build the .cxx file,
> link it into a library, install the library, run xsltproc on the .component
> file, install the transformed .component file, and install the .xml file.
> Everything can also be successfully cleaned.
>
> At present FreeBSD has been tested, and I will test Windows soon. Other
> platforms don't exist, and still have to be added in
> main/site_scons/platform.
>
> The converter is in gotoGBuild/, at the same level as main/ and test/. You
> build it with Maven by running "mvn package". Then:
>
> java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
> parsingAnalysis ../main
> Attempts to parse each gbuild module, printing out errors for those that
> couldn't be, and a summary of which could be parsed.
> What is also useful is:
> java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
> parsingAnalysis ../main 2>&1 | grep java.lang.Exception >
> /tmp/errorsByModule.csv
> Then open /tmp/errorsByModule.csv in AOO, use # as the field separator,
> and you get a table of modules that failed and a reason for each one. Sort
> by column B, and you can see how often a reason occurs, for example 21
> modules need AllLangResTarget implemented. That can inform further
> development.
>
> To actually convert a module to SCons, use one of the modules that
> previous results said could be parsed, eg. io, and run:
> java -cp target/classes org.apache.openoffice.gotoSCons.GBuildConverter
> parseModule ../main/io/Module_io.mk
> It will print the converted SCons file to standard output.
>
> Converting library names is currently broken. In main/fileaccess and
> main/site_scons I've begun with dmake's way of naming libraries, like in
> main/solenv/inc/libs.mk. GBuild re-did library naming by using 10
> layer-specific rules, and then having tons of exceptions in
> main/RepositoryFixes.mk; I am not sure which is worse. Maybe we should give
> up the pretense, and just have a table in a CSV file, with platforms as
> column headers, and library names as row headers, with the
> platform-specific name in each field, and parse it with Python's built-in
> CSV package on build startup?
>
>
>
> On Sat, Feb 1, 2020 at 8:10 PM Peter Kovacs <[hidden email]> wrote:
>
>> Hi Damjan,
>>
>> Let's try it. But I suggest to push to an own branch. There is no worth
>> in trying and getting stuck in the same spot.
>>
>> Merge is done quickly. And it is great if others can have a look, too.
>>
>> All the best
>> Peter
>>
>>
>> Am 1. Februar 2020 13:36:42 MEZ schrieb Damjan Jovanovic <
>> [hidden email]>:
>> >Hi
>> >
>> >gbuild has become an unmaintainable nightmare. While there are only 37
>> >internal dmake modules left (+ another 37 external modules), I cannot
>> >motivate myself to convert even 1 more. At my development speed of
>> >about 2
>> >lines of code every 8 hours, gbuild is a disaster that wastes
>> >unbelievable
>> >amounts of time, and it's really not a development I can say I am proud
>> >of.
>> >It has the ugliest syntax ever, it can only be debugged through
>> >experimentation, and nobody really understands it. Also, it doesn't
>> >fully
>> >work, eg. a lot of the newer targets I've added don't get cleaned on
>> >"make
>> >clean", CustomTarget fails to deliver files sometimes, etc.
>> >
>> >To that end, I've restarting playing with an old idea: to switch to the
>> >SCons build system, using a tool to automatically convert build files
>> >from
>> >gbuild to SCons.
>> >
>> >I got my previous scons build (of 1 module) up and running very
>> >quickly,
>> >and it still works. Then I continued development of my gbuild parser
>> >and
>> >automated converter, which I've called gotoSCons, and got it to parse
>> >most
>> >of the build instructions in Module_xxx.mk, Library_xxx.mk,
>> >Executable_xxx.mk and Package_xxx.mk and convert them to a SCons build
>> >file
>> >(SConscript).
>> >
>> >gotoSCons is relatively simple, about 1200 lines of Java, and I can
>> >provide
>> >it to interested parties, or put it in a Git branch. It already found 3
>> >bugs in these gbuild files:
>> >1. main/xmloff/Package_inc.mk duplicates this line:
>> >$(eval $(call
>>
>> >gb_Package_add_file,xmloff_inc,inc/xmloff/XMLTextShapeImportHelper.hxx,xmloff/XMLTextShapeImportHelper.hxx))
>> >I've committed a fix in 85bfc14eebba4af4847075b1cf1eaecfa87bcfc4
>> >2. main/bean/Module_bean.mk adds no targets.
>> >3. main/testgraphical/Module_testgraphical.mk does nothing.
>> >
>> >The gotoSCons parser is very strict, it tries hard to guarantee a
>> >correct
>> >conversion, and will immediately fail on non-deterministic sections,
>> >such
>> >as "ifeq ($(OS),WNT)". It will also immediately fail on deliverable
>> >types
>> >that I haven't implemented yet, such as AllLangResTarget_,
>> >StaticLibrary_
>> >etc., and commands within targets that I haven't implemented yet such
>> >as
>> >gb_Executable_set_targettype_gui and gb_Library_use_externals. Out of
>> >our
>> >total of 105 gbuild modules, 53 use non-deterministic sections, and
>> >will
>> >require some degree of manual conversion, but the converter should
>> >eventually be able to handle the other 52 modules. Even with the
>> >limited
>> >deliverable/command support at present, it can successfully convert 19
>> >modules. The conversion would be straightforward even if done manually,
>> >as
>> >both gbuild and SCons are high level build systems with similar
>> >concepts:
>> >includes, defines, cflags, libraries to link to, and so on (dmake is
>> >the
>> >difficult low-level tool).
>> >
>> >I love SCons. It's been easy integrating it into our build. It just
>> >works.
>> >It has beautiful clear syntax. It's understandable and debuggable. It
>> >makes
>> >development fun again. It's a well established build system, with a 20
>> >year
>> >history, supporting Python 2 and 3, supporting almost every platform
>> >out
>> >there including OS/2, supporting every version of MSVC including Visual
>> >Studio 2019, supporting advanced features like ELF sonames, library
>> >version
>> >symlinks and library name prefixes and suffixes, flex and bison targets
>> >(which we all need), automatically generating cleaning targets for
>> >every
>> >build target, can use checksums instead of timestamps to avoid
>> >unnecessary
>> >rebuilds, automatic header dependency extraction, can build a module by
>> >itself instead of the whole project, parallelizes building at a file
>> >level
>> >(across module boundaries). It can build at least some AOO modules
>> >without
>> >Cygwin. It is easy to add custom targets, unlike gbuild where it's
>> >almost
>> >impossible. It is equal to or better than gbuild in just about
>> >everything,
>> >and we don't have to maintain it going forward like we do with gbuild:
>> >its
>> >developers develop it, and we just use it.
>> >
>> >If someone would like to get involved, please let me know soon, so we
>> >can
>> >decide on an API, file layout, etc. before I've begun committing any
>> >changes (and relax, they would be committed to a separate branch until
>> >we
>> >are convinced it works well).
>> >
>> >I was the major force that led us into more gbuild, but now I think
>> >it's
>> >time to leave it. It's already coming apart at the seams, and I don't
>> >see
>> >it being developed further. It's based on ($eval), an optional extra
>> >added
>> >quite late, as an afterthought for special cases, probably not how GNU
>> >make
>> >was intended to be used at scale, and a feature absent in other
>> >"make"s.
>> >The only other project in the world that uses gbuild is LO, and I
>> >recall
>> >reading how it took them an enormous amount of work to migrate to it
>> >fully.
>> >Let's learn from their mistake?
>> >
>> >Regards
>> >Damjan
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Peter Kovacs-3
Sounds nice.

It would be great to have a debug output that can be checked to check which dependencies have been pulled. For me a declarative output would be cool to understand some code.


Also do you have an idea how to handle IDLs with scon? That would be awesome.

All the best
Peter

Am 9. Februar 2020 09:51:06 MEZ schrieb Damjan Jovanovic <[hidden email]>:

>I've implemented static libraries, added linker version scripts, and
>made
>some cleanups, and 22 modules can now be converted to SCons (up from 19
>a
>week ago).
>
>AllLangResTarget is the next big target, and it seems exceptionally
>complicated, with 3 layers of intermediate targets: .src
>--(transex3)-->
>.src --(rsc)--> .src --(cat)--> .srs --(rsc)--> .res per language in
>AllLangResTarget, with transex3 being skipped when not building
>translations. Since the .src can #include C headers, they will need a
>custom dependency scanner, in addition to the custom builder that all
>custom targets need, plus of course delivery rules. Fortunately, as
>it's
>Python, we have some flexibility as to how to implement it.
>
>On Sun, Feb 2, 2020 at 2:31 AM Damjan Jovanovic <[hidden email]>
>wrote:
>
>> Hi
>>
>> I've just pushed a "scons-build" branch.
>>
>> The build infrastructure is in main/scons. Python and SCons have to
>be
>> installed system-wide and available in $PATH. Currently we require
>Python
>> 3, but that's easy to change.
>>
>> So far main/fileaccess has been converted to SCons as a test. Its
>gbuild
>> files are still there; prj/makefile.mk determines whether to use
>gbuild
>> or scons. SCons is only used at a module level, build.pl is still the
>> launcher. You can build main/fileaccess in isolation by running
>"scons"
>> inside main/, or "scons -u" inside main/fileaccess, and clean by
>adding
>> "-c" (or "--clean") to those flags. It will correctly build the .cxx
>file,
>> link it into a library, install the library, run xsltproc on the
>.component
>> file, install the transformed .component file, and install the .xml
>file.
>> Everything can also be successfully cleaned.
>>
>> At present FreeBSD has been tested, and I will test Windows soon.
>Other
>> platforms don't exist, and still have to be added in
>> main/site_scons/platform.
>>
>> The converter is in gotoGBuild/, at the same level as main/ and
>test/. You
>> build it with Maven by running "mvn package". Then:
>>
>> java -cp target/classes
>org.apache.openoffice.gotoSCons.GBuildConverter
>> parsingAnalysis ../main
>> Attempts to parse each gbuild module, printing out errors for those
>that
>> couldn't be, and a summary of which could be parsed.
>> What is also useful is:
>> java -cp target/classes
>org.apache.openoffice.gotoSCons.GBuildConverter
>> parsingAnalysis ../main 2>&1 | grep java.lang.Exception >
>> /tmp/errorsByModule.csv
>> Then open /tmp/errorsByModule.csv in AOO, use # as the field
>separator,
>> and you get a table of modules that failed and a reason for each one.
>Sort
>> by column B, and you can see how often a reason occurs, for example
>21
>> modules need AllLangResTarget implemented. That can inform further
>> development.
>>
>> To actually convert a module to SCons, use one of the modules that
>> previous results said could be parsed, eg. io, and run:
>> java -cp target/classes
>org.apache.openoffice.gotoSCons.GBuildConverter
>> parseModule ../main/io/Module_io.mk
>> It will print the converted SCons file to standard output.
>>
>> Converting library names is currently broken. In main/fileaccess and
>> main/site_scons I've begun with dmake's way of naming libraries, like
>in
>> main/solenv/inc/libs.mk. GBuild re-did library naming by using 10
>> layer-specific rules, and then having tons of exceptions in
>> main/RepositoryFixes.mk; I am not sure which is worse. Maybe we
>should give
>> up the pretense, and just have a table in a CSV file, with platforms
>as
>> column headers, and library names as row headers, with the
>> platform-specific name in each field, and parse it with Python's
>built-in
>> CSV package on build startup?
>>
>>
>>
>> On Sat, Feb 1, 2020 at 8:10 PM Peter Kovacs <[hidden email]> wrote:
>>
>>> Hi Damjan,
>>>
>>> Let's try it. But I suggest to push to an own branch. There is no
>worth
>>> in trying and getting stuck in the same spot.
>>>
>>> Merge is done quickly. And it is great if others can have a look,
>too.
>>>
>>> All the best
>>> Peter
>>>
>>>
>>> Am 1. Februar 2020 13:36:42 MEZ schrieb Damjan Jovanovic <
>>> [hidden email]>:
>>> >Hi
>>> >
>>> >gbuild has become an unmaintainable nightmare. While there are only
>37
>>> >internal dmake modules left (+ another 37 external modules), I
>cannot
>>> >motivate myself to convert even 1 more. At my development speed of
>>> >about 2
>>> >lines of code every 8 hours, gbuild is a disaster that wastes
>>> >unbelievable
>>> >amounts of time, and it's really not a development I can say I am
>proud
>>> >of.
>>> >It has the ugliest syntax ever, it can only be debugged through
>>> >experimentation, and nobody really understands it. Also, it doesn't
>>> >fully
>>> >work, eg. a lot of the newer targets I've added don't get cleaned
>on
>>> >"make
>>> >clean", CustomTarget fails to deliver files sometimes, etc.
>>> >
>>> >To that end, I've restarting playing with an old idea: to switch to
>the
>>> >SCons build system, using a tool to automatically convert build
>files
>>> >from
>>> >gbuild to SCons.
>>> >
>>> >I got my previous scons build (of 1 module) up and running very
>>> >quickly,
>>> >and it still works. Then I continued development of my gbuild
>parser
>>> >and
>>> >automated converter, which I've called gotoSCons, and got it to
>parse
>>> >most
>>> >of the build instructions in Module_xxx.mk, Library_xxx.mk,
>>> >Executable_xxx.mk and Package_xxx.mk and convert them to a SCons
>build
>>> >file
>>> >(SConscript).
>>> >
>>> >gotoSCons is relatively simple, about 1200 lines of Java, and I can
>>> >provide
>>> >it to interested parties, or put it in a Git branch. It already
>found 3
>>> >bugs in these gbuild files:
>>> >1. main/xmloff/Package_inc.mk duplicates this line:
>>> >$(eval $(call
>>>
>>>
>>gb_Package_add_file,xmloff_inc,inc/xmloff/XMLTextShapeImportHelper.hxx,xmloff/XMLTextShapeImportHelper.hxx))
>>> >I've committed a fix in 85bfc14eebba4af4847075b1cf1eaecfa87bcfc4
>>> >2. main/bean/Module_bean.mk adds no targets.
>>> >3. main/testgraphical/Module_testgraphical.mk does nothing.
>>> >
>>> >The gotoSCons parser is very strict, it tries hard to guarantee a
>>> >correct
>>> >conversion, and will immediately fail on non-deterministic
>sections,
>>> >such
>>> >as "ifeq ($(OS),WNT)". It will also immediately fail on deliverable
>>> >types
>>> >that I haven't implemented yet, such as AllLangResTarget_,
>>> >StaticLibrary_
>>> >etc., and commands within targets that I haven't implemented yet
>such
>>> >as
>>> >gb_Executable_set_targettype_gui and gb_Library_use_externals. Out
>of
>>> >our
>>> >total of 105 gbuild modules, 53 use non-deterministic sections, and
>>> >will
>>> >require some degree of manual conversion, but the converter should
>>> >eventually be able to handle the other 52 modules. Even with the
>>> >limited
>>> >deliverable/command support at present, it can successfully convert
>19
>>> >modules. The conversion would be straightforward even if done
>manually,
>>> >as
>>> >both gbuild and SCons are high level build systems with similar
>>> >concepts:
>>> >includes, defines, cflags, libraries to link to, and so on (dmake
>is
>>> >the
>>> >difficult low-level tool).
>>> >
>>> >I love SCons. It's been easy integrating it into our build. It just
>>> >works.
>>> >It has beautiful clear syntax. It's understandable and debuggable.
>It
>>> >makes
>>> >development fun again. It's a well established build system, with a
>20
>>> >year
>>> >history, supporting Python 2 and 3, supporting almost every
>platform
>>> >out
>>> >there including OS/2, supporting every version of MSVC including
>Visual
>>> >Studio 2019, supporting advanced features like ELF sonames, library
>>> >version
>>> >symlinks and library name prefixes and suffixes, flex and bison
>targets
>>> >(which we all need), automatically generating cleaning targets for
>>> >every
>>> >build target, can use checksums instead of timestamps to avoid
>>> >unnecessary
>>> >rebuilds, automatic header dependency extraction, can build a
>module by
>>> >itself instead of the whole project, parallelizes building at a
>file
>>> >level
>>> >(across module boundaries). It can build at least some AOO modules
>>> >without
>>> >Cygwin. It is easy to add custom targets, unlike gbuild where it's
>>> >almost
>>> >impossible. It is equal to or better than gbuild in just about
>>> >everything,
>>> >and we don't have to maintain it going forward like we do with
>gbuild:
>>> >its
>>> >developers develop it, and we just use it.
>>> >
>>> >If someone would like to get involved, please let me know soon, so
>we
>>> >can
>>> >decide on an API, file layout, etc. before I've begun committing
>any
>>> >changes (and relax, they would be committed to a separate branch
>until
>>> >we
>>> >are convinced it works well).
>>> >
>>> >I was the major force that led us into more gbuild, but now I think
>>> >it's
>>> >time to leave it. It's already coming apart at the seams, and I
>don't
>>> >see
>>> >it being developed further. It's based on ($eval), an optional
>extra
>>> >added
>>> >quite late, as an afterthought for special cases, probably not how
>GNU
>>> >make
>>> >was intended to be used at scale, and a feature absent in other
>>> >"make"s.
>>> >The only other project in the world that uses gbuild is LO, and I
>>> >recall
>>> >reading how it took them an enormous amount of work to migrate to
>it
>>> >fully.
>>> >Let's learn from their mistake?
>>> >
>>> >Regards
>>> >Damjan
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: gotoSCons: automated gbuild to SCons conversion

Damjan Jovanovic
On Sun, Feb 9, 2020 at 11:47 AM Peter Kovacs <[hidden email]> wrote:

> Sounds nice.
>
> Thank you.

It would be great to have a debug output that can be checked to check which
> dependencies have been pulled. For me a declarative output would be cool to
> understand some code.
>
>
You can see within-module dependencies with "scons -u --tree=all" run in a
module:

+-fileaccess
  +-fileaccess/SConscript
  +-fileaccess/inc
  | +-fileaccess/inc/fileaccess
  |   +-fileaccess/inc/fileaccess/dllapi.h
  +-fileaccess/source
  | +-fileaccess/source/FileAccess.cxx
  | +-fileaccess/source/fileacc.xml
  +-fileaccess/util
    +-fileaccess/util/fileacc.component

Dependencies between modules can be seen in main/ with "scons
--tree=prune", which autodetects tons of headers, libraries, xml, etc. in
solver/, with transitive dependencies.

There are others, see
https://scons.org/doc/production/HTML/scons-user.html#chap-troubleshooting


>
> Also do you have an idea how to handle IDLs with scon? That would be
> awesome.
>
>
You mean the idlc + regmerge + cppmaker/javamaker toolchain? It should be
fairly easy, given I already ported part of it to Ant before
(main/solenv/ant/idl.xml), although that didn't do header dependencies in
.idl files.


> All the best
> Peter
>
>
Damjan