packaging rpms goes into endless loop (Pool:<file>.chk exists. Waiting 10 secons (<iteration>)

Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

packaging rpms goes into endless loop (Pool:<file>.chk exists. Waiting 10 secons (<iteration>)

Christian Lohmaier-2
Hi *,

doing a parallel build of instsetoo_native with --with-lang=ALL on
DEV300_m75 (build -- -P2) causes the packaging process to go into an
endless loop sooner or later. The packaging gets stuck with (for
example)

... Pool: /ooo/build/compile/instsetoo_native/util/../unxlngi6.pro/pool_rpm/ooobasis3.3-core07.pcf.check
exists. WAITING 10 seconds (80).

And never returns.

That turns up other questions in my mind:
* Apparently the process tries to use the same files for different
packages. "Why?"
* Is there high-level documentation on how rpm packging in
instsetoo_native works without having to read the perl-monster script?
What's up with the package pool, why generate md5sums for all
files?..and why does it query the finished rpm for individual
FILESIZES and sums those up, instead of querying SIZE directly?

ciao
Christian

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: packaging rpms goes into endless loop (Pool:<file>.chk exists. Waiting 10 secons (<iteration>)

Ingo Schmidt
Hi Christian,


On 04/02/10 14:54, Christian Lohmaier wrote:

> Hi *,
>
> doing a parallel build of instsetoo_native with --with-lang=ALL on
> DEV300_m75 (build -- -P2) causes the packaging process to go into an
> endless loop sooner or later. The packaging gets stuck with (for
> example)
>
> .. Pool: /ooo/build/compile/instsetoo_native/util/../unxlngi6.pro/pool_rpm/ooobasis3.3-core07.pcf.check
> exists. WAITING 10 seconds (80).
>
> And never returns.

If this happens, this is a bug, that needs to be fixed. But in the last
years of using the pool process, this never occured. Did you also change
other things in your current build environment?
And is this problem reproducable? And does it occur after you removed
the package pool completely?

The process does not wait endless. In the moment there is a lock time of
1 hour. If the lock file of a specific process exists for one hour,
another process is allowed to remove it, assuming, that the first
process is no longer alive. Packaging of one package should never take
one hour ;-)

Furthermore there is always a way to skip the pool process, so that it
never hinders you at your work. Please insert in
instsetoo_native/util/openoffice.lst for your product the line:

POOLPRODUCT    0

or simply change the value for this property in the global section at
the beginning of this file.

> That turns up other questions in my mind:
> * Apparently the process tries to use the same files for different
> packages. "Why?"

Hm, I do not know, if I understand you correctly. The same files cannot
be included into different RPMs. In this case the installation of the
product will not be possible, because rpm shows an error like: "File xyz
is already part of package abc.rpm".
The idea of the package pool is, to use the same packages for different
products. If you create an English OOo installation set and later you
want a German OOo installation set, than only the language dependent
packages have to be created. So the creation of the German installation
set will be very fast.

> * Is there high-level documentation on how rpm packging in
> instsetoo_native works without having to read the perl-monster script?

No, this packaging process is very dynamic and expanded permanently. rpm
packaging is only a very small part of it. At the moment I make very big
changes for the creation of micro-msi-packages for Windows, so that we
can support the pool process for Windows in the future, too.

> What's up with the package pool, why generate md5sums for all
> files?..and why does it query the finished rpm for individual
> FILESIZES and sums those up, instead of querying SIZE directly?

For the pool process only the md5sums of the files are important. A rpm
is created only, if its content has changed, compared with the rpm in
the pool. So for every file inside the RPM and for the spec file
required for the rpm creation, the md5sum is stored. If only one md5sum
is different in the current packaging process, it is obviously necessary
to package this RPM again and to store the new package in the pool.
The file size is saved for later usage in the Java GUI Installer that
installs the RPM and displays and calculates required disc spaces (only
in the product "OOo with JRE"). This is a platform independent way, to
save the file sizes.
Ciao

   Ingo


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: packaging rpms goes into endless loop (Pool:<file>.chk exists. Waiting 10 secons (<iteration>)

Christian Lohmaier-2
Hi Ingo, *,
(now for the mailinglist as well)

On Tue, Apr 6, 2010 at 4:40 AM, Ingo Schmidt - Sun Germany - ham02 -
Hamburg <[hidden email]> wrote:

> On 04/02/10 14:54, Christian Lohmaier wrote:
>>
>> doing a parallel build of instsetoo_native with --with-lang=ALL on
>> DEV300_m75 (build -- -P2) causes the packaging process to go into an
>> endless loop sooner or later. The packaging gets stuck with (for
>> example)
>>
>> .. Pool:
>> /ooo/build/compile/instsetoo_native/util/../unxlngi6.pro/pool_rpm/ooobasis3.3-core07.pcf.check
>> exists. WAITING 10 seconds (80).
>>
>> And never returns.
>
> If this happens, this is a bug, that needs to be fixed. But in the last
> years of using the pool process, this never occured. Did you also change
> other things in your current build environment?

No, didn't change anything. But I guess it was related to a
"close-to-disk-full" situation with slow i/o.

> And is this problem reproducable? And does it occur after you removed the
> package pool completely?

With that buildtree: Yes, not always at the same time/with the same
file, but regularily.

Not reproducible anymore after providing more diskspace.
Thus I guess it is not worth looking into it anymore, esp. as I now
disable creation of full-installsets for the additional languages (to
save time and diskspace)
I also remove the "unpacked" rpms, and only keep the _download
variant. (by patching instsetoo_native/util/makefile.mk accordingly)

> The process does not wait endless. In the moment there is a lock time of 1
> hour. If the lock file of a specific process exists for one hour, another
> process is allowed to remove it, assuming, that the first process is no
> longer alive. Packaging of one package should never take one hour ;-)

Ah, I'm way too impatient for that - a en-US build is completed in
less than one hour on that machine :-)

> Furthermore there is always a way to skip the pool process, so that it never
> hinders you at your work. Please insert in
> instsetoo_native/util/openoffice.lst for your product the line:
>
> POOLPRODUCT    0
>
> or simply change the value for this property in the global section at the
> beginning of this file.

Thanks - will remember that.

> [...]
> The idea of the package pool is, to use the same packages for different
> products. If you create an English OOo installation set and later you want a
> German OOo installation set, than only the language dependent packages have
> to be created. So the creation of the German installation set will be very
> fast.

Ah, OK thanks for the explanation - but "very fast" is relative :-)
So when I only create en-US full installset (and only do languagepacks
for the other languages), the packaging will never make use of the
pool?

>> * Is there high-level documentation on how rpm packging in
>> instsetoo_native works without having to read the perl-monster script?
>
> No, this packaging process is very dynamic and expanded permanently. rpm
> packaging is only a very small part of it. At the moment I make very big
> changes for the creation of micro-msi-packages for Windows, so that we can
> support the pool process for Windows in the future, too.

too bad :-/

> [md5sum/filesizes explanation]

Thanks a lot

ciao
Christian

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: packaging rpms goes into endless loop (Pool:<file>.chk exists. Waiting 10 secons (<iteration>)

Ingo Schmidt
Hi Christian,


> No, didn't change anything. But I guess it was related to a
> "close-to-disk-full" situation with slow i/o.

hm, interesting. I would expect that "disk-full" causes problems, but
not "close-to-disk-full" ;-)

>> The process does not wait endless. In the moment there is a lock time of 1
>> hour. If the lock file of a specific process exists for one hour, another
>> process is allowed to remove it, assuming, that the first process is no
>> longer alive. Packaging of one package should never take one hour ;-)
>
> Ah, I'm way too impatient for that - a en-US build is completed in
> less than one hour on that machine :-)

I see, for your scenario waiting one hour is not a good value. But for
nightly builds it is very comfortable.

>> The idea of the package pool is, to use the same packages for different
>> products. If you create an English OOo installation set and later you want a
>> German OOo installation set, than only the language dependent packages have
>> to be created. So the creation of the German installation set will be very
>> fast.
>
> Ah, OK thanks for the explanation - but "very fast" is relative :-)
> So when I only create en-US full installset (and only do languagepacks
> for the other languages), the packaging will never make use of the
> pool?

The package pool is filled, if you create more and more installation
sets. If you have to create every package exactly once, the package pool
cannot offer any advantage for you. But typically packages are created
more than once, because you create an installation set a second time, or
because you make a further installation set, that uses the same
packages. If you know, that you do not need the pool, simply set the
property POOLPRODUCT to 0.

Ciao

  Ingo

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]