Memory management strategy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory management strategy

Patricia Shanahan
Currently, AOO uses a lot of finite size arrays. Ideally, overflows will
be detected and lead to an error message, but that still prevents
processing of the user's documents.

Computer memories and swap spaces are still increasing in size, so the
array size limits are more likely to be the limiting factor on document
sizes.

I can see two main strategies, but I'm open to additional suggestions.

1) Keep the finite size arrays, but periodically review them to see if
the sizes should be increased.

2) Use automatically extending STL structures. If a document is too big
or uses too many instances of some array element, the symptom will be
the process as a whole running out of memory. We would have to make sure
all memory allocation failures are handled smoothly.

This somewhat ties in with exception handling strategy. As we turn what
used to be silent out-of-bounds conditions into exceptions, it is
important to avoid arbitrary catching and hiding of all exceptions.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Peter Kovacs-3
Documents that are to large for the memory available will also fail to
load in  the 1st approach.
To add array review mechanism is in my opinion only a source of errors.
I think STL should be used.

+1 for not hiding the exceptions. Maybe we could catch exception on
document level quickly and process the directly?

On 25.03.2018 16:21, Patricia Shanahan wrote:

> Currently, AOO uses a lot of finite size arrays. Ideally, overflows
> will be detected and lead to an error message, but that still prevents
> processing of the user's documents.
>
> Computer memories and swap spaces are still increasing in size, so the
> array size limits are more likely to be the limiting factor on
> document sizes.
>
> I can see two main strategies, but I'm open to additional suggestions.
>
> 1) Keep the finite size arrays, but periodically review them to see if
> the sizes should be increased.
>
> 2) Use automatically extending STL structures. If a document is too
> big or uses too many instances of some array element, the symptom will
> be the process as a whole running out of memory. We would have to make
> sure all memory allocation failures are handled smoothly.
>
> This somewhat ties in with exception handling strategy. As we turn
> what used to be silent out-of-bounds conditions into exceptions, it is
> important to avoid arbitrary catching and hiding of all exceptions.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Pedro Giffuni
In reply to this post by Patricia Shanahan
Somewhat related ...

I have been considering the use of APR pools:

http://www.apachetutor.org/dev/pools

It would be great to have the memory managed by the same technology used
in Apache httpd.

Cheers,

Pedro.



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Patricia Shanahan
On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
> Somewhat related ...
>
> I have been considering the use of APR pools:
>
> http://www.apachetutor.org/dev/pools
>
> It would be great to have the memory managed by the same technology used
> in Apache httpd.

I need to think about this. It seems very appropriate for transaction
processing. I am not so sure it is a good fit for AOO.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Peter Kovacs-3
On 25.03.2018 21:28, Patricia Shanahan wrote:

> On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
>> Somewhat related ...
>>
>> I have been considering the use of APR pools:
>>
>> http://www.apachetutor.org/dev/pools
>>
>> It would be great to have the memory managed by the same technology
>> used in Apache httpd.
>
> I need to think about this. It seems very appropriate for transaction
> processing. I am not so sure it is a good fit for AOO.
Maybe in part. It could be a good thing to force extentions / macros
into a pool concept. If the Extention / Macro ends functioning it gets
cleaned up, and we are sure that it is cleaned up completly.
Just as an Idea.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Pedro Giffuni
In reply to this post by Pedro Giffuni
> On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
> > Somewhat related ...
> >
> > I have been considering the use of APR pools:
> >
> > http://www.apachetutor.org/dev/pools
> >
> > It would be great to have the memory managed by the same technology used
> > in Apache httpd.
>
> I need to think about this. It seems very appropriate for transaction
> processing. I am not so sure it is a good fit for AOO.

We do a lot of different memory processing in AOO: in some cases it may
be more useful that in others.

For the time being I was only looking at using pools to replace some
malloc()/calloc()s in SAL but I haven't found time to do a PoC.

Pedro.


Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Damjan Jovanovic
In reply to this post by Peter Kovacs-3
A related problem to consider is the generally poor runtime environment
support for using C++ between modules. On Windows, using C++ between
different DLLs requires they were built by the same MSVC version and used
the same CRT runtime settings, and I am finding that for the main/jvmaccess
module to compile and link and be linked to from other modules in gbuild,
class typeinfos must be hidden and symbols for methods selectively
exported, on a case-by-case basis 🙈.

Exceptions in UNO components must be declared all the way from the IDL
file, so that the UNO bridge can catch them and re-create and rethrow them
on the caller's side (who is possibly calling that method using an RPC from
another process/machine). You can't just throw whatever you want, which the
C++ language itself allows. Changes to IDL files break compatibility with
clients using older versions, and are generally avoided, you can see many
examples of how an UNO interface/service got deprecated and a new one got
added instead.

Damjan


On Sun, Mar 25, 2018 at 5:18 PM, Peter Kovacs <[hidden email]> wrote:

> Documents that are to large for the memory available will also fail to
> load in  the 1st approach.
> To add array review mechanism is in my opinion only a source of errors.
> I think STL should be used.
>
> +1 for not hiding the exceptions. Maybe we could catch exception on
> document level quickly and process the directly?
>
>
> On 25.03.2018 16:21, Patricia Shanahan wrote:
>
>> Currently, AOO uses a lot of finite size arrays. Ideally, overflows will
>> be detected and lead to an error message, but that still prevents
>> processing of the user's documents.
>>
>> Computer memories and swap spaces are still increasing in size, so the
>> array size limits are more likely to be the limiting factor on document
>> sizes.
>>
>> I can see two main strategies, but I'm open to additional suggestions.
>>
>> 1) Keep the finite size arrays, but periodically review them to see if
>> the sizes should be increased.
>>
>> 2) Use automatically extending STL structures. If a document is too big
>> or uses too many instances of some array element, the symptom will be the
>> process as a whole running out of memory. We would have to make sure all
>> memory allocation failures are handled smoothly.
>>
>> This somewhat ties in with exception handling strategy. As we turn what
>> used to be silent out-of-bounds conditions into exceptions, it is important
>> to avoid arbitrary catching and hiding of all exceptions.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Peter Kovacs-3
In reply to this post by Pedro Giffuni
I would like to pick the discussion up again.

In general I prefer a Class type approach over a Function based approach
in Code methology. Since Class based has the "buildIn" RAI approach, I
think the pool approach does not make sense when classes are used. When
ever the desicion goes in favour of function based programing we should
concider the pool based memory management for data structures. Therefore
a structure that has not an owner is owned and managed by a pool instead
of a class.

That has some logic to me. Is there something I miss?


On 25.03.2018 22:59, Pedro Giffuni wrote:

>> On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
>> > Somewhat related ...
>> > > I have been considering the use of APR pools:
>> > > http://www.apachetutor.org/dev/pools
>> > > It would be great to have the memory managed by the same
>> technology used > in Apache httpd.
>>
>> I need to think about this. It seems very appropriate for transaction
>> processing. I am not so sure it is a good fit for AOO.
>
> We do a lot of different memory processing in AOO: in some cases it
> may be more useful that in others.
>
> For the time being I was only looking at using pools to replace some
> malloc()/calloc()s in SAL but I haven't found time to do a PoC.
>
> Pedro.
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Damjan Jovanovic
The Mozilla project also uses C++, also started in a similar timeframe to
StarOffice, also has a huge codebase, also uses a component-based
development methodology, and so on.

Lately, they've dealt with memory issues by developing in another language
that is memory-safe: Rust.

We could learn from them.

On Tue, Apr 17, 2018 at 7:26 AM, Peter Kovacs <[hidden email]> wrote:

> I would like to pick the discussion up again.
>
> In general I prefer a Class type approach over a Function based approach
> in Code methology. Since Class based has the "buildIn" RAI approach, I
> think the pool approach does not make sense when classes are used. When
> ever the desicion goes in favour of function based programing we should
> concider the pool based memory management for data structures. Therefore a
> structure that has not an owner is owned and managed by a pool instead of a
> class.
>
> That has some logic to me. Is there something I miss?
>
>
>
> On 25.03.2018 22:59, Pedro Giffuni wrote:
>
>> On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
>>> > Somewhat related ...
>>> > > I have been considering the use of APR pools:
>>> > > http://www.apachetutor.org/dev/pools
>>> > > It would be great to have the memory managed by the same technology
>>> used > in Apache httpd.
>>>
>>> I need to think about this. It seems very appropriate for transaction
>>> processing. I am not so sure it is a good fit for AOO.
>>>
>>
>> We do a lot of different memory processing in AOO: in some cases it may
>> be more useful that in others.
>>
>> For the time being I was only looking at using pools to replace some
>> malloc()/calloc()s in SAL but I haven't found time to do a PoC.
>>
>> Pedro.
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Memory management strategy

Peter Kovacs-2
I am not so sure that will help us.
What are the pros what the cons?

Am 17. April 2018 08:13:33 MESZ schrieb Damjan Jovanovic <[hidden email]>:

>The Mozilla project also uses C++, also started in a similar timeframe
>to
>StarOffice, also has a huge codebase, also uses a component-based
>development methodology, and so on.
>
>Lately, they've dealt with memory issues by developing in another
>language
>that is memory-safe: Rust.
>
>We could learn from them.
>
>On Tue, Apr 17, 2018 at 7:26 AM, Peter Kovacs <[hidden email]> wrote:
>
>> I would like to pick the discussion up again.
>>
>> In general I prefer a Class type approach over a Function based
>approach
>> in Code methology. Since Class based has the "buildIn" RAI approach,
>I
>> think the pool approach does not make sense when classes are used.
>When
>> ever the desicion goes in favour of function based programing we
>should
>> concider the pool based memory management for data structures.
>Therefore a
>> structure that has not an owner is owned and managed by a pool
>instead of a
>> class.
>>
>> That has some logic to me. Is there something I miss?
>>
>>
>>
>> On 25.03.2018 22:59, Pedro Giffuni wrote:
>>
>>> On 3/25/2018 10:44 AM, Pedro Giffuni wrote:
>>>> > Somewhat related ...
>>>> > > I have been considering the use of APR pools:
>>>> > > http://www.apachetutor.org/dev/pools
>>>> > > It would be great to have the memory managed by the same
>technology
>>>> used > in Apache httpd.
>>>>
>>>> I need to think about this. It seems very appropriate for
>transaction
>>>> processing. I am not so sure it is a good fit for AOO.
>>>>
>>>
>>> We do a lot of different memory processing in AOO: in some cases it
>may
>>> be more useful that in others.
>>>
>>> For the time being I was only looking at using pools to replace some
>>> malloc()/calloc()s in SAL but I haven't found time to do a PoC.
>>>
>>> Pedro.
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]