Memory Usage

Reducing the memory usage of our load modules is something that we are working on at present, so to kick off the blog with a very technical subject...

Much of the code that we develop relies on repeating group views to process lists of objects.

Group views in Gen need to have a fixed, maximum size, and memory for the group view is allocated at program load time. This means that there is a trade-off between the memory usage of a load module and the limits imposed by the group view size. In many cases you can size the group view to cater for the maximum number of rows that are needed, or repeat processing with some sort of 'start from' value.

However there are quite a few instances where this is not possible, and then the group view size becomes a fixed limit on the data that can be processed by the function.

Over the years we have increased these limits based on customer requirements, and this has resulted in an increased memory requirement for the load modules. This is particularly noticeable in Windows and UNIX environments where the server load modules are kept in memory by the Transaction Enabler. (Note that there are tuning parameters for the number of load modules that are loaded by the TE).

We have also made more use of DLLs in Windows and UNIX (Operations Libraries) and these can also increase the memory requirement if they result in a load module ending up loading a DLL for one action block and therefore also allocating the memory for all of the other action blocks in the opslib and any dependent opslibs. Our models have a lot of shared and reused code, and complex dependencies between modules has meant that many of the DLLs are loaded multiple times because of code dependencies, even if they are not required for each execution.

We are therefore working on some strategies to reduce memory usage. Ones that we have come up with so far include:

1) Delete unused views and attributes
2) Improve use of perfect view matching
3) Reduce use of uninitialised local views
4) Share group views between action blocks
5) Delay loading of DLLs
6) Delay allocation of local view storage

The first four are all standard coding techniques.

1) Maintaining an action block over many years (some of our action blocks are now 20 years old) can result in quite a few unused views. Finding and deleting these can be tedious and time-consuming, so we developed some tools in VerifIEr and genIE to automate finding and then deleting the unused views and also unused attribute views.

2) If view matching is not perfect, extra working storage is required in the generated code, so improving view matching (especially for large group views) will help.

3) In C generated code, having even one uninitialised local view will double the memory usage of all of the local views, so avoid this if you can.

4) An example of sharing memory is where many action blocks require a large local group view with the same structure. For example, we have a case where many action blocks need a 1Mb group view and each is called from the same parent. If each action block has its own local view, then each AB needs 1Mb of storage. However if the group view is defined as an exportable import view and is passed from a local view in the parent, then they all share the same storage. You then need to ensure that the view is properly initialised, but there is a big memory saving. In one load module we managed to halve the memory usage using this technique.

5) On windows we are experimenting with using the /DELAYLOAD: linker option to delay loading of opslibs. If the route through the code never requires a function in the opslib, it will not get loaded.

6) This is very much work in progress, so more on this later...