Discovering hidden errors

Since we originally developed our VerifIEr QA and code checking product over 2 years ago, the scope of checks has enlarged in some very interesting directions.

The initial checks were primarily aimed at standards enforcement, for example, object names, CBD architecture enforcement, use (or abuse) of various properties (e.g. READ properties), etc.

More recently however we have developed a number of checks for customers that are aimed at improving code quality by detecting errors that might be otherwise difficult to find.

We have run these checks on our own products (which are developed with Gen) and have been surprised at the number of potential errors that have been encountered. Usually the more serious errors are detected during testing, but sometimes not!

Examples of the checks that we have found especially useful include:

1. Local views that are referenced but never set to a value, indicating that either the local views need to be populated or the code is no longer required;
2. Hidden fields (for example fields on a GUI that are placed above the top border) that are not read-only (the user could therefore tab to the field and change its value);
3. Export views not fully mapped to an import view on a screen or window;
4. Checking the tab sequence for GUI windows and dialog boxes;

These sorts of errors are notoriously difficult to spot via code inspection and can also be missed during testing.

Gen r8 Beta Testing

October and November were busy months with 3 Edge conferences to attend and the start of the Gen r8 beta test period, so apologies for no blog posting for the past few weeks.

We have started our beta testing of the next Gen release (r8). There are a large number of new and changed features that will keep us busy with the development of the 8.0 release of our products that will support Gen r8.

A number of customers have asked about the availability of IET products to support their beta testing of Gen r8. Unfortunately we have only been able to start development of Release 8.0 of our products with the start of the beta programme and hence cannot offer a beta version of our products until this development work is complete.

Unused and Duplicate Prompts

We have been developing our products for close to 20 years now and one of the consequences has been that we have found quite a few unused and duplicate prompts in the models. We also have a multiple model architecture and a policy of migrating the entire data model to each of the development models. This results in all of the prompts also being duplicated (and unused) in all of the models.

Apart from having a large number of redundant prompts in the models, it can also make the selection of prompts in the window/screen designer tedious because the large lists of unused and duplicate prompts makes locating the desired prompt harder.

There is a Gen function in the toolset to delete unused prompts, but this requires the model to be downloaded, and ours are too big. It will also not get rid of duplicates.

We have therefore written a new genIE function to both delete unused and consolidate duplicate prompts.

The result is faster downloads because you are not downloading extra prompts and also easier selection of existing prompts in the window/screen designer.

RI Trigger Impact Analysis

The previous post discussed the need to regenerate programs that call changed RI triggers. The difficulty is in performing this impact analysis.

The Gen model does contain associations between action blocks and RI triggers (using IMPUSE associations if you are familiar with the Gen schema), but the important point to note is that these associations are maintained by the code generators. This means that if you change the data model, the IMPUSE association data in the model is not accurate until you have regenerated the affected code, which makes it useless in helping you understand what you need to regenerate!

We faced this issue when developing the impact analysis process that GuardIEn performs when it detects data model changes. To work out what RI triggers are directly affected by a data model change is straight forward, but the consequential impact on other triggers and action block / procedure steps involves a complex navigation though the data model, following cascade delete chains for example. It is also affected by the choice of generated or DBMS enforced RI rules.

RI Triggers

In my experience, one of the more misunderstood aspects of Gen development is RI triggers, and the impact of making a change in the data model.

In many cases, users think that they only need to regenerate the RI triggers once they have made their data model changes. Unfortunately it is more complex than that.

The correct process is:

1) Implement any changes to table or column names in the data structure
2) Run the Ref. Integrity Process (accessed from the toolset Design menu) to synchronise the technical design RI constraints with the data model
3) Generate the affected RI triggers
4) Re-generate all action blocks that call the re-generated RI triggers
5) Install re-generated code

Steps (1) and (2) can either be performed separately or accomplished using the Retransformation tool.

The reason why you need to regenerate the action blocks that call the RI triggers is that the code generated for a DELETE, DISASSOCIATE, TRANSFER, etc. depends on the RI rules in the model. If these rules change (for example changing a relationship from cascade delete to pendent delete), then the action blocks require regeneration as well as the RI triggers.

Where is your source code?

For your Gen applications, the answer is of course that it is stored in the Gen encyclopaedia. But what about your non-Gen code like:

• External Action Blocks
• OLE files
• Bitmaps, icons
• DDL
• Documentation
• etc.

Unfortunately we often come across projects that do not adequately manage the non-Gen source code, probably because they do not have this issue with the Gen code. There are cases of projects where they have lost their EAB source code and do not even have documentation for what the EAB did apart from the stubs in the Gen model.

Another common instance is where you use an OCX control in a GUI design. Gen creates a .ole file in the local workstation model sub-directory which contains properties for the OCX control. However this is not uploaded to the model, so if you delete the local model directory and have not saved the .ole files elsewhere, they are lost!

You should therefore take care that all project source code is properly managed. This could be as basic as ensuring that the files are stored centrally in a place where they will not be deleted, either accidentally or as part of a housekeeping routine.

Even better, the external code should be properly versioned controlled. There are many tools for this, ranging from simple and free source code control tools to more sophisticated products. Our own XOS tool has been designed specifically for managing Gen externals, including support for automatically versioning .ole files when a subset is uploaded to the encyclopaedia.

An unexpected feature of Object Migration

An unexpected feature of the CA Gen object migration utility that sometimes catches us out is when you migrate an action block and the view matching of action blocks that use it is also affected.

Consider the example where AB1 uses AB2. If you add a new import view to AB2 and view match it to an existing view in AB1:

AB1:
USE AB2
IMPORTS: temp xxx to in xxx

and then migrate just AB2 to another model, if the view temp xxx in AB1 has common ancestry between the two models, then the view match is also migrated, which in effect modifies AB1 even though this had not been selected for migrate. However AB1 does not get a modified timestamp, so it looks like AB1 has not changed even though its view matching has.

This may not necessarily create a problem, but it does sometimes cause confusion.

Gen and null columns

A recent posting on the Duick forum regarding NULL column support lead to a discussion on the Gen qualifier IS EQUIVALENT TO and a potential misinterpretation of the way that this works.

As a bid of background information, it is important to understand how a nullable column containing a NULL value behaves. Consider a table with a nullable column and these rows:

Id opt_column
1 ' ' (column has a value of spaces)
2 NULL(column is NULL
3 'X' (column has a value of X)

If you want a value of NULL to be equivalent to SPACES, and you want to read rows that have a space or null in the opt_column, then if your SQL was:

SELECT * FROM table WHERE opt_column = ' ';

would return just row 1 but

SELECT * FROM table WHERE opt_column = ' ' OR opt_column IS NULL;

would return rows 1 and 2

If you want to read rows that do not have ‘X’:

SELECT * FROM table WHERE opt_column != 'X';

would return just row 1 but

SELECT * FROM table WHERE opt_column != 'X' OR opt_column IS NULL;

would return rows 1 and 2

Once you understand the need for the IS NULL or IS NOT NULL qualifier in the SQL, you can write the READ qualifiers in the action diagram code.

The confusion arises over the use of the IS EQUIVALENT TO clause since it is likely that this does not work the way you expect!

For example, if an optional column has no value, then I think of SPACES & NULL as the same, so you would code:

READ table WHERE opt_column = SPACES OR opt_column IS NULL

However the statement READ table WHERE opt_column IS EQUIVALENT TO SPACES

gives the following SQL which is not the same:

SELECT opt_column FROM table
WHERE (opt_column = ' ' AND opt_column IS NOT NULL)

This means that if the column is NULL it will not return a row, which is the opposite of what I think you want.

IS EQUIVALENT TO is OK. For example, if you want a row with a specific value:

READ table WHERE opt_column = ‘X’ AND opt_column IS NOT NULL

Is the same as READ table WHERE opt_column IS EQUIVALENT TO ‘X’

which gives the following SQL:

SELECT opt_column FROM table
WHERE (opt_column = 'X' AND opt_column IS NOT NULL)

IS NOT EQUIVALENT also gives the desired result for a specific value but not for SPACES:

READ table WHERE opt_column NOT = ‘X’ OR opt_column IS NULL

and READ table WHERE opt_column IS NOT EQUIVALENT TO ‘X’ gives:

SELECT opt_column FROM table
WHERE (opt_column <> 'X' OR opt_column IS NULL)

But if you want a row where the column is not spaces, you would code:

READ table WHERE opt_column NOT = SPACES AND opt_column IS NOT NULL

but READ table WHERE opt_column IS NOT EQUIVALENT TO SPACES gives:

SELECT opt_column FROM table
WHERE (opt_column <> ' ' OR opt_column IS NULL)

In summary, it is best not to use EQUIVALENT with SPACES unless you want the behaviour that the generated code gives you. You would also need to be careful with a view that might have a value of spaces.

Documenting Changes

Whether you are developing software that is commercially available or for internal use, users value a clear description of the enhancements and fixes introduced in a new release or service pack. This is especially important when there are changes in behaviour or actions that need to be taken by the users to take advantage of new features.

A customer recently complemented us on the quality of our release notes and asked whether we generated them from a database.?

Unfortunately there wasn’t a magic solution. We cut and paste the descriptions of each change into a Word document and then generate the PDF file from that. That part is simple though. The harder part is to ensure that each change is documented properly and is not accidentally omitted from the release notes. We therefore document the changes using the following process:

1) Each change must be documented in a form that will make sense to the end user, explaining the business reason for the enhancement or requirement for a fix. The documentation is in the long description of the Change Request (CR) in GuardIEn.

2) Each CR should address a single problem or enhancement. You should avoid CRs that span multiple requirements (the worst cases being a single CR that has all changes in it or a CR for changes made by a developer that is not linked to the actual requirements.

3) If a new requirement is found whilst changing some code, a new CR should be created for it and the temptation to ‘hide’ the new requirement within the scope of the existing CR avoided.

4) Once the CR has been completed and tested, the description should be reviewed for accuracy and any changes in behaviour noted.

5) The Release Notes should be updated with the CR and user documentation reviewed and updated as necessary. We have a separate state in the CR life-cycle to indicate that the documentation has been updated.

Almost like having a new machine

We recently upgraded our anti-virus software to the latest 2010 release and it was immediately noticeable how much slower our machines were, especially our heavily used CSE machine. We also found the desktops to be much slower, and so the new release was de-installed and the older (2009) release re-installed. It still took up to 25% of the CPU though, and so we decided to try some alternatives. After a bit of research, we selected one of the other leading products to trial. Both desktops and the CSE run much faster and it is like having a new machine and so the upgrade can be delayed for a while!

In praise of integration

Having spent over 20 years developing our products using Gen, it is clear that one of the main benefits is the low cost of maintaining applications developed with Gen. I think that there are many reasons for this, some of which are due to inherent features of Gen and others derive from the methods and standards used by the development project. In my view, a key feature of Gen that contributes to the low cost of maintenance is the integrated nature of the analysis and design tools.

The early marketing of IEF (as Gen was called in the early days) emphasised the integrated nature of the product and IEF was called an i-CASE (integrated Computer Aided Software Engineering) tool to distinguish it from point solution CASE tools. Unfortunately many i-CASE tools were nothing of the sort and few if any came close to delivering the 100% code generation and great success of Gen. This resulted in the CASE / i-CASE market getting a bad name, through little fault of IEF.

However. having chosen the best integrated development tool, shouldn’t a Gen project maximise the benefits of that integration? The trend to only use Gen for the server and batch parts of a project concerns me. Whilst there are undoubtedly situations where Gen is not the best choice for developing the user interface, I suspect that there are others where the choice not to use Gen for the front-end has been a mistake due to the resulting increased cost of development and maintenance.

When the user interface is developed with a separate tool, the interface between the presentation layer (client) and the business logic (server) has to become much more formalised at an early stage in the life-cycle, especially when the client and server parts are developed by separate teams. Even if you are using CBD/SOA or some other development approach that advocates stable, published interfaces, there are still many situations when a rapid, iterative approach to development will benefit from having one person develop the client and its closely coupled servers at the same time and with the same tool.

The goal of 100% code generation and integrated nature of Gen means that there are boundaries to the product's capabilities. Whilst there are features that allow external code (external action blocks, OCX controls, etc.), there are still limitations on what can be accomplished with Gen. The perceived weakness of Gen for developing sophisticated user interfaces has made some Gen projects avoid Gen for the user interface or presentation layer of an application.

A few years ago, I was visiting a long standing Gen user who had used Gen very successfully to develop 3270 and batch applications. I demonstrated GuardIEn to the development manager, and then we went for lunch. He explained that they were now moving to client/server but had decided not to use Gen for the front end because they did not think that you could develop a good front end. I asked him what they were looking for, and his response was that they would like to be able to develop something that looked like GuardIEn! He did not realise that GuardIEn was a Gen developed application with the user interface created using the same Gen design tools that they had decided were inappropriate.

Now, to achieve the sophisticated look and feel of our products with Gen has not been easy. We have had to develop an add-on tool (IETeGUI) and learn how best to achieve the desired results. But is this not the case with any tool? Don’t just take the product out of the box and expect to develop a very sophisticated user interface immediately. It needs quite a bit more work than that – probably more than you would expect. It is not easy to create a great user interface with Gen, but it can and has been done, and in my view, the extra effort is more than compensated for by the significant reduction in development and maintenance effort through the use of an integrated tool with 100% code generation.

Dog Food or Champagne?

There is a saying about eating your own dog food, or the more pleasant version, drinking your own champagne. The point is that if you really believe in your own product, then you would use it yourself, and therefore I prefer the dog food analogy since you would only eat your own dog food if it was really palatable, whereas you might be prepared to drink anything that is alcoholic!

Anyway, getting back to the main point, if you are a software developer and you can use your own products, then you have a big incentive to improve them for your own benefit. This is why I was really pleased when I heard that CA would be using Gen within their development team as part of the Mainframe 2.0 initiative.

Because we develop our products with Gen, we are also able to use our own tools as well, and this positive feedback loop has resulted in many improvements and enhancements to make the ‘dog food’ as palatable as possible. An example of this is in the area of version control.

One of the most useful tools in the armoury of a developer is the ability to see what has changed in the source code. The ability to see the what, why, when and who (what has changed, why was it changed, when was the change made and who made it) makes diagnosing a problem much easier. With Gen, a single model can only contain a single version of an object, so if the object is changed, you lose the ability to see what it looked like the moment before the change, unless you have saved the previous version somehow (via migration, model copy, etc.).

Since it is impractical to save the previous version every time a change is made, often the diagnosis of a problem is made unnecessarily hard because this useful information is not available. For example, a user reports a problem in the test system that they noticed a few days ago. In the meantime, the model has been changed and you are therefore unable to see what the changes were (only that the object was last changed on a specific date/time). If you cannot reproduce the problem, you cannot then tell if the problem has been fixed, or if your test case does not properly test for the issue.

We have found the ‘minor versions’ feature of GuardIEn especially useful. This allows you to track every change made to a Gen object and see the who changed it, when it was changed and what was changed (down to properties and individual action diagram statements). When linked to a GuardIEn Change Request, you can also see why it was changed and what other objects were also affected by the same change.

I know that we would say this anyway, but we have found this capability to be invaluable in the on-going maintenance of our products.

Extending the GUI runtime

Earlier postings have discussed the benefits of a runtime, for example, simplifying the development environment and allowing developers to concentrate on the business logic. We discussed how we have extended the runtime with custom action diagram functions. However the area where we have made the most use of a customised runtime is the GUI interface.

It is no secret that the standard Gen window designer offers limited features and the resulting look and feel often leaves users underwhelmed!

The majority of our products are client/server applications with a Windows GUI presentation layer designed in Gen. A few years ago we decided that it was important to improve the usability of the products and one of the key areas that we felt needing enhancement was the GUI front-end.

Some of the areas that we felt could be improved were:

• Resizable windows and dialog boxes. These have to be designed for a minimum screen resolution (for example 800x600), but if a user has a larger monitor, it would be useful for them to be able to have larger window/dialog boxes. Two issues in a Gen application are a) if you resize a window, you then need a lot of code to reposition/resize the controls on the dialog box and b) the new size should be remembered since the user will not want to resize the dialog box each time they open it.



• List Boxes. Most windows applications utilise list boxes where the user can change column widths, re-sequence columns, sort on columns by clicking on the column heading. Often a bitmap will make the data much more readable.
  
  


• Tabs. These can reduce the complexity of an application that requries a lot of dialog boxes.

We investigated various approaches to implementing the above requirements. Most solutions involved a lot of additional code (for example to move controls around when a window is resized or to manipulate an OCX control) and would have involved a lot of effort. We also wanted to keep a common look and feel across all parts of the products and hence if we decided to enhance the products, we would have to re-code all of the windows and dialogs and not just the commonly used ones.

This is where we come back to the concept of a runtime. In our normal use of Gen, we did not have to add in a lot of window handling code – this was handled by the Gen GUI runtime. What if this runtime were extended to support our additional requirements? It transpired that a Gen list box could be ‘enhanced’ via Windows functions and still appear as a normal list box to the Gen GUI runtime. We were therefore able to extensively enhance and improve the functionality of the Gen GUI without needing to change either the action diagram code or the Gen GUI runtime.

Simple NOTE statements in the action diagram were used to provide parameters to the extended runtime and we were able to enhance the entire product line in a matter of days because the complexity was ‘hidden’ in the runtime functions and did not have to be coded by the developer.

The example below illustrates a window that utilises enhanced list boxes that can display bitmaps, it can be resized and the list boxes are resized automatically to fill the window, and various tabs are available with associated bitmaps as well.

The results of this effort are now available to other Gen sites as the IETeGUI product.

Beware hidden READ properties

We encountered a problem recently where some code worked fine on our main Oracle test system but did not work correctly on z/OS DB2.

We could not see anything wrong, so ran the code in trace using xTrace and it was then that it was noticed that there was a LIMIT 1 clause set on a READ EACH properties. This limits the rows returned to 1, but only for MS/SQL or DB2. The statement should not have had the property set, but had been copied from another READ EACH that did have the LIMIT since there was an ESCAPE after the first READ and hence it worked correctly in all databases.

This highlights the danger of using properties that affect the results set, for example, LIMIT or DISTINCT since these are not visible to the designer when viewing the action diagram.

My recommendation would be that if you decide to use these properties, add a NOTE statement indicating this above the statement. This will act as a warning if the statement is copied, and also should also indicate that the properties might need modifying.

Client/Server Versioning

Our products are designed as client/server applications where the client code resides on the user’s workstation and the server code resides on a central windows, UNIX or z/OS server.

The majority of our customers install the client code on each user’s workstation, though it is also possible to install it on a central file server. Because the changes to the workstation software are often strictly controlled and require the involvement of desktop support departments, in many sites there are restrictions on the timing and frequency that desktop software can be updated.

With a Gen client/server application, if the interface to a server changes (i.e. import/export views), then the client window manager needs to be re-generated and therefore a new version of the client software is required.

This presents two issues with the development and deployment of new releases of software.

The first issue is to prevent the client software invoking the wrong version of the server software. For example, if the server software is updated and a user attempts to access it with an older version of the client software, the software might fail (typically with a common format buffer error) or even worse, complete execution but with incorrect results.

The solution that we have adopted to the first issue is to incorporate a version number in the client software . This is passed to the server on each server call and the first thing that each server does is to verify that the client version number is compatible with the server. The version number is nearly always the same as the major release of the software and hence this ensures that the client and server software is at the same release, for example 7.7.

The second issue is to allow fixes and minor updates to the software within a release that does not require a complete refresh of the client software on all user’s workstations.

To address this requirement, we do not make any changes to the server interface (import/export views, view properties, transaction codes, etc) within a major release that would require a new client version. This means that the server code (and client code if required) can be updated to the latest service pack or with emergency fixes without needing to re-distribute or update the client code.

If there is a real need for a new client/server flow within a release, this can be accommodated with a completely new server procedure step. Updated clients can access the new p-step, but existing clients will be unaware of it and hence continue to operate unaffected. With this approach, the only consideration is that the server code will need to be deployed before the client code.

Single Logon

When launching our products (GUI client/server or local windows based ‘fat clients’) the user has to logon by providing a userid and password. These are then used to validate that the user has access to the system and optionally are also provided to the Gen client/server runtime modules for authentication at the server level, for example, to RACF on the mainframe.

Since we have multiple applications available via a program folder or accessed via the Gen toolset plug-in menu, we wanted to avoid having to ask the user to logon each time they started a new application if another one was already open.

This meant that we needed to find a way of sharing the logon credentials. This method had to be secure and reliable, so storing the values in an environment variable or file was ruled out. The solution we adopted was to use a DLL that was shared by all of the clients and the user credentials stored in the DLL’s memory with encryption.

Normally if multiple executables share the same DLL, each instance will have its own private copy of the memory, thus precluding the sharing that we were trying to achieve. The solution was to use a shared memory segment in the DLL so that the contents of the shared segment could be accessed by all executables.

The result is that once the user has logged on to one client, then all subsequent clients do not ask for the userid and password. An additional benefit of the shared DLL is that when the last client is closed, the DLL is no longer resident in memory unlike using persistent storage methods.

Memory Usage Update

A previous post described some of the techniques that we have used to reduce the memory requirements of Gen applications. One of the approaches that we have been trialling is to delay the allocation of local view storage for generated C code.

The generated C code for an action block will define a structure for the local views and the memory for these structures will be allocated for all of the action blocks linked into the load module when the load module is loaded, even if the action blocks are not called. Note that this is only strictly true for action blocks that do not have any uninitialized local views.

We have developed a source code post-processor that re-defines the local view structure so that the memory is only allocated when the action block is called and is then freed on return from the action block.

This has dramatically reduced the memory usage of the load modules, especially for co-operative servers that remain loaded by the transaction enabler.

We have been running with the modified code for a month now with no unexpected results. The benefits are a big reduction in the overall memory requirement. This allows us to improve performance by a combination of having more load modules loaded into the TE thus reducing program load time, freeing up memory that can then be allocated to the DBMS and fewer page faults.

User Defined Functions

In my last post, I described how you can extend the Gen runtime with user defined functions.

The post described how the use of user defined functions can simplify action diagrams because you can nest functions and also the input and output parameters can be specified using any attribute view of the same domain, whereas imports and exports to action blocks have to use the same entity type or workset views.

It is worth pointing out that whilst this simplifies the action diagram code, the underlying generated code will still need to ‘map’ the input and output views to additional data items that have the same length as the function’s parameters, and hence the fact that the action diagram code appears not to have this extra ‘mapping’ does not mean that there isn’t some runtime overhead associated to this data mapping.

To add a function, you have to code the logic in the same way that you would for an external action block and place the code into a library so that it is available at link time. You also have to define the function to the model, specifying the input and output parameters. These are more restrictive than for a normal action block. You are limited to one output parameter and up to 5 input parameters, and group views are not allowed. Defining the function to the model is quite a complex process, especially if you are not familiar with the Gen meta-model, so you may want to take a look at the QAT Function Manager which automates much of the tedious tasks associated with defining the function. There are also various presentations and documents produced by CA on this subject, so I do not want to repeat them here.

Once you have added your function to the model and coded the logic in C, COBOL or your other target language, you are ready to start using it in the model.

Some of the functions that we have added are:

IET_FIND. This is a replacement for the standard Gen find function which allows you to search a string using wild cards (like %) and other search operators.

IET_GET_ROW_CLICKED. This is a function that returns the clicked row given an x & y co-ordinate returned by the Gen GUI right mouse functions and a list box control. Gen only provides the x & y co-ordinates and to get the row selected, you need to adjust for the scrolling of the list box, the font height and whether the user has changed the default DPI setting in Windows.

IETeGUI functions. For IETeGUI we have developed a range of functions that allow the developer to fine-tune the GUI at execution time, for example, dynamically changing a list box column position, visibility, headings, etc. Using functions greatly simplies the coding when you want to provide detailed control over the GUI runtime.

Extending the Gen Runtime

I thought it would be worthwhile describing how we have extended the Gen runtime in the development of our tools, but first we should clarify what we mean by the ‘runtime’.

A high level language (including Gen, but also languages like C, COBOL, Java, .Net, etc,) simplifies the developer’s task by providing functions that are inherent in the programming language. Examples include printf in C, STRING in COBOL and the Gen concat function.

The actual implementation of the function by the compiler (or code generator) might be via a macro, code generation (i.e. the COBOL or C compiler generating assembler instructions to implement the statement) or via a call to a language runtime module. To the developer, it does not matter how the function is implemented. For example, the CA Gen concat function is implemented in generated COBOL code whereas in C it is implemented via a Gen runtime function call.

So what is the difference between a language’s runtime functions and those that you might procure from a 3rd party vendor or develop in-house?

From a developer’s perspective, in some languages (like C) the code to use a language function will not differ much from the code to use a 3rd party function, whereas in others (like COBOL), the in-built functions are typically referenced via language statements (i.e. STRING) whereas other functions would be used via a CALL statement. Typically a function that is invoked via simple language statements is easier to use than having to implement a separate program call to invoke the function.

In CA Gen action diagramming, the same is true. Compare the following:

SET c to numtext(substr(concat(a,b),1,5))

With the following which uses alternative routines for numtext, substr and concat.

SET map_1 TO a
SET map_2 TO b
USE my_concat
WHICH IMPORTS: map_1, map_2
WHICH EXPORTS: map_3
SET map_4 TO map_3
SET start to 1
SET end to 5
USE my_substr
WHICH IMPORTS map_4,start,end
WHICH EXPORTS map_5
SET map_6 to map_5
USE my_numtext
WHICH IMPORTS map_6
WHICH EXPORTS map_7
SET c to map_7

In the first example, one statement could achieve the desired result, whereas if we replace the use of in-built functions with external action blocks that achieve the same result, we need a lot of additional code as well as special ‘mapper’ views that are of the same type as the import and export views of the EABs.

In the above example, I have shown an extreme case where each function has its own set of import and export vies that are not shared between the functions.

Gen allows you to develop your own functions, so you could then code the above as:

SET c to my_numtext(my_substr(my_concat(a,b),1,5))

We have made use of this capability to add our own functions into the Gen model and hence simplify our code, both in terms of the number of statements needed to implement a given task, but also reducing the number of ‘mapping’ views needed.

In a subsequent posting, I will describe some of the functions that we have developed.

Windows Timers in Gen GUI

Watching paint dry is not everyone’s idea of an interesting day at work, and for certain applications this is extra tedius when you have to click a ‘refresh’ button just to see how dry that paint has become.

Various functions in our applications submit background tasks and exhibit this mindless click-to-see-if-it-has-finished scenario. After much RSI and boredom we decided it was time to implement auto-refresh and timer events in the GUI so that at least you didn’t have to click the tired old mouse so much.

To achieve this in our Gen GUI we developed an EAB which we call to register and start a timer event in Windows (WM_TIMER messages). Basically this is a Windows event interception routine which processes the timer events we are interested in, and any other events get passed on to the default window event handler (the Gen GUI & Windows runtimes).

To make something actually ‘happen’ when the WM_TIMER event is received, TIREVENT is used to trigger an event in the client PStep, and the timer event is deactivated again, as far as Windows is concerned.

The event in the PStep can then do its' thing, and if it wants to repeat the timer again (like a regular auto-refresh) it simply calls the EAB again telling it to reactivate the timer.

Pretty simple stuff in principle, but a big improvement for the poor user and their aching click-finger.

Tabs Controls in Gen GUI – an easier approach

Let’s face it, designing neat but sophisticated applications using Gen’s GUI can be challenging. The temptation to resort to OCX controls to get a good-looking app is high, but at the expense of difficult and cumbersome coding in your client PStep.

In particular, a user-friendly way to simplify groups of activities or information in a GUI would be to use a Tab control. If you were coding any other language (VB, C++ etc.) this would be an easy choice, but in Gen the result is a huge client PStep which contains all the logic for every tab page.

Better would be to split each tab page’s logic into a separate client PStep, but this simply is not possible using OCX controls in Gen.

To get the look and feel of a tab control, we’ve developed some cheats which allow exactly this… very simply, each tab IS a separate client PStep, but the tabs are actually a combination of buttons and bitmaps, designed to look like tabs!

To reduce the complexity of the Dialog Flows required, a central non-screened PStep is created which has the Dialog Flows (links) from itself to each of the tab PSteps. There are no flows between the tabs themselves. If a tab button is pressed, it simply flows back to the non-screened PStep with instructions on where to flow to next.

There are of course a few tricks to make this look slicker, including positioning and sizing of the individual tab windows, to give the impression that they are in fact the same window. Originally we achieved this by simply getting the window x,y,w,h and passing it around on the flows, but now we take advantage of IETeGUI’s resize features specifically for this purpose.

Find out more about IETeGUI here... http://www.iet.co.uk/Products/IETeGUI

Also, the tab buttons tend to ‘float’ around a bit, compared to the designed position in the Gen Toolset meaning that the buttons wouldn't always be perfectly lined up, so we put logic into each client PStep to explicitly place them at specific positions so they line up nicely. Again, we now make use of some automatic functions in IETeGUI to help with this.

Another aspect of the illusion of tabs is the border of the tab control. Our original implementation simply used carefully designed background bitmaps for those windows, and the tab buttons were placed so that they sat along the top border of the tab. If your windows are not resizeable, then this works perfectly. However, for resizeable windows, you’ll want to resize the tab’s border as the window gets resized. To achieve this we originally wrote some C code which intercepted the window’s redraw/resize events and drew the border dynamically, but now use some built-in features of IETeGUI to achieve much the same thing.

Avoiding too much logic in Window Open events

When flowing between windows in a Gen GUI application, it is good to have the window pop-up quickly, even if the data is not populated immediately, as this gives the user the feeling that something is happening.

If you put too much logic inside the Open event of the window, this perception changes because the window does not get physically displayed until the end of the Open event, thus giving the illusion that the application has hung, crashed or simply disappeared.

We’ve found that a good practise is to minimise the amount of logic (especially server calls) performed in the Open event, and if necessary use a TIREVENT within the Open event to trigger another event where the longer processing is performed. The window will then get displayed after the Open event so the user can at least see it is alive.

Coping with the 32k view size limit

You are likely to have come across the 32k view size limit for procedure step import and export views unless you are generating for platforms that do not have the limit like pure windows clients, Java or .Net.

When the user needs to scroll through simple lists of data, a paging method that uses a ‘start from’ field is usually adequate, and if you want to display a lot of data on the client, you can still repeatedly call the server to retrieve batches of data, again using a hidden ‘start from’ field.

However there are occasionally situations where the above approach does not work. Examples that we have come across include impact analysis on the server which returns a set of object ids based on a complex trawl through the database. You need to return the complete set of data and the 32k view size limit can then result in truncation of the results.

We have therefore developed some techniques for handling this.

The first technique is to limit the export view to the identifiers of the returned rows. You can then call a different server to retrieve additional details, for example, the first server returns 1000 identifiers and then a second server is used to retrieve the additional details in batches of 100.

The second technique is to reduce the number of attribute views in the export view. For each attribute in a view, Gen adds 1 byte for the state flag and for generated C code there is an additional null terminator byte. Gen also adds an additional 6 bytes for each attribute in the export view. Consider the situation where you need to return ten 1 byte attributes in a group view. For C code, each row in the group view will occupy 90 bytes. However if you were to string the ten bytes together into a single 10 byte field, then each row would occupy 18 bytes, thus allowing a much larger group view size.

The third technique is to first call the server to populate the result set and return the first batch of data. Any remaining data is stored in a temporary location, for example a file. If the results set is incomplete, the client can then call the server again repeatedly to retrieve the remainder of the data from the temporary location.

Memory Usage and OS differences

The work that we have done so far on reducing memory usage has concentrated on the windows platform, but we also run our products on UNIX (AIX & HP) and z/OS.

Each platform is different with respect to when memory is allocated, so an understanding of this is important when trying to reduce memory usage, especially if your code will operate on different platforms.

The examples that we are using relate to generated C or COBOL code and not other platforms like Java or .Net.

For C code, the storage for local views is allocated on initial load of the executable. For static modules this will be when the executable is loaded, and for DLLs, this will also be when the executable is invoked. However, once it has been invoked, the memory will remain allocated until the executable terminates. For an executable loaded by the Transaction Enabler (TE), this might be some time!

You can avoid loading a DLL that might not be invoked for each execution with the /DELAYLOAD linker option on Windows and equivalent options for UNIX (e.g. -blazy on AIX). We are experimenting with this and will report back on findings in a later posting, but so far we have seen a benefit of Windows without a noticeable overhead. The main benefit on Windows is that we can now keep many more executables loaded in the TE and thus do not have the overhead of these being swapped in and out.

On z/OS, the COBOL working storage is allocated from the heap for re-entrant code. (Normally Gen programs are compiled with the RENT option). This means that the working storage is allocated on first execution of the action block and then not freed until the executable terminates. If you have an executable with a lot of action blocks that are rarely invoked, then they will not present a memory overhead compared with C code where the storage for all action blocks will be allocated on initial load of the executable.

Memory Usage Caveat

In the Mythical Man Month, Fred Brooks provides an example of how a developer 'wasted' a few bytes of storage in the OS/360 control program by putting in a routine to handle leap years. In his view this was not necessary since the operator could reset the date once every 4 years. In those days memory was very expensive, and one of the main ways of charging for the machine, so freeing up a few bytes to the application program was seen as a good thing.

We are now used to machines with many gigabytes of memory and so you may be wondering why this blog started with a post on reducing memory usage, especially since memory is cheap and developers are expensive.

Let me start by stating that I am not advocating spending a lot of precious time trying to save a few bytes. In our case, because of large group views, large number of called action blocks and reuse of code in shared libraries, some of our server load modules were requiring over 100Mb. If you then have 20 of these loaded by the TE, you are now using 2Gb.

Paging memory to disk or swapping load modules in and out of the TE has a performance impact, so our thinking is that if you can reduce the memory usage of the application, then you can keep more load modules loaded and/or give more memory to the DBMS, which will be able to benefit from it and hence improve performance.

A second point is that most of the techniques that we are using are highly automated. We have developed tools to search for and remove unused views, and report on imperfect view matching (optionally only for repeated calls or those involving group views). There is therefore a very low overhead in implementing these good practices.

You should however consider the cost/benefit case for any coding standards that you adopt, and spending a lot of time trying to save a few bytes (or even a few Mb) is unlikely to be justifiable.

Memory Usage

Reducing the memory usage of our load modules is something that we are working on at present, so to kick off the blog with a very technical subject...

Much of the code that we develop relies on repeating group views to process lists of objects.

Group views in Gen need to have a fixed, maximum size, and memory for the group view is allocated at program load time. This means that there is a trade-off between the memory usage of a load module and the limits imposed by the group view size. In many cases you can size the group view to cater for the maximum number of rows that are needed, or repeat processing with some sort of 'start from' value.

However there are quite a few instances where this is not possible, and then the group view size becomes a fixed limit on the data that can be processed by the function.

Over the years we have increased these limits based on customer requirements, and this has resulted in an increased memory requirement for the load modules. This is particularly noticeable in Windows and UNIX environments where the server load modules are kept in memory by the Transaction Enabler. (Note that there are tuning parameters for the number of load modules that are loaded by the TE).

We have also made more use of DLLs in Windows and UNIX (Operations Libraries) and these can also increase the memory requirement if they result in a load module ending up loading a DLL for one action block and therefore also allocating the memory for all of the other action blocks in the opslib and any dependent opslibs. Our models have a lot of shared and reused code, and complex dependencies between modules has meant that many of the DLLs are loaded multiple times because of code dependencies, even if they are not required for each execution.

We are therefore working on some strategies to reduce memory usage. Ones that we have come up with so far include:

1) Delete unused views and attributes
2) Improve use of perfect view matching
3) Reduce use of uninitialised local views
4) Share group views between action blocks
5) Delay loading of DLLs
6) Delay allocation of local view storage

The first four are all standard coding techniques.

1) Maintaining an action block over many years (some of our action blocks are now 20 years old) can result in quite a few unused views. Finding and deleting these can be tedious and time-consuming, so we developed some tools in VerifIEr and genIE to automate finding and then deleting the unused views and also unused attribute views.

2) If view matching is not perfect, extra working storage is required in the generated code, so improving view matching (especially for large group views) will help.

3) In C generated code, having even one uninitialised local view will double the memory usage of all of the local views, so avoid this if you can.

4) An example of sharing memory is where many action blocks require a large local group view with the same structure. For example, we have a case where many action blocks need a 1Mb group view and each is called from the same parent. If each action block has its own local view, then each AB needs 1Mb of storage. However if the group view is defined as an exportable import view and is passed from a local view in the parent, then they all share the same storage. You then need to ensure that the view is properly initialised, but there is a big memory saving. In one load module we managed to halve the memory usage using this technique.

5) On windows we are experimenting with using the /DELAYLOAD: linker option to delay loading of opslibs. If the route through the code never requires a function in the opslib, it will not get loaded.

6) This is very much work in progress, so more on this later...

Welcome

Welcome to the IET Gen Development Blog. I decided to start this blog to share some of the tips and techniques that we have learnt at IET in developing our products with CA Gen. We are passionate about CA Gen and the main aim of this blog is to help other Gen users learn from our experiences (and mistakes!).

The blog is likely to be composed of random information and is not designed to be a structured training course in Gen development!